https://developer.hpe.comGatsbyJSTue, 28 Apr 2026 17:02:36 GMThttps://developer.hpe.com/Implementing-a-complete-Workshop-on-Demand-infrastructure-in-less-than-an-hour/https://developer.hpe.com/Implementing-a-complete-Workshop-on-Demand-infrastructure-in-less-than-an-hour/<h2>Implementing a complete Workshop-on-Demand infrastructure in less than an hour</h2> <p>December 17, 2025</p> <p>Join our next meetup session to witness how quick and easy it can be to deploy a Workshop-on-Demand infrastructure so that you can deliver your own on-demand training sessions. Bruno Cornec (a former HPE Linux Distinguished Technologist) and Frederic Passeron from HPE Developer Community will demonstrate how you can achieve this goal.</p>https://developer.hpe.com/Network-operations-spotlight-with-pycentralv2/https://developer.hpe.com/Network-operations-spotlight-with-pycentralv2/<h2>Network Operations Spotlight with PyCentralv2</h2> <p>April 08, 2026</p> <p>Return for our second session of the HPE Networking Automation Team’s Developer Meetup series! This session will focus on the Automation Team’s PyCentral software development kit (SDK) for HPE Aruba Networking Central. PyCentral is the Automation Team’s very own Python SDK built to integrate automation with network operations. PyCentral handles authentication, request formatting, and error handling with PE Aruba Networking Central's Rest APIs while exposing simple Python functions. The SDK has received numerous enhancements including the integration a full suite of support materials for configuration, monitoring, troubleshooting, and streaming. Follow along as the Automation team guides you through a series of demos showcasing the power of automating network operations with PyCentral.</p>https://developer.hpe.com/accelerate-public-sector-ai-use-cases-using-a-powerful-ml-ops-platform/https://developer.hpe.com/accelerate-public-sector-ai-use-cases-using-a-powerful-ml-ops-platform/<h2>Accelerate public sector AI use cases using a powerful ML Ops platform</h2> <p>September 21, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, learn how the French governmental agency, Pole Emploi, is collaborating with HPE to build and implement an effective and scalable AI/ML Ops platform that helps companies find and hire workers. Hear from Pole Emploi Software Engineer, François Réthoré, and HPE Software Engineer, Dietrich Zinsou, as they explore different uses cases where this AI/ML Ops platform has been put into action.</p>https://developer.hpe.com/ai-inferencing-at-the-edge-use-cases-from-earth-to-space/https://developer.hpe.com/ai-inferencing-at-the-edge-use-cases-from-earth-to-space/<h2>AI Inferencing at the Edge: Use Cases from Earth to Space</h2> <p>February 12, 2025</p> <p>High-stake operating environments can add additional challenges when it comes to implementing AI technologies. Whether your use case is focused on speeding up time-to-target while maintaining data security or handling rugged and remote computing environments outside your data center, there are important considerations you need to take into account. Join us in our next session as we explore public sector and space-deployed AI-use cases, spanning the use of RAG for knowledge retrieval to the use of computer vision for image identification.</p>https://developer.hpe.com/accelerating-scientific-research-through-high-performance-computing-democratization/https://developer.hpe.com/accelerating-scientific-research-through-high-performance-computing-democratization/<h2>Accelerating scientific research through high performance computing democratization</h2> <p>February 15, 2023</p> <p>Large, complex computer simulations of the physical world are becoming more common-place thanks to the broader availability of high performance computing resources. Contemporaneously, the advent of open-source software, data science libraries, and machine-learning frameworks mean that some aspect of software engineering is now required of domain scientists. In this session, you’ll learn how these advances necessitate collaborations amongst software engineers, data scientists, and domain scientists and explore a specific use case delivering the first realistic global simulations of the world’s oceans.</p>https://developer.hpe.com/apis-for-hpe-greenlake-for-block-storage-and-next-gen-platforms/https://developer.hpe.com/apis-for-hpe-greenlake-for-block-storage-and-next-gen-platforms/<h2>APIs for HPE GreenLake for block storage and next-gen platforms</h2> <p>March 27, 2024</p> <p>Join us for an introduction to HPE GreenLake for block storage automation using Data Services Cloud Console APIs. This session will guide you through the first steps of automating your processes using Data Services Cloud Console. The APIs covered in this talk will enable HPE GreenLake for block storage management automation using scripts as well as how they can be integrated into custom-built applications.</p>https://developer.hpe.com/automate-what-happens-next-using-hpe-opsramp-process-automation/https://developer.hpe.com/automate-what-happens-next-using-hpe-opsramp-process-automation/<h2>Automate what happens next using HPE OpsRamp Process Automation</h2> <p>May 20, 2026</p> <p>Join our next HPE Developer Community Meetup to learn more about HPE OpsRamp Process Automation. Process Automation enables you to set up and automatically run tasks related to platform events in the system. Use it to trigger workflows on signals, timers, or events to save time and reduce manual operations. It can also be used as a scheduler platform for customers who need to configure tasks that run on various schedules. In the session, you’ll have the opportunity to understand HPE OpsRamp Process Automation implementation and usecase walkthrough. We look forward to seeing you there!</p>https://developer.hpe.com/automated-continuous-deployment-of-container-based-applications-onto-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/automated-continuous-deployment-of-container-based-applications-onto-hpe-greenlake-for-private-cloud-enterprise/<h2>Automated continuous deployment of container-based applications onto HPE GreenLake for Private Cloud Enterprise</h2> <p>November 29, 2023</p> <p>You need to develop and deploy applications faster. Key to this is having an automated mechanism that takes care of application deployment, allowing you to focus your time on actual application development. With HPE GreenLake for Private Cloud Enterprise, you get built-in workflows that enable you to build and deliver applications with agility, reducing the need to worry about continuous deployment and allowing you to shift your focus to application development and business outcomes. With these DevOps CD pipelines, you can deliver applications onto your private cloud faster and in a continuous and automated fashion.</p>https://developer.hpe.com/boost-spark-ai-workloads-with-pepperdata/https://developer.hpe.com/boost-spark-ai-workloads-with-pepperdata/<h2>Boost Spark AI workloads with Pepperdata</h2> <p>October 26, 2022</p> <p>HPE combines the power and versatility of Apache Spark with the robust, enterprise-grade HPE Ezmeral Runtime Enterprise to support running analytics at scale against large data sources. But what do you do once adoption scales to the point where dozens or hundreds of data scientists and data analysts are running massive amounts of Spark applications? In this talk, we’ll learn how HPE is partnering with Pepperdata to bring detailed observability via Pepperdata’s Platform and Application Spotlight, and more importantly, deliver automated, near real-time, autonomous optimization of cluster container resources via Pepperdata’s Capacity Optimizer, without the need for Spark developers to change a single line of code.</p>https://developer.hpe.com/calling-all-citizen-developers-can-low-code-platforms-accelerate-your-impact/https://developer.hpe.com/calling-all-citizen-developers-can-low-code-platforms-accelerate-your-impact/<h2>Calling all citizen developers: Can low-code platforms accelerate your impact?</h2> <p>November 16, 2022</p> <p>Across industries, low-code citizen development platforms are unleashing creative talent to solve problems. In this session, HPE colleagues from the Office of the CTO, Education Services, and HR will provide an introduction into this powerful technology and how they have been able to build award-winning and enterprise-grade apps and automations—often with little-to-no budget. Even full-time developers can benefit from learning about low-code citizen development platforms to create “full stack” solutions outside of their areas of expertise.</p>https://developer.hpe.com/chapelcon-24/https://developer.hpe.com/chapelcon-24/<h2>ChapelCon '24 - The Chapel Event of the Year</h2> <p>June 5-7, 2024</p> <p>ChapelCon ’24 welcomes anyone with computing challenges that demand performance, particularly through parallelism and scalability. Our open-source community includes those interested in parallel programming, programming languages, and high-performance computing. Building on 10 years of experience from CHIUW (Chapel Implementers and Users Workshop), ChapelCon '24 brings together Chapel users, enthusiasts, researchers, and developers to exchange ideas, present their work, and forge new collaborations.</p>https://developer.hpe.com/chathpe-hub-enabling-secure-and-scalable-ai-transformation-at-hpe/https://developer.hpe.com/chathpe-hub-enabling-secure-and-scalable-ai-transformation-at-hpe/<h2>ChatHPE Hub: Enabling Secure and Scalable AI Transformation at HPE</h2> <p>March 19, 2025</p> <p>Join us to learn more about an internal project HPE has developed using HPE Private Cloud AI that helps its teams use Generative AI (GenAI) to achieve its business goals. In this session, we will introduce you to the ChatHPE GenAI Hub – a revolutionary platform from HPE IT that provides secure, private access to large language models (LLMs) for HPE organizations. The ChatHPE GenAI Hub empowers employees to quickly develop and deploy impactful use cases, accelerating innovation and delivering transformative insights across the organization.</p>https://developer.hpe.com/collaborating-at-scale-with-postman/https://developer.hpe.com/collaborating-at-scale-with-postman/<h2>Collaborating at scale with Postman</h2> <p>March 29, 2023</p> <p>Having evolved from everyone’s favorite desktop API endpoint testing utility to a true enterprise API development platform, Postman’s recently released version 10 now delivers a broad range of new enterprise features. In this session, you’ll learn how Postman can help include the broadest possible spectrum of stakeholders involved in API development throughout the entire API lifecycle, including development, testing, deployment and management.</p>https://developer.hpe.com/cool-ai-use-case-now-what/https://developer.hpe.com/cool-ai-use-case-now-what/<h2>Cool AI Use Case, Now What?</h2> <p>October 2, 2024</p> <p>You’ve come up with a great use case where AI technologies can help your business. Now your leadership is saying “show me something.” Join this session to understand best practices on getting started with AI and learn some shortcuts from the experts on how to get AI projects from data experiments to AI deployments. Begin your journey: Grab your toolbox, work with your data experts, build some results, and then amplify success on what you have accomplished.</p>https://developer.hpe.com/dayn-a-new-way-to-look-at-observability/https://developer.hpe.com/dayn-a-new-way-to-look-at-observability/<h2>DayN+ - A new way to look at observability</h2> <p>April 16, 2025</p> <p>Traditional monitoring systems focus mainly on metrics, predefined thresholds, and manual log analysis to monitor system and application health. With the integration of cloud technologies, IT environments have become significantly more distributed and complex. This complexity reduces visibility into resources, forcing IT operations into an inefficient, reactive problem-solving mode. This situation has led to the emergence of observability. Join this session to learn how an effective observability practice integrates people, processes, and tools to foster collaboration, break down silos, and enable proactive detection, diagnosis, and resolution of issues.</p>https://developer.hpe.com/developing-and-deploying-ai-in-the-enterprise/https://developer.hpe.com/developing-and-deploying-ai-in-the-enterprise/<h2>Developing and deploying AI in the enterprise</h2> <p>April 23, 2025</p> <p>Integrating the right software into your enterprise AI infrastructure is crucial to deploying AI applications successfully today. In this session, we'll discuss the ever-changing developer ecosystem, how machine learning workflows have evolved, and how HPE Private Cloud AI can give your teams the ability to move more quickly. Lastly, we will show how to create agentic workflows using tools like Langflow and explore recent advancements in the ecosystem with ready-to-deploy blueprints.</p>https://developer.hpe.com/determined-ai-model-training-hackathon/https://developer.hpe.com/determined-ai-model-training-hackathon/<h2>Determined AI Model Training Hackathon</h2> <p>February 20, 2023 - April 17, 2023</p> <p>The Determined Community Team is running a hackathon! Build and train an ML model using Determined to win cash prizes.</p>https://developer.hpe.com/devops-alert-tool-sprawl-complexity-burnout-help-1/https://developer.hpe.com/devops-alert-tool-sprawl-complexity-burnout-help-1/<h2><strong>Going from containers, to pods, to Kubernetes – help for your developer environments!</strong> </h2> <p>October 25, 2023</p> <p>Today, Kubernetes is the undisputed go-to platform for scaling containers. But for developers, Kubernetes can be daunting, particularly when working with the discrepancies between local and production environments. In this talk, learn how Podman and Podman Desktop bridges this gap, serving as a beginner-friendly launch pad to Kubernetes. Get a demo and learn how you can benefit to streamline your container development processes!</p>https://developer.hpe.com/devops-alert-tool-sprawl-complexity-burnout-help/https://developer.hpe.com/devops-alert-tool-sprawl-complexity-burnout-help/<h2>DevOps Alert: Tool Sprawl. Complexity. Burnout. Help!</h2> <p>September 27, 2023</p> <p>Today’s DevOps teams are finding it even more difficult to develop and deploy features to meet customers’ expectations, even with the ubiquity of free and “easy-to-implement” tools at their disposal. In this session, learn how to address DevOps challenges like tool sprawl and managing Kubernetes application operations across hybrid and multi-cloud landscapes through a centralized GitOps-centric approach.</p>https://developer.hpe.com/digital-twins-the-metaverse-and-augmented-reality-developer-insights-and-it-foundations-for-immersive-technologies-powered-by-ai/https://developer.hpe.com/digital-twins-the-metaverse-and-augmented-reality-developer-insights-and-it-foundations-for-immersive-technologies-powered-by-ai/<h2>Digital twins, the Metaverse, and augmented reality: Developer insights and IT foundations for immersive technologies powered by AI</h2> <p>June 14, 2023</p> <p>In this Munch &#x26; Learn session, learn about digital twins, the metaverse, and augmented reality, including their application in industries, the role of AI, and developer factors. You’ll get to view real-world examples showing how they can transform industries and create amazing immersive experiences.</p>https://developer.hpe.com/divide-and-conquer-with-microfrontends/https://developer.hpe.com/divide-and-conquer-with-microfrontends/<h2>Divide and conquer with MicroFrontends</h2> <p>April 26, 2023</p> <p>Micro Frontend, a design paradigm that extends the concepts of micro services to the frontend world, helps scale team development processes to deliver complex frontends. It allows teams to develop multiple micro frontends in parallel, helping build modern complex applications and empowering users to transform legacy apps in a gradual manner by rebuilding the app in parts. Join us to hear Vishal Sharma discuss using micro frontends and how to best split teams to achieve the desired results.</p>https://developer.hpe.com/embracing-large-language-model-tools-like-chatgpt-for-iterative-problem-solving/https://developer.hpe.com/embracing-large-language-model-tools-like-chatgpt-for-iterative-problem-solving/<h2>Embracing large language model tools like ChatGPT for iterative problem solving</h2> <p>May 24, 2023</p> <p>Learn how coders and project architects alike are using Large Language Model tools like ChatGPT to transform software development. Tailored for those with some developer experience, this Munch &#x26; Learn session explores the highly iterative nature of these tools and their potential to serve as your 24/7 developer assistant.</p>https://developer.hpe.com/enabling-business-automation-using-hpe-greenlake-platform-foundational-apis/https://developer.hpe.com/enabling-business-automation-using-hpe-greenlake-platform-foundational-apis/<h2>Enabling business automation using HPE GreenLake platform foundational APIs</h2> <p>March 27, 2024</p> <p>The HPE GreenLake edge-to-cloud platform, now offers application programming interfaces (APIs) for many common services, such as workspace management, identity and access management, device and subscription, locations, audit logs, and wellness. In this session, explore using the HPE GreenLake Developer Portal to access them and see how to invoke these APIs using an OpenAPI tool, such as Postman. We’ll demonstrate this using some of the most popular scripting languages (i.e. Bash, PowerShell and Python).</p>https://developer.hpe.com/engage-with-the-hpe-dev-team-at-hpe-technology-and-solution-summit-2022/https://developer.hpe.com/engage-with-the-hpe-dev-team-at-hpe-technology-and-solution-summit-2022/<h2>HPE Technology and Solutions Summit 2022</h2> <p>March 28-31, 2022</p> <p>HPE TSS 2022 helps build a collaborative, open discussion where presales consultants and solution architects from the HPE and HPE partner presales community can each learn from each other to address today’s challenges.</p>https://developer.hpe.com/enhancing-nlp-with-retrieval-augmented-generation-a-practical-demonstration/https://developer.hpe.com/enhancing-nlp-with-retrieval-augmented-generation-a-practical-demonstration/<h2>Enhancing NLP with Retrieval-Augmented Generation: A Practical Demonstration</h2> <p>September 18, 2024</p> <p>In the evolving landscape of Natural Language Processing (NLP), Retrieval-Augmented Generation (RAG) stands out as a powerful technique that enhances the NLP application by incorporating relevant external information. In this session, we will delve into the fundamentals and applications of RAG, providing a comprehensive overview of how it integrates retrieval mechanisms with generative capabilities to produce more accurate and contextually aware responses. We will then transition into a live demonstration showcasing the practical usage of RAG and the process of augmenting a generative model with external knowledge sources, showcasing how RAG improves the relevance and quality of generated outputs in real-time applications.</p>https://developer.hpe.com/exploring-hpe-greenlake-platform-apis-through-use-cases/https://developer.hpe.com/exploring-hpe-greenlake-platform-apis-through-use-cases/<h2>Exploring HPE GreenLake Platform APIs through use cases</h2> <p>December 13, 2023</p> <p>The HPE GreenLake edge-to-cloud platform empowers businesses to efficiently manage a diverse set of IT resources no matter their location. Working with the platform is made simple though its collection of RESTful APIs, which enable developers to interact with and manage different aspects of the HPE GreenLake platform. Join this session to learn how these APIs will equip your business with the tools necessary for automating the seamless integration of HPE GreenLake services into your custom applications, streamlining processes, and enhancing control and visibility over your infrastructure.</p>https://developer.hpe.com/exploring-the-hpe-sustainability-insight-center-key-features-innovations-and-sic-api-capabilities/https://developer.hpe.com/exploring-the-hpe-sustainability-insight-center-key-features-innovations-and-sic-api-capabilities/<h2>Exploring the HPE Sustainability Insight Center: Key features, innovations, and API capabilities</h2> <p>September 25, 2024</p> <p>Discover how the HPE Sustainability Insight Center can optimize energy efficiency, reduce costs, and drive impactful sustainability initiatives for your customers' organizations. This session will highlight key product features, including a unified dashboard for energy and carbon emissions reporting, real-time insights for data-driven decisions, and seamless integration with HPE GreenLake. You will have the opportunity to learn about the SIC API, exploring its use to enhance sustainable IT solutions. Join us and take the first step toward a greener, more efficient future. Register now!</p>https://developer.hpe.com/exploring-transformative-ai-use-cases-across-industries/https://developer.hpe.com/exploring-transformative-ai-use-cases-across-industries/<h2>Exploring Transformative AI Use Cases Across Industries</h2> <p>August 21, 2024</p> <p>Artificial Intelligence (AI) is rapidly reshaping industries, driving innovation, and creating new opportunities for growth and efficiency. This panel discussion brings together thought leaders, industry experts, and technologists to explore transformative AI use cases across diverse sectors, including healthcare, finance, manufacturing, and retail. In this talk, panelists will share insights on the critical factors for AI adoption, including data management, ethical considerations, integration with existing systems, and emerging trends, such as the use of AI for predictive analytics, personalization, automation, and decision-making. Book your seat for this insightful session with our AI subject matter experts.</p>https://developer.hpe.com/extraordinary-claims-require-extraordinary-engineering-and-tamo-isn’t-an-option/https://developer.hpe.com/extraordinary-claims-require-extraordinary-engineering-and-tamo-isn’t-an-option/<h2>Extraordinary claims require extraordinary engineering (and TAMO isn’t an option)</h2> <p>March 15, 2023</p> <p>So many challenges we face as a species demand an unprecedented ability to simulate and engineer at the scale where quantum mechanics dominate. In this session, Kirk Bresniker of Hewlett Packard Labs will give us his perspective on moving from quantum computing research to productization. His thoughts on approaching quantum computing development as a global community from the application down to the Qubits, as opposed to the other direction, offer insights on how to further its advancement.</p>https://developer.hpe.com/finding-vulnerabilities-in-production-with-open-source-threatmapper/https://developer.hpe.com/finding-vulnerabilities-in-production-with-open-source-threatmapper/<h2>Finding vulnerabilities in production with open source ThreatMapper</h2> <p>July 27, 2022</p> <p>In this talk, we’ll explore how open source Deepfence ThreatMapper discovers and visualizes the external and internal attack surface for your applications and infrastructure. See how it extends the goodness of Shift Left into your production platforms in our Kubernetes cluster demo as it scans for vulnerabilities and exposed secrets to identify the greatest risks.</p>https://developer.hpe.com/from-log-files-to-ai-insights-the-60-year-evolution-of-observability-and-aiops/https://developer.hpe.com/from-log-files-to-ai-insights-the-60-year-evolution-of-observability-and-aiops/<h2>From log files to AI insights: The 60-year evolution of observability and AIOps</h2> <p>January 15, 2025</p> <p>In an era where systems are increasingly complex and interconnected, the ability to monitor, understand, and optimize your IT landscape has evolved dramatically. This session takes you on a 60-year journey through the evolution of observability and the rise of Artificial Intelligence for IT Operations (AIOps).</p>https://developer.hpe.com/galadriel-an-alternative-approach-to-spire-federation/https://developer.hpe.com/galadriel-an-alternative-approach-to-spire-federation/<h2>Galadriel - An alternative approach to SPIRE federation</h2> <p>February 22, 2023</p> <p>SPIFFE and SPIRE contribute to strong identification and attestation of workloads in cloud native environments. A more scalable alternative to its current federation method, Galadriel, proposes to facilitate SPIRE federation of multiple trust domains via a central exchange hub. Join this Meetup to learn more about some of the limitations of the current federation method and how Galadriel’s architecture aims to address these, providing a simpler, more scalable solution.</p>https://developer.hpe.com/getting-started-with-aruba-central-automation/https://developer.hpe.com/getting-started-with-aruba-central-automation/<h2>Getting Started with Aruba Central automation</h2> <p>February 28, 2024</p> <p>Learn how to configure, manage, and retrieve information using the various automation interfaces of Aruba Central in this session. We’ll explain how to access and utilize the REST APIs using the built-in user interface (UI) and Postman, how to manage the webhooks, and how to view the data from the streaming APIs. These areas will be showcased in demos so that the audience can recreate the capabilities in their own environments.</p>https://developer.hpe.com/getting-started-with-hpe-greenlake-for-compute-ops-management-apis/https://developer.hpe.com/getting-started-with-hpe-greenlake-for-compute-ops-management-apis/<h2>Getting started with HPE GreenLake for Compute Ops Management APIs</h2> <p>January 31, 2024</p> <p>Join this session to learn how to take advantage of HPE GreenLake for Compute Ops Management capabilities, simplifying and unifying operations across the server lifecycle for your whole environment, no matter where your compute infrastructure lies. See how it provides a consistent, secure cloud experience that scales elastically and unifies your compute management programmatically.</p>https://developer.hpe.com/how-digital-twins-help-companies-reach-their-sustainability-goals/https://developer.hpe.com/how-digital-twins-help-companies-reach-their-sustainability-goals/<h2>How digital twins help companies reach their sustainability goals</h2> <p>June 12, 2024</p> <p>As virtual replicas of physical systems, digital twins, help companies reach their sustainability goals by optimizing resource usage, reducing waste, and improving efficiency across the datacenter - from infrastructure to workloads. They enable predictive maintenance, energy conservation, and sustainable design, paving the way to optimize usage. In this session, we’ll discuss about some of the on-going projects inside of HPE Labs and the Technology Incubations teams that investigate optimization and efficiency prediction for datacenters and associated workloads.</p>https://developer.hpe.com/how-to-fix-your-biggest-security-hole/https://developer.hpe.com/how-to-fix-your-biggest-security-hole/<h2>How to fix your biggest security hole</h2> <p>November 20, 2024</p> <p>Are your users doing everything right to keep your systems secure? What we need is an access control system that is understandable, works on premises, in multiple clouds, object stores, file systems, relational databases and specialized hardware. This access control system must be able to be deployed incrementally, integrate with existing systems, simplify audits, and support formal verification of compliance. This session will describe how you can have all of that in a simple and understandable access control system.</p>https://developer.hpe.com/hpe-ai-developer-day-bogota-colombia/https://developer.hpe.com/hpe-ai-developer-day-bogota-colombia/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Crea tu propio AI Agent con HPE &#x26; Nvidia</strong></h2> <h3>Bogota, Colombia</h3> <p>February 24, 2026</p> <p>Únase a nosotros en un taller práctico y envolvente en el que aprenderá a diseñar e implementar casos de uso de IA generativa del mundo real en HPE Private Cloud AI, con tecnología NVIDIA AI Enterprise Software. Este taller está diseñado para desarrolladores, científicos de datos e innovadores empresariales que deseen ver cómo se crean, escalan y aplican soluciones de IA de nivel empresarial para resolver retos empresariales reales.</p> <h6><a href="https://events.bizzabo.com/797906/page/5472192/columbia-information">Regístrate</a></h6>https://developer.hpe.com/hpe-ai-developer-day-monterrey-mexico/https://developer.hpe.com/hpe-ai-developer-day-monterrey-mexico/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Crea tu propio AI Agent con HPE &#x26; Nvidia</strong></h2> <h3>Monterrey, Mexico</h3> <p>January 29, 2026</p> <p>Sumérgete en una exploración práctica de HPE AI Essentials y NVIDIA AI Enterprise Software, donde adquirirás experiencia práctica con herramientas y marcos de trabajo de vanguardia. Empieza por aprender sobre el tecnol stack tecnológico y recorrer el entorno de desarrollo para comprender sus capacidades. En esta sesión utilizaremos un caso de uso de gestión de retrasos de vuelos como ejemplo. Sin embargo, con estos conocimientos podrás desarrollar tus propios agentes de AI para resolver tu caso de negocio.</p> <h6><a href="https://events.bizzabo.com/797906/page/5472191/monterrey-information">Regístrate</a></h6>https://developer.hpe.com/hpe-ai-developer-day-paris-france/https://developer.hpe.com/hpe-ai-developer-day-paris-france/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Créez votre propre agent AI avec HPE et Nvidia</strong></h2> <h3>Paris, France</h3> <p>Le 10 Mars 2026</p> <p>Construisez et déployez une véritable solution d'IA dans le cadre d'un atelier en petit groupe animé par des conseillers HPE et NVIDIA, et repartez avec les outils, les compétences et la confiance nécessaires pour renouveler l'expérience. HPE Developer Day est un événement d'une journée conçu pour vous aider à passer de vos ambitions en matière d'IA à des résultats concrets.</p> <h6><a href="https://events.bizzabo.com/797906/page/5526657/paris-information">Inscription</a></h6>https://developer.hpe.com/hpe-ai-developer-day-stockholm-sweeden/https://developer.hpe.com/hpe-ai-developer-day-stockholm-sweeden/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Build your own AI Agent with HPE &#x26; Nvidia</strong></h2> <h3>Stockholm, Sweden</h3> <p>March 24, 2026</p> <p>Join us for an immersive, hands-on workshop where you’ll learn how to design and deploy real-world generative AI use cases on HPE Private Cloud AI, powered by NVIDIA AI Enterprise Software.</p> <h6><a href="https://edgetocloud.se/event/hpe-private-cloud-ai-developer-day-framtidens-ai-losningar-for-offentlig-sektor">Register</a></h6>https://developer.hpe.com/hpe-ai-developer-day-goteborg-sweeden/https://developer.hpe.com/hpe-ai-developer-day-goteborg-sweeden/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Build your own AI Agent with HPE &#x26; Nvidia</strong></h2> <h3>Göteborg, Sweden</h3> <p>March 25, 2026</p> <p>Join us for an immersive, hands-on workshop where you’ll learn how to design and deploy real-world generative AI use cases on HPE Private Cloud AI, powered by NVIDIA AI Enterprise Software.</p> <h6><a href="https://events.bizzabo.com/797906/page/5605984/sweden-information">Register</a></h6>https://developer.hpe.com/hpe-ai-developer-days-san-jose-california/https://developer.hpe.com/hpe-ai-developer-days-san-jose-california/<h1><strong>HPE AI Developer Day</strong></h1> <h2><strong>Build your own AI Agent with HPE &#x26; Nvidia</strong></h2> <h3>San Jose, California</h3> <p>January 26, 2026</p> <p>Join us for an immersive, hands-on workshop where you’ll learn how to design and deploy real-world generative AI use cases on HPE Private Cloud AI, powered by NVIDIA AI Enterprise Software.</p> <p><a href="https://events.bizzabo.com/797906/page/5472193/bay-area-information">Register</a></p>https://developer.hpe.com/hpe-ai-foundations-workshop/https://developer.hpe.com/hpe-ai-foundations-workshop/<h2>HPE AI Foundations workshop</h2> <p>The innovation workshop is an invaluable opportunity for all IT technologists looking to learn how to design, implement, and operate a hybrid cloud AI platform. The HPE AI Foundations workshop shows you how to establish a comprehensive and extensible foundation to empower integration, strategic deployment, responsible data management &#x26; governance, machine &#x26; deep learning, and generative AI. During the workshop, you will gain the knowledge and skills necessary to understand HPE GreenLake cloud and best of breed AI ecosystem technologies incorporated within it.</p> <p><a href="https://developer.hpe.com/skillup/"><strong>Register now!</strong></a></p>https://developer.hpe.com/hpe-and-451-research-joint-webinar/https://developer.hpe.com/hpe-and-451-research-joint-webinar/<h2>Hybrid Cloud by Design : Optimizing Enterprise IT for Modern Workloads</h2> <p>September 17, 2024</p> <p>Given the varying capacity, cost, performance, security, governance, and data locality requirements of enterprise workloads, hybrid cloud estates spanning on-premises private and off-premises public environments have become the default operating model for enterprise IT. The resulting increase in IT complexity often results in infrastructure silos instead of systems that deliver a unified cloud operating experience. Join Latha Vishnubhotla, Chief Platform Officer at HPE, and Melanie Posey, Research Director, 451 Research, for insights on how IT leaders formulate their hybrid cloud strategies to deal with this challenge.</p>https://developer.hpe.com/hpe-compute-ops-management-apis-and-the-devops-ecosystem-updates-and-developer-tools/https://developer.hpe.com/hpe-compute-ops-management-apis-and-the-devops-ecosystem-updates-and-developer-tools/<h2>HPE Compute Ops Management APIs and the DevOps ecosystem: Updates and developer tools</h2> <p>June 18, 2025</p> <p>Explore the latest updates to the HPE Compute Ops Management (COM) APIs and discover essential resources for developers. This session will highlight tools such as Postman collections, GitHub projects and their role in enabling seamless DevOps integration. Additionally, we will introduce the PowerShell library for COM, featuring an overview, a live demonstration, and a roadmap for future enhancements. Join us to maximize your productivity and leverage the HPE GreenLake ecosystem for your projects.</p>https://developer.hpe.com/hpe-data-world-2021/https://developer.hpe.com/hpe-data-world-2021/<h1>HPE Data World 2021</h1> <h3>3D virtual event</h3> <p>December 7, 2021</p> <p><strong>Let us help you solve your data challenges!</strong></p> <p>In this virtual event, we will show how HPE can help you to unify, modernize, analyze and protect all of your data, from edge-to-cloud, in any and every place it's stored, so that you get a sustainable competitive advantage.</p>https://developer.hpe.com/hpe-dev-meetups-1/https://developer.hpe.com/hpe-dev-meetups-1/<h2>Meetups</h2> <h3>Streamlit - The fastest way to build and share data science apps</h3> <p>February 23, 2022</p> <p>Poor tooling slows down data science and machine learning projects. Projects often develop a unique ecosystem of bug-ridden and unmaintainable internal tools to analyze data through a patchwork of Jupyter Notebooks and Flask apps. In this talk, you’ll be introduced to <strong>Streamlit</strong>, the fastest way to build and share data apps as Python scripts to help alleviate this issue.</p>https://developer.hpe.com/hpe-dev-meetups-2/https://developer.hpe.com/hpe-dev-meetups-2/<h2>Meetups</h2> <h3>HPE + vFunction: Modernizing Legacy Applications and Data Sources Faster</h3> <p>March 30, 2022</p> <p>Today, most application migrations focus on lift-and-shift strategies to get to the cloud faster, yet neglect the parallel challenge of harnessing value from massive amounts of data.</p> <p>Come to this technology talk with HPE and vFunction to see in action how <strong>vFunction</strong> uses AI, data science, and automation to significantly expand the number of applications that can be refactored into microservices at a much faster speed and much lower risk as part of an integration for <strong>HPE Ezmeral</strong>.</p>https://developer.hpe.com/hpe-dev-meetups-3/https://developer.hpe.com/hpe-dev-meetups-3/<h2>Meetups</h2> <h3>Styra - Decoupled policy enforcement with Open Policy Agent</h3> <p>April 27, 2022</p> <p>In just a few years, the CNCF graduated Open Policy Agent (OPA) project has established itself as the de-facto standard for policy based guard rails around Kubernetes clusters - now it’s moving into microservices! In this talk, we’ll explore the benefits of decoupling policy from application logic, and how OPA can help bring order to an increasingly distributed, heterogeneous, and complex tech stack.</p>https://developer.hpe.com/hpe-dev-meetups-4/https://developer.hpe.com/hpe-dev-meetups-4/<p>Today, natural language processing (NLP) powers the latest conversational AI and translation apps. Join us to explore the collaborative experimentation process that machine learning teams leverage and the challenges they face while training these large-scale NLP models. See how Determined’s open-source deep learning training platform helps model developers train models faster and easier using tools such as resource management, fault tolerance, and model optimization.</p>https://developer.hpe.com/hpe-dev-meetups/https://developer.hpe.com/hpe-dev-meetups/<h2>Meetups</h2> <h3>Quarkus - Supersonic Subatomic Java</h3> <p>January 26, 2022</p> <p>Java based software development has been a winning proposition for the past 20+ years. Come to this technology talk to get an introduction to <strong>Quarkus</strong>, a Kubernetes native Java stack that tailors apps to make them shine in the cloud-native universe.</p>https://developer.hpe.com/hpe-dev-munch-learn-series-april-2022/https://developer.hpe.com/hpe-dev-munch-learn-series-april-2022/<h3>Chapel: Making parallel computing as easy as Py(thon), from laptops to supercomputers</h3> <p>April 20, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, meet with Brad Chamberlain, HPE, who will discuss the Chapel parallel programming language, pioneered by Cray and HPE. Chapel supports parallel programs that are similarly readable/writable as Python while providing performance and portability similar to Fortran, C, C++, and other common techniques for writing parallel and/or scalable code. Brad will also describe an open-source Python library written in Chapel that uses parallelism to accelerate common NumPy and Pandas operations used in data science.</p>https://developer.hpe.com/hpe-dev-munch-learn-series-february-2022/https://developer.hpe.com/hpe-dev-munch-learn-series-february-2022/<h3>Golden Age of AI, Dark Ages of AI Infrastructure</h3> <p>February 16, 2022</p> <p>This month, meet with Neil Conway, senior director of engineering Determined AI at HPE and learn why practical use of deep learning (DL) remains difficult, what problems DL infrastructure tools solve today, where they fall short, and how they can improve. This talk draws on academic work done at Carnegie Mellon, UC Berkeley, and UCLA, as well as experiences at Determined AI, a startup recently acquired by HPE and that builds open-source software to make deep learning engineers dramatically more productive.</p>https://developer.hpe.com/hpe-dev-munch-learn-series-january-2022/https://developer.hpe.com/hpe-dev-munch-learn-series-january-2022/<h3>Location, location, location! With data everywhere, location matters more than ever. Learn how to succeed at the Edge with HPE Ezmeral and NVIDIA</h3> <p>January 19, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, NVIDIA and HPE partner to help you accelerate your success at the Edge while enjoying a friction-free distributed cloud experience across Edge, Core, and multi-cloud. Register now to learn how you can benefit from this data-first, Edge-in revolution as Denis Vilfort from HPE and William Benton from NVIDIA describe the use of GPU accelerated Apache Spark, HPE Ezmeral Data Fabric, and the HPE GreenLake edge-to-cloud platform.</p>https://developer.hpe.com/hpe-dev-munch-learn-series-march-2022/https://developer.hpe.com/hpe-dev-munch-learn-series-march-2022/<h3>Mithril: Introducing Robust Identities into Istio by integrating with SPIRE</h3> <p>March 23, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, meet with Mithril engineering team at HPE who will discuss the fundamentals of zero trust, workload identities, and introduce the concept of Service Mesh. They will then explain how the new open source solution, Mithril, leverages the widely adopted Service Mesh solution Istio, and the secure identities framework of SPIRE to secure the Istio Service Mesh.</p>https://developer.hpe.com/hpe-dev-munch-learn-series-may-2022/https://developer.hpe.com/hpe-dev-munch-learn-series-may-2022/<h2>Why Open Source is more than Software: The example of The Linux Foundation's AgStack project</h2> <p>May 18, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, learn about open services through the intriguing use case of the Linux Foundation’s AgStack project for the world's agricultural ecosystem. Hear from AgStack Founder, Sumer Johal, and HPE AgStack Developer, Ted Dunning, as they explore how open source enables services to benefit the world.</p>https://developer.hpe.com/hpe-discover-2022/https://developer.hpe.com/hpe-discover-2022/<h2>HPE Discover 2022</h2> <h3>Join the HPE Developer Community team in the Hack Shack</h3> <p>Las Vegas, June 28-30</p> <p>From the latest insights in secure connectivity, hybrid cloud, AI and unified data analytics, HPE Discover 2022 is the best place to stay ahead of the trends and technologies that will move your business forward, faster. Join HPE experts, leading companies, and industry luminaries and learn how to accelerate your data-first modernization across edge to cloud. You don’t want to miss this opportunity to once again gather with HPE Developer Community members in the Hack Shack!</p>https://developer.hpe.com/hpe-discover/https://developer.hpe.com/hpe-discover/<h2>HPE Discover 2021</h2> <h3>Join the HPE Developer team in the Hack Shack</h3> <p>June 24, 2021</p> <p>HPE Developer will offer five Hack Shack Workshops on Day 3 at Discover 2021. In each session, experts will cover topics from Kubernetes to data fabric and security, followed by hands-on workshops. These workshops will be available to all levels with some aimed specifically at beginners. The team will be available throughout the week to answer any questions you may have.</p>https://developer.hpe.com/hpe-greenlake-and-infrastructure-as-code-iac/https://developer.hpe.com/hpe-greenlake-and-infrastructure-as-code-iac/<h2>HPE GreenLake and Infrastructure-as-Code (IaC)</h2> <p>January 25, 2023</p> <p>Discover the work that is underway at HPE to support IaC in HPE GreenLake, enabling developers to extend automation within the DevOps toolchain. It will allow them to easily spin up, tear down, and scale infrastructure, making software development, testing, and deployment of applications faster and easier.</p>https://developer.hpe.com/hpe-machine-learning-development-environment-and-the-open-source-ml-advantage/https://developer.hpe.com/hpe-machine-learning-development-environment-and-the-open-source-ml-advantage/<h2>HPE Machine Learning Development Environment and the Open Source ML Advantage</h2> <p>May 31, 2023</p> <p>Built upon the widely popular open source Determined Training Platform, the HPE Machine Learning Development Environment (MLDE) helps developers and scientists focus on innovation by removing the complexity and cost associated with machine learning model development. In this meetup, you’ll be exposed to the Determined platform and how it integrates the latest, innovative open source AI/ML technologies, including PyTorch, TensorFlow, and DeepSpeed, and new libraries to further accelerate distributed ML model training.</p>https://developer.hpe.com/hpe-private-cloud-ai-technical-demo/https://developer.hpe.com/hpe-private-cloud-ai-technical-demo/<h2>HPE Private Cloud AI Technical Demo</h2> <p>April 30, 2025</p> <p>This technical meetup will delve into the HPE Private Cloud AI platform, offering role-specific walkthroughs for administrators focusing on user and GPU management, data engineers exploring data pipeline construction with Airflow and Spark, and data scientists learning about model deployment via HPE Machine Learning Inference Software (MLIS) and AI Essentials Solution Accelerators.</p>https://developer.hpe.com/hpe-sustainability-insight-center-a-win-for-both-business-and-the-environment/https://developer.hpe.com/hpe-sustainability-insight-center-a-win-for-both-business-and-the-environment/<h2>HPE Sustainability Insight Center: A Win for both Business and the Environment</h2> <p>May 21, 2025</p> <p>Although IT sustainability is a top priority for many businesses today, significant challenges often stand in their way in achieving their goals. Often, IT sustainability reporting is disaggregated and done mainly offline, accomplished through email communications. In addition, research (Deloitte) shows that 88% of those customers aiming to achieve IT sustainability are challenged with data quality. Join this session to learn how the HPE Sustainability Insight Center addresses these challenges through a unified real-time dashboard featuring automated data feeds.</p>https://developer.hpe.com/hpe-sustainability-strategy-and-sustainability-research-at-hewlett-packard-labs/https://developer.hpe.com/hpe-sustainability-strategy-and-sustainability-research-at-hewlett-packard-labs/<h2>HPE Sustainability Strategy and Sustainability Research at Hewlett Packard Labs</h2> <p>October 19, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, learn about HPE’s sustainability strategy, Net Zero commitments, and Hewlett Packard Labs’ rich history of sustainability research that spans IT systems and beyond IT and that leads technological advancements required to achieve a low-carbon future.</p>https://developer.hpe.com/hpe-webinar-is-there-life-after-hadoop/https://developer.hpe.com/hpe-webinar-is-there-life-after-hadoop/<h2>HPE Webinar:</h2> <h2>Is there life after Hadoop?</h2> <h3>Of course there is!</h3> <h4>August 26, 2021</h4> <p>Business-critical analytical applications depend on Hadoop – but technology innovations like object-based data platforms, hybrid cloud deployment, and open-source compute options (i.e., Apache Spark and Presto SQL) have emerged, leaving enterprises questioning the future of their data lake investment. Find out how to de-risk your business with a new strategy to modernize your data analytics platform – allowing you to gain more value from your data. This 45-minute live session highlights best practices from enterprises who realized that there is life after Hadoop and explains how HPE Ezmeral software and solutions can help.</p>https://developer.hpe.com/hpedev-much-learn-1/https://developer.hpe.com/hpedev-much-learn-1/<h2>Munch &#x26; Learn series</h2> <p>What's a data fabric and how does it work?</p> <p>January 27, 2021</p> <p>Run by the HPE Developer Community, the Munch &#x26; Learn monthly technology talks feature industry technology experts who offer valuable insights into today’s most popular technologies. Take advantage of this free 60-minute session opportunity to listen to and engage with these leading technologists on a variety of subjects. Each month we will be presenting a different topic. For January 2021, meet with Ted Dunning and learn about the value of a data fabric and how it works.</p>https://developer.hpe.com/hpedev-much-learn-2/https://developer.hpe.com/hpedev-much-learn-2/<h2>Munch &#x26; Learn series</h2> <p>Explore Containerization and MLOps</p> <p>February 24, 2021</p> <p>Run by the HPE Developer Community, the Munch &#x26; Learn monthly technology talks feature industry technology experts who offer valuable insights into today’s most popular technologies. Take advantage of this free 60-minute session opportunity to listen to and engage with these leading technologists on a variety of subjects. Each month we will be presenting a different topic. For February 24, 2021, meet with Tom Phelan and learn about container architectures and how they can leverage the Kubernetes Container Orchestrator to deploy and manage stateful, as well as microservice-based, applications.</p>https://developer.hpe.com/hybrid-classical-quantum-workflows-on-hpe-supercomputers/https://developer.hpe.com/hybrid-classical-quantum-workflows-on-hpe-supercomputers/<h2>Hybrid Classical-Quantum Workflows on HPE Supercomputers</h2> <p>October 16, 2024</p> <p>Quantum computing is an active area of research that has attracted attention from key players in academia and industry. While quantum devices are expected to perform well for certain problems involving a small amount of data but high complexity, classical supercomputers continue to better serve researchers in areas with data-intensive workloads relying on software tools and infrastructure developed over decades. In conjunction with several possibly tightly integrated Noisy Intermediate-Scale Quantum (NISQ) devices, classical supercomputers would be valuable candidates for the execution of hybrid workflows while sustaining the classical portion of the computation and communication of potentially large data volumes. Join this talk to get an overview of the collaboration between Classiq and HPE on this topic.</p>https://developer.hpe.com/implementing-centralized-key-management-in-hpe-greenlake-with-thales-ciphertrust/https://developer.hpe.com/implementing-centralized-key-management-in-hpe-greenlake-with-thales-ciphertrust/<h2>Implementing Centralized Key Management in HPE GreenLake with Thales CipherTrust</h2> <p>June 26, 2024</p> <p>Driven by the proliferation in data silos across edge-to-cloud estates, deepening security compliance regulations, and the expansion of data sovereignty requirements, organizations and governments worldwide are mandating the deployment of data protection technologies, such as Centralized Key Management. In this session, you will learn about the exciting new partnership between HPE GreenLake and Thales, the worldwide leader in digital trust and data security. Here, you’ll be introduced to the new Centralized Key Management solution offering on HPE GreenLake.</p>https://developer.hpe.com/implementing-your-ai-breakthroughs-effectively-the-infrastructure-to-your-ai/https://developer.hpe.com/implementing-your-ai-breakthroughs-effectively-the-infrastructure-to-your-ai/<h2>Implementing Your AI Breakthroughs Effectively - The Infrastructure to your AI</h2> <p>November 6, 2024</p> <p>Grab a coffee together with your IT Ops Manager, as we discuss different AI infrastructure environment options to suit your use case. Consider what processes are necessary, who will be involved, and what resources you already have to build a toolbox that integrates all the AI components together seamlessly. You don’t have to be an AI infrastructure expert to get started, so let’s explore how you’ll get your AI proof of concept implemented.</p>https://developer.hpe.com/intel-innovation-2022/https://developer.hpe.com/intel-innovation-2022/<h2>Intel Innovation 2022</h2> <h4>September 27-28, 2022 San Jose, California</h4> <p>Attend the Intel Innovation developer conference and explore the latest break-throughs in tech-accelerating computing. Visit the HPE booth and hear more about what the HPE Developer Community is doing by engaging with the booth staff.</p>https://developer.hpe.com/introduction-to-hpe-greenlake-webhooks/https://developer.hpe.com/introduction-to-hpe-greenlake-webhooks/<h2>Introduction to HPE GreenLake cloud webhooks</h2> <p>February 26, 2025</p> <p>This presentation introduces a new webhook functionality within HPE GreenLake cloud; one that enables platform services and applications to seamlessly publish events to external customer HTTP endpoints. Join this session to learn about the architecture, features, and benefits of this new webhook functionality, highlighting its potential to enhance platform extensibility and streamline integrations with customer systems.</p>https://developer.hpe.com/introduction-to-hpe-networking-central-new-apis-with-postman-and-pycentralv2/https://developer.hpe.com/introduction-to-hpe-networking-central-new-apis-with-postman-and-pycentralv2/<h2>Introduction to HPE Networking Central's new APIs with Postman and PyCentralv2</h2> <p>February 25, 2026</p> <p>Join our next meetup session to learn about the Automation capabilities of HPE Networking Central that have had a big upgrade, and we’ll show you how to get started! Learn how the latest Central APIs make it easier than ever to configure and monitor your network, all within a refreshed Central interface.</p>https://developer.hpe.com/introduction-to-kubeflow/https://developer.hpe.com/introduction-to-kubeflow/<h2>Introduction to KubeFlow</h2> <p>November 30, 2022</p> <p>Learn the basics of Kubeflow, the machine learning toolkit for Kubernetes dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable. See how it works and what it brings to both data scientists and MLOps engineers. Find out what external add-ons may be important in certain use cases, and hear what Arrikto’s Enterprise Kubeflow distribution adds to the equation.</p>https://developer.hpe.com/it-new-year-resolution-building-an-ethical-and-trustworthy-ai-system/https://developer.hpe.com/it-new-year-resolution-building-an-ethical-and-trustworthy-ai-system/<h2>IT New Year’s resolution: Build an ethical and trustworthy AI system</h2> <p>January 22, 2025</p> <p>The "era of AI" provides new technological capabilities accompanied by new responsibilities. Learn how we developed AI principles to guide our organization's development decisions and how to put these principles into action throughout your AI system lifespan and follow-on projects.</p>https://developer.hpe.com/kube-con-europe-2021/https://developer.hpe.com/kube-con-europe-2021/<h2>KubeCon Europe 2021</h2> <p>May 04-07, 2021 Europe</p> <p>The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities virtually from May 4 – 7, 2021. Join containerd, CoreDNS, Envoy, Fluentd, Harbor, Helm, Jaeger, Kubernetes, Prometheus, Rook, TiKV, TUF, Vitess, Argo, CloudEvents, CNI, Contour, Cortex, CRI-O, Dragonfly, etcd, Falco, gRPC, KubeEdge, Linkerd, NATS, Notary, Open Policy Agent, OpenTracing, Operator Framework, SPIFFE, SPIRE, and Thanos as the community gathers for four days to further the education and advancement of cloud native computing.</p>https://developer.hpe.com/kube-con-na-2020/https://developer.hpe.com/kube-con-na-2020/<h2>KubeCon NA 2020</h2> <p>November 17-20, 2020 NA</p> <p>The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities virtually from November 17 – 20, 2020. Join Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, Jaeger, Vitess, TUF, OpenTracing, gRPC, CNI, Notary, NATS, Linkerd, Helm, Rook, Harbor, etcd, Open Policy Agent, CRI-O, TiKV, CloudEvents, Falco, Argo, Dragonfly, SPIFFE and SPIRE as the community gathers for four days to further the education and advancement of cloud native computing.</p>https://developer.hpe.com/kubecon-eu-2022/https://developer.hpe.com/kubecon-eu-2022/<h2>KubeCon EU 2022</h2> <h4>May 16-20, 2022 EU Valencia, Spain + Virtual</h4> <p>The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities in Valencia, Spain from 16 – 20 May 2022. Join thousands of cloud-native leaders, including Hewlett Packard Enterprise (HPE), as the community gathers for four days to further the education and advancement of cloud native computing. Visit the HPE booth to learn more about the HPE Developer Community and HPE’s open source efforts.</p>https://developer.hpe.com/lessons-in-ai-for-retailers-transforming-the-shopping-experience/https://developer.hpe.com/lessons-in-ai-for-retailers-transforming-the-shopping-experience/<h2>Lessons in AI for retailers: Transforming the shopping experience</h2> <p>May 14, 2025</p> <p>AI is reshaping the retail industry, from reducing shrink and optimizing inventory management to delivering faster, more personalized customer experiences. However, the journey from AI adoption to achieving success can feel overwhelming. Join us to learn how AI is transforming retail operations, from improving store-level decision-making to reimagining the shopping journey. Whether you're new to AI or refining your strategy, this webinar will give you actionable insights to help meet customer expectations, while protecting customer trust.</p>https://developer.hpe.com/kubecon-na-2022/https://developer.hpe.com/kubecon-na-2022/<h2>KubeCon NA 2022</h2> <h4>October 24-28, 2022 Detroit, Michigan USA + Virtual</h4> <p>The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities in Detroit, Michigan USA from 24 – 28 Oct 2022. Join thousands of cloud-native leaders, including Hewlett Packard Enterprise (HPE), as the community gathers for five days to further the education and advancement of cloud native computing. Visit the HPE booth to learn more about the HPE Developer Community and HPE’s open source efforts.</p>https://developer.hpe.com/learn-how-ai-hackers-detect-fragility-and-how-to-thwart-them-with-ai-model-resilience/https://developer.hpe.com/learn-how-ai-hackers-detect-fragility-and-how-to-thwart-them-with-ai-model-resilience/<h2>Learn how AI hackers detect fragility and how to thwart them with AI model resilience</h2> <p>March 20, 2024</p> <p>Learn how AI hackers detect fragility and how to thwart them with AI model resilience On every lab test, your AI was superhuman. But how will it fare in the real world of smog, smears, and nation state hackers? In this session, we'll explore how AI hackers can measure the fragility of today's AI models, covering the model's vulnerability under real-world conditions across applications of varying data dimensions, from signals to images to videos. We’ll then show how to engineer robustness into models and sketch out tomorrow's AI supply chain where confidence is measurable and the model's perception can be inspected.</p>https://developer.hpe.com/kubecon-na-2021/https://developer.hpe.com/kubecon-na-2021/<h2>KubeCon NA 2021</h2> <h4>Oct 11-15, 2021 NA Los Angeles + Virtual</h4> <p>The <a href="https://www.cncf.io/">Cloud Native Computing Foundation’s</a> flagship conference gathers adopters and technologists from leading open source and cloud native communities from October 11-15, 2021. Join containerd, CoreDNS, Envoy, etcd, Fluentd, Harbor, Helm, Jaeger, Kubernetes, Open Policy Agent, Prometheus, Rook, TiKV, TUF, Vitess, Argo, Buildpacks, CloudEvents, CNI, Contour, Cortex, CRI-O, Dragonfly, Falco, Flux, gRPC, KubeEdge, Linkerd, NATS, Notary, OpenTracing, Operator Framework, SPIFFE, SPIRE, and Thanos as the community gathers for four days to further the education and advancement of cloud native computing.</p>https://developer.hpe.com/leveraging-tech-to-address-global-challenges-health/https://developer.hpe.com/leveraging-tech-to-address-global-challenges-health/<h2>Leveraging Tech to Address Global Challenges &#x26; Health</h2> <p>April 19, 2023</p> <p>The power of technology, digital transformation, and partnerships can drive significant advancement in building a more sustainable and equitable world. Join us to hear Deputy Director of Hewlett Packard Enterprise Foundation, Fred Tan, who is helping to drive the Force for Good program, discuss specific initiatives HPE is involved with to achieve our mission of being able to advance the way people live and work by addressing pressing global challenges.</p>https://developer.hpe.com/leveraging-the-open-source-ml-advantage/https://developer.hpe.com/leveraging-the-open-source-ml-advantage/<h2>Leveraging the open source ML advantage</h2> <p>May 1, 2024</p> <p>Join this virtual session from HPE Data Science Community for discussions around:</p> <ul> <li>Scaling ML model training using state-of-the-art hyperparameter tuning and distributed training methods - using all open source software.</li> <li>Getting started with open-source LLMs in our playground feature: GenAI studio.</li> <li>Smoothly transitioning from foundation model selection and prompt engineering to fine-tuning and deployment.</li> </ul>https://developer.hpe.com/ll-mesh-–-democratizing-gen-ai/https://developer.hpe.com/ll-mesh-–-democratizing-gen-ai/<h2>LL-Mesh – Democratizing Gen AI</h2> <p>December 18, 2024</p> <p>LL-Mesh is a pioneering initiative by HPE aimed at democratizing Generative Artificial Intelligence (Gen AI). It empowers users to create tools and web applications using Gen AI with Low or No Coding. This approach addresses the technical challenges by simplifying the integration process. Join this session to learn how LL-Mesh platform allows for the creation of a "Mesh" of the Gen AI tools, providing orchestration capabilities through an agentic Reasoning Engine based on Large Language Models (LLMs).</p>https://developer.hpe.com/llm-finetuning-for-mere-mortals/https://developer.hpe.com/llm-finetuning-for-mere-mortals/<h2>LLM finetuning for mere mortals</h2> <p>August 28, 2024</p> <p>Everyone wants to use Large Language Models (LLMs), and for good reason. With applications ranging from content creation to automated software development, LLMs have the potential to transform nearly every industry. How can you make the most of this technology when applying it to your own use-cases? Finetuning is one highly effective approach, but can be challenging to implement correctly. In this talk, you'll learn about the challenges involved, and how us mere mortals can tackle them using HPE's new software that leverages the open-source machine learning (ML) ecosystem.</p>https://developer.hpe.com/machine-learning-data-version-control-dvc-reproducibility-and-collaboration-in-your-ml-projects/https://developer.hpe.com/machine-learning-data-version-control-dvc-reproducibility-and-collaboration-in-your-ml-projects/<h2>Machine Learning Data Version Control (DVC): Reproducibility and Collaboration in your ML Projects</h2> <p>September 28, 2022</p> <p>In this session, we’ll do a demo where you’ll learn how to manage and make your machine learning projects reproducible with our open-source tool DVC and the DVC extension for VS Code IDE. We will see how to track datasets and models, run, compare, visualize, and track machine learning experiments right in VS Code. We'll then go over our GitOps-based model registry solution. Search, share, and manage all models with full context around model lineage, version, production status, data used to train model, and more.</p>https://developer.hpe.com/machine-learns-from-data-to-be-artificially-intelligent-use-cases-lessons-learned/https://developer.hpe.com/machine-learns-from-data-to-be-artificially-intelligent-use-cases-lessons-learned/<h2>Machines learn from data to be artificially intelligent</h2> <p>August 17, 2022</p> <p>Join us for a free, 60-minute session where you can connect with experts who offer valuable insights into today’s most popular technologies. This month, hear from HPE’s own Dr. Eng Lim Goh, senior VP and CTO of AI, on the importance of sharing data to gain insights and how to do so responsibly. In his talk, Dr. Goh will illustrate how AI can advance the human condition, exploring a variety of industry use cases and lessons we’ve learned.</p>https://developer.hpe.com/ml-at-scale-‘23/https://developer.hpe.com/ml-at-scale-‘23/<h2>ML-at-Scale ‘23</h2> <p>September 12-13, 2023</p> <p>What does it mean to truly achieve machine learning (ML) at scale? Determined AI is hosting a virtual conference where you can find out. If you work on solving critical problems with ML, you won’t want to miss this. ML-at-Scale ’23 will feature talks and workshops about pushing the scale of AI – from open source development to high-performance computing.</p>https://developer.hpe.com/no-more-blind-spots-how-ebpf-transforms-observability/https://developer.hpe.com/no-more-blind-spots-how-ebpf-transforms-observability/<h2>No more blind spots: How eBPF transforms observability</h2> <p>March 26, 2025</p> <p>In the world of modern cloud-native applications, traditional observability tools often fall short, leaving critical blind spots in performance monitoring, security, and troubleshooting. Enter eBPF (Extended Berkeley Packet Filter), a revolutionary technology that allows deep, low-overhead visibility into the Linux kernel without modifying application code or adding intrusive instrumentation. Join this session to explore how eBPF can transform observability by enabling real-time insights into system behavior, network traffic, and application performance.</p>https://developer.hpe.com/nonstop-technical-boot-camp-tbc-2023/https://developer.hpe.com/nonstop-technical-boot-camp-tbc-2023/<h2>NonStop Technical Boot Camp (TBC) 2023</h2> <p>September 12-14, 2023</p> <p>Do not miss this opportunity to be the first to discover solutions for cyber resilience and hybrid cloud to help you stay ahead of the pack. At this event, systemadmins, architects, DBAs, DevOps practitioners and new comers will meet and hear from a thriving community that continues to define the gold standard for building the most reliable IT solutions! Register for the conference and book your hotel room now!</p>https://developer.hpe.com/nvidia-deep-learning-institute/https://developer.hpe.com/nvidia-deep-learning-institute/<h2>Get a free course on AI, HPC or Data Science from NVIDIA</h2> <p>Advance your knowledge in AI, accelerated computing, accelerated data science, graphics, and simulation with a free NVIDIA Deep Learning Institute course (offer valid while supplies last).</p>https://developer.hpe.com/nvidia-gtc-–-the-developer-conference-for-the-era-of-ai/https://developer.hpe.com/nvidia-gtc-–-the-developer-conference-for-the-era-of-ai/<h2>NVIDIA GTC – the developer conference for the era of AI</h2> <p>Interested in artificial intelligence and high performance application development? Hewlett Packard Enterprise (HPE) is as well and is excited to announce its participation as a Diamond sponsor of online event NVIDIA GTC22, September 19-22. NVIDIA GTC is the developer conference for the Era of AI and the Metaverse, exploring development, design and simulation, robotics, and applied real-time technologies.</p>https://developer.hpe.com/nvidia-nim-agents-blueprints-do-your-own-ai-multimodal-pdf-data-extraction-101/https://developer.hpe.com/nvidia-nim-agents-blueprints-do-your-own-ai-multimodal-pdf-data-extraction-101/<h2>NVIDIA NIM Agents Blueprints - Do your own AI: Multimodal PDF Data Extraction 101</h2> <p>October 30, 2024</p> <p>Pursuant to the launch of HPE Private Cloud AI, which leverages NVIDIA AI Enterprise and NIM Agent Blueprints, NVIDIA will highlight how it eases and accelerates the process of building a multimodal PDF data extraction workflow. using NVIDIA NeMo™ Retriever NIM microservices to unlock highly accurate insights from massive volumes of enterprise data. Join this session to learn about the enterprise-scale multimodal document retrieval workflow designed to enhance generative AI applications with RAG capabilities that can be connected to proprietary data–wherever it resides.</p>https://developer.hpe.com/observability-in-action/https://developer.hpe.com/observability-in-action/<h2>Observability in Action</h2> <p>July 26, 2023</p> <p>In this session, learn how OpsRamp approaches observability in a hybrid-cloud world to help customers answer why problems are occurring with their infrastructure and applications by examining events, metrics, logs and traces. This talk will also explain how OpsRamp avoids vendor lock-in through the use of open standards.</p>https://developer.hpe.com/open-source-at-hpe-three-pillars-many-paths/https://developer.hpe.com/open-source-at-hpe-three-pillars-many-paths/<h2>Open Source at HPE: Three Pillars, Many Paths</h2> <p>03/18/2026</p> <p>Join us for a cross-pillar conversation on how HPE uses open source to enable platforms, power science, and scale infrastructure.</p> <p>Our panel will explore how HPE engages with open source across networking, high-performance computing, and storage—showing that open source isn’t a single strategy but a spectrum: an entry point, a trust mechanism, a catalyst for scientific collaboration, and the foundation for scalable infrastructure.</p>https://developer.hpe.com/open-source-summit-europe-2022/https://developer.hpe.com/open-source-summit-europe-2022/<h2><strong>Open Source Summit Europe 2022</strong></h2> <h4>13-16 September, 2022</h4> <p>Open Source Summit is the premier event for open source developers, technologists, and community leaders to collaborate, share information, solve problems, and gain knowledge, furthering open source innovation and ensuring a sustainable open source ecosystem. It is the gathering place for open-source code and community contributors.</p>https://developer.hpe.com/opensearch-–-the-open-source-search-and-analytics-suite-you-can-run-yourself/https://developer.hpe.com/opensearch-–-the-open-source-search-and-analytics-suite-you-can-run-yourself/<h1>OpenSearch – The open-source search and analytics suite you can run yourself</h1> <p>June 22, 2022</p> <p>With systems getting larger, it’s more important than ever to be able to search through all your data. Whether that involves adding a search functionality for your users or creating a distributed log store to solve problems faster, OpenSearch can help you get there. The best part is that it’s open source, so the community drives what’s built next.</p>https://developer.hpe.com/opentelemetry-getting-started-and-the-road-to-production/https://developer.hpe.com/opentelemetry-getting-started-and-the-road-to-production/<h2>OpenTelemetry: Getting Started and The Road to Production</h2> <p>August 31, 2022</p> <p>In this talk, you will experience a complete OpenTelemetry Bootcamp. From the very basics, including what OpenTelemetry is and how it can help you, all the way to advanced concepts such as managing cost with sampling, the collector, and what to know before deploying in production. This session will serve you as the roadmap for starting with OpenTelemetry and as your guide to understanding core concepts.</p>https://developer.hpe.com/opportunities-and-challenges-of-self-hosting-genai/https://developer.hpe.com/opportunities-and-challenges-of-self-hosting-genai/<h2>Opportunities and challenges of self-hosting GenAI</h2> <p>January 30, 2024</p> <p>Join us on 30 January with TitanML as we bring together the HPE data science user group to discuss opportunities and challenges of self-hosting GenAI.</p>https://developer.hpe.com/optimizing-deep-neural-network-inference-workloads/https://developer.hpe.com/optimizing-deep-neural-network-inference-workloads/<h2>Optimizing deep neural network inference workloads</h2> <p>November 15, 2023</p> <p>AI, and specifically deep learning, is driving significant advancements in many vertical markets. Whether within the computer vision or natural language processing domain, deploying this technology at scale requires overcoming many challenges and constraints. In this session, learn about deep neural network inference optimization techniques, their pros and cons, and how to use them together to optimize inference performance.</p>https://developer.hpe.com/running-reliable-systems-part-1-an-overview-of-sre/https://developer.hpe.com/running-reliable-systems-part-1-an-overview-of-sre/<h2>Running reliable systems Part 1: An overview of SRE</h2> <p>December 7, 2022</p> <p>Learn about the Site Reliability Engineering (SRE) approach to ensuring reliability as part of the software product development process.</p>https://developer.hpe.com/running-reliable-systems-part-2-service-level-objective-slo-math/https://developer.hpe.com/running-reliable-systems-part-2-service-level-objective-slo-math/<h2>Running reliable systems Part 2: SLO Math</h2> <p>December 14, 2022</p> <p>Learn how to set Service-Level Objectives (SLOs) that reflect the expectations of your customers by using mathematical probability and implementing SLO-based reliability at many layers in the stack.</p>https://developer.hpe.com/secure-genai-adoption-for-all/https://developer.hpe.com/secure-genai-adoption-for-all/<h2>Secure GenAI Adoption for all!</h2> <p>February 21, 2024</p> <p>GenAI can bring great value to a business by providing a new interface that can be used to accelerate creativity, run knowledge-centric tasks, or simply provide more rapid access to information. However, rapid adoption of this technology has led to security snafu's, privacy issues and hallucinogenic data reaching the wrong people. In this session, we will look at how some of these risks have made it into the critical path for GenAI and how we might start to mitigate these issues through technology and processes. Finally, we will review "Project Ethan", which was highlighted on the mainstage at Discover Barcelona by HPE CTO, Fidelma Russo, to see how this project will bridge the gap from rapid innovation to secure business value. Join us for a lively presentation and discussion.</p>https://developer.hpe.com/state-of-the-nation-–-linux-distributions/https://developer.hpe.com/state-of-the-nation-–-linux-distributions/<h2>State of the Nation – Linux distributions</h2> <p>September 20, 2023</p> <p>Hear about the latest trends in Linux. Learn what happened to CentOS, the reasons why Red Hat no longer posts source code, and explore new minimal container deployment operating systems like CoreOS and SLE-Micro. In this session, Craig Lamparter of the HPE Linux Enablement Lab will discuss these topics as well as identify which Linux distributions are supported on HPE servers and why.</p>https://developer.hpe.com/storage-applications-in-kubernetes/https://developer.hpe.com/storage-applications-in-kubernetes/<h2>Storage &#x26; Applications in Kubernetes</h2> <h3>Live Webinar</h3> <p>March 17, 2022<br> 10am PT / 1pm ET / 6pm GT</p> <p>In this 30-minute interactive webinar, you’ll be introduced to Kubernetes storage fundamentals, key Kubernetes storage terminology, real-world use cases and best practices.</p>https://developer.hpe.com/sustainably-scaling-ai-adoption/https://developer.hpe.com/sustainably-scaling-ai-adoption/<h2>Sustainably Scaling AI Adoption</h2> <p>August 6, 2025</p> <p>AI-capable infrastructure configurations are introducing new computing, energy, and cooling demands on our data centers and technology stacks. Our engineering capabilities around efficiency, design, and cooling are now at the forefront of progress as we balance resources, costs, and outcomes. Join leading minds at HPE as we explore sustainable AI solutions for the enterprise and data center technologies that can enable the AI workloads of the future.</p>https://developer.hpe.com/technology-and-solutions-summit-2020-paris/https://developer.hpe.com/technology-and-solutions-summit-2020-paris/<h2>HPE Technology and Solutions Summit 2021</h2> <p>March 15-18, 2021 - Virtual</p> <p>HPE Technology and Solutions Summit 2021 is Hewlett Packard Enterprise's largest, annual and most comprehensive technical and solutions knowledge-transfer event. Since its inception in 2006, HPE TSS has established its reputation as a renowned training initiative. Presales Consultants and Solutions Architects from HPE and partner communities in Europe, Middle East and Africa benefit from direct access to HPE technology experts, engineers and Chief Technologists. You will receive up to the minute training around product developments, roadmap updates, strategic insights and be able to practice and apply your knowledge under non-disclosure.</p>https://developer.hpe.com/texas-children’s-hospital-healthcare-hackathon-sponsored-by-hpe/https://developer.hpe.com/texas-children’s-hospital-healthcare-hackathon-sponsored-by-hpe/<h2>Texas Children’s Hospital Healthcare Hackathon</h2> <p>Sponsored by HPE</p> <p>May 14-24, 2021</p> <p>Showcase your innovative, tech-based solutions to help build the hospital of the future! To celebrate the groundbreaking of its new hospital, Texas Children’s Hospital is hosting a two-week long virtual Healthcare Hackathon. HPE is proud to sponsor this event where coders can submit innovative solutions to address its biggest day-to-day challenges and win prizes. A special category for kids allows younger coders to participate, too. Register today!</p>https://developer.hpe.com/the-10th-annual-chapel-implementers-and-users-workshop-chiuw/https://developer.hpe.com/the-10th-annual-chapel-implementers-and-users-workshop-chiuw/<h2>CHIUW 2023</h2> <h3>The 10th Annual Chapel Implementers and Users Workshop</h3> <p>June 01-02, 2023</p> <p>CHIUW 2023 is the 10th annual Chapel Implementers and Users Workshop, which serves as a forum where users and developers of the general-purpose Chapel programming language (chapel-lang.org) can meet to report on work being done with Chapel, exchange ideas, and forge new collaborations. Anyone interested in parallel programming and/or Chapel is encouraged to attend CHIUW, from long-term enthusiasts to those simply curious to learn more. This year's CHIUW will be online and there will be no registration fees.</p>https://developer.hpe.com/the-9th-annual-chapel-implementers-and-users-workshop-chiuw/https://developer.hpe.com/the-9th-annual-chapel-implementers-and-users-workshop-chiuw/<h2>CHIUW 2022</h2> <h3>The 9th Annual Chapel Implementers and Users Workshop</h3> <p>June 09-10, 2022</p> <p>CHIUW 2022 is the 9th annual Chapel Implementers and Users Workshop, which serves as a forum where users and developers of the Chapel programming language (chapel-lang.org) can meet to report on work being done with Chapel, exchange ideas, forge new collaborations, and engage in coding activities. Anyone interested in parallel programming and Chapel is encouraged to attend CHIUW, from long-term enthusiasts to those simply curious to learn more. This year's CHIUW will be online and there will be no registration fees.</p>https://developer.hpe.com/the-family-of-hpe-greenlake-data-services-cloud-console-dscc-apis-–-putting-it-all-together/https://developer.hpe.com/the-family-of-hpe-greenlake-data-services-cloud-console-dscc-apis-–-putting-it-all-together/<h2>The data services family of APIs for HPE GreenLake – Putting it all together</h2> <p>April 24, 2024</p> <p>HPE recently announced a set of data services APIs for the HPE GreenLake platform for Backup and Recovery, Private Cloud Business Edition, Virtualization and common Data Services. In this session, a team of product managers and architects will present common API service interactions (e.g. hybrid services, data services, etc.) to accelerate your consumption of these APIs for use cases such as automation and monitoring. They’ll help you understand the benefits of these API modules, how to interact with them, and their relation to the Storage APIs published earlier.</p>https://developer.hpe.com/the-hack-shack-at-discover-2023/https://developer.hpe.com/the-hack-shack-at-discover-2023/<h2>HPE Discover 2023</h2> <h3>The Hack Shack at Discover 2023</h3> <p>Las Vegas, June 20-22</p> <p>This year, the Hack Shack will serve as the technology hub for all technologists and home base for the HPE Office of the CTO. Join us at HPE Discover 2023 to take advantage of opportunities to learn more about the HPE GreenLake platform and meet with HPE executives and technology experts, sharing your experiences and hopes for the platform. While you’re there, take advantage of opportunities to win prizes with activities like our Treasure Hunt and Hack Shack challenges and join us for our annual celebration party!</p>https://developer.hpe.com/the-last-mile-of-the-self-hosted-llm/https://developer.hpe.com/the-last-mile-of-the-self-hosted-llm/<h2>The last mile of the self-hosted LLM</h2> <p>November 7, 2023</p> <p>Join us on 7 November with TitanML as we bring together the HPE data science user group to discuss the latest trends, best approaches and practical experiences.</p> <p>We’ll have technical talks, live demos and plenty of opportunity to network with your community while enjoying drinks, food and making the most of the table tennis at Bounce Farringdon.</p>https://developer.hpe.com/the-open-source-advantage-exploring-machine-learning-through-thought-leadership/https://developer.hpe.com/the-open-source-advantage-exploring-machine-learning-through-thought-leadership/<h2>The open-source advantage: Exploring machine learning through thought leadership</h2> <p>October 18, 2023</p> <p>In our next Munch &#x26; Learn, Chase Christensen and Amber Graner from the HPE Ezmeral team get real about why machine learning is a big deal today. They break down what machine learning actually means and how thought leadership fits into this tech scene. They’ll explore all the perks of open-source machine learning and even name-drop some tools and libraries that are shaking things up in the field. Join us to hear them share some success stories showing how big thinkers can spark more use and development of these forward-thinking principles. To wrap things up, they’ll throw out a challenge to everyone listening, encouraging you to get involved and help shape the future of open-source machine learning!</p>https://developer.hpe.com/the-transformative-impact-of-generative-ai-on-telco-products/https://developer.hpe.com/the-transformative-impact-of-generative-ai-on-telco-products/<h2>The Transformative Impact of Generative AI on Telco Products</h2> <p>April 17, 2024</p> <p>This session will showcase a PoC for an innovative application that demonstrates a bot capable of retrieving information and executing commands on Athonet 5G Networks. Utilizing an LLM as a core Reasoning Engine, our PoC highlights the integration of advanced AI capabilities without compromising human or customer data security. We’ll demonstrate how to maximize the benefits of our products based on Generative AI by establishing automatic data feedback loops. Thanks to AI, these loops will enhance our product with the ability to self-improve through the analysis of customer interactions, thereby boosting personalization and operational efficiency.</p>https://developer.hpe.com/unleashing-ai-innovation-a-deep-dive-into-the-hpe-private-cloud-ai-software-stack/https://developer.hpe.com/unleashing-ai-innovation-a-deep-dive-into-the-hpe-private-cloud-ai-software-stack/<h2>Unleashing AI Innovation: A deep dive into the HPE Private Cloud AI Software Stack</h2> <p>January 29, 2025</p> <p>Join us for an in-depth exploration of the cutting-edge AI software stack powering the HPE Private Cloud AI (PCAI). This webinar will demonstrate how our platform enables enterprises to accelerate AI development while maintaining data privacy and operational control. Designed for data scientists, data engineers, AI/ML developers, IT managers, and business leaders, this session will reveal how PCAI revolutionizes the deployment and management of generative AI within your organization.</p>https://developer.hpe.com/unlocking-private-ai-power-insurance-fraud-detection-and-beyond/https://developer.hpe.com/unlocking-private-ai-power-insurance-fraud-detection-and-beyond/<h2>Unlocking Private AI Power: Insurance Fraud Detection and Beyond</h2> <p>February 19, 2025</p> <p>Join us to explore the potential of HPE Private Cloud AI in tackling real-world challenges, such as insurance fraud detection. We'll delve into the benefits of leveraging Private AI services. This webinar will demonstrate how one can train and deploy AI models to detect fraud in auto insurance claims, utilizing techniques like data preprocessing, model training, and deployment as an API endpoint. We will also discuss the role of generative AI in enhancing fraud detection capabilities.</p>https://developer.hpe.com/using-hpe-greenlake-edge-to-cloud-platform-to-build-a-private-cloud/https://developer.hpe.com/using-hpe-greenlake-edge-to-cloud-platform-to-build-a-private-cloud/<h2>Using HPE GreenLake edge-to-cloud platform to build a private cloud</h2> <p>January 24, 2024</p> <p>Learn how to onboard &#x26; manage your HPE hardware devices with the HPE GreenLake edge-to-cloud platform and how to configure and use RESTful API commands to automate IT management via the platform. Get to view a demo on using the web console to access an array of integrated services such as HPE GreenLake for Private Cloud Business Edition.</p>https://developer.hpe.com/using-hpe-greenlake-for-red-hat-openshift-to-migrate-modernize-and-run-your-applications-smoothly/https://developer.hpe.com/using-hpe-greenlake-for-red-hat-openshift-to-migrate-modernize-and-run-your-applications-smoothly/<h2>Using HPE GreenLake for Red Hat OpenShift to migrate, modernize and run your applications smoothly</h2> <p>May 29, 2024</p> <p>Many Applications are running in legacy virtualization environments today. Join this session to learn how to migrate these applications smoothly to Red Hat OpenShift. Learn the potential of application modernisation the platform offers: containerizing applications, moving to microservices and using the developer tooling. Also learn how HPE GreenLake offerings help you to get started and set things up as well as manage and run your environment.</p>https://developer.hpe.com/vendor-neutral-gpu-programming-in-chapel/https://developer.hpe.com/vendor-neutral-gpu-programming-in-chapel/<h2>Vendor-Neutral GPU Programming in Chapel</h2> <p>July 31, 2024</p> <p>Writing programs on modern computers requires parallelism to achieve maximum performance. This is complicated by GPUs, which provide great parallel performance at the price of more complex programming. Chapel is an open-source parallel programming language that supports portable, performant software on CPUs and GPUs using a single unified set of language features. In this talk, we will showcase Chapel's vendor-neutral GPU support and share user experiences writing GPU-enabled programs in Chapel and show how you can write vendor-neutral GPU programs today. How you can get involved in our open-source work and our future plans will conclude this talk.</p>https://developer.hpe.com/watch-party-nvidias-earth-2-digital-twins-for-weather-and-climate/https://developer.hpe.com/watch-party-nvidias-earth-2-digital-twins-for-weather-and-climate/<h2>NVIDIA Watch Party: NVIDIA'S EARTH-2: DIGITAL TWINS FOR WEATHER AND CLIMATE</h2> <h4>September 22, 2022</h4> <p>Join this interactive session to hear industry experts from NVIDIA and HPE discuss NVIDIA’s Earth-2 digital twin project, which continues to advance what is possible in the application of AI to meteorology.</p>https://developer.hpe.com/work-smarter-natural-language-access-to-your-infrastructure-via-greenlake-mcp/https://developer.hpe.com/work-smarter-natural-language-access-to-your-infrastructure-via-greenlake-mcp/<h2>Work Smarter: Natural Language Access to your infrastructure via GreenLake MCP</h2> <p>01/28/2026</p> <p>Managing infrastructure often means navigating complex consoles, learning APIs, and manually generating reports. But what if there was an easier way?</p> <p>GreenLake MCP (Model Context Protocol) servers enable AI assistants to securely access your infrastructure data, letting you ask questions in natural language. In this session, we'll explain what MCP is, show you how it works, and demonstrate practical examples: from handling device inventory to security audits.</p> <p>No special technical expertise is needed in order to get valuable insights out of this session. If you've ever wished that working with infrastructure could be simpler, this session is for you.</p>https://developer.hpe.com/world-artificial-intelligence-cannes-festival/https://developer.hpe.com/world-artificial-intelligence-cannes-festival/<h2>World Artificial Intelligence Cannes Festival</h2> <p>This event brings together the top 120 AI players in the market to showcase their latest products and technologies. In partnership with NVIDIA, HPE will showcase state-of-the-art AI technologies such as swarm learning, ML/DL development environments, and a software suite for genomic analysis. See what HPE’s AI solutions can bring to your business through demos, workshops and keynote presentations, and how HPE, together with NVIDIA and their rich ecosystem of partners and startups, can help shape your AI strategy.</p>https://developer.hpe.com/activation-and-onboarding-in-hpe-complete-care-service-itops/home/https://developer.hpe.com/activation-and-onboarding-in-hpe-complete-care-service-itops/home/<style> .action-button { display: inline-block; background-color: #01a982; color: white; padding: 12px 24px; border-radius: 5px; text-decoration: none; font-size: 16px; font-weight: bold; } .action-button:hover { background-color: #005a3c; } #flex-container { display: flex; } #flex-content { flex: 1; } .video-box { flex: 1; max-width: 40%; min-width: 200px; text-align: center; } .video-thumbnail { width: 80%; height: 200px; border-radius: 10px; box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2); } #center-align { text-align: center; } #button-icon { height: 20px; width: auto; } li { font-size: 20px; line-height: 28px; margin-bottom: 10px; } #action-link { background-color: #01a982; color: white; padding: 10px 20px; border-radius: 20px; text-decoration: none; display: inline-flex; align-items: center; gap: 10px; } #section-header, #learn-from-experts, #additional-resources, #contact { color: #01a982; font-size: 28px; margin-top: 30px; border-bottom: 3px solid #01a982; padding-bottom: 10px; } #content-list { list-style-type: decimal; padding-left: 20px; font-size: 18px; line-height: 1.8; } .video-title { display: block; margin: 0; font-size: 20px; font-weight: bold; } li { font-size: 20px; } #content-table { border-collapse: collapse; width: 100%; border: 1px solid black; } #table-header-cell { border: 1px solid black !important; padding: 8px !important; text-align: center !important; font-weight: bold !important; } #table-cell { border: 1px solid black; padding: 8px; text-align: center; } #section-title { border: 1px solid black; padding: 8px; text-align: center; font-weight: bold; background-color: #f1f1f1; } .video-title { display: block; margin: 0; font-size: 20px; font-weight: bold; } </style> <p style="font-size: 22px;"> HPE Complete Care Service - ITOps is a feature for all service levels of HPE Complete Care. It delivers proactive services to the customers by taking a strategic approach to incident management, moving beyond monitoring and alerting to leveraging observability, automation, and artificial intelligence for IT operations (AIOps) while improving IT operational efficiencies. </p> <p style="font-size: 22px;"> HPE Complete Care Service – ITOps is intended to help HPE Complete Care customers monitor, manage, and operate their entire IT environment regardless of the location of those IT assets – on-premises or cloud native. This service enables the visibility and control customers need to monitor and optimize performance, infrastructure operating costs and create observability of incidents and actions that drive the best outcomes for their IT business. </p> <p style="font-size: 22px;"> One outstanding feature that comes standard with HPE Complete Care Service - ITOps is enabled by HPE OpsRamp Software, and AI-powered SaaS solution. </p> <p style="font-size: 22px;"> Hewlett Packard Enterprise provides all HPE Complete Care Service - ITOps customers with HPE OpsRamp Software enterprise licenses that provide customers with a single pane of glass to discover, dashboard, and monitor physical assets (compute, storage, and networking resources) and virtual assets (VMs, cloud instances, storage buckets, and databases) within the HPE Complete Care Service. </p> <h3 id="learn-from-experts">Getting Started</h3> <p style="font-size: 22px;">Direct your IT operations teams to the activation and onboarding resources on this page to start taking full advantage today. You’ll quickly recognize the value of HPE Complete Care Service - ITOps powered by HPE OpsRamp Software and want to expand its functionality throughout your hybrid environment – including virtual machines, containers and workloads running on any manufacturer’s hardware.</p> <p style="font-size: 25px;">Choose between HPE-assisted or self-service activation and onboarding:</p> <h4 id='hpe-assisted'>HPE-Assisted Activation and Onboarding</h4> <p style="font-size: 24px;">Your HPE Account Service Manager (ASM) and a Global Support Remote specialist (GSR) are standing by to help guide your activation and onboarding process.</p> <p style="font-size: 22px;">Please engage your HPE Account Service Manager (ASM) to guide you through the following steps:</p> <ol id="content-list"> <li>Complete the quick setup questionnaire that your HPE ASM will share with you via email.</li> <pre><code> &#x3C;li>A GSR will then be assigned to provide guidance through the onboarding and customization of your HPE Complete Care Service - ITOps command center. This may include assistance with setting up user access to resources, gateways, auto-discovery and dashboarding, customizing monitoring templates and reports, as well as setting up patching, scripting and alert automation.&#x3C;/li> &#x3C;/ol> </code></pre> <h4 id='self-service-activation'>Self-service Activation and Onboarding</h4> <p style="font-size: 24px;">Resources for onboarding, activating, and configuring the HPE Complete Care Service - ITOps:</p> <table id="content-table"> <thead> <tr> <td id="table-header-cell">Video Tutorials</td> <td id="table-header-cell">Blog Tutorials</td> </tr> </thead> <tbody> <tr id="provisioning-activation"> <td colspan="2" id="section-title"> Activation and Onboarding </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=c0ZmdwACq2A&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/CCITOps-video1.png" alt="Step 1 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 1: Configuring a Gateway</span> </td> <td id="table-cell" rowspan="4"> <a href="https://developer.hpe.com/blog/self-activation-and-onboarding-in-hpe-complete-care-service-%E2%80%93-itops/"> Self-activation and onboarding in HPE Complete Care Service - ITOps </a> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=a1GVV-b9hCI&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/CCITOps-video2.png" alt="Step 2 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 2: Agentless SSH integration and monitoring templates</span> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=htZwkW-zG00&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/CCITOps-video3.png" alt="Step 3 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 3: Redfish Server integration and configuration</span> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=MPTq-3EA60E&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/CCITOps-video4.png" alt="Step 4 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 4: Creating a customized dashboard</span> </td> </tr> </tbody> </table> <h3 id="additional-resources">Additional Resources</h3> <ul> <li><a href="https://developer.hpe.com/platform/hpe-opsramp/home/">HPE OpsRamp APIs and Integration Capabilities</a></li> <li><a href="https://docs.opsramp.com/">HPE OpsRamp Software documentation</a></li> <li><a href="https://www.youtube.com/watch?v=p0uA79qnLuk">HPE Complete Care Service - ITOps video</a></li> <li><a href="https://www.hpe.com/psnow/doc/a00134891enw">HPE Complete Care Service - ITOps Solution Brief</a></li> <li><a href="https://www.hpe.com/us/en/collaterals/collateral.a50009342enw.html">HPE Complete Care Service - ITOps Contractual Support Service.</a></li> <li> <strong>Webinar replays</strong> <ul> <li><a href="https://www.youtube.com/watch?v=qH93yD5KSL8&list=PLtS6YX0YOX4f5TyRI7jUdjm7D9H4laNlF">No More Blind Spots: How eBPF Transforms Observability</a></li> <li><a href="https://www.youtube.com/watch?v=VnKSnf7G4-4&list=PLtS6YX0YOX4f5TyRI7jUdjm7D9H4laNlF">From log files to AI insights: The 60-year evolution of observability and AIOps</a></li> <li><a href="https://www.youtube.com/watch?v=nDa_NQPbbVY&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">DayN+ : A new way to look at observability</a></li> </ul> </li> </ul> <h3 id="contact">Any Questions About HPE Complete Care Service - ITOps?</h3> <p style="font-size: 22px;">Need help getting started with the observability service in HPE Complete Care Service - ITOps? Join our <a href="https://developer.hpe.com/slack-signup/" style="font-size: 22px;">Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C099G0ZBZTJ" style="font-size: 22px;">#hpe-complete-care-itops</a> Slack channel.</p>https://developer.hpe.com/chapel/home/https://developer.hpe.com/chapel/home/<h1>What is Chapel?</h1> <p>Chapel is a programming language designed for productive parallel computing on large-scale systems. Chapel's design and implementation have been undertaken with portability in mind, permitting Chapel to run on multicore desktops and laptops, commodity clusters, and the cloud, in addition to the high-end supercomputers for which it was designed. Chapel's design and development are being led by Cray Inc. in collaboration with contributors from academia, computing centers, industry, and the open-source community.</p> <p>Chapel supports a multithreaded execution model via high-level abstractions for data parallelism, task parallelism, concurrency, and nested parallelism. Chapel's locale type enables users to specify and reason about the placement of data and tasks on a target architecture in order to tune for locality and affinity. Chapel supports global-view data aggregates with user-defined implementations, permitting operations on distributed data structures to be expressed in a natural manner. In contrast to many previous higher-level parallel languages, Chapel is designed around a multiresolution philosophy, permitting users to initially write very abstract code and then incrementally add more detail until they are as close to the machine as their needs require. Chapel supports code reuse and rapid prototyping via object-oriented design, type inference, and features for generic programming. Existing code can be integrated into Chapel programs (or vice-versa) via interoperability features.</p> <p>Chapel was designed from first principles rather than by extending an existing language. It is an imperative block-structured language, designed to be easy to learn for users of Python, C, C++, Fortran, Java, Matlab, and the like. While Chapel builds on concepts and syntax from many previous languages, its parallel features are most directly influenced by ZPL, High-Performance Fortran (HPF), and the Cray MTA™/Cray XMT™ extensions to C and Fortran.</p> <ul> <li><a href="https://chapel-lang.org/docs/examples/index.html">Try Chapel Tutorials for Yourself</a></li> <li><a href="https://chapel-lang.org/">Learn more about Chapel</a></li> <li><a href="https://chapel-lang.org/publications/PMfPC-Chapel.pdf">Read the Introduction to Chapel Whitepaper</a></li> <li><a href="https://chapel-lang.org/performance.html">See Chapel's Performance</a></li> <li><a href="https://chapel.discourse.group/c/newsletters">Check out Chapel Newsletters</a></li> </ul> <p><a href="https://github.com/chapel-lang/chapel">Join HPE and the active Chapel community on GitHub <img src="Github" alt=""></a></p> <hr> <h1>Projects Powered by Chapel</h1> <p><a href="https://github.com/mhmerrill/arkouda">Arkouda <img src="Github" alt=""></a></p> <p>Arkouda allows a user to interactively issue from the Python3 interpreter massively parallel computations on distributed data using functions and syntax that mimic NumPy, the underlying computational library used in the vast majority of Python data science workflows. The computational heart of Arkouda is a Chapel interpreter that accepts a pre-defined set of commands from a client (currently implemented in Python) and uses Chapel's built-in machinery for multi-locale and multithreaded execution. Arkouda has benefited greatly from Chapel's distinctive features and has also helped guide the development of the language.</p> <p><a href="https://www.polymtl.ca/expertises/en/laurendeau-eric">CHAMPS <img src="Github" alt=""></a></p> <p>CHAMPS is multi-physics software oriented toward fluid dynamics, written completely in Chapel and developed to solve systems of equations in a general manner. A key theme is to easily expand the capabilities of the software while keeping good performance on distributed memory. The software currently handles 2D and 3D unstructured grids to solve RANS equations with a finite volume approach using the Spalart-Allmaras turbulence model for closure. Different spatial discretization schemes and linear solvers can be used, including a variety of solvers from the PETSc library for which an API has been developed. Other C libraries for which APIs were developed in this project include the CGNS (CFD General Notation System) library, the Intel MKL library and the METIS library. The overall performance achieved with Chapel is comparable to equivalent C/C++/MPI approaches. Future developments will include multi-fidelty simulations in aerodynamics, aero-elasticity, and aero-icing.</p> <p><a href="https://github.com/pnnl/chgl">CHGL <img src="Github" alt=""></a></p> <p>The Chapel Hypergraph Library (CHGL) is a library for hypergraph computation in the Chapel language. Hypergraphs generalize graphs, where a hypergraph edge can connect any number of vertices. Thus, hypergraphs capture high-order, high-dimensional interactions between multiple entities that are not directly expressible in graphs. CHGL is designed to provide HPC-class computation with high-level abstractions and modern language support for parallel computing on shared- and distributed memory systems.</p> <p><a href="https://www.sciencedirect.com/science/article/abs/pii/S0167739X1930946X">ChOp <img src="Github" alt=""></a></p> <p>This project aims at programming distributed algorithms for solving big instances of combinatorial optimization problems, taking into account productivity, parallel efficiency, heterogeneity, and fault tolerance. This project comprises heuristic and exact optimization algorithms, and its main application is a distributed Branch-and-Bound for solving permutation-based combinatorial problems.</p> <p><a href="https://github.com/sourceryinstitute/PAW/raw/gh-pages/PAW-ATM19/extendedAbstracts/PAW-ATM2019_abstract2.pdf">ChplUltra <img src="Github" alt=""></a></p> <p>chplUltra is designed to simulate the dynamics of ultra-light dark matter for astrophysics. Ultralight dark matter is a relatively new proposal designed to alleviate some of the challenges faced by the more traditional WIMP (weakly interacting massive particles) candidates. It has a rich phenomenology, including the formation of Bose-Einstein condensate solitons and interference effects from the wave-like behavior of the particles. chplUltra is a pseudo-spectral fixed grid code designed to evolve the Schrodinger-Poisson equations. The code uses a Chapel distributed FFT routine, built around the serial FFTW library. Using Chapel allows the authors to rapidly extend the code and to simultaneously scale it out to ~100s of nodes. The code has been run on up to 512 nodes (18k cores) on a Cray XC system.</p> <p><a href="https://cray.github.io/crayai/">CrayAI <img src="Github" alt=""></a></p> <p>CrayAI is a suite of distributed machine learning workflow libraries designed with HPC in mind. These libraries are portable, running on anything from a laptop up to a supercomputer. The core back-end of these libraries is written in Chapel, while the user-facing interface is Python. CrayAI currently consists of Cray HPO and Cray FS. Cray HPO is a distributed black-box hyperparameter optimization framework and Cray FS is a distributed feature selection library.</p> <hr> <h1>Any questions on Chapel?</h1> <p>Join the conversation by chat on our <a href="https://gitter.im/chapel-lang/chapel">Chapel Gitter Channel</a> or by posting to our <a href="https://chapel.discourse.group/latest">Chapel Discourse Forum</a>.</p>https://developer.hpe.com/determined-ai/home/https://developer.hpe.com/determined-ai/home/<p>Building and training optimized models at scale is considered the most demanding and critical stage of machine learning (ML) development. <a href="https://www.determined.ai/">Determined AI</a> accelerates time-to-production with an open-source platform to build and train models faster and easier.</p> <p>Determined AI helps researchers and scientists focus on innovation and accelerate their time-to-delivery by removing the complexity and cost associated with machine learning development. This includes making it easy to set-up, configure, manage and share workstations or AI clusters that run on-premise or in the cloud.</p> <p>Determined enables you to:</p> <ul> <li> <p><strong>Train models faster</strong> using state-of-the-art distributed training without changing your model code.</p> </li> <li> <p><strong>Automatically find high-quality models</strong> with advanced hyperparameter tuning from the creators of Hyperband.</p> </li> <li> <p><strong>Get more from your GPUs</strong> with smart scheduling and cut cloud GPU costs by seamlessly using spot instances.</p> </li> <li> <p><strong>Track and reproduce your work</strong> with experiment tracking that works out-of-the-box, covering code versions, metrics, checkpoints, and hyperparameters.</p> </li> </ul> <p>Determined integrates these features into an easy-to-use, high-performance machine learning environment — which means you can spend your time building models instead of managing infrastructure.</p> <p>Since the open-source project launched in 2020, and as a result of its focus on model training, Determined AI has quickly emerged as a leading training tool in the evolving machine learning software ecosystem. Its solution has been adopted by users across a wide range of industries, such as biopharmaceuticals, autonomous vehicles, defense contracting, and manufacturing.</p> <p>Call-to-action: <a href="https://docs.determined.ai/latest/#get-started-locally">Get Started With Determined</a></p> <h1>Projects</h1> <h3><a href="https://github.com/determined-ai/determined">Determined <img src="Github" alt=""></a></h3> <p>Determined is an open-source deep learning training platform that makes building models fast and easy.</p> <ul> <li><a href="https://docs.determined.ai/latest/">Explore the Determined AI Documentation</a></li> </ul> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based workshops available in the HPE Developer <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on Determined AI or about the Determined open-source project?</h2> <p>Join the <a href="https://determined-community.slack.com/join/shared_invite/zt-1f4hj60z5-JMHb~wSr2xksLZVBN61g_Q#/shared-invite/email">Determined AI Slack Workspace</a> and start a discussion.</p>https://developer.hpe.com/dragonhpc/home/https://developer.hpe.com/dragonhpc/home/<p>DragonHPC is a programmable distributed runtime for HPC &#x26; AI workflows. The DragonHPC distributed runtime will power complex post-exascale workflows that can scale, that are cloud-native, that can access data efficiently, that can span multiple systems, and that can be used with multiple languages on heterogeneous hardware.</p> <h2>Key advantages of using DragonHPC</h2> <p>Among other frameworks and libraries, DragonHPC addresses much of the friction developers and scientists face when approaching a computational problem:</p> <ul> <li> <p>Program across widely distributed resources (e.g., laptops, servers, cloud, supercomputers) as if they were all one computer.</p> </li> <li> <p>Implement applications and workflows across Python, C, C++, and Fortran with interoperable high-performance objects instead of being locked into a single language.</p> </li> <li> <p>Implement Python-based applications and workflows using the standard multiprocessing API instead of non-standard interfaces.</p> </li> <li> <p>Develop applications and highly dynamic workflows without the limitations of static execution graphs.</p> </li> <li> <p>Orchestrate individual or collections of Python functions, binaries, and MPI processes.</p> </li> <li> <p>Leverage RDMA and leading-edge HPC communication techniques through simple high-level interfaces and communication objects instead of communication through a filesystem.</p> </li> </ul> <p>Users can also access DragonHPC at any level of its architecture. This allows users to develop new components within Dragon that natively interoperate with existing components. Flexibility, composability, and the ability to adopt DragonHPC for applications, tools, and workflows with strict performance requirements dramatically improves user productivity.</p> <h2>Online resources for getting up and running with DragonHPC</h2> <p><a href="https://github.com/DragonHPC/dragon">Public open source code repository with example use cases <img src="Github" alt=""></a></p> <p><a href="https://dragonhpc.github.io/dragon/doc/_build/html/index.html">Public open source code repository <img src="Github" alt=""></a></p>https://developer.hpe.com/grommet/home/https://developer.hpe.com/grommet/home/<h2>Build awesome apps with Grommet</h2> <p><a href="https://v2.grommet.io/">Grommet</a> helps you create responsive and accessible mobile-first projects for the web with an <a href="https://v2.grommet.io/components">easy-to-use</a>, <a href="https://reactjs.org/">react</a>-based component library that is part design system and part development framework.</p> <p>Grommet is used by developers and designers alike to build both enterprise-class and consumer-grade applications that can be used in web, desktop, and mobile-friendly formats. It underpins many of HPE’s products such as <a href="https://developer.hpe.com/greenlake/hpe-greenlake-platform/home/">HPE GreenLake edge-to-cloud platform</a>.</p> <ul> <li><a href="https://v2.grommet.io/components">View Grommet Components</a></li> <li><a href="https://grommet.slack.com/">Join the conversation on Slack</a></li> </ul> <p><a href="https://github.com/grommet/grommet">Join HPE and the active Grommet Community on GitHub <img src="Github" alt=""></a></p> <hr> <h2>What’s new with Grommet v2?</h2> <p>HPE product designer at grommet.io, Chris Carlozzi, explains how Grommet is evolving.</p> <ul> <li><a href="https://medium.com/grommet-io/whats-new-with-grommet-2-2f1883a91acb">Read his blog post</a></li> <li><a href="https://www.youtube.com/watch?v=WOy7qdNN1Fg&#x26;t=5108s">Watch Grommet v2 Launch</a></li> </ul> <h2>Learn more about Grommet</h2> <ul> <li><a href="https://v2.grommet.io/">Go to the Grommet Homepage</a></li> <li><a href="https://codesandbox.io/s/github/grommet/grommet-sandbox?initialpath=box&#x26;module=%2Fsrc%2FBox.js">Use Grommet in CodeSandBox</a></li> <li><a href="https://storybook.grommet.io/?path=/story/components--all">Access the Grommet Storybook</a></li> <li><a href="https://v2.grommet.io/play">Check out the Grommet PlayGround</a></li> </ul> <hr> <h2>HPE Design System</h2> <p>The HPE Design System demonstrates how grommet can be themed and used to build user interfaces with your own brand. The HPE Design System is used within HPE to guide the design of the user interfaces that HPE creates.</p> <ul> <li><a href="https://design-system.hpe.design/">Go to the HPE Design System Homepage</a></li> </ul> <hr> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. Grommet workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on Grommet?</h2> <p>Join the <a href="https://grommet.slack.com/">Grommet Slack Workspace</a> and start a discussion.</p>https://developer.hpe.com/hpe-alletra/home/https://developer.hpe.com/hpe-alletra/home/<p>Shed the complexity and silos inherent in conventional hybrid cloud environments with category-defining cloud-native data infrastructure that delivers a cloud operating and consumption experience wherever data lives. Stop managing infrastructure — and start simply accessing and utilizing it, as a service and on demand.</p> <ul> <li>Get the agility of cloud — everywhere</li> <li>Run any app — without compromise</li> <li>Free your data across hybrid cloud</li> </ul> <h1>HPE Alletra Storage MP B10000 Web Service API v3</h1> <p>The HPE Alletra Storage MP B10000 platform offers a rich set of REST APIs to manage the system configuration, provision storage, and run other administrative operations.  See the links below for additional information about the REST API and how to use them:</p> <ul> <li><a href="https://www.hpe.com/support/AlletraMP-B10000-WSAPIV3-devguide">HPE Alletra Storage MP B10000: Web Services API Developer Guide v3</a></li> <li><a href="/ws/api-spec">HPE Alletra Storage MP B10000: Web Services API Documentation and v3 OpenAPI specification</a></li> <li><a href="https://www.hpe.com/psnow/doc/a00148521enw">HPE Alletra Storage MP B10000 Web Services API v3 FAQ</a></li> </ul> <p>Join the HPE DEV slack channel <a href="https://hpedev.slack.com/archives/C08URLVQRRR">#hpe-alletra-b10k-api</a> to start a discussion</p> <h1>Projects</h1> <p>Plugins, SDKs and documentation.</p> <h2><a href="https://github.com/hpe-storage/csi-driver">CSI Driver <img src="Github" alt=""></a></h2> <p>A Container Storage Interface (CSI) Driver for Kubernetes. The HPE CSI Driver for Kubernetes allows you to use a Container Storage Provider to perform data management operations on storage resources.</p> <ul> <li><a href="https://developer.hpe.com/api/hpe-nimble-csp/">View the API documentation</a></li> <li><a href="https://artifacthub.io/packages/helm/hpe-storage/hpe-csi-driver">Helm Chart</a></li> <li><a href="https://artifacthub.io/packages/olm/community-operators/hpe-csi-operator">Operator for Kubernetes</a></li> <li><a href="https://access.redhat.com/containers/#/registry.connect.redhat.com/hpestorage/csi-driver-operator">Operator for OpenShift</a></li> <li><a href="https://scod.hpedev.io/csi_driver/index.html">Visit documentation on SCOD</a></li> </ul> <h2><a href="https://github.com/hpe-storage/scod">Storage Container Orchestrator Documentation <img src="Github" alt=""></a></h2> <p>The definitive source for end-user documentation using Kubernetes and neighboring partner ecosystems with HPE Alletra.</p> <ul> <li><a href="https://scod.hpedev.io/">Explore the SCOD portal</a></li> </ul> <h2><a href="https://github.com/hpe-storage/array-exporter">Prometheus Array Exporter <img src="Github" alt=""></a></h2> <p>A Prometheus array exporter that may be deployed as a standalone binary or directly on Kubernetes. There's also an exporter for the CSI driver that may be deployed separately.</p> <ul> <li><a href="https://hpe-storage.github.io/array-exporter">Read the documentation</a></li> <li><a href="https://scod.hpedev.io/csi_driver/metrics.html">Learn about the CSI info metrics provider on SCOD</a></li> </ul> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. A CSI workshop for HPE Alletra is available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on HPE Alletra?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPE DEV Slack Workspace</a> and start a discussion in the <a href="https://hpedev.slack.com/archives/C025D75HHGC">#alletra</a> channel.</p>https://developer.hpe.com/hpe-3par-and-primera/home/https://developer.hpe.com/hpe-3par-and-primera/home/<p>Power your mission-critical apps with extreme resiliency and unprecedented simplicity</p> <ul> <li>App-aware resiliency sees beyond the walls of storage to predict/prevent disruptions</li> <li>Predictive acceleration safely consolidates every mission-critical app onto the same platform with extreme low-latency performance</li> </ul> <h2>Projects</h2> <p>SDKs, Plugins and Language Bindings</p> <h3><a href="https://github.com/HewlettPackard/hpe3par_pstoolkit">PowerShell Toolkit <img src="Github" alt=""></a></h3> <p>HPE 3PAR and Primera StoreServ Storage PowerShell Toolkit provides storage administrators the convenience of managing HPE 3PAR StoreServ Storage Systems from a Microsoft Windows PowerShell environment.</p> <ul> <li><a href="https://myenterpriselicense.hpe.com/cwp-ui/free-software/3PARPSToolkit">View Product Details</a></li> </ul> <h3><a href="https://github.com/HewlettPackard/hpe3par_pstoolkit">Chef <img src="Github" alt=""></a></h3> <p>Chef Cookbook and examples for HPE 3PAR StoreServ.</p> <ul> <li><a href="https://supermarket.chef.io/cookbooks/hpe3par">Go to Chef Supermarket</a></li> </ul> <h3><a href="https://github.com/HewlettPackard/hpe3par_ansible_module">Ansible <img src="Github" alt=""></a></h3> <p>The HPE 3PAR and Primera modules for Ansible to enable automation of storage provisioning for the HPE 3PAR and Primera StoreServ array.</p> <h3><a href="https://github.com/HewlettPackard/hpe3par_puppet_module">Puppet <img src="Github" alt=""></a></h3> <p>Puppet module and examples for HPE 3PAR StoreServ.</p> <ul> <li><a href="https://forge.puppet.com/modules/hewlettpackardenterprise/hpe3par">Go to Puppet Forge</a></li> </ul> <h3><a href="https://github.com/hpe-storage/array-exporter">Prometheus Array Exporter <img src="Github" alt=""></a></h3> <p>A Prometheus array exporter that may be deployed as a standalone binary or directly on Kubernetes. There's also an exporter for the CSI driver that may be deployed separately.</p> <ul> <li><a href="https://hpe-storage.github.io/array-exporter">Read the documentation</a></li> <li><a href="https://scod.hpedev.io/csi_driver/metrics.html">Learn about the CSI info metrics provider on SCOD</a></li> </ul> <h3><a href="https://github.com/hpe-storage/python-hpedockerplugin">Docker <img src="Github" alt=""></a></h3> <p>HPE Docker Volume Plugin, provides persistent block storage for containerized applications using HPE 3PAR and Primera StoreServ.</p> <ul> <li><a href="https://hub.docker.com/r/hpestorage/legacyvolumeplugin">Go to Docker Store</a></li> </ul> <h3><a href="https://github.com/HewlettPackard">Language SDKs <img src="Github" alt=""></a></h3> <p>Client libraries in different languages which provide access to the HPE 3PAR array over WSAPI.</p> <ul> <li><a href="https://github.com/hpe-storage/python-3parclient">Go to Python Client</a></li> <li><a href="https://github.com/HewlettPackard/hpe3par_ruby_sdk">Go to Ruby Client</a></li> </ul> <h3><a href="https://github.com/hpe-storage/">OpenStack <img src="Github" alt=""></a></h3> <p>The HPE 3PAR and Primera Cinder storage driver for use with Openstack implementations.</p> <ul> <li><a href="https://docs.openstack.org/cinder/pike/configuration/block-storage/drivers/hpe-3par-driver.html">Go to Openstack Driver</a></li> </ul> <hr> <h2>Any questions on 3PAR and Primera?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/CRU01FTRS">#3par-primera</a> channel.</p>https://developer.hpe.com/hpe-cray-programming-environment/home/https://developer.hpe.com/hpe-cray-programming-environment/home/<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>HPE Cray Programming Environment (CPE) suite offers programmers a comprehensive set of tools for developing, porting, debugging, and tuning applications. The programming environment simplifies the transition to new hardware architectures and configurations by automatically applying optimizations on HPC applications that use existing programming models with a simple recompile.</p> <p>The environment provides:</p> <ul> <li> <p>User environment (compiler drivers, hugepages, craype-api)</p> </li> <li> <p>Compilers, programming languages, and models</p> </li> <li> <p>Scalable communication libraries</p> </li> <li> <p>Scientific and math libraries</p> </li> <li> <p>Debugging tools</p> </li> <li> <p>Profiling and performance optimization tools</p> </li> </ul> <p>Learn more about the HPE Cray Programming Environment components by exploring the <a href="https://cpe.ext.hpe.com/docs/">CPE documentation</a>.</p> <h2>HPE Cray user environment</h2> <p>Our user environment provides libraries that support code compilation and development environment setup. It includes compiler drivers, the hugepages utility, and the CPE API (craype-api).</p> <p>PrgEnv modules provide wrappers (cc, CC, ftn) for both CCE and third-party compiler drivers. These wrappers call the correct compiler with appropriate options to build and link applications with relevant libraries as required by loaded modules.  Note that only dynamic linking is supported. These wrappers replace direct calls to compiler drivers in Makefiles and build scripts.</p> <p>Hugepages: Standard Linux operating system typically supports base pages of size 4KiB. For high performance parallel workloads that involve data movement operations across a high-speed network, larger page sizes generally yield improved communication performance. On HPE Cray EX supercomputing systems, the Cray Operating System (COS) supports non-standard large page sizes to offer an additional layer of optimization. HPE CPE software stack is tightly integrated with COS and exposes this feature via craype-hugepages modules. Users can leverage this important optimization by loading specific craype-hugepages modules in their build and runtime environments.</p> <p>CrayPE API (cray-api) provides software package integration with enhanced control when building multiple versions of a software product modulefile.</p> <h2>HPE Cray Compiling Environment</h2> <p>Our Fortran, C, and C++ compilers are designed to help extract maximum performance from the systems regardless of the underlying architecture supporting ARM and x86-64 (Intel and AMD) processors, as well as AMD and NVIDIA accelerators.</p> <p>The compilers identify regions of computation that are either sequential scalar or vector parallel, and automatically exploit these capabilities for the targeted system.</p> <p>The compilers give programmers optimization feedback with an annotated listing of source code for easier application tuning. Our compilers integrate with debuggers and performance tools in the HPE Cray Programming Environment suite—enhancing each other's capability to generate correct and performant code.</p> <p>The suite also includes integration with GNU, Intel, AMD, and NVIDIA programming environments, so developers can choose between multiple compilers and still use the libraries, debuggers, and performance analysis tools included in our suite to help optimize application performance.</p> <p>We focus on standards compliance for code safety, application portability, and investment protection. Our compilers support standard programming languages (Fortran, C/C++, and UPC) and standard programming models such as OpenMP and OpenACC.</p> <h2>HPE Cray Message Passing Toolkit (CMPT)</h2> <p>CMPT is a collection of libraries that provide portable, efficient, and flexible mechanisms for performing data transfers between parallel processes. It comprises HPE Cray MPI, HPE Cray OpenSHMEMX, HPE Cray PMI, and HPE Cray DSMML libraries.</p> <p>HPE Cray MPI is an MPICH ABI compatible library tuned for Intel, AMD, and ARM CPUs as well as AMD and NVIDIA GPUs. It is a highly scalable implementation, customized for low latency and high bandwidth, both on-node and off-node, for point-to-point and collective communications. It is also highly optimized and tuned for the HPE Slingshot network architecture. Strategic optimizations for MPI I/O, MPI_THREAD_MULTIPLE, remote memory access (RMA), and integration with the performance analysis tools in the suite contribute to deliver ideal application performance for today's HPC codes.</p> <p>HPE Cray OpenSHMEMX is a highly scalable implementation of the OpenSHMEM standards library interface specification. It supports Partitioned Global Address Space (PGAS) style of programming to cater to the needs of distributed memory applications with highly irregular and random communication models. It is optimized specifically for system architectures involving Intel, AMD, and ARM CPUs. HPE Cray OpenSHMEMX is tuned for exposing the rich set of features available in HPE Slingshot interconnect through operations like remote memory access (RMA), atomic memory updates, put-with-signal, scalable-collective communication, memory ordering, and effective multithreading.</p> <h2>HPE Cray Scientific and Math Libraries</h2> <p>This suite offers a comprehensive collection of highly tuned linear algebra subroutines designed to help extract maximum performance from the system with the least amount of effort.</p> <p>Customized Cray LibSci (including optimized BLAS, LAPACK, ScaLAPACK, and IRT), Cray LibSci_ACC (GPU accelerated BLAS, and LAPACK), and Cray FFTW (optimized fast Fourier transform routines) are designed to take full advantage of the underlying hardware, optimizing for both intra-node and inter-node performance on all HPE HPC systems.</p> <p>The libraries are highly tuned and optimized to select performant algorithms at runtime for a variety of HPE HPC systems. They also feature simplified interfaces into complex software (no source code changes required to access optimized algorithms) and integrate with the HPE Cray Compiling Environment for better productivity.</p> <p>NetCDF, HDF5, and Parallel NetCDF I/O libraries are built with the supported compiling environments and are integrated with the HPE Cray Compiling Environment.</p> <h2>Debugging tools</h2> <p>The HPE Cray Programming Environment offers traditional debuggers combined with new innovative techniques. Together, these technologies allow users to address debugging problems at a broader range and scale than conventional techniques. This means that programmers can spend less time debugging and more time creating. Included are:</p> <ul> <li> <p>Comparative Debugger: This market-unique tool helps programmers uncover issues by running two applications side by side. If the values of the selected data structures diverge, the user is notified that an error may exist. This capability is useful for locating errors that are introduced when applications are modified through code, compiler, or library changes, and for application porting between architectures (for example, between CPUs and GPUs) or programming models.</p> </li> <li> <p>GDB for HPC is based on the popular GDB command-line debugger used to debug applications compiled with Fortran, C, and C++ compilers with enhancements to provide a GDB debugging experience for applications that run at scale across many nodes. The tool enables users to run a traditional scalable debugging session—either by launching an application or by attaching it to an already-running application. A GDB for HPC debug session retrieves debug information from thousands of processes and presents merged backtraces and data, removing vast amounts of duplicate information.</p> </li> <li> <p>Valgrind for HPC: Parallel memory analysis tool based on Valgrind debugger used for applications compiled with Fortran, C, and C++ compilers—it aggregates common errors into a single output record for easier analysis of potential memory problems within applications that run at scale.</p> </li> <li> <p>Stack Trace Analysis Tool (STAT): Helps developers identify if an application is hung or still making progress when running. Generates a merged backtrace for applications so users can get a better insight into application behavior at a function level.</p> </li> <li> <p>Tool for Abnormal Termination Processing (ATP): When an application crashes, the tool detects a signal and generates a merged backtrace resulting in a minimal core file set so that programmers do not have to plough through an enormous number of core files when debugging the application.</p> </li> <li> <p>Sanitizers for HPC: Help developers detect memory and thread errors for easier analysis and debugging of their applications at scale by aggregating and analyzing output of LLVM sanitizers at scale.</p> </li> </ul> <p>We also offer support for traditional debugging mechanisms via integration with TotalView by Perforce and Arm Forge.</p> <h2>Profiling and performance optimization tools</h2> <p>Comprehensive collection of tools designed to reduce the time and effort associated with porting and tuning of applications on HPE and HPE Cray systems. We offer different tools and experiments to fit different developer needs and choice of interfaces for ease of use.</p> <ul> <li>Performance analysis tool (PAT) brings valuable insight when analyzing bottlenecks to improve performance of applications that run across the whole system. The tool exposes a wide set of indicators, such as computation, communication, I/O, and memory statistics and displays a program’s top time consumers and bottlenecks (via unique and critical load balance analysis) for jobs at scale. It then automatically generates observations and suggestions to improve code performance.</li> </ul> <p>As ease of use is an important feature of the tool suite, both simple and advanced interfaces are available, offering both a simple path to get started and a wealth of capability for analyzing the most complex codes.</p> <ul> <li> <p>Programmers can quickly assess the type and severity of performance issues by using our visualization tool, which complements text reports and summarizes programs’ performance data in graphs and charts, allowing users to easily drill down to get to the bottom of issues.</p> </li> <li> <p>Code parallelization assistant helps developers reveal hidden potential of their application via code restructuring. The tool extends our existing performance analysis and visualization technology by combining performance statistics and program source code visualization with our compiling environment optimization feedback. This tool can easily navigate through source code to highlight dependencies or bottlenecks during the optimization phase of program development or porting.</p> </li> </ul> <p>Using the program library provided by our compiling environment and the performance data collected by our performance, measurement, and analysis tools, users can navigate through their source code to understand which high-level loops could benefit from OpenMP parallelism.</p> <hr> <h1>Any questions on HPE Cray Programming Environment?</h1> <p>Join the <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C04TG4XJBL7">#hpe-cray-programming-environment</a> channel.</p>https://developer.hpe.com/hpe-deep-learning-cookbook/home/https://developer.hpe.com/hpe-deep-learning-cookbook/home/<p>It is a common wisdom today, that to start a deep learning exploration one needs a GPU-enabled system and one of the existing open source deep learning frameworks. But which GPU box to choose? How many GPUs to put in a system? How many systems to put in a cluster and which interconnect to use? Which framework to pick? Answers to these questions are not obvious. That’s why we decided to create HPE Deep Learning Cookbook – a set of tools to characterize deep learning workloads and to recommend optimal hardware/software (HW/SW) stack for any given workload. Our Cookbook consists of several key assets:</p> <p><strong>HPE Deep Learning Benchmarking Suite:</strong> automated benchmarking tool to collect performance measurements on various HW/SW configurations in a unified way.</p> <p><strong>HPE Deep Learning Performance Guide:</strong> a web-based tool which provides access to a knowledge base of benchmarking results. It enables querying and analysis of measured results as well as performance prediction based on analytical performance models.</p> <p><strong>Reference Designs:</strong> hardware/software recipes for selected workloads.</p> <p>Recommendations with our Deep Learning Cookbook are based on a massive collection of performance results for various deep learning workloads on different HW/SW stacks, and analytical performance models. The combination of real measurements and analytical performance models enables us to estimate the performance of any workload and to recommend an optimal hardware/software stack for that workload. Additionally, we use the Cookbook internally to detect bottlenecks in existing hardware and to guide the design of future systems for artificial intelligence and deep learning.</p> <ul> <li> <p><a href="https://community.hpe.com/t5/Behind-the-scenes-Labs/The-Deep-Learning-Cookbook/ba-p/6967323#.WhX-xVWnFhF">Read a blog post about the HPE Deep Learning Cookbook</a></p> </li> <li> <p><a href="https://youtu.be/ao_DeE9lxvk">Watch a short video introducing the HPE Deep Learning Cookbook</a></p> </li> </ul> <hr> <h1>Components of the Cookbook</h1> <h3><a href="https://github.com/HewlettPackard/dlcookbook-dlbs">HPE Deep Learning Benchmarking Suite <img src="Github" alt=""></a></h3> <p>An automated benchmarking tool, which makes it easy to run performance tests with most popular deep learning frameworks. It enables consistent and reproducible benchmark experiments on various hardware/software combinations.</p> <ul> <li><a href="https://hewlettpackard.github.io/dlcookbook-dlbs/#/index?id=deep-learning-benchmarking-suite">Read the documentation</a></li> </ul> <h3>HPE Deep Learning Performance Guide</h3> <p>A web-based tool which provides access to a knowledgebase of benchmarking results.</p> <ul> <li><a href="https://dlpg.labs.hpe.com/">Check the tool</a></li> </ul> <hr> <h1>Related resources</h1> <p>Characterisation and Benchmarking of Deep Learning</p> <p>A talk at HPC User Forum introducing HPE Deep Learning Cookbook</p> <ul> <li><a href="https://www.google.com/url?sa=t&#x26;rct=j&#x26;q=&#x26;esrc=s&#x26;source=web&#x26;cd=1&#x26;cad=rja&#x26;uact=8&#x26;ved=0ahUKEwi_78-1tdPXAhUF4WMKHf8CCoQQtwIIKDAA&#x26;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlgK0BlXdOCw&#x26;usg=AOvVaw0osgSwEOQqZ4Gg9cVx3b-r">Watch the talk</a></li> </ul> <p>HPE Introduces New Set of Artificial Intelligence Offerings</p> <ul> <li><a href="https://news.hpe.com/hpe-introduces-new-set-of-artificial-intelligence-platforms-and-services/">Read the press release</a></li> </ul>https://developer.hpe.com/hpe-ezmeral/home/https://developer.hpe.com/hpe-ezmeral/home/<p><a id="top"></a></p> <p>HPE Ezmeral Software is a unified platform that streamlines AI and GenAI development and deployment. Its open architecture and flexible deployment options empower organizations to efficiently build and deploy AI solutions across hybrid cloud environments. By providing a comprehensive suite of tools for data management, model training, and inference, HPE Ezmeral Software helps address common AI challenges such as data heterogeneity, security compliance, scalability, and management of a diverse ecosystem of AI tools.</p> <p>HPE Ezmeral Software is composed of two components:</p> <p><a href="http://www.hpe.com/datafabric">HPE Ezmeral Data Fabric</a> is a centralized data management platform that simplifies access, governance, and security across diverse data types. By federating data silos enabling instant access, HPE Ezmeral Data Fabric empowers organizations to unlock the full potential of their data for AI data-driven initiatives.</p> <p><a href="https://www.hpe.com/us/en/hpe-ezmeral-unified-analytics.html">HPE Ezmeral Unified Analytics</a> is a comprehensive platform that simplifies AI and ML development, deployment, and management across hybrid multicloud environments. By providing self-service access to diverse data sources and a unified set of tools, HPE Ezmeral Unified Analytics accelerates time-to-value for AI initiatives. Its scalable architecture and support for popular open-source frameworks enable organizations to train, tune, and deploy models efficiently, while its centralized monitoring capabilities ensure optimal performance and governance.</p> <h2>Any questions on HPE Ezmeral Software?</h2> <p>Join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C01BB50LG4W">#hpe-ezmeral-runtime-enterprise</a>, <a href="https://hpedev.slack.com/archives/C055D1EECAK">#hpe-ezmeral-unified-analytics-software</a>, and <a href="https://hpedev.slack.com/archives/CU3JRBTB7">#hpe-ezmeral-data-fabric-software</a> Slack channels.</p>https://developer.hpe.com/hpe-ezmeral-data-fabric/home/https://developer.hpe.com/hpe-ezmeral-data-fabric/home/<p>HPE Ezmeral Data Fabric centralizes different data types across on-premises, multiple clouds, and edge deployments into a single logical data store. It breaks down silos and stops the endless cycle of merging, remerging, and redeploying data silos into new silos.</p> <p>HPE Ezmeral Data Fabric allows data analysts, engineers, and scientists to deliver real-time and predictive models organizations can trust. Built-in industry-standard APIs, languages, and protocols increase productivity by preserving existing access mechanisms without low-value refactoring.</p> <p>Supports files, object, NoSQL databases, and streams</p> <ul> <li><a href="https://docs.datafabric.hpe.com/70/MapROverview/File-Store.html#MapR-XD">Data Fabric file store</a></li> <li><a href="https://docs.datafabric.hpe.com/70/MapROverview/HPE-Ezmeral-Data-Fabric-Object-Store.html">Data Fabric object store</a></li> <li><a href="https://docs.datafabric.hpe.com/70/MapROverview/maprDB-overview.html#maprDB-overview">Data Fabric database</a></li> <li><a href="https://docs.datafabric.hpe.com/70/MapROverview/c_mapr_streams.html">Data Fabric data streams</a></li> </ul> <p><a href="https://docs.datafabric.hpe.com/70/c_ecosystem_intro.html">HPE Ezmeral Ecosystem Pack includes</a> certified open-source tools and engines that integrate directly on top of the data fabric. It enables in-place analytics no matter where data is located and reduces time spent integrating and configuring open source tools.</p> <p><a href="https://docs.datafabric.hpe.com/home/">API Documentation</a> for app development.</p> <hr> <h2>Tutorials on GitHub</h2> <ul> <li><a href="https://github.com/mapr-demos/mapr-music">Music Catalog: REST and GraphQL tutorial <img src="Github" alt=""></a><a href="https://github.com/mapr-demos/mapr-smart-home"></a></li> <li><a href="https://github.com/mapr-demos/mapr-smart-home">Smart Home: IoT tutorial <img src="Github" alt=""></a></li> <li><a href="https://github.com/mapr-demos/predictive-maintenance">Ezmeral Data Fabric for Predictive Maintenance <img src="Github" alt=""></a></li> <li><a href="https://github.com/mapr-demos/customer360">Customer 360 View <img src="Github" alt=""></a></li> <li><a href="https://github.com/mapr-demos/finserv-application-blueprint">Application for Processing Stock Market Trade Data <img src="Github" alt=""></a></li> </ul> <hr> <h2>Munch &#x26; Learn</h2> <h4>What's a data fabric and how does it work?</h4> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/qi6sTvu8osk" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <hr> <h2>YouTube Videos</h2> <h4>How to size a data fabric system</h4> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/6khp9SanXhY" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <h4>Practical Erasure Coding in a Data Fabric</h4> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/-6IBKLiOb_Q" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <h4>Data Fabric File and Object Store Overview</h4> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/S19rkDF_oPs" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <p>To learn more about HPE Ezmeral Data Fabric File and Object Store, check out the HPE article <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Data-Fabric-File-and-Object-Store-Benefits-and/ba-p/7168604#.YrHKV3ZByXI">here</a>.</p> <hr> <h2>Free On-Demand Training</h2> <p>Learn for free with online courses that teach you how to build applications, secure, and administer HPE Ezmeral. Visit <a href="https://learn.software.hpe.com/">HPE Ezmeral Learn On-Demand</a> to enroll.</p> <ul> <li><a href="https://learn.software.hpe.com/page/developer-essentials">Big Data Essentials</a>.  Challenges of modernizing the IT landscape is covered in these courses on big data, open source tools, Apache Spark, Data Fabric, Database, Event Streams and more.</li> <li><a href="https://learn.software.hpe.com/page/artificial-intelligence">Artificial Intelligence &#x26; Machine Learning</a>. Learn the basics of AI and ML, machine learning project planning, and preparing and implementing the ML pipeline.</li> <li><a href="https://learn.software.hpe.com/page/security">Zero Trust &#x26; Data Security</a>. Courses outlining emerging data security threats, SPIFFE-SPIRE, what zero trust architecture is, and technical details of HPE Ezmeral Data Fabric</li> <li><a href="https://learn.software.hpe.com/page/analytics">Apache Spark &#x26; SQL Analytics</a>. Intro courses covering Apache Spark 3 essentials, Spark and Kubernetes, Apache Drill, and SQL Analytics using Apache Drill.</li> <li><a href="https://learn.software.hpe.com/page/developer-essentials">Kubernetes and Stateful Applications</a>. What applications do, how Kubernetes works, plus access an HPE Workshop On-demand covering HPE Ezmeral Runtime.</li> <li><a href="https://learn.software.hpe.com/page/data-fabric">HPE Ezmeral Data Fabric</a>. Technical overview courses on HPE Ezmeral Data Fabric architecture and implementation, database essentials, and event streams pub-sub messaging.</li> </ul> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. HPE Ezmeral Data Fabric workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on Ezmeral Data Fabric?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/CU3JRBTB7">#ezmeral-data-fabric</a> channel.</p> <p>Not a Slack user? You can also ask your questions in our <a href="https://hpe.com/forum/ezmeral">Ezmeral Forum</a>.</p>https://developer.hpe.com/hpe-greenlake/home/https://developer.hpe.com/hpe-greenlake/home/<p>Powered by the HPE GreenLake edge-to-cloud platform, HPE GreenLake helps you achieve data-first modernization by bringing the cloud experience to your distributed apps and data wherever they are. With a pay per use, scalable, point &#x26; click self-service experience that is managed for you, HPE GreenLake can help you conserve your capital, reduce complexity, and achieve a faster time to market. You can run your business on HPE GreenLake.</p> <p><a href="https://www.hpe.com/us/en/greenlake.html">Learn more about HPE GreenLake</a></p> <h2>APIs and Documentation</h2> <p>HPE GreenLake customers and partners can take advantage of our well-documented, secure, and scalable framework of APIs for HPE GreenLake found in our developer portal. Learn more about the unified HPE GreenLake experience offered through the HPE GreenLake Developer Portal by clicking on the button below.</p> <p><a href="https://developer.greenlake.hpe.com">Check out the HPE GreenLake Developer Portal</a></p> <h2>Manage workloads with the HPE GreenLake edge-to-cloud platform</h2> <p><a href="https://www.hpe.com/us/en/greenlake/demos.html">Learn how with HPE GreenLake edge-to-cloud platform demos and resources</a></p> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. HPE GreenLake workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <h2>Any questions on HPE GreenLake?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#HPEGreenLake</a> channel.</p>https://developer.hpe.com/hpe-machine-learning-development-environment/home/https://developer.hpe.com/hpe-machine-learning-development-environment/home/<p>Machine learning (ML) engineers and data scientists are on a never-ending search for new solutions that will enable them to better focus on innovation and accelerate their time to production—and this is what <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> is all about.</p> <p>By removing the complexity and cost associated with ML model development, this comprehensive platform speeds time to value for model developers by:</p> <p>• Removing the need to write infrastructure code</p> <p>• Making it easier for IT administrators to set up, manage, secure, and share AI compute clusters</p> <p>With the HPE Machine Learning Development Environment, ML practitioners can:</p> <p>• Train models faster using state-of-the-art distributed training, without changing their model code</p> <p>• Automatically find high-quality models with advanced hyperparameter tuning from the creators of state-of-the-art tuning algorithms such as Hyperband</p> <p>• Get more from their GPUs with smart scheduling, as well as reduce cloud GPU costs by seamlessly using spot instances</p> <p>• Track and reproduce their work with experiment tracking that works out of the box, covering code versions, metrics, checkpoints, and hyperparameters</p> <p>Using a comprehensive array of features integrated into an easy-to-use, high-performance ML environment, ML engineers can focus on building better models, instead of managing IT infrastructure. Using the HPE Machine Learning Development Environment that supports both cloud and on-premises deployment infrastructure, practitioners can develop models using PyTorch, TensorFlow, or Keras. HPE Machine Learning Development Environment also integrates seamlessly with today’s most popular ML tools for data preparation and model deployment.</p> <p>HPE Machine Learning Development Environment is built upon the widely popular open source training platform, <a href="https://www.determined.ai/">Determined</a>. Check out <a href="https://developer.hpe.com/platform/determined-ai/home">these related articles</a> about Determined on the HPE Developer portal <a href="https://developer.hpe.com/platform/determined-ai/home">here</a>.</p> <p>We also invite you to check out the <a href="https://hpe-mlde.determined.ai/latest/">Documentation</a> for HPE Machine Learning Development Environment.</p>https://developer.hpe.com/hpe-nimble-storage/home/https://developer.hpe.com/hpe-nimble-storage/home/<style> .button { background-color: rgba(23,235,160,1); box-sizing: border-box; color: #000000; font-size: 18px; display: inline-block; padding: 6px 12px; vertical-align: middle; overflow: hidden; text-decoration: none; text-align: center; cursor: pointer; white-space: nowrap; border-radius: 4px; border: none; margin: 0; line-height: 24px; font-weight: 700; } </style> <p>HPE Nimble Storage customers and partners have full access to the REST API of the arrays. We also provide open source projects for various automation platforms, language SDKs and container ecosystems.</p> <ul> <li> <p><a href="https://infosight.hpe.com/InfoSight/media/cms/active/public/pubs_REST_API_Reference_NOS_53x.whz">HPE Nimble Storage REST API hosted on HPE InfoSight</a></p> </li> <li> <p><a href="https://infosight.hpe.com/InfoSight/media/cms/active/public/pubs_HPE_infosight_wellness_spec.pdf">Download the HPE InfoSight Wellness API (PDF)</a></p> </li> <li> <p><a href="https://docs.cloudvolumes.hpe.com/help/rest/api-overview/">Explore the HPE Cloud Volumes REST API</a></p> </li> </ul> <h1>Projects</h1> <p>Plugins, SDKs and documentation.</p> <h3><a href="https://github.com/hpe-storage/csi-driver">CSI Driver <img src="Github" alt=""></a></h3> <p>A Container Storage Interface (CSI) Driver for Kubernetes. The HPE CSI Driver for Kubernetes allows you to use a Container Storage Provider to perform data management operations on storage resources.</p> <ul> <li> <p><a href="https://developer.hpe.com/api/hpe-nimble-csp/">View the API documentation</a></p> </li> <li> <p><a href="https://hub.helm.sh/charts/hpe-storage/hpe-csi-driver">Helm Chart</a></p> </li> <li> <p><a href="https://operatorhub.io/operator/hpe-csi-driver-operator">Operator for Kubernetes</a></p> </li> <li> <p><a href="https://access.redhat.com/containers/#/registry.connect.redhat.com/hpestorage/csi-driver-operator">Operator for OpenShift</a></p> </li> <li> <p><a href="https://scod.hpedev.io/csi_driver/index.html">Visit documentation on SCOD</a></p> </li> </ul> <h3><a href="https://github.com/hpe-storage/scod">Storage Container Orchestrator Documentation <img src="Github" alt=""></a></h3> <p>The definitive source for end-user documentation for HPE storage integration with Kubernetes, Docker and neighboring partner ecosystems, including the HPE Container Platform.</p> <ul> <li><a href="https://scod.hpedev.io/">Explore the SCOD portal</a></li> </ul> <h3><a href="https://github.com/hpe-storage/array-exporter">Prometheus Array Exporter <img src="Github" alt=""></a></h3> <p>A Prometheus array exporter that may be deployed as a standalone binary or directly on Kubernetes. There's also an exporter for the CSI driver that may be deployed separately.</p> <ul> <li><a href="https://hpe-storage.github.io/array-exporter">Read the documentation</a></li> <li><a href="https://scod.hpedev.io/csi_driver/metrics.html">Learn about the CSI info metrics provider on SCOD</a></li> </ul> <h3><a href="https://github.com/hpe-storage/flexvolume-driver">Volume Driver for Kubernetes FlexVolume Plugin <img src="Github" alt=""></a></h3> <p>Legacy FlexVolume driver for Container Provider-based storage systems, Nimble Storage and Cloud Volumes, for integration with Kubernetes FlexVolume Plugin.</p> <ul> <li><a href="https://scod.hpedev.io/flexvolume_driver/container_provider/index.html">View documentation on SCOD</a></li> </ul> <h3><a href="https://github.com/hpe-storage/common-host-utils/tree/master/cmd/dockervolumed/managedplugin">Docker Volume plugin <img src="Github" alt=""></a></h3> <p>Comprehensive Docker Volume plugin that serves as a foundation for all major container orchestration frameworks, including Docker Swarm, Kubernetes and Mesos.</p> <ul> <li> <p><a href="https://store.docker.com/plugins/nimble">Visit on the Docker Store</a></p> </li> <li> <p><a href="https://scod.hpedev.io/docker_volume_plugins/hpe_nimble_storage/index.html">View documentation on SCOD</a></p> </li> </ul> <h3><a href="https://github.com/hpe-storage/nimble-python-sdk">SDK for Python <img src="Github" alt=""></a></h3> <p>A client Python Software Development Kit for HPE Nimble Storage arrays leveraging the REST APIs.</p> <ul> <li> <p><a href="https://pypi.org/project/nimble-sdk/">Check out the package on PyPi</a></p> </li> <li> <p><a href="https://hpe-storage.github.io/nimble-python-sdk/">View documentation</a></p> </li> </ul> <h3><a href="https://github.com/hpe-storage/nimble-golang-sdk">SDK for Go <img src="Github" alt=""></a></h3> <p>A client Go Software Development Kit for HPE Nimble Storage arrays leveraging the REST APIs.</p> <h3><a href="https://github.com/hpe-storage/nimble-ansible-modules">Ansible Content Collection <img src="Github" alt=""></a></h3> <p>HPE Nimble Storage Content Collection for Ansible is a certified collection of Ansible modules to manage HPE Nimble Storage array resources.</p> <ul> <li> <p><a href="https://galaxy.ansible.com/hpe/nimble">Download from Ansible Galaxy</a></p> </li> <li> <p><a href="https://hpe-storage.github.io/nimble-ansible-modules">Read the documentation</a></p> </li> <li> <p><a href="https://cloud.redhat.com/ansible/automation-hub/hpe/nimble/">View on Red Hat Automation Hub</a></p> </li> </ul> <h3>PowerShell Toolkit</h3> <p>Windows PowerShell scripting toolkit for HPE Nimble Storage arrays.</p> <ul> <li><a href="https://www.powershellgallery.com/packages/HPENimblePowerShellToolkit/3.0.0">Visit the PowerShell Gallery</a></li> </ul> <h3><a href="https://github.com/NimbleStorage/nimble-puppet">Puppet Module <img src="Github" alt=""></a></h3> <p>Puppet module to manage Nimble Storage arrays.</p> <ul> <li><a href="https://forge.puppet.com/nimblestorage/nimblestorage">View on Puppet Forge</a></li> </ul> <h3><a href="https://github.com/NimbleStorage/automation-examples">Storage and Infrastructure Automation <img src="Github" alt=""></a></h3> <p>This repository contains examples on how you can automate daunting management tasks for HPE Nimble Storage and adjacent technologies.</p> <h3><a href="https://github.com/NimbleStorage/Nemo">Nemo <img src="Github" alt=""></a></h3> <p>Nemo uses OpenZFS to emulate the snapshot and cloning capabilities of the HPE Nimble Storage Docker Volume Plugin.</p> <h3><a href="https://github.com/NimbleStorage/nimble-sap-hana-agent">Application Snapshot Agent for SAP HANA <img src="Github" alt=""></a></h3> <p>The HPE Nimble Storage Application Snapshot Agent for SAP HANA uses the HPE Nimble Storage Snapshot Framework to support application consistent and integrated storage level snapshots of SAP HANA.</p> <h3>OpenStack Cinder Driver</h3> <p>Use HPE Nimble Storage with OpenStack Cinder.</p> <ul> <li> <p><a href="https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/nimble-volume-driver.html">Documentation</a></p> </li> <li> <p><a href="https://opendev.org/openstack/cinder">View OpenStack Cinder source</a></p> </li> <li> <p><a href="https://github.com/NimbleStorage/nimble-fuel-cinder-plugin">OpenStack Fuel Plugin</a></p> </li> </ul> <p>Nimble Storage Cinder integration with OpenStack Fuel</p> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. Nimble/CSI workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops" style="box-shadow: none;"><button type="button" class="button"><b>Try Now!</b></button></a> </div> - - - <h2>Any questions on Nimble?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C7TTAHRUN">#nimblestorage</a> channel.</p>https://developer.hpe.com/hpe-nonstop/home/https://developer.hpe.com/hpe-nonstop/home/<p>HPE NonStop is a platform that runs some of the world’s most exciting workloads in our day-to-day life. From producing luxury cars, to making payments in our grocery shopping, to helping people travel and executing massive amounts of transactions in global payments networks, HPE NonStop is the platform that lets our customers, and their engineers get their sleep, while their mission-critical applications continue relentlessly in data centres and on private clouds.</p> <h2>YouTube Videos</h2> <h4>HPE NonStop NS8: Uncompromising availability, performance and scale for mission-critical workloads</h4> <p><a href="https://www.youtube.com/watch?v=M5vq2OxwTDI"><img src="https://img.youtube.com/vi/M5vq2OxwTDI/hqdefault.jpg" alt="HPE NonStop NS8"></a></p> <h2>Resources</h2> <p><a href="https://www.hpe.com/us/en/servers/nonstop.html">HPE NonStop Home Page</a></p> <p><a href="https://www.hpe.com/psnow/doc/4aa4-2988enw">HPE NonStop Family of Systems brochure</a></p> <p><a href="https://shaniceabigail.github.io/nonstop101/">NonStop 101: Training Wheels for NonStop OS</a></p> <p><a href="https://github.com/HewlettPackard/NonStop">Sample Code for HPE NonStop products <img src="Github" alt=""></a></p>https://developer.hpe.com/hpe-oneview/home/https://developer.hpe.com/hpe-oneview/home/<h2>HPE OneView Developers Hub</h2> <p><a href="https://hpe.com/info/oneview">HPE OneView</a> takes a software-defined, programmatic approach to managing infrastructure with efficient workflow automation, a modern RESTful API, and a comprehensive partner ecosystem.</p> <p>Here you will find repositories, demos, guides and other technical resources from HPE and the Composable Ecosystem Partners to automate infrastructure management and eliminate complex manual processes.</p> <ul> <li><a href="https://github.com/hewlettpackard/">Develop together: Visit the HPE GitHub organization</a></li> <li><a href="https://hpe.com/info/composableprogram">Looking for solutions: Visit the Composable Ecosystem Partners</a></li> <li><a href="https://www.hpe.com/us/en/resources/integrated-systems/oneview-trial.html?parentPage=/us/en/products/integrated-systems/management-software">Download a free trial</a></li> </ul> <h2>HPE OneView Integrations</h2> <h3><a href="https://github.com/HewlettPackard/oneview-ansible-collection">Ansible <img src="Github" alt=""></a></h3> <p><a href="https://www.ansible.com/home">Ansible</a> by Red Hat automates the provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView.</p> <ul> <li><a href="https://www.redhat.com/en/resources/automate-container-deployment-with-hpe-datasheet">Read the Deployment Guide</a></li> <li><a href="https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=4AA6-6229ENW">Read the Accelerating DevOps White Paper</a></li> <li><a href="http://h17007.www1.hpe.com/us/en/enterprise/integrated-systems/info-library/index.aspx?cat=ci_mgmt&#x26;subcat=ansible#.XJVbZCdMEio">View Additional Resources</a></li> <li><a href="https://www.youtube.com/watch?v=PVJgUEH0Quw&#x26;feature=youtu.be">Watch the Demo</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible-collection">Ansible Docker Image</a></li> <li><a href="https://psnow.ext.hpe.com/doc/a50003411enw?jumpid=in_lit-psnow-red">Installation and user guide</a></li> </ul> <hr> <hr> <h3>Docker</h3> <p><a href="https://www.docker.com/">Docker</a> integrates with HPE OneView to bring containerization out of the cloud and onto your bare-metal infrastructure.</p> <ul> <li><a href="https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a00047301enw">Reference Configuration for CaaS on HPE Synergy</a></li> </ul> <hr> <h3><a href="https://github.com/HewlettPackard/terraform-provider-oneview">HashiCorp Terraform <img src="Github" alt=""></a></h3> <p><a href="https://www.hashicorp.com/">HashiCorp Terraform</a> provides a common workflow to provision hybrid infrastructure and applications so users can seamlessly and efficiently deploy HPE infrastructure.</p> <ul> <li><a href="https://www.terraform.io/intro/index.html">Terraform Overview</a></li> <li><a href="https://learn.hashicorp.com/terraform/getting-started/install.html">Getting started with Terraform</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">Terraform Docker Image</a></li> </ul> <hr> <h3>Microsoft</h3> <p><a href="https://www.hpe.com/us/en/product-catalog/detail/pip.5390822.html">HPE OneView for Microsoft System Center</a> integrates HPE ProLiant and HPE BladeSystem manageability features with Microsoft System Center.</p> <ul> <li><a href="https://www.hpe.com/us/en/product-catalog/detail/pip.5390822.html">View the Product</a></li> </ul> <hr> <hr> <hr> <h3>Red Hat® OpenShift®</h3> <p>Enable IT operations and application development teams to deliver applications faster using the <a href="https://www.redhat.com/en/technologies/cloud-computing/openshift">OpenShift</a> integration.</p> <ul> <li><a href="https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=A00038916ENW">Read the Synergy Reference Configuration</a></li> <li><a href="https://access.redhat.com/documentation/en-us/reference_architectures/2017/html-single/automate_red_hat_openshift_container_platform_deployment_on_hpe_proliant_servers_with_ansible_tower_and_hpe_oneview/">Read the Technical White Paper</a></li> <li><a href="http://h17007.www1.hpe.com/us/en/enterprise/integrated-systems/info-library/index.aspx?cat=ci_mgmt&#x26;subcat=ansible">View Additional Resources</a></li> </ul> <hr> <h3>VMware</h3> <p>HPE OneView for <a href="https://vmware.com/">VMware</a> vCenter seamlessly integrates manageability features with VMware virtualization solutions.</p> <ul> <li><a href="https://www.hpe.com/us/en/product-catalog/detail/pip.4152978.html">View the Product</a></li> </ul> <hr> <h2>SDKs and Language Bindings</h2> <p>Access Software Development Kits (SDKs) and language bindings for integrating HPE OneView with common programming languages and frameworks.</p> <h3><a href="https://github.com/HewlettPackard/oneview-golang">Go <img src="Github" alt=""></a></h3> <p>HPE OneView allows you to treat your physical infrastructure as code. Now you can integrate your favorite tools based in Golang with HPE OneView.</p> <ul> <li><a href="https://hub.docker.com/r/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">OneView Golang SDK Docker Image</a></li> </ul> <hr> <h3><a href="https://github.com/HewlettPackard/hpe-oneview-hubot">Hubot <img src="Github" alt=""></a></h3> <p>Interface with HPE OneView using Hubot, an automation tool that can sync with other chat services.</p> <hr> <h3><a href="https://github.com/HewlettPackard/POSH-HPOneView">PowerShell <img src="Github" alt=""></a></h3> <p>This library provides a pure Windows PowerShell interface to the HPE OneView REST APIs.</p> <hr> <h3><a href="https://github.com/HewlettPackard/oneview-python">Python <img src="Github" alt=""></a></h3> <p>This library provides a pure Python interface to the HPE OneView REST APIs.</p> <ul> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">OneView Python SDK Docker Image</a></li> </ul> <hr> <h3><a href="https://github.com/HewlettPackard/oneview-redfish-toolkit">Redfish <img src="Github" alt=""></a></h3> <p>The Redfish toolkit allows customers to take automations that use the Redfish specification and apply them to HPE OneView without need for extensive scripting.</p> <hr> <hr> <h2>HPE Synergy Image Streamer Tools</h2> <p>Access tools, reference architectures, artifact bundles, and other technical resources for implementing HPE Synergy Image Streamer.</p> <h3><a href="https://github.com/HewlettPackard?q=image-streamer">Developer Resources <img src="Github" alt=""></a></h3> <p>Resources for achieving fast software-defined control using the Image Streamer management appliance.</p> <hr> <h3><a href="https://github.com/HewlettPackard/image-streamer-reference-architectures">Reference Architectures <img src="Github" alt=""></a></h3> <p>Best practices and configuration guidance for developing and executing Image Streamer deployment plans.</p> <hr> <h3><a href="https://github.com/HewlettPackard/image-streamer-rhel">RHEL Artifacts <img src="Github" alt=""></a></h3> <p>Plan scripts and sample artifact bundles for personalizing and deploying Red Hat Enterprise Linux.</p> <hr> <h3><a href="https://github.com/HewlettPackard/image-streamer-sles">SLES Artifacts <img src="Github" alt=""></a></h3> <p>Plan scripts and sample artifact bundles for personalizing and deploying SUSE Linux Enterprise Server.</p> <hr> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. HPE OneView workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> - - - <h2>Any questions on HPE OneView?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C5TMA1TK5">#oneview</a> channel.</p>https://developer.hpe.com/hpe-oneview-global-dashboard/home/https://developer.hpe.com/hpe-oneview-global-dashboard/home/<h1>Unified View of Infrastructure Across Data Centers in Multiple Locations</h1> <p>The HPE OneView Global Dashboard provides a unified view of the health and inventory of Hewlett Packard Enterprise servers, profiles, enclosures, HPE Synergy frames, HPE 3PAR and HPE Primera storage systems across multiple appliances for ease of management.</p> <p>Aggregates critical activities from multiple appliances into a single feed to quickly identify issues occurring on monitored hardware for prompt resolution.</p> <p>Generates reports for monitored assets to view inventory, including firmware versions as well as compliance that allow you to verify that your equipment meets corporate standards.</p> <p>One click navigation of managed resources with single sign-on managing appliance.</p> <ul> <li><a href="https://hpe.com/products/ovglobaldashboard">Learn more about Global Dashboard</a></li> </ul> <h1>API Reference</h1> <p>Global Dashboard provides a public REST API to can be used to retrieve information about resources. For example, you can retrieve server-hardware information about thousands of servers from datacenters scattered across the global with a single API call.</p> <h3><a href="https://github.com/HewlettPackard/hpe-globaldashboard-swagger" title="OpenAPI (Swagger) definition of the Global Dashboard API">OpenAPI Definition <img src="Github" alt=""></a></h3> <h3><a href="http://app.swaggerhub.com/apis/hpe-global-dashboard/hpe-one_view_global_dashboard_rest_api/2.1">API Reference on SwaggerHub</a></h3>https://developer.hpe.com/hpe-opsramp/home/https://developer.hpe.com/hpe-opsramp/home/<p>The Cloud and Cloud-Native Observability solution within OpsRamp, a Hewlett Packard Enterprise company, is designed to provide organizations with deep visibility and actionable insights into their hybrid cloud and cloud-native environments. By leveraging a comprehensive set of powerful monitoring and observability capabilities, OpsRamp enables ITOps and DevOps teams to ensure the reliability, performance, and security of their business-critical services, applications and infrastructure.</p> <p>OpsRamp is a comprehensive autonomous IT operations platform designed to help service providers and enterprises modernize and streamline their IT operations. OpsRamp integrates artificial intelligence (AI), automation, and analytics to provide visibility, monitoring, and management across hybrid and multi-cloud environments. It addresses the challenges of complex IT infrastructures by offering tools for unified observability, proactive monitoring, incident management, and remediation.</p> <p>Learn more about <a href="https://www.hpe.com/us/en/opsramp.html">OpsRamp</a>.</p> <h2>API Documentation</h2> <p>OpsRamp offers a robust set of RESTful APIs designed to integrate and automate various IT operations management processes. These APIs allow developers to interact programmatically with the OpsRamp platform, enabling advanced customization, data exchange, and workflow automation across IT environments. Detailed information can be found on the <a href="https://develop.opsramp.com/v2">OpsRamp documentation</a> site.</p> <h2>Integrations</h2> <p>OpsRamp offers an extensive range of integrations that enhance its capabilities for IT operations management. These integrations span across various domains, including service management platforms like ServiceNow and Jira, monitoring and observability tools such as Prometheus, backup and recovery systems like Zerto, and cloud-native systems such as Kubernetes.</p> <p>Additionally, OpsRamp supports integration with popular automation tools like Ansible and comprehensive export options to cloud storage solutions such as AWS S3 and Azure Blob Storage. Detailed documentation on all integrations can be found at <a href="https://docs.opsramp.com/integrations/">www.docs.opsramp.com/integrations</a>.</p> <ul> <li> <p><strong>Custom integrations:</strong> OpsRamp's custom integration framework enables users to create webhook and OAuth2 integrations for data ingestion and event capture within the platform. It offers flexibility to build integrations through either the OpsRamp APIs or the web interface. This framework is designed to accommodate diverse use cases, such as integrating custom alert sources, enabling ticketing workflows, or syncing with external systems, all while providing tools for validation and auditing of integration setups. For further details, visit the <a href="https://docs.opsramp.com/integrations/a2r/custom-integration/custom-integration/">OpsRamp Custom Integration documentation</a>.</p> </li> <li> <p><strong>OpenTelemetry integration:</strong> OpsRamp's <a href="https://opentelemetry.io/">OpenTelemetry</a> integration enables organizations to achieve seamless distributed tracing across their IT environments. By leveraging OpenTelemetry, an open-source observability framework, OpsRamp provides a vendor-agnostic solution for capturing and visualizing trace data. This integration helps teams gain end-to-end visibility into the lifecycle of requests spanning microservices, APIs, and other system components, allowing for the identification of latency issues and performance bottlenecks.<br/> <br/></p> </li> </ul> <p>The platform supports easy instrumentation using OpenTelemetry’s libraries and SDKs, ensuring compatibility with a wide range of programming languages and environments. The resulting trace data can be visualized using OpsRamp's intuitive interfaces, such as flame graphs and service insights, for efficient performance analysis and troubleshooting. This integration empowers IT and development teams to optimize their distributed systems while maintaining flexibility and industry standards in their observability strategies. For further details, visit the <a href="https://docs.opsramp.com/integrations/a2r/3rd-party/opentelemetry-integration/">OpenTelemetry integration documentation</a>.</p> <ul> <li> <p><strong>Kubernetes integration:</strong> OpsRamp’s Kubernetes (K8s) integration provides robust monitoring and management of containerized environments. It supports Kubernetes clusters deployed across various cloud providers and on-premises setups. By integrating with Kubernetes, OpsRamp offers capabilities such as automated resource discovery, real-time monitoring of nodes, pods, and services, and visualization of cluster health and performance metrics. The platform enables users to customize monitoring thresholds, set alert conditions, and leverage detailed dashboards for better insight into their Kubernetes workloads. This integration helps ensure operational efficiency and scalability in managing containerized applications across diverse environments. For further details, visit the <a href="https://docs.opsramp.com/integrations/container-orchestration/kubernetes-new/">Kubernetes integration documentation</a>.</p> </li> <li> <p><strong>Ansible integration:</strong> OpsRamp's <a href="https://www.ansible.com/">Ansible</a> integration enables automation of a wide range of tasks across hybrid infrastructures. It supports agent deployment, updates, and uninstallation, but also goes further by automating diagnostic health checks and performing remediation actions to maintain system stability. Through the use of Ansible playbooks, IT teams can automate the identification and resolution of issues, enforce compliance, and ensure system health. This integration also includes secure management of sensitive data with Ansible Vault, making it a powerful tool for streamlining IT operations and ensuring consistency and efficiency across diverse environments. For further details, visit the <a href="https://docs.opsramp.com/integrations/automation-integration/ansible-integration/">Ansible integration documentation</a>.</p> </li> </ul> <h2>Resources</h2> <p>Ready to learn more about OpsRamp? Browse through the <a href="https://www.opsramp.com/resources/">OpsRamp resources portal</a>.</p> <br/> <hr> <br/> <h2>Any questions on HPE OpsRamp?</h2> <p>Join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C082P2Q3811">#hpe-opsramp</a> Slack channel.</p>https://developer.hpe.com/hpe-private-cloud-ai/home/https://developer.hpe.com/hpe-private-cloud-ai/home/<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>You’re an AI/ML practitioner, not an infrastructure manager. But AI development is bogged down by:</p> <ul> <li>Spending excessive amounts of time waiting for data that delays model training and iteration cycles</li> <li>Limited compute and storage resources that slow down exploring new model architectures, tuning, and scaling experiments</li> <li>Security risks that slow down development as well as the ability to quickly test and deploy new models</li> <li>Struggling to keep pace with an ecosystem of constantly changing tool sets and frameworks</li> </ul> <p>HPE Private Cloud AI is a pre-configured, end-to-end solution for enterprise AI that integrates high-performance compute, storage, networking and AI software so you can focus on what you need to do: build innovative AI solutions. A built-in software ecosystem of tools and frameworks streamlines lifecycles so you can deploy into production without wrangling data, tools, or infrastructure.</p> <p><a href="https://hpe.com/private-cloud-ai">Learn more about HPE Private Cloud AI</a></p> <p><img src="/img/hpe-private-cloud-ai-picture1-new.jpg" alt="HPE Private Cloud AI" title="HPE Private Cloud AI"></p> <h1>HPE Private Cloud AI developer system - Accelerate AI development</h1> <p>The HPE Private Cloud AI developer system aims to abstract away the underlying infrastructure complexity, data wrangling, and environment setup. Being freed from these tasks enables you to focus on building, training, and deploying AI solutions that prove and validate AI projects faster. How do we do that?</p> <p>This system is ready to run out-of-the-box to reduce the burden of infrastructure provisioning and management. With its built-in, end-to-end software platform, you gain direct, unified access to open-source and NVIDIA tools, enabling you to focus on critical tasks by eliminating hardware-software interoperability headaches. Got custom or third-party tools you want to use? No problem. Import them into the software model catalog along with NVIDIA Blueprints to accelerate your specialized AI workflows.</p> <p>But this development environment isn’t just for ideation; it’s designed to be the starting point for production AI. As your models mature and demand enterprise-grade scale and reliability, you can seamlessly transition AI workflows to the HPE Private Cloud family, leveraging its proven architecture and unified management for true enterprise-grade AI.</p> <h1>Technical Demos</h1> <h3>Simplify AI from Infrastructure to Model Deployment</h3> <p>Join Randy Thomasson as he demonstrates how HPE Private Cloud AI removes the complexities of AI infrastructure, streamlines data pipelines, and simplifies model deployment.</p> <a href="https://www.brighttalk.com/webcast/19535/640132?utm_source=HPE&utm_medium=brighttalk&utm_campaign=640132" target="_blank"> <p><img src="/img/simplify-ai-from-infrastructure-to-model-deployment-500-281.png" alt="Simplify AI from Infrastructure to Model"></p> </a> <h3>Simplify Data Pipelines: HPE AI Essentials Demo for Data Teams</h3> <p>See how HPE AI Essentials simplifies the creation of powerful data pipelines using Apache Airflow and Apache Spark.</p> <p><a href="https://www.youtube.com/watch?v=e0IUMJKpqGg"><img src="https://img.youtube.com/vi/e0IUMJKpqGg/hqdefault.jpg" alt="Simplify Data Pipelines"></a></p> <h3>Fast track innovation: how HPE simplifies model deployment</h3> <p>This demo showcases the built-in machine learning services that simplify and automate model development and deployment.</p> <p><a href="https://www.youtube.com/watch?v=THeg2DwrF4c"><img src="https://img.youtube.com/vi/THeg2DwrF4c/hqdefault.jpg" alt="Fast track innovation"></a></p> <h3>Building a generative AI foundation with HPE Private Cloud AI</h3> <p>In this video, Alex Ollman provides a deep dive into the HPE Private Cloud AI architecture and infrastructure needed to deploy generative AI at enterprise scale.</p> <p><a href="https://www.youtube.com/watch?v=AIG4-O9ZVRY"><img src="https://img.youtube.com/vi/AIG4-O9ZVRY/hqdefault.jpg" alt="HPE Private Cloud AI architecture for GenAI"></a></p> <h3>HPE Private Cloud AI technical demo</h3> <p>Join Randy Thomasson to delve into offering role-specific walkthroughs for administrators focusing on user and GPU management, data engineers exploring data pipeline construction, and data scientists learning about model deployment.</p> <p><a href="https://www.youtube.com/watch?v=5uRoHXD2Sks"><img src="https://img.youtube.com/vi/5uRoHXD2Sks/hqdefault.jpg" alt="HPE Private Cloud AI technical demo"></a></p> <h1>Resources</h1> <h2>HPE Private Cloud AI Documentation</h2> <ul> <li><a href="https://www.hpe.com/psnow/product-documentation?oid=1014847366&#x26;cc=my&#x26;lc=en&#x26;jumpid=in_pdp-psnow-docs">Product Documentation</a></li> <li><a href="https://hpe.com/support/PCAIUserGuide">Administration Guide</a></li> <li><a href="https://www.hpe.com/support/AIEDocs">HPE AI Essentials Software</a></li> </ul> <h2>Tutorials – HPE AI Essentials</h2> <p><a href="https://github.com/HPEEzmeral/aie-tutorials/tree/aie-1.7.0">Tutorials for AI Essentials Software on HPE Private Cloud AI <img src="Github" alt=""></a></p> <h2>Analyst Reports</h2> <ul> <li>Technical validation: <a href="https://psnow.ext.hpe.com/doc/a00146294enw">Accelerate AI business value with NVIDIA Computing for HPE</a></li> <li>Economic validation: <a href="https://www.hpe.com/psnow/doc/a00146433enw">The economic benefits of HPE Private Cloud AI with NVIDIA AI Computing by HPE</a></li> </ul> <h1>Any questions about HPE Private Cloud AI?</h1> <p>Need help getting started with the HPE Private Cloud AI, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C08MBCD6ER5">#hpe-private-cloud-ai</a> Slack channel.</p>https://developer.hpe.com/ilo-restful-api/home/https://developer.hpe.com/ilo-restful-api/home/<h1>One Interface for Server Management Automation</h1> <p>HPE Server management provides intelligent remote control automation through <a href="https://www.hpe.com/us/en/hpe-integrated-lights-out-ilo.html">HPE Integrated Lights-Out</a> (iLO) and the Redfish® iLO RESTful API. Gain even more capabilities that go beyond scripting by leveraging one API to manage your complete lifecycle of HPE Gen10, Gen10 Plus and Gen11 servers.</p> <p>A single API interface integrates server management components and full compute power. Use it with HPE iLO 5 and iLO 6 to perform remote server provisioning, configuration, inventory and monitoring industry standards through Redfish API conformance.</p> <h1>HPE Redfish API Implementation</h1> <p>Obtain simple, secure management of today’s scalable data center hardware with the Redfish® API ecosystem. It’s an open industry-standard specification and schema that helps you integrate solutions within your existing tools. Published by the Distributed Management Task Force (<a href="http://www.dmtf.org/standards/redfish">DMTF</a>), it's ideal for cloud and web-based infrastructures, which typically have large quantities of servers in heterogeneous environments.</p> <ul> <li><a href="https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/whitepaper/pdfs/4AA6-1727ENW.pdf">Read about Redfish on HPE iLO</a></li> <li><a href="https://ilorestfulapiexplorer.ext.hpe.com/">Explore the API</a></li> <li><a href="https://servermanagementportal.ext.hpe.com/">Consult the API reference documentation</a></li> <li><a href="https://youtu.be/ur9UKRV_0S8">Choose a Redfish Client Tool</a></li> </ul> <h1>GitHub and PowerShell Repositories</h1> <p>Find tools you need to help you leverage the iLO RESTful API SDKs.</p> <h2>SDKs and Language Bindings</h2> <h3><a href="https://servermanagementportal.ext.hpe.com">iLO RESTful API Documentation</a></h3> <p>HPE Reference documentation with examples to help you write Redfish client programs and scripts.</p> <hr> <h3><a href="https://github.com/HewlettPackard/python-ilorest-library">The Python library <img src="Github" alt=""></a></h3> <p>The Python library provides a rich Redfish library and examples for developers to easy interact with the iLO RESTful API.</p> <hr> <h3><a href="https://www.powershellgallery.com/packages?q=hpe*cmdlets">The PowerShell Gallery</a> and <a href="https://github.com/HewlettPackard/PowerShell-ProLiant-SDK">library <img src="Github" alt=""></a></h3> <p>The PowerShell Gallery and library provide Cmdlets and scripts to interact with the Windows PowerShell Interface to the iLO RESTful API.</p> <hr> <h3><a href="https://github.com/HewlettPackard/ilo-sdk-ruby">The Ruby library <img src="Github" alt=""></a></h3> <p>The Ruby library enables to interact the iLO RESTful API.</p> <hr> <h3><a href="https://github.com/HewlettPackard/javascript-ilorest-library">The JavaScript library <img src="Github" alt=""></a></h3> <p>The JavaScript library enables Java developers to easily integrate with the iLO RESTful API.</p> <h2>DevOps</h2> <h3><a href="https://galaxy.ansible.com/hpe/ilo">Ansible <img src="Github" alt=""></a></h3> <p><a href="https://github.com/HewlettPackard/ilo-ansible-collection/">Ansible playbooks and roles</a> for HPE iLO using the Redfish® API.</p> <hr> <h3><a href="https://github.com/HewlettPackard/chef-ilorest-cookbook">Chef Cookbook <img src="Github" alt=""></a></h3> <p>Chef Cookbook for installing the Python iLOrest library and examples.</p> <hr> <h3><a href="https://github.com/HewlettPackard/puppet-ilorest-module">Puppet module <img src="Github" alt=""></a></h3> <p>Puppet module for installing the Python iLOrest library and examples.</p> <h2>IT Operations</h2> <h3><a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest">RESTful Interface Tool <img src="Github" alt="iLOrest"></a></h3> <p><a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest">HPE iLOrest</a>, the HPE RESTful Interface Tool is an open source Redfish client scripting tool also featuring interactive and debug modes. Packaging includes Windows, many Linux flavors as well as a <a href="https://pypi.org/project/ilorest">PyPI project</a>.</p> <ul> <li><a href="https://servermanagementportal.ext.hpe.com/docs/redfishclients/ilorest-userguide/">Read the documentation guide</a></li> <li><a href="https://www.youtube.com/watch?v=xfEN95pNNfY">Watch the Demo</a></li> </ul> <hr> <h3><a href="https://github.com/HewlettPackard/nagios-hpeilo-restful-extension">Nagios- Plug-in <img src="Github" alt="Redfish Nagios plug-in"></a></h3> <p>Nagios- Plug-in for Industry Standard in IT infrastructure monitoring.</p> <hr> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="/hackshack">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. iLO/Redfish workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> - - - <h2>Any questions on iLO or Redfish?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C9RRCL9TJ">#redfish</a> channel.</p>https://developer.hpe.com/kubedirector/home/https://developer.hpe.com/kubedirector/home/<p>KubeDirector is an open source Kubernetes “custom controller” that simplifies implementing stateful scaleout application clusters on Kubernetes. Using standard Kubernetes (K8s) facilities to implement stateful scale out application clusters, it enables the transparent integration of K8s user/resource management and existing K8s clients and tools.</p> <p>KubeDirector does not tie a custom resource definition to a particular type of application, or contain hardcoded application-specific logic within the controller. Application characteristics are instead defined by metadata and an associated package of configuration artifacts, simplifying application deployment.</p> <p>KubeDirector provides:</p> <ul> <li> <p><strong>One operator for stateful apps</strong>, enabling the deployment of legacy, stateful enterprise applications without writing or implementing application-specific operators</p> </li> <li> <p><strong>An abstract framework</strong>, allowing the definition of application characteristics through metadata and configuration artifacts</p> </li> <li> <p><strong>Continuous validation of resources</strong>, automating app deployment and validation</p> </li> <li> <p><strong>Hooks for “Day two” operations</strong>, simplifying lifecycle management</p> </li> </ul> <h2>Introduction to KubeDirector</h2> <p><a href="https://kubedirector.io/">Project web site</a></p> <p><a href="https://github.com/bluek8s/kubedirector/wiki">KubeDirector Wiki</a></p> <hr> <h2>GitHub repositories</h2> <p><a href="https://github.com/bluek8s/kubedirector">KubeDirector on GitHub</a>: Main KubeDirector repository.</p> <p><a href="https://hpe-container-platform-community.github.io/learn-kubedirector/docs/">Learn Kubedirector on GitHub</a>: Get hints on how to use</p> <hr> <h2>Documentation and tutorials</h2> <p><a href="https://github.com/bluek8s/kubedirector/blob/master/doc/quickstart.md">Quickstart Guide</a></p> <p><a href="https://github.com/bluek8s/kubedirector/blob/master/HISTORY.md">Update release info</a></p> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. KubeDirector workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on KubeDirector?</h2> <p>You can raise any questions and issues you might have on our <a href="https://github.com/bluek8s/kubedirector/issues">GitHub issues</a> link.</p>https://developer.hpe.com/hpe-simplivity/adding-certificates/https://developer.hpe.com/hpe-simplivity/adding-certificates/<h1>Adding certificates to the trust store</h1> <p>Add certificates to the trust store by using the <code>Add-HPESvtCertificate</code> cmdlet. You can specify the path to the certificate file or an X509Certificate2 object. The certificate file can be a PEM or a DER file. To use the <code>Add-HPESvtCertificate</code> cmdlet, you must first authenticate.</p> <p>The following example shows how add a certificate called <code>test-node.pem</code> by path. It assumes that you have already authenticated.</p> <pre><code>PS C:\> Add-HPESvtCertificate -Path c:\temp\test-node.pem certificate : -----BEGIN CERTIFICATE----- MIIELTCCAxWgAwIBAgIJAMptn/02hezjMA0GCSqGSIb3DQEBCwUAMIGsMQswCQYD VQQGEwJVUzEWMBQGA1UECAwNTWFzc2FjaHVzZXR0czEUMBIGA1UEBwwLV2VzdGJv W8rc/ZJmzmxtOmM1DPDCL6RpINS1V4ouJIF5DHo8kzP1UjZjHQcCAwEAAaNQME4w CQYDVR0TBAIwADALBgNVHQ8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG AQUFBwMCMBUGA1UdEQQOMAyHBAqQAWeHBArPAfswDQYJKoZIhvcNAQELBQADggEB AKkXh9z6wo5Mt2zyDrh2p252AwUTmILLD7+YgGxU1stpjFaVlKL7wiEsW4/g37+J . . . -----END CERTIFICATE----- hash : 47ca6deb5880fd26ec4f546b3884aa65c78590df subject : EMAILADDRESS=security@myco.com,CN=omnicube.xxx.us-east.myco.local,O=myco,L=mycity,ST=mystate,C=US issuer : EMAILADDRESS=security@myco.com, CN=omnicubexx.us-east.myco.local, O=myco, L=mycity, ST=mystate, C=US serialno : ca6d9ffd3685ece2 </code></pre> <p>This example shows how to add a root certificate by using an X509Certificate2 object:</p> <pre><code>$newrootcert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::New($newrootpath) Add-HPESvtCertificate -Certificate $newrootcert </code></pre> <p>Optionally, you can pipe the parameter to <code>Add-HPESvtCertificate</code> instead of passing it as an argument. For example:</p> <pre><code>'test.pem' | Add-HPESvtCertificate </code></pre> <p>or</p> <pre><code>$newrootcert | Add-HPESvtCertificate </code></pre>https://developer.hpe.com/hpe-simplivity/architecture-and-object-model/https://developer.hpe.com/hpe-simplivity/architecture-and-object-model/<h1>Architecture</h1> <p>The HPE OmniStack REST API is available on every Virtual Controller. To issue a REST request, target the network address (IP address or DNS name) of the Virtual Controller. The REST API provides a secure portal into the Virtual Controller. Consequently, address REST API requests to secure port 443. The following diagram illustrates the basic architecture of the REST API:</p> <p><img src="https://developer.hpe.com/uploads/media/2018/7/svt-rest-api-arch-1532712778003.png" alt="architecture diagram"></p> <h1>Object model</h1> <p>The REST API object model enables you to monitor and manage a set of HPE OmniStack objects, including backups, datastores, hosts, clusters, policies, tasks, and virtual machines. The following table describes each of the types of objects that the REST API supports:</p> <table> <thead> <tr> <th>Object type</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>backup</code></td> <td>A complete, standalone image of a virtual machine, taken at a specific point in time. You can retrieve all of the backups that are defined in the federation. You can copy, delete, lock, rename, restore, and set the retention time for backups.</td> </tr> <tr> <td><code>datastore</code></td> <td>A repository that contains the files of one or more virtual machines. You can retrieve all of the datastores that are defined in the federation. You can create new datastores, and you can delete, resize, and set policies for existing datastores.</td> </tr> <tr> <td><code>host</code></td> <td>An HPE OmniStack host in a federation. You can retrieve all of the hosts that are defined in a federation along with their capacity, hardware, and metrics data.</td> </tr> <tr> <td><code>omnistack_cluster</code></td> <td>A logical grouping of systems that are running the HPE OmniStack software. You define an <code>omnistack_cluster</code> to enable resources to be shared efficiently across the HPE OmniStack hosts in a federation. You can retrieve all of the <code>omnistack_cluster</code> objects that are defined in a federation. You can retrieve <code>omnistack_cluster</code> metrics, throughput, and connected clusters data, and you can set the time zone for an <code>omnistack_cluster</code>.</td> </tr> <tr> <td><code>policy</code></td> <td>Contains backup rules that can be applied to an individual datastore or virtual machine. You can retrieve all the policies that are defined in the federation. You can create, delete, and rename policies. You can also create, edit, and delete the rules associated with policies.</td> </tr> <tr> <td><code>task</code></td> <td>Tracks the progress of an HPE OmniStack operation. When the status of a task is <code>COMPLETED</code>, the <code>affected_objects</code> indicate any created or modified objects. You can retrieve tasks.</td> </tr> <tr> <td><code>virtual_machine</code></td> <td>Represents a single virtual machine that has been created within an HPE OmniStack datastore. You can retrieve all of the virtual machines in the federation. You can back up, clone, and move virtual machines. You can set policies for virtual machines and retrieve virtual machine backup and metrics data.</td> </tr> </tbody> </table> <h2>REST API object type relationships</h2> <p>The REST API supports the relationship of one object to another object of a different type by referencing the ID of the related object as a value of one of the properties of the base object. The following figure shows these object type relationships:</p> <p><img src="https://developer.hpe.com/uploads/media/2018/7/svt-rest-api-object-type-relationships-1532712788492.png" alt="object type relationships"></p> <h2>REST API tasks as managed objects</h2> <p>The control plane treats tasks as true managed objects, which enables clients to efficiently determine the changes that requests have made on the system; for example, during the creation of new objects. Tasks contain meaningful information that enables clients to post-process requests in an intuitive and deterministic manner. The REST API task managed object has the following characteristics:</p> <ul> <li>A task has an ID value that enables a client to query for a task.</li> <li>The state of a task conveys where the task currently is in the processing sequence. Based on the current state, other properties of the task might not be set.</li> <li>If the task ends in failure, the <code>error_code</code> and <code>message</code> properties contain information about the failure.</li> <li>If the task completes successfully, the <code>affected_objects</code> array contains information about the object(s) that the task impacted. Each <code>affected_object</code> entry has a type-significant ID, as well as a type string. For example, if the move of a virtual machine completes successfully, the task that is associated with the move has an <code>affected_object</code> of the <code>virtual_machine</code> type, and the ID of the task is the system-assigned ID of the new virtual machine.</li> <li>Multiple <code>affected_objects</code> can be associated with a task (thus, the array of affected_objects).</li> </ul>https://developer.hpe.com/hpe-simplivity/aside/https://developer.hpe.com/hpe-simplivity/aside/<h3>About the API</h3> <p><a href="/platform/hpe-simplivity/whats-new/"><strong>What's New</strong></a></p> <p><a href="/platform/hpe-simplivity/feature-and-function-support-by-rest-api-version"><strong>Feature and function support by REST API version</strong></a></p> <p><a href="/platform/hpe-simplivity/architecture-and-object-model"><strong>Architecture and object model</strong></a></p> <p><a href="/platform/hpe-simplivity/enumerations"><strong>Enumerations</strong></a></p> <p><a href="/platform/hpe-simplivity/interactive-rest-api-reference"><strong>Interactive REST API reference</strong></a></p> <p><a href="/platform/hpe-simplivity/errors-and-exceptions"><strong>About REST API errors and exceptions</strong></a></p> <p><a href="/platform/hpe-simplivity/http-error-code-reference"><strong>HTTP error code reference</strong></a></p> <p><a href="/platform/hpe-simplivity/authorization-related-rest-api-log-messages-and-responses"><strong>Authorization-related REST API log messages and responses</strong></a></p> <h3>Explore the API</h3> <p><a href="https://developer.hpe.com/api/simplivity/"><strong>HPEOmniStack REST API Reference</strong></a></p> <h3>Use the API</h3> <p><a href="/platform/hpe-simplivity/sample-code"><strong>Getting started</strong></a></p> <p><a href="/platform/hpe-simplivity/get-backups"><strong>Getting backups</strong></a></p> <p><a href="/platform/hpe-simplivity/restore-virtual-machines"><strong>Restoring virtual machines</strong></a></p> <p><a href="/platform/hpe-simplivity/versioning"><strong>Getting the API version</strong></a></p> <p><a href="/platform/hpe-simplivity/optimize-get-requests"><strong>Optimizing GET requests</strong></a></p> <p><a href="/platform/hpe-simplivity/metrics"><strong>Getting performance and capacity metrics</strong></a></p> <p><a href="/platform/hpe-simplivity/powershell"><strong>Using the API with PowerShell</strong></a></p> <p><a href="/platform/hpe-simplivity/python"><strong>Using the API with Python</strong></a></p> <p><a href="/platform/hpe-simplivity/java"><strong>Using the API with Java</strong></a></p> <h3>Authenticating</h3> <p><a href="/platform/hpe-simplivity/authenticating-against-hpe-omnistack-api"><strong>Authenticating against the API</strong></a></p> <p><a href="/platform/hpe-simplivity/revoke-oath-2-token"><strong>Revoking OAuth 2 tokens</strong></a></p> <p><a href="/platform/hpe-simplivity/request-authentication-by-emergency-grant"><strong>Requesting authentication by emergency grant</strong></a></p> <h3>Manage certificates using PowerShell</h3> <p><a href="/platform/hpe-simplivity/hpesvtcli-powershell-commands"><strong>Installing the certificate management Powershell cmdlets</strong></a></p> <p><a href="/platform/hpe-simplivity/ps-getoathtoken"><strong>Authenticating</strong></a></p> <p><a href="/platform/hpe-simplivity/adding-certificates"><strong>Adding certificates to the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/getting-a-certificate"><strong>Getting certificates from the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/getting-a-root-certificate"><strong>Getting root certificates</strong></a></p> <p><a href="/platform/hpe-simplivity/remove-certificate"><strong>Removing certificates from the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/tls-certificate-validation"><strong>Disabling/enabling TLS certificate validation on HPE OmniStack for vSphere</strong></a></p>https://developer.hpe.com/hpe-simplivity/authenticating-against-hpe-omnistack-api/https://developer.hpe.com/hpe-simplivity/authenticating-against-hpe-omnistack-api/<h1>Authenticating against the API</h1> <hr> <p>About OAuth 2 authentication and authorization</p> <p>The REST API uses OAuth 2 authentication. To perform any operation using the REST API, you must log in and obtain a valid OAuth token. To log in, use the same username/password combination that you would use to create a session on the Virtual Controller using the <code>svt-session-start</code> CLI command. Once you have received an OAuth token, pass this token in with every REST API operation to authorize the operation. The privilege level of a REST API session is equivalent to the privilege level of a CLI session that uses the same credentials. For example, a read-only user can perform GET operations but cannot perform most POST/PUT/DELETE operations.</p> <h1>Requesting an OAuth 2 token</h1> <p>This example shows how to request an OAuth 2 token for use with the REST API using <code>curl</code>. OAuth 2 token requests are the only HTTP requests that use basic authentication. For these OAuth 2 token requests, use <code>simplivity</code> as the username and leave the password blank.</p> <p>You need a username/password combination that to use to request an OAuth 2 token.</p> <p>The following <code>curl</code> command requests an OAuth 2 token:</p> <pre><code>curl -k https://simplivity@[host]/api/oauth/token -d grant_type=password -d username=[username] -d password=[password] </code></pre> <p>Where:</p> <ul> <li><em>host</em> is the IP address of the Virtual Controller to which you want to authenticate.</li> <li><em>username</em> is the username for a user account with the appropriate privileges for the REST API operations that you plan to perform.</li> <li><em>password</em> is the password that corresponds to the <em>username</em> that you have specified.</li> </ul> <p>You must supply the <code>-k</code> switch due to the use of self-signed certificates. You can import these certificates into your local certificate store.</p> <p>This call returns a JSON response. For example:</p> <pre><code>{ "access_token":"d3c4782f-7fd9-496f-969b-29b7f7715972", "token_type":"bearer", "expires_in":81301, "scope":"read write", "updated_at":1455232016377 } </code></pre> <p>Note the value for the <code>access_token</code> in this response.</p> <p>Pass the value for the <code>access_token</code> in every HTTP request header, using the following format: <code>Authorization: Bearer [access_token]</code></p> <p>For example:</p> <p><code>Authorization: Bearer d3c4782f-7fd9-496f-969b-29b7f7715972</code></p>https://developer.hpe.com/hpe-simplivity/authorization-related-rest-api-log-messages-and-responses/https://developer.hpe.com/hpe-simplivity/authorization-related-rest-api-log-messages-and-responses/<h1>Authorization-related REST API log messages and responses</h1> <h3>Successful OAuth 2 token request log messages</h3> <p>If you request and receive an OAuth 2 token for the REST API successfully, <code>svt-rest-api.log</code> displays relevant messages. For example:</p> <pre><code>2016-04-29 14:36:48.379 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtSessionMap : User count after add: 1 2016-04-29 14:36:48.384 INFO 13430 --- [tp1872627924-22] o.s.b.a.audit.listener.AuditListener : AuditEvent [timestamp=Fri Apr 29 14:36:48 EDT 2016, principal=administrator, type=AUTHENTICATION_SUCCESS, data={details={grant_type=password, username=administrator}}] 2016-04-29 14:36:48.387 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtTokenEnhancer : enhancing accessToken 28b42cf2-4d6b-40a4-a91e-101a9e21c1cc with updated_at=1461955008387 2016-04-29 14:36:48.387 DEBUG 13430 --- [tp1872627924-22] c.s.r.security.SvtInMemoryTokenStore : Storing access token 28b42cf2-4d6b-40a4-a91e-101a9e21c1cc 2016-04-29 14:36:48.387 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtTokenEnhancer : enhancing accessToken 28b42cf2-4d6b-40a4-a91e-101a9e21c1cc with updated_at=1461955008387 2016-04-29 14:36:48.388 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtSessionMap : Adding HMS session map with OAuth=28b42cf2-4d6b-40a4-a91e-101a9e21c1cc 2016-04-29 14:36:48.389 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtSessionMap : Session count after add: 1 2016-04-29 14:36:48.389 DEBUG 13430 --- [tp1872627924-22] c.s.restapi.security.SvtSessionMap : Expired session count after add: 0 </code></pre> <h3>JSON response indicates unsuccessful OAuth 2 token request</h3> <p>If an authorization error occurs during the token request, the JSON response includes a series of fields that provide information about the cause of the failure. For example:</p> <pre><code>{ "timestamp": 1457970690535, "status": 401, "error": "Unauthorized", "exception": "com.simplivity.restapi.exceptions.SvtExceptions $UnauthorizedException", "message": "Unauthorized", "path": "/api/backups" } </code></pre> <h3>Normal OAuth 2 token audit log messages</h3> <p>During normal REST API operations, <code>svt-rest-api.log</code> displays token audit messages. For example:</p> <pre><code>2016-04-29 14:53:29.895 DEBUG 13430 --- [tp1872627924-35] c.s.restapi.security.SvtTokenServices : Time elapsed (ms) since last update of access token: 53388 2016-04-29 14:53:29.895 INFO 13430 --- [tp1872627924-35] o.s.b.a.audit.listener.AuditListener : AuditEvent [timestamp=Fri Apr 29 14:53:29 EDT 2016, principal=administrator, type=AUTHENTICATION_SUCCESS, data={details=remoteAddress=172.16.43.1, tokenType=BearertokenValue=&#x3C;TOKEN>}] </code></pre> <h3>JSON response indicates expired OAuth 2 token</h3> <p>Each OAuth 2 token expires after ten minutes of inactivity. The following example shows an HTTP 401 error response that indicates an expired token:</p> <pre><code>{ "error": "invalid_token", "message": "Access token expired due to inactivity: 4a1da31f-5405-4a93- af5c-799403ea70d6" } </code></pre> <p>Tokens expire completely after 24 hours even with continuous activity during this time period. The following example shows an HTTP 401 error response that indicates a token that has expired after 24 hours:</p> <pre><code>{ "error": "invalid_token", "message": "Access token expired: 4a1da31f-5405-4a93-af5c-799403ea70d6" } </code></pre> <p>If a token expires, you can request a new token.</p> <h3>Expired OAuth 2 token log messages</h3> <p><code>svt-rest-api.log</code> contains messages to indicate a token that has expired due to inactivity. For example:</p> <pre><code>2016-04-29 16:58:07.953 DEBUG 14950 --- [ qtp39959931-18] c.s.restapi.security.SvtTokenServices : Time elapsed (ms: 66702 since last update of access token: ec7b184f-b591-4609-bee8-96777339cf0e Chapter 2: Getting started 27 2016-04-29 16:58:09.372 DEBUG 14950 --- [ SessionTickler] c.s.restapi.security.SessionTickler : HMS Session 16bc9cec-37eb-4a11- b620-189c9d420410 tickled. 2016-04-29 16:58:11.126 DEBUG 14950 --- [ qtp39959931-18] c.s.restapi.security.SvtTokenServices : Removing token ec7b184f-b591-4609- bee8-96777339cf0e due to inactivity. inactiveTokenExpiration is 20000 2016-04-29 16:58:14.827 DEBUG 14950 --- [ SessionTickler] c.s.restapi.security.SessionTickler : HMS Session 16bc9cec-37eb-4a11- b620-189c9d420410 tickled. 2016-04-29 16:58:15.540 DEBUG 14950 --- [ qtp39959931-18] c.s.r.security.SvtInMemoryTokenStore : Removing access token ec7b184fb591- 4609-bee8-96777339cf0e 2016-04-29 16:58:16.573 DEBUG 14950 --- [ qtp39959931-18] c.s.restapi.security.SvtSessionMap : Removing session with oauth id=ec7b184f-b591-4609-bee8-96777339cf0e 2016-04-29 16:58:16.574 DEBUG 14950 --- [ qtp39959931-18] c.s.restapi.security.SvtSessionMap : Session count after remove: 0 2016-04-29 16:58:16.575 DEBUG 14950 --- [ qtp39959931-18] c.s.restapi.security.SvtSessionMap : Expired session count after remove: 0 2016-04-29 16:58:20.820 INFO 14950 --- [ qtp39959931-18] o.s.b.a.audit.listener.AuditListener : AuditEvent [timestamp=Fri Apr 29 16:58:20 EDT 2016, principal=access-token, type=AUTHENTICATION_FAILURE, data={type=org.springframework.security.authentication.BadCredentialsException, message=Access token expired due to inactivity: ec7b184f-b591-4609- bee8-96777339cf0e}] </code></pre> <p><code>svt-rest-api.log</code> contains a message to indicate a token that has expired after 24 hours. For example:</p> <pre><code>2016-04-28 10:17:27.605 INFO 23921 --- [qtp1909078861-30] o.s.b.a.audit.listener.AuditListener : AuditEvent [timestamp=Thu Apr 28 10:17:27 EDT 2016, principal=access-token, type=AUTHENTICATION_FAILURE, data={type=org.springframework.security.authentication.BadCredentialsException, message=Access token expired: 2d65e35f-6306-40ce-a8b8-5a8278e44e73}] </code></pre>https://developer.hpe.com/hpe-simplivity/errors-and-exceptions/https://developer.hpe.com/hpe-simplivity/errors-and-exceptions/<h1>About REST API errors and exceptions</h1> <h2>Shallow validation REST API errors</h2> <p>The service performs shallow validation to identify errors immediately upon the submission of an operation. In this case, the service does not create a task instance for the operation. Rather, it returns an HTTP response, such as a 400 (Bad Request) error, that includes a JSON body to explain the error.</p> <p>The following example shows the JSON body of an error response for an attempt to rename a policy using a name that is already in use:</p> <pre><code> "exception": "com.simplivity.rpc.common.TaskException", "path": "/api/policies/0fa0d890-6847-4eb0-8480-78d6fc3a9913/rename", "error_code": "13", "message": "Duplicate name exists.", "timestamp": 1460129027118, "status": "400" </code></pre> <p>In this example, the error occurred because the attempted to rename a policy using a name that already existed in the . The JSON body includes the following information:</p> <ul> <li>path: displays the affected object and attempted operation.</li> <li>error_code: displays the numeric code that uniquely identifies the error that occurred</li> <li>message: displays a description of the error that occurred</li> <li>timestamp: displays the date and time at which the error occurred</li> <li>status: Displays the status of the operation, that is, the HTTP response code</li> </ul> <p>The REST API service also performs certain additional validation checks before creating a task for the operation. These additional checks serve to identify crucial errors immediately.</p> <h2>Deep validation REST API errors</h2> <p>The REST API performs deep validation to identify errors after the submission of an operation. If an operation fails after the service has created a task instance for the operation, the task instance reports error information. In this case, you can GET the task instance, and this GET returns HTTP response 200. The following example shows the JSON body of a task instance that reports an error caused by an attempt to move a virtual machine while it is powered on:</p> <pre><code> "task": { "id": "420b50a4-99b0-d27a-4f28-ea2c9e899da0:420b50a4-99b0-d27a-4f28-ea2c9e899da0:20102ad9-69c1-46b3-b70d-171fc5df033f", "state": "FAILED", "message": "The VM power state is not acceptable.", "affected_objects": [], "error_code": 130, "start_time": "2016-05-09T20:37:08Z", "end_time": "2016-05-09T20:37:09Z" } } </code></pre> <p>The JSON body of the task instance for a failed operation includes the following information:</p> <ul> <li>id: displays the unique identifier for the task</li> <li>state: displays the state of the task, that is, FAILED</li> <li>message: displays a description of the error that occurred</li> <li>affected_objects: displays information about the objects that were the targets of the failed operation</li> <li>error_code: displays the numeric code that uniquely identifies the error that occurred</li> <li>start_time: displays the date and time at which the failed task started</li> <li>end_time: displays the date and time at which the failed task ended with an error</li> </ul> <h2>ID not found errors</h2> <p>If you specify an ID in a GET or other operation, and this ID does not exist for the object type that you have specified, the REST API service returns HTTP response 404, along with a JSON error body, such as the following example:</p> <pre><code>{ "exception": "com.simplivity.restapi.exceptions.ObjectNotFoundException", "path": "/api/backups/badid/rename", "message": "Object type: backup id: badid could not be found", "timestamp": 1460140207690, "status": "404" } </code></pre> <p>The JSON body of the response for an error reporting an ID that the REST API service did not find includes the following information:</p> <ul> <li>path: displays badid as the affected object and the attempted operation</li> <li>message: dDisplays a description of the error that occurred</li> <li>timestamp: displays the date and time at which the error occurred</li> <li>status: displays the status of the operation, that is, the HTTP response code</li> </ul>https://developer.hpe.com/hpe-simplivity/feature-and-function-support-by-rest-api-version/https://developer.hpe.com/hpe-simplivity/feature-and-function-support-by-rest-api-version/<h1>Feature and function support by REST API version</h1> <p>The tables in this section list the features and functions that each version of the REST API supports for the following object types:</p> <ul> <li>backups</li> <li>cluster_groups</li> <li>datastores</li> <li>hosts</li> <li>omnistack_clusters</li> <li>policies</li> <li>security/certificates</li> <li>tasks</li> <li>virtual_machines</li> </ul> <hr> <h1><code>backups</code></h1> <p>The REST API supports the following features and functions for the <code>backups</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>consistency_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>sent_completion_time</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>sent_duration</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>unique_size_bytes</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>unique_size_timestamp</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>virtual_machine_state</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>virtual_machine_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET virtual_disks</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET virtual_disk_partitions</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET virtual_disk_partition_files</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>DELETE</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST calculate_unique_size</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST cancel</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST copy</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>POST delete</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST lock</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>POST rename</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST restore</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST restore_file</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST restore_files</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST set_retention</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> </tbody> </table> <h1><code>cluster_groups</code></h1> <p>The REST API supports the following features and functions for the <code>cluster_groups</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST rename</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> </tbody> </table> <h1>datastores</h1> <p>The REST API supports the following features and functions for the <code>datastore</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>hypervisor_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>DELETE</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>GET standard_hosts</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST resize</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>POST set_policy</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>POST share</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>Post unshare</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> </tbody> </table> <h1>hosts</h1> <p>The REST API supports the following features and functions for the <code>host</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>can_rollback</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>current_feature_level</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>date</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>federation_ip</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>federation_mask</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>federation_mtu</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>hypervisor_management_system_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>infosight_configuration</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>life_remaining</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>management_mask</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>management_mtu</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>omnistack_cluster_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>policy_enabled</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>potential_feature_level</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>storage_ip</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>storage_mask</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>storage_mtu</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>upgrade_state</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>virtual_controller_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET capacity</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>GET hardware</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>GET metrics</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>GET virtual_controller_shutdown_status</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST cancel_virtual_controller_shutdown</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST remove_from_federation</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST shutdown_virtual_controller</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> </tbody> </table> <h1>omnistack_clusters</h1> <p>The REST API supports the following features and functions for the <code>omnistack_cluster</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>arbiter_address</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>arbiter_connected</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>cluster_feature_level</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>connected_cluster</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>hypervisor_object_parent_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_object_parent_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>infosight_configuration</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>time_zone</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>upgrade_state</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>upgrade_task_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>version</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET connected_clusters</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET metrics</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>GET throughput</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET time_zone_list</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST set_time_zone</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> </tbody> </table> <h1>policies</h1> <p>The REST API supports the following features and functions for the <code>policy</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>GET datastores</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>GET /policies/policy_schedule_report</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>GET virtual_machines</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>DELETE</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST policies</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST rename</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>POST resume</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST rules</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X (multiple rules)</td> <td>X (one rule)</td> </tr> <tr> <td>POST suspend</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST impact_report/create_rules</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST impact_report/edit_rules</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST impact_report/delete_rule</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>PUT rule</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> </tbody> </table> <h1><code>security/certificates</code></h1> <p>The REST API supports the following features and functions for the <code>security/certificates</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hash</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>certificate</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>subject</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>issuer</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>serialno</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>DELETE</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> </tbody> </table> <h1>tasks</h1> <p>The REST API supports the following features and functions for the <code>task</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> </tbody> </table> <h1>virtual_machines</h1> <p>The REST API supports the following features and functions for the <code>virtual_machine</code> object type:</p> <table> <thead> <tr> <th>Operation</th> <th>1.13</th> <th>1.12</th> <th>1.11</th> <th>1.10</th> <th>1.9</th> <th>1.8</th> <th>1.7</th> <th>1.6</th> <th>1.5</th> <th>1.4</th> <th>1.3</th> <th>1.2</th> <th>1.1</th> <th>1</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>Additional returned properties:</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>adapter_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>app_aware_vm_status</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_hypervisor_object_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>compute_cluster_parent_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>deleted_at</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>device_number</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>network_interfaces</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>network_label</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>ha_resynchronization_progress</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>host_id</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>hypervisor_allocated_capacity</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_allocated_cpu</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_consumed_cpu</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_consumed_memory</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_cpu_count</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_free_space</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_folder_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_is_template</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_management_system_name</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_total_memory</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_type</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_virtual_disk_count</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>hypervisor_virtual_machine_power_state</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> </tr> <tr> <td>modified_at</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>ipv4_addresses</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>mac_address</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>mac_generation</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>network_interfaces</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>replica_set</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>GET backups</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>GET metrics</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> </tr> <tr> <td>POST backup</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST backup_parameters</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST clone</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST move</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST set_policy</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> </tr> <tr> <td>POST validate_backup_credentials</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST policy_impact_report/apply_policy</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> <tr> <td>POST power_off</td> <td>X</td> <td>X</td> <td>X</td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> <td></td> </tr> </tbody> </table>https://developer.hpe.com/hpe-simplivity/enumerations/https://developer.hpe.com/hpe-simplivity/enumerations/<h1>Enumerations</h1> <p>Use this section as a reference for the REST API enumerations (enums).<br> The REST API always performs sorting and filtering on enumerated values using the full uppercase enumerations. The REST API ignores the value of the case parameter when you sort or filter by fields with enumerated values.</p> <hr> <h2>adapter_type</h2> <p>Used by: <code>NetworkInterface</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td><code>E1000</code>, <code>E1000E</code>, <code>PCNET32</code>, <code>VMXNET</code>, <code>VNXNET2</code>, and <code>VMXNET3</code></td> <td>The <code>adapter_type</code> assigned to the virtual machine. For more information about these values, see the VMware vSphere documentation.</td> </tr> <tr> <td><code>UNKNOWN</code></td> <td>The <code>adapter_type</code> cannot be determined.</td> </tr> </tbody> </table> <h2>app_aware_type</h2> <p>Used by: <code>backup_parameters</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>DEFAULT</td> <td>Crash-consistent.</td> </tr> <tr> <td>NONE</td> <td>Application-consistent backup using a VMware snapshot.</td> </tr> <tr> <td>VSS</td> <td>This backup is an application aware backup with Microsoft VSS.</td> </tr> </tbody> </table> <h2>app_aware_vm_status</h2> <p>Used by: <code>virtual_machines</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>CAPABLE</td> <td>The virtual machine is ready and can take application aware snapshots.</td> </tr> <tr> <td>INVALID_CREDS</td> <td>A valid set of credentials for the virtual machine is not available, which prevents application aware snapshots of this virtual machine.</td> </tr> <tr> <td>INVALID_OS</td> <td>The virtual machine is not running an appropriate operating system (OS) which prevents application aware snapshots of this virtual machine.</td> </tr> <tr> <td>UNKNOWN</td> <td>The state of the virtual machine with respect to readiness for an application aware snapshot is not known.</td> </tr> <tr> <td>UNKNOWN_FAULT</td> <td>A general failure has occurred that prevents application aware snapshots of the virtual machine.</td> </tr> <tr> <td>VALID_CREDS</td> <td>A valid set of credentials for the virtual machine is available which enables application aware snapshots of this virtual machine.</td> </tr> <tr> <td>VMWARE_TOOLS_UNAVAILABLE</td> <td>VMware Tools are not installed or not running on the virtual machine which prevents application aware snapshots of this virtual machine.</td> </tr> </tbody> </table> <h2>consistency_type</h2> <p>Used by: <code>backup</code> object, <code>BackupVMMO</code> object, Policy rule, Policy rule POST body, create_or_edit_rule, rule</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>DEFAULT</td> <td>An application consistent backup with VMware Snapshots.</td> </tr> <tr> <td>NONE</td> <td>A crash consistent backup or a backup that is not application consistent.</td> </tr> <tr> <td>VSS</td> <td>An application aware backup with Microsoft VSS.</td> </tr> </tbody> </table> <h2>ha_status</h2> <p>Used by: <code>virtual_machine</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>DEFUNCT</td> <td>Object no longer exists but is still being referenced.</td> </tr> <tr> <td>DEGRADED</td> <td>Object is not fully replicated.</td> </tr> <tr> <td>NOT_APPLICABLE</td> <td>Object is intentionally not replicated.</td> </tr> <tr> <td>OUT_OF_SCOPE</td> <td>Status not available from this location.</td> </tr> <tr> <td>SAFE</td> <td>Object is replicated.</td> </tr> <tr> <td>SYNCING</td> <td>Object is becoming replicated.</td> </tr> <tr> <td>UNKNOWN</td> <td>Status could not be determined.</td> </tr> </tbody> </table> <h2>hypervisor_type</h2> <p>Used by: <code>backup</code> object, <code>datastore</code> object, <code>omnistack_cluster</code>, <code>virtual_machine</code> object, <code>BackupMO</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>VSPHERE</td> <td>The Hypervisor Management System (HMS) associated with this backup is vSphere.</td> </tr> <tr> <td>HYPERV</td> <td>The HMS associated with this backup is Hyper-V.</td> </tr> </tbody> </table> <h2>hypervisor_virtual_machine_power_state</h2> <p>Used by: <code>virtual_machine</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>OFF</td> <td>The Hypervisor Management System (HMS) considers the virtual machine to be powered off.</td> </tr> <tr> <td>ON</td> <td>The HMS considers the virtual machine to be powered on.</td> </tr> <tr> <td>SUSPENDED</td> <td>The HMS considers the virtual machine to be suspended.</td> </tr> <tr> <td>UNKNOWN</td> <td>The hypervisor-based power state of the virtual machine cannot be determined.</td> </tr> </tbody> </table> <h2>mac_generation</h2> <p>Used by: <code>NetworkInterface</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>ASSIGNED</td> <td>The MAC address is automatically assigned when you power on the virtual machine.</td> </tr> <tr> <td>GENERATED</td> <td>The MAC address was automatically generated.</td> </tr> <tr> <td>MANUAL</td> <td>The MAC address was statically assigned.</td> </tr> <tr> <td>UNKNOWN</td> <td>The mac_generation type could not be determined.</td> </tr> </tbody> </table> <h2>media_type</h2> <p>Used by: <code>physical_drive</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>HDD</td> <td>Hard disk drive.</td> </tr> <tr> <td>SSD</td> <td>Solid-state drive.</td> </tr> <tr> <td>UNKNOWN</td> <td>The <code>media_type</code> could not be determined.</td> </tr> </tbody> </table> <h2>role</h2> <p>Used by: <code>ReplicaInfo</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>PRIMARY</td> <td>The ReplicaInfo id represents the ID of the host that contains the primary data replica of the virtual machine.</td> </tr> <tr> <td>SECONDARY</td> <td>The ReplicaInfo id represents the ID of the host that contains the secondary data replica of the virtual machine.</td> </tr> </tbody> </table> <h2>state</h2> <p>Used by:<code>backup</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>CANCELED</td> <td>The backup operation was canceled successfully.</td> </tr> <tr> <td>CANCELING</td> <td>The backup operation is responding to a manual cancellation of a backup in progress.</td> </tr> <tr> <td>DEGRADED</td> <td>The backup is in an unprotected high availability state. This situation can occur when an HPE OmniStack host in the backup replica set is replaced by another HPE OmniStack host, and the backup has been saved to one HPE OmniStack host in a datacenter with multiple HPE OmniStack hosts.</td> </tr> <tr> <td>DELETED</td> <td>The backup data was deleted before the expiration of the backup. This situation can occur when removing an HPE OmniStack host may remove the last copy of a backup in a datacenter.</td> </tr> <tr> <td>FAILED</td> <td>The backup was unsuccessful.</td> </tr> <tr> <td>NEW</td> <td>The backup operation started, but the initial backup of the virtual machine and processing of the backup on the source datacenter is not complete.</td> </tr> <tr> <td>PROTECTED</td> <td>The backup is successful and is in a protected high availability (HA) state. If the backup was a remote backup, successful replication to the remote site has also completed.</td> </tr> <tr> <td>QUEUED</td> <td>The backup is waiting to be replicated to a remote datacenter.</td> </tr> <tr> <td>REBUILDING</td> <td>The backup data is rebuilding onto a second HPE OmniStack host to ensure high availability for the backup in a multi-node datacenter.</td> </tr> <tr> <td>SAVING</td> <td>The backup replication is in progress. The state changes to <code>Protected</code>, <code>Queued</code>, <code>Failed</code>, or <code>Degraded</code>.</td> </tr> <tr> <td>UNKNOWN</td> <td>The backup state cannot be determined.</td> </tr> </tbody> </table> <h2>state</h2> <p>Used by: <code>host</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>ALIVE</td> <td>The HPE OmniStack host is healthy.</td> </tr> <tr> <td>FAULTY</td> <td>The HPE OmniStack host is in a critical error state, and operations have failed over to an alternate HPE OmniStack host in the federation. It is likely that one or more error or event messages were logged.</td> </tr> <tr> <td>MANAGED</td> <td>The Virtual Controller for this host is offline but can still be managed.</td> </tr> <tr> <td>REMOVED</td> <td>The HPE OmniStack host has been removed from the federation but is still being recognized.</td> </tr> <tr> <td>SUSPECTED</td> <td>The HPE OmniStack host has one or more components that show degraded performance.</td> </tr> <tr> <td>UNKNOWN</td> <td>The HPE OmniStack host status is indeterminate, perhaps because it is unable to communicate with other federation HPE OmniStack hosts. It is possible that one or more error or event messages were logged.</td> </tr> </tbody> </table> <h2>state</h2> <p>Used by: <code>task</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>COMPLETED</td> <td>The task has completed successfully.</td> </tr> <tr> <td>FAILED</td> <td>The task has failed.</td> </tr> <tr> <td>IN_PROGRESS</td> <td>The task is currently being processed.</td> </tr> </tbody> </table> <h2>state</h2> <p>Used by: <code>virtual_machine</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>ALIVE</td> <td>An active hypervisor-based virtual machine is associated with the <code>virtual_machine</code> object.</td> </tr> <tr> <td>DELETED</td> <td>The hypervisor-based virtual machine that is associated with the <code>virtual_machine</code> object has been deleted with at least one backup of that virtual machine still existing.</td> </tr> <tr> <td>REMOVED</td> <td>The hypervisor-based virtual machine that is associated with the <code>virtual_machine</code> object has been removed from the virtual machine inventory of the hypervisor.</td> </tr> </tbody> </table> <h2>status</h2> <p>Used by: <code>credential_validation</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>VM_POWERED_OFF</td> <td>The virtual machine is powered off, so it is not possible to determine if the credentials are valid.</td> </tr> <tr> <td>INVALID</td> <td>Credentials are invalid.</td> </tr> <tr> <td>VALID</td> <td>Credentials are valid.</td> </tr> </tbody> </table> <h2>type</h2> <p>Used by:<code>backup</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>MANUAL</td> <td>A user created this backup manually. Manual backups are not deleted automatically.</td> </tr> <tr> <td>POLICY</td> <td>An automatic policy created this backup. The backup is subject to automatic deletion when the retention time for the backup expires or the maximum number of backups is exceeded. The oldest backups are deleted first.</td> </tr> <tr> <td>UNSPECIFIED</td> <td>The backup type cannot be determined.</td> </tr> </tbody> </table> <h2>type</h2> <p>Used by: <code>omnistack_cluster</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>CLOUD</td> <td>The <code>omnistack_cluster</code> is comprised of HPE OmniStack Cloud hosts.</td> </tr> <tr> <td>OMNISTACK</td> <td>The <code>omnistack_cluster</code> is comprised of HPE OmniStack hosts that share resources and provide high availability and load-balancing services.</td> </tr> <tr> <td>UNKNOWN</td> <td>The type could not be determined.</td> </tr> </tbody> </table> <h2>upgrade_state</h2> <p>Used by: <code>host</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>FAIL</td> <td>The HPE OmniStack host upgrade task failed, and an error code and message indicate the reason for the failure.</td> </tr> <tr> <td>IN_PROGRESS</td> <td>The HPE OmniStack upgrade task is proceeding.</td> </tr> <tr> <td>NOOP</td> <td>The HPE OmniStack host upgrade is incomplete. For example, a host may have failed to upgrade successfully, and the upgrade needs to be repeated, but this host is already at the correct version and does not need to be upgraded again.</td> </tr> <tr> <td>SUCCESS</td> <td>The HPE OmniStack host upgrade task completed successfully.</td> </tr> <tr> <td>UNKNOWN</td> <td>It is not possible to determine the status of the HPE OmniStack host upgrade task.</td> </tr> </tbody> </table> <h2>upgrade_state</h2> <p>Used by: <code>omnistack_cluster</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>FAIL_CAN_ROLLBACK</td> <td>At least one software upgrade for this <code>omnistack_cluster</code> is in a state that cannot be upgraded, committed, or rolled back. This state should rarely occur. Contact Customer Support (support.hpe.com).</td> </tr> <tr> <td>FAIL_CANNOT_ROLLBACK</td> <td>One or more HPE OmniStack hosts failed the upgrade. The upgrade either rolled back automatically or failed before a rollback was required. Attempt the upgrade and, if it fails again, contact Customer Support (support.hpe.com).</td> </tr> <tr> <td>IN_PROGRESS</td> <td>The upgrade task is proceeding.</td> </tr> <tr> <td>SUCCESS_COMMIT_NEEDED</td> <td>The upgrade is ready to commit (or roll back).</td> </tr> <tr> <td>SUCCESS_COMMITTED</td> <td>The <code>omnistack_cluster</code> is committed to the current software version. Commits occur at the federation level, so all <code>omnistack_cluster</code> instances in the federation should be in this state.</td> </tr> <tr> <td>SUCCESS_MIXED_VERSION</td> <td>None of the HPE OmniStack hosts in the have an upgrade in progress, but the <code>omnistack_cluster</code> has mixed versions of software on different HPE OmniStack hosts. An upgrade is needed to make the <code>omnistack_cluster consistent</code>. This state can occur when a node with a different software version is added to the <code>omnistack_cluster</code>. An upgrade is required to ensure that all HPE OmniStack hosts are running the same version.</td> </tr> <tr> <td>UNKNOWN</td> <td>It is not possible to determine the status of the previous upgrade task.</td> </tr> </tbody> </table> <h2>virtual_machine_state</h2> <p>Used by: <code>backup</code> object, <code>BackupMO</code> object</p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>ALIVE</td> <td>An active hypervisor-based virtual machine is associated with the <code>virtual_machine</code> object.</td> </tr> <tr> <td>DELETED</td> <td>The hypervisor-based virtual machine that is associated with the <code>virtual_machine</code> object has been deleted with at least one backup of that virtual machine still existing.</td> </tr> <tr> <td>REMOVED</td> <td>The hypervisor-based virtual machine that is associated with the <code>virtual_machine</code> object has been removed from the virtual machine inventory of the hypervisor.</td> </tr> </tbody> </table> <h2>virtual_machine_type</h2> <p>Used by: <code>BackupMO</code></p> <table> <thead> <tr> <th>Element</th> <th>Definition</th> </tr> </thead> <tbody> <tr> <td>TEMPLATE</td> <td>The virtual machine instance associated with the backup is a virtual machine template.</td> </tr> <tr> <td>UNKNOWN</td> <td>The type of the virtual machine instance associated with the backup is unknown.</td> </tr> <tr> <td>VM</td> <td>The virtual machine instance associated with the backup is a virtual machine.</td> </tr> </tbody> </table>https://developer.hpe.com/hpe-simplivity/get-backups/https://developer.hpe.com/hpe-simplivity/get-backups/<h1>Getting backups</h1> <p>This example uses <code>curl</code> to retrieve a list of all the backups in the federation along with details about each backup.</p> <pre><code>curl -H "Accept: application/vnd.simplivity.v1+json" -H "Authorization: Bearer [*access_token*]" -X GET -k -i https://[*host*]/api/backups </code></pre> <p>Where:</p> <ul> <li> <p><code>-H</code> (or <code>--header</code>) enables curl to add request headers.</p> </li> <li> <p>The <code>"Accept: application/vnd.simplivity.v1+json"</code> header indicates that the response should use version 1 of the HPE OmniStack JSON extension.</p> </li> <li> <p><em>access_token</em> is the complete access token, for example, <code>f93f2059- afef-4310-9147-447645992a5d</code>.</p> </li> <li> <p><code>-X GET</code> indicates an HTTP GET operation.</p> </li> <li> <p><code>-k</code> allows the use of self-signed SSL/TLS certificates.</p> </li> <li> <p><em>host</em> is the IP address of the Virtual Controller that you want to authenticate.</p> </li> </ul> <p>The GET request returns a JSON response similar to the following example:</p> <pre><code>{ "offset": 0, "limit": 500, "count": 21, "backups": [ { "id": "f6d5980c-478c-4292-aabf-67da627dbc57", "name": "vm32_A_6-backup-2016-03-13T21:03:46-04:00", "sent": 0, "state": "PROTECTED", "type": "MANUAL", "omnistack_cluster_id": "0bf3988b-b684-4002-a04d-55bb8831b8c4", "omnistack_cluster_name": "Boston -:- OmniCube CN-2000 5.5.0", "datastore_id": "c1d172e2-bd89-4bad-ae2a-848e7fbe3dc4", "datastore_name": "ds2", "expiration_time": "NA", "virtual_machine_id": "729861fd-a006-4849-bf08-b232551989c0", "virtual_machine_name": "vm32_A_6", "size": 12574720, "application_consistent": false, "created_at": "2016-03-14T01:03:46Z" }, ... ] } </code></pre> <p>In the response, the count value indicates that there are a total of 21 backup objects (that is, resources). The response returns these backups in a JSON array of objects. The offset and limit values indicate that the response array starts at the backup with an offset of 0 and includes backups up to the maximum limit of 500 objects. These values for the offset and limit are the default values.</p>https://developer.hpe.com/hpe-simplivity/getting-a-certificate/https://developer.hpe.com/hpe-simplivity/getting-a-certificate/<h1>Getting certificates from the trust store</h1> <p>Use the <code>Get-HPESvtCertificate</code> cmdlet to list the all certificates in the trust store or to search for a specific one by the certificate's SHA1 thumbprint. To use the <code>Get-HPESvtCertificate</code>, you must be an authenticated user.</p> <p>This example shows how to get the list of certificates. It assumes that you have already authenticated.</p> <pre><code>PS C:\ Get-HPESvtCertificate | Format-List Thumbprint Subject ---------- ------- 7E084B44FCDAB6EF74AC40E8D9BD508337D96CDE CN=omnicubexx.us-east.myco.local 2F43831D62C01FD2AF448DFA8BF86E126B8190DD CN=omnicubexx.us-east.myco.local 77E44C4DFF0E565665B6CB27E77BA8FBAFFF88A2 CN==omnicubexx.us-east.myco.local </code></pre> <p>This example shows how to search by SHA1 thumbprint:</p> <pre><code>PS C:\ Get-HPESvtCertificate -Thumbprint EB2CD509BA53...D4C6 Thumbprint Subject EnhancedKeyUsageList --------------- --------- --------------------------- EB2CD509BA53...D4C6 OU=VMware Enginee... </code></pre>https://developer.hpe.com/hpe-simplivity/getting-a-root-certificate/https://developer.hpe.com/hpe-simplivity/getting-a-root-certificate/<h1>Getting root certificates</h1> <p>The <code>Get-HPESvtRootCertificate</code> cmdlet returns an array of X509Certificate2 objects from the vCenter Server. This cmdlet does not require that you authenticate.</p> <p>When you use this cmdlet, you must independently verify that the certificate is what you expect it to be because the command does not perform validation checks on the certificate.</p> <pre><code>Get-HPESvtRootCertificate -hmsHostname 192.0.2.2 | Format-list Subject : OU=VMware Engineering, O=vc-147-50-53, S=California, C=US, DC=local, DC=vsphere, CN=CA Issuer : OU=VMware Engineering, O=vc-147-50-53, S=California, C=US, DC=local, DC=vsphere, CN=CA Thumbprint : 83754908A5088D883FBC86275EF1FD964E2153C0 FriendlyName : NotBefore : 7/9/2017 10:48:12 PM NotAfter : 7/7/2027 10:48:12 PM Extensions : {System.Security.Cryptography.Oid, System.Security.Cryptography.Oid, System.Security.Cryptography.Oid, System.Security.Cryptography.Oid} </code></pre> <p>Use the <code>Get-HPESvtRootCertificate</code> cmdlet in conjunction with the <code>Add-HPESvtCertificate</code> cmdlet to populate the certificate trust store on the Virtual Controller.</p>https://developer.hpe.com/hpe-simplivity/home/https://developer.hpe.com/hpe-simplivity/home/<h2>HPE OmniStack REST API</h2> <p>The HPE OmniStack REST API enables tool developers and integrators to manage HPE OmniStack assets efficiently, intuitively, and securely. Anyone can use the REST API to gather information about, analyze, configure, and troubleshoot HPE OmniStack hosts and federations. Moreover, tools developers can use the REST API to automate these tasks.</p> <hr> <h3>About the API</h3> <p><a href="/platform/hpe-simplivity/whats-new/"><strong>What's New</strong></a></p> <p><a href="/platform/hpe-simplivity/feature-and-function-support-by-rest-api-version"><strong>Feature and function support by REST API version</strong></a></p> <p><a href="/platform/hpe-simplivity/architecture-and-object-model"><strong>Architecture and object model</strong></a></p> <p><a href="/platform/hpe-simplivity/enumerations"><strong>Enumerations</strong></a></p> <p><a href="/platform/hpe-simplivity/interactive-rest-api-reference"><strong>Interactive REST API reference</strong></a></p> <p><a href="/platform/hpe-simplivity/errors-and-exceptions"><strong>About REST API errors and exceptions</strong></a></p> <p><a href="/platform/hpe-simplivity/http-error-code-reference"><strong>HTTP error code reference</strong></a></p> <p><a href="/platform/hpe-simplivity/authorization-related-rest-api-log-messages-and-responses"><strong>Authorization-related REST API log messages and responses</strong></a></p> <hr> <h3>Use the API</h3> <p><a href="/platform/hpe-simplivity/sample-code"><strong>Getting started</strong></a></p> <p><a href="/platform/hpe-simplivity/get-backups"><strong>Getting backups</strong></a></p> <p><a href="/platform/hpe-simplivity/restore-virtual-machines"><strong>Restoring virtual machines</strong></a></p> <p><a href="/platform/hpe-simplivity/versioning"><strong>Getting the API version</strong></a></p> <p><a href="/platform/hpe-simplivity/optimize-get-requests"><strong>Optimizing GET requests</strong></a></p> <p><a href="/platform/hpe-simplivity/metrics"><strong>Getting performance and capacity metrics</strong></a></p> <p><a href="/platform/hpe-simplivity/powershell"><strong>Using the API with PowerShell</strong></a></p> <p><a href="/platform/hpe-simplivity/python"><strong>Using the API with Python</strong></a></p> <p><a href="/platform/hpe-simplivity/java"><strong>Using the API with Java</strong></a></p> <hr> <h3>Authenticating</h3> <p><a href="/platform/hpe-simplivity/authenticating-against-hpe-omnistack-api"><strong>Authenticating against the API</strong></a></p> <p><a href="/platform/hpe-simplivity/revoke-oath-2-token"><strong>Revoking OAuth 2 tokens</strong></a></p> <p><a href="/platform/hpe-simplivity/request-authentication-by-emergency-grant"><strong>Requesting authentication by emergency grant</strong></a></p> <hr> <h3>Manage certificates using PowerShell</h3> <p><a href="/platform/hpe-simplivity/hpesvtcli-powershell-commands"><strong>Installing the certificate management Powershell cmdlets</strong></a></p> <p><a href="/platform/hpe-simplivity/ps-getoathtoken"><strong>Authenticating</strong></a></p> <p><a href="/platform/hpe-simplivity/adding-certificates"><strong>Adding certificates to the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/getting-a-certificate"><strong>Getting certificates from the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/getting-a-root-certificate"><strong>Getting root certificates</strong></a></p> <p><a href="/platform/hpe-simplivity/remove-certificate"><strong>Removing certificates from the trust store</strong></a></p> <p><a href="/platform/hpe-simplivity/tls-certificate-validation"><strong>Disabling/enabling TLS certificate validation on HPE OmniStack for vSphere</strong></a></p> <hr> <h2>Projects</h2> <h3><a href="https://github.com/HewlettPackard/Docker-SimpliVity">Docker <img src="Github" alt=""></a></h3> <p>See the HPE SimpliVity REST API-based solution to stand up Docker’s environment automatically on top of the HPE SimpliVity HCI</p> <ul> <li><a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/Go-from-zero-to-Docker-in-minutes-with-HPE-hyperconverged/ba-p/6990234#.W2IksxYpDDs">Go from zero to Docker in minutes with HPE hyperconverged</a></li> </ul> <hr> <h3><a href="https://github.com/HewlettPackard/simplivity-vra-plugin">HPE SimpliVity plugin for VMware vRealize Automation <img src="Github" alt=""></a></h3> <p>Learn more about the HPE SimpliVity plugin for VMware vRealize Automation (VRA) which enables delivery and administration of hybrid cloud applications and services. The plugin is based on the HPE SimpliVity REST API.</p> <ul> <li><a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/Simplified-Management-Operate-HPE-SimpliVity-data-services/ba-p/7013599#.W2CpIH58thE">Simplified Management – Operate HPE SimpliVity data services directly through vRealize Automation</a></li> </ul> <hr> <h3><a href="http://github.com/HewlettPackard/simplivity-microfocus-hcm-plugin">HPE SimpliVity plugin for Micro Focus Hybrid Cloud Management <img src="Github" alt=""></a></h3> <p>Learn more about the HPE SimpliVity plugin for Micro Focus Hybrid Cloud Management.</p> <p>The plugin is based on the HPE SimpliVity REST API.</p> <ul> <li><a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/HPE-SimpliVity-integration-for-Micro-Focus-Hybrid-Cloud/ba-p/7021447">HPE SimpliVity integration for Micro Focus Hybrid Cloud Management</a></li> </ul> <hr> <h3>HPE SimpliVity and the Citrix Cloud</h3> <p>Learn how to host and administer any Citrix service on HPE SimpliVity</p> <ul> <li><a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/Just-announced-Automation-for-HPE-SimpliVity-and-Citrix-Cloud/ba-p/7022957#.W-L_m5NKiUl">Read the article</a></li> <li><a href="https://github.com/HewlettPackard/SimpliVity-Citrix-HyperV-Plugin">Download the HPE OmniStack for Hyper-V plugin for Citrix Cloud</a></li> <li><a href="https://github.com/HewlettPackard/SimpliVity-Citrix-VCenter-Plugin">Download the HPE OmniStack for Vsphere plugin for Citrix Cloud</a></li> </ul> <h2>Language bindings</h2> <h3><a href="https://github.com/HewlettPackard/hpe-simplivity-powershell">PowerShell certificate management cmdlets <img src="Github" alt=""></a></h3> <ul> <li><a href="https://www.powershellgallery.com/packages/HPESvtCmdlets">Download the cmdlets from the PowerShell Gallery</a></li> </ul> <h3><a href="https://github.com/atkinsroy/HPESimpliVity">PowerShell Module <img src="Github" alt=""></a></h3> <ul> <li><a href="https://github.com/atkinsroy/HPESimpliVity">Visit the priject website on GitHub</a></li> </ul> <h3><a href="https://github.com/HewlettPackard/simplivity-python">Python SDK <img src="Github" alt=""></a></h3> <ul> <li><a href="https://pypi.org/project/simplivity/">Download the library from pypi</a></li> </ul> <h3><a href="https://github.com/HewlettPackard/simplivity-go">Go SDK <img src="Github" alt=""></a></h3> <h3><a href="https://github.com/HewlettPackard/simplivity-ansible">Ansible Library <img src="Github" alt=""></a></h3> <ul> <li><a href="https://github.com/HewlettPackard/simplivity-ansible/releases/tag/v1.0.0">Download the library from GitHub</a></li> </ul>https://developer.hpe.com/hpe-simplivity/hpesvtcli-powershell-commands/https://developer.hpe.com/hpe-simplivity/hpesvtcli-powershell-commands/<h1>Installing the certificate management PowerShell module</h1> <p>You can download and install the HPE SimpliVity certificate management PowerShell module from the Powershell Gallery.</p> <p>The module works with Powershell Core (PS6) on Ubuntu or Windows 10 and on PowerShell 5.1 (PS5.1).</p> <h2>Installing from the PowerShell Gallery</h2> <p>To install directly from PowerShell, run the following command from a Powershell window:</p> <p><code>Install-Module -Name HPESvtCmdlets</code></p> <p>If you are reinstalling to obtain a newer version, you might need to use the -Force option. You can always uninstall a module using the <code>Uninstall-module</code> cmdlet.</p> <h2>Loading the module</h2> <p>After you have downloaded the module, load it by using the <code>import-module</code> cmdlet. For example:</p> <p><code>Import-Module .\HpeSvtCmdlets\HPESvtCmdlets.psm1</code></p> <p>To verify that the module imported properly, run the following command and verify that it contains the full set of cmdlets.</p> <p><code>get-module HpeSvtCmdlets</code></p> <p>You see the list of cmdlets in the current version. For example:</p> <pre><code>ModuleType Version Name ExportedCommands ---------- ------- ---- ---------------- Script 1.1.21.0 HPESvtCmdlets {Add-HPESvtCertificate, Get-HPESvtAuthToken, Get-HPESvtCertificate, Get-HPESvtRootCertificate...} </code></pre> <p>You can get help on any command by calling <code>Get-Help</code> (or just <code>help</code> for short). For example:</p> <p><code>Get-Help Add-HPESvtCertificate</code></p>https://developer.hpe.com/hpe-simplivity/http-error-code-reference/https://developer.hpe.com/hpe-simplivity/http-error-code-reference/<h2>HTTP error code reference</h2> <p>Use this reference for information about the HTTP error codes returned by the REST API. The error codes are returned in the <code>status</code> field of the JSON error response body.</p> <h3>400-Bad Request</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>InvalidBodyException</td> <td>You tried to pass invalid information in the body of a POST or PUT request.</td> </tr> <tr> <td>InvalidParameterException</td> <td>You passed an invalid query parameter.</td> </tr> <tr> <td>InvalidSortCriteriaException</td> <td>You passed an invalid limit or sort criteria query parameter.</td> </tr> <tr> <td>ObjectInUseException</td> <td>You tried to DELETE a policy that is in use by a datastore or virtual machine.</td> </tr> <tr> <td>PermissionDeniedException</td> <td>You tried to view or modify an object without the required permissions.</td> </tr> <tr> <td>TypeMismatchException</td> <td>You passed in an incorrect object type. For example, you passed in a String, but an Integer was expected.</td> </tr> </tbody> </table> <h3>401---Unauthorized</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>SvtExceptions$UnauthorizedException</td> <td>You submitted an invalid access token or the access token has expired. Tokens expire after 24 hours or after 10 minutes of inactivity.</td> </tr> </tbody> </table> <h3>404---Not Found</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>ObjectNotFoundException</td> <td>You tried to GET or act on an object, but the object cannot be found.</td> </tr> </tbody> </table> <h3>405---Method Not Allowed</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>HttpRequestMethodNotSupportedException</td> <td>You tried to perform an HTTP operation on a resource that is not supported.</td> </tr> </tbody> </table> <h3>413---Payload Too Large</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>BodyTooLargeException</td> <td>You submitted a request body that is too large.</td> </tr> </tbody> </table> <h3>415---Unsupported Media Type</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>HttpMediaTypeNotSupportedException</td> <td>You submitted a PUT or POST request and the content type that is not valid. For example: <code>Content type 'application/xml' not supported.</code></td> </tr> </tbody> </table> <h3>500---Internal Server Error</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>InternalServerException</td> <td>You submitted a request, there are unexpected problems. Check the log for more details.</td> </tr> </tbody> </table> <h3>502---Bad Gateway</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>Bad Gateway</td> <td>There are too many simultaneous REST requests.</td> </tr> </tbody> </table> <h3>504---Gateway Timeout</h3> <table> <thead> <tr> <th>Exception</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>Gateway Time-out</td> <td>The REST request took more than 60 seconds to complete.</td> </tr> </tbody> </table>https://developer.hpe.com/hpe-simplivity/interactive-rest-api-reference/https://developer.hpe.com/hpe-simplivity/interactive-rest-api-reference/<h1>Interactive REST API reference</h1> <p>The REST API service provides a comprehensive, intuitive, and interactive real-time reference as a web page on all Virtual Controllers that are running the REST API service. The reference describes each object type, along with the supported operations, request input parameters, response model, and response codes. Furthermore, the REST API reference provides the Try it out! option that enables you to perform REST API operations on a federation. You must log in to an HPE OmniStack host to use the Try it out! option.</p> <p>To access the interactive REST API reference, obtain the IP address of a Virtual Controller in the federation that you want to manage. Then, in a web browser, navigate to the following URL:</p> <pre><code> https://[Virtual Controller IP address]/api/index.html </code></pre>https://developer.hpe.com/hpe-simplivity/java/https://developer.hpe.com/hpe-simplivity/java/<h1>Using the API with Java</h1> <p>This sample Java code performs authentication, issues example GET requests, performs a POST operation (in this case, renaming a backup), and monitors the status of the operation using a task instance.</p> <pre><code>package main.java; import java.security.SecureRandom; import java.security.cert.X509Certificate; import java.util.Arrays; import javax.net.ssl.HttpsURLConnection; import javax.net.ssl.SSLContext; import javax.net.ssl.TrustManager; import javax.net.ssl.X509TrustManager; import javax.net.ssl.SSLSession; import javax.net.ssl.HostnameVerifier; import org.springframework.http.HttpEntity; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpMethod; import org.springframework.http.MediaType; import org.springframework.http.ResponseEntity; import org.springframework.util.LinkedMultiValueMap; import org.springframework.util.MultiValueMap; import org.springframework.web.client.RestTemplate; import org.json.JSONObject; import com.sun.org.apache.xerces.internal.impl.dv.util.Base64; public class RestClient { private static String access_token; private String BASE_URL; static final String HMS_USERNAME = "HMS_USER"; static final String HMS_PASSWORD = "HMS_PASS"; public RestClient(String hostIp) { BASE_URL = "https://"+hostIp+"/api/"; } // Create a trust manager that does not validate certificate chains. private void enableSSL() { TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() { public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; } public void checkClientTrusted(X509Certificate[] certs, String authType) { } public void checkServerTrusted(X509Certificate[] certs, String authType) { } } }; try { SSLContext sc = SSLContext.getInstance("TLSv1.2"); sc.init(null, trustAllCerts, new SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { return true; } }); } catch (Exception e) { } } /* * Authenticate user and retrieve access token. */ public String getAccessToken() { enableSSL(); String encoding = Base64.encode("simplivity:".getBytes()); RestTemplate restTemplate = new RestTemplate(); MultiValueMap&#x3C;String, String> body = new LinkedMultiValueMap&#x3C;String, String>(); body.add("username", HMS_USERNAME); body.add("password", HMS_PASSWORD); body.add("grant_type", "password"); HttpHeaders headers = new HttpHeaders(); headers.set("Accept", "application/json"); headers.set("Authorization", "Basic " + encoding); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>(body, headers); ResponseEntity&#x3C;String> res = restTemplate.exchange( BASE_URL+"oauth/token", HttpMethod.POST, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); access_token = (String) jsonObj.get("access_token"); System.out.println("Authenticated user and retrieved access token: "+ access_token); return access_token; } /* * Issue a GET request: GET /policies. */ public Object getPolicies() { RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON)); headers.set("Authorization", "Bearer " + access_token); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>("parameters", headers); ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies", HttpMethod.GET, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object policies= jsonObj.get("policies"); System.out.println(policies.toString()); return policies; } /* * Issue a GET request with sorting and filtering: * GET the first 100 policies * sorted in ascending order by name * and show only the name and rules fields. */ public Object getFirst100Policies() { RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.set("Accept", "application/json"); headers.set("Authorization", "Bearer " + access_token); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>("parameters", headers); ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies?fields=name,rules&#x26;limit=100&#x26;offset=0&#x26;sort=name&#x26;order=ascending", HttpMethod.GET, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object policies= jsonObj.get("policies"); System.out.println(policies.toString()); System.out.println("Limit: "+ jsonObj.get("limit")); System.out.println("Count: "+ jsonObj.get("count")); return policies; } /* * Issue a POST request: Create a new policy. */ public void createNewPolicy() { RestTemplate restTemplate = new RestTemplate(); // Set a custom media type. MediaType myMediaType = new MediaType("application", "vnd.simplivity.v1+json"); // Set the headers. HttpHeaders headers = new HttpHeaders(); headers.setAccept(Arrays.asList(myMediaType)); headers.setContentType(myMediaType); headers.set("Authorization", "Bearer " + access_token); // Form the POST body. String policyMo = "{\"name\": \"randomPolicyName4\"}"; HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;String>(policyMo, headers); // Issue the POST operation and expect a task object in return. ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies", HttpMethod.POST, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object task = jsonObj.get("task"); JSONObject taskJson = new JSONObject(task.toString()); String taskId = taskJson.getString("id"); String state = taskJson.getString("state"); // Monitor the status of the policy creation operation by using a loop to query // the task while this task is IN_PROGRESS. // The state field in the JSON response body indicates the status. while(state.equals("IN_PROGRESS")) { // Wait one second. try { Thread.sleep(1000); } catch (InterruptedException e) { e.printStackTrace(); } res = restTemplate.getForEntity(BASE_URL+"tasks/"+taskId, String.class); jsonObj = new JSONObject(res.getBody()); task = jsonObj.get("task"); taskJson = new JSONObject(task.toString()); state = taskJson.getString("state"); } System.out.println("Task object: " +task.toString()); } public static void main(String[] args) { RestClient restClient = new RestClient("10.150.1.71"); // Authenticate user and retrieve access token. restClient.getAccessToken(); // GET policies. restClient.getPolicies(); // GET first 100 policies sorted in ascending order by name // and showing only the name and rules fields. restClient.getFirst100Policies(); // POST /policies: Create a new policy. restClient.createNewPolicy(); } } </code></pre>https://developer.hpe.com/hpe-simplivity/metrics/https://developer.hpe.com/hpe-simplivity/metrics/<h1>Getting performance and capacity metrics</h1> <p>You use the REST API to view IOPs, latency, and throughput data over time. This information is available for individual <code>hosts</code>, <code>omnistack_clusters</code>, and <code>virtual_machines</code>. The metrics that the REST API provides include data about both reads and writes, as well as the date of the metrics sample.</p> <p>The REST API collects metrics at five-second intervals.</p> <p>When requesting metrics data, you can specify the following parameters:</p> <ul> <li> <p><code>resolution</code>: Specify one of the following values:</p> <ul> <li>SECOND</li> <li>MINUTE</li> <li>HOUR</li> <li>DAY</li> </ul> </li> <li> <p><code>time_offset</code>: Specify the <code>time_offset</code> from the current date and time or a <code>time_offset</code> as a specific date/time.</p> </li> <li> <p><code>range</code>: Specify the desired range as a number of seconds from the <code>time_offset</code>.</p> </li> </ul> <p>The following example shows how to find the last ten seconds of metrics data for a specific <code>omnistack_cluster</code>:</p> <pre><code>GET /api/omnistack_clusters/f25bc244-e70f-4ffb-833d-e797bc2bc231/metrics?range=10&#x26;resolution=SECOND </code></pre> <p>This request returns an array of data points similar to the following example:</p> <pre><code>{ "metrics": [ { "name": "iops", "data_points": [ { "reads": 0, "writes": 6, "date": "2016-09-08T16:11:20Z" }, { "reads": 0, "writes": 0, "date": "2016-09-08T16:11:25Z" } ] }, { "name": "throughput", "data_points": [ { "reads": 0, "writes": 537, "date": "2016-09-08T16:11:20Z" }, { "reads": 0, "writes": 0, "date": "2016-09-08T16:11:25Z" } ] }, { "name": "latency", "data_points": [ { "reads": 0, "writes": 1259, "date": "2016-09-08T16:11:20Z" }, { "reads": 0, "writes": 0, "date": "2016-09-08T16:11:25Z" } ] } ] } </code></pre> <h1>Performance chart</h1> <p>The REST API provides a simple one page performance chart that allows you to request host metrics and display them using <code>Chart.js.</code> For example:</p> <p><img src="https://developer.hpe.com/uploads/media/2018/7/svt-rest-api-perf-chart-1532710805262.png" alt="Example of the performance chart"></p> <p>The code to generate the sample page looks like this:</p> <pre><code>&#x3C;!DOCTYPE html> &#x3C;!-- Copyright © 2017 SimpliVity Corporation - All Rights Reserved --> &#x3C;html> &#x3C;head> &#x3C;script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.17.1/moment.min.js">&#x3C;/script> &#x3C;script src="lib/jquery-1.11.3.min.js">&#x3C;/script> &#x3C;script src="lib/Chart.js">&#x3C;/script> &#x3C;/head> &#x3C;body> &#x3C;canvas id="myChart" width="400" height="90%">&#x3C;/canvas> &#x3C;div id="query params"> &#x3C;form> Host: &#x3C;select name="host" id="host" onchange="getHosts()"> &#x3C;option>Fetching Hosts&#x3C;/option> &#x3C;/select> Resolution: &#x3C;select name="resolution" id="resolution"> &#x3C;option>Second&#x3C;/option> &#x3C;option>Minute&#x3C;/option> &#x3C;option>Hour&#x3C;/option> &#x3C;option>Day&#x3C;/option> &#x3C;/select> &#x3C;input type=button value="Show Metrics" onclick="fetchHostMetrics();"> &#x3C;/form> &#x3C;/div> &#x3C;script> fetchHosts(); function fetchHosts() { $.ajax({ url: 'hosts?fields=name,id', type: 'GET', dataType : "json", statusCode: { 401: function() {alert("Invalid Credentials - Login Again"); }} }) .done(function(json) { // reset the combo box with the list of hosts var hosts = json.hosts; var sel = document.getElementById('host'); sel.innerHTML = ""; hosts.forEach(function(host) { var opt = document.createElement('option'); opt.innerHTML = host.name; opt.value = host.id; sel.appendChild(opt); }) }) .fail(function( xhr, status, errorThrown ) { alert( "Sorry, there was a problem!" ); console.log( "Error: " + errorThrown ); console.log( "Status: " + status ); console.dir( xhr ); }) } function fetchHostMetrics() { // find the selected host var sel = document.getElementById('host'); var hostid = sel.value; // find the selected resolution var sel = document.getElementById('resolution'); var resolution = sel.value; $.ajax({ url: 'hosts/'+hostid+'/metrics?resolution='+resolution, type: 'GET', dataType : "json", statusCode: { 401: function() {alert("Invalid Credentiald - Login Again"); }} }) .done(function(json) {createLineChart(json)}) .fail(function( xhr, status, errorThrown ) { alert( "Sorry, there was a problem!" ); console.log( "Error: " + errorThrown ); console.log( "Status: " + status ); console.dir( xhr ); }) } function processJSONintoDatasets(json) { // create 6 datasets from the JSON var iops_reads = []; var iops_writes = []; var latency_reads = []; var latency_writes = []; var throughput_reads = []; var throughput_writes = []; // process all the metrics json.metrics.forEach(function(metric) { if (metric.name == "iops") { metric.data_points.forEach(function(point) { iops_reads.push({x:point.date, y: point.reads}); iops_writes.push({x:point.date, y: point.writes}); }) } else if (metric.name == "latency") { metric.data_points.forEach(function(point) { latency_reads.push({x: point.date, y: point.reads}); latency_writes.push({x: point.date, y: point.writes}); }) } else if (metric.name == "throughput") { metric.data_points.forEach(function(point) { throughput_reads.push({x:point.date, y: point.reads}); throughput_writes.push({x:point.date, y: point.writes}); }) } }) // create the data object var data = { datasets: [ { label: 'IOPS read', data: iops_reads, yAxisID: "y-axis-left", borderColor: 'rgba(31,120,180,1)',backgroundColor: 'rgba(31,120,180, 0.1)'}, { label: 'IOPS written', data: iops_writes, yAxisID: "y-axis-left", borderColor: "#33a02c"}, { label: 'latency read', data: latency_reads, yAxisID: "y-axis-left", borderColor: "#e31a1c" }, { label: 'latency writes', data: latency_writes, yAxisID: "y-axis-left", borderColor:"#ff7f00" }, { label: 'throughput read', data:throughput_reads, yAxisID: "y-axis-right", borderColor:"#6a3d9a" }, { label: 'throughput writes', data: throughput_writes, yAxisID: "y-axis-right", borderColor: "#b15928" } ] } // now set the common options data.datasets.forEach(function(dataset) { dataset.fill = true; dataset.lineTension = 0; }) return data; } var myLineChart; function createLineChart(json) { var ctx = document.getElementById("myChart"); var data =processJSONintoDatasets(json); // remove any existing chart before creating a new one if (myLineChart) myLineChart.destroy(); myLineChart = new Chart(ctx, { type: 'line', data: data, options: { responsive: true, title:{ display:true, text:"Metrics" }, scales: { xAxes: [{ type: "time", display: true, scaleLabel: { display: true, labelString: 'Time=(GMT)' } }], yAxes: [{ display: true, scaleLabel: { display: true, labelString:'IOPS/Latency' }, position: "left", "id": "y-axis-left" }, { display: true, scaleLabel: { display: true, labelString: 'Throughput' }, position: "right", "id": "y-axis-right" }] } } }); } &#x3C;/script> &#x3C;/body> &#x3C;/html> </code></pre> <h1>Capacity metrics</h1> <p>Each HPE OmniStack <code>host</code> and <code>omnistack_cluster</code> object returns a set of capacity metrics when you perform a GET operation. You can query for a specific period of time. The REST API provides the following capacity metrics for <code>host</code>and <code>omnistack_cluster</code> objects:</p> <ul> <li>allocated_capacity`</li> <li>capacity_savings`</li> <li>compression_ratio`</li> <li>deduplication_ratio`</li> <li>efficiency_ratio`</li> <li>free_space`</li> <li>local_backup_capacity`</li> <li>remote_backup_capacity`</li> <li>stored_compressed_data`</li> <li>stored_uncompressed_data`</li> <li>stored_virtual_machine_data`</li> <li>used_capacity`</li> <li>used_logical_capacity`</li> </ul> <p>The following example returns <code>free_space</code> (in bytes) for the last ten seconds for a specific <code>host</code>:</p> <pre><code>GET /api/hosts/422ac72f-8437-6e09-eb44-6ffc8bde0aad/capacity?fields=free_space&#x26;range=5&#x26;resolution=SECOND" </code></pre> <p>The JSON response:</p> <pre><code>{ "metrics": [ { "name": "free_space", "data_points": [ { "value": 5170371316326, "date": "2016-09-08T16:29:50Z" }, { "value": 5170369843492, "date": "2016-09-08T16:29:55Z" } ] } ] } </code></pre>https://developer.hpe.com/hpe-simplivity/optimize-get-requests/https://developer.hpe.com/hpe-simplivity/optimize-get-requests/<h1>Optimizing GET requests</h1> <p>You can increase the efficiency of the REST API by minimizing the amount of processing that occurs on the server and the amount of data that is transmitted over the network. This page describes a number of ways that you can sort and filter the output of GET requests to optimize them. You can increase the speed of these transactions by eliminating the transfer of unnecessary content.</p> <h2>Request specific fields</h2> <p>If you only need a small number of fields, you can request only these fields. The following example requests only the <code>id</code> and <code>name</code> fields:</p> <pre><code>GET /api/backups?fields=id,name </code></pre> <h2>Limit the number of objects by relationship</h2> <p>If you only want to retrieve a limited number of objects, you can use filtering parameters to return only objects associated with a particular resource. The following example queries for all backups of a specific virtual machine:</p> <pre><code>GET /api/backups?fields=id,name&#x26;virtual_machine_id=729861fd-a006-4849-bf08-b232551989c0 </code></pre> <h2>Sort returned data</h2> <p>To increase the efficiency of the returned data, you can sort these results by most fields. The following example sorts results by the <code>name</code> field:</p> <pre><code>GET /api/backups?fields=id,name&#x26;virtual_machine_id=729861fd-a006-4849-bf08-b232551989c0&#x26;sort=name </code></pre> <p>For most fields, the sort is a case-sensitive lexicographic sort. Numbers come before capital letters, which come before lowercase letters. The REST API service sorts backup names case-insensitively.</p> <h2>Request objects in sorted/filtered chunks</h2> <p>You can use the <code>offset</code> and <code>limit</code> fields to request objects in sorted/filtered chunks. You can set the limit to any integer value up to 5000. The following example sorts results by the <code>name</code> field and requests chunks of 50 results at a time:</p> <pre><code>GET /api/backups?fields=id,name&#x26;virtual_machine_id=729861fd-a006-4849-bf08- b232551989c0&#x26;sort=name&#x26;offset=50&#x26;limit=50 </code></pre> <p>The REST API always performs sorting and filtering on enumerated values using the full uppercase enumerations. The REST API ignores the value of the case parameter when you sort or filter by fields with enumerated values.</p>https://developer.hpe.com/hpe-simplivity/powershell/https://developer.hpe.com/hpe-simplivity/powershell/<h1>Using the API with Windows PowerShell</h1> <p>This sample Windows PowerShell code performs authentication, issues an example GET request, performs a POST operation (in this case, renaming a backup), and monitors the status of the operation using a task instance.</p> <p>When using Windows PowerShell to develop clients:</p> <ul> <li> <p>Allow the use of self-signed SSL certifications, as follows: <code>[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $True }</code></p> </li> <li> <p>Alternatively, you can use <code>https://github.com/Jaykul/Tunable-SSL-Validator</code>with the <code>-Insecure</code> option.</p> </li> <li> <p>Use PowerShell version 3 or above to support <code>Invoke-RestMethod</code>.</p> </li> <li> <p>HTTP Basic Authentication requires the manual creation of the base 64 encoding, as follows:</p> </li> </ul> <pre><code>$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "simplivity",""))) Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} $body = @{grant_type='password';username='username';password='password'} -Method POST </code></pre> <p>Consider the following code:</p> <pre><code>[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $True } $token = Invoke-RestMethod -Uri https://10.150.1.93/api/oauth/token -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Body $body -Method Post </code></pre> <p>Which returns a response similar to the following:</p> <pre><code>access_token : 45718cb1-528b-430f-8a6d-89e25ae9c7ea token_type : bearer expires_in : 83649 scope : read write updated_at : 1459805657628 </code></pre> <p>This example shows how to make a GET call for all backups:</p> <pre><code>$header = @{Authorization='Bearer '+$token.access_token} $backups = Invoke-RestMethod -Insecure -Header $header -Uri https://10.150.1.93/api/backups </code></pre> <p>This example finds the type of the 22nd backup: <code>$backups.backups[22].type</code></p> <p>This example renames this backup:</p> <pre><code>$header['Content-Type']='application/vnd.simplivity.v1+json' $uri = 'https://10.150.1.93/api/backups/' $uri += $backups.backups[22].id $uri += '/rename' $body = @{backup_name='new_name'} $body = $body | ConvertTo-Json TunableSSLValidator\Invoke-RestMethod -Insecure -Header $header -Uri $uri -Method Post -Body $body </code></pre> <h2>Sample code</h2> <pre><code># Set the base URL for REST API requests. $BASE_URL = 'https://[host]/api/' # Set the username and password. $hms_username = 'HMS_USER' $hms_password = 'HMS_PASS' # Allow the use of self signed certificates. [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $True } # Create a base64 encoding for HTTP Authentication. $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "simplivity",""))) # Create a JSON body with username, password, and grant_type. $body = @{grant_type='password';username=$hms_username;password=$hms_password} # Authenticate user and generate access token. $url = $BASE_URL+'oauth/token' $header = @{Authorization=("Basic {0}" -f $base64AuthInfo)} $response= Invoke-RestMethod -Uri $url -Headers $header -Body $body -Method Post $access_token = $response.access_token; # Add the access_token to the header. $header =@{Authorization='Bearer '+$access_token} # Issue a GET request: GET /backups. $url = $BASE_URL+'backups' $backups = Invoke-RestMethod -Header $header -Uri $url # Issue a POST request: Rename the second backup from the GET results. # Find the ID of the second backup from the GET results. $backupid = $backups.backups[2].id # Create a JSON body for the rename action. $body = @{backup_name='new_name'} $body = $body | ConvertTo-Json # Form the URI. $url = $BASE_URL+'backups/' $url += $backupid $url += '/rename' # Issue the POST operation and expect a task object in return. $response = Invoke-RestMethod -Header $header -Uri $url -Method Post -Body $body -ContentType 'application/vnd.simplivity.v1+json' # Monitor the status of the rename operation by using a loop to query the task while this task is IN_PROGRESS. # The state field in the JSON response body indicates the status. $taskid = $response.task.id $state = $response.task.state $url = $BASE_URL+'tasks/'+$taskid while ($state -eq 'IN_PROGRESS') { # Wait one second and try again. Start-Sleep -s 1 $response = Invoke-RestMethod -Header $header -Uri $url $state = $response.task.state } # Print out the task result. $response </code></pre>https://developer.hpe.com/hpe-simplivity/ps-getoathtoken/https://developer.hpe.com/hpe-simplivity/ps-getoathtoken/<h1>Authenticating</h1> <p>The certificate management cmdlets use OAuth to authenticate.</p> <p>Use the <code>Get-HPESvtAuthToken</code> cmdlet to obtain an OAuth access_token. The access_token is stored in an encrypted file and is automatically used by any of the Powershell cmdlets during the same session. The access_token has a ten minute idle lifetime.</p> <p>To add or delete a certificate, you must authenticate as a user in the hypervisor management system (HMS) administrator role or as the <code>svtcli</code> user. To get certificates, you need to authenticate with the HMS, but the HMS administrator role is not required.</p> <p>You need to obtain a token from each Virtual Controller where you want to manage certificate trust stores by using the Powershell cmdlets.</p> <h2>Authenticating as the HMS administrator</h2> <p>At the command line, you can pass the username then provide the password when prompted by <code>Get-HPESvtAuthToken</code>. For example:</p> <pre><code>PS C:\> Get-HPESvtAuthToken 192.0.2.2 admin-username </code></pre> <p>If you are scripting the authentication, you can combine the credentials into an object, then pass the object. The following example uses the <code>$creds</code> object to pass the credentials.</p> <pre><code># securely store the password PS C:\HPESvtCmdlets> $secpasswd = ConvertTo-SecureString "password" -AsPlainText -Force PS C:\HPESvtCmdlets> $creds = New-Object System.Management.Automation.PSCredential("admin-username", $secpasswd) PS C:\HPESvtCmdlets> Get-HPESvtAuthToken 192.0.2.2 $creds </code></pre> <p>It returns an access_token. For example:</p> <pre><code>access_token : bd96becc-71e7-4eca-8859-cbbdfb6d1cdd token_type : bearer expires_in : 86399 scope : read write updated_at : 1544475704159 </code></pre> <h2>Authenticating as the <code>svtcli</code> user</h2> <p>You might need to authenticate as the <code>svtcli</code> user when the hypervisor management system is not reachable by the Virtual Controller.</p> <p>You can use the same technique of combining the credentials into the <code>$creds</code> object and then passing <code>$creds</code> on the command line, but when you authenticate as the <code>svtcli</code> user, you must pass the <code>-emergency</code> parameter. For example:</p> <pre><code>$emergencypwd = ConvertTo-SecureString "password" -AsPlainText -Force $emergencycreds = New-Object System.Management.Automation.PSCredential("svtcli",$emergencypwd) $oauthcred = Get-HPESvtAuthToken -HostName $ovcip -credential $emergencycreds -emergency </code></pre> <p>It returns an access_token. For example:</p> <pre><code>access_token : bd96becc-71e7-4eca-8859-cbbdfb6d1cdd token_type : bearer expires_in : 86399 scope : read write updated_at : 1544475704159 </code></pre>https://developer.hpe.com/hpe-simplivity/python/https://developer.hpe.com/hpe-simplivity/python/<h1>Using the API with Python</h1> <p>This sample Python code performs authentication, issues three example GET requests, performs a POST operation (in this case, renaming a backup), and monitors the status of the operation using a task.</p> <pre><code>import requests # Set the base URL for REST API requests. url = 'https://[host]/api/' # Set the username and password. hms_username = 'HMS_USER' hms_password = 'HMS_PASS' # Authenticate user and generate access token. response = requests.post(url+'oauth/token', auth=('simplivity', ''), verify=False, data={ 'grant_type':'password', 'username':hms_username, 'password':hms_password}) access_token = response.json()['access_token'] # Add the access_token to the header. headers = {'Authorization': 'Bearer ' + access_token, 'Accept' : 'application/vnd.simplivity.v1+json'} # Issue a GET request: GET /hosts. response = requests.get(url+'hosts', verify=False, headers=headers) print(response.json()) # Issue a GET request: GET /datastores. response = requests.get(url+'datastores', verify=False, headers=headers) print(response.json()) # Issue a GET request with sorting and filtering: # GET first 100 backups of the MANUAL type # sorted in ascending order by name # and show only the name, id, and created_at fields. response = requests.get(url+'backups?type=MANUAL&#x26;fields=name%2C%20id%2C%20created_at&#x26;limit=100&#x26;offset=0&#x26;sort=name&#x26;order=ascending', verify=False, headers=headers) print(response.json()) # Issue a POST request: Rename the first backup from the GET results. # Define a container to hold the JSON response. json = response.json() # Find the ID of the first backup from the GET results. backupid = json['backups'][0]['id'] # Create a JSON body for the rename action. body = '{"backup_name":"newbackupname"}' # Specify the correct MIME type for the body. headers['Content-Type'] = 'application/vnd.simplivity.v1+json' # Issue the POST operation and expect a task object in return. response = requests.post(url+'backups/'+backupid+'/rename', data=body, verify=False, headers=headers) json = response.json() # Monitor the status of the rename operation by using a loop to query # the task while this task is IN_PROGRESS. # The state field in the JSON response body indicates the status. taskid = json['task']['id'] state = json['task']['state'] while (state == "IN_PROGRESS"): # Wait one second. time.sleep(1) response = requests.get(url+'tasks/'+taskid, verify=False, headers=headers) json = response.json() state = json['task']['state'] print(json) </code></pre>https://developer.hpe.com/hpe-simplivity/remove-certificate/https://developer.hpe.com/hpe-simplivity/remove-certificate/<h1>Removing certificates from the trust store</h1> <p>You might need to delete a certificate from the trust store on an HPE OmniStack host if the existing certificate has expired or is corrupted, and you want to replace it. If the certificate has expired or is corrupted, then you must authenticate to the HPE OmniStack Virtual Controller by using the emergency access account (<code>svtcli</code>).</p> <p>You can remove certificates from the trust store by using the <code>Remove-HPESvtCertificate</code> cmdlet and specifying the certificate's SHA1 thumbprint.</p> <p>The following sample code shows how delete a certificate from the trust store by thumbprint. It assumes that you have already authenticated.</p> <pre><code>Remove-HPESvtCertificate -Thumbprint 7E084B44FCDAB6EF74AC40E8D9BD508337D96CDE </code></pre>https://developer.hpe.com/hpe-simplivity/request-authentication-by-emergency-grant/https://developer.hpe.com/hpe-simplivity/request-authentication-by-emergency-grant/<h1>Requesting authentication by emergency grant</h1> <p>The emergency grant type functions in a way that is similar to the password grant type. However, this grant type uses the emergency local svtcli account for authentication. This functionality is similar to the <code>--emergency</code> option in CLI commands.</p> <p>This grant type is intended for use only when the Hypervisor Management System has become unavailable, and you want to access or restore backups of <code>virtual_machine</code> objects. Using this type of token, clients can only access the following limited set of API operations:</p> <ul> <li>GET /hosts</li> <li>GET /omnistack_clusters</li> <li>GET /backups</li> <li>GET /datastores</li> <li>GET /virtual_machines</li> <li>POST /backups/{bkpId}/restore</li> </ul> <p>With this grant type, the REST API does not support sorting and filtering GET requests or querying for optional properties.</p> <p>Use a <code>curl</code> command like the following to request a token for the HPE OmniStack CLI (svtcli) account from a Virtual Controller with <em>host</em> as the IP address, svtcli as the username, and <em>password</em> as the password:</p> <pre><code>curl -k https://simplivity@[host]/api/oauth/token -d grant_type=emergency -d username=svtcli -d password=[password] </code></pre> <p>Where:</p> <ul> <li><em>host</em> is the IP address of the Virtual Controller you want to authenticate to.</li> <li>svtcli is the username for the emergency local HPE OmniStack CLI account.</li> <li><em>password</em> is the password for the emergency local HPE OmniStack CLI (svtcli) account.</li> </ul> <p>You must supply the <code>-k</code> switch due to the use of self-signed certificates. You can import these certificates into your local certificate store.</p> <p>This call returns a JSON response similar to the following:</p> <pre><code>{ "access_token":"6f9166c2-bc7f-4cfc-a304-56aadd85e214", "token_type":"bearer", "expires_in":86399, "scope":"read write", "updated_at":1489421129188 } </code></pre> <p>Note the value for the <code>access_token</code> in this response. Tokens expire either after ten minutes of inactivity or after 24 hours even with continuous activity. If you pass incorrect credentials, then the client receives a response similar to the following:</p> <pre><code>{ "error":"invalid_grant", "error_description":"Invalid credentials" } </code></pre>https://developer.hpe.com/hpe-simplivity/restore-virtual-machines/https://developer.hpe.com/hpe-simplivity/restore-virtual-machines/<h1>Restoring virtual machines</h1> <p>This example uses <code>curl</code> to issue a POST request to restore a specific virtual machine from a backup. The example assumes that you have obtained an OAuth 2 token and that you have retrieved the identifier of the object that you want to perform the operation on.</p> <pre><code>curl -X POST --H "Content-Type: application/vnd.simplivity.v1+json" --header "Authorization: Bearer [access_token]" -d "{ \"virtual_machine_name\": \"new_vm_name\" }" "https://[host]/api/backups/[id]/restore?restore_original=false" </code></pre> <p>Where:</p> <ul> <li><code>-H</code> (or <code>--header</code>) enables <code>curl</code> to add request headers.</li> <li><code>"Content-Type: application/vnd.simplivity.v1+json"</code> indicates that the data in the body of the request uses version 1 of the HPE OmniStack JSON extension.</li> <li><code>access_token</code> is the complete access token. For example: <code>f93f2059-afef-4310-9147-447645992a5d</code>.</li> <li>The body of the POST request is a JSON object that includes the required <code>virtual_machine_name</code> field that specifies the desired name for the new virtual machine that you want to create from the backup.</li> <li><code>/api/backups/[id]/restore</code> is the base URI for the POST restore operation: For example: <code>/api/backups/0f123f92-2d75-4640-aab1-fa22b0a037cd/restore</code>.</li> <li><code>host</code> is the IP address of the Virtual Controller to authenticate.</li> <li>The query parameter <code>restore_original</code> indicates whether to restore the original virtual machine. Setting this parameter to <code>false</code> specifies that you want to create a new virtual machine from the backup, rather than restoring the backup to an existing virtual machine.</li> </ul> <p>The REST API service returns a <code>task</code> instance as the response to this POST request. For example:</p> <pre><code>{ "task": { "id": "422a3b5d-9125-467b-1909-9fb5f5c9c65e:422a3b5d-9125-467b-1909- 9fb5f5c9c65e:841eb5cf-19d2-4fbb-a7ec-20638373ccfc", "state": "IN_PROGRESS", "affected_objects": [], "error_code": 0, "start_time": "2016-03-15T15:43:24Z", "end_time": "1970-01-01T00:00:00Z" } } </code></pre> <p>The following <code>curl</code> command monitors the status of this operation by retrieving the task instance:</p> <pre><code>curl -H "Accept: application/vnd.simplivity.v1+json" -H "Authorization: Bearer [*access_token*]" -X GET -k -i https://[host]/api/tasks/[*task_id*] </code></pre> <p>Where:</p> <ul> <li> <p><code>-H</code> (or <code>--header</code>) enables curl to add request headers.</p> </li> <li> <p>The <code>"Accept: application/vnd.simplivity.v1+json"</code> header indicates that you want the response to use version 1 of the HPE OmniStack JSON extension.</p> </li> <li> <p><code>access_token</code> is the complete access token.</p> </li> <li> <p><code>-X GET</code> indicates an HTTP GET operation.</p> </li> <li> <p><code>-k</code> allows the use of self-signed SSL/TLS certificates.</p> </li> <li> <p><code>host</code> is the IP address of the Virtual Controller to authenticate against.</p> </li> <li> <p><code>task_id</code> is the complete ID of the task to monitor. For example: <code>422a3b5d-9125-467b-1909-9fb5f5c9c65e:422a3b5d-9125-467b-1909- 9fb5f5c9c65e:841eb5cf-19d2-4fbb-a7ec-20638373ccfc</code>.</p> </li> </ul> <p>You can issue this command multiple times, as necessary, to monitor the state of the operation. Possible states include:</p> <ul> <li><code>IN_PROGRESS</code></li> <li><code>COMPLETED</code></li> <li><code>FAILED</code></li> </ul> <p>Once the operation completes, the response to this GET request includes any affected objects. In this example, the <code>affected_objects</code> body includes the object type and ID of the new virtual machine that operation created.</p>https://developer.hpe.com/hpe-simplivity/sample-code/https://developer.hpe.com/hpe-simplivity/sample-code/<h1>Getting started</h1> <p>You can develop REST API clients in a variety of languages, including Python, Windows PowerShell, Java, and others. Before starting to create a REST API client, it is important to understand the following basic client workflow:</p> <p>1 - Identify the user credentials that you plan to use to authenticate to an HPE OmniStack host.</p> <p>2 - Authenticate to the HPE OmniStack host to generate an access token.</p> <p>3 - Store the access token for use in all REST API request headers.</p> <p>4 - Issue a GET request to get a set of instances of an object type, based on the filtering and sorting criteria that you specify.</p> <p>5 - Issue a POST request or other operation to perform an action that affects this instance.</p> <p>6 - Poll the returned task instance to monitor the status of the operation.</p>https://developer.hpe.com/hpe-simplivity/revoke-oath-2-token/https://developer.hpe.com/hpe-simplivity/revoke-oath-2-token/<h1>Revoking an OAuth 2 token</h1> <p>This example uses <code>curl</code> to illustrate how to revoke an OAuth 2 token. You might revoke a token when you are done using it and you want to free up any outstanding resources, such as vCenter Server sessions.</p> <p>The following <code>curl</code> command revokes an OAuth 2 token:</p> <pre><code>curl -k https://[host]/api/oauth/revoke -H "Authorization: Bearer 0a08c809-17ff-479f-b0a8-aedd4d8305a0" </code></pre> <p>Where:</p> <ul> <li><em>host</em> is the IP address of the Virtual Controller.</li> </ul> <p>You must supply the <code>-k</code>switch due to the use of self-signed certificates. You can import these certificates into your local certificate store.</p> <p>The call returns HTTP status 200 without any body.</p>https://developer.hpe.com/hpe-simplivity/tls-certificate-validation/https://developer.hpe.com/hpe-simplivity/tls-certificate-validation/<h1>Disabling/enabling TLS certificate validation on HPE OmniStack for vSphere</h1> <p>In some cases, it may be necessary to temporarily disable TLS certification validation while you are fixing various certificate issues. However, use these steps only when it is necessary to re-establish communication to the Virtual Controller. Re-enable validation immediately after you diagnose and fix the certificate issues.</p> <p>WARNING</p> <p><em>Removing all certificates from the trust store causes HPE SimpliVity to stop performing TLS certificate validation which is a potential security risk.</em></p> <h2>Disabling TLS certificate validation</h2> <p>If you need to turn off certificate validation across reboots, you can disable TLS validation by removing all of the certificates from the trust store by using the PowerShell certificate management cmdlets.</p> <p>When TLS is not working properly in your environment, you must connect to the Virtual Controller using the <code>svtcli</code> account because HMS entities cannot be validated when the connection is down.</p> <p>This procedure assumes that you have downloaded and installed the certificate management cmdlets.</p> <p>To begin, open a Powershell window and run the following script to remove all of the certificates from the trust store. Update the script to use the IP addresses and credentials for your system.</p> <pre><code>$ErrorActionPreference = "Stop" $hmsip = "192.0.2.2" $ovcip = "192.0.2.5" # # When the certficates are bad, use the "emergency" account # $emergencypwd = ConvertTo-SecureString "password" -AsPlainText -Force $emergencycreds = New-Object System.Management.Automation.PSCredential("svtcli",$emergencypwd) $oauthcred = Get-HPESvtAuthToken -HostName $ovcip -credential $emergencycreds -emergency # # delete all of the certificates - this turns off TLS validation # HPESvtCertificate | ForEach-Object -Process { # remove from OVC trust store Remove-HPESvtCertificate -Thumbprint $_.Thumbprint } # # The HMS user should be able to logon now # $oauthcred = Get-HPESvtAuthToken -HostName $ovcip -credential $creds </code></pre> <h2>Enabling TLS certificate validation</h2> <p>Leaving validation off is not an acceptable long term solution. Fix the issues with the HMS certificate and then re-enable TLS validation.</p> <p>This procedure assumes that you have downloaded and installed the certificate management cmdlets.</p> <p>To begin, open a Powershell window and run the following script to add the HMS certificate to the trust store to restore TLS validation. Update the script to use the IP addresses and credentials for your system.</p> <p>The script gets the vCenter certificates then adds them to the Virtual Controller's trust store. When certificate validation is turned off or not working, you must connect using the <code>svtcli</code> account.</p> <pre><code>$ErrorActionPreference = "Stop" $hmsip = "192.0.2.2" $ovcip = "192.0.2.5" $secpasswd = ConvertTo-SecureString "password" -AsPlainText -Force $creds = New-Object System.Management.Automation.PSCredential("administrator",$secpasswd) # # When the certificates are bad, use the "emergency" account # $emergencypwd = ConvertTo-SecureString "password" -AsPlainText -Force $emergencycreds = New-Object System.Management.Automation.PSCredential("svtcli",$emergencypwd) $oauthcred = Get-HPESvtAuthToken -HostName $ovcip -credential $emergencycreds -emergency # # grab vmware certs from HMS # $vmwarecerts = Get-HPESvtRootCertificate -HostName $hmsip # # add them into trust store # $vmwarecerts | ForEach-Object -Process { Add-HPESvtCertificate -Certificate $_ } # # The HMS user should still be able to logon # $oauthcred = Get-HPESvtAuthToken -HostName $ovcip -credential $creds Write-Host $oauthcred </code></pre>https://developer.hpe.com/hpe-simplivity/versioning/https://developer.hpe.com/hpe-simplivity/versioning/<h1>Getting the API version</h1> <p>The REST API is backward compatible. As new attributes and operations are added in each release, older clients continue to work without any changes. Clients should strive to be backward compatible and should query the version of the REST API they are targeting to make sure that this version supports the needed functionality. The first version of the REST API was v1, followed by v1.1, v1.2, and so on.</p> <p>You can issue a GET request to determine the version running on a Virtual Controller. To determine the version, issue a <code>GET /api/version</code> request. This request does not need to be authenticated. For example:</p> <pre><code>curl -k https://&#x3C;host>/api/version </code></pre> <p>The response includes the REST API version and the SVTFS version. For example:</p> <pre><code>{ REST_API_Version: "1.1", SVTFS_Version: "3.5.9904.284" } </code></pre> <h3>Accept header</h3> <p>Use the <code>Accept</code> header to request a particular version of the REST API response objects. For example:</p> <pre><code>Accept: application/vnd.simplivity.v1.1+json </code></pre> <p>You always receive the current version representation, even if you request an older version.</p> <h3>Content type</h3> <p>Each new version of the REST API introduces new operations. For detailed information about the operations that a particular REST API version supports, see the interactive REST API reference on the Virtual Controller you plan to interact with.</p> <p>You must specify the correct minimum version using the <code>Content-Type</code> header. For example:</p> <pre><code>Content-Type: application/vnd.simplivity.v1.1+json </code></pre> <p>The following example shows how to use <code>curl</code> to specify the desired minimum version in the <code>Content-Type</code> header within a POST request to lock a backup:</p> <pre><code>curl -k -X POST --header "Content-Type: application/vnd.simplivity.v1.1+json" "https://&#x3C;host>/api/backups/a5f3e4ae-ae54-48cd-86fc-d168a542ef3f/lock" </code></pre> <p>If you specify a <code>Content-Type</code> that is too old while issuing a POST or PUT request, the REST API returns an exception. For example:</p> <pre><code>{ 'status': '415', 'path': '/api/backups/f61a3e0f-829b-4739-86da-c82de96c2c85/lock', 'exception': 'org.springframework.web.HttpMediaTypeNotSupportedException', 'message': "Content type 'application/vnd.simplivity.v1+json' not supported", 'timestamp': '2016-05-16T18:25:01Z' } </code></pre>https://developer.hpe.com/hpe-simplivity/whats-new/https://developer.hpe.com/hpe-simplivity/whats-new/<h1>What's new in each version</h1> <p>The following sections list the features added in each version of the HPE OmniStack REST API.</p> <h2>Version 1.16 (released with HPE OmniStack 4.1.0)</h2> <p>Version 1.16 of the REST API added the following new features:</p> <ul> <li>The <code>connected_clusters</code> field for <code>GET omnistack_clusters</code> has been deprecated. Use the REST call <code>GET /api/omnistack_clusters/{clusterid}/connected_clusters</code> instead.</li> <li>You can now set the IWO status for an HPE OmniStack cluster.</li> <li>You can now create and identify single_replica datastores.</li> </ul> <h2>Version 1.15 (released with HPE OmniStack 4.0.1)</h2> <p>Version 1.15 of the REST API added the following new features:</p> <ul> <li>You can now update credentials for external stores and unregister external stores from a cluster.</li> <li>The <code>availability_zone_effective</code> and <code>availability_zone_planned</code> properties were added to the <code>omnistack_hosts</code> object.</li> <li>The REST call <code>GET /api/omnistack_clusters/throughput</code> has been deprecated. Use the replacement REST call <code>GET /api/omnistack_clusters/{clusterId}/throughput</code> instead.</li> </ul> <h2>Version 1.14 (released with HPE OmniStack 4.0.0)</h2> <p>Version 1.14 of the REST API added the following new features:</p> <ul> <li>You can now backup to an external store and restore a backup from an external store by creating a new virtual machine.</li> <li>The <code>arbiter_required</code> and <code>arbiter_configured</code> properties were added to the <code>omnistack_clusters</code> object.</li> </ul> <h2>Version 1.13 (released with HPE OmniStack 3.7.10)</h2> <p>Version 1.13 of the REST API added no new features.</p> <h2>Version 1.12 (released with HPE OmniStack 3.7.9)</h2> <p>Version 1.12 of the REST API added the following features:</p> <ul> <li>The certificates URI was renamed to security/certificates</li> <li>cluster_feature_level was added to omnistack_clusters</li> </ul> <h2>Version 1.11 (released with HPE OmniStack 3.7.8)</h2> <p>Version 1.11 of the REST API added the following features:</p> <ul> <li>Added infosight_configuration properties to the <code>hosts</code> and <code>omnistack_clusters</code> objects</li> </ul> <h2>Version 1.10 (released with HPE OmniStack 3.7.7)</h2> <p>Version 1.10 of the REST API added the following features:</p> <ul> <li>Certificate management</li> <li>Improved performance for GET backups</li> </ul> <h2>Version 1.9 (released with HPE OmniStack 3.7.6)</h2> <p>Version 1.9 of the REST API added the following features:</p> <ul> <li>Perform file level restore from backups</li> <li>Shutdown a Virtual Controller, cancel the shutdown, and check the status of the shutdown operation</li> <li>Remove an HPE OmniStack host from a federation</li> <li>Backup rule impact reporting</li> </ul> <h2>Version 1.8 (released with HPE OmniStack 3.7.5)</h2> <p>Version 1.8 of the REST API added the following features:</p> <ul> <li> <p>Share a datastore with a standard host (host without HPE OmniStack software)</p> </li> <li> <p>Stop sharing a datastore with a standard host</p> </li> <li> <p>Provide details on standard hosts that can share a datastore (unique ID, IP address, host name, sharing status, virtual machine count)</p> </li> <li> <p>Save virtual machine backup parameters (includes backup type application consistent or crash consistent and virtual machine credentials to access VSS if necessary to create an application-consistent backup with VSS instead of a VMware snapshot)</p> </li> <li> <p>Show the name of the hypervisor management system for clusters, hosts, datastores, and virtual machines</p> </li> </ul> <h2>Version 1.7 (released with HPE OmniStack 3.7.3)</h2> <p>Version 1.7 of the REST API added the following new operations for <code>backups</code> objects:</p> <ul> <li>Calculates the unique size of a specified backup</li> <li>Cancels a specific running backup</li> </ul> <h2>Version 1.6 (released with HPE OmniStack 3.7.2)</h2> <p>Version 1.6 of the REST API added the following features:</p> <ul> <li> <p>Added the following fields for <code>host</code> objects:</p> <ul> <li><code>federation_mask</code></li> <li><code>federation_mtu</code></li> <li><code>management_mask</code></li> <li><code>management_mtu</code></li> <li><code>storage_mask</code></li> <li><code>storage_mtu</code></li> </ul> </li> <li> <p>Added the following field for <code>datastore</code> and <code>omnistack_clusters</code> objects:</p> <ul> <li><code>hypervisor_management_system</code></li> </ul> </li> </ul> <h2>Version 1.5 (released with HPE OmniStack 3.7.0)</h2> <p>Version 1.5 of the REST API added the following features:</p> <ul> <li>Set backup policy for multiple <code>virtual_machines</code> with one operation</li> <li>Renamed <code>hypervisor_management_system</code> to <code>hypervisor_type</code></li> <li>Added support for an emergency <code>grant_type</code> to enable you to restore virtual machines in situations in which the Hypervisor Management System has become unavailable</li> <li>Added support for revoking OAuth tokens</li> <li>Added support for searching for objects by one or more IDs</li> </ul> <h2>Version 1.4 (released with HPE OmniStack 3.6.2)</h2> <p>Version 1.4 of the REST API added the following features:</p> <ul> <li> <p>Suspend or resume policy-based backups for a <code>host</code>, <code>omnistack_cluster</code>, or entire federation</p> </li> <li> <p>Delete multiple backups with one POST request</p> </li> <li> <p>Added <code>hypervisor_management_system</code> for <code>backup</code>, <code>datastore</code>, <code>omnistack_cluster</code>, and <code>virtual_machine</code> objects</p> </li> <li> <p>Added the following fields for <code>backup</code>, <code>datastore</code>, and <code>virtual_machine</code> objects:</p> <ul> <li><code>compute_cluster_parent_hypervisor_object_id</code></li> <li><code>compute_cluster_parent_name</code> +Added the following fields for <code>backup</code> objects:</li> <li><code>unique_size_bytes</code></li> <li><code>unique_size_timestamp</code> +<code>virtual_machine_type</code> +Added the following optional fields for <code>virtual_machine</code> objects:</li> <li><code>hypervisor_allocated_capacity</code></li> <li><code>hypervisor_cpu_count</code></li> <li><code>hypervisor_free_space</code></li> <li><code>hypervisor_is_template</code></li> <li><code>hypervisor_total_memory</code></li> <li><code>hypervisor_virtual_disk_count</code></li> </ul> </li> <li> <p>Added the following fields for <code>host</code> objects:</p> <ul> <li><code>can_rollback</code></li> <li><code>compute_cluster_parent_hypervisor_object_id</code></li> <li><code>compute_cluster_parent_name</code></li> <li><code>current_feature_level</code></li> <li><code>policy_enabled</code></li> <li><code>potential_feature_level</code></li> <li><code>upgrade_state</code></li> </ul> </li> <li> <p>Added <code>version</code> for <code>omnistack_cluster</code> objects</p> </li> <li> <p>Faster performance and additional query options for <code>omnistack_cluster</code> objects</p> </li> <li> <p>Filter <code>backup</code>, <code>datastore</code>, and <code>omnistack_cluster</code> objects using multiple values in a comma-separated (OR) list</p> </li> </ul> <h2>Version 1.3 (released with HPE OmniStack 3.6.1)</h2> <p>Version 1.3 of the added the following features:</p> <ul> <li>Association of <code>omnistack_cluster</code> objects with Hypervisor Management System (HMS) clusters</li> <li>The new <code>hypervisor_object_parent_name</code> and <code>hypervisor_object_parent_id</code> properties identify the parent of an <code>omnistack_cluster</code></li> <li>Set the retention time for one or more backups</li> <li>Cluster throughput</li> <li>Cluster connectivity</li> <li>Set the time zone for a cluster</li> <li>Request a list of valid time zones for use when setting the time zone for a cluster</li> <li>Request the amount of life remaining for solid state drives (SSDs)</li> <li>Get the upgrade status for clusters</li> <li>Increased security resulting from removing the ability to refresh OAuth 2 tokens</li> <li>Filter <code>virtual_machine</code> objects using multiple values in a comma-separated (OR) list</li> </ul> <h2>Version 1.2 (released with HPE OmniStack 3.5.3)</h2> <p>Version 1.2 of the added the following features:</p> <ul> <li>Ability to set a backup policy for a datastore</li> <li>Host capacity metrics</li> <li><code>virtual_machine</code> hypervisor power state</li> <li><code>omnistack_cluster</code> associated hypervisor object ID and time zone</li> <li>Host associated compute cluster information</li> </ul> <h2>Version 1.1 (released with HPE OmniStack 3.5.2)</h2> <p>Version 1.1 of the added the following features:</p> <ul> <li>Ability to query the REST version</li> <li>vCenter Server linked mode support</li> <li>Rename, copy, and lock backups</li> <li>Performance metrics</li> <li>Host hardware reporting</li> </ul> <h2>Version 1 (released with HPE OmniStack 3.5.1)</h2> <p>The initial version of the REST API focused on orchestration tools and included the following features:</p> <ul> <li>Management: Perform common operations, such as clone, back up, restore, move, and set policy</li> <li>Reporting: Retrieve storage utilization per node and datacenter efficiency data</li> <li>Simple query support: Retrieve and information using straight-forward, intuitive queries, and perform sorting, filtering, and paging for all object types</li> <li>Interactive documentation: Access an intuitive, comprehensive, real-time HTML5 reference</li> </ul>https://developer.hpe.com/morpheus/home/https://developer.hpe.com/morpheus/home/<p>Morpheus, a Hewlett Packard Enterprise company, is a vendor-agnostic management platform for multi-cloud orchestration, unified operations, and self-service provisioning. It bridges the gap between teams, tools, and processes, independent of where and how applications are deployed. As a centralized control plane, Morpheus enables enterprise-wide governance, policy enforcement, and self-service application provisioning access across virtually any hybrid cloud environment.</p> <p>Learn more at <a href="https://morpheusdata.com/">www.morpheusdata.com</a>.</p> <h2>API Documentation</h2> <p>The Morpheus platform offers an extensive REST API to manage the configuration of the platform as well as deploy workloads to the various clouds that the platform supports. Detailed information about the REST API and how to use it can be found in the <a href="https://apidocs.morpheusdata.com">Morpheus API documentation</a>.</p> <h2>Plugin Developer Portal</h2> <p>The Morpheus platform offers a robust plugin framework that enables developers to build integrations to extend the native platform functionality. The framework includes providers for creating custom reports, cloud integrations, UI integrations, and more. Start building integrations with your environment by browsing through the <a href="https://developer.morpheusdata.com">Morpheus developer portal</a>.</p> <h2>Morpheus Integrations</h2> <ul> <li> <p><strong>Morpheus Golang SDK:</strong> The Morpheus Golang library provides an interface for interacting with the Morpheus platform within a Golang application. Details on using the library can be found at <a href="https://github.com/gomorpheus/morpheus-go-sdk">morpheus-go-sdk <img src="Github" alt=""></a>.</p> </li> <li> <p><strong>Morpheus Terraform Provider:</strong> The Morpheus Terraform provider enables the Morpheus platform to be managed in a declarative fashion using HashiCorp Terraform. Documentation for the Terraform provider can be found at <a href="https://registry.terraform.io/providers/gomorpheus/morpheus/latest/docs">HashiCorp Terraform registry</a>.</p> </li> <li> <p><strong>Morpheus CLI:</strong> The Morpheus CLI is a Ruby-based command line tool for interacting with the Morpheus platform. The CLI can be used to programmatically manage the platform or deploy workloads using a YAML or JSON payload. Detailed information about the Morpheus CLI and how to use it can be found in the <a href="https://clidocs.morpheusdata.com">CLI documentation</a>.</p> </li> </ul> <h2>Gain hands-on experience with Morpheus</h2> <p>The Morpheus platform includes a community edition that enables users to get hands-on experience with the Morpheus platform. The platform can be quickly deployed on a single Linux server running in a home lab, datacenter, or public cloud environment. Get started with the community edition at <a href="https://morpheusdata.com/community">https://morpheusdata.com/community</a>.</p>https://developer.hpe.com/openchami/home/https://developer.hpe.com/openchami/home/<h2>Open Composable Heterogeneous Adaptable Management Infrastructure</h2> <p><strong>OpenCHAMI</strong> (Open Composable Heterogeneous Adaptable Management Infrastructure) is an open-source system management platform designed to bring <strong>cloud-like flexibility and security</strong> to High Performance Computing (HPC) environments. It provides modern tools to efficiently <strong>deploy, manage, and scale</strong> HPC clusters, making it easier for organizations to support both traditional scientific workloads and emerging <strong>AI, ML, and data-driven applications</strong>.</p> <h3><strong>Key Features</strong></h3> <ul> <li>🛡 <strong>Security-First Architecture</strong> – Implements <strong>zero-trust authentication</strong>, fine-grained <strong>access control</strong>, and <strong>OIDC-based authorization</strong> to safeguard HPC environments.</li> <li>🧩 <strong>Composable &#x26; Scalable</strong> – Modular, cloud-native design supports <strong>heterogeneous compute infrastructures</strong> across on-prem and cloud environments.</li> <li>🔧 <strong>Microservices-Based</strong> – Built on <strong>lightweight, containerized services</strong>, making deployment and scaling more efficient.</li> <li>🌐 <strong>Community-Driven &#x26; Open</strong> – Developed <strong>in collaboration with leading HPC sites</strong> under the <strong>Linux Foundation HPC initiative</strong>, ensuring transparency and innovation.</li> <li>🚀 <strong>Fast Boot</strong> - Faster and more secure boot times with cloud-init. POST + 40 seconds!</li> </ul> <h3><strong>Why OpenCHAMI?</strong></h3> <p>HPC system management has traditionally relied on <strong>monolithic, complex, and rigid</strong> solutions that are difficult to adapt to modern workloads. OpenCHAMI <strong>bridges the gap between HPC and cloud-native technologies</strong> by providing:</p> <ul> <li><strong>Security &#x26; Compliance</strong> – Designed with modern security best practices for protecting critical infrastructure.</li> <li><strong>Cloud-Like Flexibility</strong> – Enables dynamic, on-demand scaling and <strong>automation</strong> of system resources.</li> <li><strong>Open &#x26; Transparent Governance</strong> – Unlike proprietary solutions, OpenCHAMI is <strong>community-led</strong> and continuously evolving.</li> </ul> <p>Learn how OpenCHAMI simplifies HPC system management with <strong>security-first architecture, composability, and microservices-based deployment.</strong></p> <p>Watch OpenCHAMI in action <a href="https://youtu.be/UbBdbhzXjbA">here</a>:</p> <h3>Projects</h3> <h4><a href="https://github.com/openCHAMI">OpenCHAMI</a><a href="https://github.com/openCHAMI"></a></h4> <p>HPC System Management for Cloud Engineers and HPC Sysadmins</p> <p>Learn more:</p> <ul> <li><a href="https://openchami.org/">OpenCHAMI Website</a></li> <li><a href="https://openchami.org/docs/">Explore the OpenCHAMI Documentation</a></li> <li><a href="https://openchami.org/docs/tutorial/">Check the tutorials</a></li> <li><a href="https://openchami.org/blog/">Read blogs</a></li> </ul> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based workshops available in the HPE Developer <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on OpenCHAMI open-source project?</h2> <p>Join the <a href="https://join.slack.com/share/enQtMTAzNjA1NDMxMjI2NzItYTY1Yjk3NzgzNjhlZWEyZThjY2RiNGU3ZTM1MzhlMGQxYmRmZTJiMTIyMTZkYjQ4MDYzMTU0NTUzZmZmYTNmZQ">open source slack channel</a> and start a discussion.</p>https://developer.hpe.com/project-data-map/home/https://developer.hpe.com/project-data-map/home/<p>With so much data expanding exponentially at the edge, it’s challenging to find, use, manage, and share the right data when you need it. To acquire the true meaning of your data and take action within your distributed enterprise, HPE is developing Project Data Map, which will allow you to navigate to the data you’re looking for with governance, lineage and trust.</p> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe title="Brightcove Player" src="//players.brightcove.net/4119874060001/tViCJfxWJ_default/index.html?videoId=ref:2f8dce2b-5585-48e3-a9bb-95e4a7f04562" allowfullscreen="" frameborder="0" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <p>Project Data Map includes an open Common Metadata Framework, and correlates and scores metadata relationships derived implicitly from data usage and signals that continuously track pattern changes. This allows it to get smarter data, enabling precision AI and machine learning models to make the best possible decisions.</p> <p>HPE is developing Project Data Map to:</p> <ul> <li> <p>Democratize access to data, data analytics, and AI to domain experts, citizen users, and data scientists.</p> </li> <li> <p>Create a standard and open approach to meta data, enabling easier cross-vertical discovery and sharing of distributed diverse data sets.</p> </li> <li> <p>Use data governance of rights and conditions to improve efficiency, promote transparency, and enable business insight.</p> </li> <li> <p>Accelerate the data and analytics exchange.</p> </li> <li> <p>Reduce the extreme expense of cloud egress.</p> </li> <li> <p>Secure data movement from every layer of the stack, continuously and automatically.</p> </li> <li> <p>Enable a distributed enterprise to minimize unnecessary data movement for more efficient data management and help with cloud egress.</p> </li> </ul> <h2>Features</h2> <p>With an open source foundation and a global data fabric, like HPE Ezmeral software, Project Data Map provides seamless, unified access to distributed data and unified control of distributed Kubernetes clusters. Project Data Map features:</p> <ul> <li> <p><strong>A data preview browser</strong>, enabling you to see fragments of data and access its lineage to check on validity and quality.</p> </li> <li> <p><strong>Data exchange and marketplace access</strong>, providing a mechanism to publish contextual meta data to help define policy, license and exchange contracts, and provide access rights to data when criteria is met.</p> </li> <li> <p><strong>An open Common Metadata Framework</strong> that standardizes the way for you to find the most meaningful data based on personalized recommendations.</p> </li> <li> <p><strong>A data policy engine</strong>, making data sovereignty possible through dynamic policy generation and governance integrated with standard AAA services and versioned persistence on-platform and enforceable license models.</p> </li> <li> <p><strong>Data search and filter/meta data standards</strong>, automating meta data generation on ingest with the ability to provide rich meta data to describe the data and update it for personalization.</p> </li> <li> <p><strong>An Open API</strong>, allowing a service ecosystem to both consume data and orchestrate the creation of meta data for the mutual benefit of data producers and consumers.</p> </li> <li> <p><strong>Trustworthy Data Foundation</strong>, tracking and analyzing meta data from edge to core in order to shape data and pipelines with trustworthy elements throughout the entire data lifecycle.</p> </li> </ul> <h2>Resources</h2> <ul> <li> <p><a href="https://youtu.be/71_dEAWuBPw?t=1144">New Innovations: Project Edge Cluster, Project Sustainability Edge Cluster, and Project Data Map</a></p> </li> <li> <p><a href="https://youtu.be/71_dEAWuBPw?t=2305">Project Data Map demo</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=9VTLA1nxpoo">Lightboard video</a></p> </li> <li> <p><a href="https://www.hpe.com/us/en/insights/articles/getting-the-most-from-your-data-driven-transformation-2109.html">Getting the most from your data-driven transformation: 10 key principles</a></p> </li> <li> <p><a href="https://www.hpe.com/us/en/insights/articles/getting-value-from-your-data-shouldn-t-be-this-hard-2106.html">Getting value from your data shouldn't be this hard</a></p> </li> <li> <p><a href="https://community.hpe.com/t5/Advancing-Life-Work/Dataspaces-how-an-open-metadata-layer-can-establish-a/ba-p/7149075#.Yw01QXbMKUk">How an open metadata layer can establish a trustworthy data pipeline</a></p> </li> <li> <p><a href="https://www.hpe.com/us/en/insights/articles/why-the-future-of-ai-hinges-on-trust-2205.html">Why the future of AI hinges on trust</a></p> </li> </ul> <h2>Collaborations</h2> <p>Explore how Project Data Map can transform the flow of data in these real-world projects:</p> <ul> <li> <p><strong>CGIAR (formerly the Consultative Group for International Agricultural Research), Digital Green, The AgStack Foundation and HPE</strong> are on a global mission to build solutions to address the exponentially growing need for food. To learn more, watch the <a href="https://www.hpe.com/us/en/discover-more-network/series/scale-for-good.html?media-id=/us/en/resources/discover/dmn/scale-for-good/thefoodcrisiscantechnologyscaletofeedtheworld/_jcr_content.details.json&#x26;media-strategy=delegate">Scale for Good video</a>, and the <a href="https://www.youtube.com/watch?v=g0cGYXg11Os">AI for Good webinar</a>.</p> </li> <li> <p><strong>Novartis Global Health and HPE</strong> are seeking ways to develop a disease surveillance solution for dengue fever by collecting and integrating complex data sources across organizations globally, providing data transparency at scale to inform targeted response strategies. To learn more, <a href="https://www.hpe.com/us/en/newsroom/news-advisory/2021/06/hewlett-packard-enterprise-and-novartis-join-forces-to-advance-novartis-global-health-efforts.html">read the press release</a>.</p> </li> <li> <p><strong>Gaia-X and HPE</strong> are creating an interoperable data exchange in which data can be shared under the protection of European data privacy laws. Hear the <a href="https://share.transistor.fm/s/b465abf0">HPE Tech Talk podcast</a> to learn more.</p> </li> </ul> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. Project Data Map workshops are coming soon.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on Project Data Map?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C03LU2V1CSJ">#HPEDataMap</a> channel.</p>https://developer.hpe.com/smartsim/home/https://developer.hpe.com/smartsim/home/<h1>What is SmartSim?</h1> <p>Recently, there is growing interest in applying machine learning (ML) algorithms to improve scientific simulation efficiency and accuracy. New software approaches are needed to couple existing scientific applications, traditionally written in Fortran/C/C++ and MPI, to rapidly evolving ML and data analytics libraries, typically written in Python. Currently, the diversity of programming languages, dependence on file input/output (I/O), and large variance in compute resource requirements for scientific applications makes it difficult to perform online analysis, training, and inference with most ML and data analytics packages at the scale needed for numerical simulations.</p> <p>How does one connect the two programming paradigms of numerical model development and machine learning? While on the surface, this question seems to approach the problem, we believe the true difficulty (and opportunity) in bridging these workloads needs to be reformulated in terms of data exchange: How to pass data between a simulation and ML model at scale? SmartSim provides the answer to this challenge.</p> <p>SmartSim is a software framework that facilitates the convergence of numerical simulations and AI workloads on heterogeneous architectures. SmartSim enables simulations in Fortran,C, C++ and Python to execute ML models hosted within in-memory storage (DRAM) facilitating online inference at simulation runtime. In addition, SmartSim’s API orchestrates the movement of data between simulation and learning components with single-line put/get semantics. SmartSim can host ML models on CPU-only or GPU-enabled compute nodes adjacent to or co-located with the simulation. SmartSim is portable and can be run on laptops and scales to thousands of processors.</p> <p><img src="/img/smartsim-architecture.png" alt="" title="SmartSim Architecture"></p> <p>SmartSim is comprised of two libraries, a single light weight client library, the SmartRedis, that is compiled into end-users simulation and an Infrastructure Library (IL) that facilitates workflow around simulations. SmartSim users are able to run their ML models written in Python in either TensorFlow, TensorFlow-Lite, Keras, Pytorch, or any framework that can serialize to ONNX(e.g. scikit learn). In addition to online inference, SmartSim also facilitates online analysis, visualization, learning, and computational steering. Because simulation data can be held in co-located DRAM, scientists can interact with and perturb model data manually or programmatically during the course of a simulation. SmartSim enables users to execute nearly any simulation from a Jupyter notebook on HPC systems that support PBSPro, Slurm, Cobalt, as well as laptops and workstations.</p> <p>Interested to learn more? Check out these resources:</p> <ul> <li><a href="https://www.youtube.com/watch?v=JsSgq-fq44w">Watch a recent presentation to Pangeo</a></li> <li><a href="https://www.craylabs.org/docs/overview.html">Read the SmartSim documentation</a></li> <li><a href="https://github.com/CrayLabs/SmartSim">SmartSim Github open-source repository</a></li> <li><a href="https://arxiv.org/abs/2104.09355">Read our recent paper</a></li> <li><a href="https://join.slack.com/t/craylabs/shared_invite/zt-nw3ag5z5-5PS4tIXBfufu1bIvvr71UA">Join our Slack community</a></li> </ul>https://developer.hpe.com/spiffe-and-spire-projects/home/https://developer.hpe.com/spiffe-and-spire-projects/home/<p>Inspired by production infrastructure at Facebook, Google, Netflix, and more, SPIFFE is a set of open-source standards for securely authenticating software services in dynamic and heterogeneous infrastructures through platform-agnostic, cryptographic identities. SPIRE is an open-source system that implements the SPIFFE specification in a wide variety of environments.</p> <p>Together, the projects deliver a foundational capability, <em>service identity</em>, for cloud- and container-deployed microservices. They enable organizations to deploy consistent, fine-grained cross-service authentication via a “dial-tone” API across heterogeneous environments.</p> <p><a href="https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/">SPIFFE and SPIRE are graduated projects</a> from the Cloud Native Computing Foundation (CNCF). Joining the group of already <a href="https://www.cncf.io/projects/">graduated projects</a>,  including HELM and Kubernetes, SPIFFE and SPIRE projects have received contributions from Bloomberg, Google, Pinterest, Square, Uber, HPE and others and have grown to become a foundational layer within the cloud native ecosystem. These projects integrate with multiple cloud native technologies and projects, such as Istio, Envoy, gPRC, Sigstore, and OPA (Open Policy Agent).</p> <ul> <li><a href="https://spiffe.io/">Project web site at CNCF</a></li> </ul> <p>You can download an eBook that presents the SPIFFE standard for service identity, and SPIRE, the reference implementation for SPIFFE <a href="https://spiffe.io/book/">here</a>.</p> <hr> <h1>Learn from the experts</h1> <h3>Introduction to SPIFFE and SPIRE</h3> <p>In this lightboard video, Evan Gilman, co-author of O’Reilly’s book Zero Trust Networks and a maintainer for SPIRE, provides an overview of CNCF’s SPIFFE and SPIRE Projects. Evan goes into the security issues that SPIFFE/SPIRE solve and how through workload identity attestation.</p> <p><a href="https://www.youtube.com/watch?v=-XGKybqTfZo"><img src="https://img.youtube.com/vi/-XGKybqTfZo/hqdefault.jpg" alt="Introduction to Spiffe and Spire"></a></p> <h3>Service Authentication for Zero Trust Model with SPIRE</h3> <p>In this video, Evan Gilman, co-author of O’Reilly’s book Zero Trust Networks and a maintainer for SPIRE, explains how SPIRE addresses zero trust challenges in a distributed environment.</p> <p><a href="https://www.youtube.com/watch?v=QNDWRQY0t-o"><img src="https://img.youtube.com/vi/QNDWRQY0t-o/hqdefault.jpg" alt="Zero Trust challenges"></a></p> <hr> <h1>GitHub repositories <img src="Github" alt=""></h1> <ul> <li><a href="https://github.com/spiffe/spiffe">spiffe</a>: This repository includes the SPIFFE ID, SVID and Workload API specifications, example code, and tests, as well as project governance, policies, and processes.</li> <li><a href="https://github.com/spiffe/spire">spire</a>: This is a reference implementation of SPIFFE and the SPIFFE Workload API that can be run on and across varying hosting environments.</li> <li><a href="https://github.com/spiffe/go-spiffe/tree/main/v2">go-spiffe</a>: Golang client libraries.</li> <li><a href="https://github.com/spiffe/java-spiffe">java-spiffe</a>: Java client libraries</li> <li><a href="https://github.com/HewlettPackard/py-spiffe">py-spiffe</a>: Python client libraries</li> <li><a href="https://github.com/HewlettPackard/c-spiffe">c-spiffe</a>: C client libraries</li> </ul> <hr> <h1>Integrations</h1> <ul> <li><a href="https://spiffe.io/docs/latest/microservices/envoy/">Tutorial on how to configure the Envoy proxy with SPIFFE and SPIRE</a></li> <li><a href="https://istio.io/latest/docs/ops/integrations/spire/">Tutorial on how to configure Istio with SPIRE</a>. The Istio integration was contributed to by HPE engineers, and is now part of Istio, since V1.14.</li> </ul> <hr> <h1>Workshops-on-Demand</h1> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/workshops/">HPE Developer Community Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. SPIFFE and SPIRE workshops are available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <hr> <h2>Any questions on SPIFFE?</h2> <p>Join the <strong><a href="https://slack.spiffe.io/">SPIFFE Slack Workspace</a></strong> and start a discussion.</p>https://developer.hpe.com/swarm-learning/home/https://developer.hpe.com/swarm-learning/home/<p><strong>Editor’s Note: HPE Swarm Learning product is now offered only as a community edition.</strong></p> <p>HPE Swarm Learning is a decentralized, privacy-preserving Machine Learning framework. This ML framework utilizes the computing power at, or near, the distributed data sources to run the Machine Learning algorithms that train the models. It uses the security of a blockchain platform to share learnings with peers in a safe and secure manner. In HPE Swarm Learning, training of the model occurs at the edge, where data is most recent, and where prompt, data-driven decisions are mostly necessary. In this completely decentralized architecture, only the insights learned are shared with the collaborating ML peers, not the raw data. This tremendously enhances data security and privacy.</p> <p>Swarm Learning nodes works in collaboration with other Swarm Learning nodes in the network. It regularly shares its learnings with the other nodes and incorporates their insights. This process continues until the Swarm Learning nodes train the model to desired state. User can monitor the progress of the current training as shown in the below image. It shows all running Swarm nodes, loss, model metric (for example, accuracy) and overall training progress for each User ML node. On hovering over the "progress bar", one can see the number of completed epochs and the total number of epochs.</p> <p><img src="/img/guid-cb6f59c9-7cd9-4ee8-ba7c-3082f07b8491-high.png" alt="" title="HPE Swarm Learning Topology"></p> <p><img src="/img/guid-899b556f-d33f-42d1-8d0d-37f191715709-high.jpg" alt="HPE Swarm Learning architecture" title="HPE Swarm Learning architecture"></p> <p>User can now extend Swarm client to support other machine learning platforms as well. Currently Swarm client supports machine learning platforms like PyTorch and Keras (based on Tensorflow 2 in backend). Please find the instructions to extend Swarm client <a href="https://github.com/HewlettPackard/swarm-learning/blob/master/lib/src/README.md">here</a>.</p> <p>We are happy to announce <a href="https://github.com/HewlettPackard/swarm-learning/releases/tag/v2.2.0">Swarm 2.2.0 community release</a>. In this release, we have delivered key enhancements on UI/UX, which includes experiment tracking for easier “birds-eye” visualization of past training rounds, parallel Swarm installation on multiple hosts, Podman support via SLM-UI etc., that will significantly enhance user experience. We have also added powerful features to Swarm manageability framework for better management of user ML workloads.</p> <hr> <h2>Articles</h2> <ul> <li><strong><a href="https://www.nature.com/articles/s41586-021-03583-3">2021 Nature Paper - Swarm Learning for decentralized and confidential clinical machine learning</a></strong></li> <li><strong><a href="https://www.researchgate.net/publication/342495847_Swarm_Learning_as_a_privacy-preserving_machine_learning_approach_for_disease_classification">Research gate - Swarm Learning as a privacy-preserving machine learning approach for disease classification</a></strong></li> </ul> <hr> <h2>Learn more about HPE Swarm Learning and its underlying technology</h2> <p><strong>- The big shift: What is swarm learning?</strong></p> <p>Swarm learning is the next gold rush for machine intelligence - training at the edge so the edge devices get smarter and also train their peers. With no central authority, Blockchain is integrated to add control, privacy, and security. Learn more by watching <strong>What is swarm learning?</strong>:</p> <p><a href="https://www.youtube.com/watch?v=6Fep6Lw5t-U"><img src="https://img.youtube.com/vi/6Fep6Lw5t-U/hqdefault.jpg" alt="What is swarm learning?"></a></p> <hr> <h2>Listen to what the experts are saying</h2> <p>- <strong>How to optimize machine learning at the edge with HPE Swarm Learning</strong></p> <p>In this video Ronald van Loon and HPE Chief Technologist, Krishnaprasad Shastry, talk about optimizing machine learning at the edge with swarm learning. Learn more by watching <strong>How to optimize machine learning at the edge with HPE Swarm Learning</strong>:</p> <p><a href="https://www.youtube.com/watch?v=paBt6nvyTHQ"><img src="https://img.youtube.com/vi/paBt6nvyTHQ/hqdefault.jpg" alt="Optimize ML at the edge with HPE Swarm learning"></a></p> <hr> <h2>Resources<a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/swarm-learning.html"></a></h2> <ul> <li><a href="https://github.com/HewlettPackard/swarm-learning"><strong>HPE Swarm Learning community edition <img src="Github" alt=""></strong></a></li> <li><a href="https://github.com/HewlettPackard/swarm-learning/tree/master/examples"><strong>Try HPE Swarm Learning examples <img src="Github" alt=""></strong></a></li> <li><a href="https://github.com/HewlettPackard/swarm-learning/blob/master/lib/src/README.md"><strong>HPE Swarm Learning client <img src="Github" alt=""></strong></a></li> </ul> <hr> <h2>Any questions on HPE Swarm Learning?</h2> <p>Join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C04A5DK9TUK">#hpe-swarm-learning</a> channel.</p>https://developer.hpe.com/zerto/home/https://developer.hpe.com/zerto/home/<p>Zerto, a Hewlett Packard Enterprise company, empowers customers to run an always-on business by simplifying the protection, recovery, and mobility of on-premises and cloud applications. Zerto’s continuous data protection (CDP) eliminates the risks and complexity of modernization and cloud adoption across private, public, and hybrid deployments. The simple, software-only solution uses continuous data protection at scale to converge disaster recovery, backup, and data mobility. Zerto is trusted by over 9,500 customers globally and is powering offerings for Microsoft Azure, IBM Cloud, AWS (Amazon Web Services), Google Cloud, Oracle Cloud, and more than 350 managed service providers.  </p> <p>Learn more about <a href="https://www.hpe.com/us/en/zerto.html">Zerto</a></p> <h2>Welcome to the Zerto Hacker Hub!</h2> <p>On this page, will be provided the tools to help you get started in deploying and managing Zerto’s CDP, which leverages its proprietary near-synchronous replication capabilities to protect and restore virtual machines and public cloud instances. Zerto offers an extensive REST API with Swagger support that enables users to programmatically create and manage end-to-end disaster recovery workflows.  </p> <p>Get started by diving into <a href="https://github.com/ZertoPublic">our GitHub page</a> or check out the documentation at <a href="https://help.zerto.com">https://help.zerto.com</a></p> <h3>Simplify automation with Zerto’s REST Swagger APIs</h3> <p>Zerto’s REST Swagger APIs offer a powerful and user-friendly way to automate and integrate Zerto into your existing IT workflows. These APIs provide a comprehensive set of endpoints that allow you to programmatically manage, monitor, and control your Zerto environment with ease. Whether you are looking to automate routine tasks, integrate with third-party tools, or develop custom applications, Zerto’s REST APIs make it straightforward and efficient.</p> <p>With Swagger, developers can interactively explore and test the APIs, making it easier to understand their capabilities and integrate them into your systems. The Swagger interface provides clear documentation and real-time testing capabilities, ensuring that you can quickly get up to speed and start leveraging Zerto’s powerful features.</p> <p>Unlock the full potential of your disaster recovery strategy with Zerto’s REST Swagger APIs today!</p> <p>For more information and to access the Zerto REST Swagger APIs, visit the <a href="https://help.zerto.com/category/ZVM_REST_API_Swagger">Zerto API Documentation</a>.</p> <h3>Unlock powerful insights with Zerto Resiliency Observation Console</h3> <p>Meet the Zerto Resiliency Observation Console (zROC) – a dynamic, Docker-compose based software stack designed to transform your Zerto API data into rich, visual insights using Prometheus and Grafana. This innovative tool empowers you to monitor and analyze your data effortlessly.</p> <p>What sets zROC apart is its custom Prometheus exporter code, crafted to seamlessly integrate with standard Prometheus and Grafana containers. The comprehensive configuration files included in the repository ensure that you can start deriving value immediately, straight out of the box.</p> <p>While zROC comes with pre-built dashboards to get you started, we encourage you to unleash your creativity by building custom dashboards or tweaking existing ones to perfectly match your unique needs. Dive into the world of intuitive data visualization and take your monitoring capabilities to the next level.</p> <p>Explore the Zerto Resiliency Observation Console today on our <a href="https://github.com/ZertoPublic/zroc">GitHub page</a>.</p> <h3>Zerto In-Cloud for AWS</h3> <p>Zerto In-Cloud for AWS is disaster recovery for EC2 instances between regions and availability zones. Zerto In-Cloud for AWS blends the familiar Zerto approach to disaster recovery for workloads with the AWS native platform capabilities and services. Zerto In-Cloud for AWS features virtual protection groups, non-disruptive to production failover testing to isolated environments, and disaster recovery failover orchestration. </p> <p>Zerto In-Cloud for AWS supports 1000+ instance types and is 100% API-first developed to enable automation tools integration. </p> <p>Linked to our <a href="https://github.com/ZertoPublic/ZIC-AWS">GitHub here</a> are the Terraform files you can use to start experimenting with Zerto In-Cloud for AWS.  </p> <p>Ready to learn more? Read the <a href="https://www.zerto.com/wp-content/uploads/2021/11/Zerto-In-Cloud-for-AWS-Data-Sheet.pdf">Zerto-In Cloud for AWS datasheet</a>.</p> <h2>Resources</h2> <p>Ready to learn more about Zerto? Browse through the <a href="https://www.zerto.com/">Zerto portal</a>.</p> <p>Looking to try Zerto out? Sign up for our <a href="https://www.zerto.com/myzerto/labs">Hands on Labs! </a></p> <br /> <hr> <h2>Any questions about Zerto?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C03J3EGDDM0">#zerto</a> channel.</p> <p>Not a Slack user? Send us an email to <a href="mailto:zertotm@hpe.com">zertotm@hpe.com</a></p>https://developer.hpe.com/Arkouda/https://developer.hpe.com/Arkouda/https://developer.hpe.com/Chapel/https://developer.hpe.com/Chapel/https://developer.hpe.com/Cloud_Native_Computing_Foundation/https://developer.hpe.com/Cloud_Native_Computing_Foundation/https://developer.hpe.com/Debian/https://developer.hpe.com/Debian/https://developer.hpe.com/Distributed_Management_Task_Force/https://developer.hpe.com/Distributed_Management_Task_Force/https://developer.hpe.com/ETSI/https://developer.hpe.com/ETSI/https://developer.hpe.com/Gen-Z_support_on_Linux/https://developer.hpe.com/Gen-Z_support_on_Linux/https://developer.hpe.com/Grommet/https://developer.hpe.com/Grommet/https://developer.hpe.com/HPE_Deep_Learning_Benchmarking_Suite/https://developer.hpe.com/HPE_Deep_Learning_Benchmarking_Suite/https://developer.hpe.com/HPE_Software_Delivery_Repository/https://developer.hpe.com/HPE_Software_Delivery_Repository/https://developer.hpe.com/HudsonAlpha_Synergy_Contribution/https://developer.hpe.com/HudsonAlpha_Synergy_Contribution/https://developer.hpe.com/KubeEdge/https://developer.hpe.com/KubeEdge/https://developer.hpe.com/LF_EDGE/https://developer.hpe.com/LF_EDGE/https://developer.hpe.com/LinuxCOE/https://developer.hpe.com/LinuxCOE/https://developer.hpe.com/LinuxKI/https://developer.hpe.com/LinuxKI/https://developer.hpe.com/Linux_Foundation/https://developer.hpe.com/Linux_Foundation/https://developer.hpe.com/Linux_Software_Raid_Redundant_Boot/https://developer.hpe.com/Linux_Software_Raid_Redundant_Boot/https://developer.hpe.com/Mageia_Linux_Distribution/https://developer.hpe.com/Mageia_Linux_Distribution/https://developer.hpe.com/MapR_Technologies/https://developer.hpe.com/MapR_Technologies/<p>A collection of open source projects developed by MapR Technologies before it was acquired by HPE.</p>https://developer.hpe.com/Memory-Driven_Computing_Developer_Toolkit/https://developer.hpe.com/Memory-Driven_Computing_Developer_Toolkit/https://developer.hpe.com/OpenSHMEM/https://developer.hpe.com/OpenSHMEM/https://developer.hpe.com/Open_Compute_Project/https://developer.hpe.com/Open_Compute_Project/https://developer.hpe.com/Python-redfish/https://developer.hpe.com/Python-redfish/https://developer.hpe.com/SmartSim/https://developer.hpe.com/SmartSim/https://developer.hpe.com/RSFO_(Rapid_Setting_for_Oracle)/https://developer.hpe.com/RSFO_(Rapid_Setting_for_Oracle)/https://developer.hpe.com/Spark/https://developer.hpe.com/Spark/https://developer.hpe.com/Spiffe/https://developer.hpe.com/Spiffe/https://developer.hpe.com/Spire/https://developer.hpe.com/Spire/https://developer.hpe.com/Storage_drivers/https://developer.hpe.com/Storage_drivers/https://developer.hpe.com/Topology_Framework/https://developer.hpe.com/Topology_Framework/https://developer.hpe.com/determined-ai/https://developer.hpe.com/determined-ai/<p>Determined is an open-source deep learning training platform that makes building models fast and easy</p>https://developer.hpe.com/dragonhpc/https://developer.hpe.com/dragonhpc/<p>DragonHPC is a programmable distributed runtime for HPC &#x26; AI workflows.</p>https://developer.hpe.com/kubedirector/https://developer.hpe.com/kubedirector/<p>Helps deploy and manage stateful applications on Kubernetes.</p>https://developer.hpe.com/openchami/https://developer.hpe.com/openchami/<p>HPC System Management for Cloud Engineers and HPC Sysadmins</p>https://developer.hpe.com/spiffe-and-spire-projects/https://developer.hpe.com/spiffe-and-spire-projects/https://developer.hpe.com/aruba-central/home/https://developer.hpe.com/aruba-central/home/<p>Supercharge your network operations with HPE Aruba Networking Developer Hub. Find help and resources to automate networks and quickly build powerful applications by integrating with HPE Aruba Networking products. Find APIs and guides, visit the Code Exchange, and join our community of Airheads.</p> <ul> <li><a href="https://devhub.arubanetworks.com/">Check out the HPE Aruba Networking Developer Hub</a></li> <li><a href="https://community.arubanetworks.com/community-home/digestviewer?communitykey=ea467413-8db4-4c49-b5f8-1a12f193e959&#x26;tab=digestviewer">Join the conversation at the Airheads Developer Community</a></li> <li><a href="https://www.ansible.com/integrations/networks/aruba">Simplify network operations with HPE Aruba Networking and Ansible</a></li> <li><a href="https://github.com/aruba">Leverage the HPE Aruba Networking GitHub repository</a></li> </ul> <h2>Resources</h2> <ul> <li><a href="https://developer.arubanetworks.com/hpe-aruba-networking-central/docs/rest-api-getting-started">HPE Aruba Networking Central API reference</a></li> <li><a href="https://developer.arubanetworks.com/aruba-central/docs/ansible-getting-started">Getting Started with Ansible and HPE Aruba Networking Central</a></li> <li><a href="https://github.com/aruba/aruba-postman-collections">HPE Aruba Networking Central Postman collections</a></li> <li><a href="https://devhub.arubanetworks.com/code-exchange">HPE Aruba Networking Code Exchange</a></li> </ul> <h2>Any questions on Aruba?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C0164BJHKJP">#aruba-central</a> Slack channel.</p>https://developer.hpe.com/data-services-cloud-console/home/https://developer.hpe.com/data-services-cloud-console/home/<p>To eliminate the silos and complexity of data management and infrastructure, HPE provides the Data Services Cloud Console, a SaaS-based cloud console that delivers cloud operational agility and unified data operations as a service. Data Services Cloud Console also offers a unified API that gives developers access to infrastructure and data as code.</p> <p><img src="/img/a-single-destination-for-data-and-infra-services.png" alt="DSCC UI" title="DSCC UI diagram"></p> <p>The cloud console brings the cloud experience to wherever data lives and streamlines data management across your hybrid cloud. It provides a suite of cloud services across your edge, core, and public cloud to accelerate data, agility, and innovation for everyone, from data managers, to data innovators. Delivered as a SaaS-based console, Data Services Cloud Console speeds customer innovation because there is nothing to deploy, everything is always updated to the latest software, and ownership risk is reduced by virtue of it being a subscription. The console is a single pane for management of any data and infrastructure services that are found as the cloud data services and the cloud infrastructure services catalog.</p> <h3>Introduction to Data Services Cloud Console</h3> <p>To access the Data Services Cloud Console, click on this URL <a href="https://console.greenlake.hpe.com">https://console.greenlake.hpe.com</a>, and sign-in using your HPE GreenLake's account.<a href="https://common.cloud.hpe.com"></a></p> <p>For a detailed overview of the console, and to get started using it, please click on this link <a href="https://www.hpe.com/us/en/storage/data-services-cloud-console.html">https://www.hpe.com/us/en/storage/data-services-cloud-console.html</a>.</p> <p>For a quick overview of the console, please click on the videos below.</p> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe title="HPE Data Services Cloud Console Chalk Talk" src="https://www.youtube.com/embed/AxUE89X3Sy0" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe title="A Closer Look at HPE Data Services Cloud Console" src="https://www.youtube.com/embed/lzOWapX0m5U" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <h3>Data Services Cloud Console Public REST API</h3> <p>Data Services Cloud Console public REST API provides a resource for customers who are looking to enhance their infrastructure management and data-ops using the programmatic extensions from the console. More blog posts will be coming that will provide more information to help customers adopting this API, providing examples, code snippets and other helpful information.</p> <p>For detailed <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/"></a>exploration of the console's public REST API, please take a look at the following blog posts and the video of the API demo in the Youtube.</p> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe title="Introduction to HPE Data Services Cloud Console public API" src="https://www.youtube.com/embed/g3UO0S-4r6I" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <ul> <li>Blog: <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/">Getting Started with Data Services Cloud Console API</a></li> <li>Blog: <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Using HPE GreenLake Console's API Gateway for Data Services Cloud Console</a></li> <li>Blog: <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">Implementing OAuth 2.0 Flow for Data Services Cloud Console's Client Application</a></li> </ul> <h3>Any Question on Data Services Cloud Console?</h3> <p>Please join <a href="https://slack.hpedev.io/">HPEDEV Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services-cloud-console</a> channel.</p>https://developer.hpe.com/data-services-on-the-hpe-greenlake-platform/home/https://developer.hpe.com/data-services-on-the-hpe-greenlake-platform/home/<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Data Services on the HPE GreenLake edge-to-cloud platform is a group of services which are part of the service catalogues in the HPE GreenLake edge-to-cloud Platform. These services bring the cloud experiences to HPE GreenLake customers, wherever the data lives, across on-premises or public cloud throughout its lifecycle. With the streamlined ordering, provisioning, management, protection, analysis, and archiving, customers can achieve higher agility to innovate using latest trend in I.T. such as artificial intelligence. For more information on how to leverage these Data Services on the HPE GreenLake edge-to-cloud platform, please visit HPE Data Management <a href="https://www.hpe.com/us/en/storage/data-services-cloud-console.html">website</a>.</p> <h2>Introduction to Data Services on the HPE GreenLake edge-to-cloud platform</h2> <p>There are two major categories of services from Data Services on the HPE GreenLake edge-to-cloud platform:</p> <ol> <li><strong>Infrastructure Platform suites</strong> for the HPE Storage family (for more info <a href="https://www.hpe.com/us/en/greenlake/storage.html">visit</a> HPE GreenLake for Data Storage) to enable configuration, management, monitor and optimization of HPE Alletra Storage families for block, file, and object. These services known as the <a href="https://www.hpe.com/us/en/hpe-greenlake-compute-ops-management.html">HPE GreenLake Data Ops Manager</a>, <a href="https://www.hpe.com/us/en/hpe-greenlake-block-storage.html">HPE GreenLake for Block Storage</a>, <a href="https://www.hpe.com/us/en/hpe-greenlake-file-storage.html">HPE GreenLake for File Storage</a>, and <a href="https://www.hpe.com/us/en/hpe-greenlake-storage-fabric-management.html">HPE GreenLake for Storage Fabric Management.</a></li> <li><strong>Cloud Data Services suites</strong> for hybrid cloud workload lifecycle management, to enable data protection, to provide disaster recovery, to deploy self-service hybrid cloud, and to enable infrastructure analysis. These services known as <a href="https://www.hpe.com/us/en/hpe-greenlake-backup-recovery.html"></a><a href="https://www.hpe.com/us/en/hpe-greenlake-private-cloud-business-edition.html">HPE GreenLake for Private Cloud Business Edition</a>, and <a href="https://www.hpe.com/us/en/hpe-greenlake-disaster-recovery.html">HPE GreenLake for Disaster Recovery</a>.</li> </ol> <p><img src="/img/data-services-catalogues.png" alt="HPE GreenLake Data Services Catalogue"></p> <p><em>The above figure displays the list of the services that are part of the Data Services on the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=index.html">HPE GreenLake edge-to-cloud platform</a></em></p> <h2>Data Services REST APIs</h2> <h3>The history</h3> <p>In <strong>November 2021</strong>, after the release of the HPE GreenLake edge-to-cloud platform, Data Services introduce the first set of APIs to manipulate resources made available under HPE GreenLake Data Ops Manager. Today, this set of API was expanded to include Block Storage, File Storage services, and current storage Family of HPE Primera, Nimble Gen5, and Alletra (6K,9K,MP). This set of APIs, was associated with <strong>Infrastructure Platform suites</strong>, also known as the <strong>Data Services Cloud Console API v1.5 (March 2024)</strong>. This existing API conforms to the <strong>OpenAPI Standard 3.0</strong> specification, and the spec file can be downloaded in <strong>either YAML or JSON OpenAPI Specification</strong> from the following <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">website</a>. The blog posts about getting started with this set of existing APIs is available in this <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/">HPE Developer Forum</a>. There are also additional blog posts that contains information such as <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">authentication using OAuth 2.0</a> to generate the required <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/">access token</a>, <a href="https://developer.hpe.com/blog/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/">using openapi-generator-cli</a> to convert the DSCC API OpenAPI Specifications to <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-powershell-sdk/">PowerShell</a> and <a href="https://developer.hpe.com/blog/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/">Python client Library</a>, a <a href="https://developer.hpe.com/blog/new-powershell-toolkit-available-for-managing-hpe-data-services-cloud-console/">PowerShell toolkit</a> and <a href="https://developer.hpe.com/blog/automating-operations-on-dscc-using-ansible-playbooks/">an example of ansible playbooks</a> to manage this storage sources.</p> <h3>The current family of the APIs</h3> <p>With the announcements of additional API support for the Data Services suites on the HPE GreenLake edge-to-cloud platform in <strong>March 2024</strong>, Cloud Data Services introduce a family of API sets to expand the support of Data Services such as HPE GreenLake for Backup and Recovery and HPE GreenLake for Private Cloud Business Edition. These sets of API conforms to the <strong>OpenAPI Standard 3.1</strong> specification, and it can be downloaded in <strong>JSON OpenAPI Specification</strong> file from the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake developer website</a>. These later API sets use the same authentication as the existing API (OAuth 2.0 Client Credential) and the same access token for all services inside a HPE GreenLake region. There are blog posts that introduce the new API sets, and tips to use available OpenAPI tool to convert the OpenAPI spec 3.1 to 3.0 so that the spec can be converted using the open-source openapi-generator into client-Libraries of multiple scripting languages.</p> <h3>The new set of Data Services APIs</h3> <p>This set of APIs family are broken down into different groups:</p> <ol> <li><strong>Data Services :</strong> Group of APIs to accommodate common resources used for both Data Services and Infrastructure Platform suites such as Task List (Asynchronous Operations), Dual Authorization, Issues, Settings, Storage Locations, and displaying DSCC Tags</li> <li><strong>Virtualization :</strong> Group of APIs to accommodate common interaction with both on-premises and public cloud Hyper-Visors such as registration of VMware vCenter or AWS account, discovery of virtual machines or AWS EC2 instance, migration of virtual machines and many others use cases.</li> <li><strong>Private Cloud Business Edition :</strong> Group of APIs designed to view inventory of a DHCI 2.0, SimpliVity, or Azure cloud account, to perform upgrades for Storage software, hyper-Visor software, to add, update delete provisioning policies and many other use cases in HPE GreenLake for Private Cloud Business Edition.</li> </ol> <h3>The new documentation about the family of Data Services APIs</h3> <p>This is the pivoting point for the HPE GreenLake APIs for Data Services on the HPE GreenLake edge-to-cloud platform where HPE GreenLake user’s clients for API can perform more complex automation that involves all categories of services made available in the Data Services on HPE GreenLake edge-to-cloud platform. To accommodate the documentation for the combination of multiple sets of APIs, the HPE GreenLake documentation about the data services APIs had been updated to provide directions on how to access the family of the HPE GreenLake API for Data Services on HPE GreenLake edge-to-cloud platform shown in this <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003533en_us&#x26;page=ps_api_dscc.html">link</a>. The help menu inside the HPE GreenLake for those data services as of March 2024 was also updated as shown below.</p> <p><img src="/img/the-link-to-the-documentation-of-all-hpe-gl-apis-for-data-services-on-hpegl.png" alt="HPE data services help menu after the update"></p> <p><em>The above figure shows the link that was added to points to the collection of the HPE GreenLake APIs for Data Services on HPE GreenLake platform which points further to the Help Menu for using the API.</em></p> <p><img src="/img/landing-page-that-describes-all-of-the-rest-api-for-data-services-on-hpegl.png" alt="Using the API links to the GL API for the family of data services"></p> <p><em>The above figure shows the links to each set of HPE GreenLake APIs part of the family of Data Services on the HPE GreenLake platform.</em></p> <h3>The future for Data Services APIs</h3> <p>In the future, there will be some more sets of HPE GreenLake APIs related to the additional Data Services published at the HPE GreenLake Developer Website that conform with OpenAPI 3.1 specification. Furthermore, the Data Services Cloud Console API will eventually be published to conform to the OpenAPI 3.1 specifications just as the rest of the sets of OpenAPI specs. To accommodate powerful and comprehensive DataOps use cases such as presented in this blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-virtualization/">post</a> and another blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-backup-and-recovery/">post</a>, the combination of multiple HPE GreenLake APIs from different sets of Data Services on HPE GreenLake platform has been used.</p> <h3>Learn more about the Data Services APIs</h3> <p>For getting started with the Data Services APIs and improve your agility in managing your data, please look at the following blog posts, Data Services documentation, and the video of the Developer Forum webinar in the HPE YouTube channel.</p> <ol> <li>Blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services/">post</a>: Getting Started with HPE GreenLake APIs for Data Services.</li> <li>Blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-virtualization/">post</a>: Getting Started with HPE GreenLake APIs for Virtualization.</li> <li>Blog <a href="https://developer.hpe.com/blog/getting-started-with-private-cloud-business-edition-apis/">post</a>: Getting Started with HPE GreenLake APIs for HPE GreenLake Private Cloud Business Edition.</li> <li>The <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003533en_us&#x26;page=ps_api_dscc.html">link</a> to the documentation of the HPE GreenLake APIs at HPE GreenLake help menu.</li> <li>Watch the replay of the Meetup webinar “<strong>The data services family of APIs for HPE GreenLake – Putting it all together</strong>”</li> </ol> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/-ffzhc6TzoA?si=fzg4jiakx_L2GxZB" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <p><br /> <br /></p> <h3>Any questions on HPE GreenLake APIs for Data Services on HPE GreenLake platform?</h3> <p>Join the HPE Developer Community Slack <a href="https://developer.hpe.com/slack-signup/">Workspace</a>, and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services</a> Slack channel.</p>https://developer.hpe.com/hpe-greenlake-cloud-platform/home/https://developer.hpe.com/hpe-greenlake-cloud-platform/home/<p>The HPE GreenLake platform, along with its marketplace and partner ecosystem, provides customers with a consistent, unified foundation for application development and hybrid cloud innovation. This helps customers reduce costs and complexity, while enhancing stability and performance across multiple IT environments.</p> <p><a href="https://www.hpe.com/us/en/greenlake.html">Learn more about HPE GreenLake</a></p> <h2>APIs and Documentation</h2> <p>HPE GreenLake customers and partners can take advantage of our well-documented, secure, and scalable framework of APIs for HPE GreenLake found in our developer portal. Learn more about the unified HPE GreenLake experience for developers by visiting the <a href="https://developer.greenlake.hpe.com">HPE GreenLake Developer Portal</a>.</p> <h2>Demos and resources</h2> <p>Get an overview of how you can manage workloads with the HPE GreenLake platform using our <a href="https://www.hpe.com/us/en/greenlake/demos.html">quick demos</a>.</p> <h2>Test Drive for developers</h2> <p><a href="https://auth.hpe.com/hpe/cf/registration">Register for an HPE account</a> and check what is available for developers in our <a href="https://testdrive.greenlake.hpe.com/">Test Drive</a> platform.</p> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. HPE GreenLake workshops are available today.</p> <br/> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <br/> <h2>Any questions on HPE GreenLake?</h2> <p>Join the <a href="https://slack.hpedev.io/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#HPEGreenLake</a> channel.</p>https://developer.hpe.com/hpe-greenlake-for-compute-ops-management/home/https://developer.hpe.com/hpe-greenlake-for-compute-ops-management/home/<p>Compute powers business today, and organizations need compute resources to be available in more environments than ever before -- at the edge, close to where data is generated and consumed, and in the cloud where they can provide greater agility.</p> <p>Every business is facing tough challenges – to digitally transform, address security risks, and operate efficiently. Complex server management is a distraction from these challenges, consuming IT resources and slowing innovation. HPE Compute Ops Management solves these challenges by simplifying and unifying operations across the server lifecycle, for the whole environment, no matter where your compute infrastructure lives. The service provides a consistent, secure cloud experience that scales elastically and unifies compute management.</p> <p>With HPE Compute Ops Management, you can:</p> <ul> <li><strong>Unify compute management</strong></li> </ul> <p>Streamline your compute management operations using a seamless as-a-service single console experience from edge-to-cloud with self-service and real-time access to servers.</p> <ul> <li><strong>Simplify and automate</strong><br> Simplify and bring agility to your compute lifecycle management to lower your Total Cost of Ownership (TCO).</li> <li><strong>Secure cloud operations</strong><br> Securely control your distributed compute lifecycle tasks using a cloud-native architecture to manage and monitor your servers seamlessly.</li> </ul> <h3><a href="https://www.hpe.com/us/en/compute/management-software.html">Learn more about HPE Compute Ops Management</a></h3> <h2>APIs and documentation</h2> <p>HPE Compute Ops Management customers and partners can take advantage of our well-documented, secure, and scalable APIs found on the HPE GreenLake developer portal.</p> <h3><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/">Check out the HPE Compute Ops Management APIs on the HPE GreenLake Developer Portal</a></h3> <h3><a href="https://www.hpe.com/info/com-gsg">Reference the HPE Compute Ops Management Getting Started Guide for configuration and usage guidance</a></h3> <h2>Any questions on HPE Compute Ops Management?</h2> <p>Please join the <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C03QTQWC213"></a><a href="https://hpedev.slack.com/archives/C03QTQWC213">#hpe-compute-ops-management</a> channel.</p>https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise/home/https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise/home/<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> reimagines the private cloud experience with a scalable, pay-per-use, enterprise-grade solution delivered to you as a managed service across your locations — from edge to cloud. Built for both cloud-native and traditional applications, it supports the self‑service deployment of container, bare metal, and virtual machine services.</p> <p>Its design principles are centered on leveraging open standards and open systems, preventing vendor lock-in with the ability to place your workloads in the environment of your choice based on cost and performance. You also get the full advantage of modern DevOps and automation with infrastructure-as-code (IaC) configuration management, REST APIs, and cloud command shell, for streamlined infrastructure provisioning and integration with existing DevOps/CI toolchains—speeding deployments for cloud admins and developers alike.</p> <h2>HPE Private Cloud Enterprise services</h2> <h3>Containers</h3> <p>Provision Kubernetes clusters on-demand to deploy and scale containerized applications and cloud-native workloads. The default container service is built on the HPE Ezmeral Runtime Environment and based on CNCF-compliant Kubernetes. You can create container clusters using VMs and/or bare-metal compute instances to meet a range of performance requirements.</p> <p>HPE Private Cloud Enterprise also supports select third-party container platforms such as Amazon Elastic Kubernetes (EKS), so you can leverage the same container runtimes in your public and private cloud, streamlining workload portability and providing a consistent cloud-native experience across hybrid clouds for cloud or tenant admins (cloud system admins, DevOps admins, and more) and cloud consumers (developers, project contributors, and such).</p> <h3>Bare Metal</h3> <p>Provision bare-metal instances on-demand to support workloads that require the performance of a dedicated physical server. You can organize bare-metal instances into compute groups; define a unique set of resources for each group (compute instances, storage volumes, VLAN segments, IP pools, SSH keys, and more). Easily bring your sanctioned, hardened operating system images or your own virtualization or container technology stacks to meet your corporate IT standards and versioning policies.</p> <h3>Virtual machines</h3> <p>Provision virtual machines on-demand to support traditional virtualized workloads. The service supports the popular VMware ESXi™ hypervisor. Choose from a variety of virtual machine instances with different compute instance types and sizes to meet the price-performance requirements of different workloads. In addition, HPE supports predefined VM instance families aligned to compute modules with various memory, storage, and CPU characteristics. You can also define and size your VM instances to address the specific requirements of any application. HPE Private Cloud Enterprise offers integrations with popular configuration management platforms, including Ansible, Ansible Tower, Chef, and Puppet.</p> <p>To learn, more review the <a href="https://www.hpe.com/psnow/doc/a50007892enw">HPE Private Cloud Enterprise technical white paper</a>.</p> <h2>Developer experience</h2> <p>HPE Private Cloud Enterprise includes powerful DevOps tools that streamline and automate repetitive tasks with repeatable workflows, super-charging productivity and increasing agility for faster, more consistent code collaboration.</p> <ul> <li>Fast self-service access to leverage the platform and its resources</li> <li>Reduce manual DevOps tasks with automation</li> <li>Integrate with your familiar DevOps toolchains (currently GitHub)</li> <li>Easily monitor and manage the status of projects</li> </ul> <p><img src="/img/hpegl4pce-architecture-v2.png" alt="HPE Private Cloud Enterprise architecture" title="HPE Private Cloud Enterprise architecture"></p> <h3>Infrastructure as code (IaC)</h3> <p>Leverage IaC functionality offered by the HPE GreenLake Terraform Provider to provision and manage your private cloud resources. The Terraform Registry resources can be <a href="https://registry.terraform.io/providers/HPE/hpegl/latest/docs">downloaded from here</a>.</p> <p>Get some real, hands-on experience by registering for this workshops-on-demand: <a href="https://developer.hpe.com/hackshack/workshop/36">Introduction to Virtual Machine Infrastructure-as-Code using Terraform and HPE Private Cloud Enterprise</a></p> <h3>REST APIs</h3> <p>HPE GreenLake APIs provide a secure, standardized interface for automating common and repetitive tasks, programmatically configuring and provisioning services, and integrating applications with cloud services. The APIs are based on the OpenAPI specification and governed by administratively defined RBACs and strong authentication methods.</p> <h3>Cloud shell</h3> <p>Cloud shell as an interactive browser-based shell enables secure access to your HPE Private Cloud Enterprise resources. Development packages, orchestration tools and the latest IaC libraries are pre-installed as part of the cloud shell.</p> <ul> <li> <p>Access pre-loaded with latest libraries and utilities with in-built authentication</p> <ul> <li>Pre-packaged orchestration &#x26; IaC tools and templates (HPE GreenLake specific resources and tooling)</li> <li>Development packages such as Terraform, Git, Docker / Docker Compose/ Docker CLI, Golang (latest), Python 3.x</li> </ul> </li> <li> <p>Clone GitHub repo in cloud shell, debug and deploy applications into HPE Private Cloud Enterprise resources</p> </li> <li> <p>Create and manage VM and container resources via IaC using HPE GreenLake Terraform modules</p> </li> <li> <p>Each cloud shell instance backed by 1GB of persistent storage provisioned for User's $HOME directory</p> </li> </ul> <p><img src="/img/hpegl4pce-cloud-shell.png" alt="HPE Private Cloud Enterprise cloud shell" title="HPE Private Cloud Enterprise cloud shell"></p> <h3>DevOps</h3> <p>Cloud or tenant admins and cloud consumers can create DevOps projects—workspaces where authorized administrators and contributors can configure external connections such as GitHub, create and manage automated CD pipelines for external accounts, and associate Kubernetes container clusters with the deployment pipelines.</p> <p><img src="/img/hpegl4pce-devops.png" alt="HPE Private Cloud Enterprise DevOps CI/CD pipeline" title="HPE Private Cloud Enterprise DevOps CI/CD pipeline"></p> <p>For more information on the HPE GreenLake DevOps capabilities, please watch this video:</p> <p><strong>Create a CD pipeline with a release trigger with HPE Private Cloud Enterprise</strong></p> <p><a href="https://www.youtube.com/watch?v=i95-FO0bvgg"><img src="https://img.youtube.com/vi/i95-FO0bvgg/hqdefault.jpg" alt="Create a CD pipeline"></a></p> <h2>Resources</h2> <p><a href="https://www.hpe.com/psnow/doc/a50007892enw">Technical white paper</a></p> <p><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=index.html">Support documentation</a></p> <hr> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter-Notebook based workshops available in the HPE Developer <a href="https://developer.hpe.com/hackshack/">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. <a href="https://developer.hpe.com/hackshack/workshop/36">Introduction to Virtual Machine Infrastructure-as-Code using Terraform and HPE Private Cloud Enterprise</a> is available today.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div>https://developer.hpe.com/hpe-greenlake-platform/home/https://developer.hpe.com/hpe-greenlake-platform/home/<style> ul li{ font-size:27px; margin-top: 10px; } ul li:first-child { margin-top:0; } </style> <p>HPE GreenLake cloud is a modular hybrid cloud platform that enables enterprises to unlock insights and innovate faster by deploying a single cloud operating model everywhere.</p> <p><a href="https://www.hpe.com/us/en/greenlake.html">Learn more about HPE GreenLake</a></p> <h2>HPE GreenLake cloud APIs and documentation</h2> <p>HPE GreenLake customers and partners can take advantage of our well-documented, secure, and scalable framework of APIs for HPE GreenLake found in the <a href="https://developer.greenlake.hpe.com">HPE GreenLake Developer Portal</a>, covering both the platform and the cloud services that run on it.</p> <p>The HPE GreenLake cloud APIs:</p> <ul> <li>Conform to the OpenAPI 3.x specification. This makes them easy to learn, discoverable by code, and accessible with any programming language.</li> <li>Use a single endpoint. All calls go to a single unified API global domain endpoint: <em>ht<span>tps://</span>global.api.greenlake.hpe.com</em></li> <li>Use a single OAuth token to access all APIs.</li> <li>Are built to be secure and highly available.</li> <li>Are RESTful for flexible implementation.</li> </ul> <p>We provide a <a href="https://developer.greenlake.hpe.com">dedicated developer portal</a> to support you with documentation, code, community, and more.</p> <p>For a full list of HPE GreenLake APIs please visit <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">the HPE GreenLake API catalog</a>. Some of the APIs currently available are listed below:</p> <h3>Workspace management</h3> <p>Learn about the details of a workspace, and discover how to fully create, read, update, and delete (CRUD) on managed service provider (MSP) tenant workspaces. You can also access a full CRUD roster of users, enabling you to send invitations to join a workspace.</p> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/workspace/public/">Visit Workspace management API Reference</a></p> </blockquote> </blockquote> <h3>Authorization</h3> <p>Familiarize yourself with the details about a permissions role, or full CRUD on user assignments to roles.</p> <h3>Devices and subscriptions management</h3> <p>Add and modify hardware devices in a workspace, initiate subscriptions, and assign them to devices.</p> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/device-management/public">Visit device management API Reference</a></p> </blockquote> </blockquote> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/subscription-management/public">Visit subscription management API Reference</a></p> </blockquote> </blockquote> <h3>Locations</h3> <p>Create, read, and update physical locations used for shipping and support.</p> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/location-management/public">Visit locations management API Reference</a></p> </blockquote> </blockquote> <h3>Audit Log</h3> <p>Read details about user actions for accountability.</p> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/audit-logs/public/">Visit Audit log API reference</a></p> </blockquote> </blockquote> <h3>Wellness</h3> <p>Get a list of wellness events via searches and filters, organize them with tags, read event details, pull KPIs, and open support cases.</p> <blockquote> <blockquote> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/wellness/public/">Visit Wellness API reference</a></p> </blockquote> </blockquote> <p>Learn more about the unified HPE GreenLake cloud experience for developers by visiting the <a href="https://developer.greenlake.hpe.com">HPE GreenLake Developer Portal</a>.</p> <h2>Demos and resources</h2> <p>Get an overview of how you can manage workloads with HPE GreenLake cloud using our <a href="https://hpe.com/greenlake/demos">interactive demos</a>.</p> <h2>Workshops-on-Demand</h2> <p>Take advantage of our free, Jupyter Notebook-based Workshops-on-Demand available in the <a href="https://developer.hpe.com/hackshack/workshops/?activeTab=4">Hack Shack</a>. These technical workshops provide you with an in-depth, hands-on learning experience where you can interact with and learn from the experts. Designed to fit your schedule, these workshops are available 24/7 – any time, from anywhere. HPE GreenLake workshops are available today.</p> <br/> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="/hackshack/workshops"><button type="button" class="button">Try now!</button></a> </div> <br/> <h2>Any questions on HPE GreenLake APIs?</h2> <p>Join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> Slack channel.</p>https://developer.hpe.com/hybrid-observability-flex-solutions/home/https://developer.hpe.com/hybrid-observability-flex-solutions/home/<style> .action-button { display: inline-block; background-color: #01a982; color: white; padding: 12px 24px; border-radius: 5px; text-decoration: none; font-size: 16px; font-weight: bold; } .action-button:hover { background-color: #005a3c; } #flex-container { display: flex; } #flex-content { flex: 1; } .video-box { flex: 1; max-width: 40%; min-width: 200px; text-align: center; } .video-thumbnail { width: 80%; height: 200px; border-radius: 10px; box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2); } #center-align { text-align: center; } #button-icon { height: 20px; width: auto; } li { font-size: 20px; line-height: 28px; margin-bottom: 10px; } #action-link { background-color: #01a982; color: white; padding: 10px 20px; border-radius: 20px; text-decoration: none; display: inline-flex; align-items: center; gap: 10px; } #section-header, #learn-from-experts, #additional-resources, #contact { color: #01a982; font-size: 28px; margin-top: 30px; border-bottom: 3px solid #01a982; padding-bottom: 10px; } #content-list { list-style-type: decimal; padding-left: 20px; font-size: 18px; line-height: 1.8; } .video-title { display: block; margin: 0; font-size: 20px; font-weight: bold; } li { font-size: 20px; } #content-table { border-collapse: collapse; width: 100%; border: 1px solid black; } #table-header-cell { border: 1px solid black !important; padding: 8px !important; text-align: center !important; font-weight: bold !important; } #table-cell { border: 1px solid black; padding: 8px; text-align: center; } #section-title { border: 1px solid black; padding: 8px; text-align: center; font-weight: bold; background-color: #f1f1f1; } .video-title { display: block; margin: 0; font-size: 20px; font-weight: bold; } </style> <h1 id='overview'>Activation & Onboarding of Hybrid Observability in HPE GreenLake Flex Solutions</h1> <p style="font-size: 22px;"> Your HPE GreenLake Flex Solution gives you full access to the tools and intelligence inherent within HPE GreenLake cloud – bringing the cloud experience to all your IT so you can deploy and manage resources across your private and public clouds while retaining control of your data and flexibility over how you consume and manage your services. </p> <div id="flex-container"> <div id="flex-content"> <p style="font-size: 22px;"> One outstanding feature that comes standard with HPE GreenLake Flex Solutions is hybrid observability – for monitoring the compute, storage, and networking resources within your Flex solution. </p> <p style="font-size: 22px;"><a href="/greenlake/hybrid-observability-flex-solutions/why-greenLake-flex-solutions" target="_blank" style="font-size: 22px;">Hybrid observability</a> is an AI-powered SaaS solution – powered by <a href="https://www.hpe.com/us/en/opsramp.html" target="_blank" style="font-size: 22px;"> HPE OpsRamp Software</a> – that provides your IT operations teams full visibility and control to optimize uptime and infrastructure operating costs while improving process efficiencies. </p> </div> <div class="video-box"> <a href="https://www.youtube.com/watch?v=3Jp4MbsNydM" target="_blank"> <img src="/img/stepsflex/main_video_img.png" alt="HPE GreenLake Flex Solutions Video" class="video-thumbnail"> </a> </div> </div> <h3 id="learn-from-experts">Getting Started</h3> <p style="font-size: 22px;">Direct your IT operations teams to the activation and onboarding resources on this page to start taking full advantage today. You’ll quickly recognize the value of hybrid observability from HPE OpsRamp and want to expand its functionality throughout your hybrid environment – including virtual machines, containers and workloads running on any manufacturer’s hardware.</p> <p style="font-size: 25px;">Choose between HPE-assisted or self-service activation and onboarding:</p> <h4 id='hpe-assisted'>HPE-Assisted Activation and Onboarding</h4> <p style="font-size: 24px;">Your HPE Account Service Manager (ASM) and a Global Support Remote specialist (GSR) are standing by to help guide your activation and onboarding process.</p> <div id="content-container" style="display: flex; align-items: flex-start; gap: 20px;"> <div id="text-content" style="flex: 2;"> <p style="font-size: 22px;">After your hardware has been installed and configured in your environment, engage your HPE Account Service Manager (ASM) to guide you through the following steps:</p> <ol id="content-list"> <li><a href="https://forms.office.com/pages/responsepage.aspx?id=YSBbEGm2MUuSrCTTBNGV3MNwzZIB3zBHp5x5QZ1HakFUOE9NMFdWU0RYRjIzSkRDVzFCVUgzVTM1My4u" target="_blank">Complete the quick setup questionnaire</a> that your ASM will share with you via email.</li> <li><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&page=GUID-C335D790-5166-406F-B0B8-C93AB46A2C76.html#ariaid-title1" target="_blank">Create a workspace</a> within HPE GreenLake cloud. After creating your first workspace, you’ll be taken to your HPE GreenLake cloud page where you’ll see featured services, like hybrid observability powered by HPE OpsRamp.</li> <li>Invite additional users from your IT operations team by using the Identity and Access Management (IAM) tool.</li> <li><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&page=GUID-9EDAAB42-9182-488D-A06F-6E8CB4BFAB60.html" target="_blank">Provision OpsRamp</a> in your HPE GreenLake cloud.</li> <li><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&page=GUID-7CDEB995-EAAB-4A2B-BC1F-2EB6F20B594B.html%23ariaid-title1" target="_blank">Specify the region</a> where you’d like your OpsRamp SaaS instance deployed.</li> <li><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&page=GUID-46819916-2559-4A35-B10A-42E982026A6C.html" target="_blank">Launch OpsRamp using the subscription key</a> received within your welcome letter or from your ASM.</li> <li>A GSR will then be assigned to provide guidance through the onboarding and customization of your hybrid observability command center. This may include assistance with setting up gateways, auto-discovery and dashboarding, customizing monitoring templates and reports, as well as setting up patching, scripting and alert automation.</li> </ol> </div> <div id="image-content" style="flex: 1; text-align: right;"> <img src="/img/stepsflex/assisted.png" alt="Activation and Onboarding Steps" style="max-width: 100%; height: auto; border-radius: 8px;"> </div> </div> <h4 id='self-service-activation'>Self-service Activation and Onboarding</h4> <p style="font-size: 24px;">Resources for onboarding, activating, and configuring the hybrid observability service</p> <table id="content-table"> <thead> <tr> <td id="table-header-cell">Video Tutorials</td> <td id="table-header-cell">Blog Tutorials</td> </tr> </thead> <tbody> <tr id="provisioning-activation"> <td colspan="2" id="section-title"> Provisioning and activation </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=HFRDexDyr88&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 1 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 1: Provisioning the hybrid observability service in your workspace</span> </td> <td id="table-cell" rowspan="4"> <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-1-provisioning-and-activation-in-hpe-greenlake-flex-solutions/"> Hybrid observability service – Provisioning and activation in HPE GreenLake Flex Solutions </a> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=7lBc9_2TC3s&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 2 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 2: Activate hybrid observability subscription key in your workspace</span> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=oinaiu1zeBo&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 3 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 3: Assign access roles for desired users</span> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=vviJ_2B_C_o&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 4 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 4: Launching the hybrid observability service from your workspace</span> </td> </tr> <tr id="client-settings-gateway-configs"> <td colspan="2" id="section-title"> Configuring client settings and gateways </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=4jYTVIbUAU4&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 5 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 5: Preparing the client/user settings</span> </td> <td id="table-cell" rowspan="2"> <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/"> Hybrid observability service – Initial configuration to enable the discovery of resources in HPE GreenLake Flex Solutions </a> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=AtQCNt67SIA&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 6 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 6: Configuring a Gateway</span> </td> </tr> <tr id="monitoring-dashboards"> <td colspan="2" id="section-title"> Monitoring, integration, and dashboards </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=JtiWrHUqzdk&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 7 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 7: Agentless SSH integration and monitoring templates</span> </td> <td id="table-cell"> <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/"> Hybrid observability service – Enabling the monitoring of agentless SSH-enabled system in HPE GreenLake Flex Solutions </a> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=7E1-QvFdeE4&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 8 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 8: Redfish Server integration and configuration</span> </td> <td id="table-cell" rowspan="3"> <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/"> Hybrid observability service – Enabling the monitoring of physical devices in HPE GreenLake Flex Solutions </a> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=JpsGVEG_eVA&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 9 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 9: Storage array integration and configuration</span> </td> </tr> <tr> <td id="table-cell"> <a href="https://www.youtube.com/watch?v=6ahkkdmGsi8&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY"> <img src="/img/stepsflex/video_img.jpg" alt="Step 10 Image" style="width: 120px; height: 75px; margin: 0;"> </a> <span class="video-title">Part 10: Creating a customized dashboard</span> </td> </tr> </tbody> </table> <h3 id="additional-resources">Additional Resources</h3> <ul> <li><a href="https://www.hpe.com/us/en/collaterals/collateral.a50010160enw.html">Hybrid observability in HPE GreenLake Flex Solutions Service Description</a></li> <li><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us">HPE GreenLake Flex Solutions User Guide</a></li> <li><a href="https://www.hpe.com/psnow/doc/a50010620enw?jumpid=in_pdfviewer-psnow">HPE GreenLake Flex Solutions Briefcase</a></li> <li><a href="https://developer.hpe.com/platform/hpe-opsramp/home/">HPE OpsRamp APIs and Integration Capabilities</a></li> <li><a href="https://glp.docs.opsramp.com/">HPE OpsRamp documentation for Hybrid Observability in HPE GreenLake Flex Solutions</a></li> <li> <strong>Webinar replays</strong> <ul> <li><a href="https://www.youtube.com/watch?v=qH93yD5KSL8&list=PLtS6YX0YOX4f5TyRI7jUdjm7D9H4laNlF">No More Blind Spots: How eBPF Transforms Observability</a></li> <li><a href="https://www.youtube.com/watch?v=VnKSnf7G4-4&list=PLtS6YX0YOX4f5TyRI7jUdjm7D9H4laNlF">From log files to AI insights: The 60-year evolution of observability and AIOps</a></li> <li><a href="https://www.youtube.com/watch?v=nDa_NQPbbVY&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">DayN+ : A new way to look at observability</a></li> </ul> </li> </ul> <h3 id="contact">Any Questions About Hybrid Observability in HPE GreenLake Flex Solutions?</h3> <p style="font-size: 22px;">Need help getting started with the bundled hybrid observability in HPE GreenLake Flex Solutions? Join our <a href="https://developer.hpe.com/slack-signup/" style="font-size: 22px;">Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C08K4GV7YN5" style="font-size: 22px;">#hpe-greenlake-flex-observability</a> Slack channel. You can also email us at <a href="mailto:gsccitops@hpe.com" style="font-size: 22px;">gsccitops@hpe.com</a>.</p>https://developer.hpe.com/hybrid-observability-flex-solutions/why-greenLake-flex-solutions/https://developer.hpe.com/hybrid-observability-flex-solutions/why-greenLake-flex-solutions/<h1 id ="overview">Why Hybrid observability?</h1> <p style="font-size: 22px;"> IT operations teams need hybrid observability to gain unified visibility and insights across complex, distributed IT infrastructure environments. Hybrid observability enables faster issue detection and remediation, improved performance, and more efficient management. Hybrid observability within your HPE GreenLake Flex solutions is a powerful added-value service, delivered via an AI-powered SaaS-based application, to help your IT operations teams proactively ensure system performance and optimize infrastructure operating costs while improving process efficiencies. </p> <h3 id ="1-optimize-costs-and-utilization">1. Optimize Costs and Utilization?</h3> <div style="display: flex; align-items: flex-start; gap: 20px;"> <div style="font-size: 22px; line-height: 32px;"> Optimize resource utilization and forecast future requirements for more accurate budgeting, and drive better cost efficiencies and predictability. <ul> <li>Capacity and usage analytics</li> <li>Customizable dashboards and reporting</li> <li>Asset lifecycle management</li> </ul> </div> <img src="/img/stepsflex/capacity_insights.png" alt="Capacity Insights" style="width: 500px; height: auto;"> </div> <h3 id="2-assure-uptime-and-performance">2. Assure Uptime and Performance</h3> <div style="display: flex; align-items: flex-start; gap: 20px;"> <div style="font-size: 22px; line-height: 32px;"> Rapidly identify issues and the potential impact on application experience. Move from reactive to proactive to quickly resolve incidents before they impact business performance. <ul> <li>Full-stack observability</li> <li>Health and performance data</li> <li>Service dependency mapping</li> <li>Predictive analytics on trending health data with proactive alerting</li> <li>Policy-based resolution workflows</li> </ul> </div> <img src="/img/stepsflex/full_stack_observability.png" alt="Uptime and Performance" style="width: 500px; height: auto;"> </div> <h3 id="3-streamline-it-operations-and-service-delivery">3. Streamline IT Operations and Service Delivery</h3> <div style="display: flex; align-items: flex-start; gap: 20px;"> <div style="font-size: 22px; line-height: 32px;"> Policy-based automation of routine operational tasks leveraging AI and ML analytics. Focus your IT team on innovation and business-critical needs. <ul> <li>Unified visibility across hybrid estate</li> <li>Intuitive process automation</li> <li>Event and incident management</li> <li>Bi-directional integration with Service Desk</li> </ul> </div> <img src="/img/stepsflex/AI_Insights.png" alt="IT Operations and Service Delivery" style="width: 500px; height: auto;"> </div> <h3 id="4-simplify-compliance-and-governance">4. Simplify Compliance and Governance</h3> <div style="display: flex; align-items: flex-start; gap: 20px;"> <div style="font-size: 22px; line-height: 32px;"> Mitigate risk and safeguard sensitive data by automating policy enforcement with compliance and regulatory requirements. <ul> <li>Prescribed monitoring templates</li> <li>Automated OS upgrades and patching</li> <li>Role-based consoles</li> <li>Audit trails and session recordings</li> </ul> </div> <img src="/img/stepsflex/automated_patching.png" alt="Compliance and Governance" style="width: 500px; height: auto;"> </div>https://developer.hpe.com/EzmeralSoftwareForum/https://developer.hpe.com/EzmeralSoftwareForum/<p>...</p>https://developer.hpe.com/OSScontribute/https://developer.hpe.com/OSScontribute/<p>.</p>https://developer.hpe.com/Slack/https://developer.hpe.com/Slack/https://developer.hpe.com/Twitter/https://developer.hpe.com/Twitter/<p>Text</p>https://developer.hpe.com/contribute/https://developer.hpe.com/contribute/<p>.</p>https://developer.hpe.com/newsletter/https://developer.hpe.com/newsletter/<p>...</p>https://developer.hpe.com/openSource/https://developer.hpe.com/openSource/<p>...</p>https://developer.hpe.com/devtalks/https://developer.hpe.com/devtalks/<p>DevTalks are developer/data scientist focused 1-hour sessions delivered by the HPE Developer Community every other week on Wednesdays, at 5 PM CET.</p> <p>Please note that these talks are internal/partners only for now.</p> <table> <thead> <tr> <th>    Date       </th> <th>Title</th> <th>Speaker(s)</th> <th>Link         </th> </tr> </thead> <tbody> <tr> <td>09-Dec-20</td> <td>What’s a data fabric and how does it work?</td> <td>Ted Dunning</td> <td><a href="https://youtu.be/qi6sTvu8osk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>06-Jan-21</td> <td>Managing HPE Cloud Volumes with API</td> <td>Yann Allandit</td> <td><a href="https://youtu.be/aReR7DF1iIY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>20-Jan-21</td> <td>Building a dynamic Machine Learning pipeline with KubeDirector</td> <td>Denis Choukroun</td> <td><a href="https://youtu.be/AO0x7pxEw98&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>03-Feb-21</td> <td>Advanced PowerShell scripting with HPE OneView PowerShell library</td> <td>Dung Hoang Khac</td> <td><a href="https://youtu.be/qp_gmOj5OX0&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>17-Feb-21</td> <td>How to start developing NLP applications</td> <td>Rohit Rawat &#x26; Vineet Gundecha</td> <td><a href="https://vimeo.com/514054456/fc11ffd8cf">Replay</a></td> </tr> <tr> <td>03-Mar-21</td> <td>Delivering the AI Stack for the Modern Enterprise</td> <td>Vinod Iyengar (H2O.ai) &#x26; Ellen Friedman</td> <td><a href="https://youtu.be/lLxy03I3qrE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>31-Mar-21</td> <td>Industrialising and transforming Data Science scope and scale</td> <td>Doug Cackett</td> <td><a href="https://vimeo.com/532641045/d498467501">Replay</a></td> </tr> <tr> <td>14-Apr-21</td> <td>Delivering hands-on experience with Jupyter Notebooks</td> <td>Frédéric Passeron</td> <td><a href="https://youtu.be/lAlNNUkuPc8?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>28-Apr-21</td> <td>Kubernetes 101</td> <td>Didier Lalli / Nigel Poulton</td> <td><a href="https://youtu.be/PWVJKK1obKQ?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>12-May-21</td> <td>Redfish programming made easy and secure with Ansible and HPE OneView</td> <td>François Donzé</td> <td><a href="https://youtu.be/HEU29-Nf4J8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-May-21</td> <td>Simplify GPU Sharing and Utilization with Run:AI on HPE Ezmeral</td> <td>Omri Geller, Run:AI</td> <td><a href="https://youtu.be/3KMdV0CcvRE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>09-Jun-21</td> <td>Deploying end-to-end machine learning workflows​ with HPE Ezmeral ML Ops</td> <td>Denis Choukroun</td> <td><a href="https://youtu.be/MoqQTvwH0p8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>07-Jul-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/7/DevTalk-SmartSim-jul-7-2021.pdf">SmartSim – Bringing Simulation and AI Together at Scale</a></td> <td>Samuel Partee, Matt Ellis</td> <td><a href="https://youtu.be/-K5yFCP6wTg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>21-Jul-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/7/DevTalk32-TrilioVault-21-july-2021.pdf">Recovering and Porting Cloud-Native Applications in the Fast-Paced DevOps World</a></td> <td>Prashanto Kochavara, Trilio</td> <td><a href="https://youtu.be/wVnDk-JMMwI&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>04-Aug-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/8/DevTalk33-Faster-apps-and-app-delivery-with-GraphQL.pdf">Faster apps and app delivery with GraphQL</a></td> <td>Phil Prasek / Michael Watson, apollographql.com</td> <td><a href="https://youtu.be/koIPQUK-i6E&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>18-Aug-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/8/Sysdig-and-HPE-final.pdf">Security and Observability for HPE Ezmeral with Sysdig</a></td> <td>Alex Lawrence, Sysdig​</td> <td><a href="https://youtu.be/nMvrgeQRkEw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>15-Sep-21</td> <td>Learn about the upcoming HPE GreenLake Cloud Services Developer Portal</td> <td>Travis Tripp, Pulkit Sindhwani</td> <td><a href="https://psnow.ext.hpe.com/asset?id=e800d2f9-1e6c-4fd7-aef3-b76c70ca194e&#x26;preview=true">Replay</a></td> </tr> <tr> <td>29-Sep-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE_SmartIO.pdf">Discover hardware-based networking and security acceleration with HPE Pensando SmartNIC</a></td> <td>Olivier Vallois</td> <td><a href="https://www.youtube.com/watch?v=j45I21Chakg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>13-Oct-21</td> <td>Using the Aruba CX pyaoscx Python SDK</td> <td>Alvin Castro</td> <td><a href="https://youtu.be/vB3DEpRDLOk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>10-Nov-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/11/gopaddle-introduction-HPEDev-Final-10-Nov-2021.pdf">No-Code Acceleration on HPE Ezmeral</a></td> <td>Vinothini Raju, &#x26; Ashvin Kumar, gopaddle</td> <td><a href="https://youtu.be/2A195hVR5Qk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> </tbody> </table>https://developer.hpe.com/get-real-with-ai-jam-series/https://developer.hpe.com/get-real-with-ai-jam-series/<p>AI is now revolutionizing how enterprises create business value from data. New technical capabilities are disrupting infrastructure configurations, business processes and organizational structures. This now enabling companies to harness the power and value of their data more efficiently, leading to enhanced decision-making, improved customer experiences, and the creation of new business models.</p> <p>This new interactive series is focused on “keeping it real with AI” and will provide you with insights on how to make AI a functional reality in your organization. Mark your calendar to engage with HPE’s leading innovators, developers and architects as they share what they have learned traversing industry hype, while transforming AI into technical and business value.</p> <style> table { display: block; width: 100%; width: max-content; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <table> <thead> <tr> <th>     Date     </th> <th>Title</th> <th>Speaker(s)</th> <th>Link</th> </tr> </thead> <tbody> <tr> <td>06-Aug-25</td> <td>Sustainably Scaling AI Adoption</td> <td>Audrey Scribner / Anthony Delli Colli / Jason Zeiler / Lin Nease / Karim Abou Zahab</td> <td><a href="https://www.youtube.com/watch?v=AmL_k-I6dSY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>14-May-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/%5B%23007%5D+AI+Jam+Show+-+Lessons+in+AI+for+Retailers.pdf">Lessons in AI for retailers: Transforming the shopping experience</a></td> <td>Audrey Scribner / Anthony Delli Colli / Karen Rhodes / Reginald Height</td> <td><a href="https://www.youtube.com/watch?v=_8gnUunB7bM&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>23-Apr-25</td> <td>Developing and deploying AI in the enterprise</td> <td>Audrey Scribner / Anthony Delli Colli / Jimmy Whitaker / Derek Bekebrede / Alejandro Martinez</td> <td><a href="https://www.youtube.com/watch?v=Dau7swlAkJY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>12-Feb-25</td> <td>AI Inferencing at the Edge: Use cases from Earth to Space</td> <td>Audrey Scribner / Michael Greene / Mark Fernandez.</td> <td><a href="https://www.youtube.com/watch?v=9QxCAaAKN3c&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>22-Jan-25</td> <td>IT New Year’s resolution: Build an ethical and trustworthy AI system</td> <td>Audrey Scribner / Pam Wood / Sahand Ghorbanpour / Jeff Oxenberg</td> <td><a href="https://www.youtube.com/watch?v=S9SfJIUgIx4&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>06-Nov-24</td> <td>Implementing your AI breakthroughs effectively - The infrastructure to your AI</td> <td>Audrey Scribner / Anthony Delli Colli / Shep Bostin / Tyler Britten</td> <td><a href="https://www.youtube.com/watch?v=7pGOYlA5eyI&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>02-Oct-24</td> <td>Cool AI Use Case, Now What?</td> <td>Audrey Scribner / Hayden Barnes / Chad Smykay / Anthony Delli Colli</td> <td><a href="https://www.youtube.com/watch?v=gxpcBISePhE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>21-Aug-24</td> <td>Exploring Transformative AI Use Cases Across Industries</td> <td>Audrey Scribner / Anthony Delli Colli / Mark Perreira / Peter Moser / Travis Tripp</td> <td><a href="https://www.youtube.com/watch?v=XEJqcdWj790&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> </tbody> </table>https://developer.hpe.com/hpe-innovation-workshops/https://developer.hpe.com/hpe-innovation-workshops/<p>The HPE Innovation workshops are an invaluable opportunity for all IT technologists looking to learn how to design, implement, and operate a hybrid cloud AI platform.</p> <p>One of the first workshops, HPE AI Foundations, shows you how to establish a comprehensive and extensible foundation to empower integration, strategic deployment, responsible data management &#x26; governance, machine &#x26; deep learning, and generative AI.</p> <p>During the workshop, you will gain the knowledge and skills necessary to understand HPE GreenLake cloud and the best of breed AI ecosystem technologies incorporated within it.</p> <style> table { display: block; width: 100%; width: max-content; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <table> <thead> <tr> <th>    Date    </th> <th>Title</th> <th>Speaker(s)</th> <th>Company</th> <th>Link</th> </tr> </thead> <tbody> <tr> <td>13-Aug-24</td> <td>HPE AI Foundations Workshop - APAC</td> <td>Steven Fatigante</td> <td>HPE</td> <td></td> </tr> <tr> <td>14-Aug-24</td> <td>HPE AI Foundations Workshop - EMEA</td> <td>Steven Fatigante</td> <td>HPE</td> <td></td> </tr> <tr> <td>15-Aug-24</td> <td>HPE AI Foundations Workshop - AMA</td> <td>Steven Fatigante</td> <td>HPE</td> <td></td> </tr> </tbody> </table>https://developer.hpe.com/links/https://developer.hpe.com/links/<img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/hpedevprogram-qrcode-1584535759606.png" width="400"> <p>We are glad you followed that QR Code link. Now please join the HPE Developer Community using your favourite communication channel: <br /></p> <ul> <li>On <a href="https://www.yammer.com/hpe.com/groups/HPEDeveloperCommunityProgram">Yammer</a> (HPE)</li> <li>On <a href="https://www.yammer.com/presalesone/groups/HPEDeveloperCommunity">Yammer</a> (Partners)</li> <li>On <a href="https://slack.hpedev.io">Slack</a></li> <li>On <a href="https://twitter.com/HPE_Developer">Twitter</a></li> <li>On the <a href="https://developer.hpe.com">Web</a></li> <li>By <a href="mailto:hpedev@hpe.com">email</a></li> <li>Subscribe to our monthly <a href="https://developer.hpe.com/newsletter-signup">Newsletter</a></li> <li>Register to one of our <a href="/hackshack/workshops">Workshops-on-Demand</a></li> </ul> <br /> <p>Thank you!</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/rsz_1rsz_stack-waving-1557501164196.png" width="400">https://developer.hpe.com/meetups/https://developer.hpe.com/meetups/<p>Connect with the experts to dive deep and learn more about some of today’s most exciting technologies. In these meetups, speakers will discuss specific products in detail and collaborate with you to determine how they may help in your particular situation.</p> <p>Hosted by the HPE Developer Community, these meetups are held on a monthly basis at 5PM CET (8AM PST).  Read more about Meetups <a href="https://developer.hpe.com/blog/new-for-2022-hpe-dev-meetups/">in this blog</a> post.</p> <style> table { display: block; width: 100%; width: max-content; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <table> <thead> <tr> <th>    Date    </th> <th>Title</th> <th>Speaker(s)</th> <th>Company</th> <th>Link</th> </tr> </thead> <tbody> <tr> <td>20-May-26</td> <td>Automate what happens next using HPE OpsRamp Process Automation</td> <td>Singh S B, Simmaranpal / J C Prasad Vasireddy / Ravi Kiran Srirangam Sadananda</td> <td>HPE</td> <td><a href="https://hpe.zoom.us/webinar/register/3217769572987/WN_COR_W9XMQtq4FYC5JhL__g">Register</a></td> </tr> <tr> <td>08-Apr-26</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/HPEDEV+Meetup+26-04+-+Network+Operations+Spotlight+with+PyCentralv2.pdf">Network Operations Spotlight with PyCentralv2</a></td> <td>Collin Koss /Alvin Castro</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=tEQZqb3C98I%C3%9F">Replay</a></td> </tr> <tr> <td>25-Feb-26</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/HPE+Dev+Meetup+Feb+26+-+Introduction+to+HPE+Networking+Central&#x27;s+new+APIs+with+Postman+and+PyCentralv2.pdf">Introduction to HPE Networking Central's new APIs with Postman and PyCentralv2</a></td> <td>Alvin Castro / Lucas Voron</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=5QFTzwk9VrU">Replay</a></td> </tr> <tr> <td>17-Dec-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/WOD_infrastructure_less_than_an_hour.pdf">Implementing a complete Workshop-on-Demand infrastructure in less than an hour</a></td> <td>Frederic Passeron / Bruno Cornec</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=RSt7ewTulPE">Replay</a></td> </tr> <tr> <td>18-Jun-25</td> <td>HPE Compute Ops Management APIs and the DevOps ecosystem: Updates and developer tools</td> <td>Lionel Jullien</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=TKl_gRJnoxk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>30-Apr-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/HPE+PCAI+Webinar+04-30-25.pdf">HPE Private Cloud AI Technical Demo</a></td> <td>Randy Thomasson</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=5uRoHXD2Sks&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Mar-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/No+More+Blindspots.pdf">No More Blind Spots: How eBPF Transforms Observability</a></td> <td>Neil Pearson</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=qH93yD5KSL8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Feb-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/HPE+GreenLake+Cloud+Webhooks.pdf">Introduction to HPE GreenLake cloud webhooks</a></td> <td>Vandewilly Silva / Roman Nersisyan</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=CPiMcIjXyVA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>29-Jan-25</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/HPEDEV_HPE+Private+Cloud+AI+deep+dive.pdf">Unleashing AI Innovation: A deep dive into the HPE Private Cloud AI Software Stack</a></td> <td>Srikanth Venkata Seshu / Thiru Palanichamy</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=6zj7biG74uk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>18-Dec-24</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/LLM+Agentic+Tool+Mesh+%E2%80%93+Democratizing+Gen+AI+-+Meetup+18th+December.pdf">LLM Agentic Tool Mesh – Democratizing Gen AI</a></td> <td>Antonio Fin / Cong Xu</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=vDQAVIuEsVo&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>30-Oct-24</td> <td>NVIDIA NIM Agents Blueprints - Do your own AI: Multimodal PDF Data Extraction 101</td> <td>Miguel Martinez</td> <td>NVIDIA</td> <td></td> </tr> <tr> <td>25-Sep-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE_SIC_KashishPahwa.pdf">Exploring the HPE Sustainability Insight Center: Key features, innovations, and API capabilities</a></td> <td>Kevin Tronkowski / Kashish Pahwa / Pallavi Pathak</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=jzYjTrnbmsE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>28-Aug-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Kevin+HPE+Developer+Community+Presentation.pdf">LLM finetuning for mere mortals</a></td> <td>Kevin Musgrave</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=GpjCOrlYt0M&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>31-Jul-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Vendor-Neutral-GPU-Programming-in-Chapel-Meetup-31-July-2024-v1.0.pdf">Vendor-Neutral GPU Programming in Chapel</a></td> <td>Jade Abraham / Engin Kayraklioglu</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=nj-WqhGEy24&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Jun-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Implementing+Centralized+Key+Management+in+HPE+GreenLake+with+Thales+CipherTrust+Hackshack-v1.pdf">Implementing Centralized Key Management in HPE GreenLake with Thales CipherTrust</a></td> <td>Jeff Ceason</td> <td>Thales DIS CPL</td> <td><a href="https://www.youtube.com/watch?v=A77yVzNtIKs&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>29-May-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/SlidesMeetup-May24.zip">Using HPE GreenLake for Red Hat OpenShift to migrate, modernize and run your applications smoothly</a></td> <td>Russell Clark / Ian Lawson</td> <td>HPE / Red Hat</td> <td><a href="https://www.youtube.com/watch?v=ScHOpRElCBE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>24-Apr-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/DSCC+API+Family.pdf">The data services family of APIs for HPE GreenLake – Putting it all together</a></td> <td>Paul Murray / Edgard Lopez / Ronald Dharma</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=-ffzhc6TzoA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>27-Mar-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/GLPAPI.pdf">Enabling business automation using HPE GreenLake Platform foundational APIs</a></td> <td>HPE Dev Team</td> <td>HPE</td> <td><a href="https://youtu.be/Uv0kKyVfjgg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>28-Feb-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE-Developer-Central.pdf">Getting Started with HPE Aruba Networking Central automation</a></td> <td>Alvin Castro / Karthik Satheesh Kumar</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=R4oo-7ACuLg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>31-Jan-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Getting_Started_With_HPE_GL_for_ComputeOpsMgmt.pdf">Getting started with HPE GreenLake for Compute Ops Management APIs</a></td> <td>Sebastian Schagerer</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=uwpxzHNXKvE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>29-Nov-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Dev+Community+Meetup-PCE+DevOps+-+WhiteBackground.pdf">Automated continuous deployment of container-based applications onto HPE GreenLake for Private Cloud Enterprise</a></td> <td>Pavan Belur Gopalakrishna Upadhya, Ruma Nandi, Sonu Sudhakaran</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=WlFohNTqv8w&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>25-Oct-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/%5BHPE%5D+Simplifying+containers+and+Kubernetes+on+your+laptop+with+Podman+Desktop.pdf">Going from containers, to pods, to Kubernetes – help for your developer environments!</a></td> <td>Cedric Clyburn / Evan Shortiss</td> <td>Red Hat</td> <td><a href="https://www.youtube.com/watch?v=Hy4MsJgxG8k&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>27-Sep-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Nethopper-AvoidK8sToolSprawlWithKAOPS-092723%5B34%5D.pdf">DevOps Alert: Tool Sprawl. Complexity. Burnout. Help!</a></td> <td>Chris Munford</td> <td>Nethopper</td> <td><a href="https://www.youtube.com/watch?v=zlRJUl_aF04&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Jul-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Observability+in+Action+26th+July+2023.pdf">Observability in Action</a></td> <td>Neil Pearson</td> <td>OpsRamp/HPE</td> <td><a href="https://www.youtube.com/watch?v=ogBb_IPXorM&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>31-May-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE+Developer+Meetup+-+May+.pdf">HPE Machine Learning Development Environment and the open source ML advantage</a></td> <td>Isha Ghodgaonkar</td> <td>Determined AI/HPE</td> <td><a href="https://www.youtube.com/watch?v=e8Fv4IWl38s&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Apr-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Divide+and+Conquer+with+Micro-frontends.pdf">Divide and conquer with Micro Frontends</a></td> <td>Vishal Sharma</td> <td>ThoughtWorks</td> <td><a href="https://www.youtube.com/watch?v=LbCw7Z7KT1U&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>29-Mar-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE+Postman+Intro.pdf">Collaborating at scale with Postman</a></td> <td>Bryan Cross</td> <td>Postman</td> <td><a href="https://www.youtube.com/watch?v=LuXNpIEzYgg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>22-Feb-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE+Dev+meetup+Feb+23+-+Project+Galadriel.pdf">Galadriel - An alternative approach to SPIRE federation</a></td> <td>Maximiliano Churichi/Max Lambrecht</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=IfaBCBXJBdw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>25-Jan-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/iac-developer-2023.pdf">HPE GreenLake and Infrastructure-as-Code (IaC)</a></td> <td>Eamonn O’Toole</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=zUo8Ag2IXqk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>17-Jan-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE+Ezmeral+Unified+Analytics_HPEDEV+-+Read-Only.pdf">HPE Ezmeral Unified Analytics</a></td> <td>Matt Morris</td> <td>HPE</td> <td><a href="https://www.youtube.com/watch?v=1Z4fNOHGYlk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>14-Dec-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Part+2_Running+Reliable+systems_SLO+Math.pdf">Running reliable systems - Part 2: Service level objective (SLO) Math</a></td> <td>Leonid Yankulin</td> <td>Google Cloud</td> <td><a href="https://www.youtube.com/watch?v=ZDxptOGs-ow&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>7-Dec-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Part+1_+Running+Reliable+systems_+SRE+Overview.pdf">Running reliable systems - Part 1: An Overview of Site Reliability Engineering (SRE)</a></td> <td>Leonid Yankulin</td> <td>Google Cloud</td> <td><a href="https://www.youtube.com/watch?v=XhhqEjUaLjE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>30-Nov-22</td> <td>Introduction to Kubeflow</td> <td>Souheil Inati/Chase Christensen</td> <td>Arrikto</td> <td></td> </tr> <tr> <td>26-Oct-22</td> <td>Boost Spark AI workloads with Pepperdata</td> <td>Heidi Carson/Kirk Lewis</td> <td>Pepperdata</td> <td><a href="https://www.youtube.com/watch?v=N36DTliNmck&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>28-Sep-22</td> <td>Machine Learning Data Version Control (DVC): Reproducibility and Collaboration in your ML Projects</td> <td>Alex Kim</td> <td>Iterative</td> <td><a href="https://www.youtube.com/watch?v=sgkN09LkCP4&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>31-Aug-22</td> <td>OpenTelemetry: Getting Started and The Road to Production</td> <td>Michael Haberman</td> <td>Aspecto</td> <td><a href="https://www.youtube.com/watch?v=odi9isyZOrU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>27-Jul-22</td> <td>Finding vulnerabilities in production with open source ThreatMapper</td> <td>Owen Garrett</td> <td>Deepfence</td> <td><a href="https://www.youtube.com/watch?v=r62VLwT6w3Y&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>22-Jun-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/opensearch-project.pdf">OpenSearch – The open-source search and analytics suite you can run yourself</a></td> <td>David Tippett</td> <td>AWS</td> <td><a href="https://www.youtube.com/watch?v=KdssEOIdO_0&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>25-May-22</td> <td>Scaling Language Training to Trillion-parameter Models on a GPU Cluster</td> <td>Evan Sparks</td> <td>Hewlett Packard Enterprise</td> <td><a href="https://www.youtube.com/watch?v=rIPqCvvMmms&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>27-Apr-22</td> <td>Decoupled policy enforcement with Open Policy Agent</td> <td>Anders Eknert</td> <td>Styra</td> <td><a href="https://www.youtube.com/watch?v=_0XJnr8U0sU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;t=15s">Replay</a></td> </tr> <tr> <td>30-Mar-22</td> <td>HPE + vFunction: Modernizing Legacy Applications and Data Sources Faster</td> <td>Samantha Cartwright / Amir Rapson</td> <td>vFunction</td> <td><a href="https://www.youtube.com/watch?v=UvcyIjzml7s&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>23-Feb-22</td> <td>Streamlit - The fastest way to build and share data science apps</td> <td>Arnaud Miribel</td> <td>Streamlit.io</td> <td><a href="https://youtu.be/sdgTYy3BJiM&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>26-Jan-22</td> <td>Quarkus - Supersonic Subatomic Java</td> <td>Dimitris Andreadis</td> <td>RedHat</td> <td><a href="https://www.youtube.com/watch?v=mY1z9OC0y54&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> </tbody> </table>https://developer.hpe.com/munch-and-learn/https://developer.hpe.com/munch-and-learn/<p>Join our free, one-hour webinars to hear from and engage with renowned technologists as they share thought-leadership insights on popular HPE and open source technologies. Get your questions answered while sharing a recipe or two.</p> <p>Hosted by the HPE Developer Community, these gatherings are held on a monthly basis at 5PM CET (8AM PST). You can read more about Munch &#x26; Learn Technology Talks <a href="https://developer.hpe.com/blog/hpe-dev-launches-its-munch-learn-technical-talks">in this blog</a> post.</p> <style> table { display: block; width: max-content !important; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; font-weight: normal !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <table> <thead> <tr> <th>       Date </th> <th>Title</th> <th>Speaker(s)</th> <th>Link</th> </tr> </thead> <tbody> <tr> <td>18-Mar-26</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/munch-and-learn_Open-Source-at-HPE_Three+Pillars.pdf">Open Source at HPE: Three Pillars, Many Paths</a></td> <td>Ria Farrell Schalnat / Alvin Castro /Dr. Andrew Shao –Shaun Tancheff</td> <td><a href="https://www.youtube.com/watch?v=c3L13ELQ51o">Replay</a></td> </tr> <tr> <td>28-Jan-26</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/Munch-and-Learn-GreenLake-MCP-Jan2026.pdf">Work Smarter: Natural Language Access to your infrastructure via GreenLake MCP</a></td> <td>Vandewilly Silva / Roman Nersyan</td> <td><a href="https://www.youtube.com/watch?v=-RYmpXaGl2o">Replay</a></td> </tr> <tr> <td>21-May-25</td> <td>HPE Sustainability Insight Center: A Win for both Business and the Environment</td> <td>Kashish Pahwa / Kevin Tronkowski</td> <td><a href="https://www.youtube.com/watch?v=1H0M-c7g744&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>16-Apr-25</td> <td>DayN+ - A new way to look at observability</td> <td>Raj Mistry</td> <td><a href="https://www.youtube.com/watch?v=nDa_NQPbbVY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>19-Mar-25</td> <td>ChatHPE Hub: Enabling Secure and Scalable AI Transformation at HPE</td> <td>Jose Mejias / Manmeet Dhillon</td> <td><a href="https://www.youtube.com/watch?v=cAXwX3Yb62c&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>19-Feb-25</td> <td>Unlocking Private AI Power: Insurance Fraud Detection and Beyond</td> <td>Jordan Nanos</td> <td><a href="https://www.youtube.com/watch?v=BYzF2Twg8UY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>15-Jan-25</td> <td>From log files to AI insights: The 60-year evolution of observability and AIOps</td> <td>Neil Pearson</td> <td><a href="https://www.youtube.com/watch?v=VnKSnf7G4-4&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>20-Nov-24</td> <td>How to fix your biggest security hole</td> <td>Ted Dunning</td> <td><a href="https://www.youtube.com/watch?v=mgla3ovDlXA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>16-Oct-24</td> <td><a href="https://hpe-developer-portal.s3.us-east-1.amazonaws.com/Munch_and_Learn_Classiq_HPE_final.pdf">Hybrid Classical-Quantum Workflows on HPE Supercomputers</a></td> <td>Aniello Esposito / Erik Garcell / Tamuz Danzig</td> <td><a href="https://www.youtube.com/watch?v=dO_5qXqfu5o&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>18-Sep-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/RAG+-Munch+%26+Learn.pptx.zip">Enhancing NLP with Retrieval-Augmented Generation: A Practical Demonstration</a></td> <td>Rajesh P E / Umang Kedia</td> <td><a href="https://www.youtube.com/watch?v=TQPvK-pFAD8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>21-Aug-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/August+21+2024+AI+Munch+and+Learn.pdf">Exploring Transformative AI Use Cases Across Industries</a></td> <td>Panel Discussion</td> <td><a href="https://www.youtube.com/watch?v=XEJqcdWj790&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>12-Jun-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/M%26L+Data+Center+Digital+Twins_June_2024+rev+01.pdf">How digital twins help companies reach their sustainability goals</a></td> <td>Cullen Bash / Gallig Renaud</td> <td><a href="https://www.youtube.com/watch?v=WUlH17ruIek&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>17-Apr-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE+Developer+Munch+%26+Learn+-+The+Transformative+Impact+of+Generative+AI+on+Telco+Products+-+April+2024.pdf">The Transformative Impact of Generative AI on Telco Products</a></td> <td>Antonio Fin</td> <td><a href="https://www.youtube.com/watch?v=61ghjMnc-JU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>20-Mar-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HackShack-HPE-Trust-ML.pdf">Learn how AI hackers detect fragility and how to thwart them with AI model resilience</a></td> <td>Soumyendu Sarkar / Ashwin Ramesh Babu / Sajad Mousavi</td> <td><a href="https://www.youtube.com/watch?v=CMASNlKuTao&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>21-Feb-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/SecureGenAIAdoption.pdf">Secure GenAI Adoption for all!</a></td> <td>Glyn Bowden / Tom Phelan / Saad Zaher</td> <td><a href="https://www.youtube.com/watch?v=FSEMz8fvpYE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>24-Jan-24</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/munch-n-learn_build_private_cloud_HPE_GL_edge-to-cloud-Platform.pdf">Using HPE GreenLake edge-to-cloud platform to build a private cloud</a></td> <td>Don Wake</td> <td><a href="https://www.youtube.com/watch?v=Rm1z2pHtyw0&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>13-Dec-23</td> <td>Exploring HPE GreenLake Platform APIs through use cases</td> <td>Pallavi Pathak</td> <td><a href="https://www.youtube.com/watch?v=IM7W89Vt7zQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>15-Nov-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Inference-Optimization-HPEDevelopers-Nov15-2023.pdf">Optimizing deep neural network inference workloads</a></td> <td>Hana Malha / Lindsey Hillesheim</td> <td><a href="https://www.youtube.com/watch?v=Ck5dVgp68uA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>18-Oct-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/The+Open+Source+advantage+Exploring+Machine+Learning+Through+Thought+Leadership.pdf">The open-source advantage: Exploring machine learning through thought leadership</a></td> <td>Amber Graner / Chase Christensen</td> <td><a href="https://www.youtube.com/watch?v=BgirPJNDtxs&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>20-Sep-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Linux_Trends_07.pdf">State of the Nation – Linux distributions</a></td> <td>Craig Lamparter</td> <td><a href="https://www.youtube.com/watch?v=dFYLyy7oL-Q&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;t=3s">Replay</a></td> </tr> <tr> <td>14-Jun-23</td> <td>Digital Twins, the Metaverse, and Augmented Reality: Developer Insights and IT Foundations for Immersive Technologies Powered by AI</td> <td>James Hopfenblatt / Marcus Bonner / Garry Orsolini</td> <td><a href="https://www.youtube.com/watch?v=T1aWHB0-4kA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>24-May-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/ChatGPTCollaboration.pdf">A New Era of Software Development: Embracing Large Language Model Tools like ChatGPT for Iterative Problem Solving</a></td> <td>Jeff Fougere / Andrew Nieuwsma / Jim Schreckengast</td> <td><a href="https://www.youtube.com/watch?v=zAm5CpOHfH4&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>19-Apr-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/Tech+for+Good+presentation.pdf">Leveraging Tech to Address Global Challenges &#x26; Health</a></td> <td>Fred Tan</td> <td><a href="https://www.youtube.com/watch?v=Wu04-dz81Pc&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>15-Mar-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/BRESNIKER+-+Dev+Community+-+Extra.pdf">Quantum computing insights from HPE Labs: Extraordinary Claims Require Extraordinary Engineering (and TAMO isn’t an option)</a></td> <td>Kirk Bresniker</td> <td><a href="https://www.youtube.com/watch?v=wVY7uZstDWA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>15-Feb-23</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/2023_OSS_MunchandLearn.pdf">Accelerating scientific research through high performance computing democratization</a></td> <td>Andrew Shao / Scott Bachman</td> <td><a href="https://www.youtube.com/watch?v=DnmhTj1PVIU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>16-Nov-22</td> <td>Calling all citizen developers: Can low-code platforms accelerate your impact?</td> <td>Jeffrey Fougere / Richard Kerridge / Colin Lue King</td> <td><a href="https://www.youtube.com/watch?v=zc_54fq8PoY&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>19-Oct-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/HPE_Munch%26Learn_Sustainability_final.pdf">HPE Sustainability Strategy and Sustainability Research at Hewlett Packard Labs</a></td> <td>Tiffani Jarnigan / Cullen Bash</td> <td><a href="https://www.youtube.com/watch?v=SUgdVsncWrk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>21-Sep-22</td> <td>Accelerate public sector AI use cases using a powerful ML Ops platform</td> <td>François Réthoré / Dietrich Zinsou</td> <td><a href="https://www.youtube.com/watch?v=5pejLKu32Js&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>17-Aug-22</td> <td>Machines learn from data to be artificially intelligent</td> <td>Dr. Eng Lim Goh</td> <td><a href="https://youtu.be/3KOFDciS3WU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>18-May-22</td> <td>Why Open Source is more than Software: The example of The Linux Foundation's AgStack project</td> <td>Ted Dunning / Sumer Johal</td> <td><a href="https://www.youtube.com/watch?v=dnhjRF5dr6M&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>20-Apr-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/ChapelForHPEMunchAndLearn.pdf">Chapel: Making parallel computing as easy as Py(thon), from laptops to supercomputers</a></td> <td>Brad Chamberlain</td> <td><a href="https://www.youtube.com/watch?v=7Qk8T7_bevo&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1">Replay</a></td> </tr> <tr> <td>23-Mar-22</td> <td>Mithril: Introducing Robust Identities into Istio by integrating with SPIRE</td> <td>Praneetha Manthravadi / Mithril engineering team</td> <td><a href="https://youtu.be/xhd8MhG4Vvw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>16-Feb-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/munch-and-learn-feb-2022.pdf">Golden Age of AI, Dark Ages of AI Infrastructure</a></td> <td>Neil Conway</td> <td><a href="https://youtu.be/ktZFLD-9qgw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>19-Jan-22</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/JanuaryMunchAndLearn.zip">Location, location, location! Succeed at the Edge with HPE Ezmeral and NVIDIA</a></td> <td>Denis Vilfort / William Benton</td> <td><a href="https://www.youtube.com/watch?v=C5HfiLatauQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>08-Dec-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/DecemberMunchAndLearn-Jeff.pdf">Redfish: Past, Present and Future</a></td> <td>Jeff Hilland / Jeff Autor / François Donzé</td> <td><a href="https://www.youtube.com/watch?v=Q1Qeb24lpKg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>17-Nov-21</td> <td>The Great Unification: Building Analytic pipelines with Apache Spark Workloads</td> <td>Matt Hausmann / Donald (Don) Wake / Matthew Morris</td> <td><a href="https://youtu.be/TxZP_T9CC5Y&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>22-Sep-21</td> <td>Digital Transformation.Next: Data &#x26; Analytics Workloads</td> <td>Matt Maccaux / Randy Thomasson</td> <td><a href="https://youtu.be/Q4kJKCS7rbo&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>25-Aug-21</td> <td>Kubernetes 101</td> <td>Nigel Poulton / Didier Lalli</td> <td><a href="https://youtu.be/PWVJKK1obKQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>28-Jul-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/7/HPE-Munch-and-Learn-7-28-july-2021.pdf">How to make data consumable for real-world data science</a></td> <td>Ellen Friedman / Ted Dunning</td> <td><a href="https://youtu.be/4WKjRqflF7M&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>30-Jun-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/4/fundamentals-of-microservices-1625131973756.pdf">Microservices Architecture 101</a></td> <td>Owen Garrett</td> <td><a href="https://youtu.be/qyyxQU37ZyQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>19-May-21</td> <td>Data Science Unplugged: Part 2</td> <td>Doug Cackett / Ellen Friedman</td> <td><a href="https://youtu.be/Va4tSr__Yok&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>21-Apr-21</td> <td>Building a foundation for zero trust with SPIFFE</td> <td>Daniel Feldman / Frederick Kautz</td> <td><a href="https://youtu.be/G1ceKr16nn8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>24-Mar-21</td> <td>Data Science Unplugged: Part 1</td> <td>Doug Cackett / Ellen Friedman</td> <td><a href="https://youtu.be/Inh6eXM0EbA&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>24-Feb-21</td> <td>Explore Containerization and MLOps</td> <td>Tom Phelan / Nigel Poulton</td> <td><a href="https://youtu.be/9PvKpe7yMpI&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> <tr> <td>27-Jan-21</td> <td><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/munch-and-learn-dunning-1611939333032.pdf">What’s a data fabric and how does it work?</a></td> <td>Ted Dunning / Ellen Friedman</td> <td><a href="https://youtu.be/qi6sTvu8osk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Replay</a></td> </tr> </tbody> </table>https://developer.hpe.com/01-hpe-greenlake/https://developer.hpe.com/01-hpe-greenlake/<p>Discover the unified HPE GreenLake experience through well-documented, secure, and scalable APIs.</p>https://developer.hpe.com/02-hpe-greenlake-testdrive/https://developer.hpe.com/02-hpe-greenlake-testdrive/<p>Browse our YouTube channel to learn new things and expand your skill set.</p>https://developer.hpe.com/03-role/https://developer.hpe.com/03-role/<p>HPE Developer Day is a focused one-day event designed to help you move from AI ambition to real-world results. Check it out!</p>https://developer.hpe.com/05-meetups/https://developer.hpe.com/05-meetups/<p>Connect with the experts to dive deep and learn more about some of today’s most exciting technologies.</p>https://developer.hpe.com/04-munch-learn-technology-talks/https://developer.hpe.com/04-munch-learn-technology-talks/<p>Attend our monthly community meetups where you can interact with experts regarding popular new technologies and get your questions answered.</p>https://developer.hpe.com/06-workshops-on-demand/https://developer.hpe.com/06-workshops-on-demand/<p>Familiarize yourself with popular HPE and open source technologies through on-demand technical training. These workshops use Jupyter Notebooks to give you a unique, hands-on learning experience.</p>https://developer.hpe.com/07-spiffe-spire-graduates-enabling-greater-security-solutions/https://developer.hpe.com/07-spiffe-spire-graduates-enabling-greater-security-solutions/<p>Deploying a local Large Language Model (LLM) architecture using S3‑compatible object storage and vLLM as the inference engine provides a scalable, cost‑efficient, and secure foundation for enterprise AI adoption.</p>https://developer.hpe.com/08-hpe-hackshack/https://developer.hpe.com/08-hpe-hackshack/<p>Explore our Virtual Experience to learn about different technologies and have some fun in the process.</p>https://developer.hpe.com/09-featured-platform-1/https://developer.hpe.com/09-featured-platform-1/<p>Hewlett Packard Enterprise (HPE) is the leading contributor to Cloud Native Computing Foundation’s (CNCF) SPIFFE and SPIRE open source projects.</p>https://developer.hpe.com/10-featured-platform-2/https://developer.hpe.com/10-featured-platform-2/<p>Determined AI accelerates innovation with open source AI solutions to build and train models faster and easier.</p>https://developer.hpe.com/https://developer.hpe.com/<h1>HPE Developer Community</h1> <p>Where you’ll find all things software at HPE. Join us to collaborate and build applications and integrations with HPE products using the latest software and open source technologies.</p>https://developer.hpe.com/EzmeralLearnOnDemand/https://developer.hpe.com/EzmeralLearnOnDemand/<p>ok</p>https://developer.hpe.com/MunchandLearn/https://developer.hpe.com/MunchandLearn/<p>Join our free, one-hour webinars to hear from and engage with renowned technologists as they share thought-leadership insights on popular HPE and open source technologies.</p>https://developer.hpe.com/Workshops-on-Demand/https://developer.hpe.com/Workshops-on-Demand/https://developer.hpe.com/skillup-1/https://developer.hpe.com/skillup-1/<p>Connect with the experts to dive deep and learn more about some of today’s most exciting technologies.</p>https://developer.hpe.com/skillup-2/https://developer.hpe.com/skillup-2/<p>This informative series will provide you with insights on how to make AI a functional reality in your organization, transforming industry hype into technical and business value.</p>https://developer.hpe.com/skillup-3/https://developer.hpe.com/skillup-3/<p>Done</p>https://developer.hpe.com/skillup-4/https://developer.hpe.com/skillup-4/<p>Unlock unlimited access to strategically curated training on HPE technologies, IT, business, and power skills. Learn continuously. Lead confidently.</p>https://developer.hpe.com/skillup/https://developer.hpe.com/skillup/<p>Browse replays of our most successful technology talks.</p>https://developer.hpe.com/cloud-architect/home/https://developer.hpe.com/cloud-architect/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>In a highly distributed environment with diverse needs, cloud architects today are challenged to design, support and maintain a hybrid cloud infrastructure that provides maximum efficiency and cost effectiveness. On this page, we focus on the HPE GreenLake edge-to-cloud platform and provide practical tips to address key issues.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **Customize solutions to your specific needs:** <pre><code>*In an Everything-as-a-Service model.* </code></pre> </div> <div class="column"> ### Building a Foundation **Today’s cloud strategy** <pre><code>*A mix of public cloud, private cloud, and legacy infrastructure* </code></pre> </div> </div> <div class="row"> <div class="column"> - See how you can [get hands-on experience with the new HPE GreenLake for Data Fabric](/blog/hpe-ezmeral-early-access-gives-developers-hands-on-experience-with-the-new-hpe-greenlake-for-data-fabric/) <pre><code>- Learn how to [curate and expose service catalog items using HPE GreenLake for private cloud](/blog/curate-and-expose-service-catalog-items-using-hpe-greenlake-for-private-cloud/) - Discover how you can [deploy an ML model in the HPE GreenLake Platform ML Ops service](/blog/mlops-–-deploying-an-ml-model-in-greenlake-platform-mlops-service/) </code></pre> </div> <div class="column"> - Explore the service [HPE GreenLake for Compute Ops Management](/greenlake/hpe-greenlake-for-compute-ops-management/home/) <pre><code>- Learn about [Data Services Cloud Console](/greenlake/data-services-cloud-console/home/) - Learn to work with [HPE Aruba Central](/greenlake/aruba-central/home/) </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>Supporting an increasingly distributed workforce and data</strong></p> <ul> <li> <p>Learn of <a href="https://www.hpe.com/us/en/newsroom/press-release/2022/12/hpe-greenlake-adds-application-analytics-and-developer-services-to-modernize-workloads-across-the-hybrid-cloud.html">new applications and developer services</a> that drive a data-first modernization strategy across edge to cloud</p> </li> <li> <p>See how <a href="https://community.hpe.com/t5/the-cloud-experience-everywhere/modern-private-cloud-made-easy-hpe-unveils-hpe-greenlake-for/ba-p/7169009#.Y9tUKOzMJGM">HPE GreenLake for Private Cloud Enterprise makes a modern private cloud easy</a></p> </li> <li> <p>Explore the industry’s first <a href="/greenlake/hpe-greenlake-for-data-fabric/home/">hybrid analytics-ready data fabric</a></p> </li> </ul> <p><strong>Dealing with networking concerns</strong></p> <ul> <li> <p>Read how <a href="https://community.arubanetworks.com/community-home/digestviewer/viewthread?GroupId=67&#x26;MessageKey=c59874f5-eab5-44e4-b7c0-59e7762410e8&#x26;CommunityKey=e1202040-11b3-4eea-9f57-d903f67db2f9">Aruba Central is now part of HPE GreenLake</a></p> </li> <li> <p>Learn <a href="/blog/how-to-create-a-virtual-network-in-hpe-greenlake-for-private-cloud/">how to create a virtual network in HPE GreenLake for Private Cloud</a></p> </li> <li> <p>Secure connections by <a href="/blog/configuring-azure-ad-with-greenlake-cloud-platform-and-aruba-central/">configuring Azure AD as the SAML IDP with HPE GreenLake Cloud Platform and Aruba Central</a></p> </li> </ul> <p><strong>Addressing security and access management</strong></p> <ul> <li> <p>See <a href="/blog/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/">how to use an API access token for HPE GreenLake for Compute Ops Management</a></p> </li> <li> <p>Read <a href="/blog/understanding-hpe-greenlake-identity-access-management/">Understanding HPE GreenLake Central Identity &#x26; Access Management</a></p> </li> <li> <p>Explore the CNCF <a href="/platform/spiffe-and-spire-projects/home/">SPIFFE and SPIRE projects</a></p> </li> <li> <p>View video explaining how to <a href="https://www.hpe.com/h22228/video-gallery/us/en/10b3350c-52fe-45ca-9232-71ddc7185c77/video?jumpId=in_videogallery_e8826c6e-76e3-4c2e-b751-a257a47788f5_gaiw">access iLO through HPE GreenLake Compute Ops Management</a></p> </li> <li> <p><a href="/blog/api-console-for-data-services-cloud-console/">Use HPE GreenLake Console’s API Gateway for Data Services Cloud Console</a></p> </li> </ul> <p><strong>Automation and orchestration of cloud infrastructure</strong></p> <ul> <li> <p>Explore automation through <a href="/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/">HPE GreenLake and Infrastructure-as-code using Terraform</a></p> </li> <li> <p>See <a href="/blog/how-to-integrate-chef-automation-with-hpe-greenlake-for-private-cloud/">how to integrate Chef automation with HPE GreenLake for private cloud</a></p> </li> </ul> <p><strong>Controlling costs and managing cloud governance</strong></p> <ul> <li> <p>Check out <a href="https://docs.consumption.support.hpe.com/HPE_GreenLake_Billing_Docs/HPE_GreenLake_Billing_User_Guide/Viewing_monthly_usage_and_charges">HPE GreenLake Usage and Analytics documentation</a></p> </li> <li> <p>Refer to the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=HPE-Consumption-Analytics.html">HPE GreenLake Cloud Consumption Analytics User Guide</a></p> </li> </ul> <p><strong>Monitoring and observability</strong></p> <ul> <li> <p>See how <a href="https://www.hpe.com/us/en/newsroom/blog-post/2022/03/unified-server-management-from-edge-to-cloud-drives-greater-performance-and-efficiency.html">unified server management from edge-to-cloud drives greater performance and efficiency</a></p> </li> <li> <p>Learn <a href="/blog/how-to-monitor-hpe-compute-ops-management/">how to monitor HPE GreenLake for Compute Ops Management infrastructure with Grafana Metrics Dashboards</a></p> </li> <li> <p>Read <a href="/blog/greenlake-platform-infrastructure-monitoring-with-apache-skywalking/">HPE GreenLake for Private Cloud Enterprise monitoring with Apache SkyWalking and OpenTelemetry</a></p> </li> </ul> <p><strong>Filling skills gaps</strong></p> <ul> <li> <p>Explore the <a href="/skillup/">HPE Developer Skill Up page</a></p> </li> <li> <p>Take a <a href="https://testdrive.greenlake.hpe.com/">Test Drive</a></p> </li> <li> <p>Discover <a href="https://education.hpe.com/ww/en/training/portfolio/greenlake.html">HPE GreenLake training and certification</a></p> </li> <li> <p>Join in on <a href="https://community.arubanetworks.com/community-home?CommunityKey=403fcb2e-4306-4eee-9bc7-bd6d0309ff7e">Aruba Central Community discussions</a></p> </li> <li> <p>Learn more about streamlining hybrid cloud management by reading <a href="https://www.hpe.com/us/en/collaterals/collateral.a00112667.Streamlining-hybrid-cloud-management.html?rpv=cpf&#x26;parentPage=/us/en/greenlake/hybrid-multi-cloud-services">HPE GreenLake Management Services</a>, checking out this <a href="https://www.hpe.com/us/en/services/remote-infrastructure-monitoring.html">site</a>, or going through this <a href="https://community.hpe.com/t5/the-cloud-experience-everywhere/hybrid-cloud-simplified-a-podcast-intro-to-hpe-greenlake-hybrid/ba-p/7163196#.Y9tX2uzMJGM">podcast</a></p> </li> </ul> <p><strong>Case studies</strong></p> <ul> <li> <p><a href="https://www.hpe.com/h22228/video-gallery/us/en/67b545ce-0d1c-43c7-8f85-ee03b4bf1754/western-canada-lottery-corporation--winning-with-insight-and-new-ways-to-play/video/">Western Canada Lottery Corporation: winning with insight and new ways to play</a></p> </li> <li> <p><a href="https://www.hpe.com/psnow/doc/a50001216enw?jumpid=in_hpesitesearch">HPE GreenLake helps Wibmo adopt agile, scalable infrastructure model</a></p> </li> <li> <p>More HPE GreenLake <a href="https://www.hpe.com/us/en/greenlake/customer-stories.html">customer stories</a></p> </li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Meetups</a></p> <div class="row"> <div class="column"> A series of in-depth talks on open source developer technologies. </div> <div class="column"> - [HPE GreenLake and Infrastructure-as-Code (IaC)](https://www.youtube.com/watch?v=zUo8Ag2IXqk&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&ab_channel=HewlettPackardEnterprise) <pre><code>- [HPE GreenLake for Data Fabric](https://www.youtube.com/watch?v=rzLxGZIraTg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HewlettPackardEnterprise) </code></pre> </div> </div> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> - [HPE Ezmeral Data Fabric 101 – Get to know the basics around the data fabric](/hackshack/workshop/26) <pre><code>- [Introduction to Virtual Machine Infrastructure-as-Code using Terraform and HPE GreenLake for Private Cloud Enterprise](/hackshack/replays/36) - [Creating a Zero Trust Model for Microservices Architectures with SPIRE and Envoy](/hackshack/replays/32) </code></pre> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Demos, resources, APIs and documentation</div> <div class="row"> <div class="column"> <pre><code>The [HPE GreenLake developer portal](https://developer.greenlake.hpe.com/) offers documentation and API information. You can also find additional resources on the demos and resources page. </code></pre> </div> <div class="column"> - [HPE GreenLake edge-to-cloud platform demos and resources](https://www.hpe.com/us/en/greenlake/demos.html) <pre><code>- [HPE GreenLake API and documentation portal](https://developer.greenlake.hpe.com/) - [HPE GreenLake Test Drive](https://www.hpe.com/us/en/greenlake/demos.html) </code></pre> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> <pre><code>Ping us with your comments, questions, and requests for information. </code></pre> </div> <div class="column"> - [HPE Developer Community Slack]( https://slack.hpedev.io/) </div> </div>https://developer.hpe.com/data-driven-developer/home/https://developer.hpe.com/data-driven-developer/home/<p>N/A</p>https://developer.hpe.com/cloud-datacenter-owner/home/https://developer.hpe.com/cloud-datacenter-owner/home/<p>Deliver modern, scalable and secure on-prem and cloud infrastructures managed through centralized dashboards that monitor multiple vendors and services.</p>https://developer.hpe.com/data-ml-engineer/home/https://developer.hpe.com/data-ml-engineer/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>It’s your job to ensure the right data powers the right applications at the right time and in the right place.</p> <p>With an increased number and variety of workloads, how can you address all aspects of data logistics and processing that can make or break the success of any data-intensive project, including analytics and AI/machine learning? And do it easily and reliably?</p> <p>On this page, we provide content to help you meet these challenges. You will find a rotating selection of foundational material, ideas to help you get inspired, as well as practical tips on key issues to improve efficiency and performance. You’ll also learn what Hewlett Packard Enterprise (HPE) offers.</p> <p>The roles of the Data/ML Engineer and Data Scientist can overlap. You may also find content of interest to you on the <a href="/role/data-scientist/home/">Data Scientist</a> page. Content on this page changes as new material becomes available or new topics arise, so check back regularly.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **A sampler of new ideas related to data/ML engineering:** <pre><code>*Learn how industry innovation may affect your job.* </code></pre> </div> <div class="column"> ### Building a Foundation **Key to data science projects is a unifying data infrastructure to handle logistics and the containerization of applications** <pre><code>*Simplify operations and workflows with the right data fabric and orchestrate containerized applications with open source Kubernetes.* </code></pre> </div> </div> <div class="row"> <div class="column"> - Unit testing isn’t just for code: you need to unit test your data. [Watch Deequ: Unit Tests for Data](https://www.youtube.com/watch?v=2f_JewK79GI) <pre><code>- Data locality helps support GPUs and other accelerators from a data point of view. Read [How fine-grained data placement helps optimize application performance](https://developer.hpe.com/blog/how-fine-grained-data-placement-helps-optimize-application-performance/) - Better connections between data producers and data consumers make data science more successful. Read [Getting value from your data shouldn’t be this hard](https://www.hpe.com/us/en/insights/articles/getting-value-from-your-data-shouldn-t-be-this-hard-2106.html) </code></pre> </div> <div class="column"> - Study the technical paper [HPE Ezmeral Data Fabric: Modern infrastructure for data storage and management](https://www.hpe.com/psnow/doc/a00110846enw) <pre><code>- Read [What’s your superpower for data management?](https://community.hpe.com/t5/HPE-Ezmeral-Uncut/What-s-your-superpower-for-data-management/ba-p/7100920#.Ya5RTb3ML0p) - View the [HPE Ezmeral Data Fabric platform page](https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/) - Read [Kuberneticized machine learning and AI using Kubeflow](https://developer.hpe.com/blog/kubernetized-machine-learning-and-ai-using-kubeflow/) - Learn how management of large scale Kubernetes clusters is made easier with [HPE Ezmeral Runtime Enterprise](https://developer.hpe.com/platform/hpe-ezmeral/home/) </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>What can I do to lower the entry barriers to developing new AI/ML/data science projects?</strong></p> <ul> <li> <p>AI/ML projects can and should be run on the same system as analytics projects: Read “Chapter 3: AI and Analytics Together” in the free eBook <a href="https://www.hpe.com/us/en/resources/software/ai-and-analytics-systems.html">AI and Analytics at Scale: Lessons from Real-World Production Systems</a></p> </li> </ul> <p><strong>Who should be included on the team to ensure the success of the project?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/The-New-Data-Science-Team-Who-s-on-First/ba-p/7154783#.Ybi1pb3MI2y">The New Data Science Team: Who’s on First?</a></p> </li> </ul> <p><strong>How do I handle data movement?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/A-better-approach-to-major-data-motion-Efficient-built-in/ba-p/7135056#.Ya5Xqb3ML0p">A better approach to major data motion: built-in data mirroring</a></p> </li> <li> <p>Watch the webinar <a href="https://www.hpe.com/h22228/video-gallery/us/en/5a1ff1b7-faf8-43f2-98a3-d5b7331616b6/video?jumpid=em_4pbhacrk27_aid-520049397&#x26;utm_source=RE">Data Motion at Scale: the Untold Story</a></p> </li> </ul> <p><strong>What makes it easier to deal with edge computing in large-scale systems?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/To-the-edge-and-back-again-Meeting-the-challenges-of-edge/ba-p/7132609#.Ya5X3r3ML0o">To the edge and back again: Meeting the challenges of edge computing</a></p> </li> </ul> <p><strong>How do I ensure data trust and security?</strong></p> <ul> <li> <p>New approaches are improving the connection between data producers and data consumers. See how in the video <a href="https://www.youtube.com/watch?v=9VTLA1nxpoo">Dataspaces: connecting to data you can trust</a></p> </li> <li> <p>Learn about the <a href="https://developer.hpe.com/platform/spiffe-and-spire-projects/home/">SPIFFE and SPIRE projects</a> that are hosted by the CNCF Foundation</p> </li> </ul> <p><strong>How are others doing this?</strong></p> <p>Check out these real-world case studies</p> <ul> <li> <p><a href="https://www.hpe.com/psnow/doc/a50003176enw?jumpid=in_lit-psnow-red">Accelerating Autonomous Car Development with Ready Access to Global Data Fabric</a></p> </li> <li> <p><a href="https://www.hpe.com/psnow/doc/a50003827enw">Accelerating Data Insight for a Better Work Life</a></p> </li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Munch &#x26; Learn technology talk</a></p> <div class="row"> <div class="column"> Monthly meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks. </div> <div class="column"> - [What’s a data fabric and how does it work?](https://www.youtube.com/watch?v=qi6sTvu8osk) <pre><code>- [Data Science Unplugged: Part 1](https://www.youtube.com/watch?v=Inh6eXM0EbA) - [Data Science Unplugged: Part 2](https://www.youtube.com/watch?v=Va4tSr__Yok) - [How to make data consumable for real-world data science](https://www.youtube.com/watch?v=4WKjRqflF7M) </code></pre> </div> </div> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> - [HPE Ezmeral Data Fabric 101 – Get to know the basics around the data fabric](https://hackshack.hpedev.io/workshop/26) </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Documentation</div> <div class="row"> <div class="column"> <pre><code>The [HPE Ezmeral Data Fabric platform page](https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/) offers documentation and API information along with informative videos and tutorials. Additional documentation can be found here. </code></pre> </div> <div class="column"> - [HPE Ezmeral Data Fabric 7.0 documentation](https://docs.datafabric.hpe.com/70/index.html) </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> <pre><code>Ping us with your comments, questions, and requests for information. </code></pre> </div> <div class="column"> - [HPE Dev Slack]( https://slack.hpedev.io/) </div> </div>https://developer.hpe.com/data-scientist/home/https://developer.hpe.com/data-scientist/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>Data science is an intriguing field, and recognition of its potential is rapidly expanding. But there are challenges. You need access to the right data and the flexibility to use a variety of tools of your own choice. The pipeline for data preparation and for model and application deployment needs to be reliable and efficient. And you need to increase the likelihood that stakeholders and IT managers will green-light new projects.</p> <p>On this page, we provide a range of content – for advanced data scientists to those just getting started – to help you meet these challenges. You will find a rotating selection of foundational material, ideas to help you get inspired, as well as practical tips on key issues that help make your data science projects easier to build and more likely to be successful. You’ll also learn what Hewlett Packard Enterprise (HPE) offers.</p> <p>The roles of the Data/ML Engineer and Data Scientist can overlap. You may also find content of interest to you on the <a href="/role/data-ml-engineer/home/">Data/ML Engineer</a> page. Content on this page changes as new material becomes available or new topics arise, so check back regularly.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **A sampler of new ideas** <pre><code>*Learn how innovation may affect your job.* </code></pre> </div> <div class="column"> ### Building a Foundation **Optimize data access** <pre><code>*The right data infrastructure gives you direct access to data via a wide range of APIs for a choice in tools.* </code></pre> </div> </div> <div class="row"> <div class="column"> - [Watch Deequ: Unit Tests for Data](https://www.youtube.com/watch?v=2f_JewK79GI) to learn why unit testing isn’t just for code <pre><code>- Read [Swarm Learning: Turn Your Distributed Data into Competitive Edge](https://www.hpe.com/us/en/pdfViewer.html?docId=a50000344&#x26;parentPage=/us/en/products/compute/hpc/deep-learning&#x26;resourceTitle=Swarm+Learning:+Turn+Your+Distributed+Data+into+Competitive+Edge+technical+white+paper) to see how innovative architectures take advantage of the increasingly distributed nature of data </code></pre> </div> <div class="column"> - Study the technical paper [HPE Ezmeral Data Fabric: Modern infrastructure for data storage and management](https://www.hpe.com/psnow/doc/a00110846enw) <pre><code>- Learn best practices in [Getting the most from your data-driven transformation: 10 key principles](https://www.hpe.com/us/en/insights/articles/getting-the-most-from-your-data-driven-transformation-2109.html) - View the [HPE Ezmeral Data Fabric platform ](https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/) page </code></pre> </div> </div> <hr> <div class="row"> <div class="column"> **Working together** <pre><code>*Domain expertise helps frame questions, identify useful data, and take action on insights.* </code></pre> </div> <div class="column"> **Containerization of applications** <pre><code>*The open source Kubernetes framework orchestrates containers.* </code></pre> </div> </div> <div class="row"> <div class="column"> - Discover [The New Data Science Team: Who’s on First?](https://community.hpe.com/t5/HPE-Ezmeral-Uncut/The-New-Data-Science-Team-Who-s-on-First/ba-p/7154783#.Ybi1pb3MI2y) and learn how multiple roles contribute to a successful data science project <pre><code>- Combine diverse data sets to advance healthcare. Watch [Data Saves Lives](https://www.hpe.com/us/en/discover-more-network/events/discover-2021/results.html/types/sessions/search/b4372?media-id=/us/en/resources/discover/las-vegas-2021/B4372/_jcr_content.details.json) to learn how - Watch [Data Feeds People](https://www.hpe.com/us/en/discover-more-network/events/discover-2021/results.html/types/sessions/search/b4373) to see how combined stakholder expertise puts advanced agricultural knowledge to work in the field - literally </code></pre> </div> <div class="column"> - Read [Kuberneticized machine learning and AI using Kubeflow](https://developer.hpe.com/blog/kubernetized-machine-learning-and-ai-using-kubeflow/) <pre><code>- Learn how management of large scale Kubernetes clusters is made easier with [HPE Ezmeral Runtime Enterprise](https://developer.hpe.com/platform/hpe-ezmeral/home/) </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>How do I find and get access to the right data?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Avoiding-pitfalls-Tips-for-better-data-science/ba-p/7144228">Avoiding pitfalls: Tips for better data science</a></p> </li> <li> <p>Better connections between data producers and data consumers make data science more successful. Check out <a href="https://www.hpe.com/us/en/insights/articles/getting-value-from-your-data-shouldn-t-be-this-hard-2106.html">Getting value from your data shouldn’t be this hard</a></p> </li> </ul> <p><strong>What can I do to lower the entry barriers to developing new AI/ML/data science projects?</strong></p> <ul> <li> <p>AI/ML projects can and should be run on the same system as analytics projects: Read “Chapter 3: AI and Analytics Together” in the free eBook <a href="https://www.hpe.com/us/en/resources/software/ai-and-analytics-systems.html">AI and Analytics at Scale: Lessons from Real-World Production Systems</a></p> </li> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Second-project-advantage-Lowering-barriers-to-AI-and-machine/ba-p/7154034#.YZYX2elKhE4">2nd project advantage: lowering barriers to AI and machine learning</a></p> </li> </ul> <p><strong>How do I optimize data logistics and preparation efforts to keep them from overwhelming the data science project?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Budgeting-time-for-AI-ML-projects/ba-p/7090807#.YZYZVelKhE4">Budgeting time for AI/ML projects</a></p> </li> <li> <p>See content for the <a href="/role/data-ml-engineer/home/">Data/ML Engineer role</a></p> </li> </ul> <p><strong>What makes it easier to deal with edge computing in large-scale systems?</strong></p> <ul> <li> <p>Read <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/To-the-edge-and-back-again-Meeting-the-challenges-of-edge/ba-p/7132609#.Ya5X3r3ML0o">To the edge and back again: Meeting the challenges of edge computing</a></p> </li> </ul> <p><strong>How are others doing this?</strong></p> <p>Check out these real-world case studies</p> <ul> <li> <p><a href="https://www.hpe.com/psnow/doc/a50003176enw?jumpid=in_lit-psnow-red">Accelerating Autonomous Car Development with Ready Access to Global Data Fabric</a></p> </li> <li> <p><a href="https://www.hpe.com/psnow/doc/a50003827enw">Accelerating Data Insight for a Better Work Life</a></p> </li> <li> <p><a href="https://developer.hpe.com/blog/architecting-the-worlds-largest-biometric-identity-system-the-aadhaar-ex/">Architecting the World’s Largest Biometric Identity System: The Aadhaar Experience</a></p> </li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Munch &#x26; Learn technology talk</a></p> <div class="row"> <div class="column"> Monthly meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks. </div> <div class="column"> - [What’s a data fabric and how does it work?](https://www.youtube.com/watch?v=qi6sTvu8osk) <pre><code>- [Data Science Unplugged: Part 1](https://www.youtube.com/watch?v=Inh6eXM0EbA) - [Data Science Unplugged: Part 2](https://www.youtube.com/watch?v=Va4tSr__Yok) - [How to make data consumable for real-world data science](https://www.youtube.com/watch?v=4WKjRqflF7M) </code></pre> </div> </div> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> - [HPE Ezmeral Data Fabric 101 – Get to know the basics around the data fabric](https://hackshack.hpedev.io/workshop/26) </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Documentation</div> <div class="row"> <div class="column"> The [HPE Ezmeral Data Fabric platform page](https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/) offers documentation and API information along with informative videos and tutorials. Additional documentation can be found here. </div> <div class="column"> - [HPE Ezmeral Data Fabric 7.0 documentation](https://docs.datafabric.hpe.com/70/index.html) </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> <pre><code>Ping us with your comments, questions, and requests for information. </code></pre> </div> <div class="column"> - [HPE Dev Slack]( https://slack.hpedev.io/) </div> </div>https://developer.hpe.com/developer/home/https://developer.hpe.com/developer/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>Every software developer, computer programmer, software engineer, and coder shares the same goal: to create the best software for customers. This involves user experience, security, serviceability, and reusability, as well as commercial success. In addition to knowing the languages and tools at your disposal, it also requires an understanding of your deployment model - considering the teams in place, the corporate context and the delivery mechanism. And in this quickly evolving environment, it’s not always easy to keep up.</p> <p>Here, we cover the latest software trends, new programming languages, and new techniques and methodologies to help you improve the way you build software for your customers – whether it’s through open source projects or commercial offerings.</p> <p>The role of the Developer may overlap with other roles like the <a href="https://developer.hpe.com/role/data-driven-developer/home/">Data-driven Developer</a> or even the <a href="https://developer.hpe.com/role/data-scientist/home/">Data Scientist</a>. On this page, we focus on developers who build the applications that generate data, which Data Scientists can use to support business outcomes.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **Build the right user experience** <pre><code>*Delivering a consistent, enjoyable user experience (UX) is key to developing successful applications today.* </code></pre> </div> <div class="column"> ### Building a Foundation **Kubernetes** <pre><code>*Open source Kubernetes is the de-facto standard for managing containerized applications at scale. But it can be complex and take time and experience to fully master.* </code></pre> </div> </div> <div class="row"> <div class="column"> - Bringing UX designers into a project early can significantly improve your chances of success. Read [Wow – A practiced and perfected design process.](https://developer.hpe.com/blog/wow-a-practiced-and-perfected-design-process-part-1-uncovering-the-merit/) <pre><code>- Implementing validated designs isn’t always easy. See how [HPE DesignSystem](https://design-system.hpe.design/), based on open source Grommet, can help you get there faster and easier. </code></pre> </div> <div class="column"> - Learn how to migrate an existing application by reading [Deploying Complex Stateful Applications on Kubernetes with KubeDirector](https://developer.hpe.com/blog/deploying-complex-stateful-applications-on-kubernetes-with-kubedirector/) or [Best Practices for Migrating Your Apps to Containers and Kubernetes](https://developer.hpe.com/blog/best-practices-for-migrating-your-apps-to-containers-and-kubernetes/) <pre><code>- Read [Introducing a multi-vendor CSI driver for Kubernetes](https://developer.hpe.com/blog/introducing-a-multi-vendor-csi-driver-for-kubernetes/) and [Using Raw Block and Ephemeral Inline Volumes on Kubernetes](https://developer.hpe.com/blog/using-raw-block-and-ephemeral-inline-volumes-on-kubernetes/) to learn how to leverage the CSI driver for persistent storage for your applications </code></pre> </div> </div> <hr> <div class="row"> <div class="column"> **Containerized apps** <pre><code>*Complex applications can be assembled from small, independent containerized modules with REST API-based interaction. Choose the right platform for your deployment.* </code></pre> </div> <div class="column"> **Zero-Trust Security** <pre><code>*Securing an application in heterogeneous, distributed environments is a daunting task. A CNCF-incubated project, SPIFFE/SPIRE, can help.* </code></pre> </div> </div> <div class="row"> <div class="column"> <pre><code>Modern, cloud-native application development relies heavily on containers and Kubernetes. When determining the right platform for your use, be sure to check out [HPE Ezmeral Runtime Enterprise.](https://developer.hpe.com/platform/hpe-ezmeral/home/) </code></pre> </div> <div class="column"> - [Learn about the concept of Zero-Trust security in the ebook Solving the Bottom Turtle](https://spiffe.io/book/) <pre><code>- [Discover some of the more recently added SPIFFE/SPIRE capabilities](https://developer.hpe.com/blog/top-13-capabilities-within-spiffe-and-spire-released-in-2019/) </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>How can I easily prototype my designs?</strong></p> <ul> <li> <p>Learn about the <a href="https://developer.hpe.com/platform/grommet/home/">open source React framework Grommet</a></p> </li> <li> <p><a href="https://designer.grommet.io/">Test the Grommet Designer </a></p> </li> </ul> <p><strong>How do I best handle parallel programming?</strong></p> <ul> <li> <p>Explore <a href="https://developer.hpe.com/platform/chapel/home/">Chapel</a>, an open source programming language invented by parallel programming engineers from Cray Computer, now part of Hewlett Packard Enterprise</p> </li> <li> <p><a href="https://chapel.discourse.group/latest">Join the Chapel community</a></p> </li> <li> <p>Read about the HPE Cray Programming Environment <a href="https://community.hpe.com/t5/Advantage-EX/Make-your-apps-run-faster-with-HPE-Cray-Programming-Environment/ba-p/7144400?jumpId=in_videogallery_0f352d12-b606-4fae-97db-be760f0fda7a_gaiw">here</a> or follow <a href="https://www.hpe.com/h22228/video-gallery/us/en/4ee817bd-3edc-45db-aed0-ae054d9d8712/take-the-frustration-out-of-hpc-software-development-with-hpe-cray-programming-environment-/video/">this</a> 25mn seminar</p> </li> </ul> <p><strong>How do I secure workloads in distributed environments?</strong></p> <ul> <li> <p>Learn about the CNCF project, <a href="https://developer.hpe.com/platform/spiffe-and-spire-projects/home/">SPIFFE/SPIRE</a>, and how to achieve zero-trust security for your application</p> </li> <li> <p><a href="https://slack.spiffe.io/">Join the SPIFFE/SPIRE community</a></p> </li> </ul> <p><strong>How can I deploy my stateful application on my modern Kubernetes platform?</strong></p> <ul> <li> <p>If migrating a legacy application to Kubernetes is one of your concerns, <a href="https://developer.hpe.com/blog/best-practices-for-migrating-your-apps-to-containers-and-kubernetes/">read about the KubeDirector project</a></p> </li> </ul> <p><strong>How can I focus more about the application, and less about the data?</strong></p> <ul> <li> <p><a href="https://www.hpe.com/us/en/resources/software/ai-and-analytics-systems.html">Read the eBook</a> to learn how to simplify using and sharing your data with a common infrastructure</p> </li> <li> <p>Make sure you check our <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/">HPE Ezmeral Data Fabric platform</a> pages</p> </li> </ul> <p><strong>How do you handle collaboration and sharing of code in your organization?</strong></p> <p>Mastering Git (including GitHub, GitLab, etc.) is a key element of the modern developer toolkit and mandatory for being a contributor or consumer of open source projects. We have designed this series of blog posts to help with Git wizardry:</p> <ul> <li> <p><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-git/">Get started with Git and get involved in the open source community</a></p> </li> <li> <p><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun/">Learn to share with the community using Git</a></p> </li> <li> <p><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-3-contributing-back-to-th/">Start contributing back to the community</a></p> </li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Munch &#x26; Learn technology talk</a></p> <div class="row"> <div class="column"> Monthly meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks. </div> <div class="column"> - [Microservices Architecture 101](https://www.youtube.com/watch?v=qyyxQU37ZyQ) <pre><code>- [Building a foundation for zero trust with SPIFFE](https://www.youtube.com/watch?v=G1ceKr16nn8) - [Kubernetes 101](https://www.youtube.com/watch?v=PWVJKK1obKQ&#x26;feature=youtu.be) </code></pre> </div> </div> <hr> <p><br><br> <a href="https://developer.hpe.com/campaign/meetups/" style="font-weight: 700; font-size: 27px">Meetups</a></p> <div class="row"> <div class="column"> A series of in-depth talks on open source developer technologies. </div> <div class="column"> - [Quarkus - Supersonic Subatomic Java](https://hpe.zoom.us/webinar/register/5716414626617/WN_VS7nBF_qQze0G64XLzBilw) <pre><code>- [Streamlit - The fastest way to build and share data science apps](https://hpe.zoom.us/webinar/register/2016414625150/WN_FzzTDsTjQBSw-UFwD6UTdw) </code></pre> </div> </div> <hr> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> <pre><code>- [Kubernetes 101 - Introduction to Kubernetes Concepts](https://hackshack.hpedev.io/workshop/24) - [SPIFFE - SPIRE 101 – An introduction to SPIFFE server and SPIRE agent security concepts](https://hackshack.hpedev.io/workshop/27) - [API 101 - REST API basics and the value they provide](https://hackshack.hpedev.io/workshop/9) - [Python 101 - Introduction to Python programming language](https://hackshack.hpedev.io/workshop/15) - [Streamline app development with open source Grommet](https://hackshack.hpedev.io/workshop/14) - [RUST 101 - Introduction to the Rust programming language](https://hackshack.hpedev.io/workshop/16) - [GIT 101 – Get involved in the open source community](https://hackshack.hpedev.io/workshop/17) </code></pre> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> Ping us with your comments, questions, and requests for information. We recommend starting with the HPE Developer Slack workspace. </div> <div class="column"> <pre><code>- [HPE Developer Slack](https://slack.hpedev.io/) - [Grommet Slack](https://grommet.slack.com/) - [Chapel Discourse Forum](https://chapel.discourse.group/latest) </code></pre> </div> </div>https://developer.hpe.com/devops-sre/home/https://developer.hpe.com/devops-sre/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>Setting up the software delivery pipeline, streamlining processes, and ensuring it all works is your responsibility. And automation is key to your success. Here, you’ll not only find tips and tricks to help you deliver what’s expected from you but also ideas that will help you expand your capabilities.</p> <p>You may also find content of interest to you on the <a href="/role/data-scientist/home/">Data Scientist</a> and <a href="/role/developer/home/">Developer pages</a>, so be sure to check them out.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **Solve future problems now** <pre><code>*Learn how industry innovation affects you* </code></pre> </div> <div class="column"> ### Building a Foundation **Setting up the pipeline** <pre><code>*Ease operations and workflows* </code></pre> </div> </div> <div class="row"> <div class="column"> - Check out [Infrastructure-as-code on HPE GreenLake using Terraform](/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/) <pre><code>- Read [Kubernetes Cluster as Code - Part 1](/blog/kubernetes-clusters-as-code-part1/) - Better understand [HPE GreenLake Central Identity &#x26; Access Management](/blog/understanding-hpe-greenlake-identity-access-management/) - Explore [From DevOps to AIOps with the power of HPC: Why and how it’s time to make the move](https://community.hpe.com/t5/tech-insights/from-devops-to-aiops-with-the-power-of-hpc-why-and-how-it-s-time/ba-p/7116668#.Y9tcoezMJGM) </code></pre> </div> <div class="column"> - Read [Lift and Transform Apps with HPE CSI Driver for Kubernetes](/blog/lift-and-transform-apps-with-hpe-csi-driver-for-kubernetes/) <pre><code>- Learn more about [Autopilot Kubernetes Deployments on HPE Ezmeral Runtime Enterprise](/blog/autopilot-kubernetes-deployments-on-hpe-ezmeral-runtime-enterprise/) - Check out how to [Scale Kubernetes Clusters using HPE GreenLake Terraform Provider](/blog/scale-kubernetes-cluster-using-hpe-greenlake-terraform-provider/) - Watch [Microservices Architecture 101](https://www.youtube.com/watch?v=qyyxQU37ZyQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;feature=youtu.be&#x26;ab_channel=HewlettPackardEnterprise) </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>Leveraging automation for maximum efficiency, repeatability, and cost effectiveness</strong></p> <ul> <li> <p>Watch <a href="https://www.youtube.com/watch?v=zUo8Ag2IXqk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HewlettPackardEnterprise">HPE GreenLake and Infrastructure-as-Code (IaC)</a></p> </li> <li> <p>Read <a href="/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/">Infrastructure-as-code on HPE GreenLake using Terraform</a></p> </li> <li> <p>Study the technical paper <a href="/blog/infrastructure-as-code-with-hpe-oneview-and-ansible-by-red-hat/">Infrastructure as code with HPE OneView and Ansible by Red Hat</a></p> </li> </ul> <p><strong>Securing and automating app development</strong></p> <ul> <li> <p>Watch <a href="https://www.youtube.com/watch?v=XhhqEjUaLjE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HewlettPackardEnterprise">Running Reliable systems Part 1: An overview of Site Reliability Engineering (SRE)</a></p> </li> <li> <p>Watch <a href="https://www.youtube.com/watch?v=ZDxptOGs-ow&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;ab_channel=HewlettPackardEnterprise">Running Reliable systems Part 2: Service level objective (SLO) math</a></p> </li> <li> <p>View <a href="https://www.youtube.com/watch?v=G1ceKr16nn8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;feature=youtu.be&#x26;ab_channel=HewlettPackardEnterprise">Building a foundation for zero trust with SPIFFE</a></p> </li> <li> <p>Learn about <a href="/blog/galadriel-a-spire-federation-alternative/">Galadriel – a SPIRE Federation Alternative</a></p> </li> <li> <p>Read <a href="/blog/the-advent-of-ephemeral-infrastructure-as-code/">The advent of ephemeral Infrastructure-as-Code</a></p> </li> <li> <p>Check out <a href="/blog/deploying-mongodb-application-on-hpe-greenlake-for-containers/">A guide to deploying MongoDB applications using HPE GreenLake for Containers</a></p> </li> <li> <p>Read how to <a href="/blog/secure-containerized-and-traditional-apps-concurrently/">Secure containerized and traditional apps concurrently</a></p> </li> </ul> <p><strong>Setting up pipelines and streamlining processes</strong></p> <ul> <li> <p>Explore the HPE GreenLake for Compute Ops Management REST API – <a href="/blog/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-curl-and-postman/">using cURL and Postman</a> or by <a href="/blog/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-python-and-powershell/">using Python and PowerShell</a></p> </li> <li> <p>Learn how <a href="https://community.hpe.com/t5/the-cloud-experience-everywhere/big-news-for-devops-teams-new-capabilities-in-hpe-greenlake-for/ba-p/7178793#.Y9te5uzMJGM">HPE GreenLake for Private Cloud Enterprise offers expanded container deployment</a> options for Kubernetes to support DevOps and CI/CD environments</p> </li> </ul> <p><strong>Monitoring and ensuring applications are stable</strong></p> <ul> <li> <p>Review <a href="/blog/greenlake-platform-infrastructure-monitoring-with-apache-skywalking/">HPE GreenLake for Private Cloud Enterprise monitoring with Apache SkyWalking and OpenTelemetry</a></p> </li> <li> <p>Read <a href="/blog/monitor-application-performance-across-hybrid-cloud-environment-using-apache-skywalking-and-service-mesh/">Addressing hybrid cloud application challenges using HPE GreenLake for Private Cloud Enterprise – Part 2: Application monitoring</a></p> </li> <li> <p>Check out <a href="/blog/containers-best-practices-for-running-in-production/">Containers: Best practices for running in production</a></p> </li> </ul> <p><strong>Modernizing and migrating legacy apps</strong></p> <ul> <li> <p>View <a href="https://www.youtube.com/watch?v=UvcyIjzml7s&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;ab_channel=HewlettPackardEnterprise">HPE + vFunction: Modernizing Legacy Applications and Data Sources Faster</a></p> </li> <li> <p>Read <a href="/blog/application-modernization-with-the-application-workbench/">Application modernization with the Application Workbench</a></p> </li> <li> <p>Check out our <a href="/blog/best-practices-for-migrating-your-apps-to-containers-and-kubernetes/">Best practices for migrating your apps to containers and Kubernetes</a></p> </li> <li> <p>See how to <a href="/blog/lift-and-transform-apps-with-hpe-csi-driver-for-kubernetes/">Lift and transform apps with HPE CSI Driver for Kubernetes</a></p> </li> </ul> <p><strong>Case studies</strong></p> <ul> <li> <p><a href="https://www.hpe.com/us/en/collaterals/collateral.a50000267.T%C3%BCbingen-University-Hospital-future-proofs-its-SAP-environment-case-study.html?rpv=cpf&#x26;parentPage=/us/en/greenlake/customer-stories">Tübingen University Hospital future-proofs its SAP environment case study</a></p> </li> <li> <p><a href="https://www.hpe.com/us/en/collaterals/collateral.a50006818.Packaging-giant-builds-platform-for-sustainable-growth--EPL-Limited.html?rpv=cpf&#x26;parentPage=/us/en/customer-case-studies/modal-fragment/e6818">Packaging giant builds platform for sustainable growth – EPL Limited</a></p> </li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Munch &#x26; Learn technology talk</a></p> <div class="row"> <div class="column"> Monthly meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks. </div> <div class="column"> - [Calling all citizen developers: Can low-code platforms accelerate your impact?](https://www.youtube.com/watch?v=zc_54fq8PoY&list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&index=1&ab_channel=HewlettPackardEnterprise) <pre><code>- [Kubernetes 101](https://www.youtube.com/watch?v=PWVJKK1obKQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;feature=youtu.be&#x26;ab_channel=HewlettPackardEnterprise) - [Microservices Architecture 101](https://www.youtube.com/watch?v=qyyxQU37ZyQ&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;feature=youtu.be&#x26;ab_channel=HewlettPackardEnterprise) </code></pre> </div> </div> <hr> <p><br><br> <a href="/campaign/meetups/" style="font-weight: 700; font-size: 27px">Meetups</a></p> <div class="row"> <div class="column"> A series of in-depth talks on open source developer technologies. </div> <div class="column"> <ul> <li> <p><a href="https://www.youtube.com/watch?v=zUo8Ag2IXqk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">HPE GreenLake and Infrastructure-as-Code (IaC)</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=XhhqEjUaLjE&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HewlettPackardEnterprise">Running reliable systems - Part 1: An Overview of Site Reliability Engineering (SRE)</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=ZDxptOGs-ow&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=2&#x26;ab_channel=HewlettPackardEnterprise">Running reliable systems - Part 2: Service level objective (SLO) Math</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=odi9isyZOrU&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;ab_channel=HewlettPackardEnterprise">OpenTelemetry: Getting started and the road to production</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=r62VLwT6w3Y&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;ab_channel=HewlettPackardEnterprise">Finding vulnerabilities in production with open source ThreatMapper</a></p> </li> <li> <p><a href="https://www.youtube.com/watch?v=UvcyIjzml7s&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;index=1&#x26;ab_channel=HewlettPackardEnterprise">HPE + vFunction: Modernizing Legacy Applications and Data Sources Faster</a></p> </li> </ul> </div> </div> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> <ul> <li> <p><a href="/hackshack/workshop/24">Kubernetes 101 – Introduction to Kubernetes Concepts</a></p> </li> <li> <p><a href="/hackshack/workshop/33">Docker 101 – Introduction to Docker Concepts</a></p> </li> <li> <p><a href="/hackshack/workshop/31">Ansible 101 – Introduction to Ansible Concepts</a></p> </li> <li> <p><a href="/hackshack/replays/36">Introduction to Virtual Machine Infrastructure-as-Code using Terraform and HPE GreenLake for Private Cloud Enterprise</a></p> </li> <li> <p><a href="/hackshack/replays/32">Creating a Zero Trust Model for Microservices Architectures with SPIRE and Envoy</a></p> </li> <li> <p><a href="/hackshack/workshop/0">Intro to the HPE Ezmeral Runtime Enterprise REST API</a></p> </li> <li> <p><a href="/hackshack/workshop/9">API 101 - API basics and the value they provide</a></p> </li> <li> <p><a href="/hackshack/workshop/21">StackStorm 101 - Introduction to the StackStorm Framework</a></p> </li> </ul> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Documentation</div> <div class="row"> <div class="column"> <pre><code>The [HPE GreenLake API portal](https://developer.greenlake.hpe.com/) offers documentation and API information along with informative videos and tutorials. </code></pre> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> <pre><code>Ping us with your comments, questions, and requests for information. </code></pre> </div> <div class="column"> <pre><code>- [HPE Developer Community Slack]( https://slack.hpedev.io/) - [HPE GreenLake Test Drive](https://testdrive.greenlake.hpe.com/) </code></pre> </div> </div>https://developer.hpe.com/open-source-advocate/home/https://developer.hpe.com/open-source-advocate/home/<style> .row { display: grid; grid-template-columns: 1fr 1fr; column-gap: 50px; } </style> <p>To stay on the cutting edge, you know your organization needs the agility afforded by open source software and community collaboration. And that’s what you’ll find here. Hewlett-Packard Enterprise (HPE) has always had a culture of open collaboration and giving back to the community. It’s in our DNA. Open source collaboration is the whole idea behind the HPE Developer Community, offering a place where HPE and external engineers can come together to accelerate innovation.</p> <p>Our open source projects today focus on common problems customers are trying to solve; making hybrid IT simple, enabling cloud economics, programming data analyses on laptops through supercomputers, and securing digital infrastructure. But they go beyond providing innovative enterprise solutions to tackling global issues, such as ensuring the right people can access the right data wherever it’s needed to combat disease and hunger. We do so by contributing to projects focused on cloud-native application development, data fabric, and high performance computing (HPC), investing in future growth areas such as AI, machine learning, data science, and edge computing.</p> <p>Explore community-powered innovation here as we highlight a few of the latest open source contributions from HPE. We encourage you to join these community efforts, contributing where and when you can.</p> <br> <hr style="background: #7630EA; height: 5px; border: none"> <br> <div class="row"> <div class="column"> ### Get Inspired **Data science de facto standards emerge** <pre><code>*Explore open standards for AI/ML tool and application development.* </code></pre> </div> <div class="column"> ### Building a Foundation **Secure from the start** <pre><code>*Start with a zero-trust policy where everyone needs to be authenticated and authorized.* </code></pre> </div> </div> <div class="row"> <div class="column"> - Improve scientific simulation efficiency and accuracy at scale using high-performance compute simulations and the open source framework of [SmartSim](https://developer.hpe.com/platform/smartsim/home/) <pre><code>- Accelerate time-to-production with [Determined AI](https://developer.hpe.com/platform/determined-ai/home/), an open source deep learning training platform to build and train models faster and easier </code></pre> </div> <div class="column"> <pre><code>- Use [SPIFFE](https://developer.hpe.com/platform/spiffe-and-spire-projects/home/) (Secure Production Identity Framework for Everyone) to securely authenticate services in dynamic and heterogeneous environments through platform-agnostic, cryptographic identification. This ensures distributed workloads can continuously establish mutual trust and encrypted communications within and across organizational boundaries, especially when communicating over untrusted networks </code></pre> </div> </div> <hr> <div class="row"> <div class="column"> **Prototype-then-port parallel processing apps** <pre><code>*ML/DL and AI apps requiring large-scale simulation often need the power of parallel processing, but access to HPC hardware can be limited.* </code></pre> </div> <div class="column"> **Connect seamlessly to trustworthy data** <pre><code>*Dataspaces provide the building blocks to integrate diverse data sets from multiple distributed owners and gain broader discovery and access with improved governance and trust.* </code></pre> </div> </div> <div class="row"> <div class="column"> <pre><code>- Consider writing your app in [Chapel](https://developer.hpe.com/platform/chapel/home/), which allows you to prototype your app on a laptop and then run it at scale on larger systems </code></pre> </div> <div class="column"> <pre><code>- Create a standard and open approach to meta data, enabling easier cross-vertical discovery and sharing of distributed diverse data sets with [HPE Dataspaces](https://www.hpe.com/us/en/what-is/dataspaces.html) </code></pre> </div> </div> <hr> <div class="row"> <div class="column"> **Apply open source solutions to solve global issues** <pre><code>*In an interconnected world, technology can enhance and protect our existence* </code></pre> </div> <div class="column"> **Modernize the SQL stack for analytics transformations** <pre><code>*Enterprises increasingly require data analytics services with cloud-like operations* </code></pre> </div> </div> <div class="row"> <div class="column"> <pre><code>- [AgStack](https://github.com/agstack) Foundation aims to improve global agriculture efficiency through the creation, maintenance and enhancement of free, reusable, open and specialized digital infrastructure for data and applications </code></pre> </div> <div class="column"> <pre><code>- Current on-premises SQL technologies don’t account for the new requirements around hybrid cloud and scale. [Presto](https://github.com/prestodb/presto) supports SQL runtimes to increase speed </code></pre> </div> </div> <br> <hr style="background: #33DAC8; height: 5px; border: none"> <br> <h3>Addressing Key Concerns</h3> <p><strong>I have data everywhere, including supplier and customer sites. I need to take advantage of it for AI and machine learning applications. What open source technologies can help me do this?</strong></p> <ul> <li>View this video to learn how <a href="https://www.hpe.com/h22228/video-gallery/us/en/505e5730-c199-4ca8-9ee7-d70ba6f6f7fa/dataspaces--connecting-you-to-data-you-can-trust/video/">HPE Dataspaces</a> connects data consumers with data producers offering a clean and trusted exchange of data</li> </ul> <p><strong>How can open, interconnected data access solve global issues?</strong></p> <ul> <li>Explore how HPE’s research partnership with <a href="https://www.cgiar.org/how-we-work/governance/system-organization/">CGIAR System Organization</a>, <a href="https://agstack.org/">The AgStack Foundation</a> and <a href="https://www.digitalgreen.org/">Digital Green</a> enables farmers to find the data they need to advance their agricultural goals in this <a href="https://community.hpe.com/t5/Advancing-Life-Work/Dataspaces-how-an-open-metadata-layer-can-establish-a/ba-p/7149075#.YjUz4hDMLlx">article</a></li> </ul> <p><strong>Where can I find an open source tool that helps me build and train ML models faster and easier?</strong></p> <ul> <li> <p>Read <a href="https://developer.hpe.com/blog/determined-ai-is-joining-hewlett-packard-enterprise/">Determined AI is Joining Hewlett Packard Enterprise</a></p> </li> <li> <p>Learn more about <a href="https://developer.hpe.com/platform/determined-ai/home/">Determined AI</a></p> </li> </ul> <p><strong>My machine-learning application requires the power of parallel processing. But access to HPC machines is limited. Is there a way I can prototype my app on less expensive hardware and then port it over to a cluster or supercomputer?</strong></p> <ul> <li> <p>Explore <a href="https://developer.hpe.com/platform/chapel/home/">Chapel</a>, an open source programming language that aims to solve the “prototype then port” issue, developed by Cray, an HPE company</p> </li> <li> <p>Learn more about <a href="https://chapel-lang.org/">Chapel</a></p> </li> <li> <p><a href="https://chapel.discourse.group/latest">Join the Chapel community</a></p> </li> </ul> <p><strong>How do I secure an application in heterogeneous, distributed environments?</strong></p> <ul> <li> <p>Learn how the CNCF-incubated project, <a href="https://developer.hpe.com/platform/spiffe-and-spire-projects/home/">SPIFFE/SPIRE</a>, can help in the eBook <a href="https://spiffe.io/book/">Solving the Bottom Turtle</a></p> </li> <li> <p>Explore some of the <a href="https://developer.hpe.com/blog/top-13-capabilities-within-spiffe-and-spire-released-in-2019/">top capabilities of SPIFFE/SPIRE</a></p> </li> <li> <p><a href="https://slack.spiffe.io/">Join the SPIFFE/SPIRE community</a></p> </li> </ul> <p><strong>How can I deploy my stateful application on my modern, open Kubernetes platform?</strong></p> <ul> <li> <p>Learn about and contribute to the HPE-sponsored open source project <a href="https://kubedirector.io/">KubeDirector</a></p> </li> <li> <p>Deploy cloud-native and non-cloud-native apps running on bare-metal or virtualized infrastructure, on-premises, on any cloud and at the edge with KubeDirector-powered <a href="https://developer.hpe.com/platform/hpe-ezmeral/home/">HPE Ezmeral Runtime Enterprise</a></p> </li> </ul> <p><strong>How can I manage my systems using open source standards?</strong></p> <ul> <li> <p>Simply and securely manage your data center hardware with the <a href="https://developer.hpe.com/platform/ilo-restful-api/home/">Redfish API ecosystem</a></p> </li> <li> <p>Learn why the <a href="https://www.hpe.com/psnow/doc/4AA6-1727ENW?jumpid=in_lit-psnow-red">industry is transitioning to Redfish</a></p> </li> </ul> <p><strong>Where can I find open source design system and development framework tools to create responsive, accessible mobile-first apps?</strong></p> <ul> <li>Build awesome apps with the HPE-sponsored open source project, <a href="/platform/grommet/home/">Grommet</a></li> </ul> <br> <hr style="background: #FF8300; height: 5px; border: none"> <br> <h3>Skill Up</h3> <p><a href="https://developer.hpe.com/campaign/munch-and-learn/" style="font-weight: 700; font-size: 27px">Munch &#x26; Learn technology talk</a></p> <div class="row"> <div class="column"> Monthly meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks. </div> <div class="column"> - Mithril: Introducing Robust Identities into Istio by integrating with SPIRE <pre><code>- [Golden Age of AI, Dark Ages of AI Infrastructure](https://www.youtube.com/watch?v=ktZFLD-9qgw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY) - [Redfish: Past, Present and Future](https://www.youtube.com/watch?v=Q1Qeb24lpKg&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY) - [How to make data consumable for real-world data science](https://www.youtube.com/watch?v=4WKjRqflF7M&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HPETechnology) - [Building a foundation for zero trust with SPIFFE](https://www.youtube.com/watch?v=G1ceKr16nn8&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY&#x26;ab_channel=HPETechnology) - [What’s a data fabric and how does it work?](https://www.youtube.com/watch?v=qi6sTvu8osk&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY) </code></pre> </div> </div> <hr> <p><br><br> <a href="https://developer.hpe.com/campaign/meetups/" style="font-weight: 700; font-size: 27px">Meetups</a></p> <div class="row"> <div class="column"> A series of in-depth talks on open source developer technologies. </div> <div class="column"> <pre><code>- Decoupled policy enforcement with Open Policy Agent - HPE + vFunction: Modernizing Legacy Applications and Data Sources Faster - [Quarkus - Supersonic Subatomic Java](https://hpe.zoom.us/webinar/register/5716414626617/WN_VS7nBF_qQze0G64XLzBilw) - [Streamlit - The fastest way to build and share data science apps](https://hpe.zoom.us/webinar/register/2016414625150/WN_FzzTDsTjQBSw-UFwD6UTdw) </code></pre> </div> </div> <hr> <hr> <p><br><br> <a href="https://hackshack.hpedev.io/workshops" style="font-weight: 700; font-size: 27px">Workshops-on-Demand</a></p> <div class="row"> <div class="column"> Free, in-depth, hands-on workshops that allow you to explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. </div> <div class="column"> <pre><code>- [Kubernetes 101 - Introduction to Kubernetes Concepts](https://hackshack.hpedev.io/workshop/24) - [SPIFFE - SPIRE 101 – An introduction to SPIFFE server and SPIRE agent security concepts](https://hackshack.hpedev.io/workshop/27) - [API 101 - REST API basics and the value they provide](https://hackshack.hpedev.io/workshop/9) - [Python 101 - Introduction to Python programming language](https://hackshack.hpedev.io/workshop/15) - [Streamline app development with open source Grommet](https://hackshack.hpedev.io/workshop/14) - [RUST 101 - Introduction to the Rust programming language](https://hackshack.hpedev.io/workshop/16) - [GIT 101 – Get involved in the open source community](https://hackshack.hpedev.io/workshop/17) - [Stackstorm 101 – Introduction to the Stackstorm automation features](https://developer.hpe.com/hackshack/workshop/21) - [Jupyter Notebooks 101 – A simple how to on Jupyter Notebooks](https://developer.hpe.com/hackshack/workshop/25) </code></pre> </div> </div> <hr> <p><br><br> <strong>Documentation</strong></p> <div class="row"> <div class="column"> Find documentation, API information, videos and tutorials on our [open source platform pages](/opensource/). </div> <div class="column"> <pre><code>- [Chapel](https://chapel-lang.org/) - [Determined AI](/platform/determined-ai/home/) - [Grommet](/platform/grommet/home/) - [KubeDirector](/platform/hpe-ezmeral/home/) - [SmartSim](/platform/smartsim/home/) - [SPIFFE/SPIRE](/platform/spiffe-and-spire-projects/home/) </code></pre> </div> </div> <hr> <p><br><br></p> <div style="font-weight: 700; font-size: 27px">Engage</div> <div class="row"> <div class="column"> Ping us with your comments, questions, and requests for information. We recommend starting with the HPE Developer Slack workspace. </div> <div class="column"> <pre><code>- [HPE Developer Slack](https://slack.hpedev.io/) - [Grommet Slack](https://grommet.slack.com/) - [Chapel Discourse Forum](https://chapel.discourse.group/latest) </code></pre> </div> </div>https://developer.hpe.com/build-data-pipelines-to-inform-descision-making-processes/home/https://developer.hpe.com/build-data-pipelines-to-inform-descision-making-processes/home/<p>Data is growing in all dimensions, including its importance to a business. Building an end-to-end data analytics pipeline is the only way to connect the dots. Now that data is distributed over multiple sites in different locations with data processing taking place outside the traditional data center environment, you need to find new ways to make information available from the edge to the cloud quickly, securely, and reliably.</p> <p>Today’s advanced data fabric can help you build the data pipelines you need to inform your real-time decision-making processes. When you use <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/If-HPE-Ezmeral-Data-Fabric-is-the-answer-what-is-the-question/ba-p/7092812#.YUjRDmZKj0q">HPE Ezmeral Data Fabric</a>, you can build data pipelines that support modern workloads that demand data to be handled wherever it is – from edge-to-core and edge-to-cloud, in any direction, at any speed, and in any format. Check out the resources found here on the HPE Developer portal to get started. Ping us on <a href="https://slack.hpedev.io/">Slack</a> if you have any questions or need more information.</p> <h2>Documentation and tutorials</h2> <p>The <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/#tutorials">HPE Ezmeral Data Fabric platform page</a> on the HPE Developer portal offers documentation and API information along with informative videos and tutorials.</p> <h2>Munch &#x26; Learn</h2> <p>The Munch &#x26; Learn Technology Talks are monthly community meetups where you can hear from experts on the newest technologies. Catch up on any you may have missed and register for upcoming talks <a href="https://developer.hpe.com/campaign/munch-and-learn/">here</a>. For those interested in data science, you’ll find a number of interesting sessions:</p> <ul> <li><a href="https://www.youtube.com/watch?v=qi6sTvu8osk">What’s a data fabric and how does it work?</a></li> <li><a href="https://www.youtube.com/watch?v=Inh6eXM0EbA">Data Science Unplugged: Part 1</a></li> <li><a href="https://www.youtube.com/watch?v=Va4tSr__Yok">Data Science Unplugged: Part 2</a></li> <li><a href="https://www.youtube.com/watch?v=4WKjRqflF7M">How to make data consumable for real-world data science</a></li> </ul> <h2>Workshops-on-Demand</h2> <p>The <a href="/hackshack/workshops">Workshops-on-Demand</a> are free, Jupyter Notebook-based workshops that offer an in-depth, hands-on learning experience. Explore details of a technology by interacting with it. Designed to fit your schedule, these workshops are available 24/7 – from anywhere at any time. Check out these interesting HPE Ezmeral Data Fabric workshops:</p> <ul> <li><a href="/hackshack/workshop/28">Building a dynamic Machine Learning pipeline with KubeDirector</a></li> <li><a href="/hackshack/workshop/29">Deploying end-to-end machine learning workflows with HPE Ezmeral MLOps</a></li> </ul> <h2>Documentation and tutorials</h2> <p>The HPE Developer blog offers many articles and tutorials to help you learn about data fabric and build data pipelines that deliver you the data you need, where and when you need it. Explore our rich library of articles:</p> <p>(Blog feed)</p>https://developer.hpe.com/design-a-website-with-interactive/home/https://developer.hpe.com/design-a-website-with-interactive/home/https://developer.hpe.com/employ-devlops-practives-to-accelerate/home/https://developer.hpe.com/employ-devlops-practives-to-accelerate/home/https://developer.hpe.com/deploy-apps-across-private-public-and-hybrid/home/https://developer.hpe.com/deploy-apps-across-private-public-and-hybrid/home/https://developer.hpe.com/provision-and-configure-cloud-based-and-on-prem-services/home/https://developer.hpe.com/provision-and-configure-cloud-based-and-on-prem-services/home/https://developer.hpe.com/secure-your-infrastructure/home/https://developer.hpe.com/secure-your-infrastructure/home/https://developer.hpe.com/transform-manual-into-automated/home/https://developer.hpe.com/transform-manual-into-automated/home/https://developer.hpe.com/what-other-use-cases-are-missing/home/https://developer.hpe.com/what-other-use-cases-are-missing/home/https://developer.hpe.com/contribute/https://developer.hpe.com/contribute/<ul> <li> <p>Read instructions found in <a href="https://developer.hpe.com/blog/be-an-hpe-dev-blogger/">Be a Blogger</a>.</p> </li> <li> <p>Create a <a href="https://github.com/signup">Personal GitHub account</a> or sign into your GitHub account.</p> </li> <li> <p>Review tips offered in the &#x3C;a target='_blank'rel="noopener noreferrer" href='<a href="https://github.com/hpe-dev-incubator/hpe-dev-portal/blob/master/docs/ContributorGuide-v2.md&#x27;%3EHPE">https://github.com/hpe-dev-incubator/hpe-dev-portal/blob/master/docs/ContributorGuide-v2.md'>HPE</a> Developer External Contributor Guide</a> regarding the Content Management System (CMS).</p> </li> <li> <p>Download and review the <a href="https://brandcentral.hpe.com/uploads/media/2026/2/writing-style-guide-aug-2025v2-1755635696557-7-1771886948866.pdf">HPE Brand Writing Style Guide</a> for accurate and relevant styling.</p> </li> <li> <p>Click <strong>Start Now</strong> button to connect to our Content Management System to fork our repository and start working on your blog.</p> </li> </ul> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <style> .button { background-color: rgba(23,235,160,1); box-sizing: border-box; color: #000000; font-size: 18px; display: inline-block; padding: 6px 12px; vertical-align: middle; overflow: hidden; text-decoration: none; text-align: center; cursor: pointer; white-space: nowrap; border-radius: 4px; border: none; margin: 0; line-height: 24px; font-weight: 700; } </style> <div class="w3-container w3-center w3-margin-bottom"> <a target='\_blank'rel="noopener noreferrer" href="https://developer.hpe.com/admin"><button type="button" class="button">Start Now</button></a> </div>https://developer.hpe.com/osscontribute/https://developer.hpe.com/osscontribute/<p>In the world of open source, you need to be fluent with <a href="https://git-scm.com/">Git</a>. If you are not yet, we recommend you read the following 3-part blog series:</p> <ul> <li><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-git/">Getting started with Git</a></li> <li><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun/">Sharing with the community</a></li> <li><a href="https://developer.hpe.com/blog/get-involved-in-the-open-source-community-part-3-contributing-back-to-th/">Contributing back</a></li> </ul> <p>We also offer a Workshop-on-Demand that you can take to get hands-on experience with Git. Feel free to register by clicking below:</p> <ul> <li><a href="https://developer.hpe.com/hackshack/workshop/17">Getting started with Git</a></li> </ul> <p>When you get started with open source, it means that you will either contribute to an existing project or create your own project. If you want to create your own project, don’t forget to create a public repo for it. For HPE employees, be aware that there is an open source review process you need to follow before releasing any code to open source. Make sure you check this <a href="https://opensource.corp.hpecorp.net/osrp_process_upstream.html">HPE-only website</a>.</p> <p>The HPE Developer Community has highlighted a number of contributors/maintainers in a series of blog posts. Read about their respective journeys and join the team in our Hall of Fame:</p> <ul> <li><a href="https://developer.hpe.com/blog/open-source-contributor-explains-how-kubedirector-empowers-data-intensive-apps/">Kartik Mathur</a> (KubeDirector)</li> <li><a href="https://developer.hpe.com/blog/becoming-a-linux-kernel-contributor-following-the-journey-of-souptick-joarder/">Souptick Joarder</a> (Linux kernel)</li> <li><a href="https://developer.hpe.com/blog/meet-linux-distinguished-technologist-and-open-source-evangelist-bruno-cornec/">Bruno Cornec</a> (MondoRescue)</li> <li><a href="https://developer.hpe.com/blog/spire-maintainer-agustn-martnez-fay-reveals-his-passion-for-information-/">Agustín Martínez Fayó</a> (SPIRE)</li> <li><a href="https://developer.hpe.com/blog/chapel-technical-lead-brad-chamberlain-opens-up-about-open-source/">Brad Chamberlain</a> (Chapel)<a href="https://developer.hpe.com/blog/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/" title="https://developer.hpe.com/blog/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/"></a></li> <li><a href="https://developer.hpe.com/blog/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/" title="https://developer.hpe.com/blog/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/">Shimrit Yacobi</a> (Grommet)</li> </ul> <p>One last piece of advice. When joining an existing open source project in order to contribute, you will have to make yourself known to other contributors and maintainers. This might take time. Be patient. Start small by documenting issues in the code or in the documentation and propose a solution. You can also review and comment on proposed changes. In most cases, there will be a Slack or a Gitter forum dedicated to the project. Don’t hesitate to join and start a discussion there. Once your name becomes associated with good feedback and proposals, it will be a lot easier for you to contribute code and get it approved and merged.</p> <p>And that’s when it gets really exciting.</p> <p>We estimate that there are 200 million packages on GitHub today, so finding the right place to engage is not that easy. If you are looking for a good project to contribute to, we suggest the following list (in alphabetical order). Enjoy the journey and check out our <strong><a href="https://developer.hpe.com/opensource">Open Source</a></strong> page to learn more about some of the key projects HPE supports.</p> <style> table { display: block; width: 100%; width: max-content; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <table> <thead> <tr> <th>   Name/Repo</th> <th>   Description</th> </tr> </thead> <tbody> <tr> <td><a href="https://github.com/agstack">AgStack</a></td> <td>AgStack consists of an open repository to create and publish models, with free and easy access to public data, interoperable frameworks for cross-project use and topic-specific extensions and toolboxes. It leverages existing technologies, such as agriculture standards (AgGateway, UN-FAO, CAFA, USDA and NASA-AR); public data (Landsat, Sentinel, NOAA and Soilgrids; models (UC-ANR IPM), and open source projects like Hyperledger, Kubernetes, Open Horizon, Postgres, Django and more.</td> </tr> <tr> <td><a href="https://github.com/apache/logging-log4j2">Apache Logging (log4j)</a></td> <td>Apache Log4j is a Java-based logging utility originally written by Ceki Gülcü. It is part of the Apache Logging Services, a project of the Apache Software Foundation.</td> </tr> <tr> <td><a href="https://github.com/apache/airflow">Airflow</a></td> <td>Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.</td> </tr> <tr> <td><a href="https://github.com/arrow-kt/arrow">Arrow</a></td> <td>Arrow is a library for Typed Functional Programming in Kotlin.</td> </tr> <tr> <td><a href="https://github.com/gchq/Bailo">Bailo</a></td> <td>Managing the lifecycle of machine learning to support scalability, impact, collaboration, compliance and sharing.</td> </tr> <tr> <td><a href="https://github.com/apache/calcite">Calcite</a></td> <td>Apache Calcite is a dynamic data management framework.</td> </tr> <tr> <td><a href="https://github.com/chapel-lang/chapel">Chapel</a></td> <td>Chapel is a modern programming language designed for productive parallel computing at scale. Chapel's design and implementation have been undertaken with portability in mind, permitting Chapel to run on multicore desktops and laptops, commodity clusters, and the cloud, in addition to the high-end supercomputers for which it was originally created.</td> </tr> <tr> <td><a href="https://github.com/HewlettPackard/cmf">CMF</a></td> <td>Common metadata framework (CMF) addresses the problems associated with tracking of pipeline metadata from distributed sites and tracks code, data and metadata together for end-to-end traceability.</td> </tr> <tr> <td><a href="https://github.com/hpe-storage/csi-driver">HPE CSI</a></td> <td>A Container Storage Interface (CSI) Driver for Kubernetes. The HPE CSI Driver for Kubernetes allows you to use a Container Storage Provider (CSP) to perform data management operations on storage resources.</td> </tr> <tr> <td><a href="https://github.com/datahub-project/datahub">DataHub</a></td> <td>The Metadata Platform for the Modern Data Stack.</td> </tr> <tr> <td><a href="https://github.com/determined-ai/determined">Determined AI </a></td> <td>Determined is an open source deep learning training platform that makes building models fast and easy.</td> </tr> <tr> <td><a href="https://github.com/apache/drill">Drill</a></td> <td>Apache Drill is a distributed MPP query layer that supports SQL and alternative query languages against NoSQL and Hadoop data storage systems. It was inspired, in part, by Google's Dremel.</td> </tr> <tr> <td><a href="https://github.com/apache/druid">Druid</a></td> <td>Druid is a high performance real-time analytics database. Druid's main value add is to reduce time to insight and action.</td> </tr> <tr> <td><a href="https://github.com/falcosecurity/falco">Falco</a></td> <td>The Falco Project, originally created by Sysdig, is an incubating CNCF open source cloud native runtime security tool. Falco makes it easy to consume kernel events, and enrich those events with information from Kubernetes and the rest of the cloud native stack. Falco can also be extended to other data sources by using plugins. Falco has a rich set of security rules specifically built for Kubernetes, Linux, and cloud-native. If a rule is violated in a system, Falco will send an alert notifying the user of the violation and its severity.</td> </tr> <tr> <td><a href="https://github.com/fluent">Fluent</a></td> <td>Fluentd is a cloud native logging solution to unify data collection and consumption.</td> </tr> <tr> <td><a href="https://github.com/grafana">Grafana</a></td> <td>The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.</td> </tr> <tr> <td><a href="https://github.com/grommet/">Grommet</a></td> <td>A react-based framework that provides accessibility, modularity, responsiveness, and theming in a tidy package.</td> </tr> <tr> <td><a href="https://github.com/istio">Istio</a></td> <td>An open platform to connect, manage, and secure microservices.</td> </tr> <tr> <td><a href="https://github.com/jaegertracing/jaeger">Jaeger</a></td> <td>Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and donated to Cloud Native Computing Foundation. It can be used for monitoring microservices-based distributed systems.</td> </tr> <tr> <td><a href="https://github.com/JuliaLang">Julia</a></td> <td>Julia is a high-level, high-performance, dynamic programming language. While it is a general-purpose language and can be used to write any application, many of its features are well suited for numerical analysis and computational science.</td> </tr> <tr> <td><a href="https://github.com/jupyter/notebook">Jupyter Notebook</a></td> <td>The Jupyter notebook is a web-based notebook environment for interactive computing.</td> </tr> <tr> <td><a href="https://github.com/apache/kafka">Kafka</a></td> <td>Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.</td> </tr> <tr> <td><a href="https://github.com/bluek8s/kubedirector">KubeDirector</a></td> <td>KubeDirector uses standard Kubernetes (K8s) facilities of custom resources and API extensions to implement stateful scaleout application clusters. This approach enables transparent integration with K8s user/resource management and existing K8s clients and tools.</td> </tr> <tr> <td><a href="https://github.com/kubernetes/kubernetes">Kubernetes</a></td> <td>Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides basic mechanisms for deployment, maintenance, and scaling of applications. Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.</td> </tr> <tr> <td><a href="https://github.com/lf-edge">LFEdge</a></td> <td>LF Edge aims to establish an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system.</td> </tr> <tr> <td><a href="https://github.com/torvalds/linux">Linux</a></td> <td>The Linux operating system.</td> </tr> <tr> <td><a href="https://github.com/HewlettPackard/LinuxKI">Linux KI</a></td> <td>The LinuxKI Toolset (or LinuxKI for short) is an opensourced advanced mission critical performance troubleshooting tool for Linux. It is designed to identify performance issues beyond the typical performance metrics and results in faster root cause for many performance issues.</td> </tr> <tr> <td><a href="https://github.com/llvm/llvm-project">LLVM</a></td> <td>The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.</td> </tr> <tr> <td><a href="https://github.com/ODIM-Project/ODIM">ODIM</a></td> <td>Open Distributed Infrastructure Management (ODIM). A bold collaborative open source initiative to bring together a critical mass of infrastructure management and orchestration stakeholders to define and execute the collaborative work in several areas.</td> </tr> <tr> <td><a href="https://github.com/OpenLineage/OpenLineage">OpenLineage</a></td> <td>An Open Standard for lineage metadata collection.</td> </tr> <tr> <td><a href="https://github.com/open-metadata/OpenMetadata">OpenMetaData</a></td> <td>Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.</td> </tr> <tr> <td><a href="https://github.com/open-telemetry">OpenTelemetry</a></td> <td>OpenTelemetry is a vendor-neutral standard way to collect telemetry data for applications, their supporting infrastructures, and services.</td> </tr> <tr> <td><a href="https://github.com/openshmem-org">OpenSHMEM</a></td> <td>OpenSHMEM is an effort to create a specification for a standardized API for parallel programming in the Partitioned Global Address Space. Along with the specification the project is also creating a reference implementation of the API.</td> </tr> <tr> <td><a href="https://github.com/openbmc">OpenBMC</a></td> <td>OpenBMC is a Linux distribution for management controllers used in devices such as servers, top of rack switches or RAID appliances. It uses Yocto, OpenEmbedded, systemd, and D-Bus to allow easy customization for your platform.</td> </tr> <tr> <td><a href="https://github.com/prestodb">PrestoDB</a></td> <td>Presto is a distributed SQL query engine for big data.</td> </tr> <tr> <td><a href="https://github.com/prometheus">Prometheus</a></td> <td>Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.</td> </tr> <tr> <td><a href="https://github.com/rstudio">R Studio</a></td> <td>RStudio is an integrated development environment (IDE) for the R programming language.</td> </tr> <tr> <td><a href="https://github.com/CrayLabs/SmartSim">SmartSim</a></td> <td>SmartSim is a workflow library that makes it easier to use common Machine Learning (ML) libraries, like PyTorch and TensorFlow, in High Performance Computing (HPC) simulations and applications. SmartSim launches ML infrastructure on HPC systems alongside user workloads.</td> </tr> <tr> <td><a href="https://github.com/apache/spark">Spark</a></td> <td>Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.</td> </tr> <tr> <td><a href="https://github.com/spiffe">SPIFFE/SPIRE</a></td> <td>SPIRE (the SPIFFE Runtime Environment) is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms. SPIRE exposes the SPIFFE Workload API, which can attest running software systems and issue SPIFFE IDs and SVIDs to them. This in turn allows two workloads to establish trust between each other, for example by establishing an mTLS connection or by signing and verifying a JWT token. SPIRE can also enable workloads to securely authenticate to a secret store, a database, or a cloud provider service.</td> </tr> <tr> <td><a href="https://github.com/apache/zookeeper">Zookeeper</a></td> <td>ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.</td> </tr> </tbody> </table>https://developer.hpe.com/evangelist/https://developer.hpe.com/evangelist/<p>Are you a fan of HPE Developer? Become an HPE Developer Evangelist and help us spread the word on how developers, data scientists, and IT technologists can write applications and develop integrations for Hewlett Packard Enterprise enabled environments. As an HPE Developer Evangelist, you’ll be the primary HPE Developer contact for your region/country for colleagues, HPE partners and customers. You’ll forward communications to them, alerting them of upcoming meetups, Munch &#x26; Learn sessions and promoting the newest Workshops-on-Demand.</p> <p>Being an HPE Developer Evangelist means you’ll get special attention. When you represent HPE Developer at selected local events, you’ll be supported with swag to help promote how HPE focuses on open, collaborative software development. You’ll be able to propose new subjects for meetups and workshops, influencing our roadmap. And you can rely on the HPE Developer Slack Channel and Newsletter to provide you with the support you need to help answer questions and identify the newest blog posts, workshops, and events to promote.</p> <p>What are you waiting for? Sign up today!</p>https://developer.hpe.com/hackshackhome-1/https://developer.hpe.com/hackshackhome-1/https://developer.hpe.com/hackshackhome-10/https://developer.hpe.com/hackshackhome-10/https://developer.hpe.com/hackshackhome-11/https://developer.hpe.com/hackshackhome-11/https://developer.hpe.com/hackshackhome-12/https://developer.hpe.com/hackshackhome-12/https://developer.hpe.com/hackshackhome-14/https://developer.hpe.com/hackshackhome-14/https://developer.hpe.com/hackshackhome-13/https://developer.hpe.com/hackshackhome-13/https://developer.hpe.com/hackshackhome-15/https://developer.hpe.com/hackshackhome-15/https://developer.hpe.com/hackshackhome-16/https://developer.hpe.com/hackshackhome-16/https://developer.hpe.com/hackshackhome-17/https://developer.hpe.com/hackshackhome-17/https://developer.hpe.com/hackshackhome-18/https://developer.hpe.com/hackshackhome-18/https://developer.hpe.com/hackshackhome-19/https://developer.hpe.com/hackshackhome-19/https://developer.hpe.com/hackshackhome-2/https://developer.hpe.com/hackshackhome-2/https://developer.hpe.com/hackshackhome-20/https://developer.hpe.com/hackshackhome-20/https://developer.hpe.com/hackshackhome-21/https://developer.hpe.com/hackshackhome-21/https://developer.hpe.com/hackshackhome-22/https://developer.hpe.com/hackshackhome-22/https://developer.hpe.com/hackshackhome-23/https://developer.hpe.com/hackshackhome-23/https://developer.hpe.com/hackshackhome-24/https://developer.hpe.com/hackshackhome-24/https://developer.hpe.com/hackshackhome-25/https://developer.hpe.com/hackshackhome-25/https://developer.hpe.com/hackshackhome-26/https://developer.hpe.com/hackshackhome-26/https://developer.hpe.com/hackshackhome-28/https://developer.hpe.com/hackshackhome-28/https://developer.hpe.com/hackshackhome-27/https://developer.hpe.com/hackshackhome-27/https://developer.hpe.com/hackshackhome-29/https://developer.hpe.com/hackshackhome-29/https://developer.hpe.com/hackshackhome-3/https://developer.hpe.com/hackshackhome-3/https://developer.hpe.com/hackshackhome-30/https://developer.hpe.com/hackshackhome-30/https://developer.hpe.com/hackshackhome-31/https://developer.hpe.com/hackshackhome-31/https://developer.hpe.com/hackshackhome-32/https://developer.hpe.com/hackshackhome-32/https://developer.hpe.com/hackshackhome-33/https://developer.hpe.com/hackshackhome-33/https://developer.hpe.com/hackshackhome-34/https://developer.hpe.com/hackshackhome-34/https://developer.hpe.com/hackshackhome-35/https://developer.hpe.com/hackshackhome-35/https://developer.hpe.com/hackshackhome-36/https://developer.hpe.com/hackshackhome-36/https://developer.hpe.com/hackshackhome-37/https://developer.hpe.com/hackshackhome-37/https://developer.hpe.com/hackshackhome-38/https://developer.hpe.com/hackshackhome-38/https://developer.hpe.com/hackshackhome-39/https://developer.hpe.com/hackshackhome-39/https://developer.hpe.com/hackshackhome-4/https://developer.hpe.com/hackshackhome-4/https://developer.hpe.com/hackshackhome-40/https://developer.hpe.com/hackshackhome-40/https://developer.hpe.com/hackshackhome-41/https://developer.hpe.com/hackshackhome-41/https://developer.hpe.com/hackshackhome-42/https://developer.hpe.com/hackshackhome-42/https://developer.hpe.com/hackshackhome-43/https://developer.hpe.com/hackshackhome-43/https://developer.hpe.com/hackshackhome-44/https://developer.hpe.com/hackshackhome-44/https://developer.hpe.com/hackshackhome-45/https://developer.hpe.com/hackshackhome-45/https://developer.hpe.com/hackshackhome-46/https://developer.hpe.com/hackshackhome-46/https://developer.hpe.com/hackshackhome-47/https://developer.hpe.com/hackshackhome-47/https://developer.hpe.com/hackshackhome-48/https://developer.hpe.com/hackshackhome-48/https://developer.hpe.com/hackshackhome-49/https://developer.hpe.com/hackshackhome-49/https://developer.hpe.com/hackshackhome-5/https://developer.hpe.com/hackshackhome-5/https://developer.hpe.com/hackshackhome-50/https://developer.hpe.com/hackshackhome-50/https://developer.hpe.com/hackshackhome-51/https://developer.hpe.com/hackshackhome-51/https://developer.hpe.com/hackshackhome-52/https://developer.hpe.com/hackshackhome-52/https://developer.hpe.com/hackshackhome-53/https://developer.hpe.com/hackshackhome-53/https://developer.hpe.com/hackshackhome-54/https://developer.hpe.com/hackshackhome-54/https://developer.hpe.com/hackshackhome-55/https://developer.hpe.com/hackshackhome-55/https://developer.hpe.com/hackshackhome-56/https://developer.hpe.com/hackshackhome-56/https://developer.hpe.com/hackshackhome-57/https://developer.hpe.com/hackshackhome-57/https://developer.hpe.com/hackshackhome-58/https://developer.hpe.com/hackshackhome-58/https://developer.hpe.com/hackshackhome-59/https://developer.hpe.com/hackshackhome-59/https://developer.hpe.com/hackshackhome-6/https://developer.hpe.com/hackshackhome-6/https://developer.hpe.com/hackshackhome-60/https://developer.hpe.com/hackshackhome-60/https://developer.hpe.com/hackshackhome-7/https://developer.hpe.com/hackshackhome-7/https://developer.hpe.com/hackshackhome-8/https://developer.hpe.com/hackshackhome-8/https://developer.hpe.com/hackshackhome-9/https://developer.hpe.com/hackshackhome-9/https://developer.hpe.com/hackshackhome/https://developer.hpe.com/hackshackhome/https://developer.hpe.com/hpediscover2022/https://developer.hpe.com/hpediscover2022/https://developer.hpe.com/munchlearn/https://developer.hpe.com/munchlearn/https://developer.hpe.com/treasurehunt/https://developer.hpe.com/treasurehunt/https://developer.hpe.com/workshop/https://developer.hpe.com/workshop/https://developer.hpe.com/hpe-private-cloud-ai-interact-with-sql-database-using-natural-language/https://developer.hpe.com/hpe-private-cloud-ai-interact-with-sql-database-using-natural-language/Thu, 23 Apr 2026 11:35:26 GMT<p>In today's data-driven world, two fundamental languages enable us to interact with information: <strong>Natural Language</strong> and <strong>SQL (Structured Query Language)</strong>. Natural language is the way humans naturally communicate. It allows us to express ideas, ask questions, and convey intentions effortlessly, whether through speech or text. On the other hand, <strong>SQL</strong> is a specialized language designed for managing and querying structured data stored in databases. While SQL is powerful for precise data retrieval, it often requires technical expertise and familiarity with database schemas.</p> <p>Bridging the gap between these two languages is crucial for making data accessible and actionable. This is where <strong>generative AI (GenAI)</strong> comes into play. By leveraging advanced artificial intelligence (AI) models, we can translate natural language queries into SQL commands, enabling anyone, regardless of technical background to unlock valuable insights from complex, structured datasets. Using GenAI to interpret and generate queries democratizes data analysis, accelerates decision-making, and helps organizations harness their data's full potential for strategic advantage.</p> <p>This blog post walks you through steps to deploy and configure various tools, required to demonstrate NL to SQL use case (using manufacturing dataset) on <strong>HPE Private Cloud AI</strong>. By leveraging these technologies, organizations can unlock valuable insights from their data.</p> <h2>HPE Private Cloud AI</h2> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (HPE PCAI)</a> offers a comprehensive, turnkey AI solution designed to address key enterprise challenges, from selecting the appropriate large language models (LLMs) to efficiently hosting and deploying them. Beyond these core functions, HPE Private Cloud AI empowers organizations to take full control of their AI adoption journey by offering a curated set of pre-integrated <em><a href="https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/">NVIDIA Inference Microservices (NIM)</a></em> LLMs, along with a powerful suite of AI tools and frameworks for data engineering, analytics, and data science.</p> <p><strong><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie112hen_us&#x26;page=About/aie-overview.html">HPE AI Essentials</a></strong> is a software and data foundation layer designed to accelerate the development, deployment, and management of artificial intelligence (AI) and GenAI applications. It is part of the <strong>HPE Private Cloud AI</strong> portfolio and provides a curated, ready-to-run suite of open-source and proprietary tools, enabling organizations to move from AI pilots to production quickly.</p> <h2>Architecture</h2> <p><img src="/img/screenshot-2026-04-23-151822.png" alt=""></p> <h3>Prerequisites</h3> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>HPE AI Essentials version 1.12+, which has PrestoMCP</li> <li>OpenWebUI version v0.6.31+, which supports the MCP server as an external tool.</li> <li>Role 'Private Cloud AI Administrator' assigned to the user. This is required to import PostgreSQL framework.</li> <li>Hugging Face user access token, to download the LLM.</li> </ul> <h2>Prepare data source</h2> <p>You may connect to your existing database to HPE PCAI using Data Services Connector, available in HPE AI Essentials. Otherwise, you may choose to deploy a database using <em>Import Framework</em> feature, described in next section.</p> <h3>Deploy database and load data</h3> <p>A sample Helm chart for PostgreSQL is available in the GitHub repo, <a href="https://github.com/ai-solution-eng/frameworks/tree/main/postgresql">ai-solution-eng/frameworks.</a></p> <p>Deploy PostgreSQL on HPE Private Cloud AI using <em>Import Framework</em></p> <p><img src="/img/screenshot-2026-04-23-184640.png" alt=""></p> <p>Provide the Name, Description and Icon (from GitHub repo).</p> <p><img src="/img/screenshot-2026-04-23-184320.png" alt=""></p> <p>Upload the Helm chart that has been downloaded from GitHub.</p> <p><img src="/img/screenshot-2026-04-23-184131.png" alt=""></p> <p>Update the PostgreSQL password in line #38.</p> <p><img src="/img/screenshot-2026-04-23-184251.png" alt=""></p> <p>Review and submit.</p> <p><img src="/img/screenshot-2026-04-23-184352.png" alt=""></p> <p>Within a few minutes, you'll find that the PostgreSQL framework is in 'Ready' state.</p> <h3>Load database file</h3> <p>Use the script, <em><a href="https://github.com/ai-solution-eng/ai-solution-demos/blob/main/nl-to-sql-mcp-manufacturing/create_manufacturing_data.py">create_manufacturing_data.py</a></em> to create a sample data. Execute the script to create a new ".db" file, which will be used in the following steps.</p> <pre><code class="language-shell">python ./create_manufacturing_data.py </code></pre> <p>On HPE AI Essentials, open your Jupyter notebook server and upload the generated ".db" file and *<a href="https://github.com/ai-solution-eng/ai-solution-demos/blob/main/nl-to-sql-mcp-manufacturing/loaddata.py">loaddata.py </a>*script. You will need to update the password (that was provided while Importing Postgres), in line #9 of <em>loaddata.py</em> script.</p> <p>After uploading these two files, open a terminal in the Jupyter notebook Server. Execute the following commands,</p> <pre><code class="language-shell">pip install psycopg2 python loaddata.py </code></pre> <h3>Connect database to HPE AI Essentials</h3> <ul> <li> <p>Navigate to <strong>Data Engineering > Data Sources</strong></p> </li> <li> <p>In <strong>Structured Data</strong> click on <strong>Add New Data Source</strong></p> </li> <li> <p>Select <strong>PostgreSQL</strong> and click <strong>Create Connection</strong></p> </li> <li> <p>Fill in the details as below:</p> <ul> <li>Name* : manufacturingdb</li> <li>Connection URL*: jdbc:postgresql://postgresql.postgres.svc.cluster.local:5432/manufacturing</li> <li>Connection User*: postgres</li> <li>Connection Password*: "YOUR_DEFINED_PASSWORD"</li> <li>Click on 'PostgreSQL Advanced Settings'</li> <li>Case Insensitive Name Matching: Tick</li> </ul> </li> </ul> <p><img src="/img/screenshot-2026-04-23-191659.png" alt=""></p> <h3>Explore the data catalog</h3> <p>Once the data is loaded and database is connected, you can explore the available data via the Data Catalog.</p> <ul> <li>Navigate to <strong>Data Engineering > Data Catalog</strong></li> <li>Select <strong>manufacturingdb</strong></li> <li>Tick <strong>public</strong> schema</li> <li>See three tables, machine_metrics, machines and operators.</li> <li>Select one of the table, click <strong>Data Preview</strong> to get an overview of the data.</li> </ul> <p><img src="/img/screenshot-2026-04-23-153224.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-153549.png" alt=""></p> <h2>LLM deployment</h2> <p>Deploy the <strong>Qwen/Qwen3-8B-Instruct</strong> LLM using HPE MLIS (Machine Learning Inference Software) framework, available in HPE AI Essentials. You may replace Qwen/Qwen3-8B-Instruct with any other model of your choice.</p> <p>In HPE MLIS, <strong>packaged models</strong> -> create packaged model.</p> <p><img src="/img/screenshot-2026-04-23-154513.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-154632.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-154714.png" alt=""></p> <p>Under 'Advanced' tab, set the following,</p> <p><strong>Environment Variables:</strong></p> <p>HUGGING_FACE_HUB_TOKEN: &#x3C;&#x3C;**Your Hugging Face Token**>></p> <p><strong>Arguments:</strong> <em>--model Qwen/Qwen3-8B --enable-reasoning --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --port 8080</em></p> <p><img src="/img/screenshot-2026-04-23-200031.png" alt=""></p> <p>After creating the packaged model, you may deploy the model.</p> <p><strong>Deployments</strong> -> Create deployment</p> <p><img src="/img/screenshot-2026-04-23-200126.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-200220.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-200318.png" alt=""></p> <p><img src="/img/screenshot-2026-04-23-200419.png" alt=""></p> <p>After the model gets deployed, go to <strong>HPE AI Essentials -> GenAI -> Model Endpoints.</strong></p> <p>Click on 'Action -> Generate API Token'. Copy the Model Endpoint and the API token to a text file, as you will need them in next steps.</p> <p><img src="/img/screenshot-2026-04-23-161519.png" alt=""></p> <h2>Chat interface</h2> <p>Use the Open WebUI framework as the chat interface. Follow the steps to configure Open WebUI</p> <h4>Connect to LLM</h4> <ul> <li>Open Open WebUI</li> <li>Navigate to <strong>Admin Panel >> Settings >> Connections</strong></li> <li>Add new 'OpenAI API' connection -- provide the Model Endpoint URL and API token saved in previous step.</li> </ul> <p><img src="/img/screenshot-2026-04-23-161912.png" alt=""></p> <h4>Configure MCP server</h4> <p>Navigate in Open WebUI to <strong>Admin Panel >> Settings >> External Tools</strong></p> <ul> <li>Add a new Tool Server</li> <li>Change the type to MCP Streamable HTTP</li> <li>Add the URL of ezPresto MCP server, this can be retrieved from Data Engineering >> Data Sources >> MCP server. Open the menu and click on copy endpoint. </li> <li>Add the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie112hen_us&#x26;page=Security/k8s-secret-auth-token.html">JWT token</a> of the user you want to have the Presto Connections. The MCP agent gets access to all datasources this user has access to.</li> <li>Provide the ID and Name, PrestoMCP</li> </ul> <p><img src="/img/screenshot-2026-04-23-162520.png" alt=""></p> <p>As final step in Open WebUI, you need to <strong>Create a Model</strong> that is leveraging the <strong>Qwen3 8b</strong> base model and has the MCP Server as a Tool.</p> <ul> <li>Click in OpenWebUI on Workspace. Click <strong>New Model</strong>.</li> <li>Provide your Model a Name for example <em>Manufacturing</em> select Qwen/Qwen3-8B as Base Model</li> <li>Edit Visibility to <em>Public</em> in case you want the model to be available for everyone to chat</li> <li>Add a System Prompt for example: "Always use 'manufacturingdb' catalog and the schema 'public' for SQL queries. Syntax: catalog.schema.table is how you reference a table in presto"</li> <li>Click on the Advance Params and set <em>Function Calling</em> to <em>Native</em>, otherwise it will only make one call. You can edit this within your chat as well. Underneath Tools tick the <em>PrestoMCP Tool</em> and click Save&#x26;Create.</li> </ul> <p><img src="/img/screenshot-2026-04-23-163728.png" alt=""></p> <h2>Natural Language to SQL</h2> <p>In Open WebUI, click on 'New Chat', select 'Manufacturing' and enable the 'PRESTOMCP' tool inside the chat interface.</p> <p><img src="/img/screenshot-2026-04-23-164049.png" alt=""></p> <p>Using chat, you may now interact with the SQL database using natural language.</p> <p><img src="/img/screenshot-2026-04-23-190106.png" alt=""></p> <h2>Conclusion</h2> <p>By using the tools in HPE AI Essentials; HPE MLIS's robust model management, PrestoMCP and Open WebUI's intuitive chat interface, one can create a powerful ecosystem for transforming natural language queries into actionable insights. This comprehensive approach democratizes data access, allowing users to effortlessly interact with complex datasets through conversational interfaces.</p> <p>By enabling natural language to SQL translation, organizations can unlock the full potential of their data which accelerates decision-making, fostering data-driven culture, and gaining valuable insights without requiring deep technical expertise. As AI and data technologies continue to evolve, such implementations will become essential tools for making data more accessible, understandable, and impactful across all levels of an organization.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more guides and best practices on leveraging HPE Private Cloud AI for your AI use cases.</p>https://developer.hpe.com/chapel-project-seeks-new-funding/https://developer.hpe.com/chapel-project-seeks-new-funding/Tue, 21 Apr 2026 17:11:24 GMT<p>external blog</p>https://developer.hpe.com/redefining-storage-operations-with-ai-and-mcp/https://developer.hpe.com/redefining-storage-operations-with-ai-and-mcp/Sun, 19 Apr 2026 19:15:10 GMT<h3>The shift from scripts to conversations in storage management</h3> <p>Interacting with storage infrastructure today often means navigating dashboards, writing scripts, or manually stitching together API calls. While powerful, these approaches can slow down operations and create a gap between intent and execution, especially when quick insights or actions are needed.</p> <p>What if you could request:</p> <p><em>Which arrays are running low on capacity?</em></p> <p><em>Create a volume for this workload.</em></p> <p><em>And have those actions carried out reliably?</em></p> <p>In this post, I’ll show you how to use the Model Context Protocol (MCP) with <strong>Visual Studio Code</strong> and <strong>GitHub Copilot</strong> to enable natural language interaction with storage systems. By leveraging Data Service Cloud Console (DSCC)'s open API specification, you can expose storage operations as AI-understandable capabilities, turning everyday management tasks into simple, conversational workflows.</p> <h2>From queries to actions: AI-driven storage control</h2> <h3>The foundation: Why OpenAPI matters</h3> <p>Before MCP and AI came into the picture, there is a critical enabler, a strong OpenAPI specification.</p> <p>The well-designed OpenAPI spec provides:</p> <ul> <li>Clearly defined endpoints</li> <li>Structured request/response schemas</li> <li>Consistent naming and semantics</li> </ul> <p>This structure is what makes it possible to:</p> <ul> <li>Programmatically understand capabilities</li> <li>Automatically generate tools</li> <li>Reliably map user intent to actions</li> </ul> <p>Without a solid API foundation, exposing capabilities to AI would be inconsistent and error-prone.</p> <h3>The shift: From APIs to intent-driven operations</h3> <p>Most storage platforms already provide rich APIs. However, APIs are typically structured and require users to:</p> <ul> <li>Understand endpoints and payloads</li> <li>Refer to documentation</li> <li>Write and execute requests</li> </ul> <p><img src="/img/hdd.png" alt="" title="High level workflow"></p> <p>MCP serves as the translation layer, converting human intent into executable API calls.</p> <p><strong>Note:</strong> While this blog article demonstrates the setup using <strong>Visual Studio Code</strong> and <strong>GitHub Copilot</strong>, MCP clients are not limited to Visual Studio Code. Any compatible AI client can interact with the MCP server. For more information, refer to this <a href="https://github.com/ivo-toby/mcp-openapi-server">page</a>.</p> <h3>Setting up MCP with Visual Studio Code</h3> <p>To get started, you configure an MCP server locally and connect it to your API layer. Before setting up and using the MCP server, ensure the following prerequisites are met:</p> <ul> <li>Install Node.js (version 18.0 or later)</li> <li>Install and sign in to GitHub Copilot with an active license</li> </ul> <h4>Step 1: Configuring an Model Context Protocol (MCP) server</h4> <p>In Visual Studio Code, start by opening an empty folder. Inside this folder, create a ‘.vscode’ directory to store your workspace configurations. Within the .vscode folder, add a JSON configuration file with the following details:</p> <pre><code class="language-json">{ "servers": { "fleet-openapi": { "type": "stdio", "command": "npx", "args": [ "-y", "@ivotoby/openapi-mcp-server", "--api-base-url", "https://fleetscale-app.qa.cds.hpe.com", "--headers", "Authorization:Bearer BEARER_TOKEN", "--openapi-spec", "https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.yaml", "--name", "fleet-openapi", "--tools", "all", "--tag", "storage-systems", "--tag", "headroom", "--tag", "capacity", "--tag", "performance", ] } } } </code></pre> <p>Generate the access token as mentioned in this <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">blog</a>. Replace the BEARER_TOKEN with the generated access token. Bearer tokens are short-lived and security-scoped; they should not be treated as static secrets. When a token expires, update it in the JSON file, reload the Visual Studio Code window, and restart the MCP server.</p> <ul> <li><strong>server-name</strong> → A unique identifier for the MCP server instance.</li> <li><strong>type</strong> → Specifies how the MCP server communicates (e.g., stdio, http).</li> <li><strong>command</strong> → The executable used to start the MCP server (e.g., npx).</li> <li><strong>args</strong> → Command-line arguments passed to the MCP server at startup.</li> <li><strong>api-base-url</strong> → Base endpoint of the backend service that the MCP server will call.</li> <li><strong>openapi-spec</strong> → Path to the OpenAPI file used to generate tools.</li> <li><strong>headers</strong> → HTTP headers (like authentication tokens) sent with API requests.</li> <li><strong>tools</strong> → Controls which capabilities (APIs/actions) are exposed to the AI.</li> <li><strong>"tools": "all"</strong> → Exposes all available tools without filtering.</li> <li><strong>tag</strong> → Filters tools based on OpenAPI tags (logical grouping).</li> </ul> <h5>Restricting Operations for Safer Execution:</h5> <p>Since MCP enables AI-driven interaction with your storage systems, it’s important to control what actions are allowed, especially in production environments.</p> <p>Using the <strong>--operation</strong> flag, you can restrict the MCP server to specific HTTP methods.</p> <pre><code class="language-json">{ "servers": { "fleet-openapi": { "type": "stdio", "command": "npx", "args": [ "-y", "@ivotoby/openapi-mcp-server", "--api-base-url", "https://fleetscale-app.qa.cds.hpe.com", "--headers", "Authorization:Bearer BEARER_TOKEN", "--openapi-spec", "https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.yaml", "--name", "fleet-openapi", "--tools", "all", "--operation", "get", "--tag", "storage-systems", "--tag", "headroom", "--tag", "capacity", "--tag", "performance", ] } } } </code></pre> <p>This limits MCP to read-only APIs, preventing any create, update, or delete actions.</p> <p>This is useful for:</p> <ul> <li>Safe initial adoption</li> <li>Monitoring and reporting use cases</li> <li>Security-sensitive environments</li> </ul> <p>You can gradually enable additional operations as needed, with proper guardrails in place.</p> <p>This configuration starts the MCP server locally, loads the OpenAPI specification, and exposes the APIs as MCP tools. It also passes the required authentication headers for API calls. To apply the changes, reload the Visual Studio Code window using <strong>Ctrl + Shift + P → “Developer: Reload Window.”</strong></p> <p><strong>Note:</strong> Ensure your organization approves the use of MCP and AI tools, and that all integrations comply with internal security and cybersecurity policies before use.</p> <h4>Step 2: Connect with GitHub Copilot and start the MCP server</h4> <p>In Visual Studio Code, install the GitHub Copilot extension and sign in with your GitHub account to start using it.</p> <p>To start the MCP server, open the Command Palette (<strong>Ctrl+Shift+p</strong>) and select ‘<em><strong>MCP: List Servers</strong></em>’. You should see the server configured in your JSON file (for example, <em>fleet-openapi</em>) listed there.</p> <p><img src="/img/2listservers.png" alt="" title="List Servers"></p> <p>Select the configured server and click “<em><strong>Start Server</strong></em>”. Once the server is up and running, you should see a confirmation in the console similar to the example below.</p> <p><img src="/img/3serverstart.png" alt="" title="MCP server running"></p> <p>The GitHub Copilot extension allows you to choose from supported LLMs based on your subscription and configuration, as shown below.</p> <p><img src="/img/4listllms.png" alt="" title="List the LLM models"></p> <p>GitHub Copilot can leverage the tools discovered by the MCP extension in Visual Studio Code. You can also view the discovered tools using the Command Palette (<strong>Ctrl + Shift + P</strong>) by searching for MCP-related commands.</p> <p><img src="/img/5listtools.png" alt="" title="List the tools"></p> <h4>Step 3: Interacting using natural language</h4> <p>Once everything is set up, you can begin issuing natural language queries.</p> <p>Examples:</p> <ul> <li>List all storage arrays</li> <li>Identify arrays above 80% capacity</li> <li>Create a 100GB volume on a specific array</li> <li>Check overall system health</li> </ul> <p>These queries are interpreted, mapped to MCP tools, and executed via DSCC APIs.</p> <p>For example, consider the following prompt where the user asks to list storage system details:</p> <p><img src="/img/6liststoragesystems.png" alt="" title="List storage systems-part1"></p> <p><img src="/img/7liststoragesystems.png" alt="" title="List Storage systems-part2"></p> <p><strong>Note:</strong> The behaviour and responses may vary depending on the LLM used, as different models can interpret prompts and execute actions with varying accuracy and reasoning.</p> <h5>What happens behind the scenes</h5> <ul> <li>The user enters a natural language query (in this case, requesting storage system details)</li> <li>Copilot interprets the intent and identifies the relevant operation (e.g., list storage systems)</li> <li>The MCP client discovers available tools from the MCP server and selects the most relevant one based on the intent.</li> <li>The MCP server maps the request to the corresponding API (derived from the OpenAPI specification) and executes it via DSCC, passing the required authentication headers configured in the server JSON.</li> <li>The API returns a structured response, which is then relayed back through the MCP server.</li> <li>Structured response is returned</li> <li>The response is presented to the user, often enriched with additional insights (for example, highlighting potential issues such as performance headroom overutilization).</li> </ul> <p>This removes the need to manually:</p> <ul> <li>Write API calls</li> <li>Look up documentation</li> <li>Handle request formatting</li> <li>Parse responses</li> </ul> <p><strong>Important:</strong> Please ensure that the user does not have admin or superuser privileges, as the project is still in its early stages and requires further enhancements in RBAC and guardrails. For safer execution, use the <em>operation</em> tag appropriately.</p> <h4>Relationship to HPE GreenLake MCP documentation</h4> <p>The DSCC MCP integration follows the same core patterns as the GreenLake MCP setup.</p> <p>You can refer to the GreenLake documentation for:</p> <ul> <li>MCP server setup</li> <li>Tool definitions</li> <li>Request/response flow</li> </ul> <p>HPE GreenLake MCP references can be found <a href="https://developer.greenlake.hpe.com/docs/greenlake/mcp-server/public">here</a>.</p> <p>While DSCC APIs and authorization differ, the overall MCP concepts and configuration remain largely the same and can be reused with minimal changes.</p> <h4>Conclusion: From APIs to conversation</h4> <p>What makes this transformation possible is not just MCP or AI, it’s the combination of a strong OpenAPI foundation and a protocol that can leverage it effectively. By layering MCP on top of a well-defined API ecosystem like DSCC, you can unlock a new interaction model where users express intent and systems handle execution.</p> <p>With tools like Visual Studio Code and GitHub Copilot, users can now manage storage systems using natural language, reducing complexity and speeding up operations.</p> <h4>Key takeaway</h4> <p>Great APIs don’t just enable integrations—they enable entirely new ways of interaction.</p> <p>And MCP doesn’t replace APIs—it enhances how we interact with them.</p> <p>It does so by combining:</p> <ul> <li>OpenAPI specifications</li> <li>MCP server</li> <li>AI assistants</li> </ul> <p>This enables a shift from script-driven operations to intent-driven interactions.</p> <h4>Call to action:</h4> <ul> <li>Start by exploring your existing OpenAPI specifications. Identify a few key operations and expose them through an MCP server. Try interacting with them using natural language in Visual Studio Code with GitHub Copilot.</li> <li>If you have questions or want to explore deeper integrations, feel free to reach out <a href="anusha.y@hpe.com">anusha.y@hpe.com</a> or continue experimenting. This is just the beginning of conversational infrastructure.</li> <li>Please check out the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more articles on this topic.</li> </ul>https://developer.hpe.com/integrating-dagster-as-a-modern-data-orchestration-framework-in-hpe-private-cloud-ai/https://developer.hpe.com/integrating-dagster-as-a-modern-data-orchestration-framework-in-hpe-private-cloud-ai/Sun, 19 Apr 2026 06:11:38 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (PCAI)</a> provides a curated set of pre‑integrated orchestration and machine‑learning (ML) frameworks, including <em>Airflow</em>, <em>Kubeflow</em>, <em>Spark</em> and <em>Ray</em>, to streamline the development and operationalization of AI workloads. However, teams that require stronger data‑centric orchestration, asset lineage, and reproducibility may find gaps in the existing toolchain. Traditional task‑based orchestrators such as <em>Airflow</em> don’t always provide the asset‑level visibility, modularity, or developer‑friendly workflow needed for modern data engineering practices.</p> <p>This blog post introduces <em>Dagster</em> as an additional, asset‑oriented orchestration framework that augments the existing HPE Private Cloud AI tool stack without replacing any component. Using the <em>Import Framework</em>, <em>Dagster</em> can be deployed within minutes and integrated seamlessly into the HPE Private Cloud AI environment. <em>Dagster</em>’s modular architecture cleanly separates the infrastructure layer from the user code layer, allowing the user code package to be built as an independent container image, pushed to the local PCAI image registry, and deployed entirely within the HPE Private Cloud AI boundary. This integration approach reinforces data protection and supports strict data sovereignty requirements. Once integrated, <em>Dagster</em> provides advanced orchestration capabilities, including asset-level lineage tracking, deterministic reproducibility, and comprehensive observability. Its developer‑focused design enables teams to build, execute, and monitor data assets with greater reliability and maintainability, making it a valuable optional addition within HPE Private Cloud AI for workflows that benefit from modern data‑centric orchestration.</p> <h3>Why Dagster?</h3> <p><a href="https://dagster.io/"><em>Dagster</em></a> is a modern data orchestration framework centered on the concept of data assets. Rather than focusing primarily on tasks, <em>Dagster</em> encourages teams to model pipelines as interconnected datasets with explicit lineage and dependencies. Its strong developer oriented features, such as type‑checking, local development tooling, built‑in testing, and rich observability, make it especially effective for contemporary data stacks involving <a href="https://www.getdbt.com/"><em>dbt</em></a>, cloud warehouses, and ML workflows. It’s widely used by data engineering and analytics teams to make pipelines more reliable, observable, and maintainable.</p> <p><em>Dagster</em> does not replace existing orchestrators such as <em>Airflow</em>, <em>Kubeflow</em>, or <em>Ray</em>. Instead, it complements them by offering a more data‑centric option for teams that prioritize lineage, observability, and long‑term maintainability. In practice, <em>Dagster</em>, <em>Airflow</em>, and other orchestration tools often coexist within the same ecosystem, each bringing strengths suited to different workflow styles. <em>Dagster</em> excels in asset‑driven, lineage‑aware environments, while tools like <em>Airflow</em> remain reliable choices for task‑driven scheduling with extensive integrations. Rather than competing, these tools work together to give teams the flexibility to choose the orchestration model that best fits each part of their data platform.</p> <p>The following sections describe the process for integrating <em>Dagster</em> into HPE Private Cloud AI using the <em>Import Framework</em>. Once integrated, <em>Dagster</em> becomes an additional orchestration framework within the PCAI environment, providing users with more options to select the approach that best aligns with their workflow and orchestration requirements.</p> <h3>Prerequisites</h3> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>HPE Private Cloud AI version 1.5.0 or later, running HPE AI Essentials version 1.9.1 or later.</li> <li>Access to an HPE Private Cloud AI workspace (with the <em>Private Cloud AI Administrator</em> role), allowing to perform administrative operations.</li> <li>Docker Engine version 27.3.1 or later, including the default docker CLI, which will be used for building and pushing <em>Dagster</em> user code images.</li> </ul> <p>The deployment examples in the following sections use the <em>kubectl</em> CLI and kubeconfig to display deployment details in the PCAI Kubernetes (K8s) cluster for illustration purposes only. Direct cluster access via <em>kubectl</em> is generally not required, as the full framework setup can be completed through the <em>Import Framework</em> UI.</p> <h3>Integrate <em>Dagster</em> framework using the <em>Import Framework</em></h3> <p>The official <a href="https://github.com/dagster-io/dagster/tree/master/helm"><em>Dagster</em> Helm charts</a> contain the main <em>dagster</em> chart and the <em>dagster-user-deployments</em> subchart. The <em>dagster</em> chart deploys the core <em>Dagster</em> infrastructure, specifically the <em>Dagster Webserver</em> and the <em>Dagster daemon</em>, using prebuilt images available from <em>DockerHub</em>. The <em>dagster-user-deployments</em> subchart, by contrast, is responsible for deploying <em>Dagster</em> user code, which contains the user-defined pipelines and asset definitions. Because this code is user-specific, customers must build their own user code image and use that image when deploying <em>Dagster</em> user code. In many cases, customers prefer to store this image locally within their environment.</p> <p>The following sections describe how to build a sample user code image, deploy <em>Harbor</em> and configure it as the local image registry within HPE Private Cloud AI, and push the built <em>Dagster</em> user code image to <em>Harbor</em> so it can be used for the <em>Dagster</em> deployment.</p> <h4>Build the <em>Dagster</em> user code image</h4> <p>This section describes the process for building the <em>Dagster</em> user code image in preparation for the <em>Dagster</em> deployment. For demonstration purposes, the <em><a href="https://github.com/dagster-io/dagster/tree/master/examples/deploy_k8s">'iris_analysis'</a></em> example project from the official <em><a href="https://github.com/dagster-io/dagster">Dagster GitHub repository</a></em> is used as the sample for the user code image build.</p> <pre><code class="language-shell">$ cd examples/deploy_k8s/ $ ls -al total 24 drwxrwxr-x 3 guoping guoping 4096 mars 20 14:01 . drwxrwxr-x 37 guoping guoping 4096 mars 19 11:13 .. -rw-rw-r-- 1 guoping guoping 516 mars 20 14:01 Dockerfile drwxrwxr-x 2 guoping guoping 4096 mars 19 11:13 iris_analysis -rw-rw-r-- 1 guoping guoping 361 mars 19 11:13 pyproject.toml -rw-rw-r-- 1 guoping guoping 0 mars 19 11:13 py.typed -rw-rw-r-- 1 guoping guoping 477 mars 19 11:13 README.md </code></pre> <p>The following <em>Dockerfile</em> is used to build the sample <em>Dagster</em> user code image. In addition to installing the required libraries (e.g., <em>dagster</em>, <em>dagster-postgres</em>, <em>dagster-k8s</em>, and <em>pandas</em>), be sure to update the file paths and port settings to match you own <em>Dagster</em> project.</p> <pre><code class="language-shell">$ cat Dockerfile FROM python:3.11 COPY . / ENV PYTHONUNBUFFERED=1 RUN pip install --upgrade pip RUN \ pip install \ dagster \ dagster-postgres \ dagster-k8s \ pandas WORKDIR /iris_analysis/ EXPOSE 80 </code></pre> <p>Run the following <em>Docker</em> command to build the image <em>'pcaidemo/user-code-example'</em> with the tag <em>'1.12.19'</em>.</p> <pre><code class="language-shell">$ docker build . -t pcaidemo/user-code-example:1.12.19 [+] Building 188.9s (8/10)                                                                                                                                           docker:default  => [internal] load build definition from Dockerfile                                                                                                                           0.1s  => => transferring dockerfile: 555B                                                                                                                                           0.0s  => [internal] load metadata for docker.io/library/python:3.11                                                                                                                 6.7s  => [auth] library/python:pull token for registry-1.docker.io                                                                                                                  0.0s  => [internal] load .dockerignore                                                                                                                                              0.0s [+] Building 271.3s (9/10)                                                                                                                                           docker:default  => [internal] load build definition from Dockerfile                                                                                                                           0.1s  => => transferring dockerfile: 555B                                                                                                                                           0.0s  => [internal] load metadata for docker.io/library/python:3.11                                                                                                                 6.7s  => [auth] library/python:pull token for registry-1.docker.io                                                                                                                  0.0s  => [internal] load .dockerignore                                                                                                                                              0.0s [+] Building 280.0s (11/11) FINISHED                                                                                                                                 docker:default  => [internal] load build definition from Dockerfile                                                                                                                           0.1s  => => transferring dockerfile: 555B                                                                                                                                           0.0s  => [internal] load metadata for docker.io/library/python:3.11                                                                                                                 6.7s  => [auth] library/python:pull token for registry-1.docker.io                                                                                                                  0.0s  => [internal] load .dockerignore                                                                                                                                              0.0s  => => transferring context: 2B                                                                                                                                                0.0s  => [internal] load build context                                                                                                                                              0.1s  => => transferring context: 2.08kB                                                                                                                                            0.1s  => [1/5] FROM docker.io/library/python:3.11@sha256:ff461875d046c85ecc529e93cf2a0004f29df70566194936214115b36703d866                                                         158.7s  => => resolve docker.io/library/python:3.11@sha256:ff461875d046c85ecc529e93cf2a0004f29df70566194936214115b36703d866                                                           0.0s  => => sha256:ff461875d046c85ecc529e93cf2a0004f29df70566194936214115b36703d866 10.32kB / 10.32kB                                                                               0.0s  => => sha256:990d2ceca3883d62ee15b9e7c06da32a9f9a6bb95d5c0b47548581e6e0a38d50 2.32kB / 2.32kB                                                                                 0.0s  => => sha256:e1388005fc3d7fd4f5611bd6b70464d8dc602a6189c9d5689def96add4b74a3a 6.35kB / 6.35kB                                                                                 0.0s  => => sha256:ee3a0e7d77f0c84203cab438fcf345647c8121bbd80506a3c692f8608a14c4f4 67.78MB / 67.78MB                                                                              17.9s  => => sha256:8f6ad858d0a46fa8ee628532c70b8dc82d06179d543b0b09ec19fc03d4c5b373 49.30MB / 49.30MB                                                                              13.4s  => => sha256:b012eb15dff0bce418c03ec940325aee6aa4300d771c325728855697e620c63a 25.62MB / 25.62MB                                                                               5.8s  => => sha256:8688d0f2f567884eb217c6f80efa063bdb13a1951e92e6c5cac1ae5b736f5e1b 236.08MB / 236.08MB                                                                            42.6s  => => sha256:66063df90a44c93620e1790b680bad5509bc860518ee257a157d6262916b680a 6.09MB / 6.09MB                                                                                15.9s  => => extracting sha256:8f6ad858d0a46fa8ee628532c70b8dc82d06179d543b0b09ec19fc03d4c5b373                                                                                     31.6s  => => sha256:1589b6e505d3cd8ceb2b87cebc53c22b3bd9b858def90d5c108605bbd58d8b28 23.98MB / 23.98MB                                                                              24.7s  => => sha256:8cd65a420aac1587da1c19e7cc7bd6f61b226f69fe9ec6f9d3f6215b9bf33cf2 249B / 249B                                                                                    18.8s  => => extracting sha256:b012eb15dff0bce418c03ec940325aee6aa4300d771c325728855697e620c63a                                                                                     10.8s  => => extracting sha256:ee3a0e7d77f0c84203cab438fcf345647c8121bbd80506a3c692f8608a14c4f4                                                                                     24.9s  => => extracting sha256:8688d0f2f567884eb217c6f80efa063bdb13a1951e92e6c5cac1ae5b736f5e1b                                                                                     66.5s  => => extracting sha256:66063df90a44c93620e1790b680bad5509bc860518ee257a157d6262916b680a                                                                                      1.4s  => => extracting sha256:1589b6e505d3cd8ceb2b87cebc53c22b3bd9b858def90d5c108605bbd58d8b28                                                                                      4.2s  => => extracting sha256:8cd65a420aac1587da1c19e7cc7bd6f61b226f69fe9ec6f9d3f6215b9bf33cf2                                                                                      0.0s  => [2/5] COPY . /                                                                                                                                                             4.1s  => [3/5] RUN pip install --upgrade pip                                                                                                                                       19.0s  => [4/5] RUN     pip install         dagster         dagster-postgres         dagster-k8s         pandas                                                                     82.4s  => [5/5] WORKDIR /iris_analysis/                                                                                                                                              0.1s  => exporting to image                                                                                                                                                         8.5s  => => exporting layers                                                                                                                                                        8.3s  => => writing image sha256:e5ccb2007d4d8eb9ab2d964c08a45d343a6775ceb7d012021902dc0dc51dc247                                                                                   0.0s  => => naming to docker.io/pcaidemo/user-code-example:1.12.19  </code></pre> <p>Run the below command to verify the image <em>'pcaidemo/user-code-example'</em> has been built with its proper tag <em>'1.12.19'</em>.</p> <pre><code class="language-shell">$ docker images REPOSITORY                   TAG       IMAGE ID       CREATED              SIZE pcaidemo/user-code-example   1.12.19   e5ccb2007d4d   About a minute ago   1.51GB </code></pre> <h4>Set up <em>Harbor</em> as a local image registry</h4> <p><em><a href="https://goharbor.io/">Harbor</a></em> is an open-source container registry designed for cloud-native environments like K8s. It securely stores and manages container images with policies and role-based access control (RBAC), ensures images are scanned and free from vulnerabilities, and signs images as trusted.</p> <p>You can install <em>Harbor</em> and set it up as a local image registry in HPE Private Cloud AI by following the instructions in the blog post <a href="https://developer.hpe.com/blog/setting-up-harbor-as-a-local-container-registry-in-hpe-private-cloud-ai/">Setting up Harbor as a local container registry in HPE Private Cloud AI</a>.</p> <p>After <em>Harbor</em> is deployed via the <em>Import Framework</em>, an imported* Harbor* tile appears under <strong>Tools &#x26; Frameworks</strong>.</p> <p><img src="/img/harbor.png" alt=""></p> <p>Click <em><strong>Open</strong></em> to launch the <em>Harbor</em> console. After creating a project (e.g., <em>'pcaidemo'</em>) under <strong>Projects</strong> and a user (e.g., <em>'pcai-admin'</em>) with the <em>Maintainer</em> role under <strong>Administration -> Users</strong>, add the user to the project <em>'pcaidemo'</em> from its <em>Members</em> tab.</p> <p><img src="/img/harbor-user.png" alt=""></p> <h4>Push the <em>Dagster</em> user code image to <em>Harbor</em></h4> <p>From the <em>Linux</em> client, run the following command to log in to the <em>Harbor</em> registry using the user credentials configured above.</p> <pre><code class="language-shell">$ docker login harbor.ai-application.pcai0104.ld7.hpecolo.net Username: pcai-admin Password: WARNING! Your credentials are stored unencrypted in '/home/guoping/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded </code></pre> <p>Tag the built image with the <em>Harbor</em> registry URL (e.g., <em>'harbor.ai-application.pcai0104.ld7.hpecolo.net'</em>) and the project name <em>'pcaidemo'</em>. Run the following command to push the image to the <em>Harbor</em> registry.</p> <pre><code class="language-shell">$ docker tag pcaidemo/user-code-example:1.12.19 harbor.ai-application.pcai0104.ld7.hpecolo.net/pcaidemo/user-code-example:1.12.19 $ docker push harbor.ai-application.pcai0104.ld7.hpecolo.net/pcaidemo/user-code-example:1.12.19 The push refers to repository [harbor.ai-application.pcai0104.ld7.hpecolo.net/pcaidemo/user-code-example] 5f70bf18a086: Pushed 72974f5579e5: Pushed e4701c8f7c5b: Pushed 2b8a04da403b: Pushed 4ec5e33e2b38: Pushed 5c262981bdb5: Pushed 30d39f2c6455: Pushed 6afcfc3ecd04: Pushed 817e939a94eb: Pushed dd6e353abeff: Pushed c5864b4cf4c9: Pushed 1.12.19: digest: sha256:b877e86abeea7c509dfb029a1d9fba51c45aaa9e84ca84399a92e79c2e2ac442 size: 2634 </code></pre> <p>Run the following command to view the repositories from the <em>Harbor</em> registry.</p> <pre><code class="language-shell">$ curl -k -sS --user 'pcai-admin:&#x3C;hidden>' https://harbor.ai-application.pcai0104.ld7.hpecolo.net/v2/_catalog | jq { "repositories": [ "pcaidemo/user-code-example" ] } </code></pre> <p>In the <em>Harbor</em> console, under the <em>Repositories</em> tab of the project <em>'pcaidemo'</em>, the image <em>'pcaidemo/user-code-example'</em> appears in the list.</p> <p><img src="/img/harbor-pacidemo-user-code.png" alt=""></p> <p>This user code image, once pushed to the local <em>Harbor</em> registry, will be used for the <em>Dagster</em> user code deployment.</p> <h4>Deploy <em>Dagster</em> framework</h4> <p>Based on the official <a href="https://github.com/dagster-io/dagster/tree/master/helm"><em>Dagster</em> Helm charts</a>, a revised version, available in the GitHub repository <a href="https://github.com/GuopingJia/pcai-helm-examples/tree/main/dagster">pcai-helm-examples</a>, provides HPE Private Cloud AI compatible deployment configurations. This updated chart includes the required <em>Istio VirtualService</em> and <em>Kyverno ClusterPolicy</em> manifests to ensure alignment with PCAI’s service mesh and policy controls. It also incorporates modifications for pulling the user code image from the local <em>Harbor</em> registry.</p> <p>Follow the steps below to deploy <em>Dagster</em> into HPE Private Cloud AI using the <em>Import Framework</em>.</p> <ul> <li>In the PCAI left navigation panel, select <strong>Tools &#x26; Frameworks</strong>. Click <em><strong>Import Framework</strong></em>.</li> </ul> <p><img src="/img/pcai-tools-frameworks-import-framework.png" alt=""></p> <ul> <li>By following the Import Framework wizard workflow, <em>Dagster</em> can be deployed into the PCAI environment within minutes.</li> </ul> <p><img src="/img/import-framework-dagster.png" alt=""></p> <ul> <li>Run the following commands to verify the <em>Dagster</em> deployment in the namespace 'dagster' of the PCAI K8s cluster.</li> </ul> <pre><code class="language-shell">$ kubectl get all -n dagster NAME                                                                  READY   STATUS    RESTARTS   AGE pod/dagster-daemon-66c46866f8-sc4n7                                   1/1     Running   0          30h pod/dagster-dagster-user-deployments-k8s-example-user-code-1-8k86gs   1/1     Running   0          30h pod/dagster-dagster-webserver-66447f9b57-9rzmh                        1/1     Running   0          30h pod/dagster-postgresql-0                                              1/1     Running   0          30h NAME                                  TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE service/dagster-dagster-webserver     ClusterIP   10.96.1.6    &#x3C;none>        80/TCP     30h service/dagster-postgresql            ClusterIP   10.96.3.66   &#x3C;none>        5432/TCP   30h service/dagster-postgresql-headless   ClusterIP   None         &#x3C;none>        5432/TCP   30h service/k8s-example-user-code-1       ClusterIP   10.96.3.96   &#x3C;none>        3030/TCP   30h NAME                                                                       READY   UP-TO-DATE   AVAILABLE   AGE deployment.apps/dagster-daemon                                             1/1     1            1           30h deployment.apps/dagster-dagster-user-deployments-k8s-example-user-code-1   1/1     1            1           30h deployment.apps/dagster-dagster-webserver                                  1/1     1            1           30h NAME                                                                                  DESIRED   CURRENT   READY   AGE replicaset.apps/dagster-daemon-66c46866f8                                             1         1         1       30h replicaset.apps/dagster-dagster-user-deployments-k8s-example-user-code-1-85d9d56f44   1         1         1       30h replicaset.apps/dagster-dagster-webserver-66447f9b57                                  1         1         1       30h NAME                                  READY   AGE statefulset.apps/dagster-postgresql   1/1     30h </code></pre> <h3>Access <em>Dagster</em> framework</h3> <p>After <em>Dagster</em> is deployed via the <em>Import Framework</em>, an <em>imported</em> <em>Dagster</em> tile appears under <strong>Tools &#x26; Frameworks</strong>.</p> <p><img src="/img/dagster.png" alt=""></p> <h4>Connect to <em>Dagster</em> deployment</h4> <p>Clicking <em><strong>Open</strong></em> on the <em>Dagster</em> tile launches the <em>Dagster</em> Webserver and directs you to the <strong>Overview</strong> page.</p> <p><img src="/img/dagster-overview.png" alt=""></p> <p>Navigate to <strong>Deployment</strong> and open the <em>Code locations</em> tab. The entry <em>'k8s-example-user-code-1'</em> shows the user code image <em>'harbor.ai-application.pcai0104.ld7.hpecolo.net/pcaidemo/user-code-example:1.12.19'</em> pulled from the <em>Harbor</em> registry.</p> <p><img src="/img/dagster-deployment.png" alt=""></p> <p>In the <em>Harbor</em> console, under the <em>Logs</em> tab of the project <em>'pcaidemo'</em>, you can see the artifact pull operations for the image <em>'pcaidemo/user-code-example'</em>.</p> <p><img src="/img/harbor-audit-logs.png" alt=""></p> <h4>Materialize <em>Dagster</em> assets</h4> <p>In the <em>Dagster</em> Webserver, navigate to <strong>Catalog</strong>, select the asset <em>'iris_dataset_size'</em>, and click the <em><strong>Materialize selected</strong></em> button. This triggers the materialization process for the asset <em>'iris_dataset_size'</em>.</p> <p><img src="/img/dagster-catalog.png" alt=""></p> <p>Run the following command to view the K8s job that was started on the cluster to materialize the asset.</p> <pre><code class="language-shell">$ kubectl get jobs -n dagster NAME STATUS COMPLETIONS DURATION AGE dagster-run-044deadf-141d-4b25-89b1-dd74f0a44f89 Complete 1/1 8s 25s </code></pre> <p>Once the K8s job completes, the asset <em>'iris_dataset_size'</em> appears with a <em>Materialized</em> status in the <strong>Catalog</strong>.</p> <p><img src="/img/dagster-catalog-materialization.png" alt=""></p> <p>Click the asset <em>'iris_dataset_size'</em> to view its overview.</p> <p><img src="/img/dagster-catalog-materialization-overview.png" alt=""></p> <p>With this materialized data asset, you can begin capturing how data is transformed, joined, filtered, and aggregated throughout each stage of the data pipeline, providing a complete record of the data’s journey and its lineage. For more details on these advanced capabilities, refer to the <a href="https://docs.dagster.io/"><em>Dagster</em> documentation</a>.</p> <p>Click <em><strong>Wipe materializations</strong></em> from the selected asset to remove its materializations.</p> <p><img src="/img/dagster-catalog-materialization-wipe.png" alt=""></p> <h3>Conclusion</h3> <p>This blog post explored the pre-curated orchestration toolchain available within PCAI and introduced <em>Dagster</em> as a modern, asset-centric framework that can be integrated seamlessly into the HPE Private Cloud AI environment via the <em>Import Framework</em>. When deployed alongside existing orchestration services such as <em>Airflow</em>, <em>Kubeflow</em>, and <em>Ray</em>, <em>Dagster</em> operates as an additional, fully compatible orchestration layer within PCAI. Its modular architecture and clear separation between infrastructure and user code allow all user-defined pipeline definitions to be deployed and executed locally within the HPE Private Cloud AI environment, ensuring strong data sovereignty guarantees. By aligning naturally with PCAI's service model and operational patterns, <em>Dagster</em> enriches the platform with a clean, asset-oriented orchestration approach that enhances pipeline reliability while remaining fully compliant with PCAI’s security and governance expectations.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud AI and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/siva-observe-any-observable-using-opsramp-—-part-2-setting-up-the-stack/https://developer.hpe.com/siva-observe-any-observable-using-opsramp-—-part-2-setting-up-the-stack/Fri, 17 Apr 2026 12:36:31 GMT<p><em>Part 2 of a series on vendor-neutral observability with HPE OpsRamp and OpenTelemetry</em></p> <hr> <h2>Introduction</h2> <p>In <a href="https://developer.hpe.com/blog/siva-observe-any-observable-using-opsramp-open-telemetry-as-the-universal-ingestion-standard/">Part 1</a>, I described the full architecture of the observability stack — five components wired together through open protocols to deliver end-to-end telemetry from a Redfish hardware domain into HPE OpsRamp. Now it is time to build that stack and verify every component before sending a single signal anywhere.</p> <p>This article covers installation, configuration, and verification of every component in the local stack: <a href="https://github.com/DMTF/Redfish-Interface-Emulator">the Redfish emulator</a>, <a href="https://github.com/open-telemetry">OpenTelemetry Collector(OTel)</a>, <a href="https://github.com/prometheus/prometheus">Prometheus</a>, <a href="https://github.com/grafana/grafana">Grafana</a>, and <a href="https://github.com/jaegertracing/jaeger">Jaeger</a>. I deliberately exclude the HPE OpsRamp configuration from this article — I want to establish that the local stack is fully functional and producing real data before introducing the cloud management plane. That separation of concerns makes troubleshooting dramatically easier.</p> <p>By the end of this article, you will have a running stack that you can interact with through five different browser UIs and a smoke test that validates every component automatically.</p> <hr> <h2>Prerequisites: What you need before starting</h2> <p>Before beginning, you will need an EC2 instance running Ubuntu 22.04 or later with the following:</p> <ul> <li>At least 4GB RAM and 2 vCPUs (the full stack with all containers needs headroom)</li> <li>Docker and Docker Compose installed</li> <li>Python 3.11 or later with <code>uv</code> package manager</li> <li>Ports 3000, 5000, 9090, 16686, 4317, 4318, 8888, 8889, 55679, 13133 open in your security group</li> </ul> <p>I use EC2 Instance Connect in the browser as my terminal throughout this series, so there is no local SSH key management required.</p> <h3>Installing uv</h3> <p><code>uv</code> is the modern Python package manager I use throughout this project. <a href="https://github.com/astral-sh/uv">uv</a> is significantly faster than pip and handles virtual environments cleanly.</p> <pre><code class="language-bash">curl -LsSf https://astral.sh/uv/install.sh | sh source ~/.bashrc uv --version </code></pre> <h3>Installing Docker on Ubuntu</h3> <p>If Docker is not already installed, use the convenience script:</p> <pre><code class="language-bash">curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo usermod -aG docker ubuntu newgrp docker docker --version docker compose version </code></pre> <hr> <h2>The project structure</h2> <p>I organize everything under a single directory to keep the two PoC versions isolated from each other.</p> <pre><code class="language-bash">~/sivabala/siva-sdk/ ├── poc-v1/ ← original PoC (v1) │ └── ... ├── poc-v2/ ← this article's stack │ ├── docker-compose.yml │ ├── otel-collector-config.yaml │ ├── prometheus.yml │ ├── grafana/ │ │ └── provisioning/ │ │ ├── datasources/ │ │ └── dashboards/ │ └── opsramp-sdk/ │ ├── pyproject.toml │ └── src/opsramp_sdk/redfish/ │ ├── agent.py │ ├── event_listener.py │ ├── opsramp_resources.py │ └── resource_context.py ├── start-v1.sh ├── start-v2.sh ├── stop-all.sh └── status.sh </code></pre> <p>I first create the base directory structure:</p> <pre><code class="language-bash">mkdir -p ~/sivabala/siva-sdk/poc-v2/opsramp-sdk/src/opsramp_sdk/redfish mkdir -p ~/sivabala/siva-sdk/poc-v2/grafana/provisioning/datasources mkdir -p ~/sivabala/siva-sdk/poc-v2/grafana/provisioning/dashboards mkdir -p ~/sivabala/siva-sdk/poc-v2/grafana/dashboards </code></pre> <hr> <h2>Component 1: Docker Compose — the orchestration layer</h2> <p>All five local components run in Docker containers managed by a single <code>docker-compose.yml</code>. This file defines the services, their relationships, port mappings, and shared network.</p> <pre><code class="language-yaml"># docker-compose.yml services: # 1. Redfish emulator — hardware simulation redfish-emulator: image: dmtf/redfish-interface-emulator:latest container_name: redfish-emulator ports: - "5000:5000" restart: unless-stopped healthcheck: test: ["CMD", "curl", "-sf", "http://127.0.0.1:5000/redfish/v1/"] interval: 10s timeout: 5s retries: 5 start_period: 15s networks: - redfish-poc # 2. OpenTelemetry Collector — signal processing and routing otel-collector: image: otel/opentelemetry-collector-contrib:latest container_name: otel-collector command: ["--config=/etc/otel/config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel/config.yaml:ro ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP - "8888:8888" # Collector self-metrics - "8889:8889" # Prometheus scrape endpoint - "55679:55679" # zpages debug UI - "13133:13133" # health check restart: unless-stopped depends_on: redfish-emulator: condition: service_healthy networks: - redfish-poc # 3. Jaeger — distributed trace visualization jaeger: image: jaegertracing/all-in-one:latest container_name: jaeger environment: - COLLECTOR_OTLP_ENABLED=true ports: - "16686:16686" # Jaeger UI - "14317:4317" # OTLP gRPC for Jaeger restart: unless-stopped networks: - redfish-poc # 4. Prometheus — metrics storage and remote write prometheus: image: prom/prometheus:latest container_name: prometheus command: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" - "--web.enable-lifecycle" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro - prometheus_data:/prometheus ports: - "9090:9090" restart: unless-stopped depends_on: - otel-collector networks: - redfish-poc # 5. Grafana — local dashboard visualization grafana: image: grafana/grafana:latest container_name: grafana environment: - GF_SECURITY_ADMIN_PASSWORD=admin - GF_USERS_ALLOW_SIGN_UP=false volumes: - grafana_data:/var/lib/grafana - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro ports: - "3000:3000" restart: unless-stopped depends_on: - prometheus - jaeger networks: - redfish-poc volumes: prometheus_data: grafana_data: networks: redfish-poc: driver: bridge </code></pre> <p>I start all containers:</p> <pre><code class="language-bash">cd ~/sivabala/siva-sdk/poc-v2 sudo docker compose up -d </code></pre> <hr> <h2>Component 2: OpenTelemetry Collector — the pipeline configuration</h2> <p>The OpenTelemetry (OTel) Collector configuration defines receivers, processors, exporters, and pipelines. I will focus here on the local stack configuration — the HPE OpsRamp exporters are introduced in Part 4.</p> <pre><code class="language-yaml"># otel-collector-config.yaml extensions: zpages: endpoint: 0.0.0.0:55679 health_check: endpoint: 0.0.0.0:13133 path: "/health" receivers: # Receives OTLP signals from the Python agent otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 # Polls Redfish endpoints directly for health metrics httpcheck: targets: - endpoint: http://redfish-emulator:5000/redfish/v1/ method: GET - endpoint: http://redfish-emulator:5000/redfish/v1/Chassis/Chassis-1/Power method: GET - endpoint: http://redfish-emulator:5000/redfish/v1/Chassis/Chassis-1/Thermal method: GET collection_interval: 15s # Scrapes the Collector's own metrics prometheus/self: config: scrape_configs: - job_name: opentelemetry-collector scrape_interval: 30s static_configs: - targets: - localhost:8888 processors: resource: attributes: - action: insert key: deployment.environment value: "redfish-poc-v2" - action: insert key: resource.type value: "RESOURCE" batch: send_batch_size: 512 timeout: 5s exporters: debug: verbosity: normal prometheus: endpoint: "0.0.0.0:8889" namespace: "redfish" send_timestamps: true metric_expiration: 3m resource_to_telemetry_conversion: enabled: true otlp/jaeger: endpoint: "jaeger:4317" tls: insecure: true service: extensions: [zpages, health_check] pipelines: metrics/agent: receivers: [otlp] processors: [resource, batch] exporters: [debug, prometheus] metrics/httpcheck: receivers: [httpcheck] processors: [resource, batch] exporters: [debug, prometheus] metrics/self: receivers: [prometheus/self] processors: [batch] exporters: [prometheus] logs: receivers: [otlp] processors: [resource, batch] exporters: [debug] traces: receivers: [otlp] processors: [resource, batch] exporters: [debug, otlp/jaeger] telemetry: logs: level: warn </code></pre> <hr> <h2>Component 3: Prometheus — metrics configuration</h2> <p>The <code>prometheus.yml</code> configures Prometheus to scrape the OTel Collector's Prometheus exporter endpoint. The <code>remote_write</code> section is added in Part 4.</p> <pre><code class="language-yaml"># prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: "otel-collector" static_configs: - targets: ["otel-collector:8889"] metric_relabel_configs: - source_labels: [__name__] regex: "redfish_.*" action: keep </code></pre> <hr> <h2>Component 4: Grafana — datasource provisioning</h2> <p>Grafana is pre-configured with Prometheus and Jaeger datasources through provisioning files, eliminating manual setup.</p> <pre><code class="language-yaml"># grafana/provisioning/datasources/datasources.yaml apiVersion: 1 datasources: - name: Prometheus type: prometheus uid: prometheus access: proxy url: http://prometheus:9090 isDefault: true editable: true - name: Jaeger type: jaeger uid: jaeger access: proxy url: http://jaeger:16686 editable: true </code></pre> <hr> <h2>Component 5: The Python SDK — project setup</h2> <p>The Python agent uses <code>uv</code> for dependency management. The <code>pyproject.toml</code> defines all OTel dependencies.</p> <pre><code class="language-toml"># opsramp-sdk/pyproject.toml [project] name = "opsramp-sdk" version = "2.0.0" description = "OpsRamp SDK v2 — OTel-native resource and telemetry ingestion" requires-python = ">=3.11" dependencies = [ "requests>=2.31.0", "opentelemetry-api>=1.24.0", "opentelemetry-sdk>=1.24.0", "opentelemetry-exporter-otlp-proto-grpc>=1.24.0", "opentelemetry-instrumentation-requests>=0.45b0", "python-dotenv>=1.0", ] [project.scripts] opsramp-redfish-agent-v2 = "opsramp_sdk.redfish.agent:main" opsramp-redfish-events-v2 = "opsramp_sdk.redfish.event_listener:main" opsramp-redfish-provision = "opsramp_sdk.redfish.opsramp_resources:provision_all_resources" [build-system] requires = ["uv_build>=0.10.9,&#x3C;0.11.0"] build-backend = "uv_build" [tool.uv] package = true [dependency-groups] dev = [ "pytest>=8.0", "pytest-timeout>=2.3", "python-dotenv>=1.0", ] </code></pre> <p>Install dependencies:</p> <pre><code class="language-bash">cd ~/sivabala/siva-sdk/poc-v2/opsramp-sdk uv sync </code></pre> <hr> <h2>Environment configuration: The .env file</h2> <p>All credentials and endpoints are stored in a <code>.env</code> file that is loaded automatically by each Python module.</p> <pre><code class="language-bash"># poc-v2/.env # HPE OpsRamp endpoints OPSRAMP_API_URL=https://api.opsramp.com OPSRAMP_TENANT_ID=&#x3C;your-tenant-id> # Resource API credentials (scope: global:manage) — used for provisioning OPSRAMP_CLIENT_ID=&#x3C;client-id> OPSRAMP_CLIENT_SECRET=&#x3C;client-secret> # Collector credentials (scope: logs:write, metrics:write) — used by OTel Collector OPSRAMP_COLLECTOR_CLIENT_ID=&#x3C;collector-client-id> OPSRAMP_COLLECTOR_CLIENT_SECRET=&#x3C;collector-client-secret> # Redfish emulator REDFISH_BASE_URL=http://localhost:5000 # OTel Collector OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 # Agent behaviour POLL_INTERVAL_SEC=15 LOG_LEVEL=INFO SKIP_PROVISION=false # Event listener EVENT_SIMULATOR_ENABLED=true SIM_INTERVAL_SEC=30 </code></pre> <hr> <h2>Verification: Confirming every component is healthy</h2> <p>I wrote a comprehensive smoke test that validates every component automatically. Here is how to run it and what to look for:</p> <h3>Step 1: Verify Docker containers</h3> <pre><code class="language-bash">sudo docker compose ps </code></pre> <p>Expected output — all five containers running:</p> <pre><code>NAME STATUS redfish-emulator Up (healthy) otel-collector Up jaeger Up prometheus Up grafana Up </code></pre> <p>If <code>otel-collector</code> shows <code>Restarting</code>, check its logs:</p> <pre><code class="language-bash">sudo docker logs otel-collector 2>&#x26;1 | grep -i error | tail -10 </code></pre> <p>The most common issue is an invalid <code>address:</code> field in the <code>telemetry.metrics</code> section of the collector config. Remove that field if present.</p> <h3>Step 2: Verify the Redfish emulator</h3> <pre><code class="language-bash">curl -s http://localhost:5000/redfish/v1/ | python3 -m json.tool | head -10 </code></pre> <p>Expected: JSON response with <code>RedfishVersion</code>, <code>Chassis</code>, <code>Systems</code>, <code>Managers</code> links.</p> <pre><code class="language-bash"># Verify all 13 chassis are present curl -s http://localhost:5000/redfish/v1/Chassis | \ python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Chassis count: {d[\"Members@odata.count\"]}')" </code></pre> <p>Expected: <code>Chassis count: 13</code></p> <pre><code class="language-bash"># Verify power data for Chassis-1 curl -s http://localhost:5000/redfish/v1/Chassis/Chassis-1/Power | \ python3 -c " import sys,json d=json.load(sys.stdin) for pc in d.get('PowerControl',[]): print(f'Consumed: {pc.get(\"PowerConsumedWatts\")}W') " </code></pre> <h3>Step 3: Verify the OTel Collector</h3> <pre><code class="language-bash"># Health check endpoint curl -s http://localhost:13133/health </code></pre> <p>Expected: <code>{"status":"Server available"}</code></p> <pre><code class="language-bash"># zpages pipeline status — shows all active pipelines curl -s http://localhost:55679/debug/pipelinez | grep -o "pipeline[^&#x3C;]*" | head -10 </code></pre> <pre><code class="language-bash"># Verify httpcheck metrics are flowing curl -s http://localhost:8889/metrics | grep "redfish_httpcheck_status" | head -5 </code></pre> <p>Expected: Multiple lines with <code>redfish_httpcheck_status{...} 1</code> (1 = endpoint up)</p> <h3>Step 4: Verify Prometheus</h3> <pre><code class="language-bash"># Check scrape targets are healthy curl -s http://localhost:9090/api/v1/targets | \ python3 -c " import sys,json d=json.load(sys.stdin) for t in d['data']['activeTargets']: print(f'{t[\"health\"]:10s} {t[\"scrapeUrl\"]}') " </code></pre> <p>Expected: <code>up</code> for all targets.</p> <pre><code class="language-bash"># Query httpcheck metrics curl -s "http://localhost:9090/api/v1/query?query=redfish_httpcheck_status" | \ python3 -c " import sys,json d=json.load(sys.stdin) print(f'httpcheck series: {len(d[\"data\"][\"result\"])}') " </code></pre> <h3>Step 5: Verify Grafana</h3> <p>Open <code>http://&#x3C;EC2-IP>:3000</code> in your browser. Log in with <code>admin</code> / <code>admin</code>.</p> <p>Navigate to <strong>Connections → Data sources</strong> and verify both Prometheus and Jaeger datasources show a green "Data source connected" status.</p> <h3>Step 6: Run the automated smoke test</h3> <p>I wrote a comprehensive shell script that validates all endpoints automatically:</p> <pre><code class="language-bash">chmod +x ~/sivabala/siva-sdk/poc-v2/smoke-test.sh bash ~/sivabala/siva-sdk/poc-v2/smoke-test.sh </code></pre> <p>The script checks every component and reports pass/fail with HTTP status codes. A clean stack shows zero failures before starting the agent.</p> <hr> <h2>The zpages debug UI: Live pipeline visibility</h2> <p>One of the most useful tools for understanding what the OTel Collector is doing in real time is the zpages debug interface. I added this specifically for demonstration purposes.</p> <p>Open <code>http://&#x3C;EC2-IP>:55679/debug/pipelinez</code> in your browser. You will see the complete pipeline topology — every receiver, processor, and exporter — with live counters showing signals flowing through each stage.</p> <p>The key pages are:</p> <ul> <li><code>/debug/pipelinez</code> — Pipeline status with signal counts</li> <li><code>/debug/tracez</code> — Live trace sampling — watch spans arrive in real time</li> <li><code>/debug/servicez</code> — Collector version and running extensions</li> <li><code>/debug/rpcz</code> — gRPC connection stats from the agent</li> </ul> <p>This is the single best tool for demonstrating the OTel Collector pipeline to an audience because it shows the data moving in real time without requiring any additional tooling.</p> <hr> <h2>Browser access summary</h2> <p>After a successful stack startup, you should have access to five browser UIs:</p> <table> <thead> <tr> <th>UI</th> <th>URL</th> <th>What it shows</th> </tr> </thead> <tbody> <tr> <td>Redfish API</td> <td><code>http://&#x3C;EC2-IP>:5000/redfish/v1/</code></td> <td>Raw hardware data</td> </tr> <tr> <td>zpages</td> <td><code>http://&#x3C;EC2-IP>:55679/debug/pipelinez</code></td> <td>Live pipeline</td> </tr> <tr> <td>Grafana</td> <td><code>http://&#x3C;EC2-IP>:3000</code></td> <td>Dashboards</td> </tr> <tr> <td>Jaeger</td> <td><code>http://&#x3C;EC2-IP>:16686</code></td> <td>Trace waterfall</td> </tr> <tr> <td>Prometheus</td> <td><code>http://&#x3C;EC2-IP>:9090</code></td> <td>Metrics queries</td> </tr> </tbody> </table> <hr> <h2>Conclusion and what comes next</h2> <p>The local stack is now running and verified. Using the steps above, I now have five healthy containers, a working Redfish emulator producing real hardware data, an OTel Collector routing signals through its pipelines, and Prometheus, Grafana, and Jaeger ready to visualize that data.</p> <p>Importantly, I have verified each component independently before combining them with the Python agent or HPE OpsRamp. This incremental verification approach is what makes debugging tractable when something goes wrong.</p> <p>In Part 3, I will introduce two powerful testing tools — <code>otel-cli</code> and <code>promtool</code> — and show how to use them to test the Redfish emulator, validate the collector pipeline, and trigger test traces manually. These tools let you probe every part of the stack without writing a single line of Python.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more insights on <a href="https://www.hpe.com/us/en/opsramp.html">HPE OpsRamp</a> (Hybrid Cloud Observability) and practical ideas to apply it in your daily operations.</p>https://developer.hpe.com/reflections-on-30-years-of-hpc-programming-so-many-hardware-advances-so-little-adoption-of-new-languages/https://developer.hpe.com/reflections-on-30-years-of-hpc-programming-so-many-hardware-advances-so-little-adoption-of-new-languages/Thu, 09 Apr 2026 22:39:07 GMT<p>external blog</p>https://developer.hpe.com/2026-april-06/https://developer.hpe.com/2026-april-06/Mon, 06 Apr 2026 13:00:00 GMThttps://developer.hpe.com/ai-devcon-2026-why-the-future-of-enterprise-ai-is-a-data-and-storage-engineering-problem/https://developer.hpe.com/ai-devcon-2026-why-the-future-of-enterprise-ai-is-a-data-and-storage-engineering-problem/Fri, 03 Apr 2026 20:51:38 GMT<p>AI DevCon 2026, held on 12–13 March 2026 at the NIMHANS Convention Centre in Bengaluru, brought together developers, data engineers, platform architects, DevOps leaders, and AI practitioners working at every layer of the stack. While the conference agenda spanned topics such as GenAI, agentic AI, DevOps with AI, edge inference, and open-source tooling, one theme consistently surfaced across keynotes, technical talks, and hands‑on sessions: AI systems succeed or fail based far more on data engineering and infrastructure than on model sophistication.</p> <p>Across both days, speakers repeatedly demonstrated that many production AI failures—hallucinations, incorrect answers, unreliable agents, or non‑deterministic behavior—are not caused by faulty models. Instead, they originate upstream, in how data is stored, curated, versioned, refreshed, and retrieved. For organizations building enterprise AI systems, this reframes the challenge: AI reliability has become a storage‑adjacent engineering concern.</p> <p><strong>From models to momentum: The shift toward data-centric AI engineering</strong></p> <p>Several sessions, particularly in the <strong>Succeeding with AI and GenAI &#x26; LLMs</strong> tracks, emphasized the industry’s shift away from model‑centric thinking. Large language models(LLMs) are increasingly commoditized—open, swappable, or accessible via APIs. What differentiates successful AI deployments is no longer which model is used, but how effectively organizations manage the data that feeds those models.</p> <p>Presenters traced the evolution of engineering pipelines across eras. Traditional analytics relied on structured data warehouses. Machine learning systems introduced feature pipelines and offline training loops. In contrast, modern GenAI systems depend on knowledge pipelines—continuous data ingestion, semantic chunking, embedding generation, vector indexing, and retrieval‑augmented generation (RAG). Each step builds directly on stored data, and each step is sensitive to data quality, freshness, and lineage.</p> <p>The key insight shared repeatedly was simple but profound: LLMs reason over whatever context they are given—and they do so with complete confidence, even when that context is outdated, duplicated, or wrong.</p> <p><strong>Where GenAI systems actually break in production</strong></p> <p>Real production examples shared at AI DevCon highlighted how GenAI systems fail in subtle but damaging ways. Many RAG implementations rely on naïve chunking strategies—splitting documents by token count rather than by semantic structure. This fragments meaning, separates definitions from constraints, and causes incomplete context to be retrieved at inference time. Weak or outdated embeddings further amplify the problem, surfacing text that is lexically similar but semantically incorrect.</p> <p>Perhaps the most concerning failure mode discussed was knowledge freshness drift. Policies, documentation, and procedures evolve continuously, yet embeddings and vector indexes are often refreshed infrequently. As a result, AI systems confidently return answers that reflect last quarter’s truth, not today’s reality. One real‑world case discussed involved an airline chatbot providing incorrect refund guidance based on outdated stored content—an error that ultimately resulted in legal liability.</p> <p>These failures make an uncomfortable truth unavoidable: RAG systems are only as reliable as the stored data they retrieve, and storage systems that lack strong versioning, provenance, and refresh guarantees directly undermine AI correctness.</p> <p><strong>The emergence of semantic data quality pipelines</strong></p> <p>To address these challenges, multiple speakers introduced what is best described as a semantic data quality layer—a new architectural component that sits between raw storage and AI models. Unlike traditional ETL pipelines focused on schema and format, this layer focuses on meaning.</p> <p>Incoming data is semantically validated, deduplicated, enriched, and scored before it is allowed to influence AI outputs. Low‑value or low‑trust content is filtered out entirely. Embedding similarity is used not just for retrieval, but also for semantic deduplication and drift detection. In practice, presenters showed that more than half of raw data often never reaches the AI model, because it is irrelevant, redundant, or insufficiently trustworthy.</p> <p>This reframes data engineering for AI as a quality‑first discipline. Instead of maximizing data volume, successful teams aggressively curate data relevance, ensuring that models see less data—but better data.</p> <p><strong>Agentic AI raises the stakes for infrastructure</strong></p> <p>Agentic AI, a major focus of Day 2, further increases pressure on data and storage foundations. Unlike traditional AI applications, agents are long‑running, stateful, and action‑oriented. They retrieve context, call tools, store intermediate reasoning artifacts, and revisit prior decisions. This creates access patterns that are far more dynamic than classic read‑heavy inference workloads.</p> <p>From an infrastructure perspective, agents introduce continuous read‑write cycles, ephemeral knowledge states, and frequent context recomposition. Storage systems must handle high concurrency, small object access, frequent updates, and strong consistency—while still supporting compliance, auditability, and isolation. The message from the conference was clear: agentic AI magnifies any weaknesses in the underlying data platform.</p> <p><strong>Why this matters deeply for HPE Storage</strong></p> <p>For HPE Storage teams, the themes from AI DevCon 2026 are directly relevant. Nearly every AI failure described at the conference maps back to core storage concerns: data freshness, version control, metadata quality, lineage tracking, governance, and performance consistency at scale. In the GenAI era, storage is no longer a passive repository—it actively shapes AI outcomes.</p> <p>As enterprises deploy RAG systems and AI agents on top of proprietary data, storage becomes the system of record for organizational truth. Vector databases and embeddings are derivative views; the authoritative content still resides in object, file, or block storage. If that foundational data is duplicated, stale, or poorly governed, AI systems will reflect those flaws—no matter how advanced the model.</p> <p>From an HPE perspective, this creates a strategic opportunity. Enterprise customers increasingly need storage platforms that are AI‑aware by design: optimized for rapid re‑indexing, capable of exposing rich metadata, supportive of frequent refresh cycles, and resilient under embedding‑heavy workloads. Storage that enables strong provenance, policy‑driven lifecycle management, and auditability becomes a trust anchor for AI—not just an infrastructure component.</p> <p><strong>Storage as an AI trust layer</strong></p> <p>The conference made it clear that trust is the critical currency of enterprise AI adoption. Models may generate answers, but organizations are accountable for correctness. Storage platforms that support immutable records, versioned datasets, and explainable data sourcing directly contribute to AI trust and regulatory readiness.</p> <p>For HPE Storage engineering teams, aligning storage capabilities with AI pipelines—rather than treating AI as an external consumer—positions HPE to play a foundational role in customers’ AI strategies. The opportunity is not to compete with AI models or vector databases, but to enable them to operate reliably, securely, and at enterprise scale.</p> <p><strong>Closing thoughts</strong></p> <p>AI DevCon 2026 reinforced a pivotal industry transition. As AI systems become more capable and autonomous, correctness and reliability are moving upstream—from model architectures into data pipelines and infrastructure design. In this new reality, storage engineering is inseparable from AI engineering.</p> <p>For HPE, this is not a peripheral trend but a defining moment. The teams that design, build, and evolve HPE Storage platforms are increasingly shaping how enterprise AI systems behave in the real world. By embracing this responsibility, HPE can help ensure that the AI systems built on top of its platforms are not just powerful—but dependable.</p>https://developer.hpe.com/Streamlining-hybrid-cloud-Announcing-the-unified-HPE-Terraform-provider v1.1.0/https://developer.hpe.com/Streamlining-hybrid-cloud-Announcing-the-unified-HPE-Terraform-provider v1.1.0/Thu, 02 Apr 2026 23:06:40 GMT<p>external blog</p>https://developer.hpe.com/siva-observe-any-observable-using-opsramp-open-telemetry-as-the-universal-ingestion-standard/https://developer.hpe.com/siva-observe-any-observable-using-opsramp-open-telemetry-as-the-universal-ingestion-standard/Mon, 30 Mar 2026 12:23:01 GMT<p><em>Part 1 of a series on vendor-neutral observability with HPE OpsRamp and OpenTelemetry</em></p> <hr> <h2>Introduction</h2> <p>I have spent considerable time thinking about a fundamental question in modern infrastructure management: what does it mean for a platform to be truly observable? Not just monitorable — but observable in the sense that any signal, from any source, using any open standard, can flow into a single management plane without vendor lock-in.</p> <p>In this article, I want to share what I built to answer that question: a proof-of-concept that demonstrates how HPE OpsRamp can serve as a universal observability backend for any infrastructure domain, using OpenTelemetry as the sole ingestion standard. I will walk you through the objective, the architecture, every component in the stack, and exactly how they are wired together.</p> <h2>What is OpenTelemetry?</h2> <p>OpenTelemetry (OTel) is a <a href="https://www.cncf.io/">CNCF</a> open-source project that provides a vendor-neutral standard for collecting and exporting telemetry — metrics, logs, and traces — from any application or infrastructure component. It defines a common data model, APIs, SDKs, and a wire protocol (OTLP) that any observability backend can consume.</p> <p>Rather than instrumenting your code differently for every monitoring tool, you instrument once with OpenTelemetry and route signals wherever you need them.</p> <blockquote> <p><strong>Project home:</strong> <a href="https://opentelemetry.io">opentelemetry.io</a></p> </blockquote> <p>By the end of this article you will understand the full picture — the what and the why. In subsequent articles I will go deeper into the how: installation, verification, testing, and signal ingestion.</p> <hr> <h2>The objective: Observe anything, from anywhere, using open standards</h2> <p>The central premise of this proof-of-concept is deceptively simple.</p> <p><strong>Before ingestion:</strong> HPE OpsRamp has no knowledge of the target infrastructure. No resources pre-created, no templates configured, no proprietary agents installed.</p> <p><strong>After ingestion:</strong> HPE OpsRamp automatically has a complete operational picture — resources with full domain attributes, topology relationships, live metrics, structured logs, distributed traces, and correlated alerts — all derived purely from OpenTelemetry signals.</p> <p>The domain does not matter. I used Redfish — the DMTF standard for hardware management — purely as a convenient resource simulator. The same architecture works for Kubernetes workloads, VMware vSphere, bare-metal servers, IoT sensors, or any system that can emit OpenTelemetry signals. Redfish is the stand-in, not the subject.</p> <p>The key constraint I imposed on myself: <strong>no HPE OpsRamp proprietary agents, no vendor-specific SDKs for signal collection, no webhook-based event ingestion</strong>. Everything flows through open standards.</p> <hr> <h2>The stack: Five components, one open pipeline</h2> <p>The architecture consists of five components arranged in a clean signal pipeline. Let me introduce each one and explain its role before describing how they interconnect.</p> <p><img src="/img/five-stack.jpg" alt="The stack: Five components, one open pipeline" title="The stack: Five components, one open pipeline"></p> <h3>The DMTF Redfish emulator — resource simulation layer</h3> <p>The DMTF Redfish Interface Emulator is an open-source Python application that implements the Redfish API specification. It simulates a complete hardware infrastructure: 13 chassis, 7 compute systems, and 13 BMC managers — 34 resources in total — each exposing real Redfish endpoints for power consumption, thermal readings, fan speeds, memory, CPU, and hardware events.</p> <p>I chose Redfish because it is a genuine industry standard used in real data centers, it produces rich structured data, and it runs entirely in Docker with no external dependencies. Every chassis exposes endpoints like <code>/redfish/v1/Chassis/Chassis-1/Power</code> and <code>/redfish/v1/Chassis/Chassis-1/Thermal</code> that return real numeric data suitable for metric collection.</p> <p>The emulator runs on port 5000 and is the sole source of truth for infrastructure state in this stack.</p> <h3>The OTel Python agent — instrumentation layer</h3> <p>This is the custom Python application I wrote for this proof-of-concept. It is the heart of the open-standard ingestion story.</p> <p>The agent uses the <strong>OpenTelemetry Python SDK</strong> to instrument Redfish polling in a way that produces three distinct signal types:</p> <p><strong>Metrics</strong> — The agent creates one <code>MeterProvider</code> per chassis and one per system. Each provider has its own <code>Resource</code> object carrying the chassis or system identity. Power consumption, thermal temperatures, fan speeds, memory, and CPU metrics are emitted as OTel gauge instruments with full resource context attached.</p> <p><strong>Logs and events</strong> — Redfish hardware events are captured as OTel <code>LogRecord</code> objects. Each log record carries the host.name of the originating resource, enabling HPE OpsRamp to associate the log with the correct resource entry. A separate <code>LoggerProvider</code> per resource ensures the association is precise.</p> <p><strong>Traces</strong> — Every poll cycle is wrapped in an OTel trace span using <code>RequestsInstrumentor</code> for automatic HTTP instrumentation. The trace waterfall shows exactly which Redfish API calls were made, their latency, and any errors — giving full visibility into the agent's operational behavior.</p> <p>The agent sends all three signal types to the OTel Collector using <strong>OTLP/gRPC</strong> on port 4317.</p> <h3>The OTel Collector — processing and routing layer</h3> <p>The OpenTelemetry Collector is the central routing engine of the stack. It is the only component that speaks to both the local observability tools and HPE OpsRamp simultaneously.</p> <p>The collector runs three parallel pipelines:</p> <p>The <strong>metrics pipeline</strong> receives OTLP metrics from the agent and the httpcheck receiver, processes them through a resource attribute processor, and exports them to Prometheus on port 8889. Prometheus then pushes to HPE OpsRamp via remote_write.</p> <p>The <strong>logs pipeline</strong> receives OTLP log records from both the agent and the event listener, batches them for efficiency, and exports them directly to HPE OpsRamp's log ingestion endpoint using OTLP/gRPC with OAuth2 authentication managed by the oauth2client extension.</p> <p>The <strong>traces pipeline</strong> receives OTLP spans and exports them to Jaeger for local visualization.</p> <p>The Collector also runs the <strong>httpcheck receiver</strong> — a built-in receiver that polls the Redfish API endpoints directly, producing endpoint health metrics (up/down status and response time) without any agent code.</p> <h3>Prometheus — metrics buffering and remote write</h3> <p>Prometheus serves a dual purpose in this stack. Locally, it provides a queryable time-series database that Grafana uses for dashboards. Remotely, it acts as the push mechanism for metrics into HPE OpsRamp via <code>remote_write</code> — HPE OpsRamp's recommended ingestion path for Prometheus-format metrics.</p> <p>The <code>remote_write</code> configuration in <code>prometheus.yml</code> includes the HPE OpsRamp endpoint, OAuth2 credentials, and the critical resource association labels: <code>type="RESOURCE"</code> and <code>uuid="&#x3C;resourceUUID>"</code> that tell HPE OpsRamp which managed resource each metric belongs to.</p> <h3>HPE OpsRamp — the observability backend</h3> <p>HPE OpsRamp is the management plane that consumes all signals and provides the operational intelligence layer: resource lifecycle management, topology visualization, metric correlation, log management, alert generation, and AIOps capabilities.</p> <p>HPE OpsRamp is not configured in advance for this domain. It learns everything about the Redfish infrastructure from the signals it receives, augmented by a one-time resource provisioning step that uses the HPE OpsRamp REST API to create resource entries with full domain attributes before telemetry begins.</p> <hr> <h2>The communication map: How everything is wired</h2> <p>Understanding the signal flow is essential before diving into installation and code. Here is the complete communication topology.</p> <h3>Signal flow: Metrics</h3> <pre><code class="language-bash">Redfish Emulator :5000 │ HTTP GET (every 15s) ▼ OTel Agent (agent.py) │ OTLP/gRPC :4317 (metrics per chassis/system) ▼ OTel Collector :4317 │ Prometheus exporter :8889 ▼ Prometheus :9090 │ remote_write HTTPS │ Authorization: Bearer &#x3C;token> │ labels: type="RESOURCE", uuid="&#x3C;resourceUUID>" ▼ HPE OpsRamp Metrics Explorer </code></pre> <h3>Signal flow: Logs and events</h3> <pre><code class="language-bash">Redfish Emulator :5000 │ HTTP GET (EventService polling) │ HTTP POST (event push to :9999) ▼ Event Listener (event_listener.py) │ OTLP/gRPC :4317 (LogRecords with host.name, type, uuid) ▼ OTel Collector :4317 │ OTLP/gRPC :443 (batched, gzip compressed) │ Authorization: Bearer &#x3C;oauth2 token> │ Header: tenantId: &#x3C;tenantId> ▼ HPE OpsRamp Log Management </code></pre> <h3>Signal flow: Traces</h3> <pre><code class="language-bash">OTel Agent (RequestsInstrumentor auto-instrumentation) │ OTLP/gRPC :4317 (spans for every HTTP call) ▼ OTel Collector :4317 │ OTLP/gRPC to Jaeger :14317 ▼ Jaeger UI :16686 </code></pre> <h3>Signal flow: Resource provisioning</h3> <pre><code class="language-bash">HPE OpsRamp REST API (one-time, on agent startup) │ POST /tenancy/auth/oauth/token → Bearer token │ POST /api/v2/tenants/{id}/resources → resourceUUID per resource │ POST /api/v2/tenants/{id}/topologies → parent-child relationships ▼ ResourceContext (in-memory registry: hostname → uuid) │ ▼ All subsequent OTel signals carry: type="RESOURCE", uuid="&#x3C;resourceUUID>" </code></pre> <h3>Operational protocols</h3> <table> <thead> <tr> <th>Hop</th> <th>Protocol</th> <th>Port</th> <th>Auth</th> <th>Direction</th> </tr> </thead> <tbody> <tr> <td>Agent → Collector</td> <td>OTLP/gRPC</td> <td>4317</td> <td>None (same host)</td> <td>Push</td> </tr> <tr> <td>Collector → HPE OpsRamp logs</td> <td>OTLP/gRPC</td> <td>443</td> <td>OAuth2 Bearer</td> <td>Push</td> </tr> <tr> <td>Prometheus → HPE OpsRamp metrics</td> <td>HTTPS remote_write</td> <td>443</td> <td>OAuth2 Bearer</td> <td>Push</td> </tr> <tr> <td>Collector httpcheck → Redfish</td> <td>HTTP</td> <td>5000</td> <td>None</td> <td>Pull</td> </tr> <tr> <td>Agent → Redfish</td> <td>HTTP</td> <td>5000</td> <td>None</td> <td>Pull</td> </tr> <tr> <td>HPE OpsRamp REST API</td> <td>HTTPS</td> <td>443</td> <td>OAuth2 Bearer</td> <td>Push</td> </tr> </tbody> </table> <hr> <h2>The resource association problem and how I solved it</h2> <p>This deserves special attention because it is the most important design decision in the entire stack.</p> <p>HPE OpsRamp can receive metrics and logs as raw telemetry and display them in its explorer views. But to associate those signals with a specific managed resource — to show them in the resource's Metrics tab or Logs tab, and to enable topology-aware correlation — HPE OpsRamp requires two mandatory attributes on every signal:</p> <pre><code>type = "RESOURCE" uuid = "5ce1fab8-b706-46bc-8941-47eb32a8f571" </code></pre> <p>The <code>type</code> must be exactly the string <code>"RESOURCE"</code> in uppercase. The <code>uuid</code> must be the <code>resourceUUID</code> that HPE OpsRamp assigned when the resource was created via its REST API.</p> <p>I solved this with a module called <code>resource_context.py</code> — a shared in-memory registry that maps each resource hostname to its full identity attributes. Phase 1 (provisioning) populates this registry. Phase 2 (telemetry) reads from it to enrich every metric data point, log record, and trace span.</p> <p>The critical timing constraint is that <code>agent_resource</code> — the OTel Resource object for the agent — must be constructed <strong>after</strong> Phase 1 completes, not at module import time. If built at import time, the registry is empty and <code>uuid</code> is an empty string. This is a subtle but consequential bug I had to identify and fix explicitly.</p> <hr> <h2>The observability stack in context</h2> <p>What makes this architecture compelling for a demonstration is not any single component — it is the combination. Each component is an open standard or open-source tool:</p> <ul> <li><strong>Redfish</strong> — DMTF open standard for hardware management</li> <li><strong>OpenTelemetry</strong> — CNCF open standard for telemetry</li> <li><strong>Prometheus</strong> — CNCF open-source metrics engine</li> <li><strong>Jaeger</strong> — CNCF open-source distributed tracing</li> <li><strong>Grafana</strong> — Open-source visualization</li> <li><strong>HPE OpsRamp</strong> — Commercial management plane consuming all of the above</li> </ul> <p>The PoC demonstrates that HPE OpsRamp can serve as the observability backend for any infrastructure domain instrumented with OpenTelemetry, without requiring its proprietary agents for signal collection. The ingestion path is fully open standard.</p> <hr> <h2>Conclusion and what comes next</h2> <p>I have laid out the full architecture — the components, the signal flows, the protocols, and the resource association mechanism. The picture should be clear: any infrastructure domain that can be instrumented with OpenTelemetry can be observed in HPE OpsRamp using this pattern.</p> <p>In Part 2, I will walk through the complete installation and verification of every component in the local stack — the Redfish emulator, OTel Collector, Prometheus, Grafana, and Jaeger — running in Docker Compose on a single EC2 instance. I will show exactly how to verify each component is healthy before sending a single signal to HPE OpsRamp.</p> <p>Stay tuned to the HPE Developer Community blog for more insights on HPE HPE OpsRamp (Hybrid Cloud Observability) and practical ideas to apply it in your daily operations.</p> <h2>Want to know more?</h2> <ul> <li><strong>OpenTelemetry project:</strong> <a href="https://opentelemetry.io">opentelemetry.io</a></li> <li><strong>OTel specification:</strong> <a href="https://opentelemetry.io/docs/specs/otel/">opentelemetry.io/docs/specs/otel</a></li> <li><strong>OTel Python SDK:</strong> <a href="https://opentelemetry-python.readthedocs.io/">opentelemetry-python.readthedocs.io</a></li> <li><strong>OTel Collector:</strong> <a href="https://opentelemetry.io/docs/collector/">opentelemetry.io/docs/collector</a></li> <li><strong>OTLP protocol:</strong> <a href="https://opentelemetry.io/docs/specs/otlp/">opentelemetry.io/docs/specs/otlp</a></li> <li><strong>DMTF Redfish standard:</strong> <a href="https://www.dmtf.org/standards/redfish">dmtf.org/standards/redfish</a></li> <li><strong>Redfish emulator:</strong> <a href="https://github.com/DMTF/Redfish-Interface-Emulator">github.com/DMTF/Redfish-Interface-Emulator</a></li> <li><strong>HPE OpsRamp OTLP integration:</strong> <a href="https://docs.opsramp.com/integration/opentelemetry/">docs.opsramp.com</a></li> <li><strong>Prometheus remote_write:</strong> <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write">prometheus.io/docs</a></li> <li><strong>Jaeger tracing:</strong> <a href="https://www.jaegertracing.io">jaegertracing.io</a></li> </ul>https://developer.hpe.com/7-questions-for-champs-developers-empowering-academic-r-d-to-create-cutting-edge-cfd-apps-in-chapel/https://developer.hpe.com/7-questions-for-champs-developers-empowering-academic-r-d-to-create-cutting-edge-cfd-apps-in-chapel/Thu, 26 Mar 2026 23:06:40 GMT<p>external blog</p>https://developer.hpe.com/cascading-dropdown-lists-on-hpe-morpheus-enterprise-forms/https://developer.hpe.com/cascading-dropdown-lists-on-hpe-morpheus-enterprise-forms/Fri, 20 Mar 2026 11:16:09 GMT<h2>Introduction</h2> <p>Authors frequently need to capture additional information during the provisioning of workloads and services. This information is typically collected through user interface controls such as drop-down lists, often populated dynamically from external systems. Downstream, these selections are consumed by automation workflows that tailor deployments and integrate with ancillary systems such as CMDBs, IPAM platforms, or other operational tooling.</p> <p>HPE Morpheus Enterprise provides extensive flexibility for customizing form elements across instance provisioning, workflows, and service catalog items. Drop-down Option Lists can be populated using a variety of Option Source types, including REST-based endpoints and plugin-backed integrations.</p> <p>However, in real-world environments, form inputs are rarely independent. The selection made in one field often determines the valid values in another.</p> <p>This is where cascading (interdependent) drop-down lists become essential.</p> <p>By dynamically filtering or populating one field based on the selection of another, cascading drop-downs introduce context awareness into provisioning forms. This reduces user error, improves data integrity, and ensures that deployments align with environmental constraints and governance requirements.</p> <p>This article demonstrates how to implement cascading drop-down lists in Morpheus using both REST-based and plugin-based Option Lists.</p> <h2>HPE Morpheus terms explained</h2> <p>Some of the terminology used in this article may be misleading or confusing due to its ubiquitous use across a wide range of products and technology domains, including ITSM, virtualization platforms, and service orchestration systems. This section explains the common terms used for HPE Morpheus Enterprise concepts throughout the article:</p> <p><strong>Option List</strong><br/> A list of <em><strong>name</strong></em> and <em><strong>value</strong></em> pairs used to populate UI controls such as drop-downs, radio lists, and type-ahead fields. In the HPE Morpheus Enterprise UI, Option Lists are defined under <em><strong>Library > Options > Option Lists</strong></em>.</p> <p><strong>Option Source</strong><br/> The underlying data source used to populate an Option List. The Option Source type may be static data (JSON or CSV), REST response data, LDAP query results, HPE Morpheus Enterprise API data, or a plugin-based provider. Option Sources are defined as part of creating Option Lists.</p> <p><strong>Input</strong><br/> A web UI control used in an HPE Morpheus Enterprise wizard. A wizard typically contains multiple Input controls, such as when provisioning a VM Instance. Input types include checkbox, hidden value, number, password, radio list, select list, text, text area, and type-ahead. Inputs are defined under <em><strong>Library > Options > Inputs</strong></em></p> <p>A select list Input is populated by a corresponding Option List. This article focuses exclusively on select list Inputs.</p> <p><strong>Forms</strong><br/> HPE Morpheus Forms are collections of Inputs organized in sections called Field Groups. Forms are used exclusively with Catalog Items in the HPE Morpheus Enterprise Service Catalog. Forms are created and configured under <em><strong>Library > Options > Forms</strong></em> in the UI. Forms <em><strong>will not be covered</strong></em> by this article.</p> <p><strong>HPE Morpheus Plugin</strong><br/> A compiled <em><strong>.jar</strong></em> file containing logic that extends the functionality of HPE Morpheus. Plugins are typically written in Groovy and compiled using the Java toolchain.</p> <p><strong>Option Source Provider</strong><br/> A plugin class responsible for retrieving and constructing the data used to populate Option Lists programmatically.</p> <p><strong>Wizard</strong> This article uses the term <strong>wizard</strong> to indicate a section of the web user interface where the end user provides input values. An example of this would be the input fields when a user provisions an instance. In this article an operational workflow wizard is used to show and test Input controls.</p> <h2>Demo environment</h2> <p>To illustrate how to reference external data sources on wizard drop-down Inputs, and then make them interdependent, this article makes use of two demo lab VMs. These include a <em><strong>JSON Server</strong></em> VM and an <em><strong>HPE Morpheus Enterprise</strong></em> appliance. This lab has been tested on HPE Morpheus Enterprise 8.0.</p> <p>To demonstrate how a plugin can supply the data via custom integration, a development environment assumes plugin compile capability as described in the article <a href="https://developer.hpe.com/blog/morpheus-plugin-tutorial-how-to-build-and-compile/">A Beginner’s Guide to Building and Compiling HPE Morpheus Enterprise Plugins</a>.</p> <p>For the purposes of this article, the <em><strong>JSON Server</strong></em> was setup on a clean Debian 12 install, on the same network segment as the <em><strong>HPE Morpheus Enterprise</strong></em> appliance. Data for the <em><strong>JSON Server</strong></em> web endpoint below is supplied by a <em><strong>locations.json</strong></em> file. The content for this file can be found <a href="https://github.com/neilvrhpe/OptionSourceDemo/blob/main/locations.json">here</a>.</p> <p><em><strong>JSON Server</strong></em> <em><strong>v1.0.0</strong></em> was setup using the following commands:</p> <p><code>apt update</code></p> <p><code>apt install nodejs npm -y</code></p> <p><code>npm install json-server</code></p> <p><code>vi locations.json</code></p> <p><code>npx json-server --host 0.0.0.0 --port 80 locations.json</code></p> <p>(Bear in mind that minimal Debian doesn't install with the <em><strong>sudo</strong></em> command by default. Prepend <em><strong>sudo</strong></em> to administrative commands where appropriate on your OS distribution)</p> <p>In this demo, the <em><strong>JSON Server</strong></em> web endpoint responds on <em><strong><a href="http://demojsonserver">http://demojsonserver</a></strong></em> and renders this:</p> <p><img src="/img/jsonserver_home.png" alt=""></p> <p>The 3 sub-pages: <em><strong>/countries</strong></em>, <em><strong>/states</strong></em> and <em><strong>/cities</strong></em> reflect the base objects in the <em><strong>locations.json</strong></em> file:</p> <p><img src="/img/locations_json.png" alt=""></p> <p>As you can see, the endpoint <em><strong><a href="http://demojsonserver/countries">http://demojsonserver/countries</a></strong></em> provides the 3 countries in the* <strong>locations.json</strong>* file:</p> <p><img src="/img/countries_json.png" alt=""></p> <h2>Option Source Types</h2> <p>There are several types of Option Lists. These represent the different ways by which an Option Source can populate the resulting drop-down control. Consider the TYPE input on the NEW OPTION LIST wizard:</p> <p><img src="/img/option_list_types.png" alt=""></p> <p><strong>REST:</strong> Web endpoint requests.</p> <p><strong>Morpheus Api:</strong> HPE Morpheus Enterprise platform elements like VMs or Networks directly.</p> <p><strong>LDAP:</strong> Use LDAP Search Filter syntax to populate Option Lists.</p> <p><strong>Manual:</strong> Define Option List data with CSV or JSON.</p> <p><strong>Plugin:</strong> OptionSourceProvider classes within uploaded plugins supply data in name / value pairs.</p> <p>In this article, I am focusing on the <em><strong>REST Web</strong></em> endpoints and an <em><strong>OptionSourceProvider Plugin</strong></em>.</p> <h2>Creating the Option Lists</h2> <p>To create the <em><strong>Option Lists</strong></em> in <em><strong>HPE Morpheus Enterprise</strong></em>, navigate to <em><strong>Library > Options > Options Lists</strong></em>. Click <em><strong>Add</strong></em>. The following dialog will be displayed:</p> <p><img src="/img/add_optionlist.png" alt=""></p> <p>For you to obtain proper results, replace the URL hostname with the appropriate hostname or IP address of the <em><strong>JSON Server</strong></em> in your environment. Then create three Option Lists that reflect the values below:</p> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>Countries</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>REST</td> </tr> <tr> <td><strong>SOURCE URL:</strong></td> <td><a href="http://demojsonserver/countries">http://demojsonserver/countries</a></td> </tr> <tr> <td><strong>SOURCE METHOD:</strong></td> <td>GET</td> </tr> <tr> <td><strong>REAL TIME:</strong></td> <td>Checked</td> </tr> </tbody> </table> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>States</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>REST</td> </tr> <tr> <td><strong>SOURCE URL:</strong></td> <td><a href="http://demojsonserver/states">http://demojsonserver/states</a></td> </tr> <tr> <td><strong>SOURCE METHOD:</strong></td> <td>GET</td> </tr> <tr> <td><strong>REAL TIME:</strong></td> <td>Checked</td> </tr> </tbody> </table> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>Cities</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>REST</td> </tr> <tr> <td><strong>SOURCE URL:</strong></td> <td><a href="http://demojsonserver/cities">http://demojsonserver/cities</a></td> </tr> <tr> <td><strong>SOURCE METHOD:</strong></td> <td>GET</td> </tr> <tr> <td><strong>REAL TIME:</strong></td> <td>Checked</td> </tr> </tbody> </table> <p>Verify that the three Option Lists reflect the below:</p> <p><img src="/img/option_lists.png" alt=""></p> <h2>Create the Inputs</h2> <p>On their own, Option Lists aren't useful as wizard inputs. To use the values in a UI control, the Option List is attached to a list-based Input Type. These include HTML drop-downs, option lists and type-ahead fields. <br /> Inputs are also variables, with the <em><strong>fieldName</strong></em> property as the variable name. This means that an Input with a <em><strong>fieldName</strong></em> of <em><strong>country</strong></em> will have it's selected value stored as <em><strong>input.country</strong></em>. These variables will be used for filtering in the next section.<br /> To create the wizard Inputs, navigate to <em><strong>Library > Options > Inputs</strong></em>. Click <em><strong>Add</strong></em>.</p> <p><img src="/img/create_input.png" alt=""></p> <p>Create 3 Inputs that correspond to the Option Lists above, with the following values:</p> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>Country</td> </tr> <tr> <td><strong>FIELD NAME:</strong></td> <td>country</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>Select List</td> </tr> <tr> <td><strong>LABEL:</strong></td> <td>Country</td> </tr> <tr> <td><strong>OPTION LIST:</strong></td> <td>Countries</td> </tr> </tbody> </table> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>State</td> </tr> <tr> <td><strong>FIELD NAME:</strong></td> <td>state</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>Select List</td> </tr> <tr> <td><strong>LABEL:</strong></td> <td>State</td> </tr> <tr> <td><strong>OPTION LIST:</strong></td> <td>States</td> </tr> </tbody> </table> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>City</td> </tr> <tr> <td><strong>FIELD NAME:</strong></td> <td>city</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>Select List</td> </tr> <tr> <td><strong>LABEL:</strong></td> <td>City</td> </tr> <tr> <td><strong>OPTION LIST:</strong></td> <td>Cities</td> </tr> </tbody> </table> <p>Inputs should be created as shown below:</p> <p><img src="/img/inputs.png" alt=""></p> <p>The rest of the available Input types are not driven by Option Source list data. These include HTML text, textarea, checkbox, hidden, number and password. Text based Inputs can be validated using regular expressions.</p> <h2>How to test Option List Inputs</h2> <p>A simple way to test Inputs in HPE Morpheus Enterprise is to create an Operational Workflow. These workflows can use Inputs as Workflow wizard Inputs. Navigate to <em><strong>Library > Automation > Workflows</strong></em>. Click <strong><em>Add</em> > <em>Operational Workflow</em></strong>. Provide <em><strong>Test Inputs</strong></em> as the <em><strong>Name</strong></em> and add <em><strong>Country</strong></em>, <em><strong>State</strong></em> and <em><strong>City</strong></em> to the type-ahead <em><strong>Inputs</strong></em> field:</p> <p><img src="/img/new_workflow.png" alt=""></p> <p>Under <strong><em>Library</em> > <em>Automation</em> > <em>Workflows</em></strong>, click the name of the <em><strong>Test Inputs</strong></em> workflow. Click the <em><strong>EXECUTE</strong></em> button. Check that all three drop-downs contain data from the JSON Server endpoints:</p> <p><img src="/img/execute_workflow.png" alt=""></p> <h2>Filtering data</h2> <p>When Option Lists are populated by Option Source data, the list items are stored against an inherent <em><strong>results</strong></em> object. This <em><strong>results</strong></em> construct consists of a list of <em><strong>name</strong></em> and <em><strong>value</strong></em> pairs.</p> <p>Consider the current country JSON data:</p> <pre><code class="language-json">[ { "value": 1, "name": "United States" }, { "value": 2, "name": "Canada" }, { "value": 3, "name": "Australia" } ] </code></pre> <p>If the JSON keys in the list are not exactly <em><strong>name</strong></em> and <em><strong>value</strong></em>, then the drop-down will not be correctly populated. Should the JSON keys be different, an additional step is needed to populate the values correctly. This step is covered in the next step.</p> <p>When the Option List is populated, each entry in the JSON Option Source list will be added onto the <em><strong>results</strong></em> object, causing the corresponding HTML page <em><strong>select</strong></em> tag to be populated with <em><strong>option</strong></em> tags.</p> <p>Navigate to <em><strong>Library > Automation > Workflows</strong></em> and click on the name of the <em><strong>Test Inputs</strong></em> workflow. Click the <em><strong>EXECUTE</strong></em> button. Using developer tools on your browser, inspect the <em><strong>country</strong></em> drop-down HTML element on the web UI page. This reveals that the drop-down control is populated with country name and value IDs.</p> <p><img src="/img/dropdown_html_after.png" alt=""></p> <h3>Filter the state by country using a translation script</h3> <p>At this stage, the <em><strong>state</strong></em> drop-down contains all states in the data source, regardless of which <em><strong>country</strong></em> is selected. Similarly, <em><strong>cities</strong></em> also remain unfiltered, regardless of the selected <em><strong>country</strong></em> and <em><strong>state</strong></em>.</p> <p>This section will look at the two available mechanisms for filtering Option List data based on the values of other UI wizard Inputs, <em><strong>Translation Scripts</strong></em> and <em><strong>Request Scripts</strong></em>.</p> <p>Translation scripts use javascript syntax and consist of these components:</p> <ul> <li>An <em><strong>input</strong></em> object map that represents the selected values of other inputs on the same UI wizard</li> <li>A <em><strong>data</strong></em> list/array object that contains the raw data from the Option Source</li> <li>A <em><strong>results</strong></em> list/array object that the script needs to add <em><strong>name</strong></em> and <em><strong>value</strong></em> objects onto</li> <li><em><strong>name</strong></em> and <em><strong>value</strong></em> objects that get added onto the <em><strong>results</strong></em> object. E.g., {"name": "USA", "value": 1}</li> </ul> <p>Navigate to <em><strong>Library > Options > Options Lists</strong></em> and edit the previously created <em><strong>States</strong></em> Option List using the corresponding pencil icon on the right:</p> <p><img src="/img/edit_optionslist.png" alt=""></p> <p>Add the following code to the <em><strong>TRANSLATION SCRIPT</strong></em> field and click <em><strong>SAVE CHANGES</strong></em>:</p> <pre><code class="language-js">for (var i = 0; i &#x3C; data.length; i++) { if (data[i].countryId == input.country) { results.push({"name": data[i].name, "value": data[i].id}) } } </code></pre> <p><img src="/img/translation_script_code.png" alt=""></p> <p>The above script loops through the state <em><strong>data</strong></em> set and pushes entries onto the <em><strong>results</strong></em> list. <br /> This time there is no <em><strong>value</strong></em> field in the data. The <em><strong>id</strong></em> field is used for the value instead. It is necessary to provide <em><strong>name</strong></em> and <em><strong>value</strong></em> objects via translation scripts if the JSON data doesn't specifically use <em><strong>name</strong></em> and <em><strong>value</strong></em> keys. <br /> The conditional if statement ensures that the selected value of the <em><strong>country</strong></em> Input matches the <em><strong>countryId</strong></em> of the JSON list entry before it can be added to the <em><strong>results</strong></em> list:</p> <p><img src="/img/country_id.png" alt=""></p> <p>Inputs use <em><strong>DEPENDENT FIELD</strong></em> to trigger an Option List refresh when another field changes. Supply the <em><strong>FIELD NAME</strong></em> value of the other Input (country in this case) to trigger the refresh of the state Input. <br /> To set up the refresh trigger of the <em><strong>state</strong></em> drop-down field, navigate to <em><strong>Library > Options > Inputs</strong></em> and edit the <em><strong>State</strong></em> Input. Set the value of the <em><strong>DEPENDENT FIELD</strong></em> input to <strong>country</strong>. Click <em><strong>SAVE CHANGES</strong></em>:</p> <p><img src="/img/dependent_on_country.png" alt=""></p> <p>Navigate back to <em><strong>Library > Automation > Workflows</strong></em> and open the workflow execution dialog for the <em><strong>Test Inputs</strong></em> workflow again. This time, <em><strong>states</strong></em> are filtered by the selected <em><strong>country</strong></em> value:</p> <p><img src="/img/filtered_states.png" alt=""></p> <h3>Filter the city by state using a request script</h3> <p>Some REST web endpoints support filtering by URL parameters. As an example, consider the HPE Moprpheus Enterprise REST API endpoint for servers. Here is an example where the GET request URL is used to filter the list of servers on their <em><strong>parentServerId</strong></em>:</p> <p><code>https://some.morpheus.appliance/api/servers?parentServerId=42</code></p> <p>In this case you don't need to use a request script. For example, to filter <em><strong>cities</strong></em> by a <em><strong>state</strong></em> id of 3, the URL would look like this:</p> <p><code>http://demojsonserver/cities?stateId=3</code></p> <p>Here is the GET request result:</p> <p><img src="/img/cities_filtered_by_url.png" alt=""></p> <p>To implement URL request parameter filtering on the <em><strong>cities</strong></em> Option Source, navigate to <em><strong>Library > Options > Options Lists</strong></em> and edit the <em><strong>Cities</strong></em> Option Source. Populate the* <strong>REQUEST SCRIPT</strong>* text box with the below code:</p> <pre><code class="language-js">results.push({ name: 'stateId', value: data.state || "NoState" }); </code></pre> <p><img src="/img/request_script.png" alt=""></p> <p>This line of code effectively sets the <em><strong>stateId</strong></em> request parameter to the value of the <em><strong>state</strong></em> Form Input, or to "NoState" if no state is selected. The reason for this is that a blank <strong>stateId</strong> request parameter value causes JSON Server to remove the filter entirely, thus showing all entries, instead of no entries.</p> <p>To trigger the refresh of <em><strong>cities</strong></em> upon the selection of a <em><strong>state</strong></em>, navigate to <em><strong>Library > Options > Inputs</strong></em> and edit the <strong>State</strong> input using the pencil icon on the right. Set the value of the <em><strong>DEPENDENT FIELD</strong></em> to <strong>state</strong>. Click <em><strong>SAVE CHANGES</strong></em>:</p> <p><img src="/img/dependent_on_state.png" alt=""></p> <p>The selection of <em><strong>city</strong></em> is now based on <em><strong>state</strong></em>, which is based on the selected <em><strong>country</strong></em>. Navigate back to <em><strong>Library > Automation > Workflows</strong></em> and open the workflow execution dialog for the <em><strong>Test Inputs</strong></em> workflow again. This time, <em><strong>cities</strong></em> are filtered by the selected <em><strong>state</strong></em> value:</p> <p><img src="/img/country_state_city.png" alt=""></p> <h2>Explore, compile, and upload the Option Source plugin</h2> <p>HPE Morpheus Enterprise uses plugins to extend platform functionality, usually onto 3rd party platforms like hypervisors or IPAM systems. This is achieved through Groovy code projects that compile to java archives (.jar files). The .jar files are uploaded via the HPE Morpheus Enterprise UI or API. Plugins implement domain-specific class files called providers. To programmatically populate Option Lists from plugins, you need to implement an Option Source Provider.<br /> This section explores an example of an Option Source plugin by adding zip codes to the above countries, states, cities example. <br /></p> <p><img src="/img/plugin_providers_diag.png" alt=""></p> <p>For more information pertaining to the anatomy of HPE Morpheus Enterprise Plugins, please refer to official plugin documentation at <a href="https://developer.morpheusdata.com">developer.morpheusdata.com</a> or have a look at the blog article <a href="https://developer.hpe.com/blog/morpheus-plugin-tutorial-how-to-build-and-compile/">A Beginner’s Guide to Building and Compiling HPE Morpheus Enterprise Plugins</a>.</p> <p>Download or clone the plugin repository from <a href="https://github.com/neilvrhpe/OptionSourceDemo">https://github.com/neilvrhpe/OptionSourceDemo</a>.</p> <p>Open the project directory using a simple IDE, like Visual Studio code, or even a text editor tool. Expand the <em><strong>src > main > groovy > com > hpe > morpheus > demo</strong></em> directory. View the <em><strong>OptionsSourceDemoPlugin.groovy</strong></em> class file:</p> <p><img src="/img/plugin_class.png" alt=""></p> <p>This file is the HPE Morpheus Enterprise appliance's entry point into the plugin. <em><strong>HPE Morpheus Enterprise plugins</strong></em> always register one or more <em><strong>provider classes</strong></em>. The <em><strong>provider classes</strong></em> supply the plugin functionality. The method call on line 30, <em><strong>this.registerProvider</strong></em>, registers the <em><strong>OptionSourceProvider</strong></em>, which will provide a list of zip codes to the <em><strong>ZipCodes Option List</strong></em> in the next section.</p> <p>View the <em><strong>DemoOptionSourceProvider.groovy</strong></em> class file:</p> <p><img src="/img/provider_class.png" alt=""></p> <p>This class extends <em><strong>AbstractOptionSourceProvider</strong></em>, which enables the plugin to provide a list of <em><strong>name</strong></em> and <em><strong>value</strong></em> pairs for an Option List through a collection of methods. The methods are made available to the platform via the <em><strong>getMethodNames</strong></em> method (line 21). In this example, there is only one method called <em><strong>listZipCodes</strong></em>, which is defined on line 45. It returns static <strong>name</strong> and <strong>value</strong> pairs, although the plugin provides flexibility on how the list is built. Data can easily be retrieved from other systems via SDKs, APIs, or database connections.</p> <p>Open the project directory in a command line terminal and compile the plugin with the relevant <em><strong>gradlew</strong></em>(Linux) or <em><strong>gradlew.bat</strong></em>(Windows) script using the <em><strong>shadowJar</strong></em> argument:</p> <p><img src="/img/compile_plugin.png" alt=""></p> <p>The compiled <em><strong>.jar</strong></em> file will be found in the <em><strong>build/libs</strong></em> subdirectory:</p> <p><img src="/img/jar_file.png" alt=""></p> <p>Upload the <em><strong>.jar</strong></em> file to the <em><strong>Administration > Integrations > Plugins > Add</strong></em> dialog. The plugin should appear in the list as shown below:</p> <p><img src="/img/uploaded_plugin.png" alt=""></p> <p>Edit the uploaded plugin using the pencil icon to confirm that the <em><strong>Option Source Provider</strong></em> is present:</p> <p><img src="/img/plugin_provider.png" alt=""></p> <h2>Add the zip code Option List and Input</h2> <p>At this stage, only the name of the city is added to the <em><strong>City</strong></em> drop-down. This is due to the missing <em><strong>value</strong></em> property, as seen earlier with the <em><strong>state</strong></em> Option List.</p> <p>Add the following code snippet to the <em><strong>cities</strong></em> Option List <em><strong>TRANSLATION SCRIPT</strong></em> field by navigating to <strong>Library</strong> > <strong>Options</strong> > <strong>Options Lists</strong> and clicking the corresponding pencil icon on the right:</p> <pre><code class="language-js">for (var x = 0; x &#x3C; data.length; x++) { results.push({"name": data[x].name,"value": data[x].name}) } </code></pre> <p><img src="/img/cities_translation_script.png" alt=""></p> <p>The above code sets the <em><strong>name</strong></em> property of a city to both the <em><strong>name</strong></em> and <em><strong>value</strong></em> of the city drop-down entry. You can verify this by opening the execution dialog of the <em><strong>Test Inputs</strong></em> workflow. Inspecting the HTML once a <strong>city</strong> is selected confirms this:</p> <p><img src="/img/cities_dropdown_html.png" alt=""></p> <p>The <em><strong>listZipCodes</strong></em> method returns <em><strong>name</strong></em> and <em><strong>value</strong></em> pairs with the <em><strong>city name</strong></em> and the* <strong>zip code value</strong>*. As the Request Script will match using the <em><strong>city name</strong></em>, the value of the drop-down has to be the <em><strong>city name</strong></em> (not the city <em><strong>id</strong></em>).</p> <p>Navigate to <em><strong>Library > Options > Option</strong></em> <em><strong>Lists</strong></em> and click <em><strong>Add</strong></em>. Supply the field values as shown below:</p> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>ZipCodes</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>Plugin</td> </tr> <tr> <td><strong>OPTION LIST:</strong></td> <td>Option Source Demo: listZipCodes</td> </tr> </tbody> </table> <p><strong>TRANSLATION SCRIPT:</strong></p> <pre><code class="language-js">for (var i = 0; i &#x3C; data.length; i++) { if (data[i].name == input.city) { results.push({"name": data[i].value, "value": data[i].value}) } } </code></pre> <hr> <p>The above populates the <em><strong>ZipCodes</strong></em> Option List with the <em><strong>zip code</strong></em> as both the <em><strong>name</strong></em> and <em><strong>value</strong></em> (the drop-down <em><strong>label</strong></em> and <em><strong>value</strong></em>). Click <em><strong>SAVE CHANGES</strong></em></p> <p><img src="/img/zipcodes_option_list.png" alt=""></p> <p>Navigate to <em><strong>Library > Options > Inputs</strong></em> and click <em><strong>Add</strong></em>. Use the below field values:</p> <hr> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td><strong>NAME:</strong></td> <td>ZipCode</td> </tr> <tr> <td><strong>FIELD NAME:</strong></td> <td>zipCode</td> </tr> <tr> <td><strong>DEPENDENT FIELD:</strong></td> <td>city</td> </tr> <tr> <td><strong>VISIBILITY FIELD:</strong></td> <td>city</td> </tr> <tr> <td><strong>TYPE:</strong></td> <td>Select List</td> </tr> <tr> <td><strong>LABEL:</strong></td> <td>Zip Code</td> </tr> <tr> <td><strong>OPTION LIST:</strong></td> <td>ZipCodes</td> </tr> </tbody> </table> <hr> <p><img src="/img/zipcodes_input.png" alt=""></p> <p>Navigate to <em><strong>Library > Automation > Workflows</strong></em> and edit the <em><strong>Test Inputs</strong></em> workflow. As before, add the ZipCode Input to the workflow and save.</p> <p><img src="/img/zipcode_workflow_input.png" alt=""></p> <p>Click the name of the <em><strong>Test Inputs</strong></em> workflow and click <em><strong>EXECUTE</strong></em>. The <em><strong>Zip Code drop-down</strong></em> now appears in the workflow execution dialog, once the <strong>city</strong> is selected:</p> <p><img src="/img/zipcode_dropdown.png" alt=""></p> <p>To implement cascading visibility, set the corresponding <em><strong>VISIBILITY FIELD</strong></em> values on the other inputs to make the visibility of each subsequent drop-down depend on the one before. <em><strong>State to country</strong></em>, and <em><strong>city to state</strong></em>.</p> <h2>Summary</h2> <p>This post only explores HPE Morpheus Enterprise Inputs and Option lists and Option Source Provider Plugins. Under <em><strong>Library > Options > Forms</strong></em>, you can find more advanced Forms as collections of inputs. These form controls enrich functionality on UI forms and enable the customization of the inputs for workloads and services.</p> <p>At the more advanced end of the spectrum are other Plugin Provider Types that model core infrastructure components. These include integrations for Clouds, Networks, Storage systems, and many others. Such Providers tend to be more complex because they interact deeply with HPE Morpheus Enterprise’s provisioning, synchronization, and lifecycle management layers. Understanding how these Provider Types fit together is key to building powerful, production-grade Plugins.</p> <p>Explore the following resources for more information on the different Plugin/Provider types:</p> <p><a href="https://developer.morpheusdata.com/">https://developer.morpheusdata.com</a> (The official plugin documentation)</p> <p><a href="https://share.morpheusdata.com/">https://share.morpheusdata.com</a> (Follow the repository link under the details page of a plugin to see the corresponding source code)</p> <p><a href="https://github.com/hewlettpackard">https://github.com/hewlettpackard</a> (Several repositories with source code for various plugins and automation code samples)</p> <p><a href="https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX">https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX</a> (Plugin code generator demo video)</p>https://developer.hpe.com/7-questions-for-akihiro-hayashi-early-chapel-gpu-support-through-multiresolution-abstractions/https://developer.hpe.com/7-questions-for-akihiro-hayashi-early-chapel-gpu-support-through-multiresolution-abstractions/Wed, 18 Mar 2026 17:10:29 GMT<p>external blog</p>https://developer.hpe.com/implementing-a-local-llm-using-s3-based-model-storage-and-vllm-in-hpe-private-cloud-ai/https://developer.hpe.com/implementing-a-local-llm-using-s3-based-model-storage-and-vllm-in-hpe-private-cloud-ai/Tue, 17 Mar 2026 14:18:20 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Deploying a local Large Language Model (LLM) architecture using S3‑compatible object storage and <em>vLLM</em> as the inference engine provides a scalable, cost‑efficient, and secure foundation for enterprise AI adoption. This approach enables organizations to operationalize AI workloads while maintaining full control over data, performance, and model lifecycle management.</p> <p>This blog post outlines the implementation of a fully local LLM deployment within the HPE Private Cloud AI (PCAI) environment. It deploys <em>MinIO</em>, a high-performance, S3-compatible object storage platform optimized for cloud-native and containerized workloads, as the local model store, providing fast, parallel access to large model artifacts. The architecture then uses <em>vLLM</em> as the optimized inference engine to deliver high-throughput model execution and efficient GPU utilization. Together, these components form a fully self-hosted LLM pipeline within the PCAI environment, enabling organizations to serve models reliably, scale efficiently, and run secure, high‑performance generative AI workloads entirely on‑premises without dependence on external cloud services. </p> <h3>HPE Private Cloud AI</h3> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (PCAI)</a> is a unified, private, full-stack AI cloud platform built for enterprises that require complete control over how they deploy, govern, and scale AI. It is designed to give organizations ownership of their AI environment while enabling rapid innovation across diverse use cases. PCAI directly addresses the most pressing challenges enterprises face as AI adoption accelerates: maintaining data sovereignty, ensuring security, reducing operational complexity, and avoiding the escalating costs and inefficiencies of fragmented, multi-vendor AI infrastructure.</p> <p>PCAI provides a comprehensive, turnkey foundation for end-to-end enterprise AI. It delivers a secure, scalable, and ready-to-use private cloud environment that includes a curated set of pre-built NVIDIA NIM-optimized LLMs and an integrated suite of AI/ML tools and frameworks for data engineering, analytics, and data science. This creates a consistent, governed operational layer for building and running AI services, ensuring that organizations retain full sovereignty over their data, models, and infrastructure while benefiting from a modern, production-ready AI platform.</p> <h3>AI managed by you</h3> <p>At the core of PCAI’s <em>'AI managed by you'</em> philosophy are two complementary approaches that give enterprises both flexibility and operational rigor when deploying and operating AI services:</p> <ul> <li><em>Import Framework</em></li> <li><em>Machine Learning Inference Software (MLIS)</em></li> </ul> <h4><em>Import Framework</em></h4> <p>The PCAI Import Framework feature offers an open, extensible mechanism for organizations to integrate any AI application, framework or third-party tool into the PCAI environment. It enables customers to import partner or open-source AI frameworks and add domain-specific or business-specific AI applications. Once imported, these components are managed through PCAI’s unified lifecycle management, ensuring consistent deployment, monitoring, and governance across the entire private cloud platform.</p> <p>This Import Framework feature is what makes PCAI truly AI-application agnostic. Customers are not limited to prepackaged tools and can freely build the AI ecosystem that fits their strategy.</p> <h4><em>Machine Learning Inference Software (MLIS)</em></h4> <p>HPE Machine Learning Inference Software (MLIS) is an enterprise‑grade platform designed to streamline and operationalize large-scale deployment, management, and monitoring of machine learning (ML) models. It supports the full AI service lifecycle, including model registry configuration, LLM model onboarding, deployment creation, and high-throughput inference serving. Delivered as a prepackaged PCAI‑integrated application, similar to components such as <em>Kubeflow</em> and <em>Ray</em>, MLIS provides a controlled execution layer that handles model versioning, GPU scheduling, performance tuning, and comprehensive observability across availability, latency, and compliance metrics.</p> <p>A core capability of PCAI is its vendor-agnostic support for heterogeneous LLM models. Beyond pre-built NVIDIA NIM models, PCAI can deploy open-source LLMs (e.g., Hugging Face), third-party or proprietary models, and artifacts stored in external or internal object stores such as <em>MinIO</em>. This flexibility enables organizations to consolidate diverse model sources within a unified deployment and governance framework while maintaining enterprise-grade reliability, visibility, and operational consistency.</p> <p>The following sections detail the implementation of a local LLM deployment within the PCAI environment using the Import Framework feature and MLIS.</p> <h3>Prerequisites</h3> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>HPE Private Cloud AI version 1.5.0 or later, running HPE AI Essentials version 1.9.1 or later.</li> <li>Access to an HPE Private Cloud AI workspace (with the <em>Private Cloud AI Administrator</em> role), allowing to performe administrative operations.</li> </ul> <p>The deployment examples in the following sections use the <em>kubectl</em> CLI and <em>kubeconfig</em> to interact with the PCAI Kubernetes (K8s) cluster. However, direct cluster access via <em>kubectl</em> is generally not required.</p> <h3>Setting up model storage using <em>MinIO</em></h3> <p><a href="https://www.min.io/"><em>MinIO</em></a> is a high‑performance, S3‑compatible object storage platform designed for cloud‑native and containerized workloads, offering a lightweight, K8s-friendly architecture that makes it exceptionally easy to deploy and scale. Beyond simple S3 compatibility, <em>MinIO</em> provides enterprise‑grade durability through distributed erasure coding, high aggregate throughput for parallel I/O, and consistent operational behavior across local, on‑premises, and cloud deployments. These characteristics make it particularly effective for AI/ML and LLM pipelines, where fast, reliable access to large, immutable model artifacts is critical for efficient training and inference execution.</p> <p>The following sections describe how to deploy <em>MinIO</em> within the PCAI environment using the Import Framework and configure it as a local model repository for storing and managing LLM artifacts.</p> <h4>Deploy <em>MinIO</em> via <em>Import Framework</em></h4> <p>Using a revised <em>MinIO</em> Helm chart, available in the GitHub repository <a href="https://github.com/GuopingJia/pcai-helm-examples/tree/main/minio">pcai-helm-examples</a>, <em>MinIO</em> can be deployed into the PCAI environment through the Import Framework by following the steps outlined below. This Helm chart is derived from the official <a href="https://github.com/minio/minio/tree/master/helm/minio">MinIO charts</a> and augmented with the required <em>Istio</em> <em>VirtualService</em> and Kyverno <em>ClusterPolicy</em> manifests to ensure compatibility with PCAI’s service mesh and policy controls.</p> <ul> <li>In the PCAI left navigation panel, select <strong>Tools &#x26; Frameworks</strong>. Click <em><strong>Import Framework</strong></em>.</li> </ul> <p><img src="/img/pcai-tools-frameworks-import-framework.png" alt=""></p> <ul> <li>By following the Import Framework wizard workflow, <em>MinIO</em> can be deployed into the PCAI environment within minutes.</li> </ul> <p><img src="/img/import-framework-minio.png" alt=""></p> <p>Run the following commands to verify the <em>MinIO</em> deployment in the namespace <em>'minio'</em> of the PCAI K8s cluster:</p> <pre><code class="language-shell">$ kubectl get all -n minio NAME          READY   STATUS    RESTARTS   AGE pod/minio-0   1/1     Running   0          4h30m pod/minio-1   1/1     Running   0          4h30m pod/minio-2   1/1     Running   0          4h30m pod/minio-3   1/1     Running   0          4h30m NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE service/minio           ClusterIP   10.96.2.234   &#x3C;none>        9000/TCP   4h30m service/minio-console   ClusterIP   10.96.1.181   &#x3C;none>        9001/TCP   4h30m service/minio-svc       ClusterIP   None          &#x3C;none>        9000/TCP   4h30m NAME                     READY   AGE statefulset.apps/minio   4/4     4h30m $ kubectl get pvc -n minio NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE export-minio-0   Bound    pvc-8b5e262b-3667-4da8-a886-8e8596b1cbe0   500Gi      RWO            gl4f-filesystem   &#x3C;unset>                 4h30m export-minio-1   Bound    pvc-04770c71-301c-47f1-b92f-d7461c8bfca0   500Gi      RWO            gl4f-filesystem   &#x3C;unset>                 4h30m export-minio-2   Bound    pvc-527387c5-0ca8-4991-8d0f-340dfdead8b4   500Gi      RWO            gl4f-filesystem   &#x3C;unset>                 4h30m export-minio-3   Bound    pvc-d52bfe41-b7a4-4f06-8275-d2b566198012   500Gi      RWO            gl4f-filesystem   &#x3C;unset>                 4h30m </code></pre> <h4>Access <em>MinIO</em> console via its endpoint</h4> <p>After <em>MinIO</em> is deployed via the Import Framework, an imported <em>MinIO</em> tile appears under <strong>Tools &#x26; Frameworks</strong>.</p> <p><img src="/img/import-framework-minio-imported.png" alt=""></p> <p>Click <em><strong>Open</strong></em> from the <em>MinIO</em> tile will open the <em>MinIO</em> console login page.</p> <p><img src="/img/import-framework-minio-login.png" alt=""></p> <h4>Create a dedicated bucket</h4> <p>After logging into the <em>MinIO</em> console, click <em><strong>Create a Bucket</strong></em> to create a S3 bucket for LLM models.</p> <p><img src="/img/create-minio-bucket.png" alt=""></p> <p>Enter <em>Name</em>, e.g., <em>'s3-ai-models'</em> and select optional features for <em>Versioning</em>, <em>Object Locking</em>, or <em>Quota</em>. Click <em><strong>Create Bucket</strong></em>.</p> <p><img src="/img/create-ai-model-bucket.png" alt=""></p> <h4>Upload model artifacts to the bucket</h4> <p>As an example LLM model, the <em>Qwen3-0.6B-Base</em> has been retrieved from the <a href="https://huggingface.co/Qwen/Qwen3-0.6B-Base"><em>Hugging Face</em></a> and stored locally in the directory <em>'Qwen3-0.6B-Base'</em>.</p> <pre><code class="language-shell">$ git clone https://huggingface.co/Qwen/Qwen3-0.6B-Base Cloning into 'Qwen3-0.6B-Base'... remote: Enumerating objects: 46, done. remote: Counting objects: 100% (3/3), done. remote: Compressing objects: 100% (3/3), done. remote: Total 46 (delta 0), reused 0 (delta 0), pack-reused 43 (from 1) Unpacking objects: 100% (46/46), done. Checking out files: 100% (10/10), done. $ ls -al Qwen3-0.6B-Base/ total 1175909 drwxr-xr-x 1 GUJ 1049089 0 Mar 4 18:54 . drwxr-xr-x 1 GUJ 1049089 0 Mar 4 18:52 .. drwxr-xr-x 1 GUJ 1049089 0 Mar 4 18:54 .git -rw-r--r-- 1 GUJ 1049089 1554 Mar 4 18:52 .gitattributes -rw-r--r-- 1 GUJ 1049089 757 Mar 4 18:52 config.json -rw-r--r-- 1 GUJ 1049089 144 Mar 4 18:52 generation_config.json -rw-r--r-- 1 GUJ 1049089 11544 Mar 4 18:52 LICENSE -rw-r--r-- 1 GUJ 1049089 1823241 Mar 4 18:52 merges.txt -rw-r--r-- 1 GUJ 1049089 1192135096 Mar 4 18:54 model.safetensors -rw-r--r-- 1 GUJ 1049089 3030 Mar 4 18:52 README.md -rw-r--r-- 1 GUJ 1049089 7334926 Mar 4 18:53 tokenizer.json -rw-r--r-- 1 GUJ 1049089 9916 Mar 4 18:53 tokenizer_config.json -rw-r--r-- 1 GUJ 1049089 2776833 Mar 4 18:53 vocab.json </code></pre> <p>In the <em>MinIO</em> console, select the created bucket <em>'s3-ai-models'</em>.</p> <p><img src="/img/s3-ai-model-bucket.png" alt=""></p> <p>Click <em><strong>Upload</strong></em>, choose <em>Upload Folder</em>, and select the <em>'Qwen3-0.6B-Base'</em> directory to upload the model from local storage.</p> <p><img src="/img/import-framework-minio-ai-bucket.png" alt=""></p> <p>After few minutes, the bucket <em>'s3-ai-models'</em> displays the uploaded LLM model weights along with the associated configuration and tokenizer files from the local directory.</p> <p><img src="/img/import-framework-minio-ai-model.png" alt=""></p> <h4>Configure access credentials</h4> <p>In the <em>MinIO</em> console, navigate to <strong>Access Keys</strong>, click <em><strong>Create access key +</strong></em>. Specify the <em>Expiry</em>, <em>Name</em>, <em>Description</em> and <em>Comments</em> fields. Click <em><strong>Create</strong></em>.</p> <p><img src="/img/import-framework-minio-create-access-key.png" alt=""></p> <p>Store the newly generated <em>Access Key</em> and <em>Secret Key</em> in a secure location.</p> <p><img src="/img/bucket-access-key.png" alt=""></p> <p>These <em>Access Key</em> and <em>Secret Key</em> values will be required in the subsequent configuration steps to enable secure model retrieval.</p> <h4>Connect S3 data source</h4> <p>The following steps describe how to register <em>MinIO</em> as a S3 data source in PCAI. Instead of accessing <em>MinIO</em> directly through its internal service endpoint, PCAI connects to a configured S3-compatible <em>MinIO</em> endpoint using the provided access and secret keys to retrieve model artifacts. This S3 data source serves as the object-storage backend for model ingestion and runtime access, and can also be accessed by external clients such as <em>Spark</em> or <em>Kubeflow</em> notebooks to ensure consistent, interoperable data access across the platform.</p> <ul> <li>In the PCAI left navigation panel, go to <strong>Data Engineering</strong> and select <em>Data Sources</em>.</li> </ul> <p><img src="/img/data-sources.png" alt=""></p> <ul> <li>Select <strong>Object Store Data</strong> tab. Click <em><strong>Add New Object Store</strong></em>.</li> </ul> <p><img src="/img/object-store-data.png" alt=""></p> <ul> <li>Locate the <strong>MinIO S3</strong> tile. Click <em><strong>Add MinIO S3</strong></em>.</li> </ul> <p><img src="/img/add-minio-s3-data-source-type.png" alt=""></p> <ul> <li>Enter <em>Name</em> as <em>'s3-minio'</em>, specify <em>Endpoint</em> (for example, <em>'<a href="http://minio.minio.svc.cluster.local:90000">http://minio.minio.svc.cluster.local:90000</a>'</em>), provide the <em>Access Key</em> and <em>Secret Key</em> created earlier, and select <em>Insecure</em> option. Click <em><strong>Add</strong></em>.</li> </ul> <p><img src="/img/add-minio-s3-data-source.png" alt=""></p> <ul> <li>A new tile named <em>'s3-minio'</em>, displaying its endpoint URL (for example, <em>'<a href="http://s3-minio-service.ezdata-system.svc.cluster.local:30000">http://s3-minio-service.ezdata-system.svc.cluster.local:30000</a>'</em>) now appears on the <strong>Data Sources</strong> page.</li> </ul> <p><img src="/img/minio-s3-data-source.png" alt=""></p> <h3>Model registry configuration and deployment in <em>MLIS</em></h3> <p>HPE Machine Learning Inference Software (MLIS) is natively integrated into PCAI to provide a production-ready, standardized runtime for large-scale AI inference, and the following sections detail the configuration of the S3-based model registry, the creation of a LLM model using this registry, and its deployment through <em>vLLM</em> within MLIS.</p> <h4>Define a local S3 model registry</h4> <ul> <li>In the PCAI left navigation panel, go to <strong>Tools &#x26; Frameworks</strong>, and select the <strong>HPE MLIS</strong> tile. Click <em><strong>Open</strong></em>.</li> </ul> <p><img src="/img/pcai-mlis.png" alt=""></p> <ul> <li>Navigate to <strong>Registries</strong>. Click <em><strong>Create Registry</strong></em>.</li> </ul> <p><img src="/img/create-new-registry.png" alt=""></p> <ul> <li>Select <strong>Internal S3 registry</strong> as the model registry provider. Click <em><strong>Continue</strong></em>.</li> </ul> <p><img src="/img/mlis-internal-s3-registry.png" alt=""></p> <ul> <li>Enter <em>Name</em> as <em>'s3-minio-registry'</em>, choose <em>Object store</em> as <em>'s3-minio'</em>, and select <em>Bucket</em> as <em>'s3-ai-models'</em>. Click <em><strong>Create registry</strong></em>.</li> </ul> <p><img src="/img/create-s3-minio-registry.png" alt=""></p> <ul> <li>A registry named <em>'s3-minio-registry'</em> now appears on the <strong>Registries</strong> page.</li> </ul> <p><img src="/img/s3-minio-registry.png" alt=""></p> <h4>Create a packaged model</h4> <ul> <li>In MLIS, navigate to <strong>Packaged Models</strong> and click <em><strong>Create Packaged Model</strong></em>. Under the <strong>Your model</strong> tab, enter <em>Name</em> as <em>'qwen3-06b-basae'</em> and <em>Description</em> as <em>'Qwen3-0.6B-Base'</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-s3-packaged-model.png" alt=""></p> <ul> <li>Under the <strong>Storage</strong> tab, set <em>Registry</em> to <em>'s3-minio-registry'</em>, choose <em>Model format</em> as <em>'Custom'</em>, specify <em>image</em> as <em>'vllm/vllm-openai:latest'</em>, set <em>URL</em> to <em>'s3://s3-ai-models/Qwen3-0.6B-Base'</em>, and select <em>Model category</em> as <em>'llm'</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-s3-packaged-model-storage.png" alt=""></p> <ul> <li>Under the <strong>Resources</strong> tab, select <em>Resource Template</em>, for example as <em>'gpu-tiny'</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-s3-packaged-model-resources.png" alt=""></p> <ul> <li>Under the <strong>Advanced</strong> tab, set <em>Arguments</em> to <em>'--model /mnt/models --port 8080'</em>. Click <em><strong>Done</strong></em>.</li> </ul> <p><img src="/img/create-s3-packaged-model-advanced.png" alt=""></p> <p>MLIS retrieves the model from the mount point <em>'/mnt/models'</em>.</p> <h4>Create model deployment</h4> <ul> <li>In MLIS, navigate to <strong>Deployments</strong> and click <em><strong>Create Deployment</strong></em>. Under the <strong>Deployment</strong> tab, enter <em>Name</em> as <em>'s3-minio-registry'</em> and select <em>Namespace</em> as <em>'project-user-guoping-jia'</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-model-deployment.png" alt=""></p> <ul> <li>Under the <strong>Packaged Model</strong> tab, select <strong>Packaged model</strong> as <em>'qwen3-06b-base'</em> from the drop-down. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-model-deployment-packaged-model.png" alt=""></p> <ul> <li>Under the <strong>Scaling</strong> tab, select <em>Auto scaling template</em>, such as <em>'fixed-1'</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/create-model-deployment-scaling.png" alt=""></p> <ul> <li>Under the <strong>Advanced</strong> tab, add <em>Environment Variables</em>, for example <em>'AIOLI_DISABLE_LOGGER = 1'</em>. Click <em><strong>Done</strong></em>.</li> </ul> <p><img src="/img/create-model-deployment-advanced.png" alt=""></p> <ul> <li>After a few minutes, the deployment <em>'s3-minio-registry'</em> appears in <em>Ready</em> status on the <strong>Deployments</strong> page, along with its configured model deployment endpoint, for example: <em>'<a href="https://s3-minio-registry.project-user-guoping-jia.serving.ai-application.pcai0109.dc15.hpecolo.net/">https://s3-minio-registry.project-user-guoping-jia.serving.ai-application.pcai0109.dc15.hpecolo.net/</a>'</em>.</li> </ul> <p><img src="/img/create-model-deployment-s3-minio-registry.png" alt=""></p> <ul> <li>Click the <strong>...</strong> menu next to the deployment <em>'s3-minio-registry'</em> and select <strong>Open</strong>. The <strong>Timeline</strong> tab displays all model deployment details.</li> </ul> <p><img src="/img/create-model-deployment-s3-minio-registry-open.png" alt=""></p> <p>You have successfully configured a new registry backed by the local <em>MinIO</em> S3 data source, packaged the model, and deployed it through MLIS. With managed access tokens provided during configuration, the resulting inference service endpoint can be integrated into a wide range of AI tooling, including the <em>VSCode AI Toolkit</em> and LLM-serving frameworks such as <em>Open WebUI</em>, to enable code generation workflows and virtual assistant capabilities.</p> <h3>Conclusion</h3> <p>This blog post discussed and demonstrated the implementation of a fully local, privacy-preserving LLM deployment using <em>MinIO</em> for centralized S3-compatible model storage and <em>vLLM</em> for high-performance inference, integrated into the PCAI environment through the Import Framework and MLIS. This architecture enables scalable, secure, and cost-efficient LLM operations while eliminating reliance on external APIs or third-party model-hosting services.</p> <p>A local LLM stack provides a robust foundation for advanced AI initiatives, including <em>RAG</em> pipelines, domain-specific fine-tuning, agent-based systems, and multimodal model integration. By combining S3-compatible storage with <em>vLLM</em>'s optimized inference runtime, organizations gain a flexible and extensible platform capable of evolving with emerging AI capabilities without requiring major architectural changes or introducing new vendor dependencies.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud AI and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/managing-application-accounts-on-hpe-ilo7-servers-with-ilorest-a-complete-guide/https://developer.hpe.com/managing-application-accounts-on-hpe-ilo7-servers-with-ilorest-a-complete-guide/Sun, 15 Mar 2026 14:32:16 GMT<h2>Introduction</h2> <p>Starting with <strong>iLO 7</strong>, HPE introduced a new security paradigm for host-based applications that need to communicate with the Integrated Lights-Out (iLO) management processor: <strong>Application Accounts</strong> (appaccounts). Unlike traditional iLO user accounts that humans use to log in via the iLO web GUI or SSH, application accounts are purpose-built credentials that allow <em>software running on the host OS</em> to authenticate with iLO securely — with the secret material stored inside the server's <strong>Trusted Platform Module (TPM)</strong>.</p> <p>The <code>ilorest</code> command-line tool provides a dedicated <code>appaccount</code> command that lets administrators create, delete, check the existence of, view details of, and reactivate these application accounts. This guide walks through every capability in depth, with real-world examples and explanations of the internal mechanics.</p> <hr> <h2>Terminology: Application account vs. Application token</h2> <p>Before diving in, it's important to understand two related but distinct concepts:</p> <table> <thead> <tr> <th align="left">Term</th> <th align="left">What It Is</th> <th align="left">Where It Lives</th> <th align="left">Lifecycle</th> </tr> </thead> <tbody> <tr> <td align="left"><strong>Application account</strong> (appaccount)</td> <td align="left">The application's identity record in iLO. Includes app name, ID, and authorization configuration.</td> <td align="left"><strong>iLO</strong> (Redfish API)</td> <td align="left">Created / deleted explicitly by the admin</td> </tr> <tr> <td align="left"><strong>Application token</strong> (apptoken)</td> <td align="left">The cryptographic secret the application uses to authenticate. Has a <strong>defined lifespan</strong>.</td> <td align="left"><strong>TPM</strong> (hardware)</td> <td align="left">Created with the account; expires and is renewed via <code>reactivate</code></td> </tr> </tbody> </table> <p><strong>When to use which term:</strong></p> <ul> <li>Use <strong><code>appaccount create</code></strong> / <strong><code>appaccount delete</code></strong> when you're managing the <em>full lifecycle</em> of an application's registration — creating or removing it from both TPM and iLO.</li> <li>Use <strong><code>appaccount reactivate</code></strong> when the <em>token has expired</em> but the account itself is still valid — you're renewing the secret, not recreating the account.</li> <li>Use <strong><code>appaccount exists</code></strong> / <strong><code>appaccount details</code></strong> to inspect the state of accounts and their tokens across both stores.</li> </ul> <p>Think of it this way: the <strong>appaccount</strong> is the "identity card" and the <strong>apptoken</strong> is the "password" on that card. The card persists, but the password expires and needs periodic renewal.</p> <hr> <h2>Which applications are supported?</h2> <blockquote> <p><strong>Important:</strong> As of today, application accounts can only be created for the following HPE-recognized applications:</p> <table> <thead> <tr> <th align="left">Application</th> <th align="left">Description</th> </tr> </thead> <tbody> <tr> <td align="left"><strong>SUM</strong> (Smart Update Manager)</td> <td align="left">HPE's firmware and driver update tool</td> </tr> <tr> <td align="left"><strong>SUT</strong> (Smart Update Tools)</td> <td align="left">Automated OS-level update agent</td> </tr> <tr> <td align="left"><strong>AMS</strong> (Agentless Mgmt Service)</td> <td align="left">Collects host OS information for iLO</td> </tr> <tr> <td align="left"><strong>iLOrest</strong> (self-registered)</td> <td align="left">HPE's RESTful interface tool itself</td> </tr> </tbody> </table> <p><strong>Custom or third-party applications are not yet supported.</strong> You cannot register an arbitrary application with its own <code>hostappid</code> / <code>hostappname</code> / <code>salt</code> unless it is one of the recognized HPE tools listed above.</p> <p>When using the <code>--self</code> flag, iLOrest creates a self-registered account with the reserved ID prefix <code>00b5</code>, which is exclusively allocated to iLOrest's own self-registration mechanism.</p> </blockquote> <hr> <h2>Prerequisites</h2> <p>Before running any <code>appaccount</code> command, ensure the following:</p> <ol> <li><strong>iLO 7 v1.11.00 or later firmware</strong> is installed on the server. The command explicitly checks the iLO generation and will fail with an <code>IncompatibleiLOVersionError</code> on iLO5 or iLO6.</li> <li><strong>Virtual NIC (VNIC) is enabled</strong> in iLO. All appaccount operations communicate with iLO over the internal VNIC interface at <code>https://16.1.15.1</code>. If VNIC is not enabled or misconfigured, you will see a <code>VnicExistsError</code>.</li> <li><strong>Root or Administrator privileges</strong> are required on the host OS. The command checks for <code>root</code> on Linux and <code>Administrator</code> on Windows; unprivileged users are blocked with a <code>UserNotAdminError</code>.</li> <li><strong>iLO Administrator credentials</strong> (<code>-u</code> / <code>-p</code>) are required for create, reactivate, and delete operations. Even for self-registered account deletion, credentials are strongly recommended to ensure the account is fully removed from both TPM and iLO (see the detailed explanation under the <strong>Deleting an application account</strong> section).</li> </ol> <blockquote> <p><strong>Note on CHIF vs. VNIC:</strong> The CHIF (Channel Interface) driver used in iLO 5 and iLO 6 is <strong>not exposed in iLO7</strong>. All host-to-iLO communication on iLO 7 is performed exclusively through the <strong>VNIC (Virtual NIC)</strong> interface. The appaccount command uses the VNIC for both local TPM operations and iLO REST API calls.</p> </blockquote> <hr> <h2>Understanding dual storage: TPM + iLO</h2> <p>Every application account lives in <strong>two places simultaneously</strong>:</p> <table> <thead> <tr> <th align="left">Storage</th> <th align="left">What it holds</th> <th align="left">Access mechanism</th> </tr> </thead> <tbody> <tr> <td align="left"><strong>TPM</strong> (Trusted Platform Module)</td> <td align="left">The <strong>apptoken</strong> — cryptographic secret for authentication. Has an expiry lifecycle.</td> <td align="left">VNIC driver call from the host OS (no REST auth needed)</td> </tr> <tr> <td align="left"><strong>iLO</strong> (Redfish REST API)</td> <td align="left">The <strong>appaccount</strong> — identity record iLO uses to authorize REST API requests.</td> <td align="left">Authenticated REST session (username + password)</td> </tr> </tbody> </table> <p>This distinction is critical: <strong>TPM operations are local VNIC driver calls</strong> that don't require iLO REST authentication, while <strong>iLO-side operations always require a valid authenticated REST session</strong>. This has real implications for commands like <code>delete --self</code>, as explained below.</p> <p>When everything is in sync, both locations agree. But situations like a <strong>TPM clear</strong> can cause the two to go out of sync — the appaccount still exists in iLO but the apptoken is no longer in TPM. The <code>appaccount</code> command handles these <strong>orphaned accounts</strong> transparently during the <code>create</code> workflow.</p> <hr> <h2>Subcommands at a glance</h2> <table> <thead> <tr> <th align="left">Subcommand</th> <th align="left">Purpose</th> <th align="left">Does it require iLO Credentials?</th> </tr> </thead> <tbody> <tr> <td align="left"><code>create</code></td> <td align="left">Create a new appaccount (apptoken in TPM + appaccount in iLO). Silently cleans up orphans.</td> <td align="left"><strong>Yes</strong> — always</td> </tr> <tr> <td align="left"><code>delete</code></td> <td align="left">Remove an appaccount from both TPM and iLO</td> <td align="left"><strong>Yes</strong> — always recommended</td> </tr> <tr> <td align="left"><code>exists</code></td> <td align="left">Check whether an appaccount / apptoken exists in TPM or iLO</td> <td align="left">No</td> </tr> <tr> <td align="left"><code>details</code></td> <td align="left">List all appaccounts with their TPM / iLO presence status</td> <td align="left">No (more complete with credentials)</td> </tr> <tr> <td align="left"><code>reactivate</code></td> <td align="left">Renew an <strong>expired apptoken</strong> in TPM (token rotation). Does <strong>not</strong> handle orphans.</td> <td align="left"><strong>Yes</strong> — always</td> </tr> </tbody> </table> <blockquote> <p><strong>Note on <code>delete --self</code>:</strong> The CLI allows running <code>delete --self</code> without credentials, but this is <strong>not recommended</strong>. Without credentials, only the TPM token is deleted. The iLO-side account <strong>cannot</strong> be removed without an authenticated REST session, leaving an <strong>orphaned account in iLO</strong>. Always provide <code>-u</code> and <code>-p</code> for a complete deletion.</p> </blockquote> <hr> <h2>1. Creating an application account</h2> <h3>What it does</h3> <p>The <code>appaccount create</code> subcommand generates a new apptoken and saves it in TPM, while simultaneously registering the corresponding appaccount in iLO. This is the primary way to set up application-level authentication.</p> <p>Behind the scenes, the command:</p> <ol> <li><strong>Validates prerequisites</strong> — checks for root/admin, VNIC access, iLO7+.</li> <li><strong>Silently handles orphans</strong> — if a previous account with the same ID exists in iLO but <em>not</em> in TPM (e.g., after a TPM clear), the command <strong>automatically and silently cleans up</strong> the stale iLO account, then proceeds to create a fresh account in both stores as if nothing happened. The user sees only the success message — no extra steps are required.</li> <li><strong>Generates and saves</strong> — calls <code>generate_save_token()</code> to create the apptoken in TPM and register the appaccount in iLO in a single operation.</li> </ol> <h3>Why <code>create</code> is the recommended recovery path after TPM clear</h3> <p>After a TPM clear, all apptokens are wiped but the corresponding appaccounts remain in iLO (orphaned). Rather than requiring a manual delete-then-create sequence, the <code>create</code> command detects this situation automatically:</p> <pre><code>TPM clear occurs → apptokens wiped → appaccounts remain in iLO (orphaned) ↓ ilorest appaccount create (with same parameters) ↓ Orphaned iLO account silently deleted → fresh account created in both TPM and iLO ↓ "Application account has been generated and saved successfully." </code></pre> <p>The user doesn't need to know about the orphan — <code>create</code> handles it transparently.</p> <h3>Syntax</h3> <p>There are two modes:</p> <h4>Mode 1: Named application (for SUM, SUT, or AMS)</h4> <pre><code>ilorest appaccount create --hostappname &#x3C;name> --hostappid &#x3C;id> --salt &#x3C;salt> -u &#x3C;ilo_user> -p &#x3C;ilo_password> </code></pre> <p>All three parameters (<code>--hostappname</code>, <code>--hostappid</code>, <code>--salt</code>) are <strong>mandatory</strong> in this mode. They must correspond to one of the recognized HPE applications (SUM, SUT, AMS). The <code>hostappid</code> is a hex string of 4 or more characters that uniquely identifies the application.</p> <p><strong>Example — Creating an account for SUM:</strong></p> <pre><code class="language-shell">ilorest appaccount create --hostappname SUM --hostappid a1b2c3d4 --salt sumsecret -u admin -p iLOpassw0rd </code></pre> <p><strong>Expected output:</strong></p> <pre><code>Application account has been generated and saved successfully. </code></pre> <h4>Mode 2: Self-registration (for iLOrest itself)</h4> <pre><code>ilorest appaccount create --self -u &#x3C;ilo_user> -p &#x3C;ilo_password> </code></pre> <p>The <code>--self</code> flag tells iLOrest to create an account for <em>itself</em>, using the reserved ID prefix <code>00b5</code>. You <strong>cannot</strong> combine <code>--self</code> with <code>--hostappname</code>, <code>--hostappid</code>, or <code>--salt</code> — doing so will produce an error.</p> <p><strong>Example — Self-registering iLOrest:</strong></p> <pre><code class="language-shell">ilorest appaccount create --self -u admin -p iLOpassw0rd </code></pre> <p><strong>Expected output:</strong></p> <pre><code>Application account has been generated and saved successfully. </code></pre> <h3>What if the account already exists?</h3> <p>If you try to create an account that already exists in both TPM and iLO, the command does <strong>not</strong> fail — it simply informs you:</p> <pre><code>Application account already exists for the specified host application. </code></pre> <h3>Error scenarios</h3> <table> <thead> <tr> <th align="left">Error</th> <th align="left">Meaning</th> </tr> </thead> <tbody> <tr> <td align="left"><code>SavinginTPMError</code></td> <td align="left">The apptoken could not be written to TPM. Delete the account and retry.</td> </tr> <tr> <td align="left"><code>SavinginiLOError</code></td> <td align="left">The appaccount could not be registered in iLO. Delete and retry.</td> </tr> <tr> <td align="left"><code>InvalidCredentialsError</code></td> <td align="left">The iLO username / password is wrong.</td> </tr> <tr> <td align="left"><code>GenerateAndSaveAccountError</code></td> <td align="left">A general failure occurred during generation. Retry later.</td> </tr> </tbody> </table> <p>In all failure cases, the error message also suggests: <em>"Alternatively, you can use the <code>--no_app_account</code> option in the <strong>login</strong> Command to log in using your iLO user account credentials."</em></p> <hr> <h2>2. Deleting an application account</h2> <h3>What it does</h3> <p>The <code>appaccount delete</code> subcommand removes an application account from <strong>both</strong> TPM (the apptoken) and iLO (the appaccount). The command follows a two-step process:</p> <ol> <li><strong>Delete the apptoken from TPM</strong> — removes the cryptographic secret via a VNIC driver call. This does <strong>not</strong> require iLO REST authentication.</li> <li><strong>Delete the appaccount from iLO</strong> — removes the account via an authenticated <code>DELETE</code> call to <code>/redfish/v1/AccountService/Oem/Hpe/AppAccounts/&#x3C;id></code>. This <strong>requires a valid iLOrest session</strong>.</li> </ol> <p>If the account exists in only one location (e.g., iLO but not TPM after a TPM clear), the command still succeeds as long as it was removed from at least one.</p> <h3>Why credentials are always recommended</h3> <p>iLO does not allow any modifications to its REST API resources without authentication. This means:</p> <ul> <li><strong>TPM deletion</strong> (apptoken) works without credentials — it's a local VNIC driver operation.</li> <li><strong>iLO deletion</strong> (appaccount) requires credentials to create a REST session, query the account list, and perform the DELETE call.</li> </ul> <p>If you run <code>delete --self</code> without <code>-u</code> / <code>-p</code>, the command will:</p> <ol> <li>✅ Successfully delete the apptoken from TPM.</li> <li>❌ <strong>Silently skip</strong> the iLO-side appaccount deletion (no REST session available).</li> <li>✅ Report "success" because the TPM deletion succeeded.</li> </ol> <p><strong>The result is an orphaned appaccount left in iLO.</strong> While this orphan is harmless (it can't be used without the matching TPM token) and will be silently cleaned up by a subsequent <code>create</code>, it clutters the account list. To ensure a <strong>clean, immediate deletion</strong>, always provide credentials.</p> <h3>Credential rules</h3> <ul> <li><strong>Deleting another application's account</strong> (e.g., SUM, SUT, AMS): iLO Administrator credentials (<code>-u</code> and <code>-p</code>) are <strong>mandatory</strong>. The command will reject the request without them.</li> <li><strong>Deleting your own self-registered account</strong> (<code>--self</code> or ID containing <code>00b5</code>): Credentials are not enforced by the CLI, but are <strong>strongly recommended</strong> for a complete deletion.</li> </ul> <h3>Short appid expansion</h3> <p>When you provide a 4-character <code>hostappid</code>, the command automatically expands it to the full ID using the <code>ExpandAppId()</code> function. This makes it convenient to use the short IDs shown in the <code>details</code> output.</p> <h3>Examples</h3> <p><strong>Delete a SUT account (credentials required):</strong></p> <pre><code class="language-shell">ilorest appaccount delete --hostappid a1b2 -u admin -p iLOpassw0rd </code></pre> <p><strong>Delete the iLOrest self-registered account (credentials recommended):</strong></p> <pre><code class="language-shell">ilorest appaccount delete --self -u admin -p iLOpassw0rd </code></pre> <p><strong>Expected output on success:</strong></p> <pre><code>Application account has been deleted successfully. </code></pre> <p><strong>If the account does not exist:</strong></p> <pre><code>The application account you are trying to delete does not exist. </code></pre> <hr> <h2>3. Checking if an application account exists</h2> <h3>What it does</h3> <p>The <code>appaccount exists</code> subcommand checks whether an appaccount/apptoken is present in <strong>either</strong> TPM or iLO (or both). It is a read-only operation.</p> <p>Internally, it:</p> <ol> <li>Expands a short (4-char) <code>hostappid</code> to the full ID if possible.</li> <li>Checks for the apptoken in <strong>TPM</strong> (this also detects <em>expired/inactive</em> tokens — they still "exist").</li> <li>Checks for the appaccount in <strong>iLO</strong> by querying the REST API (requires credentials for a complete check).</li> </ol> <p>If found in either location, the account is reported as existing.</p> <blockquote> <p><strong>Note:</strong> Without credentials, only the TPM check is performed. Provide <code>-u</code> / <code>-p</code> for a complete existence check across both stores.</p> </blockquote> <h3>Examples</h3> <p><strong>Check by application ID:</strong></p> <pre><code class="language-shell">ilorest appaccount exists --hostappid a1b2 </code></pre> <p><strong>Check the self-registered iLOrest account:</strong></p> <pre><code class="language-shell">ilorest appaccount exists --self </code></pre> <p><strong>Output when found:</strong></p> <pre><code>Application account exists for this host application. </code></pre> <p><strong>Output when not found:</strong></p> <pre><code>Application account does not exist for this hostapp. </code></pre> <p>The command returns a distinct exit code (<code>ACCOUNT_DOES_NOT_EXIST_ERROR</code>) when the account is not found, which is useful for scripting and automation.</p> <hr> <h2>4. Viewing application account details</h2> <h3>What it does</h3> <p>The <code>appaccount details</code> subcommand provides a <strong>consolidated view</strong> of all application accounts, showing which ones have apptokens in TPM, appaccounts in iLO, or both. This is the most powerful diagnostic tool for understanding the state of appaccounts on a server.</p> <p>Internally, the command:</p> <ol> <li>Calls <code>ListAppIds()</code> to enumerate all apptokens known to TPM.</li> <li>Queries the iLO REST API at <code>/redfish/v1/AccountService/Oem/Hpe/AppAccounts/</code> to find accounts registered in iLO.</li> <li><strong>Merges</strong> the two lists — any appaccount found only in iLO (orphaned after a TPM clear) is included with <code>ExistsInTPM: no</code>.</li> </ol> <blockquote> <p><strong>Note:</strong> Without credentials, only TPM-side data is returned. Providing <code>-u</code> / <code>-p</code> enables the iLO REST query, giving you the complete picture.</p> </blockquote> <h3>Viewing all accounts</h3> <pre><code class="language-shell">ilorest appaccount details --hostappid all -u admin -p iLOpassw0rd </code></pre> <p><strong>Sample output (table format):</strong></p> <pre><code>Application name: SUM Application Id: **c3d4 App account exists in TPM: yes App account exists in iLO: yes Application name: SUT Application Id: **e5f6 App account exists in TPM: yes App account exists in iLO: yes Application name: iLOrest Application Id: **00b5 App account exists in TPM: yes App account exists in iLO: yes </code></pre> <blockquote> <p><strong>Security note:</strong> Application IDs are <strong>masked</strong> — only the last 4 characters are shown, prefixed with <code>**</code>.</p> </blockquote> <h3>JSON output (for scripting and automation)</h3> <pre><code class="language-shell">ilorest appaccount details --hostappid all --json -u admin -p iLOpassw0rd </code></pre> <p><strong>Sample JSON output:</strong></p> <pre><code class="language-json">[ { "ApplicationName": "SUM", "ApplicationID": "**c3d4", "ExistsInTPM": true, "ExistsIniLO": true }, { "ApplicationName": "SUT", "ApplicationID": "**e5f6", "ExistsInTPM": true, "ExistsIniLO": true } ] </code></pre> <h3>Filtering: TPM-only or iLO-only</h3> <p><strong>Show only apptoken (TPM) status:</strong></p> <pre><code class="language-shell">ilorest appaccount details --hostappid all --only_token </code></pre> <p><strong>Show only appaccount (iLO) status:</strong></p> <pre><code class="language-shell">ilorest appaccount details --hostappid all --only_account -u admin -p iLOpassw0rd </code></pre> <h3>Viewing a specific account</h3> <pre><code class="language-shell">ilorest appaccount details --hostappid a1b2 </code></pre> <p>Supports both 4-char short IDs and full IDs.</p> <h3>Viewing the self-registered account</h3> <pre><code class="language-shell">ilorest appaccount details --self </code></pre> <p>If no self-registered account exists:</p> <pre><code>No self-registered iLOrest app account found. Use 'appaccount details --hostappid all' to see all app accounts. </code></pre> <h3>Detecting orphaned accounts</h3> <p>After a TPM clear, you might see:</p> <pre><code>Application Name: SUM Application Id: **c3d4 App account exists in TPM: no App account exists in iLO: yes </code></pre> <p>This tells you the SUM apptoken has been wiped from TPM but the appaccount persists in iLO. To restore it, simply run <code>appaccount create</code> with the same parameters — the orphaned iLO account will be cleaned up automatically and silently.</p> <hr> <h2>5. Reactivating an expired apptoken</h2> <h3>What it does</h3> <p>The <code>appaccount reactivate</code> subcommand is designed <strong>exclusively</strong> for renewing <strong>expired or inactive apptokens</strong> as part of the token <strong>expiry and rotation</strong> lifecycle. Application tokens stored in TPM have a defined lifespan. When a token expires, the application can no longer authenticate with iLO. The <code>reactivate</code> command renews the token in place without requiring a full delete-and-recreate cycle.</p> <h3>What reactivate does not do</h3> <ul> <li>❌ <strong>Does not handle orphaned accounts.</strong> If the apptoken has been removed from TPM entirely (e.g., after a TPM clear), <code>reactivate</code> will fail with an error. Use <code>appaccount create</code> instead — it handles orphan cleanup automatically and silently.</li> <li>❌ <strong>Does not create new accounts.</strong> It only renews existing, expired tokens.</li> <li>❌ <strong>Does not modify the iLO-side appaccount.</strong> Only the TPM-side apptoken is renewed.</li> </ul> <h3>When to use reactivate vs. create</h3> <table> <thead> <tr> <th align="left">Scenario</th> <th align="left">What happened</th> <th align="left">Recommended command</th> </tr> </thead> <tbody> <tr> <td align="left">Token expired (still in TPM)</td> <td align="left">Apptoken's lifespan has elapsed</td> <td align="left"><code>appaccount reactivate</code></td> </tr> <tr> <td align="left">Periodic token rotation</td> <td align="left">Scheduled credential renewal</td> <td align="left"><code>appaccount reactivate</code></td> </tr> <tr> <td align="left">TPM was cleared (orphan in iLO)</td> <td align="left">Apptoken wiped; appaccount remains in iLO</td> <td align="left"><code>appaccount create</code> (silently cleans up the orphan)</td> </tr> <tr> <td align="left">Account does not exist at all</td> <td align="left">Fresh setup</td> <td align="left"><code>appaccount create</code></td> </tr> <tr> <td align="left">Account needs to be removed</td> <td align="left">Decommissioning an application</td> <td align="left"><code>appaccount delete</code></td> </tr> </tbody> </table> <h3>How it works internally</h3> <ol> <li>Checks whether the apptoken exists in TPM. The <code>_check_exists_in_tpm()</code> function returns <code>True</code> for both active and inactive/expired tokens — an expired token still <em>exists</em>, it's just no longer valid for authentication.</li> <li>If the token does <strong>not</strong> exist in TPM at all (it was wiped, not just expired), the command raises an error and directs the user to use <code>create</code> instead.</li> <li>If the token exists but is expired/inactive, it calls <code>reactivate_token()</code> to renew it in place.</li> </ol> <h3>Syntax</h3> <pre><code>ilorest appaccount reactivate --hostappname &#x3C;name> --hostappid &#x3C;id> --salt &#x3C;salt> -u &#x3C;ilo_user> -p &#x3C;ilo_password> </code></pre> <p>Or for self-registered:</p> <pre><code>ilorest appaccount reactivate --self -u &#x3C;ilo_user> -p &#x3C;ilo_password> </code></pre> <p>iLO Administrator credentials are <strong>always</strong> required — no exceptions.</p> <h3>Examples</h3> <p><strong>Reactivate an expired SUM token:</strong></p> <pre><code class="language-shell">ilorest appaccount reactivate --hostappname SUM --hostappid a1b2c3d4 --salt sumsecret -u admin -p iLOpassw0rd </code></pre> <p><strong>Output on success:</strong></p> <pre><code>Application account has been reactivated successfully. </code></pre> <p><strong>If the token doesn't exist in TPM (e.g., after TPM clear):</strong></p> <pre><code>The application account you are trying to reactivate does not exist. If the account was orphaned after a TPM clear, please use 'appaccount delete' followed by 'appaccount create' to recreate it. </code></pre> <blockquote> <p><strong>Tip:</strong> If you see this error after a TPM clear, just run <code>appaccount create</code> with the same parameters. It will silently clean up the orphaned iLO account and create everything fresh.</p> </blockquote> <hr> <h2>Orphaned accounts: what they are and how they're handled</h2> <h3>What is an orphaned account?</h3> <p>An orphaned account is one where the <strong>appaccount exists in iLO</strong> but the <strong>apptoken is missing from TPM</strong>. The account can't be used for authentication because the secret is gone.</p> <h3>How orphans occur</h3> <table> <thead> <tr> <th align="left">Trigger</th> <th align="left">Result</th> </tr> </thead> <tbody> <tr> <td align="left">TPM clear (via BIOS or iLO)</td> <td align="left">All apptokens wiped from TPM; appaccounts remain in iLO</td> </tr> <tr> <td align="left">Failed creation (partial write)</td> <td align="left">Apptoken may exist in TPM but not in iLO, or vice versa</td> </tr> <tr> <td align="left">Deleting <code>--self</code> without credentials</td> <td align="left">Apptoken removed from TPM; appaccount left in iLO (no REST session)</td> </tr> </tbody> </table> <h3>How the tool handles orphans</h3> <p>The <code>appaccount create</code> command is the <strong>primary and recommended recovery mechanism</strong> for orphaned accounts. When you run <code>create</code> for an application that has an orphaned account:</p> <ol> <li>It detects that the appaccount exists in iLO but the apptoken is missing from TPM.</li> <li>It <strong>silently deletes</strong> the orphaned appaccount from iLO.</li> <li>It creates a fresh apptoken in TPM and a fresh appaccount in iLO.</li> <li>The user sees only: <code>"Application account has been generated and saved successfully."</code></li> </ol> <p><strong>No manual cleanup is required.</strong> The entire orphan recovery is invisible to the user.</p> <h3>Summary by subcommand</h3> <table> <thead> <tr> <th align="left">Subcommand</th> <th align="left">Orphan behavior</th> </tr> </thead> <tbody> <tr> <td align="left"><code>create</code></td> <td align="left"><strong>Silently cleans up</strong> the orphaned iLO account, then creates fresh in both stores. User sees a normal success message.</td> </tr> <tr> <td align="left"><code>delete</code></td> <td align="left">Attempts deletion from both TPM and iLO independently. Succeeds if removed from at least one.</td> </tr> <tr> <td align="left"><code>details</code></td> <td align="left">Shows orphans clearly — <code>ExistsInTPM: no</code> alongside <code>ExistsIniLO: yes</code>.</td> </tr> <tr> <td align="left"><code>exists</code></td> <td align="left">Reports the account as existing if found in either store.</td> </tr> <tr> <td align="left"><code>reactivate</code></td> <td align="left"><strong>Does not handle orphans.</strong> Fails if the apptoken is missing from TPM. Only works on expired tokens still in TPM.</td> </tr> </tbody> </table> <hr> <h2>Credential encoding support</h2> <p>For security-sensitive automation pipelines, the <code>appaccount</code> command supports <strong>encoded credentials</strong>. When the <code>--encode</code> flag is active, the provided <code>-u</code> (username) and <code>-p</code> (password) values are treated as encoded strings and decoded internally using the <code>Encryption.decode_credentials()</code> utility before use.</p> <p>This prevents plaintext passwords from appearing in process listings, shell history, or CI/CD logs.</p> <hr> <h2>Session management under the hood</h2> <p>The <code>appaccount</code> command communicates with iLO in two fundamentally different ways:</p> <h3>1. VNIC driver calls (for TPM operations)</h3> <p>Operations like <code>generate_save_token</code>, <code>delete_token</code>, <code>token_exists</code>, <code>reactivate_token</code>, and <code>ListAppIds</code> are performed through the <strong>VNIC driver</strong> — a direct host-to-iLO channel that does not go through the REST API. These calls do not require REST authentication.</p> <blockquote> <p><strong>Note:</strong> In iLO7, the CHIF (Channel Interface) driver used in iLO5/iLO6 is <strong>not available</strong>. All host-to-iLO communication is through VNIC exclusively.</p> </blockquote> <h3>2. Redfish REST API over VNIC (for iLO account operations)</h3> <p>Querying, creating, and deleting appaccounts in iLO is done via the Redfish REST API at <code>https://16.1.15.1</code> (the VNIC IP). <strong>iLO requires authentication for all REST API operations</strong> — both reads and writes.</p> <p>Since <code>appaccount</code> runs without a prior <code>ilorest login</code> session, it creates <strong>temporary REST sessions</strong> on demand:</p> <ul> <li><code>_create_ilo_session()</code> — POSTs to <code>/redfish/v1/SessionService/Sessions/</code> with the provided credentials, obtaining an <code>X-Auth-Token</code>.</li> <li><code>_delete_ilo_session()</code> — DELETEs the session when done.</li> </ul> <p>These temporary sessions are always cleaned up in a <code>finally</code> block, even if an error occurs — preventing session exhaustion on iLO.</p> <blockquote> <p><strong>Key takeaway:</strong> Any operation that touches the iLO REST API (create, delete from iLO, details with iLO data, orphan cleanup) requires credentials. Operations that only touch TPM (token existence check, token deletion, token reactivation) use the VNIC driver and don't need REST credentials — though the CLI still requires them for create and reactivate to ensure the full operation succeeds.</p> </blockquote> <hr> <h2>Quick reference: Common workflows</h2> <h3>Workflow 1: First-time setup of iLOrest self-registration</h3> <pre><code class="language-shell"># Create the self-registered account (credentials required) ilorest appaccount create --self -u admin -p iLOpassw0rd # Verify it was created ilorest appaccount exists --self # View the details ilorest appaccount details --self </code></pre> <h3>Workflow 2: Recovery after TPM clear</h3> <p>No manual cleanup needed — just re-run <code>create</code>. Orphans are handled silently.</p> <pre><code class="language-shell"># Simply recreate the account (orphaned iLO account is cleaned up automatically) ilorest appaccount create --hostappname SUM --hostappid a1b2c3d4 --salt sumsecret -u admin -p iLOpassw0rd # Verify recovery ilorest appaccount details --hostappid a1b2 -u admin -p iLOpassw0rd </code></pre> <h3>Workflow 3: Token expiry — reactivating an expired apptoken</h3> <pre><code class="language-shell"># Reactivate the expired token (token must still exist in TPM, just expired) ilorest appaccount reactivate --hostappname SUM --hostappid a1b2c3d4 --salt sumsecret -u admin -p iLOpassw0rd # Verify the account is active ilorest appaccount exists --hostappid a1b2 </code></pre> <h3>Workflow 4: Audit and cleanup</h3> <pre><code class="language-shell"># List all accounts in JSON (credentials enable iLO-side data) ilorest appaccount details --hostappid all --json -u admin -p iLOpassw0rd # Delete a stale account (credentials required for iLO-side removal) ilorest appaccount delete --hostappid e5f6 -u admin -p iLOpassw0rd # Confirm deletion ilorest appaccount exists --hostappid e5f6 </code></pre> <h3>Workflow 5: Deleting the self-registered account</h3> <pre><code class="language-shell"># Always provide credentials for a complete deletion from both TPM and iLO ilorest appaccount delete --self -u admin -p iLOpassw0rd # Verify ilorest appaccount exists --self </code></pre> <blockquote> <p>⚠️ <strong>Reminder:</strong> Running <code>delete --self</code> without credentials only removes the apptoken from TPM. The appaccount in iLO will remain as an orphan until the next <code>create --self</code> silently cleans it up. Always include credentials for an immediate, complete deletion.</p> </blockquote> <hr> <h2>Error reference</h2> <table> <thead> <tr> <th align="left">Error</th> <th align="left">Cause</th> <th align="left">Resolution</th> </tr> </thead> <tbody> <tr> <td align="left"><code>IncompatibleiLOVersionError</code></td> <td align="left">Running on iLO5 or iLO6</td> <td align="left">Upgrade to iLO7 firmware</td> </tr> <tr> <td align="left"><code>VnicExistsError</code></td> <td align="left">VNIC not enabled or misconfigured</td> <td align="left">Enable VNIC in iLO settings; verify host OS NIC config</td> </tr> <tr> <td align="left"><code>UserNotAdminError</code></td> <td align="left">Not running as root / Administrator</td> <td align="left">Use <code>sudo</code> (Linux) or run as Administrator (Windows)</td> </tr> <tr> <td align="left"><code>UsernamePasswordRequiredError</code></td> <td align="left">Missing <code>-u</code> or <code>-p</code> for a command that requires them</td> <td align="left">Provide iLO admin credentials</td> </tr> <tr> <td align="left"><code>InvalidCommandLineError</code></td> <td align="left">Wrong flag combination (e.g., <code>--self</code> with <code>--hostappid</code>)</td> <td align="left">Check syntax; use <code>-h</code> for help</td> </tr> <tr> <td align="left"><code>NoAppAccountError</code></td> <td align="left">Account / token not found for the given ID</td> <td align="left">Verify <code>hostappid</code>; use <code>create</code> if the token was wiped</td> </tr> <tr> <td align="left"><code>SavinginTPMError</code></td> <td align="left">TPM write failed during creation</td> <td align="left">Delete the account and retry</td> </tr> <tr> <td align="left"><code>SavinginiLOError</code></td> <td align="left">iLO REST API write failed during creation</td> <td align="left">Delete the account and retry</td> </tr> <tr> <td align="left"><code>AppAccountExistsError</code></td> <td align="left">Account already exists (create) or ID not found (details)</td> <td align="left">Use <code>exists</code> to verify; delete first if needed</td> </tr> <tr> <td align="left"><code>ReactivateAppAccountTokenError</code></td> <td align="left">Token reactivation failed</td> <td align="left">Retry; check iLO logs for details</td> </tr> <tr> <td align="left"><code>GenBeforeLoginError</code></td> <td align="left">Cannot determine iLO version</td> <td align="left">Ensure VNIC is enabled on the iLO7 server</td> </tr> </tbody> </table> <hr> <h2>Security best practices</h2> <ol> <li><strong>Always provide iLO credentials</strong> for create, delete, and reactivate operations to ensure both TPM and iLO stay in sync.</li> <li><strong>Never embed plaintext credentials in scripts.</strong> Use the <code>--encode</code> flag or integrate with a secrets manager.</li> <li><strong>Audit appaccounts regularly.</strong> Use <code>appaccount details --hostappid all --json -u admin -p &#x3C;password></code> and feed the output into your SIEM or compliance tools.</li> <li><strong>Clean up unused accounts.</strong> If an application (e.g., SUM) is no longer deployed on a server, delete its appaccount with credentials.</li> <li><strong>After TPM clear, just re-run <code>create</code>.</strong> The command silently handles orphaned iLO accounts — no manual cleanup needed.</li> <li><strong>Monitor token expiry.</strong> When an application suddenly fails to authenticate with iLO, check if its apptoken has expired using <code>appaccount details</code> and reactivate if needed.</li> <li><strong>Restrict iLO admin credentials.</strong> Only users who manage appaccounts should have iLO admin access.</li> </ol> <hr> <h2>In summary</h2> <p>The <code>appaccount</code> command in iLOrest is a powerful, security-first tool for managing application-level authentication on iLO7+ servers. By leveraging dual storage — <strong>apptokens in TPM</strong> for hardware-backed secret security and <strong>appaccounts in iLO</strong> for REST API authorization — it provides a robust foundation for automated server management.</p> <p>Key strengths of the design include:</p> <ul> <li><strong>Transparent orphan recovery</strong> — the <code>create</code> command silently cleans up stale iLO accounts after a TPM clear, requiring no manual intervention.</li> <li><strong>Token lifecycle management</strong> — the <code>reactivate</code> command handles token expiry and rotation without disrupting the account.</li> <li><strong>Comprehensive diagnostics</strong> — the <code>details</code> command provides a unified view of both stores, making it easy to spot inconsistencies.</li> </ul> <p>Currently limited to <strong>SUM, SUT, AMS, and iLOrest</strong> (with custom application support not yet available), it covers the core HPE management stack comprehensively. Whether you're automating firmware updates with SUM, collecting host data with AMS, or scripting iLO configuration with iLOrest, the <code>appaccount</code> command ensures your application credentials are secure, auditable, and recoverable.</p> <hr> <h2>Get started today</h2> <h3>Ready to secure your iLO7 application authentication? Here's how to take the first step:</h3> <ol> <li><strong>Verify your environment</strong> — Confirm that your server is running iLO7 firmware and that VNIC is enabled.</li> <li><strong>Create your first appaccount</strong> — Run <code>ilorest appaccount create --self -u &#x3C;admin> -p &#x3C;password></code> to register iLOrest itself.</li> <li><strong>Audit your existing accounts</strong> — Run <code>ilorest appaccount details --hostappid all --json -u &#x3C;admin> -p &#x3C;password></code> to see what's already registered.</li> <li><strong>Integrate into your automation</strong> — Add appaccount create/delete steps to your provisioning scripts and CI/CD pipelines so every server is configured consistently.</li> <li><strong>Bookmark this guide</strong> — Refer back to the <a href="#quick-reference-common-workflows">Workflows</a> and <a href="#error-reference">Error reference</a> sections when troubleshooting.</li> </ol> <p>Interested in learning more....stay tuned with the <a href="https://developer.hpe.com/blog/">HPE Developer Community</a>....</p>https://developer.hpe.com/announcing-chapel-2-8/https://developer.hpe.com/announcing-chapel-2-8/Thu, 12 Mar 2026 20:40:09 GMT<p>external blog</p>https://developer.hpe.com/llm-observability-and-cost-management-on-hpe-private-cloud-ai/https://developer.hpe.com/llm-observability-and-cost-management-on-hpe-private-cloud-ai/Fri, 06 Mar 2026 06:06:26 GMT<p>LLM (Large language Model) observability and cost management are critical for deploying reliable, secure, and financially sustainable AI applications. By tracking metrics like token usage, latency, and output quality, teams can prevent runaway costs, reduce hallucinations, and ensure regulatory compliance.</p> <p><a href="https://www.litellm.ai/">LiteLLM</a> and <a href="https://langfuse.com/">Langfuse</a> together provide a powerful, open-source stack for LLM observability and cost management (tokenomics), allowing developers to unify, trace, and monitor API usage across hundreds of models. LiteLLM acts as the proxy/SDK to normalize requests and track usage, while Langfuse records these interactions for detailed analysis of token usage, latency, and costs.</p> <p>This blog post walks you through deployment and configuration of LiteLLM and Langfuse on HPE Private Cloud AI. By leveraging these technologies, organizations can perform token-level cost tracking, granular tracing, output streaming and cost analysis of LLMs and AI applications deployed on HPE Private Cloud AI.</p> <h2>HPE Private Cloud AI</h2> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (HPE PCAI)</a> offers a comprehensive, turnkey AI solution designed to address key enterprise challenges, from selecting the appropriate LLMs to efficiently hosting and deploying them. Beyond these core functions, HPE Private Cloud AI empowers organizations to take full control of their AI adoption journey by offering a curated set of pre-integrated <em>NVIDIA Inference Microservices (NIM)</em> LLMs, along with a powerful suite of AI tools and frameworks for data engineering, analytics, and data science.</p> <p>HPE Machine Learning Inference Software (MLIS) is an enterprise-grade solution designed to simplify the deployment, management, and monitoring of machine learning (ML) models at scale. It specifically targets the complexities of moving models from development into production, with a particular focus on large language models.</p> <p>HPE AI Essentials (AIE) Software is the integrated software layer that provides the tools for building, deploying, and managing generative AI applications, including HPE MLIS. It provides a flexible <strong>Import Framework</strong> that enables organizations to deploy their own applications or third-party solutions, like LiteLLM and Langfuse.</p> <p><img src="/img/screenshot-2026-03-06-121708.png" alt=""></p> <h2>Deploy Langfuse and LiteLLM via Import Framework</h2> <h3>1. Prepare the Helm charts for Langfuse</h3> <p>Obtain the Helm chart for Langfuse from the <a href="https://github.com/langfuse/langfuse-k8s">GitHub repository</a> and implement the prerequisites. Here's the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie18hen_us&#x26;page=ManageClusters/importing-applications.html">reference document</a> for the import framework prerequisites.</p> <p>These updates are implemented in the revised Langfuse Helm charts, and are available in the <a href="https://github.com/ai-solution-eng/frameworks/tree/main/langfuse">GitHub repository</a>. With these customizations, <em>Langfuse</em> can now be deployed on HPE Private Cloud AI using <strong>Import Framework</strong>.</p> <h3>2. Deploy and configure Langfuse</h3> <p>Use import framework in HPE Private Cloud AI to deploy Langfuse.</p> <p><img src="/img/screenshot-2026-03-06-152537.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-152941.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-153003.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-153018.png" alt=""></p> <p>After few minutes, Langfuse gets deployed in HPE Private Cloud AI and will be in <strong>Ready</strong> state.</p> <p><img src="/img/screenshot-2026-03-06-153639.png" alt=""></p> <h3>3. Configure Langfuse and create API Keys</h3> <p>Access the Langfuse application deployed on HPE Private Cloud AI by creating a new sign-in account. Set up your <strong>organization</strong> and <strong>project</strong> in Langfuse and create a new API key for this project. <em>Project Settings->Project API Keys-> create new API keys</em></p> <p><img src="/img/screenshot-2026-03-06-153759.png" alt=""></p> <p>Secure the generated API keys, these will be used while deploying LiteLLM.</p> <h3>4. Prepare the Helm charts for LiteLLM</h3> <p>Obtain the Helm chart for LiteLLM from <a href="https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm">litellm-helm</a> repository and implement the prerequisites. Here's the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie18hen_us&#x26;page=ManageClusters/importing-applications.html">reference document</a> for the import framework prerequisites.</p> <p>These updates are implemented in the revised LiteLLM Helm charts, and are available in the <a href="https://github.com/ai-solution-eng/frameworks/tree/main/litellm-helm">GitHub repository</a>.</p> <h3>5. Deploy and configure LiteLLM</h3> <p>Use the <strong>Import Framework</strong> in HPE Private Cloud AI to deploy LiteLLM.</p> <p><img src="/img/screenshot-2026-03-06-160236.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-160303.png" alt=""></p> <p>Set the default username/password (UI_USERNAME/UI_PASSWORD) for LiteLLM application and Langfuse details (LANGFUSE_HOST, LANGFUSE_PUBIC_KEY, LANGFUSE_SECRET_KEY - Obtained in Step#3) in <em>values.yaml</em> as shown below.</p> <p><img src="/img/screenshot-2026-03-06-160517.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-160546.png" alt=""></p> <p>After few minutes, LiteLLM will be deployed in HPE Private Cloud AI and will be in <strong>Ready</strong> state.</p> <p><img src="/img/screenshot-2026-03-06-161531.png" alt=""></p> <h2>6. Deploy LLM in HPE MLIS</h2> <p>HPE MLIS is accessed by clicking on <strong>HPE MLIS</strong> tile in <strong>Tools &#x26; Frameworks</strong> tab.</p> <p><img src="/img/screenshot-2026-03-06-162245.png" alt=""></p> <p>To deploy a pre-packaged LLM (llama-3.1-8b-instruct) in HPE MLIS, you need to create a new deployment as shown below.</p> <p><img src="/img/screenshot-2026-03-06-162547.png" alt=""></p> <p>Click on <strong>Create Deployment</strong>, give a name to the new deployment, choose the appropriate packaged model, and set the scaling factor.</p> <p><img src="/img/screenshot-2026-03-06-162617.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-162630.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-162649.png" alt=""></p> <p><img src="/img/screenshot-2026-03-06-162710.png" alt=""></p> <p>After few minutes, the deployment status will be <strong>Ready</strong>.</p> <p><img src="/img/screenshot-2026-03-06-162732.png" alt=""></p> <h3>7. LLM endpoint and API keys</h3> <p>LLM endpoint details can be obtained via <strong>GenAI</strong>-><strong>Model Endpoints</strong>.</p> <p><img src="/img/screenshot-2026-03-06-163934.png" alt=""></p> <p>Generate an API token for the LLM via, <strong>Actions</strong> -> <strong>Generate API Token</strong>.</p> <p><img src="/img/screenshot-2026-03-06-164059.png" alt=""></p> <p><img src="" alt=""></p> <p>After generating and securing the API token, you will configure LiteLLM with the LLM details.</p> <h3>8. Configure LiteLLM</h3> <p>Launch the LiteLLM application deployed on HPE Private Cloud AI and sign-in using the credentials set in <em>values.yaml</em></p> <p><img src="/img/screenshot-2026-03-06-165129.png" alt=""></p> <p>Add the LLM information, Models + EndPoints -> Add Model and provide the LLM details like Provider, LLM Model Name, API Base and OpenAI API Key.</p> <p><img src="/img/screenshot-2026-03-06-165511.png" alt=""></p> <p>You can associate the cost to this model by updating <strong>Input Cost (per 1M tokens)</strong> and <strong>Output Cost (per 1M tokens)</strong> inside <strong>Model Settings</strong>.</p> <p><img src="/img/screenshot-2026-03-04-114358.png" alt=""></p> <p>Now, create a new virtual key in LiteLLM to access the model,</p> <p><img src="/img/screenshot-2026-03-06-165904.png" alt=""></p> <p>Using the LiteLLM virtual key and the LiteLLM URL, you can access the LLM (meta/llama-3.1) and use it in any AI application.</p> <p><em>Note: In this example, security is not handled for LiteLLM endpoint URL and anyone can hit a request to proxy URL. You can protect the UI by putting it behind outh2-proxy.</em></p> <p>Sample code snippet to call meta/llama via LiteLLM. (Replace your LiteLLM API key in the code)</p> <pre><code class="language-python">import requests import json import os LITELLM_PROXY_API_KEY = "sk-***********" url = 'https://litellm.ai-application.pcai0109.dc15.hpecolo.net/chat/completions' headers = { 'Content-Type': 'application/json', 'Authorization': f'Bearer {LITELLM_PROXY_API_KEY}' } data = { "model": "openai/meta/llama-3.1-8b-instruct", "messages": [ { "role": "user", "content": "Describe Angkor Wat in 300 words" } ] } response = requests.post(url, headers=headers, json=data, verify=False) print(json.dumps(response.json(), indent=2)) </code></pre> <h3>9. LLM observability and cost analysis in Langfuse</h3> <p>Access the Langfuse application deployed on HPE Private Cloud AI and log in using the credentials. The traces of the LLM calls will appear under <strong>Observability</strong> -> <strong>Tracing</strong>.</p> <p><img src="/img/screenshot-2026-03-09-113852.png" alt=""></p> <p>The home page of the project shows various metrics from LLM traces, which provides details on LLM usage, associated costs, etc.</p> <p><img src="/img/screenshot-2026-03-09-114413.png" alt=""></p> <p>Dashboards with custom widgets can be created in Langfuse to observe various parameters of LLM traces. A sample custom dashboard created in Langfuse is shown below.</p> <p><img src="/img/screenshot-2026-03-09-114052.png" alt=""></p> <h2>Conclusion</h2> <p>By combining capabilities of LiteLLM and Langfuse with HPE AIE's robust model management, HPE Private Cloud AI empowers organizations to observe perform cost management of LLMs in their AI solutions. This integrated approach ensures data privacy, operational control, and scalability for deployments.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more guides and best practices on leveraging HPE Private Cloud AI for your AI.</p>https://developer.hpe.com/2026-march-04/https://developer.hpe.com/2026-march-04/Wed, 04 Mar 2026 13:00:00 GMThttps://developer.hpe.com/sivabala-building-opsramps-mcp-server-a-step-by-step-implementation-guide/https://developer.hpe.com/sivabala-building-opsramps-mcp-server-a-step-by-step-implementation-guide/Wed, 25 Feb 2026 22:44:12 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>Introduction</h2> <p>In my <a href="https://developer.hpe.com/blog/sivabala-model-context-protocol-mcp-the-universal-connector-for-ai-applications/">previous article</a>, I explored the Model Context Protocol (MCP) as the universal connector for AI applications. Now, let's roll up our sleeves and dive into the actual implementation of OpsRamp's MCP server, transforming monitoring data into AI-accessible intelligence.</p> <p>This isn't just another code walkthrough – it's a practical guide that takes you from project setup to a fully functional MCP server that exposes OpsRamp's monitoring capabilities to AI applications like Claude Desktop.</p> <h2>The OpsRamp challenge: Bridging monitoring and AI</h2> <p>OpsRamp's monitoring platform generates thousands of alerts daily, tracks hundreds of devices, and processes massive volumes of operational telemetry. Yet despite this wealth of data, operations teams working with it have found themselves manually correlating information, parsing through dashboards, and spending valuable time translating monitoring data into actionable insights.</p> <p>The challenge wasn't lack of data—it was the cognitive overhead of making sense of that data quickly and accurately. What if operations teams could simply ask questions like:</p> <ul> <li>"Show me all critical alerts from the last hour"</li> <li>"What's the current status of our AWS EC2 instances?"</li> <li>"Which devices need attention today?"</li> </ul> <p>To make this vision a reality, we needed to build an MCP server that bridges OpsRamp's monitoring platform with AI applications.</p> <h2>Setting up the project with UV</h2> <p>Before diving into code walkthrough, let's set up a modern Python project using UV – a fast, reliable Python package installer and resolver that's become the go-to choice for modern Python development.</p> <h3>Why UV?</h3> <p>UV offers several advantages over traditional Python package management:</p> <ul> <li><strong>Lightning fast</strong>: Up to 10-100x faster than pip</li> <li><strong>Reliable</strong>: Deterministic dependency resolution</li> <li><strong>Modern</strong>: Built-in virtual environment management</li> <li><strong>Simple</strong>: Straightforward commands and workflows</li> </ul> <h3>Installing UV</h3> <p>To install UV on our system, start with this:</p> <pre><code class="language-bash"># On macOS and Linux curl -LsSf https://astral.sh/uv/install.sh | sh # On Windows powershell -c "irm https://astral.sh/uv/install.ps1 | iex" </code></pre> <h3>Creating the project structure</h3> <p>Once that is complete, you can begin creating your OpsRamp MCP server project:</p> <pre><code class="language-bash"># Create a new project directory mkdir opsramp-mcp-server cd opsramp-mcp-server # Initialize a new UV project uv init # Create the virtual environment uv venv # Activate the virtual environment # On macOS/Linux: source .venv/bin/activate # On Windows: .venv\Scripts\activate </code></pre> <h3>Installing dependencies</h3> <p>Create a <code>pyproject.toml</code> file with our project dependencies:</p> <pre><code class="language-toml">[project] name = "opsramp-mcp-server" version = "0.1.0" description = "MCP server for OpsRamp monitoring platform" readme = "README.md" requires-python = ">=3.10" dependencies = [ "mcp>=0.9.0", "aiohttp>=3.9.0", "pydantic>=2.0.0", ] [project.scripts] opsramp-mcp-server = "opsramp_mcp_server:run_main" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" </code></pre> <p>Install the dependencies:</p> <pre><code class="language-bash">uv pip install -e . </code></pre> <h3>Project structure</h3> <p>Your project structure should look like this:</p> <pre><code class="language-text">opsramp-mcp-server/ ├── .venv/ # Virtual environment (created by UV) ├── opsramp_mcp_server.py # Main server implementation ├── pyproject.toml # Project configuration ├── README.md # Documentation └── .env # Environment variables (don't commit!) </code></pre> <h2>Architecture overview: The three-layer approach</h2> <p>The MCP server being set up via the instructions in this guide follows a clean three-layer architecture:</p> <ol> <li><strong>Authentication layer</strong>: Handles OAuth 2.0 token management and API security</li> <li><strong>API Communication layer</strong>: Manages HTTP requests and responses with OpsRamp</li> <li><strong>MCP Interface layer</strong>: Exposes tools and resources through the MCP protocol</li> </ol> <p>This separation ensures maintainability, testability, and clear boundaries between concerns.</p> <h2>Implementation deep dive: Building the server</h2> <p>Now let's walk through the implementation step by step, understanding each component and design decision.</p> <h3>Step 1: Understanding the MCP Python SDK</h3> <p>Before diving into the imports, it's important to understand the MCP Python SDK.</p> <p>Anthropic provides an official Python SDK that simplifies building MCP servers by handling the protocol details, message serialization, and transport layer complexity. The SDK provides:</p> <ul> <li><strong>Server framework</strong>: Core classes for building MCP servers</li> <li><strong>Type definitions</strong>: Strongly-typed interfaces for tools, resources, and prompts</li> <li><strong>Transport layers</strong>: Built-in support for stdio (standard input/output) communication</li> <li><strong>Protocol handling</strong>: Automatic serialization and deserialization of MCP messages</li> </ul> <p>This allows developers to focus on business logic rather than protocol implementation details.</p> <h3>Step 2: Imports and logging configuration</h3> <pre><code class="language-python">#!/usr/bin/env python3 """ OpsRamp MCP Server Provides access to OpsRamp OpsQL API with OAuth 2.0 client credentials authentication """ import asyncio import json import logging import os import sys from typing import Any, Dict, Optional from datetime import datetime, timedelta import aiohttp import mcp.server.stdio import mcp.types as types from mcp.server import Server </code></pre> <p><strong>Understanding the MCP imports:</strong></p> <p><strong><code>import mcp.server.stdio</code></strong></p> <p>This module provides the standard input/output transport layer for MCP servers. It enables communication between your server and AI applications through stdin/stdout streams, which is the standard way MCP servers communicate with host applications like Claude Desktop. This transport mechanism allows your server to run as a subprocess that AI applications can interact with.</p> <p><strong><code>import mcp.types as types</code></strong></p> <p>This module contains all the type definitions for MCP protocol objects including <code>Tool</code>, <code>Resource</code>, <code>Prompt</code>, and <code>TextContent</code>. These types ensure type safety and provide clear interfaces for defining what your server exposes to AI applications. For example, <code>types.Tool</code> is used to define each tool with its name, description, and input schema, while <code>types.TextContent</code> represents the text responses your tools return.</p> <p><strong><code>from mcp.server import Server</code></strong></p> <p>The <code>Server</code> class is the core of your MCP server implementation. It provides decorators like <code>@server.list_tools()</code> and <code>@server.call_tool()</code> that you use to register and implement your server's capabilities. The Server instance handles all the protocol-level details like capability negotiation, message routing, and error handling, allowing you to focus on implementing your specific tools and resources.</p> <p><strong>Key decisions here:</strong></p> <ul> <li><strong>Async from the ground up</strong>: We use <code>asyncio</code> and <code>aiohttp</code> for non-blocking I/O, essential for handling multiple concurrent AI requests efficiently</li> <li><strong>Comprehensive logging</strong>: Detailed logging helps troubleshoot issues in production environments</li> <li><strong>Type hints</strong>: Using Python's type system makes code more maintainable and catches errors early</li> </ul> <p>The logging configuration writes to both a file and console, crucial for debugging MCP servers since they run as background processes:</p> <pre><code class="language-python">logging.basicConfig( level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(filename)s:%(lineno)d - %(funcName)s - %(message)s', handlers=[ logging.FileHandler('opsramp_opsql_mcp.log'), logging.StreamHandler(sys.stdout) ] ) </code></pre> <h3>Step 3: The OpsRampClient class—authentication layer</h3> <p>The <code>OpsRampClient</code> class handles all interactions with OpsRamp's API, starting with OAuth 2.0 authentication:</p> <pre><code class="language-python">class OpsRampClient: """Client for OpsRamp API with OAuth 2.0 authentication""" def __init__(self, base_url: str, client_id: str, client_secret: str, tenant_id: str): self.base_url = base_url.rstrip('/') self.client_id = client_id self.client_secret = client_secret self.tenant_id = tenant_id self.access_token: Optional[str] = None self.token_expires_at: Optional[datetime] = None self.session = aiohttp.ClientSession() </code></pre> <p><strong>Design considerations:</strong></p> <ul> <li><strong>Token caching</strong>: The system stores the access token and its expiration time to avoid unnecessary authentication requests</li> <li><strong>Session reuse</strong>: A single <code>aiohttp.ClientSession</code> is maintained for connection pooling and better performance</li> <li><strong>Tenant isolation</strong>: The tenant ID ensures proper data isolation in multi-tenant environments</li> </ul> <h3>Step 4: OAuth 2.0 token management</h3> <p>The <code>_get_access_token</code> method implements intelligent token management:</p> <pre><code class="language-python">async def _get_access_token(self) -> str: """Get or refresh the OAuth 2.0 access token""" # Check if we have a valid token if (self.access_token and self.token_expires_at and datetime.now() &#x3C; self.token_expires_at - timedelta(minutes=5)): logger.debug("Using existing access token") return self.access_token # Request new token token_url = f"{self.base_url}/auth/oauth/token" data = { 'grant_type': 'client_credentials', 'client_id': self.client_id, 'client_secret': self.client_secret } headers = { 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': 'application/json' } async with self.session.post(token_url, data=data, headers=headers) as response: if response.status == 200: token_data = await response.json() self.access_token = token_data['access_token'] expires_in = token_data.get('expires_in', 3600) self.token_expires_at = datetime.now() + timedelta(seconds=expires_in) logger.info("Successfully obtained access token") return self.access_token else: error_text = await response.text() raise Exception(f"Failed to get access token: {response.status} - {error_text}") </code></pre> <p><strong>Smart token management:</strong></p> <ul> <li><strong>5-minute buffer</strong>: We refresh tokens 5 minutes before expiration to prevent race conditions</li> <li><strong>Automatic refresh</strong>: Expired tokens are automatically refreshed transparently</li> <li><strong>Error handling</strong>: Failed authentication attempts are logged with detailed error messages</li> </ul> <h3>Step 5: Authenticated API requests</h3> <p>The <code>_make_request</code> method wraps all API calls with authentication:</p> <pre><code class="language-python">async def _make_request(self, method: str, endpoint: str, **kwargs) -> Dict[str, Any]: """Make an authenticated request to the OpsRamp API""" access_token = await self._get_access_token() headers = kwargs.get('headers', {}) headers.update({ 'Authorization': f'Bearer {access_token}', 'Accept': 'application/json' }) kwargs['headers'] = headers url = f"{self.base_url}{endpoint}" logger.debug(f"Making {method} request to: {url}") async with self.session.request(method, url, **kwargs) as response: response_text = await response.text() if response.status >= 200 and response.status &#x3C; 300: try: return await response.json() except json.JSONDecodeError: return {"response": response_text} else: raise Exception(f"API request failed: {response.status} - {response_text}") </code></pre> <p><strong>Robust request handling:</strong></p> <ul> <li><strong>Automatic authentication</strong>: Every request includes a valid Bearer token</li> <li><strong>Flexible parameters</strong>: The <code>**kwargs</code> pattern allows passing any HTTP parameters</li> <li><strong>JSON parsing with fallback</strong>: Attempts to parse JSON, falls back to raw text if needed</li> <li><strong>Comprehensive error reporting</strong>: Failed requests include status codes and response bodies</li> </ul> <h3>Step 6: OpsQL query execution</h3> <p>The <code>execute_opsql_query</code> method provides access to OpsRamp's query language:</p> <pre><code class="language-python">async def execute_opsql_query(self, query: str) -> Dict[str, Any]: """Execute an OpsQL query""" logger.info(f"Preparing to execute OpsQL query: {query[:100]}") endpoint = f"/v3/api/opsql/{self.tenant_id}/queries" headers = { 'Content-Type': 'application/json' } result = await self._make_request('POST', endpoint, json=json.loads(query), headers=headers) logger.info("OpsQL query executed successfully") return result </code></pre> <p><strong>OpsQL integration:</strong></p> <ul> <li><strong>Tenant-scoped queries</strong>: Queries are automatically scoped to the authenticated tenant</li> <li><strong>JSON payload</strong>: Queries are sent as structured JSON for parsing and validation</li> <li><strong>Result streaming</strong>: Large result sets are handled efficiently through async I/O</li> </ul> <h3>Step 7: MCP server initialization</h3> <p>Now create the MCP server instance:</p> <pre><code class="language-python">server = Server("opsramp-mcp-server") </code></pre> <p>This single line creates an MCP server with the name "opsramp-mcp-server" that will be visible to AI applications.</p> <h3>Step 8: Tool registration</h3> <p>The <code>@server.list_tools()</code> decorator registers available tools with the MCP protocol:</p> <pre><code class="language-python">@server.list_tools() async def handle_list_tools() -> list[types.Tool]: """List available tools""" logger.info("Listing available tools") tools = [ types.Tool( name="execute_opsql_query", description="Execute an OpsQL query against the OpsRamp platform", inputSchema={ "type": "object", "properties": { "query": { "type": "string", "description": "The OpsQL query to execute" } } } ), types.Tool( name="get_alerts", description="Get alerts from OpsRamp", inputSchema={ "type": "object", "properties": { "query_filter": { "type": "string", "description": "Query filter for alerts" }, "limit": { "type": "integer", "description": "Maximum number of alerts to return", "default": 100 } } } ), types.Tool( name="get_minimal_resource_details", description="Get Minimal Resource Details from OpsRamp", inputSchema={ "type": "object", "properties": { "query_filter": { "type": "string", "description": "Query filter for alerts" } } } ), types.Tool( name="get_release_version", description="Get release version from OpsRamp", inputSchema={ "type": "object" } ), types.Tool( name="get_alert_statistics_dashboard", description="Get Alert Statistics dashboard from OpsRamp", inputSchema={ "type": "object" } ) ] return tools </code></pre> <p><strong>Tool design principles:</strong></p> <ul> <li><strong>Clear descriptions</strong>: Each tool has a human-readable description that AI models can understand</li> <li><strong>JSON Schema validation</strong>: Input schemas ensure AI applications provide properly formatted data</li> <li><strong>Intuitive naming</strong>: Tool names follow verb-noun patterns (get_alerts, execute_query)</li> <li><strong>Progressive disclosure</strong>: Simple tools (get_release_version) require no parameters, while complex ones (execute_opsql_query) have detailed schemas</li> </ul> <h3>Step 9: Tool execution handler</h3> <p>The <code>@server.call_tool()</code> decorator implements the actual tool logic. This is where the magic happens – where AI requests transform into actual OpsRamp API calls:</p> <pre><code class="language-python">@server.call_tool() async def handle_call_tool( name: str, arguments: dict[str, Any] | None ) -> list[types.TextContent]: """Handle tool calls""" global opsramp_client if not opsramp_client: return [types.TextContent( type="text", text="Error: OpsRamp client not initialized. Please check your configuration." )] try: if name == "execute_opsql_query": query = arguments.get("query", "") if not query.strip(): return [types.TextContent( type="text", text="Error: Query cannot be empty" )] result = await opsramp_client.execute_opsql_query(query) return [types.TextContent( type="text", text=f"Query executed successfully:\n\n```json\n{json.dumps(result, indent=2)}\n```" )] elif name == "get_alerts": endpoint = f"/api/v2/tenants/{opsramp_client.tenant_id}/alerts/search" query_filter = arguments.get("query_filter", "") limit = arguments.get("limit", 100) result = await opsramp_client._make_request( method="GET", endpoint=endpoint, params={"query": query_filter, "limit": limit} ) return [types.TextContent( type="text", text=f"Query executed successfully:\n\n```json\n{json.dumps(result, indent=2)}\n```" )] elif name == "get_minimal_resource_details": endpoint = f"/api/v2/tenants/{opsramp_client.tenant_id}/resources/minimal" query_filter = arguments.get("query_filter", "") result = await opsramp_client._make_request( method="GET", endpoint=endpoint, params={"query": query_filter} ) return [types.TextContent( type="text", text=f"Query executed successfully:\n\n```json\n{json.dumps(result, indent=2)}\n```" )] elif name == "get_release_version": endpoint = f"/tenancy/api/v3/release-version" result = await opsramp_client._make_request(method="GET", endpoint=endpoint) return [types.TextContent( type="text", text=f"Query executed successfully:\n\n```json\n{json.dumps(result, indent=2)}\n```" )] elif name == "get_alert_statistics_dashboard": dashboard_link = "https://pod7.opsramp.com/portal/dashboards/4dfc7792-d03d-11ec-9e13-0242ac120006" return [types.TextContent( type="text", text=f"Alert Statistics Dashboard: {dashboard_link}" )] else: return [types.TextContent( type="text", text=f"Unknown tool: {name}" )] except Exception as e: logger.error(f"Error in tool call {name}: {e}") return [types.TextContent( type="text", text=f"Error executing {name}: {str(e)}" )] </code></pre> <p><strong>Robust tool execution:</strong></p> <ul> <li><strong>Validation first</strong>: Input parameters are validated before making API calls</li> <li><strong>Consistent error handling</strong>: All errors are caught and returned as user-friendly messages</li> <li><strong>JSON formatting</strong>: Results are formatted in readable JSON with syntax highlighting</li> <li><strong>Comprehensive logging</strong>: Every tool call is logged for debugging and audit trails</li> </ul> <h3>Step 10: Query examples helper</h3> <p>The <code>get_query_examples</code> function provides AI applications with sample queries to learn from:</p> <pre><code class="language-python">def get_query_examples(category: str) -> str: """Get OpsQL query examples by category""" examples = { "alerts": [ '''{ "objectType": "alert", "fields": ["id", "clientId", "component", "currentState"], "filterCriteria": "currentState=critical" }''' ], "devices": [ '''{ "fields": ["ipAddress", "installedAppName", "name", "clientName"], "objectType": "resource", "filterCriteria": "availableAppName = 'AWS' and installedAppName = 'OpsRamp PM'", "pageNo": 1, "pageSize": 50 }''' ], "topology": [ '''{ "objectType": "topology", "filterCriteria": "(intAppId = 'INTG-73e5a7fa-8674-4f33-ae53-26a2b9c049ea')", "fields": ["id", "sourceId", "targetId", "relation"] }''' ] } if category == "all": result = "" for cat, queries in examples.items(): result += f"\n**{cat.upper()} Examples:**\n" for i, query in enumerate(queries, 1): result += f"{i}. {query}\n" return result elif category in examples: result = f"**{category.upper()} Examples:**\n" for i, query in enumerate(examples[category], 1): result += f"{i}. {query}\n" return result return "Category not found. Available categories: alerts, devices, topology, all" </code></pre> <p><strong>Learning by example:</strong></p> <ul> <li><strong>Categorized samples</strong>: Examples are organized by domain (alerts, devices, topology)</li> <li><strong>Real-world queries</strong>: Each example represents actual use cases</li> <li><strong>Progressive complexity</strong>: Examples range from simple to complex</li> <li><strong>AI-friendly format</strong>: Structured to help AI models understand query patterns</li> </ul> <h3>Step 11: Main entry point and server startup</h3> <p>The <code>main</code> function orchestrates server initialization and lifecycle:</p> <pre><code class="language-python">async def main(): """Main entry point""" global opsramp_client # Get configuration from environment variables base_url = os.getenv("OPSRAMP_BASE_URL", "https://develop.opsramp.com") client_id = os.getenv("OPSRAMP_CLIENT_ID") client_secret = os.getenv("OPSRAMP_CLIENT_SECRET") tenant_id = os.getenv("OPSRAMP_TENANT_ID") if not all([client_id, client_secret, tenant_id]): logger.error("Missing required environment variables:") logger.error("- OPSRAMP_CLIENT_ID") logger.error("- OPSRAMP_CLIENT_SECRET") logger.error("- OPSRAMP_TENANT_ID") logger.error("Optional: OPSRAMP_BASE_URL") return # Initialize the OpsRamp client opsramp_client = OpsRampClient(base_url, client_id, client_secret, tenant_id) logger.info(f"Starting OpsRamp MCP Server") logger.info(f"Base URL: {base_url}") logger.info(f"Tenant ID: {tenant_id}") # Run the server async with mcp.server.stdio.stdio_server() as (read_stream, write_stream): await server.run( read_stream, write_stream, server.create_initialization_options() ) def run_main(): """Entry point wrapper""" asyncio.run(main()) if __name__ == "__main__": run_main() </code></pre> <p><strong>Startup sequence:</strong></p> <ol> <li><strong>Environment validation</strong>: Checks for required configuration before starting</li> <li><strong>Client initialization</strong>: Creates the OpsRamp client with credentials</li> <li><strong>STDIO transport</strong>: Uses standard input/output for communication with AI applications</li> <li><strong>Graceful error handling</strong>: Missing configuration results in helpful error messages</li> </ol> <h2>Configuration: Environment variables</h2> <p>Create a <code>.env</code> file for your configuration (and add it to <code>.gitignore</code>):</p> <pre><code class="language-bash">OPSRAMP_BASE_URL=https://develop.opsramp.com OPSRAMP_CLIENT_ID=your_client_id_here OPSRAMP_CLIENT_SECRET=your_client_secret_here OPSRAMP_TENANT_ID=your_tenant_id_here </code></pre> <p><strong>Security best practices:</strong></p> <ul> <li><strong>Never commit credentials</strong>: Add <code>.env</code> to <code>.gitignore</code> immediately</li> <li><strong>Use environment-specific configs</strong>: Different values for dev/staging/prod</li> <li><strong>Rotate secrets regularly</strong>: Change client secrets periodically</li> <li><strong>Minimal permissions</strong>: Use credentials with only necessary API access</li> </ul> <h2>Testing the server locally</h2> <p>Before integrating with Claude Desktop, test your server locally:</p> <pre><code class="language-bash"># Set environment variables export OPSRAMP_BASE_URL=https://develop.opsramp.com export OPSRAMP_CLIENT_ID=your_client_id export OPSRAMP_CLIENT_SECRET=your_client_secret export OPSRAMP_TENANT_ID=your_tenant_id # Run the server python opsramp_mcp_server.py </code></pre> <p>Watch the log file for connection and authentication messages:</p> <pre><code class="language-bash">tail -f opsramp_opsql_mcp.log </code></pre> <p>You should see log entries like:</p> <pre><code class="language-text">2025-08-12 10:30:15 - __main__ - INFO - Starting OpsRamp MCP Server 2025-08-12 10:30:15 - __main__ - INFO - Base URL: https://develop.opsramp.com 2025-08-12 10:30:15 - __main__ - INFO - Tenant ID: client_123 2025-08-12 10:30:16 - __main__ - INFO - Successfully obtained access token </code></pre> <h2>Key architectural decisions explained</h2> <p>Let's revisit some crucial design choices that make this implementation robust:</p> <h3>Asynchronous design</h3> <p>I chose <code>asyncio</code> and <code>aiohttp</code> for several compelling reasons:</p> <ul> <li><strong>Concurrent request handling</strong>: Multiple AI applications can query simultaneously without blocking</li> <li><strong>Efficient I/O</strong>: Network requests don't block the event loop</li> <li><strong>Scalability</strong>: The server can handle many concurrent connections with minimal resource overhead</li> <li><strong>Natural fit for MCP</strong>: The MCP protocol itself is designed for async communication</li> </ul> <h3>Token management strategy</h3> <p>The OAuth 2.0 implementation includes intelligent token caching:</p> <ul> <li><strong>5-minute expiration buffer</strong>: Prevents edge cases where tokens expire mid-request</li> <li><strong>Automatic refresh</strong>: Transparent to calling code</li> <li><strong>Thread-safe</strong>: Single token instance shared across all requests</li> </ul> <h3>Error handling philosophy</h3> <p>Every layer includes comprehensive error handling:</p> <ul> <li><strong>Validation errors</strong>: Caught early with clear messages</li> <li><strong>Network errors</strong>: Logged with full context</li> <li><strong>Authentication errors</strong>: Specific messages for credential issues</li> <li><strong>API errors</strong>: Include status codes and response bodies</li> </ul> <h3>Logging strategy</h3> <p>Multi-level logging provides visibility without overwhelming:</p> <ul> <li><strong>DEBUG level</strong>: Detailed request/response information</li> <li><strong>INFO level</strong>: Important lifecycle events</li> <li><strong>ERROR level</strong>: Failures that need attention</li> <li><strong>File + console</strong>: Useful for both development and production</li> </ul> <h2>What's next?</h2> <p>Congratulations! You've successfully built a production-ready MCP server that exposes OpsRamp's monitoring capabilities to AI applications. The implementation demonstrates key patterns:</p> <ul> <li><strong>Clean separation of concerns</strong> with distinct layers for auth, API, and MCP</li> <li><strong>Robust error handling</strong> at every level</li> <li><strong>Comprehensive logging</strong> for troubleshooting</li> <li><strong>Async-first design</strong> for performance</li> <li><strong>Type safety</strong> through Python type hints</li> </ul> <p>In my next article, <strong>Part 3: Testing and Integration</strong>, I'll explore:</p> <ul> <li>Testing the server with MCP Inspector</li> <li>Integrating with Claude Desktop</li> <li>Real-world usage scenarios and queries</li> <li>Debugging tips and common pitfalls</li> <li>Performance optimization strategies</li> <li>Production deployment considerations</li> </ul> <p>The journey from monitoring data to AI-accessible intelligence continues. Stay tuned for how you can bring this MCP server to life with real AI interactions!</p> <hr> <p><em>Part 3 of this series will demonstrate testing the OpsRamp MCP server with MCP Inspector and integrating it with Claude Desktop for real-world AI-powered operations workflows.</em></p>https://developer.hpe.com/simplifying-permission-management-using-kubernetes-clusterrole-aggregation-in-hpe-private-cloud-ai/https://developer.hpe.com/simplifying-permission-management-using-kubernetes-clusterrole-aggregation-in-hpe-private-cloud-ai/Tue, 17 Feb 2026 07:36:43 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>When operating Kubernetes (K8s), Role‑Based Access Control (RBAC) serves as a foundational security mechanism, mapping users and workloads to define permissions and enforcing the principle of least privilege. However, as clusters evolve and new access requirements arise, managing K8s Roles and ClusterRoles through manual updates becomes increasingly difficult and error-prone.</p> <p>This blog post introduces ClusterRole aggregation as an effective way to simplify that challenge. It explains the key concepts and advantages of aggregated ClusterRoles and shows how they streamline permission management by reducing manual RBAC updates on existing roles. The post also provides practical examples of applying ClusterRole aggregation in the HPE Private Cloud AI (PCAI) environment, demonstrating how this approach makes RBAC administration more scalable, maintainable, and efficient.</p> <h3>What is K8s RBAC?</h3> <p>K8s RBAC is a native authorization framework integrated directly into the K8s API server. It is composed of four primary object types: <em>Roles</em>, which define permissions for namespaced resources; <em>ClusterRoles</em>, which define permissions at the cluster scope; <em>RoleBindings</em>, which associate a role with a user, group, or ServiceAccount within a specific namespace; and <em>ClusterRoleBindings</em>, which bind subjects to a ClusterRole across the entire cluster. Together, these constructs provide fine‑grained, declarative control over resource access and enforce least-privilege authorization across the environment.</p> <p>Despite its flexibility, K8s RBAC introduces several operational challenges, largely due to its mixed namespace and cluster-scoped permission model. Because RBAC permissions must span both namespaced and global resources, teams often struggle to maintain strict least‑privilege boundaries without resorting to overly broad ClusterRoles, raising the risk of misconfiguration and privilege escalation. The high granularity of K8s API resources and verbs, combined with the separation of Roles/ClusterRoles from their bindings, makes it difficult to understand the effective permissions granted to a subject. As multiple teams modify RBAC objects over time, policies tend to drift, accumulate inconsistencies, and unintentionally propagate privileges through shared bindings. In large-scale environments such as PCAI, K8s RBAC management becomes complex, error‑prone, and operationally fragile.</p> <p>To address these challenges as environments grow and the number of roles increases, K8s provides <em>ClusterRole aggregation</em>, a mechanism designed to simplify and streamline permission management across the cluster.</p> <p> </p> <h3>What is ClusterRole aggregation?</h3> <p><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles">ClusterRole aggregation</a>, introduced in K8s v1.9, is a mechanism that automatically aggregates several ClusterRoles into one combined ClusterRole based on label selectors. A controller, running in the cluster control plane, watches for ClusterRole objects that define an <em>aggregationRule</em> section. This rule specifies a set of label selectors that the controller uses to match other ClusterRoles whose rules should be merged into the <em>rules</em> field of the aggregated ClusterRole. The resulting ClusterRole is dynamically constructed by combining the permissions of all matching rules.</p> <p>K8s ships with several built-in <em>user-facing</em> roles, such as <em>view</em>, <em>edit</em> and <em>admin</em>, implemented using this aggregation mechanism. These default roles represent common permission tiers, ranging from ready-only access to full namespac-level administrative capabilities. They are automatically assembled by the controller using labels of the form <em>'rbac.authorization.k8s.io/aggregate-to-'</em>.</p> <p>When additional permissions are required, cluster administrators can define them as standalone ClusterRoles and apply the appropriate label. The controller will automatically incorporate these roles into the corresponding aggregated roles (e.g., <em>edit</em>, <em>view</em>), eliminating the need to manually modify existing ClusterRoles whenever new access requirements arise. This approach shifts the operational focus toward managing small, purpose‑specific roles that are automatically composed into higher‑level permission sets, making RBAC policy management more scalable, maintainable, and efficient.</p> <p>The following sections will show you some practical examples of permission management using ClusterRole aggregation in the HPE PCAI environment.</p> <h3>HPE Private Cloud AI</h3> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (PCAI)</a> is a turnkey, enterprise‑ready platform that integrates HPE and NVIDIA technologies to simplify and accelerate AI workload deployment on a K8s foundation. By leveraging standard K8s constructs, PCAI deploys AI models, inference services, and supporting AI/ML components into dedicated K8s namespaces, ensuring clean resource separation, scalability, and lifecycle management.</p> <p>As part of its user‑centric design, PCAI automatically provisions a default Jupyter notebook environment for each authenticated user. This Jupyter notebook runs as a containerized Pod within the user’s personal K8s namespace, providing an isolated workspace for experimentation, data preparation, and model development.</p> <h4>Kubeflow Notebooks</h4> <p>As part of the pre‑integrated toolset in PCAI, <a href="https://www.kubeflow.org/">Kubeflow</a> is deployed along with its associated custom resource definitions (CRDs) and built‑in ClusterRoles. These ClusterRoles follow K8s standard role patterns, <em>kubeflow-view</em>, <em>kubeflow-edit</em>, and <em>kubeflow-admin</em>, and can be assigned by cluster administrators to users or ServiceAccounts to manage access control within the cluster.</p> <pre><code class="language-shell"> # kubectl get clusterroles | grep -e "kubeflow-edit" -e "kubeflow-view" -e "kubeflow-admin" kubeflow-admin 2025-11-20T03:25:34Z kubeflow-edit 2025-11-20T03:25:34Z kubeflow-view 2025-11-20T03:25:34Z </code></pre> <p>When a user logs into PCAI, a default Jupyter notebook named <em>'default-notebook'</em> is already available under <strong>Notebook Servers</strong>. This Jupyter notebook is pre-created using the tensorflow image through Kubeflow Notebooks and is deployed within the user's dedicated project namespace, such as <em>'project-user-guoping-jia'</em>.</p> <p><img src="/img/kubeflow-notebooks.png" alt=""></p> <p>Within this namespace, a RoleBinding named <em>'default-editor'</em> associates the ServiceAccount <em>'default-editor'</em> with the Kubeflow ClusterRole <em>'kubeflow-edit'</em>. This ClusterRole grants the standard set of permissions required for typical Kubeflow operations.</p> <pre><code class="language-shell"># kubectl get serviceaccount -n project-user-guoping-jia default-editor NAME                      SECRETS   AGE default-editor            0         24d # kubectl get rolebinding -n project-user-guoping-jia default-editor NAME                       ROLE                                   AGE default-editor             ClusterRole/kubeflow-edit              24d </code></pre> <p>With this default RoleBinding in place, the user obtains the <em>'kubeflow-edit'</em> permissions within the namespace and can run <em>kubectl</em> commands from the Jupyter notebook terminal to access most K8s objects, including Pods, Deployments, Services, and more.</p> <p><img src="/img/notebook-server-terminal.png" alt=""></p> <h4>Permission restriction in Jupyter notebooks</h4> <p>While the user can access most K8s objects in the project namespace, certain operations remain restricted. For example, they cannot list all Secrets or perform privileged actions such as executing commands inside a running Pod’s container. These elevated permissions are sometimes necessary, for instance, when verifying private container image configurations that rely on specific Secrets or when troubleshooting issues in a failed Pod.</p> <pre><code class="language-shell"> (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i list secrets no (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i get pods --subresource=exec no (base) guoping-jia@default-notebook-0:~$ kubectl get secrets Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot list resource "secrets" in API group "" in the namespace "project-user-guoping-jia" (base) guoping-jia@default-notebook-0:~$ kubectl exec -it default-notebook-0 -- sh Error from server (Forbidden): pods "default-notebook-0" is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot create resource "pods/exec" in API group "" in the namespace "project-user-guoping-jia" </code></pre> <h3>Permission management for Jupyter notebook terminal access</h3> <p>This section explains how to grant additional permissions in a Jupyter notebook environment using ClusterRole aggregation to simplify RBAC permission management.</p> <p>Considering the fact that the ClusterRole <em>'kubeflow-edit'</em> is bound to the ServiceAccount of every authenticated PCAI user, modifying this shared ClusterRole to add additional permissions is not an appropriate approach. Doing so would grant all users elevated privileges in their Jupyter notebooks, which violates the principle of least privilege.</p> <h4>Granting additional permissions</h4> <p>Rather than modifying the existing Kubeflow ClusterRoles, this section describes how to create a custom aggregated ClusterRole that can be easily extended by combining multiple smaller, purpose-specific ClusterRoles.</p> <p>All ClusterRole and RoleBinding YAML manifests referenced here are available in <a href="https://github.com/GuopingJia/aggregate-clusterroles">my GitHub repository</a>. If you plan to configure this for your Jupyter notebook environment, remember to replace the project's user namespace with your own.</p> <ul> <li>Create an aggregated ClusterRole</li> </ul> <p>Create the following aggregated ClusterRole named <em>'custom-kubeflow-edit'</em>.</p> <pre><code class="language-shell"> # cat custom-kubeflow-edit.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:   name: custom-kubeflow-edit aggregationRule:   clusterRoleSelectors:   - matchLabels:       rbac.authorization.kubeflow.org/aggregate-to-custom-kubeflow-edit: "true" rules: [] # kubectl apply -f custom-kubeflow-edit.yaml clusterrole.rbac.authorization.k8s.io/custom-kubeflow-edit created # kubectl get clusterroles custom-kubeflow-edit NAME CREATED AT custom-kubeflow-edit 2026-02-17T10:43:00Z </code></pre> <p>Notice that the <em>'rules'</em> field in this aggregated ClusterRole is intentionally left empty, with only the label <em>'rbac.authorization.kubeflow.org/aggregate-to-custom-kubeflow-edit: "true"'</em> defined. The control plane automatically fills the <em>'rules'</em> field by merging permissions from other ClusterRoles that match this label. Any rules you manually add to an aggregated ClusterRole will be overwritten. To modify or extend permissions, you must update or create the individual ClusterRole objects selected by the aggregation label.</p> <p>Run the following command to inspect the <em>'rules'</em> section of the deployed ClusterRole <em>'custom-kubeflow-edit'</em>.</p> <pre><code class="language-shell"># kubectl get clusterrole custom-kubeflow-edit -o jsonpath='{.rules}' | jq . null </code></pre> <ul> <li>Create a focused ClusterRole <em>'custom-list-secrets'</em> to grant permission for listing Secrets.</li> </ul> <pre><code class="language-shell"> # cat custom-list-secrets.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:   name: custom-list-secret   labels:     rbac.authorization.kubeflow.org/aggregate-to-custom-kubeflow-edit: "true" rules:   - apiGroups:     - ""     resources:     - secrets     verbs:     - get     - list # kubectl apply -f custom-list-secrets.yaml clusterrole.rbac.authorization.k8s.io/custom-list-secret created </code></pre> <p>The ClusterRole includes the label <em>'rbac.authorization.kubeflow.org/aggregate-to-custom-kubeflow-edit: "true"'</em>.</p> <p>Run the following command to verify that permission to list Secrets has been incorporated and merged into the aggregated ClusterRole <em>'custom-kubeflow-edit'</em>.</p> <pre><code class="language-shell"># kubectl get clusterrole custom-kubeflow-edit -o jsonpath='{.rules}' | jq . [ { "apiGroups": [ "" ], "resources": [ "secrets" ], "verbs": [ "get", "list" ] } ] </code></pre> <ul> <li>Create another focused ClusterRole <em>'custom-exec-pods'</em> to grant permission for executing commands inside a running Pod's container.</li> </ul> <pre><code class="language-shell"># cat custom-exec-pods.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:   name: custom-exec-pods   labels:     rbac.authorization.kubeflow.org/aggregate-to-custom-kubeflow-edit: "true" rules:   - apiGroups:     - "*"     resources:     - pods/exec     verbs:     - "*" # kubectl apply -f custom-exec-pods.yaml clusterrole.rbac.authorization.k8s.io/custom-exec-pods created </code></pre> <p>Run the following command to verify that the exec-command permissions have also been incorporated and merged into the aggregated ClusterRole <em>'custom-kubeflow-edit'</em>.</p> <pre><code class="language-shell"># kubectl get clusterrole custom-kubeflow-edit -o jsonpath='{.rules} ' | jq . [ { "apiGroups": [ "*" ], "resources": [ "pods/exec" ], "verbs": [ "*" ] }, { "apiGroups": [ "" ], "resources": [ "secrets" ], "verbs": [ "get", "list" ] } ] </code></pre> <ul> <li>Create a <em>custom</em> RoleBinding named <em>'custom-editor'</em> to bind the aggregated ClusterRole <em>'custom-kubeflow-edit'</em> to the ServiceAccount <em>'default-editor'</em> in the user namespace <em>'project-user-guoping-jia'</em>.</li> </ul> <pre><code class="language-shell"> # cat custom-rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:   name: custom-editor   namespace: project-user-guoping-jia roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: custom-kubeflow-edit subjects: - kind: ServiceAccount   name: default-editor   namespace: project-user-guoping-jia # kubectl apply -f custom-rolebinding.yaml rolebinding.rbac.authorization.k8s.io/custom-editor created # kubectl get rolebinding -n project-user-guoping-jia custom-editor NAME ROLE AGE custom-editor ClusterRole/custom-kubeflow-edit 84s 2026-02-17T10:43:00Z </code></pre> <ul> <li>Verify the added permissions</li> </ul> <p>From the Jupyter notebook terminal, you should now be able to list Secrets and execute <em>bash</em> commands inside a running Pod's container within the namespace.</p> <pre><code class="language-shell"> (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i list secrets yes (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i get pods --subresource=exec yes (base) guoping-jia@default-notebook-0:~$ kubectl get secrets NAME                    TYPE                             DATA   AGE access-token            Opaque                           1      24d af-cluster-airflowui    Opaque                           6      24d hpe-imagepull-secrets   kubernetes.io/dockerconfigjson   1      24d imagepull               kubernetes.io/dockerconfigjson   1      24d ngc-cli-secret          Opaque                           2      24d (base) guoping-jia@default-notebook-0:~$ kubectl get pods NAME                                       READY   STATUS    RESTARTS   AGE default-notebook-0                         2/2     Running   0          45h fs-65cbb7b876-x9564                        2/2     Running   0          24d ml-pipeline-ui-artifact-696cff4647-46slx   2/2     Running   0          24d (base) guoping-jia@default-notebook-0:~$ kubectl exec -it default-notebook-0 -- bash root@default-notebook-0:/# pwd / root@default-notebook-0:/# exit exit (base) guoping-jia@default-notebook-0:~$ </code></pre> <h4>Granting user access to other namespaces</h4> <p>As another practical example of permission management, this section covers a more advanced requirement: granting a user access to a namespace other than their default one. In this additional namespace, where AI applications may be deployed, for example through the PCAI <em>Import Framework</em>, the user may need access for debugging or inspection.</p> <p>By default, the ClusterRole and RoleBinding configuration applied in the Jupyter notebook environment restricts each authenticated user to their own namespace. They cannot access other namespaces, such as <em>'custom-ns'</em>, unless additional permissions are explicitly granted.</p> <pre><code class="language-shell"> # kubectl create ns custom-ns namespace/custom-ns created # kubectl get ns custom-ns custom-ns                               Active   3s </code></pre> <p>The following error appears when attempting to access the newly created namespace from the Jupyter notebook terminal.</p> <pre><code class="language-shell"> (base) guoping-jia@default-notebook-0:~$ kubectl get ns custom-ns Error from server (Forbidden): namespaces "custom-ns" is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot get resource "namespaces" in API group "" in the namespace "custom-ns" (base) guoping-jia@default-notebook-0:~$ kubectl get pods -n custom-ns Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot list resource "pods" in API group "" in the namespace "custom-ns" </code></pre> <p>User access to the other namespace can be granted by creating a RoleBinding named <em>'default-editor'</em>.</p> <pre><code class="language-shell"> # cat ns-default-editor.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:   name: default-editor   namespace: custom-ns roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: kubeflow-edit subjects: - kind: ServiceAccount   name: default-editor   namespace: project-user-guoping-jia # kubectl apply -f ns-default-editor.yaml rolebinding.rbac.authorization.k8s.io/default-editor created # kubectl get rolebinding -n custom-ns NAME             ROLE                        AGE default-editor   ClusterRole/kubeflow-edit   10s </code></pre> <p>The RoleBinding <em>'default-editor'</em> binds the Kubeflow ClusterRole <em>kubeflow-edit</em> to the ServiceAccount <em>default-editor</em> in the namespace <em>custom-ns</em>.</p> <p>Run the following commands to verify that the user can now access the namespace <em>'custom-ns'</em>.</p> <pre><code class="language-shell"> (base) guoping-jia@default-notebook-0:~$ kubectl get ns custom-ns NAME        STATUS   AGE custom-ns   Active   13m </code></pre> <p>However, with the default RoleBinding <em>'default-editor'</em> in place, the user still cannot list Secrets in the namespace <em>'custom-ns'</em>, nor can they execute the commands inside any running Pod's container within that namespace.</p> <pre><code class="language-shell">(base) guoping-jia@default-notebook-0:~$ kubectl auth can-i list secrets -n custom-ns no (base) guoping-jia@default-notebook-0:~$ kubectl get secrets -n custom-ns Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot list resource "secrets" in API group "" in the namespace "custom-ns" (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i get pods --subresource=exec -n custom-ns no (base) guoping-jia@default-notebook-0:~$ kubectl exec -it mybox-84bdf9578-b6tkh -n custom-ns -- bash Error from server (Forbidden): pods "mybox-84bdf9578-b6tkh" is forbidden: User "system:serviceaccount:project-user-guoping-jia:default-editor" cannot create resource "pods/exec" in API group "" in the namespace "custom-ns" </code></pre> <p>These additional permissions can be granted in the namespace <em>'custom-ns'</em> by creating another RoleBinding named <em>'custom-editor'</em>.</p> <pre><code class="language-shell"> # cat ns-custom-editor.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:   name: custom-editor   namespace: custom-ns roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: custom-kubeflow-edit subjects: - kind: ServiceAccount   name: default-editor   namespace: project-user-guoping-jia # kubectl apply -f ns-custom-editor.yaml rolebinding.rbac.authorization.k8s.io/custom-editor created # kubectl get rolebinding -n custom-ns NAME             ROLE                               AGE custom-editor    ClusterRole/custom-kubeflow-edit   8s default-editor   ClusterRole/kubeflow-edit          12m </code></pre> <p>The RoleBinding <em>'custom-editor'</em> binds the aggregated ClusterRole <em>'custom-kubeflow-edit'</em>, created in the previous section, to the ServiceAccount <em>'default-editor'</em> in the namespace <em>'custom-ns'</em>.</p> <p>With the RoleBinding <em>'custom-editor'</em> in place, and with <em>'custom-kubeflow-edit'</em> ClusterRole already incorporating the two additional permissions, you can now verify that both listing Secrets and executing a <em>bash</em> command inside a running Pod's container work as expected in the namespace <em>'custom-ns'</em>.</p> <pre><code class="language-shell">(base) guoping-jia@default-notebook-0:~$ kubectl auth can-i list secrets -n custom-ns yes (base) guoping-jia@default-notebook-0:~$ kubectl auth can-i get pods --subresource=exec -n custom-ns yes (base) guoping-jia@default-notebook-0:~$ kubectl create deploy nginx -n custom-ns --image=nginx  deployment.apps/nginx created (base) guoping-jia@default-notebook-0:~$ kubectl get pods -n custom-ns NAME                     READY   STATUS    RESTARTS   AGE nginx-5869d7778c-b4tcw   1/1     Running   0          56s (base) guoping-jia@default-notebook-0:~$ kubectl exec -it nginx-5869d7778c-b4tcw -n custom-ns -- bash root@nginx-5869d7778c-b4tcw:/# ls -al   total 4 drwxr-xr-x.    1 root root   39 Feb  5 14:49 . drwxr-xr-x.    1 root root   39 Feb  5 14:49 .. lrwxrwxrwx.    1 root root    7 Jan  2 12:35 bin -> usr/bin drwxr-xr-x.    2 root root    6 Jan  2 12:35 boot drwxr-xr-x     5 root root  360 Feb  5 14:49 dev drwxr-xr-x.    1 root root   32 Feb  5 14:49 etc drwxr-xr-x.    2 root root    6 Jan  2 12:35 home lrwxrwxrwx.    1 root root    7 Jan  2 12:35 lib -> usr/lib lrwxrwxrwx.    1 root root    9 Jan  2 12:35 lib64 -> usr/lib64 drwxr-xr-x.    2 root root    6 Feb  2 00:00 mnt drwxr-xr-x.    2 root root    6 Feb  2 00:00 opt dr-xr-xr-x. 3149 root root    0 Feb  5 14:49 proc drwx------.    2 root root   37 Feb  2 00:00 root lrwxrwxrwx.    1 root root    8 Jan  2 12:35 sbin -> usr/sbin drwxr-xr-x.    2 root root    6 Feb  2 00:00 srv dr-xr-xr-x.   13 root root    0 Nov 14 17:04 sys drwxrwxrwt.    2 root root    6 Feb  2 00:00 tmp drwxr-xr-x.    1 root root   66 Feb  2 00:00 usr drwxr-xr-x.    1 root root   19 Feb  2 00:00 var root@nginx-5869d7778c-b4tcw:/# exit exit (base) guoping-jia@default-notebook-0:~$ </code></pre> <p>If additional permission are needed in the namespace, you can follow the same approach by defining a purpose-specific ClusterRole with the appropriate aggregation label. Once the ClusterRole is applied, its permissions will be automatically merged into the aggregated ClusterRole.</p> <h3>Conclusion</h3> <p>This blog post explored and demonstrated how permission management for accessing K8s resources in the HPE Private Cloud AI environment can be simplified and streamlined through ClusterRole aggregation. When additional permissions are required, they can be defined as independent, purpose‑built ClusterRoles, which are then automatically incorporated into an aggregated ClusterRole by applying the appropriate labels. This approach eliminates the need to modify existing ClusterRoles for each new permission request and significantly reduces RBAC maintenance overhead. By relying on smaller, focused roles that aggregates cleanly, permission management becomes more flexible, scalable, and easier to maintain.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud AI and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/why-storage-is-important-for-kv-cache/https://developer.hpe.com/why-storage-is-important-for-kv-cache/Mon, 16 Feb 2026 08:29:33 GMT<style> table { display: table; width: 100%; max-width: 100%; margin: 20px auto; border-collapse: collapse; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border: 1px solid grey; } th, td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border: 1px solid grey; text-align: left !important; font-weight: normal !important; padding: 10px !important; } th { text-align: center !important; font-weight: bold !important; background-color: #f5f5f5; font-weight: bold !important; } </style> <p>Although KV (<em>Key-value)</em> cache is usually described as an LLM inference optimization, it is actually best understood as a specialized, high‑performance storage layer that holds intermediate attention states. This article explores this aspect of KV cache and its relationship with storage.</p> <br/> <hr> <br/> <h2><span style="color:blue; font-family:Arial; font-size:1em"> What KV cache is</span></h2> <p>You can think of KV cache as a volatile GPU–resident, key–value store that stores per-token features so they don’t need to be recomputed. In essence, it acts like a memory tier within a multi-layer storage hierarchy.</p> <div align="center"> Here’s how to map KV cache to conventional storage concepts: <table> <thead> <tr> <th>Storage concept</th> <th>LLM equivalent</th> <th>Explanation</th> </tr> </thead> <tbody> <tr> <td>Registers</td> <td>Tensor cores, attention units</td> <td>Compute engines</td> </tr> <tr> <td>L1/L2 cache</td> <td>KV cache slices currently in use</td> <td>Immediate access attention data</td> </tr> <tr> <td>RAM</td> <td>Overall KV cache across all layers</td> <td>Working set for model inference</td> </tr> <tr> <td>SSD / object storage</td> <td>Prompt, documents</td> <td>Fed in before KV cache populates</td> </tr> <tr> <td>Cold storage</td> <td>Archived corpora, vector DB, documents</td> <td>Retrieved only as needed</td> </tr> </tbody> </table> </div> <blockquote> <p><span style="color:blue; font-family:Arial; font-size:1em"> <em>You can think of the KV cache as a model's “High-Bandwidth memory (HBM) scratchpad.”</em></span></p> </blockquote> <br/> <hr> <br/> <h2><span style="color:blue; font-family:Arial; font-size:1em"> Why KV cache is important for AI and generative AI</span></h2> <h3><span style="color:blue; font-family:Arial; font-size:1em">Why KV cache is essential to Retrieval Augmented Generation (RAG) and inference in general</span></h3> <p>KV cache is essential for RAG because it lets the model handle long-retrieved contexts without having to recompute attention over all those tokens at every decoding step. When a large block of retrieved text is inserted into a prompt, the model encodes it once, stores the keys and values, and then reuses them during generation. This means new tokens only attend to the cached prefix instead of reprocessing thousands of context tokens repeatedly. As the retrieved context grows, KV cache keeps latency stable, prevents compute from exploding with sequence length, and makes long-context RAG both feasible and efficient.</p> <h3><span style="color:blue; font-family:Arial; font-size:1em"> Why KV cache is important for agentic AI</span></h3> <h4>Agentic AI requires</h4> <p>A couple of things to note about KV cache when used in agentic AI situations include its:</p> <ul> <li> <p>Long contexts, long chains of thought, and multi-turn loops: Agents concatenate system prompts + chat history + retrieved chunks + tool outputs. Every added token expands the KV working set. The model then rereads prior tokens’ K/V at each step.</p> </li> <li> <p>Sensitivity to bandwidth: The attention kernel’s speed is often limited by HBM bandwidth, not FLOPs. If KV reads stall, per token latency increases, tail latencies widen, and throughput collapses under concurrency.</p> </li> <li> <p>Persistence across actions and Memory of past steps: Agent steps (plan, call tools, reflect) frequently reuse the same conversation context. KV reuse avoids recomputing attention for the past—saving both time and power.</p> </li> </ul> <p>This means agents spend minutes inside a single inference session. That makes KV cache the central runtime storage system for agent state. If the KV cache is slow, insufficient, or mismanaged:</p> <ul> <li>The agent must recompute attention resulting in a huge latency spike</li> <li>The model must truncate context resulting in memory loss</li> <li>Multi-step reasoning grinds to a halt</li> </ul> <br/> <hr> <br/> <h3><span style="color:blue; font-family:Arial; font-size:1em">Why KV cache is a storage problem</span></h3> <p>When examined closely, the KV cache becomes a storage issue.</p> <p>When a large language model generates text one token at a time, it keeps a memory of everything it has already seen. This memory lives in the KV cache a set of tensors that store the keys and values for every layer and every past token.</p> <p>At first, this cache is small. A few tokens, a few layers, a bit of GPU memory. But as the conversation grows longer—say from 1,000 tokens to 32,000—this “memory” expands linearly with the size of the context. The model must keep every past key/value vector around so that new tokens can attend back to them. And suddenly, the KV cache, not the model weights, becomes the largest memory consumer.</p> <p>As GPUs have limited High-Bandwidth Memory (HBM), when the KV cache grows too large, systems must decide whether:</p> <ul> <li>Moving it (offloading) to CPU RAM or even to fast storage helps with capacity but reduces speed, because every new token must fetch pieces of that cache across slower connection interconnects.</li> <li>Compressing it saves space but may reduce accuracy.</li> <li>Storing everything in GPU memory maintains performance but limits the number of users served simultaneously.</li> </ul> <p>It is easy to see that KV cache becomes not just a compute issue but a storage orchestration challenge—balancing capacity, bandwidth, latency, and cost. Long-context models, multi-user serving, and high-throughput inference all rely on how cleverly we can store, move, compress, or reuse this cache. Ultimately, generating text efficiently depends as much on memory engineering as on math.</p> <p>As KV cache is a storage orchestration challenge, the ability to move back and forward the KV cache in a fast storage system is a critical strategy for managing the KV cache. Here's more information that details the reasons behind this.</p> <h4>KV cache requires extremely high bandwidth (HBM class)</h4> <p>KV cache access patterns behave in such a way that:</p> <ul> <li>Every new token must retrieve all previous keys and values</li> <li>Access frequency is done per layer × per head</li> </ul> <p>This means KV cache cannot be used if in CPU memory, SSD, or network storage. To be used, it must remain in GPU HBM, which is effectively the highest performance “storage tier” available.</p> <h4>KV cache often can't fit in GPU memory</h4> <p>For effectiveness, KV cache must remain in GPU memory, but unfortunately, it often can’t fit there. The size of the KV cache increases linearly with both the context length and the number of layers, creating capacity and bandwidth challenges.</p> <p>For example:</p> <ul> <li>A 70B model can require tens of gigabytes of KV cache just for 32k tokens.</li> <li>Larger contexts (100k–1M tokens) would require storage-tier thinking, such as sharding, compression, paging, etc.</li> </ul> <p>When the KV cache exceeds the HBM memory, it must be stored elsewhere (CPU memory first, then fast storage).</p> <h4>KV cache often doesn’t reside in a single GPU</h4> <p>But there is an additional complexity. The KV cache often doesn’t reside in a single GPU. When models run across multiple GPUs, the KV cache is sharded across them. KV cache “follows” the model parallelism:</p> <ul> <li>Tensor parallelism: shard across layers/heads</li> <li>Sequence parallelism: shard by tokens</li> <li>Context parallelism: shard by ranges of inputs</li> </ul> <p>This makes KV management similar to distributed file systems:</p> <ul> <li>Placement</li> <li>Replication</li> <li>Access path optimization</li> <li>Paging / eviction</li> </ul> <blockquote> <p><span style="color:blue; font-family:Arial; font-size:1em"> <em>In conclusion, KV cache pagination requires a memory-tiered storage</em></span></p> </blockquote> <p>Indeed, recent trends see the development of storage systems optimized for tiering with KV-cache. Storage systems are, therefore, part of the KV-cache pagination management:</p> <ul> <li>GPU HBM → hot KV</li> <li>CPU RAM → warm KV</li> <li>NVMe / SSD → cold KV</li> </ul> <hr> <br/> <h3><span style="color:blue; font-family:Arial; font-size:1em">Finally, why RDMA/GPUDirect (for objects or files) is important for KV cache</span></h3> <p>This diagram shows the end-to-end RAG + inference flow and where RDMA / GPUDirect optimizes movement into the GPU—before the KV cache becomes active.</p> <p><img src="/img/screenshot-2026-02-16-at-10.41.09.png" alt="" title="Figure 1 - RDMA/GDS role in a RAG pipeline"></p> <p>RDMA / GPUDirect help:</p> <ul> <li> <p>Storage → GPU ingestion:</p> <ul> <li>RDMA (object/file) + GPUDirect allow NIC or storage to DMA directly into GPU memory, bypassing CPU copies.</li> <li>This accelerates document load, embedding pipelines, vector index updates, and LLM input streaming.</li> </ul> </li> <li> <p>Before KV activation:</p> <ul> <li>KV cache is populated after tokenization and initial forward passes occur on GPU. RDMA/GPUDirect primarily reduce the time to first token by accelerating data arrival to the GPU.</li> </ul> </li> <li> <p>During RAG loops:</p> <ul> <li>Frequent retrieval (top k) + reranking benefits from GPU-resident vector DB and RDMA reads from warm storage. The faster the context assembly, the sooner the LLM can append to KV cache.</li> </ul> </li> </ul> <p>In other words, RDMA/GPUDirect accelerate the front half (docs/embeddings/context into GPU memory). Once generation starts, the KV cache dominates the hot path, acting as the L1/L2 like working store of the decoder.</p> <br/> <hr> <br/> <h2><span style="color:blue; font-family:Arial; font-size:1em">Conclusion</span></h2> <p>KV cache management is a storage orchestration challenge that affects capacity, bandwidth, latency, and cost, and efficient text generation depends as much on memory engineering as on algorithms. Fast storage is essential for maximizing KV cache performance. By placing the right portions of the cache on fast, low‑latency. RDMA- and GDS-enabled storage - such as HPE Alletra MP X10000 - and offloading overflow to cost‑efficient tiers, organizations can balance speed, scale, and efficiency.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blogs</a>and AI Tips for more guides and best practices on AI and Storage for AI.</p>https://developer.hpe.com/2026-february-09/https://developer.hpe.com/2026-february-09/Mon, 09 Feb 2026 13:00:00 GMThttps://developer.hpe.com/7-questions-for-oliver-alvarado-rodriguez-exploiting-chapels-distributed-arrays-for-graph-analysis-through-arachne/https://developer.hpe.com/7-questions-for-oliver-alvarado-rodriguez-exploiting-chapels-distributed-arrays-for-graph-analysis-through-arachne/Thu, 22 Jan 2026 00:38:36 GMT<p>external blog</p>https://developer.hpe.com/automating-application-delivery-to-mks-clusters-using-gitops-with-argocd/https://developer.hpe.com/automating-application-delivery-to-mks-clusters-using-gitops-with-argocd/Tue, 13 Jan 2026 13:52:48 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>While Kubernetes (K8s) is widely adopted, enterprises often face challenges when deploying and managing application workloads across K8s clusters, including configuration drift, inconsistent deployment practices, and limited operational visibility. As environments grow, manual processes introduce variability and make compliance and rollback procedures increasingly difficult. A GitOps model addresses these challenges by using Git as the <em>single source of truth</em> for all declarative configurations and maintaining alignment through continuous reconciliation. This approach enhances reliability, strengthens auditability, and enables predictable, repeatable deployments across complex K8s environments.</p> <p>This blog post describes how to automate application deployments to Morpheus Kubernetes Service (MKS) clusters using a GitOps workflow powered by <em>Argo CD</em>. While the implementation example focuses on HPE Private Cloud Enterprise, the same approach applies seamlessly to HPE Morpheus Enterprise as long as a MKS cluster is provisioned. By taking advantage of <em>Argo CD</em>’s real‑time monitoring and alerting features, the solution provides clear visibility into application deployment health and enforces strict alignment between the live cluster state and the declarative configuration stored in Git. This ensures reliable, consistent, and version‑controlled application delivery across both environments.</p> <h3>What is GitOps?</h3> <p>GitOps is an operational framework that extends core DevOps principles, such as version control, collaboration, compliance, and CI/CD (continuous integration and continuous delivery), and applies them to infrastructure automation. In a GitOps workflow, Git acts as the single source of truth for all declarative infrastructure and application configurations. The entire desired state of a Kubernetes (K8s) cluster, including resources such as <em>Deployments</em>, <em>Services</em>, <em>ConfigMaps</em>, and more, is stored in Git. A GitOps controller continuously compares the cluster's actual state with the desired state defined in the repository and reconciles any differences to ensure consistency. Several well-known GitOps tools exist, including <em>Argo CD</em>, <em>Flux CD</em>, <em>Jenkins X</em>, and <em>Spinnaker</em>. For the purpose of this blog, <em>Argo CD</em> will be used as the primary tool for demonstrating GitOps automation.</p> <h3>What is Argo CD?</h3> <p><em><a href="https://argoproj.github.io/cd/">Argo CD</a></em> is a declarative, GitOps-driven continuous delivery platform for K8s. It automates consistent and repeatable application deployments by using Git as the single source of truth for all configuration artifacts. <em>Argo CD</em> continuously monitors the live state of applications running in a K8s cluster and compares it against the desired state defined in a Git repository. When developers push changes to Git, <em>Argo CD</em> detects the updates and synchronizes them to the cluster.</p> <p>Synchronization can be configured to run automatically, commonly used for development and test environments, or manually, which is typically preferred for production workflows. By defining the target environment state in Git, <em>Argo CD</em> ensures that the applications deployed in the K8s cluster remain aligned with the declared configuration.</p> <p>Beyond synchronization, <em>Argo CD</em> provides real-time insights into application health, deployment status, and configuration drift through its monitoring and alerting capabilities. It integrates seamlessly with existing CI/CD pipelines and enforces GitOps best practices throughout the application deployment lifecycle.</p> <p>The following sections provide a technical walkthrough for automating application deployments to MKS clusters using a GitOps workflow built around <em>Argo CD</em>. This blog specifically demonstrates the end‑to‑end process in a <a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> environment, including installing <em>Argo CD</em> on a MKS cluster, connecting application source repositories, configuring deployment parameters, and monitoring the application state within the cluster.</p> <h3>Prerequisites</h3> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>A MKS cluster has been provisioned from an HPE Private Cloud Enterprise workspace. You can refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a> to provision an MKS cluster.</li> <li>The <em>kubectl</em> CLI tool has been properly installed, along with the kubeconfig file used to access the MKS cluster.</li> </ul> <h3>Install Argo CD</h3> <p>You can install <em>Argo CD</em> using either the provided <a href="https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml">installation YAML file</a> with <em>kubectl</em> or the <a href="https://argoproj.github.io/argo-helm">Helm charts</a> with <em>helm</em>. Refer to the <a href="https://argo-cd.readthedocs.io/en/stable/getting_started/">Argo CD installation documentation</a> for detailed instructions.</p> <p>For MKS clusters in HPE Private Cloud Enterprise, you can automate the installation by creating a <em>Shell</em> script and add it as a Morpheus task. Running this task from the MKS master node will deploy <em>Argo CD</em> to the cluster.</p> <p>Below is the <em>Shell</em> script used to install <em>Argo CD</em>.</p> <pre><code class="language-shell">#!/bin/bash curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 chmod 700 get_helm.sh ./get_helm.sh helm repo add argo https://argoproj.github.io/argo-helm if [ -f "/etc/kubernetes/admin.conf" ]; then helm install argocd argo/argo-cd --version 9.1.3 -n argocd --create-namespace --kubeconfig /etc/kubernetes/admin.conf else echo "Its worker node.!! skiping installation" fi </code></pre> <p>Follow the steps below to install <em>Argo CD</em> on the MKS cluster.</p> <ol> <li>In the Morpheus Dashboard, navigate to <strong>Library -> Automation -></strong> <em>Tasks</em> tab. Click <em><strong>+ Add</strong></em>.</li> </ol> <p><img src="/img/add-task.png" alt=""></p> <ol start="2"> <li>Enter NAME as <em>Install ArgoCD</em> and select TYPE as <em>Shell Script</em>. Enable <em>SUDO</em>, select SOURCE as <em>Local</em>, paste the <em>Shell</em> script into CONTENT. Click <em><strong>Save changes</strong></em>.</li> </ol> <p><img src="/img/install-argocd-task.png" alt=""></p> <ol start="3"> <li>Navigate to <strong>Infrastructure -> Clusters</strong>. Click the target MKS cluster NAME, e.g., <em>mks-test</em>.</li> </ol> <p><img src="/img/mks-test-cluster.png" alt=""></p> <ol start="4"> <li>Open the <em>Nodes</em> tab. Click the MKS master node, e.g., <em>mks-test-master</em>.</li> </ol> <p><img src="/img/mks-test-nodes.png" alt=""></p> <ol start="5"> <li>Click <em><strong>Actions</strong></em>. Select <em>Run Task</em>.</li> </ol> <p><img src="/img/mks-test-master-node.png" alt=""></p> <ol start="6"> <li>Select <em>Install ArgoCD</em> as TASK.</li> </ol> <p><img src="/img/mks-test-task-execute.png" alt=""></p> <ol start="7"> <li>Open <em>History</em> tab. Click info (<strong>"i"</strong>) icon to view the logs for <em>Run Task: Install ArgoCD</em>.</li> </ol> <p><img src="/img/mks-test-task-history.png" alt=""></p> <p>You can verify that <em>Argo CD</em> has been successfully deployed to the namespace <em>'argocd'</em>.</p> <pre><code class="language-shell">$ kubectl get ns argocd NAME STATUS AGE argocd Active 52s $ k get all -n argocd NAME READY STATUS RESTARTS AGE pod/argocd-application-controller-0 1/1 Running 0 48s pod/argocd-applicationset-controller-579f778f57-m4t4d 1/1 Running 0 48s pod/argocd-dex-server-7d99b44d96-bjvxn 1/1 Running 0 48s pod/argocd-notifications-controller-57dd69d5d9-5dtw2 1/1 Running 0 48s pod/argocd-redis-9ff9dddb8-x58gp 1/1 Running 0 48s pod/argocd-redis-secret-init-9hqtt 0/1 Completed 0 52s pod/argocd-repo-server-56dbf9bd9-xbx8z 1/1 Running 0 48s pod/argocd-server-5d56b98664-q2l5f 1/1 Running 0 48s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/argocd-applicationset-controller ClusterIP 172.30.219.75 &#x3C;none> 7000/TCP 50s service/argocd-dex-server ClusterIP 172.30.32.229 &#x3C;none> 5556/TCP,5557/TCP 50s service/argocd-redis ClusterIP 172.30.65.186 &#x3C;none> 6379/TCP 50s service/argocd-repo-server ClusterIP 172.30.1.116 &#x3C;none> 8081/TCP 50s service/argocd-server ClusterIP 172.30.64.166 &#x3C;none> 80/TCP,443/TCP 50s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/argocd-applicationset-controller 1/1 1 1 50s deployment.apps/argocd-dex-server 1/1 1 1 49s deployment.apps/argocd-notifications-controller 1/1 1 1 50s deployment.apps/argocd-redis 1/1 1 1 49s deployment.apps/argocd-repo-server 1/1 1 1 50s deployment.apps/argocd-server 1/1 1 1 49s NAME DESIRED CURRENT READY AGE replicaset.apps/argocd-applicationset-controller-579f778f57 1 1 1 50s replicaset.apps/argocd-dex-server-7d99b44d96 1 1 1 49s replicaset.apps/argocd-notifications-controller-57dd69d5d9 1 1 1 50s replicaset.apps/argocd-redis-9ff9dddb8 1 1 1 49s replicaset.apps/argocd-repo-server-56dbf9bd9 1 1 1 49s replicaset.apps/argocd-server-5d56b98664 1 1 1 49s NAME READY AGE statefulset.apps/argocd-application-controller 1/1 49s NAME STATUS COMPLETIONS DURATION AGE job.batch/argocd-redis-secret-init Complete 1/1 3s 53s </code></pre> <p>Keep in mind that the <em>'argocd-server'</em> service is configured as the <em>ClusterIP</em> type, meaning it is accessible only from within the cluster.</p> <h3>Configure Argo CD access</h3> <p>To use <em>Argo CD</em> for application deployments, the <em>Argo CD</em> service must be accessible from outside the MKS cluster. This can be achieved through several methods, including <em>port forwarding</em>, <em>NodePort</em> or <em>LoadBalancer</em> type services, or a manual virtual private network (VPN) setup.</p> <p>In this blog, I show an alternative approach: exposing the <em>Argo CD</em> service to the public Internet using <em>Tailscale</em>. For background on <em>Tailscale</em> and the service-exposure workflow, refer to the blog post <a href="https://developer.hpe.com/blog/exposing-grafana-service-using-tailscale-for-mks-monitoring-in-hpe-private-cloud-enterprise/">Exposing Grafana service using Tailscale for MKS monitoring in HPE Private Cloud Enterprise</a>.</p> <p>Follow the steps below to expose the <em>Argo CD</em> service endpoint using <em>Tailscale</em>.</p> <ol> <li>Update the <em>Argo CD</em> service type from <em>ClusterIP</em> to <em>LoadBalancer</em>.</li> </ol> <pre><code class="language-shell">$ kubectl edit svc/argocd-server -n argocd service/argocd-server edited </code></pre> <ol start="2"> <li>Confirm that the service type has been updated successfully.</li> </ol> <pre><code class="language-shell">$ kubectl get svc -n argocd NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argocd-applicationset-controller ClusterIP 172.30.219.75 &#x3C;none> 7000/TCP 2m47s argocd-dex-server ClusterIP 172.30.32.229 &#x3C;none> 5556/TCP,5557/TCP 2m47s argocd-redis ClusterIP 172.30.65.186 &#x3C;none> 6379/TCP 2m47s argocd-repo-server ClusterIP 172.30.1.116 &#x3C;none> 8081/TCP 2m47s argocd-server LoadBalancer 172.30.64.166 172.20.20.242 80:30147/TCP,443:31601/TCP 2m47s </code></pre> <ol start="3"> <li>Create an <em>Ingress</em> YAML manifest and apply it to the cluster.</li> </ol> <pre><code class="language-shell">$ cat ingress-argocd.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-argocd namespace: argocd annotations: tailscale.com/funnel: "true" spec: defaultBackend: service: name: argocd-server port: number: 443 ingressClassName: tailscale tls: - hosts: - argocd $ kubectl apply -f ingress-argocd.yaml ingress.networking.k8s.io/ingress-argocd created </code></pre> <ol start="4"> <li>Verify that the <em>Ingress</em> resource <em>'ingress-argocd'</em> has been created and assigned an address.</li> </ol> <pre><code class="language-shell">$ kubectl get ingress -n argocd NAME CLASS HOSTS ADDRESS PORTS AGE ingress-argocd tailscale * argocd.qilin-beta.ts.net 80, 443 8s </code></pre> <ol start="5"> <li>In the <em>Tailscale</em> admin console, a new machine named <em>'argocd'</em> should appear under the <em>Machines</em> tab.</li> </ol> <p><img src="/img/tailscale-argocd.png" alt=""></p> <ol start="6"> <li>Open a browser and point to the <em>Argo CD</em> Funnel URL, e.g., <em>'argocd.qilin-beta.ts.net '</em>.</li> </ol> <p><img src="/img/argocd-login.png" alt=""></p> <p>Log in using the username <em>'admin'</em>. Run the following command to retrieve the admin password.</p> <pre><code class="language-shell">$ kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d </code></pre> <h3>Deploy applications using <em>Argo CD</em></h3> <p>You can now use the <em>Argo CD</em> service endpoint to connect your application's code repository and begin deploying applications to the MKS cluster.</p> <h4>Connect application repository</h4> <p>The sample <em>WordPress</em> application's Helm charts are available in the GitHub repository <a href="https://github.com/GuopingJia/helm-demo">helm-demo</a>.</p> <p>In this section, I will outline the steps for connecting that repository to <em>Argo CD</em>.</p> <ul> <li>In the <em>Argo CD</em> UI, navigate to <em><strong>Settings -> Repository</strong></em>. Click <em><strong>+ CONNECT REPO</strong></em>.</li> </ul> <p><img src="/img/argocd-git.png" alt=""></p> <ul> <li>Select <em>VIA HTTP/HTTPS</em> as the connection method, choose <em>git</em> as Type, set Project to <em>default</em>, optional name as <em>wordpress</em>, and enter Repository URL, e.g., <em><a href="https://github.com/GuopingJia/helm-demo">https://github.com/GuopingJia/helm-demo</a></em>. Click <em><strong>CONNECT</strong></em>.</li> </ul> <p><img src="/img/argocd-git-connect.png" alt=""></p> <ul> <li>After a few moments, the repository's CONNECTION STATUS should display as <em>Successful</em>.</li> </ul> <p><img src="/img/argocd-git-wordpress.png" alt=""></p> <h4>Create application</h4> <p>Follow the steps below to create a new application for deployment.</p> <ol> <li>In the <em>Argo CD</em> UI, navigate to <em><strong>Applications</strong></em>. Click <em><strong>CREATE APPLICATION</strong></em>.</li> </ol> <p><img src="/img/argocd-application.png" alt=""></p> <ol start="2"> <li>Enter Application Name as <em>wordpress-app</em>, set Project Name to <em>default</em> and choose <em>Automatic</em> for SYNC POLICY. For SOURCE, select Repository URL, set Revision to <em>HEAD</em>, and specify Path as <em>wordpress</em>. For DESTINATION, choose Cluster URL as <em><a href="https://kubernetes.default.svc">https://kubernetes.default.svc</a></em> and set Namespace to <em>wordpress</em>. Enable additional options such as ENABLE AUTO-SYNC, SELF HEAL, AUTO-CREATE NAMESPACE and RETRY. Click <em><strong>CREATE</strong></em>.</li> </ol> <p><em><strong>Note</strong></em>: If you choose <em>Manual</em> under SYNC POLICY, <em>Argo CD</em> will detect the changes in the Git repository but will not apply them automatically. You must manually click <em>SYNC</em> on Applications page to apply updates.</p> <p><img src="/img/argocd-application-create.png" alt=""></p> <ol start="3"> <li>The <em>wordpress-app</em> deployment will begin immediately.</li> </ol> <p><img src="/img/argocd-application-pogressing.png" alt=""></p> <ol start="4"> <li>After a short time, the application should complete deployment with APP HEALTH showing <em>Healthy</em> and SYNC STATUS showing <em>Synced</em>.</li> </ol> <p><img src="/img/argocd-application-running.png" alt=""></p> <h4>Make changes to application repository</h4> <p>Now let’s update the application repository by increasing the number of replicas for the <em>WordPress</em> application from the default value of <em>1</em> to <em>2</em>.</p> <p><img src="/img/git-repo-changes.png" alt=""></p> <p>Once the change is committed, <em>Argo CD</em> will detect the update and automatically begin synchronizing it to the MKS cluster.</p> <p><img src="/img/argocd-application-syncing.png" alt=""></p> <p>After the sync completes, run the following command to verify that <em>WordPress</em> is now running with 2 PODs in the cluster.</p> <pre><code class="language-shell">$ kubectl get all -n wordpress NAME READY STATUS RESTARTS AGE pod/wordpress-app-df478fc99-gwffz 1/1 Running 0 23s pod/wordpress-app-df478fc99-qrrg4 1/1 Running 0 22s pod/wordpress-app-mariadb-0 1/1 Running 0 162m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/wordpress-app LoadBalancer 172.30.18.82 172.20.20.243 80:30704/TCP,443:32092/TCP 162m service/wordpress-app-mariadb ClusterIP 172.30.18.199 &#x3C;none> 3306/TCP 162m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/wordpress-app 2/2 2 2 162m NAME DESIRED CURRENT READY AGE replicaset.apps/wordpress-app-69fdf9dcd9 0 0 0 162m replicaset.apps/wordpress-app-df478fc99 2 2 2 23s NAME READY AGE statefulset.apps/wordpress-app-mariadb 1/1 162m </code></pre> <h3>Access WordPress application</h3> <p>You can expose the <em>WordPress</em> application using <em>Tailscale</em>.</p> <ol> <li>Create an <em>Ingress</em> YAML manifest.</li> </ol> <pre><code class="language-shell">$ cat ingress-wordpress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-wordpress namespace: wordpress annotations: tailscale.com/funnel: "true" spec: defaultBackend: service: name: wordpress-app port: number: 443 ingressClassName: tailscale tls: - hosts: - wordpress </code></pre> <ol start="2"> <li>Apply the <em>Ingress</em> to the namespace <em>wordpress</em>.</li> </ol> <pre><code class="language-shell">$ kubectl apply -f ingress-wordpress.yaml ingress.networking.k8s.io/ingress-wordpress created </code></pre> <ol start="3"> <li>Verify that the <em>Ingress</em> resource <em>'ingress-wordpress'</em> has been created and assigned an address, e.g., <em>'wordpress.qilin-beta.ts.net'</em>.</li> </ol> <pre><code class="language-shell">$ kubectl get ingress -n wordpress NAME CLASS HOSTS ADDRESS PORTS AGE ingress-wordpress tailscale * wordpress.qilin-beta.ts.net 80, 443 7s </code></pre> <ol start="4"> <li>In the <em>Tailscale</em> admin console, a new machine named <em>'wordpress'</em> should appear under the <em>Machines</em> tab.</li> </ol> <p><img src="/img/tailscale-wordpress.png" alt=""></p> <p>You can now access the <em>WordPress</em> application by opening its <em>Tailscale Funnel</em> URL in a browser.</p> <p><img src="/img/wordpress-ui.png" alt=""></p> <h3>Conclusion</h3> <p>This blog post provided a detailed walkthrough of how to automate application deployments to MKS clusters using a GitOps workflow with <em>Argo CD</em>. HPE Private Cloud Enterprise was selected as the example environment for this demonstration. It covered the process of deploying <em>Argo CD</em> through a Morpheus task executed on the MKS master node in the cluster, configuring application deployments via a publicly accessible <em>Argo CD</em> endpoint exposed with <em>Tailscale</em>, and monitoring the real-time state of applications running in the MKS cluster.</p> <p>It's important to remember that while <em>Argo CD</em> manages continuous delivery (CD), it still relies on a separate continuous integration (CI) pipeline. The CI pipeline is responsible for testing and building the application whenever developers update the source code. During this phase, the application is validated, container images are built and pushed to an image registry, and the CI system can update the configuration repository, often a separate repository from the application's source code and connected to <em>Argo CD</em>. These updates then trigger <em>Argo CD</em> to synchronize the desired state with the cluster. This workflow represents a common GitOps pattern used across many organizations. By integrating seamlessly with existing CI/CD systems, <em>Argo CD</em> helps teams implement GitOps practices effectively.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/streamlining-server-management-bulk-onboarding-to-hpe-compute-ops-management-with-ilorest-1/https://developer.hpe.com/streamlining-server-management-bulk-onboarding-to-hpe-compute-ops-management-with-ilorest-1/Fri, 09 Jan 2026 08:04:01 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/firmware_updates/part1/firmware_update_part1">Server Management portal</a>.</big></p> <br>https://developer.hpe.com/monitoring-hpe-greenlake-servers-running-gpus-using-grafana-cloud/https://developer.hpe.com/monitoring-hpe-greenlake-servers-running-gpus-using-grafana-cloud/Fri, 09 Jan 2026 06:36:37 GMT<p>External blog</p>https://developer.hpe.com/2026-january-07/https://developer.hpe.com/2026-january-07/Wed, 07 Jan 2026 13:00:00 GMThttps://developer.hpe.com/announcing-chapel-2-7/https://developer.hpe.com/announcing-chapel-2-7/Thu, 18 Dec 2025 22:17:48 GMT<p>external blog</p>https://developer.hpe.com/hpe-morpheus-enterprise-case-study-fix-network-pool-links-via-a-custom-task-plugin/https://developer.hpe.com/hpe-morpheus-enterprise-case-study-fix-network-pool-links-via-a-custom-task-plugin/Thu, 18 Dec 2025 11:45:51 GMT<p><img src="" alt=""></p> <p>A previous <a href="https://developer.hpe.com/blog/morpheus-plugin-tutorial-how-to-build-and-compile/">blog</a> post explored the basics around building and compiling HPE Morpheus Enterprise Plugins. This post expands on the subject by implementing a minimal Custom Task Plugin. The logic will target IP Pools, where links between the host entries and the VM workloads are broken or missing.</p> <h2>Problem statement</h2> <p>When IP Pools are added to Networks, or changed, after Instance provisioning, where a Network Interface falls within the same <em><strong>Network</strong></em>/<em><strong>NetworkPool</strong></em>, the <em><strong>NetworkPoolIp</strong></em> entry is not linked to a <em><strong>ComputeServer</strong></em> object.</p> <p>When a server is provisioned to a Network where an IP Pool is attached, reference links are created via the <em><strong>refType</strong></em> and <em><strong>refId</strong></em> properties of the <em><strong>NetworkPoolIp</strong></em> object, as shown below:</p> <p><img src="/img/morphblog_linknetworks_reftype_refid_small.png" alt="refType and refId relationship"></p> <p>Upon provisioning, the <em><strong>refType</strong></em> property is set to the literal value of 'ComputeServer' and the <em><strong>refId</strong></em> property is set to the <em><strong>Id</strong></em> of the <em><strong>ComputeServer</strong></em> object itself.</p> <p>When a <em><strong>NetworkPool</strong></em> is migrated/changed, or added to the <em><strong>Network</strong></em> after Instance provisioning, <em><strong>NetworkPoolIp</strong></em> records are created or synchronized from IPAM, without the <em><strong>refId</strong></em> link populated. This causes orphan host entries when workloads are later deleted.</p> <p>To reproduce this, provision a single VM Instance into a simple lab, then add the IP Pool to the related Network, afterward.</p> <h2>Normal behavior</h2> <p>First, consider the normal day-to-day use case. The Network is associated with the IP Pool as shown below:</p> <p><img src="/img/morphblog_linknetwork_network_with_pool.png" alt="Network with IP Pool"></p> <p>When provisioning a VM Instance into this Network, a Host Record entry is created in the IP Pool:</p> <p><img src="/img/morphblog_linknetwork_instance_provisioned.png" alt="Instance provisioned"></p> <p><img src="/img/morphblog_linknetwork_host_record.png" alt="Host record created"></p> <p>Querying the Host Record via the REST API shows the link back to the <em><strong>ComputeServer</strong></em> within the Instance:</p> <p><img src="/img/morphblog_linknetwork_api_host_record.png" alt="Host record via REST API"></p> <h2>Provision without IP Pool</h2> <p>To illustrate the broken reference issue, remove the Network-to-IP-Pool association:</p> <p><img src="/img/morphblog_linknetwork_network_without_pool.png" alt="Network without IP Pool"></p> <p>Then provision a VM Instance to the Network:</p> <p><img src="/img/morphblog_linknetwork_provision_vm.png" alt="Provision VM to Network"></p> <p><img src="/img/morphblog_linknetwork_provisioned_vm.png" alt="Provisioned VM to Network"></p> <p>Next, create a Host Record entry manually within the IP Pool:</p> <p><img src="/img/morphblog_linknetworks_manual_host_record.png" alt="New Pool IP"></p> <p>Finally, add the IP Pool back onto the Network:</p> <p><img src="/img/morphblog_linknetwork_network_pool_readded.png" alt="Network Pool re-added onto Network"></p> <p>This time, the REST API response shows that the <em><strong>refId</strong></em> property is NOT populated on the <em><strong>ComputeServer</strong></em>, despite the matching hostname on the VM:</p> <p><img src="/img/morphblog_linknetwork_api_host_record_nolink.png" alt="API host record with no link"></p> <h2>Steps to compile and upload the Custom Task Plugin</h2> <p>Download or clone the Plugin repository from <a href="https://github.com/neilvrhpe/link-network-hosts">https://github.com/neilvrhpe/link-network-hosts</a>. Open the project directory and compile with the relevant <em><strong>gradlew</strong></em> (Linux) or <em><strong>gradlew.bat</strong></em> (Windows) script using the <em><strong>shadowJar</strong></em> argument:</p> <p><img src="/img/morphblog_linknetworks_compile_jar.png" alt="Compile the Plugin jar file"></p> <p>The compiled <em><strong>.jar</strong></em> file will be found in the <em><strong>build/libs</strong></em> subdirectory:</p> <p><img src="/img/morphblog_linknetworks_compiled_jar.png" alt="Compiled jar file in build/libs subdirectory"></p> <p>Upload the <em><strong>.jar</strong></em> file to the <em><strong>Administration > Integrations > Plugins > Add</strong></em> dialog. The Plugin should appear in the list as shown below:</p> <p><img src="/img/morphblog_linknetworks_Plugin_uploaded.png" alt="Plugin uploaded and shown in list"></p> <h2>Setup the Custom Task Workflow</h2> <p>Uploading the Plugin in the previous step introduces a new <em><strong>TaskType</strong></em> to the HPE Morpheus Enterprise appliance. This can be seen under the edit dialog of the uploaded Plugin:</p> <p><img src="/img/morphblog_linknetworks_new_tasktype.png" alt="New Task Type introduced"></p> <p>To use this new Custom Task Type in the UI, provide an <em><strong>OptionSource</strong></em> Input. Create the corresponding <em><strong>OptionList</strong></em> under <em><strong>Library > Options > Option Lists > Add</strong></em>. The type is <em><strong>Plugin</strong></em> and the Option List will be <em><strong>Link Network Hosts: getNetworkPools</strong></em>, as provided by the Plugin that was uploaded in the previous step. Provide 'ChooseNetworkPool' as the name, so that it matches the name in the next step:</p> <p><img src="/img/morphblog_linknetworks_create_optionsource.png" alt="Create the Option Source"></p> <p>Next, create the <em><strong>Input</strong></em> that represents the <em><strong>OptionList</strong></em> entries to the end user dropdown in the UI. This is done using the <em><strong>Library > Options > Inputs > Add</strong></em> button. Then provide 'networkPool' as the <em><strong>Field Name</strong></em> for the Custom Task to reference in the next step. Choose <em><strong>Select List</strong></em> as the <em><strong>Type</strong></em> and use <em><strong>ChooseNetworkPool</strong></em> as the <em><strong>Option List</strong></em> field value:</p> <p><img src="/img/morphblog_linknetworks_form_input.png" alt="Create the form input for the Option Source"></p> <p>To set up the Task, navigate to <em><strong>Library > Automation > Tasks > Add</strong></em>. Provide a Task <em><strong>Name</strong></em> and a <em><strong>Network Pool Id</strong></em> value of <em><strong>&#x3C;%=customOptions.networkPool%></strong></em>. This reference will insert the value from the Input created in the previous step.</p> <p><img src="/img/morphblog_linknetworks_task_setup.png" alt="Custom Task setup"></p> <h3>MATCH FULL FQDN</h3> <p>Checking this box will include the full domain name suffixed to the hostname in the match comparison. Unchecked, only the hostname portion is matched. Matching is case insensitive.</p> <h3>REPLACE EXISTING LINKS</h3> <p>Checking this box will overwrite any existing <em><strong>refId</strong></em> links when the hostname matches.</p> <p>Next, create an <em><strong>Operational Workflow</strong></em> to run the Task with the correct Input context (Pool ID). Create the Workflow under <em><strong>Library > Automation > Add > Operational Workflow</strong></em>. Provide a name for the Workflow, add the <em><strong>Task from the previous step</strong></em> under <em><strong>Tasks</strong></em> and add the <em><strong>ChooseNetworkPool</strong></em> Input under <em><strong>Inputs</strong></em>.</p> <p><img src="/img/morphblog_linknetworks_workflow_setup.png" alt="Operational Workflow setup"></p> <h2>Run the Workflow against the Network Pool</h2> <p>Review the REST API call to confirm that the host record <em><strong>refId</strong></em> property is not currently set against the provisioned <em><strong>ComputeServer</strong></em>:</p> <p><img src="/img/morphblog_linknetworks_link_ref_missing.png" alt="Confirm missing refId link to ComputeServer"></p> <p>Under <em><strong>Library > Automation > Workflows</strong></em>, click the name of the Workflow created in the previous step to view the Workflow details:</p> <p><img src="/img/morphblog_linknetworks_workflow_details.png" alt="View Workflow details"></p> <p>The <em><strong>Execute</strong></em> button brings up the Workflow execution dialog. Select the IP Pool that the test VM Instance is deployed to. The <em><strong>Execution Config/Context</strong></em> can be ignored, as this Task will always run on the local HPE Morpheus Enterprise appliance in its own context:</p> <p><img src="/img/morphblog_linknetworks_workflow_execute.png" alt="Execute the Workflow"></p> <p>Under the <em><strong>Executions</strong></em> tab, view the output of the Task, showing which <em><strong>ComputeServer</strong></em> objects have been allocated to Host Records within the IP Pool, via the <em><strong>refId</strong></em> property:</p> <p><img src="/img/morphblog_linknetworks_workflow_executed.png" alt="Workflow execution results"></p> <p>Re-running the REST API call confirms that the <em><strong>refType</strong></em> and <em><strong>refId</strong></em> link was created:</p> <p><img src="/img/morphblog_linknetwork_api_host_record_linked.png" alt="RefType and refId link created"></p> <h2>Workflow via the API</h2> <p>In large environments it would be impractical to execute the Workflow for each IP Pool by hand in the UI. For these scenarios, execute the Workflow via the REST API. Provide the <em><strong>id</strong></em> of the <em><strong>Workflow</strong></em> in the request URL and the <em><strong>id</strong></em> of the IP Pool to the <em><strong>networkPool</strong></em> body parameter to execute the POST request:</p> <p><img src="/img/morphblog_linknetwork_execute_via_api.png" alt="Execute Workflow via API"></p> <h2>Next steps</h2> <p>From here, it would make sense to explore the unused possibilities around Custom Tasks. Whereas this example only ever runs on the local HPE Morpheus Enterprise appliance, Tasks can run on different targets (remote, resource) and against different contexts (instances, servers).</p> <p>In essence, this post only explores a very particular use case to potentially assist in IPAM migrations and day 2 IPAM adoption.</p> <p>At the more advanced end of the spectrum are Provider Types that model core infrastructure components. These include integrations for Clouds, Networks, Storage systems, and many others. Such Providers tend to be more complex because they interact deeply with HPE Morpheus Enterprise’s provisioning, synchronization, and lifecycle management layers. Understanding how these Provider Types fit together is key to building powerful, production-grade Plugins.</p> <p>Explore the following resources for more information on the different Plugin/Provider types:</p> <p><a href="https://developer.morpheusdata.com">https://developer.morpheusdata.com</a></p> <p><a href="https://share.morpheusdata.com">https://share.morpheusdata.com</a> (follow the repository link under the Plugin details to see the source code of a Plugin)</p> <p><a href="https://github.com/hewlettpackard">https://github.com/hewlettpackard</a> <a href="https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX">https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX</a></p>https://developer.hpe.com/transformers-from-scratch-in-chapel-and-c/https://developer.hpe.com/transformers-from-scratch-in-chapel-and-c/Sat, 13 Dec 2025 03:07:14 GMT<p>external blog</p>https://developer.hpe.com/2025-december-08/https://developer.hpe.com/2025-december-08/Mon, 08 Dec 2025 13:00:00 GMThttps://developer.hpe.com/reflections-on-chapelcon-25/https://developer.hpe.com/reflections-on-chapelcon-25/Fri, 05 Dec 2025 20:07:37 GMT<p>external blog</p>https://developer.hpe.com/bringing-ai-assistants-to-hpe-greenlake-with-mcp-servers/https://developer.hpe.com/bringing-ai-assistants-to-hpe-greenlake-with-mcp-servers/Wed, 03 Dec 2025 18:40:27 GMT<p><img src="" alt=""></p> <h1>Overview</h1> <p>Modern cloud platforms generate vast amounts of operational data. GreenLake customers often find themselves toggling between multiple interfaces, running complex API queries, and piecing together information from various services to understand their infrastructure state. What if your AI assistant could directly interact with your GreenLake environment in a secure, controlled manner?</p> <p>The Model Context Protocol (MCP) makes this possible by providing a standardized way for AI assistants to access external data sources and tools. This article explores how MCP servers bring intelligent, conversational access to GreenLake APIs while maintaining strict security boundaries.</p> <h2>The Challenge: Bridging AI and Enterprise APIs</h2> <p>AI assistants have transformed how we work with information, but they face a fundamental limitation when it comes to enterprise platforms: they don't have direct access to your live operational data. When you ask questions about your GreenLake environment, the AI can only provide general guidance, not specific insights based on your actual workspaces, devices, or audit logs.</p> <p>Traditional approaches to this problem involve building custom integration layers, managing authentication flows, and maintaining bespoke code for each API endpoint. This creates several challenges:</p> <ul> <li><strong>Development overhead</strong>: Each integration requires custom code and ongoing maintenance</li> <li><strong>Security concerns</strong>: Storing credentials and managing access control adds complexity</li> <li><strong>Limited reusability</strong>: Custom integrations rarely work across different tools or AI platforms</li> <li><strong>Slow iteration</strong>: Adding new capabilities means writing more integration code</li> </ul> <h2>Introducing the Model Context Protocol</h2> <p>The <a href="https://modelcontextprotocol.io/">Model Context Protocol</a> is an open standard that enables AI applications to securely connect to external data sources and tools. Think of it as a universal adapter that allows AI assistants to "plug in" to your enterprise systems in a controlled, standardized way.</p> <p>MCP operates on a client-server architecture:</p> <p><img src="/img/mcp-architecture.png" alt="mcp architecture" title="mcp architecture"></p> <p>The protocol defines three core primitives:</p> <ol> <li><strong>Resources</strong>: Data sources that can be read (configuration files, API responses, documents)</li> <li><strong>Tools</strong>: Functions that can be invoked (API calls, calculations, data transformations)</li> <li><strong>Prompts</strong>: Reusable templates for common operations</li> </ol> <p>For GreenLake integration, MCP servers primarily expose <strong>tools</strong> that map to API endpoints, allowing AI assistants to query workspaces, retrieve device information, search audit logs, and more.</p> <h2>Why MCP Servers Are a Natural Fit for GreenLake?</h2> <p>MCP servers bring a set of benefits that align well with GreenLake’s secure, enterprise-scale architecture. At a high level, they provide local execution, per-user credentials, and a standardized interface for AI tools to safely interact with GreenLake APIs.</p> <h3>Local Execution, Complete Control</h3> <p>An MCP server runs entirely on your machine. It doesn’t open network ports or send your credentials to an external service. Instead, it acts as a controlled bridge between your AI assistant and the GreenLake APIs.</p> <p>This design provides:</p> <ul> <li><strong>Data stays local</strong>: API responses are processed on your device before the AI sees them.</li> <li><strong>Credential isolation</strong>: OAuth secrets never leave your environment.</li> <li><strong>Read-only safety</strong>: Only GET operations are exposed, preventing accidental changes.</li> <li><strong>Firewall friendly behavior</strong>: Only outbound HTTPS requests are sent to GreenLake APIs.</li> </ul> <p><em>Takeaway: MCP servers give AI assistants visibility into your GreenLake environment without sacrificing security, privacy, or control.</em></p> <h3>Personal API Client Pattern</h3> <p>MCP servers use your personal API credentials, similar to how you would interact with GreenLake through the CLI or web console. This means:</p> <ul> <li><strong>Workspace scoping</strong>: You only access resources within your authorized workspaces</li> <li><strong>Role-based access</strong>: Your existing RBAC permissions apply to all API operations</li> <li><strong>Individual accountability</strong>: All actions are attributed to your user account</li> <li><strong>No shared secrets</strong>: Each user maintains their own credentials</li> </ul> <h3>Standardized Tool Interface</h3> <p>Once configured, the MCP server exposes GreenLake APIs as standardized tools that any MCP-compatible AI assistant can use. This provides:</p> <ul> <li><strong>Consistent experience</strong>: The same tools work in Claude Desktop, VS Code, and other MCP clients</li> <li><strong>Discoverable capabilities</strong>: AI assistants automatically learn available operations</li> <li><strong>Type-safe parameters</strong>: Input validation happens before API calls</li> <li><strong>Structured responses</strong>: API data is returned in predictable formats</li> </ul> <h3>Operating Modes: Static vs. Dynamic</h3> <p>MCP servers support two operating modes depending on the size and complexity of the API surface.</p> <h4>Static Mode</h4> <ul> <li><strong>Best for</strong>: APIs with &#x3C;50 endpoints</li> <li><strong>Tool Model</strong>: One tool per endpoint</li> <li><strong>Validation</strong>: Compile-time, type-safe</li> <li><strong>Performance</strong>: Fast execution</li> <li><strong>Ideal Use</strong>: Smaller, well-defined API sets</li> </ul> <h4>Dynamic Mode</h4> <ul> <li><strong>Best for</strong>: APIs with 50+ endpoints</li> <li><strong>Tool Model</strong>: Three meta-tools handle everything</li> <li><strong>Validation</strong>: Runtime schema validation</li> <li><strong>Performance</strong>: Lower memory footprint</li> <li><strong>Ideal Use</strong>: Large, evolving API sets</li> </ul> <p>Both modes expose the same capabilities — the difference lies in how efficiently the server manages larger API collections.</p> <h2>Security Architecture</h2> <p>Security is paramount when connecting AI assistants to production infrastructure. MCP servers for GreenLake implement defense-in-depth principles.</p> <h3>Authentication and Authorization</h3> <p><img src="/img/authn-seq-diagram.png" alt="authn authz sequence diagram" title="Authentication - sequence diagram"></p> <p>The MCP server handles OAuth2 authentication using the client credentials flow, automatically managing token lifecycle and refresh operations. Your credentials are read from local environment variables or configuration files, never transmitted to the AI service.</p> <h3>Read-Only Operations</h3> <p>MCP servers for GreenLake deliberately expose only GET operations from the OpenAPI specifications. This design choice ensures no unintended modifications, as write options are excluded, allowing users to query extensively without risk of changes. This design choice reduces the blast radius, so even if misconfigured, the server cannot modify infrastructure.</p> <h3>Network Isolation</h3> <p>The MCP server communicates with AI assistants via standard input/output (stdio) rather than network protocols. This means:</p> <ul> <li><strong>No listening ports</strong>: The server doesn't expose network services</li> <li><strong>Process-level isolation</strong>: Communication happens through OS process pipes</li> <li><strong>No remote access</strong>: The server cannot be accessed from other machines</li> <li><strong>Firewall friendly</strong>: Only outbound HTTPS to GreenLake APIs</li> </ul> <h3>Credential Management</h3> <p>Following security best practices, MCP servers support multiple credential sources in priority order:</p> <ol> <li>Environment variables (recommended for development)</li> <li>Local configuration files with restricted permissions</li> <li>Secure credential stores (platform keychain integration)</li> </ol> <p>Credentials are never logged, and API responses are sanitized to remove sensitive tokens before being shared with the AI assistant.</p> <h2>Real-World Use Case</h2> <p>MCP servers unlock powerful workflows by combining AI reasoning with live GreenLake data:</p> <h3>Audit Log Analysis</h3> <p>Investigating security events becomes conversational: "Show me all failed login attempts from the last 24 hours and identify patterns."</p> <p>The MCP server enables:</p> <ol> <li>Querying audit logs with time-based filters</li> <li>Natural language parsing of filter syntax</li> <li>Pattern recognition across multiple log entries</li> <li>Correlation with workspace and user data</li> <li>Summary reports with actionable recommendations</li> </ol> <h3>Intelligent Device Inventory Management</h3> <p>Managing thousands of devices across distributed infrastructure becomes conversational: "Show me all unassigned compute devices in the production environment". "Which devices haven't been updated in 30 days and are approaching warranty expiration?" "Generate an inventory report showing all PCI-compliant devices with their support levels."</p> <p>The MCP server enables:</p> <ol> <li>Querying devices with complex filter expressions</li> <li>Natural language parsing of device attributes</li> <li>Pattern recognition across device types and states</li> <li>Automated capacity planning and forecasting</li> </ol> <h3>Subscriptions</h3> <p>MCP servers unlock powerful workflows by combining AI reasoning with live GreenLake data:</p> <p>"Show me all subscriptions expiring in the next 90 days and their renewal status." "Which subscriptions have less than 20% utilization and could be right-sized?"</p> <p>The <strong>Subscriptions</strong> MCP server enables comprehensive subscription analytics. You can query subscriptions with time-based filters, monitor expirations, and verify licenses. The <strong>Subscriptions</strong> MCP server can identify underutilized resources and forecast renewal costs. This transforms subscription management from a reactive process into strategic planning conversations with your data.</p> <h3>User Management</h3> <p>Security teams identify access risks conversationally:</p> <p>"Show me all active users who haven't logged in for 180 days and should be reviewed for deactivation." "How many new users were onboarded last month and what's their login activity?"</p> <p>These capabilities support comprehensive user lifecycle management. The <strong>Users</strong> MCP server can detect dormant accounts, analyze user growth trends to inform capacity planning, track onboarding metrics, and perform month-over-month comparisons to identify usage patterns.</p> <h2>Getting Started</h2> <p>Setting up an MCP server for GreenLake requires three steps: obtaining API credentials, configuring the server, and connecting your AI assistant.</p> <h3>Prerequisites</h3> <ul> <li>GreenLake workspace with API access</li> <li>Python 3.10 or higher</li> <li>An MCP-compatible AI client (Claude Desktop, VS Code with Claude Code extension)</li> </ul> <h3>Step 1: Obtain API Credentials</h3> <p>From the GreenLake console:</p> <ol> <li>Navigate to <strong>Settings</strong> > <strong>API Clients</strong></li> <li>Create a new API client with appropriate scopes</li> <li>Note the <strong>Client ID</strong>, <strong>Client Secret</strong>, and <strong>Workspace ID</strong></li> </ol> <h3>Step 2: Generate and Configure the MCP Server</h3> <p>For this example, we'll set up an audit logs MCP server. The MCP generator tool (used internally by HPE to accelerate development) creates production-ready servers from OpenAPI specifications, though the generated servers can be deployed and configured independently.</p> <pre><code class="language-bash"># Clone or download the pre-generated MCP server git clone https://github.com/HewlettPackard/gl-mcp cd src/audit-logs # Install dependencies using uv (fast Python package manager) uv sync # Configure environment variables cat > .env.local &#x3C;&#x3C; EOF GREENLAKE_API_BASE_URL=https://global.api.greenlake.hpe.com GREENLAKE_CLIENT_ID=your-client-id GREENLAKE_CLIENT_SECRET=your-client-secret GREENLAKE_WORKSPACE_ID=your-workspace-id MCP_TOOL_MODE=static GREENLAKE_LOG_LEVEL=INFO EOF # Test the server make test </code></pre> <h3>Step 3: Connect Your AI Assistant</h3> <p>For Claude Desktop, add the server to your configuration file (<code>~/Library/Application Support/Claude/claude_desktop_config.json</code> on macOS). See the following code sample:</p> <pre><code class="language-json">{ "mcpServers": { "greenlake-audit-logs": { "command": "uv", "args": ["run", "python", "__main__.py"], "cwd": "/path/to/greenlake-mcp-servers/audit-logs", "env": { "GREENLAKE_API_BASE_URL": "https://global.api.greenlake.hpe.com", "GREENLAKE_CLIENT_ID": "your-client-id", "GREENLAKE_CLIENT_SECRET": "your-client-secret", "GREENLAKE_WORKSPACE_ID": "your-workspace-id", "MCP_TOOL_MODE": "static" } } } } </code></pre> <p>For VS Code with the Claude Code extension, create or update <code>.vscode/mcp.json</code> in your workspace. See the following code sample:</p> <pre><code class="language-json">{ "servers": { "greenlake-audit-logs": { "type": "stdio", "command": "uv", "args": ["run", "python", "__main__.py"], "cwd": "/path/to/greenlake-mcp-servers/audit-logs", "env": { "GREENLAKE_API_BASE_URL": "https://global.api.greenlake.hpe.com", "GREENLAKE_CLIENT_ID": "your-client-id", "GREENLAKE_CLIENT_SECRET": "your-client-secret", "GREENLAKE_WORKSPACE_ID": "your-workspace-id" } } } } </code></pre> <p>Restart your AI client, and the MCP server will automatically connect.</p> <h3>Example Interaction</h3> <p>Once configured, you can interact naturally:</p> <p><strong>You</strong>: "Show me the 10 most recent audit log entries"</p> <p><strong>Claude</strong> (using the <strong>getAuditLogs</strong> tool with <strong>limit=10</strong>):</p> <pre><code class="language-shell">Here are the 10 most recent audit log entries: 1. [2025-01-15 14:32:15] User Management - User admin@example.com logged in 2. [2025-01-15 14:28:43] Device Management - Device server-01 status changed to active 3. [2025-01-15 14:15:22] Workspace - Workspace Production updated ... </code></pre> <p>The AI automatically invokes the appropriate MCP tool, formats parameters, makes the API call through the server, and presents results in a readable format.</p> <h2>Architecture Deep Dive</h2> <p>Understanding the internal architecture helps appreciate how MCP servers maintain security while providing powerful capabilities.</p> <h3>Component Overview</h3> <p><img src="/img/component-overview.png" alt="component overview" title="Component Overview"></p> <h3>Request Flow</h3> <ol> <li><strong>AI Request</strong>: The AI assistant formulates a tool invocation (e.g., "get audit logs with category 'User Management'")</li> <li><strong>MCP Protocol</strong>: The request is serialized as JSON-RPC and sent via stdio to the MCP server</li> <li><strong>Tool Resolution</strong>: The server's tool registry identifies the corresponding tool implementation</li> <li><strong>Parameter Validation</strong>: Input parameters are validated against the tool's schema</li> <li><strong>Authentication Check</strong>: The auth manager verifies token validity and freshness</li> <li><strong>API Call</strong>: The HTTP client constructs and sends the HTTPS request to GreenLake</li> <li><strong>Response Processing</strong>: The raw API response is parsed and formatted</li> <li><strong>MCP Response</strong>: Structured data is returned to the AI via JSON-RPC</li> <li><strong>AI Presentation</strong>: The assistant formats and presents results to the user</li> </ol> <h3>Tool Implementation</h3> <p>Each MCP tool implements a consistent interface:</p> <pre><code class="language-python">class GetAuditLogsTool(BaseTool): """Tool for querying GreenLake audit logs.""" name = "getAuditLogs" description = "Retrieve audit logs with optional filtering" input_schema = { "type": "object", "properties": { "filter": {"type": "string", "description": "OData filter expression"}, "limit": {"type": "integer", "default": 100}, "offset": {"type": "integer", "default": 0} } } async def execute(self, filter: str = None, limit: int = 100, offset: int = 0): """Execute the audit log query.""" # Parameter validation params = {"limit": limit, "offset": offset} if filter: params["filter"] = filter # Make authenticated API call response = await self.http_client.get("/audit-log/v1/logs", params=params) # Return structured data return response </code></pre> <p>This abstraction allows the MCP server to expose dozens of API endpoints with minimal code duplication.</p> <h3>Dynamic Mode Optimization</h3> <p>For large APIs, dynamic mode provides significant performance benefits. Instead of individual tool files for each endpoint, three meta-tools handle all operations:</p> <p><strong>1. list_endpoints</strong> - Fast discovery</p> <pre><code class="language-python"># Returns: ["GET /audit-logs/v1/logs", "GET /devices/v1/servers", ...] # Allows AI to browse available operations </code></pre> <p><strong>2. get_endpoint_schema</strong> - On-demand schema loading</p> <pre><code class="language-python"># Input: "GET /audit-logs/v1/logs" # Returns: Full parameter schema, response types, descriptions # AI learns how to use an endpoint only when needed </code></pre> <p><strong>3. invoke_dynamic_tool</strong> - Validated execution</p> <pre><code class="language-python"># Input: endpoint, parameters # Validates parameters against schema # Makes API call # Returns structured response </code></pre> <p>This architecture scales efficiently to APIs with hundreds of endpoints without overwhelming the AI's context window.</p> <h2>What's Next?</h2> <p>We are actively working on extending our GreenLake MCP Servers capabilities. Stay tuned for future updates.</p> <h2>Conclusion</h2> <p>MCP servers bridge the gap between AI assistants and enterprise APIs, enabling natural language interaction with GreenLake without sacrificing security or control. By running locally and using personal API credentials, they provide a secure, auditable way to extend AI capabilities into your infrastructure management workflows.</p> <p>The read-only nature of these servers makes them ideal for exploration, reporting, and analysis tasks. Combined with the power of large language models, they transform how teams interact with cloud platforms, shifting from manual API navigation to conversational queries and automated insights.</p> <p>Whether you're investigating audit logs, documenting infrastructure, or generating compliance reports, MCP servers make GreenLake data accessible where you need it: in your AI-powered development environment.</p>https://developer.hpe.com/hpe-morpheus-enterprise-software-community-edition-available-now/https://developer.hpe.com/hpe-morpheus-enterprise-software-community-edition-available-now/Tue, 02 Dec 2025 09:29:28 GMT<!--\\\[if gte mso 9]><xml> <o:OfficeDocumentSettings> <o:AllowPNG/> </o:OfficeDocumentSettings> </xml><!\\\[endif]--> <!--\\\[if gte mso 9]><xml> <w:WordDocument> <w:View>Normal</w:View> <w:Zoom>0</w:Zoom> <w:TrackMoves/> <w:TrackFormatting/> <w:PunctuationKerning/> <w:ValidateAgainstSchemas/> <w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid> <w:IgnoreMixedContent>false</w:IgnoreMixedContent> <w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText> <w:DoNotPromoteQF/> <w:LidThemeOther>EN-US</w:LidThemeOther> <w:LidThemeAsian>X-NONE</w:LidThemeAsian> <w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript> <w:Compatibility> <w:BreakWrappedTables/> <w:SnapToGridInCell/> <w:WrapTextWithPunct/> <w:UseAsianBreakRules/> <w:DontGrowAutofit/> <w:SplitPgBreakAndParaMark/> <w:EnableOpenTypeKerning/> <w:DontFlipMirrorIndents/> <w:OverrideTableStyleHps/> </w:Compatibility> <m:mathPr> <m:mathFont m:val="Cambria Math"/> <m:brkBin m:val="before"/> <m:brkBinSub m:val="&#45;-"/> <m:smallFrac m:val="off"/> <m:dispDef/> <m:lMargin m:val="0"/> <m:rMargin m:val="0"/> <m:defJc m:val="centerGroup"/> <m:wrapIndent m:val="1440"/> <m:intLim m:val="subSup"/> <m:naryLim m:val="undOvr"/> </m:mathPr></w:WordDocument> </xml><!\\\[endif]--> <!--\\\[if gte mso 9]><xml> <w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false" DefSemiHidden="false" DefQFormat="false" DefPriority="99" LatentStyleCount="376"> <w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/> <w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 2"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 3"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 4"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 5"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 6"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 7"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 8"/> <w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 9"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 6"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 7"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 8"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 9"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 1"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 2"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 3"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 4"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 5"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 6"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 7"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 8"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 9"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal Indent"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footnote text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="header"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footer"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index heading"/> <w:LsdException Locked="false" Priority="35" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="caption"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="table of figures"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="envelope address"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="envelope return"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footnote reference"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation reference"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="line number"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="page number"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="endnote reference"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="endnote text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="table of authorities"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="macro"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="toa heading"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 5"/> <w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Closing"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Signature"/> <w:LsdException Locked="false" Priority="1" SemiHidden="true" UnhideWhenUsed="true" Name="Default Paragraph Font"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Message Header"/> <w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Salutation"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Date"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text First Indent"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text First Indent 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Note Heading"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Block Text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Hyperlink"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="FollowedHyperlink"/> <w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/> <w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Document Map"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Plain Text"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="E-mail Signature"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Top of Form"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Bottom of Form"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal (Web)"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Acronym"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Address"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Cite"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Code"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Definition"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Keyboard"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Preformatted"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Sample"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Typewriter"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Variable"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal Table"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation subject"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="No List"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 6"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 7"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 8"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 4"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 5"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 6"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 7"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 8"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Contemporary"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Elegant"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Professional"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Subtle 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Subtle 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 1"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 2"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 3"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Balloon Text"/> <w:LsdException Locked="false" Priority="39" Name="Table Grid"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Theme"/> <w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/> <w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading"/> <w:LsdException Locked="false" Priority="61" Name="Light List"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/> <w:LsdException Locked="false" Priority="70" Name="Dark List"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/> <w:LsdException Locked="false" SemiHidden="true" Name="Revision"/> <w:LsdException Locked="false" Priority="34" QFormat="true" Name="List Paragraph"/> <w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/> <w:LsdException Locked="false" Priority="30" QFormat="true" Name="Intense Quote"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/> <w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/> <w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/> <w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/> <w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/> <w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/> <w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/> <w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/> <w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/> <w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/> <w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/> <w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/> <w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/> <w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/> <w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/> <w:LsdException Locked="false" Priority="19" QFormat="true" Name="Subtle Emphasis"/> <w:LsdException Locked="false" Priority="21" QFormat="true" Name="Intense Emphasis"/> <w:LsdException Locked="false" Priority="31" QFormat="true" Name="Subtle Reference"/> <w:LsdException Locked="false" Priority="32" QFormat="true" Name="Intense Reference"/> <w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/> <w:LsdException Locked="false" Priority="37" SemiHidden="true" UnhideWhenUsed="true" Name="Bibliography"/> <w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/> <w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/> <w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/> <w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/> <w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/> <w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/> <w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 1"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 1"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 1"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 2"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 2"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 2"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 3"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 3"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 3"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 4"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 4"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 4"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 5"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 5"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 5"/> <w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 6"/> <w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/> <w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/> <w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/> <w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/> <w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 6"/> <w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 6"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 1"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 1"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 1"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 2"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 2"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 2"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 3"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 3"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 3"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 4"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 4"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 4"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 5"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 5"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 5"/> <w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 6"/> <w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/> <w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/> <w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/> <w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/> <w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 6"/> <w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 6"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Mention"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Smart Hyperlink"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Hashtag"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Unresolved Mention"/> <w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Smart Link"/> </w:LatentStyles> </xml><!\\\[endif]--> <!--\\\[if gte mso 10]> <style> /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:8.0pt; mso-para-margin-left:0in; line-height:107%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} </style> <!\\\[endif]--> <!--StartFragment--> <p>I'm thrilled to announce the availability of the <strong>HPE Morpheus Enterprise Software Community Edition.</strong> HPE Morpheus Enterprise is a powerful and flexible solution designed to accelerate your multi-cloud and hybrid IT automation journey. Whether you’re just starting out or looking to optimize your existing cloud infrastructure, this is your chance to explore HPE Morpheus Enterprise and learn why <a href="https://www.hpe.com/psnow/doc/a00141838enw">ISG in its latest quadrant report named HPE Morpheus the leader in Hybrid Cloud Management Platform</a>. </p> <p><strong>What is HPE Morpheus Enterprise?</strong></p> <p>HPE Morpheus Enterprise is a cutting-edge software platform that empowers IT teams to unify management, streamline automation, and simplify operations across private and public clouds. It's built to help organizations modernize their infrastructure while improving agility and reducing complexity. With features tailored for VM lifecycle management, self-service provisioning, and policy-based governance, HPE Morpheus Enterprise is the ultimate tool for managing hybrid cloud environments. Here's <a href="http://www.hpe.com/morpheus">a link to the hpe.com page</a> where you can learn more by reading other content and watch videos. </p> <p><strong>Community Edition highlights</strong></p> <p>The HPE Morpheus Enterprise Software Community Edition includes all the features in HPE Morpheus VM Essentials, enabling you to:</p> <ul> <li>Deploy virtual machines quickly and efficiently through intuitive self-service workflows.</li> <li>Automate day-to-day VM operations with seamless provisioning and lifecycle management.</li> <li>Leverage policies for governance and compliance, ensuring consistent management across environments.</li> <li>Experiment with the platform at no cost and discover how it can transform your IT infrastructure.</li> </ul> <p>Whether you’re an IT or VM administrator, a DevOps engineer, or a cloud architect, the Community Edition is the perfect way to experience the power of HPE Morpheus Enterprise firsthand. However, note that the Community Edition is a non-commercial, "<em>personal use"</em> license. It has limited features and supplies a 3-socket license. If you are interested in testing Morpheus Enterprise in your environment, your HPE representative can assist you in finding ways to do that.</p> <p><strong>Get started today</strong></p> <p>Ready to dive in? The HPE Morpheus Enterprise Software Community Edition is available now, and I can’t wait to see how you use it to innovate and streamline your IT operations. Join the HPE Community and explore the possibilities with this exciting release! Send an email to <strong><a href="mailto:morpheus_communityedition@hpe.com">morpheus_communityedition@hpe.com</a></strong> with your request. If you are eligible, I'll send you an invite to get the license and download the binaries. </p> <!--EndFragment-->https://developer.hpe.com/configuring-saml-sso-authentication-with-hpe-greenlake-a-guide-for-the-top-3-identity-providers-and-passwordless-integration-for-hpecomcmdlets/https://developer.hpe.com/configuring-saml-sso-authentication-with-hpe-greenlake-a-guide-for-the-top-3-identity-providers-and-passwordless-integration-for-hpecomcmdlets/Wed, 19 Nov 2025 12:17:54 GMT<p>NONE</p>https://developer.hpe.com/morpheus-plugin-tutorial-how-to-build-and-compile/https://developer.hpe.com/morpheus-plugin-tutorial-how-to-build-and-compile/Wed, 19 Nov 2025 08:43:36 GMT<h2>Introduction</h2> <p>HPE Morpheus Enterprise is a hybrid cloud platform that unifies diverse products and technologies into a consistent workload-lifecycle orchestration, governance, and control framework.</p> <p>This makes HPE Morpheus Enterprise ideally positioned to integrate with a broad ecosystem of cloud-related service vendors. These integrations are enabled through technology-specific plugin providers. HPE Morpheus Enterprise is extendable with custom plugins for clouds, task types, UI tabs, reports, approvals, cypher, IPAM, backups and more.</p> <p>This article covers the process of generating and compiling a basic HPE Morpheus Enterprise generic plugin project on Windows 11. To understand how the workflow fits together, this blog will cover:</p> <ul> <li>Generating a new project using the plugin code generator</li> <li>Unzipping and opening the project in an IDE</li> <li>Exploring main plugin file components</li> <li>Compiling the plugin on Windows</li> <li>Uploading the compiled plugin to HPE Morpheus Enterprise</li> <li>Compiling the plugin remotely on Linux, using Visual Studio Code</li> <li>Compiling the plugin using Docker</li> </ul> <h2>JDK Prerequisite</h2> <p>For the labs in this document, we assume a <strong>Windows 11 host</strong> with internet access and <strong>Visual Studio Code</strong> installed.</p> <p>You’ll also need to have <strong>Java JDK 11 or 17</strong> installed. The vendor distribution of Java is not important — both OpenJDK and Oracle JDK are supported.</p> <p>When using JDK 17, the project’s compile <strong>compatibility level is set to version 1.11</strong> to maintain compatibility with earlier environments.</p> <ol> <li> <p>Open a Windows command prompt (Press Win + R or click Start, type cmd, press enter)</p> </li> <li> <p>To install OpenJDK 17, run the following command and click yes to provide administrative privileges where needed:</p> <pre><code class="language-bash">winget install jdkbuild.openjdk.17.jdk </code></pre> <p>To verify your OpenJDK install, run:</p> <pre><code class="language-bash">java -version </code></pre> </li> </ol> <p><img src="/img/0installjdk.png" alt="Installing and testing Java JDK" title="Install and test java"></p> <p>Another build dependency is the <strong>Gradle build system</strong>. This will automatically be provided by the <strong>Gradle wrapper configuration</strong> under the <strong>gradle directory</strong> of the project we will generate.</p> <h2>Creating a plugin project</h2> <p>Creating a project that compiles code into usable plugins can be a daunting task, especially for developers who are not familiar with Java, Groovy, or Gradle.</p> <p>To simplify this process and make it easier for potential plugin builders to get started, the HPE Morpheus Enterprise engineering team created the <strong>Morpheus Plugin Code Generator</strong>.</p> <p>We’ll use this handy tool as a starting point to create our new plugin project. The preferred source language for HPE Morpheus Enterprise plugins is <strong>Groovy</strong>.</p> <p>Groovy features a concise, flexible syntax and includes many helper methods that make coding easier. It’s fully interoperable with Java and compiles to the same JVM bytecode.</p> <ol> <li> <p>Using a web browser, navigate to <a href="https://developer.morpheusdata.com/">https://developer.morpheusdata.com/</a>. Click the Get Started Now button.</p> <p><img src="/img/1developer_getting_started_button.png" alt="Launch plugin code generator button" title="Launch plugin code generator"></p> </li> <li> <p>For this lab, we provide the following field values:</p> <ul> <li><strong>Name</strong>: Plugin Demo</li> <li><strong>Code</strong>: pluginDemo</li> <li><strong>Providers</strong>: Generic Integration</li> </ul> <p><img src="/img/2developer_generate_plugin.png" alt="Generate plugin project" title="Generate plugin project"></p> </li> <li> <p>Unzip the plugin project for use in an IDE. For this example, we will unzip the plugin to the Windows Documents folder.</p> <p><img src="/img/3unzip_plugin_project.png" alt="Extract code project" title="Extract code project"></p> </li> </ol> <h2>Authoring plugin projects in an IDE</h2> <p>Adding logic and complexity to a working plugin is an exercise in object-oriented programming. Writing code in plain text editors can be tedious, time-consuming, and error prone. To make development easier, we use an IDE such as Visual Studio Code.</p> <p>Although we use Visual Studio Code in this example, several more powerful Java/Groovy IDEs are available, including IntelliJ IDEA, Eclipse, and NetBeans.</p> <ol> <li> <p>Open <strong>Visual Studio Code</strong> and select <strong>Open Folder</strong> from the <strong>File</strong> menu.</p> <p><img src="/img/4vsc_open_folder.png" alt="Visual Studio Code Open Folder" title="VS Code Open Folder"></p> </li> <li> <p>Respond <strong>Yes</strong> to the trust prompt. If this is the first project opened of its type, VS Code will prompt for the installation of java related extension packs. Respond by clicking <strong>Install</strong>.</p> <p><img src="/img/5trust_and_install_extensions.png" alt="Trust and install extensions prompt" title="Trust and install extensions prompt"></p> </li> <li> <p>The extension pack will take a while to install and build/configure the project. This can be seen at the bottom left of the VS Code window.</p> </li> </ol> <p><img src="/img/61configuring.png" alt="Configuring the extension pack in Visual Studio Code" title="Extension pack setup"></p> <p>Wait for the <strong>Gradle: configure project</strong> message to disappear and the <strong>Java: Ready</strong> message to remain.</p> <p><img src="/img/62configured.png" alt="Project successfully configured in Visual Studio Code" title="Project build"></p> <ol> <li>Open the <strong>Explorer</strong> view by clicking the corresponding icon at the top left. The <strong>Welcome</strong> and any open <strong>Extension</strong> tabs can now be closed.</li> </ol> <p><img src="/img/7explorer_view.png" alt="Explorer view in Visual Studio Code" title="Open explorer view"></p> <h2>Important project files</h2> <p>We will explore some of the most important files briefly. In different article, we will cover the project structure and the build files in more detail.</p> <ul> <li><strong>gradlew.bat</strong><br> OS shell wrapper script used for compiling the plugin on Windows.</li> <li><strong>gradlew</strong><br> OS shell wrapper script used for compiling on Linux.</li> <li><strong>gradle.properties</strong><br> Variables used in the in the gradle build. Typically version numbers.</li> <li><strong>build.gradle</strong><br> This is the actual build script. Build dependencies are declared here. This configures how the .jar file is built.</li> <li><strong>PluginDemoPlugin.groovy</strong><br> i.The main plugin entry point class that HPE Morpheus Enterprise will load.<br> ii. Extends <strong>com.morpheusdata.core.Plugin</strong><br> iii. Specified in the build.gradle file as the <strong>Plugin-Class</strong><br> iv. Registers <strong>provider</strong> classes that add functionality to the plugin</li> <li><strong>PluginDemoGenericProvider</strong><br> A provider class that add generic functionality. Many types of providers can be added to plugins.</li> <li><strong>pluginDemoShow.hbs</strong><br> Handlebars markup to display UI elements in the HPE Morpheus Enterprise web UI.</li> </ul> <h2>Compiling locally on Windows</h2> <p>As a first compile option, we will look at the local Windows environment.</p> <ol> <li> <p>Start by opening a terminal. You may need to click the ellipses to exposes the entire menu.</p> <p><img src="/img/81terminal.png" alt="Opening a terminal in Visual Studio Code" title="Open terminal"></p> </li> <li> <p>Run the Windows wrapper batch script command:</p> <pre><code class="language-batch">.\gradlew.bat clean build </code></pre> <p>The <strong>shadowJar</strong> parameter can be used instead <strong>build</strong>.</p> <p><img src="/img/8gradlew_bat.png" alt="Running the gradlew.bat command in terminal" title="Build/compile command"></p> </li> <li> <p>Find the generated <strong>.jar</strong> file that was generated on successful build under the <strong>build > libs</strong> directory. Use the <strong>.jar</strong> file suffixed with <strong>-all</strong>. This file contains all compilation dependencies and is therefore safer to upload into HPE Morpheus Enterprise.</p> <p><img src="/img/9.-jar-file.png" alt="Browse to the generated jar file" title="Browse to the jar file"></p> </li> <li> <p>Upload the plugin to the UI by navigating to <strong>Administration > Integrations > Plugins > Add</strong>. Drag the file onto the dialog or browse to the .jar file and click Upload.</p> <p><img src="/img/10.-upload-jar.png" alt="Upload jar file to the UI" title="Upload jar file to the UI"></p> </li> </ol> <p>A successful upload will add the plugin name to the list:</p> <p><img src="/img/11.-uploaded.png" alt="Plugin successfully uploaded to the list" title="Uploaded plugin into list"></p> <p>To view the registered providers of an uploaded plugin, click the edit pencil to view the dialog.</p> <p><img src="/img/12.-plugin-providers.png" alt="Plugin providers dialog in HPE Morpheus Enterprise" title="View plugin details"></p> <h2>Compiling the plugin on Linux</h2> <p>Another compile option is to connect to a <strong>remote Linux</strong> machine using an <strong>SSH</strong> session from <strong>Visual Studio Code.</strong> As a prerequisite for this, unzip the initial project to your Linux machine and ensure remote SSH connectivity. The <strong>Java JDK v17</strong> will need to be present in this environment as well.</p> <ol> <li> <p>Click the <strong>Open Remote Window</strong> at the bottom left of the Visual Studio Code window. In the top popup option box, choose <strong>Connect to Host</strong>.</p> <p><img src="/img/13.-connect-to-host.png" alt="Connect to remote host in Visual Studio Code" title="Connect to remote host"></p> </li> <li> <p>Choose <strong>'+' Add New SSH Host</strong> and enter your <strong>user@host</strong> combo. Choose any SSH configuration file on the system to save connection details to.</p> <p><img src="/img/13.-specify-host.png" alt="Specify SSH host configuration" title="Provide user host pair"></p> </li> <li> <p>Click the <strong>Open Remote Window</strong> at the bottom left of the Visual Studio Code window <strong>again</strong>. Choose <strong>Connect to Host again</strong>. This time, <strong>select the host</strong> that was entered and saved in the previous step.</p> <p><img src="/img/13choose_host.png" alt="Choose the configured host connection" title="Choose host connection"></p> </li> <li> <p>Satisfy any possible certificate security prompts and enter your password when prompted. A new Visual Studio Code window will be opened. As with the previous exercise, <strong>Open Folder</strong> from the main <strong>File menu</strong>.</p> <p><img src="/img/13open_folder.png" alt="Open folder on remote system" title="Open remote folder"></p> </li> <li> <p>Select your <strong>Plugin Demo</strong> folder and click <strong>OK</strong>. You will be prompted to supply the password again. As before, allow for the installation of the required Java Extension packs and wait for the <strong>Gradle: configure project</strong> message to disappear.</p> <p><img src="/img/61configuring.png" alt="Configuring project on remote system" title="Configuring project"></p> <p>The <strong>Java: Ready</strong> status message should remain in the bottom status bar.</p> <p><img src="/img/62configured.png" alt="Project configured successfully on remote system" title="Project configured"></p> </li> <li> <p>From the top main menu, choose <strong>terminal > New Terminal</strong>.</p> <p><img src="/img/14open-terminal.png" alt="Open new terminal in remote Visual Studio Code" title="Open new terminal"></p> </li> <li> <p>Ensure that the gradle <strong>wrapper script</strong> is <strong>executable</strong> by running the following:</p> <pre><code class="language-bash">chmod +x ./gradlew </code></pre> </li> <li> <p>This time, we run the <strong>gradlew</strong> script, <strong>instead of gradlew.bat</strong>.</p> <pre><code class="language-bash">./gradlew clean build </code></pre> <p><img src="/img/15linux_compile.png" alt="Compile using gradlew on Linux" title="Compile using gradlew"></p> </li> </ol> <h2>Compiling using Docker</h2> <p>To avoid the need for a dedicated local development environment or specific JDK installations, you can compile the plugin inside a <strong>Docker container</strong>. Using the official Gradle image, the build process runs in an isolated environment that already includes the correct JDK version and Gradle tooling. This ensures consistent results across different systems and eliminates dependency issues.</p> <p>The prerequisite to this is a <strong>working Docker installation</strong>.</p> <pre><code class="language-bash">sudo docker run --rm \ -u "$(id -u):$(id -g)" \ -v "$PWD":/home/gradle/project \ -v gradle-cache:/home/gradle/.gradle \ -w /home/gradle/project \ gradle:8.10.2-jdk17 \ bash -lc "chmod +x ./gradlew &#x26;&#x26; ./gradlew clean build" </code></pre> <h2>Next steps</h2> <p>From here, we can explore the <strong>mechanics of the interfaces</strong> exposed by the HPE Morpheus Enterprise Plugin Core. These interfaces are organized into various <strong>provider types</strong>, each defining a specific kind of integration point within HPE Morpheus Enterprise.</p> <p>In essence, a <em>provider type</em> represents a particular extension area in the platform — such as a <strong>custom tab</strong>, <strong>analytics page</strong>, <strong>dashboard widget</strong>, or c<strong>ustom report</strong>. These allow developers to inject new functionality directly into the UI or automation workflows.</p> <p>At the more advanced end of the spectrum are provider types that model <strong>core infrastructure components</strong>. These include integrations for <strong>clouds</strong>, <strong>networks</strong>, <strong>storage systems</strong>, and many others. Such providers tend to be more complex because they interact deeply with HPE Morpheus Enterprise’s provisioning, synchronization, and lifecycle management layers. Understanding how these provider types fit together is key to building powerful, production-grade plugins.</p> <p>Explore the following resources for more information on the different plugin/provider types:</p> <ul> <li><a href="https://developer.morpheusdata.com">https://developer.morpheusdata.com</a></li> <li><a href="https://share.morpheusdata.com">https://share.morpheusdata.com</a> (follow the repository link under the plugin details to see the source code of a plugin)</li> <li><a href="https://github.com/hewlettpackard">https://github.com/hewlettpackard</a></li> <li><a href="https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX">https://youtu.be/1twoNvPoEV4?si=elUEzCYGo88TIffX</a></li> </ul>https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-8-striving-toward-adoptability/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-8-striving-toward-adoptability/Thu, 13 Nov 2025 07:17:53 GMT<p>external blog</p>https://developer.hpe.com/2025-nov-03/https://developer.hpe.com/2025-nov-03/Mon, 03 Nov 2025 13:00:00 GMThttps://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-7-minimalist-language-designs/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-7-minimalist-language-designs/Wed, 15 Oct 2025 18:23:22 GMT<p>external blog</p>https://developer.hpe.com/open-sourcing-workshops-on-demand-part-2-how-to-deploy-the-infrastructure/https://developer.hpe.com/open-sourcing-workshops-on-demand-part-2-how-to-deploy-the-infrastructure/Thu, 09 Oct 2025 12:50:13 GMT<p>In the first <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">article</a> of this series, I described the reasons behind the decision to open source our Workshops-on-Demand (WoD) project and gave you a comprehensive picture of the project's overall infrastructure. In this second article, I will explain how to deploy it.</p> <p>The overall infrastructure can run on physical servers or Virtual Machines. We usually designate one server for the frontend and a second server for the backend. You could also decide to separate every single component of each side.</p> <p><img src="/img/howto-wod-5.png" alt="" title="wod infrastucture"></p> <h2>How to deploy your own Workshops-on-Demand infrastructure...</h2> <p>As explained in the previous <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">article</a>, the project is split into multiple repositories from the architectural and public / private aspects. Since the publication if of the previous article, additionnal repositories showed up as the installation process evolved over time. The architecture is divided between the frontend, api-db, and backend. The project admins will need to decide whether they are willing to develop and propose public-only content to the participants or add any proprietary and private content.</p> <p>I will start with the simpliest scenario: A public-only approach. Then I will dive into the specificities related the private approach.</p> <h3>Public-only deployment: No private backend nor private workshops</h3> <p><strong>Important Note:</strong></p> <p><strong>This part is compulsory for any type of deployment. Public only or public + private.</strong></p> <p>First, you need a repository to clone. The Workshops-on-Demand GitHub projects can be found <a href="https://github.com/Workshops-on-Demand/">here</a>. We have packaged the project over several GitHub repos. Each repository handles a specific role in the overall architecture.</p> <p>Here's a quick look at what can be found in each:</p> <p><img src="/img/wod-repository-2025.png" alt="" title="WOD Repositories"></p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-notebooks">wod-notebooks</a>:</strong> Public Workshops-on-Demand based on Jupyter Notebooks.</p> <ul> <li>You can test them live at <a href="https://hackshack.hpedev.io/workshops">https://hackshack.hpedev.io/workshops</a></li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-install">wod-install</a>:</strong> Installer part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-backend">wod-backend</a>:</strong> Back-end part of our Workshops-on-Demand setup.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend">wod-frontend</a>:</strong> Frontend part of the Workshops-on-Demand project.</p> <ul> <li>Based on NGINX and NodeJS technologies, it provides the participtants' Registration Portal used to enable booking of the workshops.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db">wod-api-db</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>Open API 3.0 based api used to manage the Workshops-on-Demand project. It also provides a database hosting the different status of participants, workshops, and students.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-private">wod-private</a>:</strong> Example Private configuration for Workshops-on-Demand (WoD).</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend-private">wod-frontend-private</a>:</strong> Private Frontend part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db-private">wod-api-db-private</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>This provide examples for creating your own cutomization layer on top of the public standard WoD Backend / WoD Notebooks content. Do not put any confidential data here as this is a public repository!</li> </ul> <p><strong>Note</strong>: There are now 9 repositories available for now.</p> <p><img src="/img/wod-blogserie2-2repos.png" alt="" title="Workshops-on-Demand repositories"></p> <p>The Workshops-on-Demand project provides:</p> <ul> <li> <p>An Installer that allows you to install either Backend, Api-DB server, or Frontend using a single line of command.</p> </li> <li> <p>A complete JupyterHub server with some add-ons (additional JupyterHub kernels, Ansible galaxies, and PowerShell libraries) on your system, ready to use.</p> </li> <li> <p>A Postfix server used for the procmail API</p> </li> <li> <p>An Ansible engine to allow automation</p> </li> <li> <p>A Fail2Ban service</p> </li> <li> <p>An Admin user to manage everything</p> </li> <li> <p>A set of scripts to handle different tasks such as:</p> <ul> <li>Notebooks deployment</li> <li>JupyterHub compliancy</li> <li>Users compliancy</li> <li>Security Management</li> <li>Workshops updates</li> </ul> </li> </ul> <h4>Backend server preparation:</h4> <p>The installation process is handled by a dedicated repo : wod-install. This repo needs to be cloned on every single machine constituting the WoD architecture. Before cloning the wod-install repository, you will need to prepare the server that will host the backend features. When ready, you will proceed with the cloning and then the installation process.</p> <h5>Prerequesites:</h5> <p>In order to setup the backend server, you will need:</p> <ul> <li>A fresh OS install on physical / virtualized server running Ubuntu 24.04 or Centos 7.9 leveraging any deployment mechanism of your choice.(e.g. iLO, vagrant, etc.). You may even use this vagrant file to automatically generate a complete setup leveraging vagrant, libvirt and QEMU/KVM.</li> <li>A Linux account with sudo priviledges on your Linux distro. Name it <code>install</code></li> </ul> <p><strong>Note</strong>: In order to support 100 concurrent users, you will need:</p> <ul> <li>2 cpus or more machine</li> <li>128 GB of RAM</li> <li>500 GB of storage</li> </ul> <p>As a point of interest, we are currently using an HPE ProLiant DL360 Gen10 server on our different production sites.</p> <p>When done with OS installation and preparation</p> <ul> <li>From the WoD-backend server (aka JupyterHub server), as the install user, you will need to clone the wod-install repo first.</li> </ul> <pre><code class="language-shellsession">install$ git clone https://github.com/Workshops-on-Demand/wod-install.git install$ cd wod-install/ </code></pre> <ul> <li>Examine default installation parameters and adapt when necessary accordingly. Files are self-documented.</li> </ul> <p>Look at the following files within ansible/group_vars directory.</p> <ul> <li><code>all.yml</code> file</li> </ul> <pre><code class="language-shellsession">vi all.yml --- # We create fixed user accounts to provide an isolated execution environment to run the jupyter notebooks # They are called studentXXX where XXX is set between USERMIN and USERMAX defined below potentially with the addition of an offset (UIDBASE) for their uid/gid # Their home directory is located under /student and is thus named /student/studentXXX # Corresponding JupyterHub accounts are also created # # USERMIN indicates the starting ID of the Linux and Jupyter user account range # USERMIN: 1 # # USERMAX indicates the ending ID of the Linux and Jupyter user account range # USERMAX: 20 # # UIDBASE is the offset used to create the Linux user account IDs # Example when creating user 35 with UIDBASE of 2000, the uid created is 2035 # UIDBASE: 2000 # # GIDBASE is the offset used to create the Linux group IDs # Example: When creating user 35 with GIDBASE of 2000, the gid created is 2035 # GIDBASE: 2000 # # Set CLEAN to true if you want all Liunx &#x26; Jupyter user accounts to be removed before ansible check # CLEAN: false # # VAULTPWD is the password used to manage the Ansible vault # VAULTPWD: VeryComplexPasswd1234! # # NOCHECKSSH are ssh options used to dialog with appliances # By default, avoid checking Host keys and Host file as they may change on a regular base # NOCHECKSSH: -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null # # Branding management - Use if you want to customize Logo and Notebooks branding # BRANDING: "WoD Developer" BRANDINGWOD: "WoD Developer" BRANDINGLOGO: "" BRANDINGURL: "https://wod.io" # # Survey management - Use if you want to ask for feedback on your workshops - Look at existing conclusion notebooks SURVEYURL: TBD SURVEYCHALURL: TBD # # JPHUB is the directory used to install the JupyterHub stack (a Python venv) # JPHUB: /opt/jupyterhub # # # These variables are defined in Ansible playbooks. Do not change without knowing what you are doing. # STUDDIR: "{{ ansible_env.STUDDIR }}" WODBEDIR: "{{ ansible_env.WODBEDIR }}" WODPRIVDIR: "{{ ansible_env.WODPRIVDIR }}" WODNOBO: "{{ ansible_env.WODNOBO }}" WODAPIDBDIR: "{{ ansible_env.WODAPIDBDIR }}" WODFEDIR: "{{ ansible_env.WODFEDIR }}" SCRIPTDIR: "{{ WODBEDIR }}/scripts" ANSIBLEDIR: "{{ WODBEDIR }}/ansible" # This is the predefined structure for a private repo WODPRIVNOBO: "{{ WODPRIVDIR }}/notebooks" SCRIPTPRIVDIR: "{{ WODPRIVDIR }}/scripts" ANSIBLEPRIVDIR: "{{ WODPRIVDIR }}/ansible" </code></pre> <ul> <li><code>wod-backend</code> file</li> </ul> <pre><code class="language-shellsession">vi wod-backend # # These variables are located lower in the ansible tree to allow different values required for different backends while keeping a single frontend # # BASESTDID is the offset used to create users in the DB. It is required that each backend has a different non overlapping value. # Overlap is defined by BASESTDID + USERMAX (from all.yml) # # Example: # for student 35 in location A having BASESTDID to 0 the user is create as id 35 # for student 35 in location B having BASESTDID to 2000 the user is create as id 2035 # There is no overlap as long as you do not create more than 2000 users which should be the value of USERMAX in that case. # # This is different from the offset UIDBASE used for Linux uid # BASESTDID: 0 # # POSTPORT is the Postfix Port on which the smtp service is listening to receive API mail requests from the frontend # POSTPORT: "10025" # # In case you are using an LDAP server to use, flag as such the corresponding workshops in the DB and use the following values: # LDAPSRVNAME: ldap.example.org LDAPDMN: example.org LDAPPWD: MotDePasseLDAPCompliquéAussi123!!!## LDAPPORT: "389" # # For various existing public WoDs - These are needed. Adapt but do not remove! # SSHPORT-WKSHP-Docker101: 14101 SSHPORT-WKSHP-Ansible101: 16001 HTTPPORT-WKSHP-Docker101: 14151 HTTPPORT-WKSHP-Ansible101: 16051 HTTPPORT-WKSHP-Spark101: 17161 HTTPPORT-WKSHP-Concourse101: 19061 HTTPPORT-WKSHP-ML101: 18061 HTTPPORT-WKSHP-DataVisu101: 22161 CONCOURSEPORT-WKSHP-Concourse101: 19001 CONCOURSEPORT2-WKSHP-Concourse101: 19031 IP-WKSHP-DataVisu101: x.y.z.t IP-WKSHP-Concourse101: x.y.z.t IP-WKSHP-Docker101: x.y.z.t IP-WKSHP-Ansible101: x.y.z.t IP-WKSHP-Spark101: x.y.z.t IP-WKSHP-ML101: x.y.z.t IP-WKSHP-StackStorm101: x.y.z.t SPARKPORT-WKSHP-Spark101: 17101 SPARKPORT2-WKSHP-Spark101: 17131 MLPORT-WKSHP-ML101: 18101 MLPORT2-WKSHP-ML101: 18031 DATAVISUPORT1-WKSHP-DataVisu101: 22101 DATAVISUPORT2-WKSHP-DataVisu101: 22131 </code></pre> <ul> <li><code>wod-system</code> file</li> </ul> <pre><code class="language-shellsession">vi wod-system # # Backend API management # # Do not change, as the port is fixed in the JupyterHub install # WODBEAPIURL: http://{{ WODBEFQDN }}:8000 # # Replace with a random one - TODO Do that automatically at install time # WODBETOKEN: 2c0246e2c8564dc6ac7b12c544b25d77 # # You may want to use these variables if you have an OPNSense server as a security FW and are allowing http comm internally # #OPNSENSEKEY: #OPNSENSESEC: #OPNSENSEIP: #OPNSENSEPORT: # # Front-end API management # # Do not change, as the port is fixed in the JupyterHub install # WODFEAPIURL: https://{{ WODAPIDBFQDN }}/api # # Adapt to your setup - Used by installer to setup the frontend # WODFEAPIUSER: moderator WODFEAPIPWD: MotDePasseCompliquéAussi125!!!## </code></pre> <p>See the example below for a backend server.</p> <h3>Backend server installation :</h3> <p><a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/INSTALL.md#for-private-based-workshops-on-demand-private-backend--private-workshops-or-if-you-need-to-modify-defaults"></a>Once you are done with the files, you can can proceed with the installation itself. The installation is based on a common install script <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/install/install.sh">install.sh </a>that allows the deployment of the different parts of the solution. The script is located under the <code>wod-install/install/</code> directory.</p> <p>It can be called as follows:</p> <p><code>install.sh [-h][-t type][-g groupname][-b backend][-f frontend][-a api-db][-e external][-u user][-s sender]</code></p> <p>As you can see on the command line, the -t parameter will define whether you install a backend, an api-db, or a frontend server. Having this information, the script will clone the relevant repository for the installation. If <code>t=backend</code>, then the <code>wod-backend</code> repository is cloned as part of the installation process, and the relevant installation scripts are called. Same goes for api-db, and frontend servers.</p> <p><code>-h</code> parameter provides help.</p> <pre><code class="language-shellsession">install@wod-backend2-u24:~/wod-install/install$ sudo ./install.sh -h install.sh called with -h install.sh [-h][-t type][-i ip][-g groupname][-b backend[:beport:[beproto]][-n number][-j backendext[:beportext[:beprotoext]]][-f frontend[:feport[:feproto]]][-w frontendext[:feportext[:feprotoext]]][-a api-db[:apidbport[:apidbproto]]][-e api-dbext[:apidbportext[:apidbprotoext]]][-u user][-p postport][-k][-c][-s sender] where: -a api-db is the FQDN of the REST API/DB server potentially with a port (default 8021) potentially with a proto (default http) example: api.internal.example.org if empty using the name of the frontend -b backend is the FQDN of the backend JupyterHub server, potentially with a port (default 8000). potentially with a proto (default http) if empty uses the local name for the backend If you use multiple backend systems corresponding to multiple locations, use option -n to give the backend number currently being installed, starting at 1. When installing the api-db server you have to specify one or multiple backend servers, using their FQDN separated with ',' using the same order as given with the -n option during backend installation. -e api-dbext is the FQDN of the REST API server accessible externally potentially with a port (default 8021) potentially with a proto (default http) example: api.external.example.org if empty using the name of the api-db useful when the name given with -a doesn't resolve from the client browser -f frontend is the FQDN of the frontend Web server potentially with a port (default 8000). potentially with a proto (default http) example: fe.external.example.org if empty using the name of the backend -g groupname is the ansible group_vars name to be used example: production, staging, test, ... if empty using 'production' -i ip IP address of the backend server being used if empty, try to be autodetected from FQDN of the backend server Used in particular when the IP can't be guessed (Vagrant) or when you want to mask the external IP returned by an internal one for /etc/hosts creation -j backext is the FQDN of the backend JupyterHub server accessible externally potentially with a port (default 8000). potentially with a proto (default http) example: jupyterhub.external.example.org if empty using the name of the backend useful when the name given with -b doesn't resolve from the client browser -k if used, force the re-creation of ssh keys for the previously created admin user if not used keep the existing keys in place if any (backed up and restored) if the name of the admin user is changed, new keys systematically re-created -c if used, force insecured curl communications this is particularly useful for self-signed certificate on https services if not used keep curl verification, preventing self-signed certificates to work -n if used, this indicates the number of the backend currently installed used for the backend installation only, when multiple backend systems will be used in the configuration example (single backend server install on port 9999): -b be.int.example.org:9999 example (first of the 2 backends installed): -b be1.int.example.org:8888 -n 1 example 'second of the 2 backends installed): -b be2.int.example.org:8888 -n 2 example (install of the corresponding api-db server): -b be.int.example.org:8888,be2.int.example.org:8888 -p postport is the port on which the postfix service is listening on the backend server example: -p 10030 if empty using default (10025) -s sender is the e-mail address used in the WoD frontend to send API procmail mails to the WoD backend example: sender@example.org if empty using wodadmin@localhost -t type is the installation type valid values: appliance, backend, frontend or api-db if empty using 'backend' -u user is the name of the admin user for the WoD project example: mywodadmin if empty using wodadmin -w frontext is the FQDN of the frontend JupyterHub server accessible externally potentially with a port (default 8000). potentially with a proto (default http) example: frontend.external.example.org if empty using the name of the frontend useful to solve CORS errors when external and internal names are different Full installation example of a stack with: - 2 backend servers be1 and be2 using port 8010 - 1 api-db server apidb on port 10000 using https - 1 frontend server front on port 8000 - all declared on the .local network - internal postfix server running on port 9000 - e-mail sender being wodmailer@local - ansible groupname being test - management user being wodmgr On the be1 machine: ./install.sh -a apidb.local:10000:https -f front.local:8000 \ -g test -u wodmgr -p 9000 -s wodmailer@local\ -b be1.local:8010 -n 1 -t backend \ On the be2 machine: ./install.sh -a apidb.local:10000:https -f front.local:8000 \ -g test -u wodmgr -p 9000 -s wodmailer@local\ -b be2.local:8010 -n 2 -t backend \ On the apidb machine: ./install.sh -a apidb.local:10000:https -f front.local:8000 \ -g test -u wodmgr -p 9000 -s wodmailer@local\ -b be1.local:8010,be2.local:8010 -t api-db \ On the frontend machine: ./install.sh -a apidb.local:10000:https -f front.local:8000 \ -g test -u wodmgr -p 9000 -s wodmailer@local\ -t frontend \ </code></pre> <p><code>install.sh</code> performs the following tasks:</p> <ul> <li> <p>Calls the <code>install-system-&#x3C;&#x3C; distribution name >>.sh</code> script</p> </li> <li> <p>Installs minimal required (<code>ansible, git, jq, openssh server, npm</code>)</p> </li> <li> <p>Creates an admin user as defined upper (default is <code>wodadmin</code>) with sudo rights</p> </li> <li> <p>Calls the <code>install-system-common.sh</code> script that performs the following tasks:</p> <ul> <li>Cleanup</li> <li>Github repos cloning (leveraging install.repo file) : public Backend and public Private repos</li> <li>Create ssh keys for wodadmin</li> <li>Creates GROUPNAME variables</li> <li>Creates Ansible inventory files</li> </ul> </li> <li> <p>Calls the <code>install_system.sh</code> script with the type (backend, frontend, etc..) that performs the following tasks:</p> <ul> <li>Install the necessary stack based on selected type</li> <li>Create a <code>wod.sh</code> script in <code>wod-backend</code> directory to be used by all other scripts</li> <li>Source the <code>wod.sh</code> file</li> <li>Setup Ansible-galaxies (<code>community.general</code> and <code>posix</code>)</li> <li>Setup Ansible and call the playbook <code>install_&#x3C;type>.yml</code> followed by the <code>ansible\_check\_&#x3C;type>.yml</code></li> </ul> </li> </ul> <p>At the end of the installation process:</p> <ul> <li>You will have a JupyterHub server running on port 8000</li> <li>You will get a new <code>wodadmin</code> user (Default admin)</li> <li>You will get a set of 20 students (Default value)</li> </ul> <p>All playbooks are self-documented. Please check for details.</p> <p><strong>Note: A wod-install.log is available under the home folder of the install user under <code>.wod-install</code>. It contains the installation log along with a another file containing the wodadmin credentials.</strong></p> <p>I'll leave it to you to handle the necessary port redirection and SSL certificates management when needed. In our case, I went for a simple yet efficient solution based on an OPNSense Firewall along with a HAProxy setup to manage ports'redirection, HTTP to HTTPS Redirection, SSL Certificates. The backend also includes a Fail2ban service for login security management.</p> <p>At this point, you should be able to access your JupyterHub environment with a few pre-installed set of kernels like <code>Bash, Python, ansible, ssh, PowerShell</code>.</p> <p>You can then start developing new notebooks for your public based environment. And if you don't know how to do this, I will explain how in a future article.</p> <p>If you need to develop private content that cannot be shared with the wider Open Source Community because of dedicated IP, the next section in this article will explain how to handle this.</p> <h3><strong>How to handle private-content based Workshops-on-Demand</strong></h3> <h4><em>(private backend + private workshops on top of default public backend and notebooks)</em></h4> <p>The principle remains similar, with a few differences explained below.</p> <ul> <li> <p>After cloning the wod-install repository, you will fork the following public private <a href="https://github.com/Workshops-on-Demand/wod-private.git">repo</a> on Github under your own Github account (we will refer to it as <code>Account</code>).</p> </li> <li> <p>Next, clone the forked repo.</p> </li> <li> <p>Edit the <code>all.yml</code> and <code>&#x3C;groupname></code> files to customize your setup. This variable <code>&#x3C;groupname></code> defines possible backend server in your environement. By default, the project comes with a sample working file named <code>production</code> in <code>ansible/group-vars</code>. But you could have multiple. In the case I've presented, I have defined <code>sandbox</code>, <code>test</code>, <code>staging</code> and several <code>production</code> files, all defining a different backend environment. These files will be used to override the default values specified by the public version delivered as part of the default public installation.</p> </li> <li> <p>Commit and push changes to your repo.</p> </li> <li> <p>Create an <code>install.priv</code> file located in <code>install</code> directory when using a private repo (consider looking at <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/install/install.repo">install.repo</a> file for a better understanding of the variables).</p> </li> <li> <p>Define the WODPRIVREPO and WODPRIVBRANCH variables as follows:</p> <ul> <li><code>WODPRIVBRANCH="main"</code></li> <li><code>WODPRIVREPO="git@github.com:Account/Private-Repo.git wod-private"</code></li> </ul> </li> </ul> <p><strong>Note:</strong> When using a token</p> <p>Please refer to the following <a href="https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token">url</a> to generate a <code>token</code> file in <code>install</code> directory of WoD-backend:</p> <ul> <li> <p>Edit the <code>install.priv</code> file located in <code>install</code> directory of wod-install:</p> <ul> <li>Create line before variable declaration: <code>token=`cat $EXEPATH/token` </code></li> <li>Use the token in the url WODPRIVREPO="git clone <a href="https://user:$token@github.com/Account/wod-private.git%C2%A0wod-private">https://user:$token@github.com/Account/wod-private.git wod-private</a>"</li> </ul> </li> </ul> <p>You are now ready to perform the installation again to support a private repository.</p> <p>Please note that this setup phase can be concurrent with the public setup phase. Indeed, the install script should detect the presence of the private repository owing to the presence of the install.priv file. It will automatically adjust the different scripts and variables to add the relevant content. It will actually overload some of the variables with private ones.</p> <p>You now have a working Workshops-on-Demand backend server in place. In order to install the api-db server as well as the frontend server, please follow the same steps.</p> <p>In order to setup the api-db server, you will need:</p> <ul> <li>A fresh OS install on physical / virtualized server running Ubuntu 24.04 or Centos 7.9 leveraging any deployment mechanism of your choice.(e.g. iLO, vagrant, etc.). You may even use this vagrant file to automatically generate a complete setup leveraging vagrant, libvirt and QEMU/KVM.</li> <li>A Linux account with sudo priviledges on your Linux distro. Name it <code>install</code></li> <li>2 cpus or more machine</li> <li>16 GB of RAM</li> <li>60 GB of storage</li> </ul> <p>In order to setup the frontend server, you will need:</p> <ul> <li>A fresh OS install on physical / virtualized server running Ubuntu 24.04 or Centos 7.9 leveraging any deployment mechanism of your choice.(e.g. iLO, vagrant, etc.). You may even use this vagrant file to automatically generate a complete setup leveraging vagrant, libvirt and QEMU/KVM.</li> <li>A Linux account with sudo priviledges on your Linux distro. Name it <code>install</code></li> <li>2 cpus or more machine</li> <li>8 GB of RAM</li> <li>60 GB of storage</li> </ul> <p>Congratulations! you should have now a running Workshops-on-Demand infrastructure. The next article in the series will help you better understand the lifecycle of the backend server. How does a workshop registration work from the backend server's side? How do you manage this server on a daily basis? How and when do you need to update it ? All these questions will be answered in the next article. And from there, I will help you move finally to a workshop's creation process.</p> <p>If you need support for this installation process, use our dedicated <a href="https://hpedev.slack.com/archives/C01B60X8SSD">slack channel</a>.</p> <p>Please be sure to check back <a href="https://developer.hpe.com/blog">HPE Developer blog site</a> to read all the articles in this series. Also, check out the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a> <a href="https://developer.hpe.com/hackshack/replays/42">Data Visualization 101</a> is now available! Stay tuned for additional Workshops-on-Demand in our catalog.</p>https://developer.hpe.com/2025-oct-07/https://developer.hpe.com/2025-oct-07/Tue, 07 Oct 2025 13:00:00 GMThttps://developer.hpe.com/exposing-grafana-service-using-tailscale-for-mks-monitoring-in-hpe-private-cloud-enterprise/https://developer.hpe.com/exposing-grafana-service-using-tailscale-for-mks-monitoring-in-hpe-private-cloud-enterprise/Fri, 03 Oct 2025 20:28:43 GMT<p>This blog post describes how to expose the <em>Grafana</em> service running in an MKS cluster within HPE Private Cloud Enterprise to the public Internet using <em>Tailscale</em> alongside <em>MetalLB</em>. Without the usual complexity of networking or intricate security configurations, the exposed <em>Grafana</em> dashboard becomes accessible both within the on-premises environment and from external networks. This approach offers a simple and effective way to monitor MKS clusters running in the HPE Private Cloud Enterprise environment.</p> <h2>Overview</h2> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> is a fully managed <em>Infrastructure as a Service</em> (IaaS) offering that brings a modern, cloud-like experience to on-premises environments. It combines the flexibility of hybrid cloud with the enterprise-grade control and security required by enterprise IT.</p> <p>Through integration with <a href="https://www.hpe.com/us/en/morpheus-enterprise-software.html">HPE Morpheus Enterprise</a>, which serves as the cloud management and orchestration layer, HPE Private Cloud Enterprise delivers a unified self-service interface for provisioning virtual machines (VMs), creating containers, and deploying applications, all governed by role-based access control (RBAC). This integration now enables support for the Morpheus Kubernetes Service (MKS) feature, allowing users to deploy and manage Kubernetes (K8s) clusters with built-in automation and observability capabilities. You can refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a> to learn more about provisioning MKS clusters in HPE Private Cloud Enterprise.</p> <p>As you begin deploying applications in MKS clusters, networking quickly emerges as one of the key challenges. Traditional methods such as port forwarding, <em>NodePort</em> or <em>LoadBalancer</em> services, or manual virtual private network (VPN) setups can be cumbersome to configure, difficult to secure, and often require deep networking expertise. How can these applications be made accessible, both within the HPE Private Cloud Enterprise environment and from external networks, without the added complexity?</p> <p>The following sections describe how to expose services, running in MKS clusters within HPE Private Cloud Enterprise, to the public Internet using <em>Tailscale</em> and <em>MetalLB</em>, offering a streamlined and secure alternative to conventional approaches.</p> <h2>Prerequisites</h2> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>An MKS cluster has been provisioned from a HPE Private Cloud Enterprise workspace. You can refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning an MKS cluster in HPE Private Cloud Enterprise</a> to provision an MKS cluster.</li> <li>The <em>kubectl</em> CLI tool, together with the <em>kubeconfig</em> file for accessing the MKS cluster.</li> <li>The <em>helm</em> CLI tool, version 3.12.0 or later.</li> </ul> <h2><em>MetalLB</em> and <em>Tailscale</em></h2> <p><em><a href="https://metallb.io/">MetalLB</a></em> is a software solution that provides a network load balancer implementation for K8s clusters using standard routing protocols. By installing <em>MetalLB</em>, it supports the <em>LoadBalancer</em>-type services by assigning external IPs to services within the K8s clusters. This makes applications easily reachable within your private network, without needing any special hardware or cloud services.</p> <p><em><a href="https://tailscale.com/">Tailscale</a></em> is a mesh VPN service that uses the <a href="https://www.wireguard.com/">WireGuard</a> protocol to securely connect devices across different networks. Instead of routing traffic through a central server like traditional VPNs, <em>Tailscale</em> creates encrypted peer-to-peer connections between devices. These connections form a private network called a <em>tailnet</em>, where each device receives a unique <em>Tailscale</em> IP address for direct communication. A tailnet provides a secure, interconnected space of users, devices, and resources, all managed through Tailscale's admin console, where you can configure access controls, <em>DNS</em> settings, <em>TLS</em> certificates, and more.</p> <p>By utilizing the external IP addresses assigned by <em>MetalLB</em> to <em>LoadBalancer</em>-type services within the local private network, <em>Tailscale</em> securely exposes these services via publicly accessible URLs, without revealing their underlying service IP addresses.</p> <h2>Set up the load balancer with <em>MetalLB</em></h2> <p>You can install <em>MetalLB</em> and set up the load balancer in the MKS cluster by following the instructions in the blog post <a href="https://developer.hpe.com/blog/exposing-an-application-using-ingress-and-tls-termination-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/">Setting up the load balancer with MetalLB</a>.</p> <p>Run the following command to view the deployed <em>MetalLB</em> in the <em>metallb-system</em> namespace of the MKS cluster <em>mks-test</em>.</p> <pre><code class="language-shell">$ kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/metallb-controller-8474b54bc4-gdgmx 1/1 Running 0 14d pod/metallb-speaker-2f8zj 4/4 Running 0 14d pod/metallb-speaker-qgg5p 4/4 Running 0 14d pod/metallb-speaker-qsv45 4/4 Running 0 14d pod/metallb-speaker-xhhcv 4/4 Running 0 14d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/metallb-webhook-service ClusterIP 172.30.168.138 &#x3C;none> 443/TCP 14d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/metallb-speaker 4 4 4 4 4 kubernetes.io/os=linux 14d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/metallb-controller 1/1 1 1 14d NAME DESIRED CURRENT READY AGE replicaset.apps/metallb-controller-8474b54bc4 1 1 1 14d </code></pre> <p>You can view the virtual IP address range "172.20.40.240-172.20.40.250" defined in the custom resource definition (CRD) <em>IPAddressPool</em>, along with the layer 2 service IP address announcement specified in the CRD resource <em>L2Advertisement</em>.</p> <pre><code class="language-shell">$ kubectl get ipaddresspool -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES cfe-pool true false ["172.20.40.240-172.20.40.250"] $ kubectl get l2advertisement -n metallb-system NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES cfe-l2advert ["cfe-pool"] </code></pre> <h2>Deploy <em>Tailscale</em></h2> <h3>Install <em>Tailscale</em> client</h3> <p>In order to use <em>Tailscale</em>, you need to install the <em>Tailscale</em> client on your device. The <em>Tailscale</em> client is open source and available for various platforms, such as <em>Linux</em>, <em>Windows</em>, <em>macOS</em>, <em>iOS</em>, and <em>Android</em>, etc. It is used, via its admin console, to connect various devices securely to your private <em>Tailscale</em> network (<em>tailnet</em>). It is the bridge between your device and the rest of your tailnet.</p> <p>Here is the admin console of my Windows <em>Tailscale</em> client installed using the package available from the <a href="https://tailscale.com/download">Tailscale download page</a>. It uses a <em>Tailscale</em> account by choosing <em>GitHub</em> as the identity provider. You can integrate your <em>Tailscale</em> account using your own identity provider for secure single sign-on (SSO) login and multi-factor authentication (MFA).</p> <p><img src="/img/tailscale-machines.png" alt=""></p> <p>My Windows laptop <em>guoping</em> is connected to the tailnet associated with my <em>GitHub</em> identity provider.</p> <h3>Generate <em>Tailscale</em> auth key</h3> <p>After installing the <em>Tailscale</em> client, you need to generate an auth key from the <em>Tailscale</em> admin console.</p> <ol> <li>Navigate to <strong>Settings</strong> -> <strong>Keys</strong>. Click <em><strong>Generate auth key</strong></em>.</li> </ol> <p><img src="/img/tailscale-settings-keys.png" alt=""></p> <ol start="2"> <li>Enter <em>Description</em> as <em>mks-auth-key</em> and set <em>Expiration</em>. Click <em><strong>Generate key</strong></em>.</li> </ol> <p><img src="/img/tailscale-generate-auth-key.png" alt=""></p> <ol start="3"> <li>Copy and save the generated new key.</li> </ol> <p><img src="/img/tailscale-auth-key.png" alt=""></p> <p>Create a <em>Secret</em> YAML manifest file <em>'tailscale-auth.yaml'</em> using the generated auth key.</p> <pre><code class="language-shell">apiVersion: v1 kind: Secret metadata: name: tailscale-auth namespace: tailscale stringData: TS_AUTHKEY: tskey-auth-&#x3C;hidden> </code></pre> <p>Apply the <em>Secret</em> to the namespace <em>tailscale</em> of the MKS cluster. This secret will be used to securely join the cluster to your <em>Tailscale</em> network.</p> <pre><code class="language-shell">$ kubectl create ns tailscale $ kubectl apply -f tailscale-auth.yaml </code></pre> <h3>Create tag <em>k8s-operator</em></h3> <p>You need to create a tag named <em>k8s-operator</em> in the <em>Tailscale</em> admin console. This tag is used by <em>Tailscale</em> to authenticate and identify the <em>Tailscale</em> K8s operator that will be deployed to the MKS cluster.</p> <ol> <li>Navigate to <strong>Access controls -></strong> <em>Tags</em> tab. Click <em><strong>Create tag</strong></em>.</li> </ol> <p><img src="/img/tailscale-create-tag.png" alt=""></p> <ol start="2"> <li>Enter <em>Tag name</em> as <em>k8s-operator</em> and select <em>Tag owner</em> as <em>autogroup:member</em>. Click <em><strong>Save tag</strong></em>.</li> </ol> <p><img src="/img/tailscale-tag-k8s-operator.png" alt=""></p> <h3>Generate <em>Tailscale</em> OAuth client</h3> <p>You need to generate an OAuth client in the <em>Tailscale</em> admin console.</p> <ol> <li>Navigate to <strong>Settings</strong> -> <strong>OAuth clients</strong>. Click <em><strong>Generate OAuth client</strong></em>.</li> </ol> <p><img src="/img/tailscale-oauth-client.png" alt=""></p> <ol start="2"> <li>Under <strong>Devices</strong>, select <em>Core</em> with <em>Read and Write</em> and add tag <em>k8s-operator</em>. Under Keys, select <em>Auth Keys</em> with <em>Read and Write</em> and add the tag <em>k8s-operator</em>. Click <em><strong>Generate client</strong></em>.</li> </ol> <p><img src="/img/tailscale-oauth-client-k8s-operator.png" alt=""></p> <ol start="3"> <li>Copy and save the generated <em>Client ID</em> and <em>Client secret</em>.</li> </ol> <p><img src="/img/tailscale-oauth-client-details.png" alt=""></p> <h3>Deploy <em>Tailscale</em> K8s operator</h3> <p>You can now install the <em>Tailscale</em> K8s operator to the namespace <em>tailscale</em> of the MKS cluster using <em>Helm</em>, along with the generated <em>Tailscale</em> OAuth client, specifically the <em>Client ID</em> and <em>Client secret</em>.</p> <pre><code class="language-shell">$ helm repo add tailscale https://pkgs.tailscale.com/helmcharts $ helm repo update $ helm search repo tailscale NAME CHART VERSION APP VERSION DESCRIPTION tailscale/tailscale-operator 1.86.5 v1.86.5 A Helm chart for Tailscale Kubernetes operator $ helm upgrade --install tailscale-operator tailscale/tailscale-operator \ --namespace=tailscale --set-string oauth.clientId=&#x3C;hidden> \ --set-string oauth.clientSecret=tskey-client-&#x3C;hidden> --wait Release "tailscale-operator" does not exist. Installing it now. NAME: tailscale-operator LAST DEPLOYED: Wed Sep 24 15:02:41 2025 NAMESPACE: tailscale STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: You have successfully installed the Tailscale Kubernetes Operator! Once connected, the operator should appear as a device within the Tailscale admin console: https://login.tailscale.com/admin/machines If you have not used the Tailscale operator before, here are some examples to try out: * Private Kubernetes API access and authorization using the API server proxy https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy * Private access to cluster Services using an ingress proxy https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress * Private access to the cluster's available subnets using a subnet router https://tailscale.com/kb/1441/kubernetes-operator-connector You can also explore the CRDs, operator, and associated resources within the tailscale namespace: $ kubectl explain connector $ kubectl explain proxygroup $ kubectl explain proxyclass $ kubectl explain recorder $ kubectl explain dnsconfig If you're interested to explore what resources were created: $ kubectl --namespace=tailscale get all -l app.kubernetes.io/managed-by=Helm $ kubectl --namespace=tailscale get all -l app.kubernetes.io/managed-by=Helm </code></pre> <p>Type the following command to check the <em>Tailscale</em> operator deployment details.</p> <pre><code class="language-shell">$ kubectl get all -n tailscale NAME READY STATUS RESTARTS AGE pod/operator-945796556-cgg86 1/1 Running 0 41s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/operator 1/1 1 1 41s NAME DESIRED CURRENT READY AGE replicaset.apps/operator-945796556 1 1 1 41s </code></pre> <p>Once the <em>Tailscale</em> operator is successfully installed and running, a new machine named <em>tailscale-operator</em> appears under the <strong>Machines</strong> tab in your <em>Tailscale</em> admin console.</p> <p><img src="/img/tailscale-operator-machine.png" alt=""></p> <h2>Expose <em>Grafana</em> service</h2> <p>As part of the MKS cluster provisioning process, both <em>Prometheus</em> and <em>Grafana</em> are installed and configured in the namespace <em>monitoring</em>. Use the following command to view the details.</p> <pre><code class="language-shell">$ kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/alertmanager-main-0 2/2 Running 0 4d pod/alertmanager-main-1 2/2 Running 0 4d pod/alertmanager-main-2 2/2 Running 0 4d pod/blackbox-exporter-84d969fb75-msbqd 3/3 Running 0 4d pod/grafana-6698fc66bb-9rjk2 1/1 Running 0 4d pod/kube-state-metrics-6f5f95b6bf-6b77k 3/3 Running 0 4d pod/node-exporter-74nzh 2/2 Running 0 4d pod/node-exporter-89m4q 2/2 Running 0 4d pod/node-exporter-c699g 2/2 Running 0 4d pod/node-exporter-prmwt 2/2 Running 0 4d pod/node-exporter-vdfvj 2/2 Running 0 4d pod/prometheus-adapter-599c88b6c4-nd7xd 1/1 Running 0 4d pod/prometheus-adapter-599c88b6c4-zh2z5 1/1 Running 0 4d pod/prometheus-k8s-0 2/2 Running 0 4d pod/prometheus-k8s-1 2/2 Running 0 4d pod/prometheus-operator-75486dd88-pjdjh 2/2 Running 0 4d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager-main ClusterIP 172.30.103.195 &#x3C;none> 9093/TCP,8080/TCP 4d service/alertmanager-operated ClusterIP None &#x3C;none> 9093/TCP,9094/TCP,9094/UDP 4d service/blackbox-exporter ClusterIP 172.30.165.12 &#x3C;none> 9115/TCP,19115/TCP 4d service/grafana ClusterIP 172.30.211.119 &#x3C;none> 3000/TCP 4d service/kube-state-metrics ClusterIP None &#x3C;none> 8443/TCP,9443/TCP 4d service/node-exporter ClusterIP None &#x3C;none> 9100/TCP 4d service/prometheus-adapter ClusterIP 172.30.199.24 &#x3C;none> 443/TCP 4d service/prometheus-k8s ClusterIP 172.30.54.40 &#x3C;none> 9090/TCP,8080/TCP 4d service/prometheus-operated ClusterIP None &#x3C;none> 9090/TCP 4d service/prometheus-operator ClusterIP None &#x3C;none> 8443/TCP 4d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-exporter 5 5 5 5 5 kubernetes.io/os=linux 43d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/blackbox-exporter 1/1 1 1 4d deployment.apps/grafana 1/1 1 1 4d deployment.apps/kube-state-metrics 1/1 1 1 4d deployment.apps/prometheus-adapter 2/2 2 2 4d deployment.apps/prometheus-operator 1/1 1 1 4d NAME DESIRED CURRENT READY AGE replicaset.apps/blackbox-exporter-84d969fb75 1 1 1 4d replicaset.apps/grafana-6698fc66bb 1 1 1 4d replicaset.apps/kube-state-metrics-6f5f95b6bf 1 1 1 4d replicaset.apps/prometheus-adapter-599c88b6c4 2 2 2 4d replicaset.apps/prometheus-operator-75486dd88 1 1 1 4d NAME READY AGE statefulset.apps/alertmanager-main 3/3 4d statefulset.apps/prometheus-k8s 2/2 4d </code></pre> <p>To expose the <em>Grafana</em> service, change its service type from <em>ClusterIP</em> to <em>LoadBalancer</em> by running the commmand <em>'kubectl edit svc grafana -n monitoring'</em>. Once updated, the <em>Grafana</em> service appears as a <em>LoadBalancer</em> type and is assigned with an <em>EXTERNAL-IP</em> address, such as <em>'172.20.40.241'</em>.</p> <pre><code class="language-shell">$ kubectl get svc grafana -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 172.30.211.119 172.20.40.241 3000:31469/TCP 4d </code></pre> <p>You can now expose the <em>Grafana</em> service and make it publicly accessible using <a href="https://tailscale.com/kb/1223/funnel">Tailscale Funnel</a>. <em>Tailscale</em> Funnel exposes a local service running in the MKS cluster via a unique <em>Funnel URL</em>, formatted as <em>&#x3C;service-name>.&#x3C;tailscale domain></em>. When someone accesses the Funnel URL, the request is routed to the Funnel relay server, which then establishes an encrypted TCP tunnel to the local service. This ensures the data remains secure and the service's IP address stays hidden. The Funnel relay server cannot decrypt any data transmitted through the tunnel.</p> <p><em>Tailscale</em> Funnel is disabled by default. To enable it, use the <em>Tailscale</em> CLI command <em>'tailscale funnel'</em>. The <em>Tailscale</em> CLI tool is included when you install the <em>Tailscale</em> client.</p> <p>After <em>Tailscale</em> Funnel is enabled, you need to create a tag named <em>k8s</em> in the <em>Tailscale</em> admin console.</p> <p><img src="/img/tailscale-tag-k8s.png" alt=""></p> <p>You also need to add a node attribute, under <strong>Access controls -></strong> <em>Node attributes</em> tab, in the admin console.</p> <p><img src="/img/tailscale-node-attribute-k8s.png" alt=""></p> <p>Create the following <em>Ingress</em> YAML manifest file with the annotation <em>tailscale.com/funnel: "true"</em> and <em>ingressClassName: tailscale</em>. Apply it to the <em>monitoring</em> namespace.</p> <pre><code class="language-shell">$ cat ingress-grafana.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-grafana namespace: monitoring annotations: tailscale.com/funnel: "true" spec: defaultBackend: service: name: grafana port: number: 3000 ingressClassName: tailscale tls: - hosts: - grafana $ kubectl apply -f ingress-grafana.yaml ingress.networking.k8s.io/ingress-grafana created </code></pre> <p>After a few minutes, the deployed Ingress <em>ingress-grafana</em> displays its assigned Funnel URL <em>grafana.qilin-beta.ts.net</em>.</p> <pre><code class="language-shell">$ kubectl get ingress -n monitoring NAME CLASS HOSTS ADDRESS PORTS AGE ingress-grafana tailscale * grafana.qilin-beta.ts.net 80, 443 36s </code></pre> <p>The <strong>Machines</strong> tab of the <em>Tailscale</em> admin console shows the newly added device <em>grafana</em>, as well as its <em>Tailscale</em> IP address <em>100.110.103.12</em>.</p> <p><img src="/img/grafana-machine.png" alt=""></p> <h2>Access <em>Grafana</em> dashboard</h2> <p>You can start your browser by pointing to the Funnel URL <em>'grafana.qilin-beta.ts.net '</em>. After logging in, you can navigate to one of the pre-configured <em>Grafana</em> dashboards, e.g., <em>Kubernetes/API server</em>.</p> <p><img src="/img/grafana-funnel.png" alt=""></p> <p>You can access the exposed <em>Grafana</em> service from your mobile phone using the same Funnel URL to monitor your MKS cluster.</p> <p><img src="/img/grafana-mobile.png" alt=""></p> <h2>Conclusion</h2> <p>This blog post showed the steps for exposing the <em>Grafana</em> service, running within the MKS cluster in HPE Private Cloud Enterprise, using <em>Tailscale</em> and <em>MetalLB</em>. Without opening firewall ports or configuring reverse proxies, the <em>Grafana</em> dashboard becomes publicly accessible via its Funnel URL. Whether you are developing, debugging or showcasing applications in MKS clusters within the HPE Private Cloud Enterprise environment, this approach offers a simple and secure way to expose services.</p> <p>However, when applying the <em>Tailscale</em> setup in production environments, several security considerations remain important. Although traffic to the exposed Grafana service is encrypted end-to-end via <em>HTTPS</em>, it is recommended to enforce strict access controls using <em>Tailscale</em> ACLs. Additionally, you should configure <em>Tailscale</em> to automatically disable Funnel if the connection drops, ensuring services are not unintentionally exposed. With recent updates, <em>Tailscale</em> supports custom domain integration for Funnel URLs, ideal for creating production-grade public endpoints for your services.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/deploying-a-kubernetes-cluster-using-app-blueprint-with-ansible-integration-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/deploying-a-kubernetes-cluster-using-app-blueprint-with-ansible-integration-in-hpe-greenlake-for-private-cloud-enterprise/Fri, 03 Oct 2025 07:25:49 GMT<p>This blog post provides a detailed step-by-step guide on how to provision a Kubernetes (K8s) cluster using an app blueprint within the HPE Private Cloud Enterprise environment. Together with other key Morpheus components, such as <em>Ansible Integration</em> and <em>Automation Task and Workflow</em>, an app blueprint for provisioning K8s clusters can be created. Once configured, this app blueprint enables the provisioning of K8s clusters directly in the Morpheus platform in HPE Private Cloud Enterprise.</p> <h2>Overview</h2> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> is a fully managed <em>Infrastructure as a Service</em> (IaaS) offering that brings a modern, cloud-like experience to on-premises environments. It combines the flexibility of hybrid cloud with the enterprise-grade control and security required by enterprise IT.</p> <p>Through integration with <a href="https://www.hpe.com/us/en/morpheus-enterprise-software.html">HPE Morpheus Enterprise</a>, which serves as the cloud management and orchestration layer, HPE Private Cloud Enterprise delivers a unified self-service interface for provisioning virtual machines (VMs), creating containers, and deploying applications, all governed by role-based access control (RBAC).</p> <p>Among the key Morpheus components, <em>Ansible Integration</em> and <em>Automation Task and Workflow</em> can be used to create an app blueprint for provisioning K8s clusters using Ansible playbooks available from the <em>GitHub</em> repository. It automatically creates a list of required virtual machine (VM) instances and deploys K8s on top of these VM instances. This blog post walks through the process of creating an app blueprint and using it for K8s cluster provisioning in HPE Private Cloud Enterprise.</p> <h2>Prerequisites</h2> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>Access to a HPE Private Cloud Enterprise workspace with the <em>'Private Cloud Tenant Owner'</em> role, allowing administrative actions in the <em><strong>Virtual Machines</strong></em> service.</li> <li>The group named <em>'CFE Department B Group'</em> and the network <em>'Green-Net'</em> have already been created.</li> </ul> <h2>Create app blueprint</h2> <p>To create an app blueprint, you need to log in to HPE GreenLake Cloud and launch the HPE Morpheus Enterprise Dashboard. For a detailed walkthrough of this process, refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a>.</p> <h3>1. Add Ansible integration</h3> <ul> <li>From Morpheus Dashboard, navigate to <strong>Administration</strong> > <strong>Integrations</strong>.</li> </ul> <p><img src="/img/morpheus-intg.png" alt=""></p> <ul> <li>Click <em><strong>+New Integration</strong></em> and select <em>Ansible</em>.</li> </ul> <p><img src="/img/morpheus-ansible-intg.png" alt=""></p> <ol start="3"> <li>Enter NAME as 'cfe-ansible-k8s' and specify ANSIBLE GIT URL, DEFAULT BRANCH, PLAYBOOKS PATH, ROLES PATH, GROUP VARIABLES PATH, DESCRIPTION, and HOST VARIABLES PATH. Click <em><strong>SAVE CHANGES</strong></em>.</li> </ol> <p><img src="/img/k8s-ansible-intg.png" alt=""></p> <p><em><strong>Note</strong></em>: Sample Ansible playbooks are available from this <a href="https://github.com/guopingjia/ansible-k8s"><em>GitHub</em> repository</a> for provisioning a K8s cluster with one master and one worker node, using the native K8s distribution.</p> <h3>2. Create tasks and workflows</h3> <h4>Create tasks for K8s master and worker</h4> <ol> <li>Navigate to <strong>Library</strong> -> <strong>Automation</strong> -> <em>Tasks tab</em>. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>cfe-k8s-master</em> and select TYPE as <em>Ansible Playbook</em>. Then specify ANSIBLE REPO as <em>cfe-ansible-k8s</em> and PLAYBOOK as <em>master.yml</em>. Click <em><strong>SAVE CHANGES</strong></em>.</li> </ol> <p><img src="/img/k8s-master-task.png" alt=""></p> <ol start="3"> <li>Repeat <em>step 1</em> and <em>step 2</em> to create a task for K8s worker as name <em>cfe-k8s-worker</em>.</li> </ol> <p><img src="/img/k8s-worker-task.png" alt=""></p> <h3>Create workflows for K8s master and worker</h3> <ol> <li>Navigate to <strong>Library</strong> -> <strong>Automation</strong> -> <em>Workflows</em> tab.</li> <li>Click <em><strong>+Add</strong></em> and select <em>Provisioning Workflow</em>.</li> </ol> <p><img src="/img/morpheus-workflow.png" alt=""></p> <ol start="3"> <li>Enter NAME as <em>cfe-k8s-master</em> and select PLATFORM as <em>Linux</em>. Then search and select the task <em>cfe-k8s-master</em>. Click <em><strong>SAVE CHANGES</strong></em>.</li> </ol> <p><img src="/img/k8s-master-workflow.png" alt=""></p> <ol start="4"> <li>Repeat <em>step 1</em> to <em>step 3</em> to create a workflow for K8s worker as name <em>cfe-k8s-worker</em>.</li> </ol> <p><img src="/img/k8s-worker-workflow.png" alt=""></p> <h2>Create app blueprint</h2> <ol> <li>Navigate to <strong>Library</strong> -> <strong>Blueprints</strong> -> <em>App Blueprints</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>CEF-K8s-Ubuntu</em> and select TYPE as <em>Morpheus</em>. Click <em><strong>Next</strong></em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-summary.png" alt=""></p> <ol start="3"> <li>Click <em><strong>+</strong></em> (next to <em>CFE-K8s-Ubuntu</em>) and select <em>Tier Name</em> as <em>App</em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-tier-name.png" alt=""></p> <ol start="4"> <li>Click <em>App</em> and edit its CONFIGURATION with NAME as <em>CFE-K8s-master</em> and BOOT ORDER as <em>0</em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-master-config.png" alt=""></p> <ol start="5"> <li>Click <em><strong>+</strong></em> again (next to <em>CFE-K8s-Ubuntu</em>) and select <em>Tier Name</em> as <em>App</em>. Then click <em>App</em> and edit its CONFIGURATION with NAME as <em>CFE-K8s-worker</em> and BOOT ORDER as <em>1</em>. Under <strong>Connected Tiers</strong>, select <em>CFE-K8s-master</em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-worker-config.png" alt=""></p> <ol start="6"> <li>Click <em><strong>+</strong></em> (next to <em>CFE-K8s-master</em>) and select <em>vmware</em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-vmware.png" alt=""></p> <ol start="7"> <li>Click <em><strong>+</strong></em> (next to <em>vmware</em>) and select <em>Group</em>, <em>Cloud</em> and <em>Environment</em>. Click <em><strong>Add config</strong></em>.</li> </ol> <p><img src="/img/k8s-app-blueprint-vmware-config.png" alt=""></p> <ol start="8"> <li>Click the added config and configure NAME, DESCRIPTION, LAYOUT, PLAN, VOLUMES, NETWORKS and IMAGE.</li> </ol> <p><img src="/img/k8s-app-blueprint-master.png" alt=""></p> <ol start="9"> <li>Repeat <em>step 6</em> to <em>step 8</em> to configure K8s worker instance settings.</li> </ol> <p><img src="/img/k8s-app-blueprint-worker.png" alt=""></p> <ol start="10"> <li>Click <em><strong>Complete</strong></em>. The final app blueprint structure and configuration display.</li> </ol> <p><img src="/img/k8s-app-blueprint-final.png" alt=""></p> <ol start="11"> <li>Review and click <em><strong>Save</strong></em>.</li> </ol> <h2>Deploy K8s cluster</h2> <p>Perform the following steps to provision a K8s cluster using the app blueprint <em>CFE-K8s-Ubuntu</em>:</p> <ol> <li>Navigate to <strong>Provisioning</strong> -> <strong>Apps</strong>. Click <em><strong>+Add</strong></em></li> <li>Select blueprint <em>CFE-K8S-UBUNTU</em>. Click <em><strong>Next</strong></em>.</li> </ol> <p><img src="/img/k8s-app-template.png" alt=""></p> <ol start="3"> <li>Enter NAME as <em>CEF-K8s</em> and select GROUP, DEFAULT CLOUD, and ENVIRONMENT. Click <em><strong>Next</strong></em>.</li> </ol> <p><img src="/img/k8s-app-summary.png" alt=""></p> <ol start="4"> <li>Click the config under <em>CFE-K8s-master</em> and wait for green check mark to appear (this indicates that all entries are up to date).</li> </ol> <p><img src="/img/k8s-app-master-status.png" alt=""></p> <ol start="5"> <li>Click the config under <em>CFE-K8s-worker</em> and wait for green check mark to appear (this indicates that all entries are up to date). Click <em><strong>Next</strong></em>.</li> </ol> <p><img src="/img/k8s-app-worker-status.png" alt=""></p> <ol start="6"> <li>Review and click <em><strong>Complete</strong></em>.</li> </ol> <p><img src="/img/k8s-details.png" alt=""></p> <p>After a few minutes, the K8s cluster <em>CFE-K8s</em> is successfully provisioned and displays a <em>Running</em> status.</p> <p><img src="/img/k8s-cluster.png" alt=""></p> <ol start="7"> <li>Click cluster <em>CFE-K8s</em>.</li> </ol> <p><img src="/img/k8s-cluster-details.png" alt=""></p> <p>The K8s cluster <em>CFE-K8s</em> has been provisioned using the app blueprint <em>CFE-K8s-Ubuntu</em> with 2 instances.</p> <h2>Access K8s cluster</h2> <p>Provision an Ubuntu VM instance with <em>kubectl</em> and <em>helm</em> installed, and set it up as jumphost by adding a <em>DNAT</em> rule to the <em>Router</em>.</p> <p><img src="/img/k8s-jumpserver.png" alt=""></p> <p>Copy the <em>kubeconfig</em> file of the K8s cluster 'CFE-K8s' from its master node at IP <em>172.20.20.116</em>, and save it locally as <em>config</em>. You can then access the cluster 'CFE-K8s' using the command <em>'kubectl --kubeconfig=./config get nodes'</em>.</p> <p><img src="/img/k8s-access.png" alt=""></p> <h2>Delete K8s cluster</h2> <p>Perform the following steps to remove the K8s cluster <em>CFE-K8s</em> once it is no longer required.</p> <ol> <li>Navigate to <strong>Provisioning</strong>-> <strong>Apps</strong>.</li> <li>Select <em>CFE-K8s</em> and click <em><strong>DELETE</strong></em>.</li> </ol> <p><img src="/img/k8s-delete.png" alt=""></p> <p>After a few minutes, the cluster is successfully deleted, and all associated VM instances are removed.</p> <h2>Conclusion</h2> <p>This blog post provided a step-by-step walkthrough for provisioning a K8s cluster using an app blueprint integrated with Ansible in the HPE Private Cloud Enterprise environment. With the support of the Morpheus Kubernetes Service (MKS), HPE Private Cloud Enterprise now empowers users to deploy and manage K8s clusters with built-in automation and observability capabilities.</p> <p>You can refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a> for a guide to provisioning MKS clusters using predefined MKS cluster layouts. Whether you prefer the flexibility of app blueprints or the streamlined structure of cluster layouts, HPE Private Cloud Enterprise gives you the freedom to choose the approach that best fits your operational needs.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/provisioning-k3s-clusters-using-custom-cluster-layouts-in-hpe-private-cloud-enterprise/https://developer.hpe.com/provisioning-k3s-clusters-using-custom-cluster-layouts-in-hpe-private-cloud-enterprise/Fri, 26 Sep 2025 15:47:53 GMT<p>This blog post outlines the steps to create a custom cluster layout for provisioning a Kubernetes (K8s) cluster using <em>K3s</em>, a lightweight K8s distribution, within the HPE Private Cloud Enterprise environment. By utilizing a list of key Morpheus components, such as <em>Node Type</em>, <em>File Template</em>, <em>Option List</em>, <em>Input</em>, <em>Task</em>, <em>Workflow</em>, and <em>Cluster Layout</em>, a custom cluster layout that incorporates the <em>K3s</em> install script can be created. Once configured, this custom cluster layout enables provisioning and management of K3s clusters directly from the the Morpheus Clusters page.</p> <p>Like the Morpheus Kubernetes Service (MKS), K3s clusters benefit from a curated set of built-in operations, including kubeconfig download, cluster scaling, K3s version upgrade, and cluster cleanup. These integrated capabilities simplify and streamline cluster management tasks, making K8s administration more efficient and user-friendly in HPE Private Cloud Enterprise.</p> <h2>Overview</h2> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> is a fully managed <em>Infrastructure as a Service</em> (IaaS) offering that brings a modern, cloud-like experience to on-premises environments. It combines the flexibility of hybrid cloud with the enterprise-grade control and security required by enterprise IT.</p> <p>Through the integration with <a href="https://www.hpe.com/us/en/morpheus-enterprise-software.html">HPE Morpheus Enterprise</a>, which serves as the cloud management and orchestration layer, HPE Private Cloud Enterprise offers a unified self-service interface for provisioning virtual machines (VMs), creating containers, and deploying applications, all governed by role-based access control (RBAC). This integration now supports the Morpheus Kubernetes Service (MKS), enabling users to provision and manage MKS clusters using a set of prebuilt MKS cluster layouts based on the native K8s distribution. To learn more about provisioning MKS clusters in HPE Private Cloud Enterprise, refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a>. Additionally, users can define custom cluster layouts to provision K8s clusters using third-party K8s distributions such as <em>Amazon EKS Anywhere</em> or <em>K3s</em>.</p> <p>The following sections will guide you through the process of creating a custom cluster layout and using it to provision a K3s cluster within HPE Private Cloud Enterprise. Once the cluster is provisioned, a curated list of built-in operations becomes available from the cluster's <em>Actions</em> menu. Among these, you will learn how to upgrade K3s version using one of the supported actions. These integrated features streamline key cluster management tasks, making cluster administration easier, faster, and more consistent.</p> <h2>Prerequisites</h2> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>Access to an HPE Private Cloud Enterprise workspace with the '<em>Private Cloud Tenant Owner'</em> role, allowing administrative actions in the <em><strong>Virtual Machines</strong></em> service.</li> <li>The group named <em>'Department B Group'</em> and the network <em>'Green_network'</em> have already been created.</li> <li>HPE Morpheus Enterprise running <em>version 8.0.5</em> or higher.</li> <li>The MKS feature is enabled in HPE Private Cloud Enterprise. You can confirm the presence of the <em>Clusters</em> menu from the <em><strong>Infrastructure</strong></em> tab.</li> </ul> <h2>Create custom cluster layout</h2> <p>To create a custom cluster layout, you need to log in to <em>HPE GreenLake Cloud</em> and launch the <em>HPE Morpheus Enterprise Dashboard</em>. For a detailed walkthrough of this process, refer to the blog post <a href="https://developer.hpe.com/blog/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/">Provisioning MKS clusters in HPE Private Cloud Enterprise</a>.</p> <h3>1. Create node types</h3> <p>From the HPE Morpheus Enterprise <strong>Dashboard</strong>, perform the following steps to create node types for both the K3s master and worker nodes.</p> <h4>Create a K3s primary master node type</h4> <ul> <li>Navigate to **Library > Blueprints > ** <em>Node Types</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>k3s-primary-master</em>, choose <em>Virtual Image</em> option and select VM IMAGE as <em>Morpheus Ubuntu 20.04 20250218 (vmdk) - 1178</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-node-type-primary-master.png" alt=""></p> <h4>Create a K3s secondary master node type</h4> <ul> <li>Navigate to **Library > Blueprints > ** <em>Node Types</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>k3s-secondary-master</em>, choose <em>Virtual Image</em> option and select VM IMAGE as <em>Morpheus Ubuntu 20.04 20250218 (vmdk) - 1178</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-node-type-secondary-master.png" alt=""></p> <h4>Create a K3s worker node type</h4> <ul> <li>Navigate to **Library > Blueprints > ** <em>Node Types</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>k3s worker</em>, choose <em>Virtual Image</em> option and select VM IMAGE as <em>Morpheus Ubuntu 20.04 20250218 (vmdk) - 1178</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-node-type-worker.png" alt=""></p> <h3>2. Create file template</h3> <h4>Create a file template with K8s Secret YAML</h4> <ul> <li>Navigate to **Library > Templates > ** <em>File Templates</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME and FILE NAME, specify FILE PATH, select PHASE and provide TEMPLATE. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/file-template.png" alt=""></p> <h4>Configure the K3s primary master node</h4> <ul> <li>Navigate to **Library > Blueprints > ** <em>Node Types</em> tab. Click <em>Edit</em> to <em>k3s-primary-master</em>.</li> <li>Under <strong>VMware VM Options</strong>, search and add <em>morpheus-sa-file-template</em> to FILE TEMPLATES. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/primary-master-node-type-edit.png" alt=""></p> <h3>3. Create option lists</h3> <h4>Create an option list for K3s versions</h4> <ul> <li>Navigate to **Library > Options > ** <em>Option Lists</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Versions</em>, select TYPE as <em>Manual</em> and provide DATASET. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/option-list-k3s-version.png" alt=""></p> <h4>Create an option list for K3s networking</h4> <ul> <li>Navigate to **Library > Options > ** <em>Option Lists</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Networking Options</em>, select TYPE as <em>Manual</em> and provide DATASET. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/option-list-k3s-networking.png" alt=""></p> <h3>4. Create inputs</h3> <h4>Create an input for K3s version</h4> <ul> <li>Navigate to **Library > Options > ** <em>Inputs</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Version</em> and FIELD NAME as <em>k3sversion</em>, set REQUIRE FIELD and HELP BLOCK, select TYPE as <em>Select List</em> with LABEL as <em>Version</em> and choose OPTION LIST as <em>K3s Versions</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/input-k3s-version.png" alt=""></p> <h4>Create an input for K3s networking</h4> <ul> <li>Navigate to **Library > Options > ** <em>Inputs</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Networking</em> and FIELD NAME as <em>k3snetworking</em>, set REQUIRE FIELD and HELP BLOCK, select TYPE as <em>Select List</em> with LABEL as <em>Networking</em> and choose OPTION LIST as <em>K3s Networking Options</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/input-k3s-networking.png" alt=""></p> <h4>Create an input for K3s cluster VIP address</h4> <ul> <li>Navigate to **Library > Options > ** <em>Inputs</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Cluster VIP Address</em> and FIELD NAME as <em>k3svipaddress</em>, set REQUIRE FIELD and HELP BLOCK, and select TYPE as <em>Text</em> with LABEL as <em>VIP Address</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/input-k3s-vip.png" alt=""></p> <h4>Create an input for K3s cluster Classless Inter-Domain Routing (CIDR)</h4> <ul> <li>Navigate to **Library > Options > ** <em>Inputs</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Cluster CIDR</em> and FIELD NAME as <em>k3sclustercidr</em>, set REQUIRE FIELD and HELP BLOCK, and select TYPE as <em>Text</em> with LABEL as <em>Cluster CIDR</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/input-k3s-cluster-cidr.png" alt=""></p> <h4>Create an input for K3s service CIDR</h4> <ul> <li>Navigate to **Library > Options > ** <em>Inputs</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s Service CIDR</em> and FIELD NAME as <em>k3sservicecidr</em>, set REQUIRE FIELD and HELP BLOCK, and select TYPE as <em>Text</em> with LABEL as <em>Service CIDR</em>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/input-k3s-service-cidr.png" alt=""></p> <h3>5. Create automation task and workflow</h3> <h4>Create an automation task using K3s install script</h4> <ul> <li>Navigate to **Library > Automation > ** <em>Tasks</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s HA Install Script</em>, select TYPE as <em>Shell Script</em>, choose <em>SUDO</em> option, select SOURCE as <em>Local</em>, and provide CONTENT. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-install-task.png" alt=""></p> <p><strong>Note:</strong> The sample K3s HA install script is available from the <a href="https://github.com/GuopingJia/k3s-demo/blob/main/K3s-Install-Script.sh"><em>GitHub</em> repository</a>.</p> <h4>Create an automation workflow</h4> <ul> <li>Navigate to <strong>Library</strong> > <strong>Automation</strong> > <em>Workflows</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3s HA Install</em>, select PLATFORM as <em>Linux</em>, search and add <em>K3s HA Install Script</em> under <strong>Provision</strong>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-install-workflow.png" alt=""></p> <h3>6. Create K3s custom cluster layouts</h3> <p>The following section outlines the creation of two custom cluster layouts: one for a K3s high-availability (HA) setup and another for a single-master K3s configuration.</p> <h4>Create a K3s HA cluster layout</h4> <ul> <li>Navigate to <strong>Library</strong> > <strong>Blueprints</strong> > <em>Cluster Layouts</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3S HA Cluster</em>, select CLUSTER TYPE as <em>Kubernetes Cluster</em> and TECHNOLOGY as <em>VMware</em>, search and add <strong>Inputs</strong>, <strong>Master Nodes</strong> and <strong>Worker Nodes</strong>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-ha-cluster-layout.png" alt=""></p> <h4>Create a single-master K3s cluster layout</h4> <ul> <li>Navigate to <strong>Library</strong> > <strong>Blueprints</strong> > <em>Cluster Layouts</em> tab. Click <em><strong>+Add</strong></em>.</li> <li>Enter NAME as <em>K3S Single-Master Cluster</em>, select CLUSTER TYPE as <em>Kubernetes Cluster</em> and TECHNOLOGY as <em>VMware</em>, search and add <strong>Inputs</strong>, <strong>Master Nodes</strong> and <strong>Worker Nodes</strong>. Click <em><strong>Save changes</strong></em>.</li> </ul> <p><img src="/img/k3s-single-master-cluster-layout.png" alt=""></p> <h2>Provision a K3s cluster</h2> <p>The following section outlines the process to provision a K3s cluster using the custom cluster layout <em>K3S HA Cluster</em>.</p> <ul> <li>Navigate to <strong>Infrastructure > Clusters</strong>. Click <em><strong>+Add Cluster</strong></em>.</li> <li>Select <em>Cluster Type</em> as <em>KUBERNETES CLUSTER</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/k3s-cluster-type.png" alt=""></p> <ul> <li>Select <em>Group</em> as <em>Department B Group</em>. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/k3s-group.png" alt=""></p> <ul> <li>Select CLOUD, enter CLUSTER NAME as <em>k3s-ha</em>, and configure optional DESCRIPTION, RESOURCE NAME and LABELS. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/k3s-cluster-name.png" alt=""></p> <ul> <li>Select LAYOUT as <em>K3S HA Cluster</em> and PLAN, configure VOLUMES, select NETWORKS, VERSION and NETWORKING, and configure VIP ADDRESS, CLUSTER CIDR and SERVICE CIDR. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/k3s-cluster-master-config.png" alt=""></p> <ul> <li>Specify NUMBER OF WORKERS, along with PLAN, VOLUMES, and NETWORKS. You may retain the default settings or reuse the values previously configured. Click <em><strong>Next</strong></em>.</li> </ul> <p><img src="/img/k3s-cluster-worker-config.png" alt=""></p> <ul> <li>Skip this step for Automation settings. The cluster review screen displays. Click <em><strong>Complete</strong></em>.</li> </ul> <p><img src="/img/k3s-cluster-review.png" alt=""></p> <p>After a few minutes, the K3s cluster <em>k3s-ha</em> is created using the specified custom cluster layout: <em>K3S HA Cluster</em>.</p> <p><img src="/img/k3s-ha-cluster.png" alt=""></p> <h2>Verify K3s cluster</h2> <ul> <li>Navigate to <strong>Infrastructure > Clusters</strong>.</li> <li>Click the K3s cluster <em>k3s-ha</em>.</li> </ul> <p><img src="/img/k3s-ha-cluster-details.png" alt=""></p> <p>The K3s cluster page displays the count of controller and worker nodes, all showing an <em>Ok</em> status.</p> <ul> <li>Click the <em>History</em> tab.</li> </ul> <p><img src="/img/k3s-ha-cluster-history.png" alt=""></p> <p>It contains all the details of the K3s cluster provisioning process, starting from the master nodes to the worker nodes.</p> <h2>Access K3s cluster</h2> <ul> <li>Navigate to <strong>Infrastructure > Clusters</strong>.</li> <li>Click the K3s cluster <em>k3s-ha</em>.</li> <li>From <em>Control</em> tab, type command, e.g., <em>get nodes</em>.</li> </ul> <p><img src="/img/k3s-ha-cluster-access.png" alt=""></p> <p>It shows both master and worker nodes in the cluster, together with their K8s versions and status.</p> <p>From the cluster's <em><strong>Actions</strong></em> menu, you can click <em>Upgrade Cluster</em> to upgrade the K3s cluster to its new version, e.g., <em>1.31.13</em>.</p> <p><img src="/img/k3s-ha-cluster-upgrade.png" alt=""></p> <h2>Conclusion</h2> <p>This blog post walked through the process of creating custom cluster layouts and using them to provision K3s clusters within HPE Private Cloud Enterprise. Supporting both native and third-party K8s distributions, along with preconfigred MKS cluster layouts and user-defined custom cluster layouts, HPE Private Cloud Enterprise offers flexible provisioning options to suit diverse deployment needs. Once a cluster is operational, administrators can simplify day-to-day management using the built-in <em>Actions</em> menu, enabling downloading kubeconfig files, scaling worker nodes, upgrading cluster version, and decommissioning unused clusters. These integrated capabilities make K8s administration more streamlined, reliable, and intuitive across the enterprise.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/self-activation-and-onboarding-in-hpe-complete-care-service-–-itops/https://developer.hpe.com/self-activation-and-onboarding-in-hpe-complete-care-service-–-itops/Fri, 26 Sep 2025 10:01:32 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>One of the HPE Complete Care Service - ITOps slogans is <strong>Smart decisions start with good data</strong>. HPE Complete Care service - ITOps is an HPE service that helps customers monitor, manage, and operate their entire IT environment regardless of the location of those IT assets — on-premises or cloud native.</p> <p>This service includes a cloud-based observability service, powered by <a href="https://www.hpe.com/us/en/opsramp.html">HPE OpsRamp Software</a>, a SaaS-based IT Operations Management (ITOM) solution. HPE OpsRamp Software is an AI-powered command center that simplifies and optimizes IT operations. HPE OpsRamp Software gives customers complete control over their hybrid IT environment to discover, map, monitor, and troubleshoot infrastructure health and service performance, all from a single centralized command center.</p> <p>HPE OpsRamp Software leverages OpenTelemetry and the extended Berkeley Packet Filter (eBPF) technologies to collect telemetry data. It enables real time monitoring, spot anomalies, predict potential issues, so customers can stay ahead of performance issues before they impact their business.</p> <p>HPE Complete Care service – ITOps provides customers with enterprise licenses, 24x7 access to HPE OpsRamp platform and 24x7 access to HPE OpsRamp experts. The number of licenses is determined by the configuration, such as the number of devices.</p> <p>The HPE Account Service Manager (ASM) will reach out to the HPE Complete Care customers to gather information from the customers to configure the observability service enabled by HPE OpsRamp software and begin the onboarding process of the HPE OpsRamp Software command center.</p> <p>After you get access to the HPE OpsRamp Software command center for the HPE Complete Care service – ITOps, you can get started with the journey to <strong>self-activate</strong> the observability service enabled by HPE OpsRamp Software.</p> <p>This blog post walks you through the sequence of steps through a series of <strong>audio-visual learning experience</strong> that helps you get started with the setup of the HPE OpsRamp Software command center to discover, monitor, and observe the health, performance and availability of systems and devices included in your HPE Complete Care Service – ITOps contract.</p> <h2>Assumptions</h2> <p>In this scenario, it is assumed that the HPE Account Service Manager (ASM) has reached out to you to complete the HPE Complete Care Service – ITOps request form (provided by your HPE ASM) to setup and configure your HPE OpsRamp Software command center instance.</p> <h2>Prerequisites</h2> <p>Once you get your HPE OpsRamp Software command center instance, for a <strong>self-activation</strong> of the instance to monitor, control and manage resources, there are some setup requirements that must be fulfilled:</p> <ol> <li> <p>Installing a gateway collector appliance in a virtual environment as a prerequisite to enable discovery of systems and resources before they can be monitored.</p> </li> <li> <p>Installing integration modules to discover, monitor and manage agentless SSH-enabled resources and physical compute devices.</p> </li> <li> <p>Creating and customizing modern dashboards to easily identify anomalies and troubleshoot issues quickly.</p> </li> </ol> <h2>Installing and configuring a gateway collector appliance</h2> <p>Resources need to be discovered before they can be monitored, and metrics are collected. With the HPE OpsRamp Software, you discover, monitor, and manage infrastructure resources (compute, storage, network) using an <strong>agentless</strong> method with a <strong>gateway collector appliance</strong> installed <strong>within your firewall</strong> environment. This appliance can be a virtual machine or a cloud-native application that runs on your own Kubernetes environment.</p> <p>There is <a href="https://www.youtube.com/watch?v=c0ZmdwACq2A&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">a short video on YouTube</a> that walks through the installation of the gateway collector appliance in the HPE Complete Care Service – ITOps command center instance. For details about installing and registering a gateway collector appliance, you can refer to the blog post <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/">Initial configuration to enable the discovery of resources</a>, section <em><strong>Installing and configuring a gateway collector appliance</strong></em>.</p> <p>To learn more about the gateway collector appliance installation and activation procedures, and deployment requirements, refer to the <a href="https://docs.opsramp.com/platform-features/">HPE OpsRamp Software documentation</a>.</p> <blockquote> <p><strong>Note:</strong> Agent-based methods can also be used for the discovery of resources running Linux and Microsoft Windows Operating Systems. To learn more about agents, refer to the <a href="https://docs.opsramp.com/platform-features/agents/">HPE OpsRamp Software Agent documentation</a>.</p> </blockquote> <h2>Installing an agentless SSH-enabled integration module</h2> <p>HPE OpsRamp Software command center supports agentless monitoring. Agentless monitors use the <strong>gateway collector appliance</strong> to discover resources via SSH and monitor IT infrastructure agentless resources to track their health, performance, and availability.</p> <p>The SSH Agentless Integration module discovers Linux/Unix-based systems <strong>without installing an agent</strong>, by securely connecting to the device over SSH via the gateway collector appliance. A <strong>Monitoring Template</strong> then needs to be assigned to the target agentless system for collecting data for monitoring the metrics and resource availability.</p> <p>There is <a href="https://www.youtube.com/watch?v=a1GVV-b9hCI&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">a short video on YouTube</a> that walks through the installation of the agentless SSH-enables system integration module in the HPE Complete Care Service – ITOps command center instance. For details about installing and configuring an agentless SSH integration module, you can refer to the blog post <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">Enabling the monitoring of agentless SSH-enabled systems</a>.</p> <h2>Discovering and monitoring physical computing devices via the Redfish API</h2> <p>The <a href="https://docs.opsramp.com/integrations/compute/server-hardware-monitoring-redfish/redfish-server/">Redfish - Server integration module</a> monitors and manages physical servers via the Redfish API, a modern, standardized interface for out-of-band hardware management.</p> <p>The Redfish Server integration module enables the <strong>discovery</strong> of physical server hardware and its components (CPU, memory, storage, power supplies, fans, temperature, and so on). Once discovered, Redfish Server monitors are <strong>automatically</strong> applied to the resource via predefined <strong>Global Monitoring Templates</strong> and <strong>Global Device Management Policies</strong> to manage the health and status of the server’s hardware components.</p> <p>There is <a href="https://www.youtube.com/watch?v=htZwkW-zG00&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">a short video on YouTube</a> that walks through the installation of the Redfish - Server integration module in the HPE Complete Care Service – ITOps command center instance. For details about installing a Redfish – Server integration module, you can refer to the blog post <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">Enabling the monitoring of physical devices</a>.</p> <h2>Creating and customizing modern dashboards</h2> <p>Once monitoring tools have collected metrics, the next step in creating actionable insights is to visualize them in a dashboard. A dashboard is a collection of charts based on tiles and widgets for visualizing metrics data measured over intervals of time.</p> <p>The purpose of using and creating customizable dashboards is to easily identify anomalies so IT operations can identify them and troubleshoot issues quickly.</p> <p>HPE <strong>recommends leveraging the modern dashboard 2.0</strong> for its advanced capabilities.</p> <p>There is <a href="https://www.youtube.com/watch?v=MPTq-3EA60E&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">a short video on YouTube</a> that walks through the creation and customization of a modern dashboard 2.0 in the HPE Complete Care Service – ITOps command center instance. For details about creating and customizing a modern dashboard 2.0, you can refer to the blog post <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">Enabling the monitoring of physical devices</a>, section <em><strong>Dashboard</strong></em>.</p> <p>To learn more about <strong>dashboard 2.0</strong>, see the <a href="https://docs.opsramp.com/platform-features/feature-guides/dashboards/dashboard20/">HPE OpsRamp Software Dashboards documentation</a>.</p> <h2>Summary</h2> <p>The HPE Complete Care Service – ITOps, powered by HPE OpsRamp Software, discovers the compute, networks, and storage infrastructure, the applications, and workloads they host, and their dependencies. It then observes and monitors the health, performance, and capacity events, metrics, logs, traces, and network flows providing customers with true end-to-end visibility.</p> <p>This blog post walked you through the sequence of steps, leveraging a series of video tutorials, for a self-activation of the HPE Complete Care Service – ITOps command center instance. This set of videos goes through the setup of the HPE OpsRamp Software command center to discover, monitor, and observe the health, performance, and availability of resources through a gateway collector appliance, and creation of a modern dashboard 2.0.</p> <p>For <strong>the HPE-assisted activation and onboarding of the HPE Complete Care Service – ITOps</strong>, refer to the <a href="https://developer.hpe.com/platform/activation-and-onboarding-in-hpe-complete-care-service-itops/home/">HPE Complete Care Service – ITOps landing page</a>.</p> <p>To resolve issues with HPE Complete Care Service – ITOps command center instance, contact the HPE support team. Refer to the <a href="https://www.hpe.com/us/en/collaterals/collateral.a50009342enw.html">HPE Contractual Support service documentation</a> for detailed information.</p>https://developer.hpe.com/open-sourcing-workshops-on-demand-part-5-create-a-workshop/https://developer.hpe.com/open-sourcing-workshops-on-demand-part-5-create-a-workshop/Fri, 26 Sep 2025 08:24:03 GMT<p>In this article that is part of our series dedicated on <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">open sourcing of our Workshops-on-Demand project</a>, I will focus on the steps necessary to build up a new workshop. In my previous posts, I have already covered most of the pieces on how to set up the infrastructure to support the workshops. Now let's focus a little more on the content creation.</p> <h1>Overview</h1> <p>Let's start with a simple flowchart describing the 10000-foot view of the creation process:</p> <p><img src="/img/wod-b-process.png" alt="" title="Workshop&#x27;s creation flow."></p> <p>As you can see, there's no rocket science here. Just common sense. Depending on the workshop you wish to create, some obvious requirements should show up. A workshop based on a programmatic language, for instance, may require the relevant kernel to be set up on the JupyterHub server. The following <a href="https://gist.github.com/chronitis/682c4e0d9f663e85e3d87e97cd7d1624">page</a> lists all available kernels.</p> <p>Some workshops might need a specific infrastructure set up in order to run. A Kubernetes 101 workshop, for instance, could not exist without the presence of a proper Kubernetes cluster. The same thing goes for any HPE-related solutions.</p> <p>In setting up the infrastructure, there are a number of things you need at a minimum. The design of this open-sourcing project makes it very easy to deploy a development and test environment, a staging environment, and at least one production environment.</p> <p>Let's look at how each of these environments is defined :</p> <ol> <li><strong>Development Environment:</strong> This is where application/system development tasks, such as designing, programming, debugging of a workshop, etc., take place.</li> <li><strong>Test Environment:</strong> As the name implies, this is where the workshop testing is conducted to find and fix errors.</li> <li><strong>Staging Environment:</strong> Here, all the work done in the development environment is merged into the built system (often used to automate the process of software compilation) before it is moved into the production environment.</li> <li><strong>Production Environment:</strong> The last environment in workshop development, this is where new builds/updates are moved into production for end users.</li> </ol> <p>When the HPE Developer Community began implementing their Workshop-on-Demand program, they originally only had a development and a test &#x26; staging environment on one end and a production environment on the other.</p> <p>In this post, I won't focus on the subject selection process. I'll leave that to you to figure it out. I will, however, talk a little bit again about the infrastructure, especially the dedicated scripts and variables that you need to create to support the lifecycle of the workshop. As usual, there are two sides to the workshop's creation--what should be done on the backend and what needs to be done mainly for the api db server.</p> <p><img src="/img/wod-blogserie3-archi3.png" alt="" title="WOD Overview."></p> <h2>What is a workshop? What do you need to develop?</h2> <p>Let me use an example to better explain this. There's this engineer, Matt, who has a great deal of knowledge that he would like to share. He was kind enough to agree to working with me on creating a new workshop. After our first meeting, where I explained the creation process and the expectations, we were able to quickly start designing a workshop that would help him do this.</p> <p>We defined what was needed:</p> <ul> <li>A set of notebooks that will be used by the student</li> <li>Containing instructions cells in markdown and run code cells leveraging the relevant kernel. If you are not familiar with Jupyter notebooks, a simple <a href="https://developer.hpe.com/hackshack/workshop/25">101 workshop</a> is available in our Workshops-on-Demand 's catalog.</li> <li>A student range for the workshop.</li> </ul> <p>A workshop should contain at least:</p> <ul> <li>0-ReadMeFirst.ipynb</li> <li>1-WKSHP-LAB1.ipynb</li> <li>2-WKSHP-LAB2.ipynb</li> <li>3-WKSHP-Conclusion.ipynb</li> <li>LICENCE.MD</li> <li>A pictures folder (if any screenshot is required in lab instructions)</li> <li>A README.md (0-ReadMeFirst.ipynb in md format)</li> <li>A wod.yml</li> </ul> <p>To make the workshop compliant to our platform, Matt just needs to provide a final file that contains a set of metadata that will be used for the workshop's integration into the infrastructure. this file is called <strong>wod.yml</strong>.</p> <p>Matt can leverage a simple <strong>wod.yml</strong> file containing them and that can be later parsed in order to feed the database with the relevant info. Quite handy, no? Moreover, the same script that will create the workshop entry in the database can also be used to update it.</p> <p>Here is an example of such a file:</p> <pre><code class="language-yaml">%YAML 1.1 # Meta data for the GO101 Workshop to populate seeder --- name: 'GO 101 - A simple introduction to Go Programming Language' description: 'Go, also called Golang or Go language, is an open source programming language that Google developed. Software developers use Go in an array of operating systems and frameworks to develop web applications, cloud and networking services, and other types of software. This workshop will drive you through the basics of this programming language.' active: true capacity: 20 priority: 1 range: [151-170] reset: false ldap: false location: 'fully qualified domain name of default JupyterHub server' replayId: 31 varpass: false compile: false workshopImg: 'https://us-central1-grommet-designer.cloudfunctions.net/images/frederic-passeron-hpe-com/WOD-GO-101-A-simp-introduction-to-Go-programming-language.jpeg' badgeImg: 'https://us-central1-grommet-designer.cloudfunctions.net/images/frederic-passeron-hpe-com/go101-a-simple-introduction-to-go-programming-language.jpg' beta: false category: ['Open Source'] duration: 4 alternateLocation: ['fully qualified domain name of an alternate JupyterHub server'] presenter: 'Matthew Doddler' role: 'FullStack developer' avatar: '/img/SpeakerImages/MattD.jpg' replayLink: 'https://hpe-developer-portal.s3.amazonaws.com/Workshops-on-Demand-Coming-Soon-Replay.mp4' </code></pre> <p>The following file will be used to update the <strong>workshops table</strong> in the database. Let's have a look at what a new entry could look like:</p> <p><img src="/img/wod-db-go-1.png" alt="" title="GO 101 Workshop DB screenshot"></p> <p><img src="/img/wod-db-go-2.png" alt="" title="GO 101 Workshop DB screenshot"></p> <p>As a contributor, Matt should be able to provide all the following details.</p> <ul> <li><strong>ID:</strong> A workshop ID to be used by backend server automation and Replays table to reference the associated replay video of the workshop (automatically created at the import of the wod.yml file process)</li> <li><strong>name:</strong> The workshop's name as it will will be displayed on the registration portal</li> <li><strong>notebook:</strong> The name of the folder containing all the workshop's notebooks (automatically created at the import of the wod.yml file process)</li> <li><strong>description:</strong> The workshop's abstract as it will will be displayed on the registration portal</li> <li><strong>avatar, role and replayLink</strong> are superseded by entries in the replay table (I will explain later)</li> <li><strong>replayId:</strong> This entry links the dedicated replay video to the workshop and enables the presence of the replay in the learn more page of the workshop (automatically created at the import of the wod.yml file process)</li> <li><strong>category:</strong> The workshops' registration portal proposes several filters to display the catlog's content. You can view all workshops, the most poular ones, or by category. Use this field to sort workshops accordingly.</li> <li><strong>duration:</strong> All workshops are time bombed. You will define here the time allocated to perform the workshop</li> <li><strong>active:</strong> Tag to set to enable visibility of the workshop's tile in the registration portal</li> <li><strong>workshopImg:</strong> As part of the lifecycle of the workshop, several emails are sent to the student. A workshop image is embedded in the first emails</li> <li><strong>session type:</strong> Workshops-on-Demand by default (automatically created at the import of the wod.yml file process)</li> </ul> <p>The following fields are required by the infrastructure. In this example, I will work as the infrastructure Admin with Matt to define them.</p> <ul> <li><strong>capacity:</strong> The number of maximum concurrent students allowed to take on the workshop</li> <li><strong>range:</strong> The range between which students get picked at registration time</li> <li><strong>reset and ldap</strong> entries are to be used by backend server automation if dedicated reset scripts and ldap authentication are required by the workshop</li> <li><strong>location:</strong> If your setup includes multiple JupyterHub servers, use this field to allocate workshops according to your needs.</li> <li><strong>compile:</strong> This entry will be filled with the name of a script to be compiled at deployment time. This feature allows the admin to hide login scripts and credentials in non-editable executable files.</li> <li><strong>varpass:</strong> This defines whether or not a workshop requires a password variable needs to be leveraged</li> <li><strong>badgeImg:</strong> As part of the lifecycle of the workshop, several emails are sent to the student. In the final email, a badge is included. It allows the student to share its accomplishment on social media like linkedin for instance.</li> <li><strong>beta:</strong> Not implemented yet :-)</li> <li><strong>alternateLocation:</strong> (Future development) The purpose is to allow automation of the relocation of a workshop in case of primary location's failure</li> <li><strong>replayLink:</strong> YouTube link of the recorded video to be used as a replay</li> <li><strong>replayid:</strong> This ID is used to link the correct video to the workshop. This is the replayId present in the workshops table (automatically created at the import of the wod.yml file process)</li> <li><strong>monoAppliance:</strong> Some workshops require a single dedicated appliance</li> <li><strong>multiAppliance:</strong> Some workshops require multiple dedicated appliances</li> </ul> <p><em><strong>Note:</strong></em> Both WorkshopImg and BadgeImg are delivered by the frontend web server.</p> <p>If you feel you need more details about the registration process, please take a look at the <strong>Register Phase</strong> paragraph in <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">the following introductionary blog</a>.</p> <p>Matt will create a simple workshop that does not require any infrastructure but the JupyterHub itself. As far as the infrastructure's requirements, a new kernel was needed. No additional scripts were required for this workshop.</p> <p>As an admin of the Workshops-on-Demand infrastructure, I have to perform several tasks on a development environment and a staging environment:</p> <h3>On the backend server:</h3> <p>Testing and validating installation of the new kernel on the staging backend server by:</p> <ol> <li>Creating a new branch for this test</li> <li>Modifying the <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/install_backend.yml#L326">backend server installation yaml file </a>to include the new kernel</li> <li>Validating the changes by testing a new backend install process</li> </ol> <p>Pushing the changes to the GitHub repo:</p> <ol> <li>Create a user for the workshop developer on the test/dev and staging backend servers</li> <li>Provide to the developer the necessary information to connect to the test/dev and staging backend servers</li> <li>Copy in the developer's home folder a workshop template containing examples of introduction, conclusion, and lab notebooks, allowing him to start his work</li> <li>Give the developer the wod-notebook repo url for him to fork the repo and work locally on his machine (when the workshop does not require an appliance but just a Jupyter kernel for instance)</li> <li>When ready, a pull request can be made. The admin can then review and accept it. The admin can then perform the necessary steps required to prepare the infrastructure to host the workshop</li> </ol> <h3>On the api-db server:</h3> <p>Connecting to the api-db server as wodadmin user:</p> <ol> <li>Switch to the relevant branch for the new workshop and perform a git remote update / rebase in the relevant notebook directory.</li> <li>Move to wod-api-db/scripts directory</li> <li>Update the database by running the wod-update-db.sh script.</li> </ol> <p>This script will update the Database workshops table with the new workshop's entry.</p> <p>As the developer of the Workshops-on-Demand content, Matt had to perform several tasks:</p> <h3>On the backend server:</h3> <ol> <li>Log on to the backend server and clone the notebook repo in his home folder, or as explained earlier, fork the repo on his laptop and work from there</li> <li>Create a new branch for his workshop following the naming convention defined with the admin</li> <li>Leverage the template provided by me to build up the content of his workshop</li> <li>Test the workshop locally on his laptop or on the dev server leveraging the <code>wod-test-action.sh</code> script</li> <li>Test the workshop using the staging registration portal</li> <li>When all tests are green, create a pull request to merge content with the master repo</li> </ol> <p>As an admin, I would need to check the pull request and accept it. Once done, the test/dev and staging environments will require an update.</p> <ol> <li> <p>Log on to the backend server as wodadmin and update the notebook repository.</p> </li> <li> <p>Run a wod-deliver to update the relevant backend server.</p> <pre><code>git remote update git rebase wod-deliver </code></pre> </li> </ol> <p>The very same processes will apply to the move to production phase.</p> <h1>Complex workshop example:</h1> <p>If you've ever had to develop a workshop for something you're not as familiar with, you'll want to meet with SMEs and explain what your goals are and how to achieve them. Once you have a clearer understanding of the technology involved, you can move on to determine what the best platform would be on which to run the workshop. I can give you an example here focusing on Ansible, an automation tool. I was not originally familiar with it before I developed the Ansible 101 Workshop. In the steps below, I'll explain how I pulled it together..</p> <p>As the admin for the HPE Developer Community's Workshops-on-Demand infrastructure, I had to perform several tasks in order to set up this workshop. I had to determine what the workshop required from a backend and fully test it.</p> <h5>On the backend server:</h5> <p>The workshop will require:</p> <ul> <li> <p>A dedicated server running Linux to allow the student to run Ansible playbooks against during the workshops' labs.</p> <ul> <li>This means preparing a server (VM or physical) : We will consider it as an appliance</li> <li>Updating the relevant variable file to associate the IP address of the server to the workshop (there could be multiple servers too associated to a given workshop)</li> </ul> </li> <li> <p>A set of scripts under wod-backend/scripts or wod-private/scripts folders depending on the nature of the workshop to manage the workshop's lifecycle</p> <ul> <li> <p>Some generic scripts applying to all workshops' appliances : general setup phase setting up common requirements for any appliance (student users creation, ssh keys, etc..) up to some specific ones dedicated to a given workshop</p> <ul> <li> <p>create-appliance.sh (ssh keys, ldap setup)</p> </li> <li> <p>setup-appliance.sh [WKSHP-NAME] (Student setup, Appliance Setup, Workshop setup) calls:</p> <ul> <li>setup-globalappliance.sh (global / generic setup)</li> <li>setup-[WKSHP-NAME].sh (Prepare appliance with workshop's reqs, Docker image for instance)</li> </ul> </li> <li> <p>create-[WKSHP-NAME].sh (called at deployement time to instantiate the necessary appliance(s) requiered by the workshop</p> </li> <li> <p>reset-appliance (reset ssh keys and students credentials on appliance</p> </li> <li> <p>cleanup-[WKSHP-NAME].sh (takes care of cleanup some workshop's specifics)</p> </li> <li> <p>reset-[WKSHP-NAME].sh (reset of the workshop's appliance, docker compose down of a container for instance)</p> </li> </ul> </li> </ul> </li> <li> <p>A set of variables to be leveraged by the notebooks. These variables are to be set in yml format. They will be parsed at deployment time to set student ids, appliance IP addresses, and other relevant parameters like ports, or simulated hardware information</p> </li> </ul> <p>When all the scripts are functional and the necessary actions have been performed both on backend and frontend servers, some functional tests can be conducted using cli and later webui as described earlier for the simple workshop example.</p> <p>Testing the workshop:</p> <ul> <li> <p>One can leverage the wod-test-action.sh script to test a workshop lifecycle action from deployment (CREATE) to CLEANUP, RESET, or PURGE.</p> <pre><code class="language-shell">dev@dev3:~$ wod-test-action.sh Syntax: wod-test-action.sh &#x3C;CREATE|CLEANUP|RESET|PURGE|PDF|WORD> WKSHOP [MIN[,MAX] ACTION is mandatory </code></pre> </li> </ul> <p><code>Note: The available trace under ~/.mail/from will detail the different steps of the action and allow you to troubelshoot any issue.</code></p> <p>When all tests have validated the workshop, it can follow the move to prod cycle.</p> <p>You should now have a better understanding of the necessary tasks associated to the creation of a workshop. As you can see, it requires steps on the various sides of the infrastructure.</p> <p>This was the last blog post of the series. The Workshops-on-Demand project is available <a href="https://github.com/Workshops-on-Demand">here</a>. Further update of the documentation will occur on this github repository.</p> <p>If we can be of any help in clarifying any of this, please reach out to us on <a href="https://developer.hpe.com/slack-signup/">Slack</a>. Please be sure to check back at <a href="https://developer.hpe.com/blog">HPE DEV</a> for a follow up on this. Also, don't forget to check out also the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a>! Willing to collaborate with us? Contact us so we can build more workshops!</p>https://developer.hpe.com/announcing-chapel-2-6/https://developer.hpe.com/announcing-chapel-2-6/Thu, 18 Sep 2025 23:37:13 GMT<p>external blog</p>https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-6-performance-of-higher-level-languages/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-6-performance-of-higher-level-languages/Thu, 18 Sep 2025 00:14:11 GMT<p>external blog</p>https://developer.hpe.com/7-questions-for-marjan-asgari-optimizing-hydrological-models-with-chapel/https://developer.hpe.com/7-questions-for-marjan-asgari-optimizing-hydrological-models-with-chapel/Mon, 15 Sep 2025 18:34:43 GMT<p>external blog</p>https://developer.hpe.com/ai-agents-as-the-meeting-whisperers-1/https://developer.hpe.com/ai-agents-as-the-meeting-whisperers-1/Tue, 09 Sep 2025 15:01:50 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 7 of our <strong>Gen AI for PM series</strong></p> <h2>Introduction</h2> <p>Meetings often move fast, ideas overlap, action items get lost, and key insights can slip through the cracks. This is where <strong>AI agents step in as the meeting whisperers</strong>.</p> <p>Working quietly in the background, they <strong>listen, summarize, and highlight what matters most</strong>, from decisions and deadlines to risks and follow-ups. Instead of spending hours sifting through notes or trying to recall what was agreed upon, project managers (PMs) get <strong>concise, structured outputs</strong> that keep everyone aligned.</p> <p>With AI handling the details, teams can focus on the conversation itself, making meetings more productive and outcomes far clearer.</p> <p><strong>Less meetings, more impact</strong></p> <p>If you’re a project manager, this stat will hit home: PMs spend <strong>30–50% of their time in meetings.</strong></p> <p>Some days, it feels like an endless relay of Zoom calls, each one blending into the next. You jot down notes, chase action items, and try to keep everyone aligned. But by the end of the week, you're plagued with meeting fatigue with precious little time left for strategic work.</p> <p>This is where the <strong>AI agent steps in as your “meeting whisperer”</strong> capturing what matters, automating the follow-up, and helping you cut meetings in half without losing alignment.</p> <p><strong>As an example:</strong> The side-by-side illustration highlights the difference AI makes in managing meetings. On the left, a frazzled project manager is shown juggling <strong>sticky notes and back-to-back Zoom calls</strong>, struggling to keep track of decisions and action items. On the right, the same manager looks calm and focused while an <strong>AI agent projects live meeting notes and action items directly onto a dashboard.</strong></p> <p>This contrast shows how <strong>AI meeting whisperers</strong> turn chaos into clarity—freeing managers from the scramble of manual note-taking and giving them structured, real-time outputs that keep the whole team aligned.</p> <center><img src="/img/7.1.png" width="600" height="550" alt="PM meeting" title="PM meeting"></center> <h3>Section 1: Agents as real-time note-takers and action trackers</h3> <p>Most PMs spend as much energy capturing what’s said in meetings as they do participating in them. That’s double work.</p> <p><strong>AI agents change this dynamic</strong></p> <ul> <li>They <strong>transcribe meetings</strong> live and highlight key points.</li> <li>They <strong>tag action items</strong> and assign them to owners.</li> <li>They <strong>log commitments</strong> into project management tools automatically.</li> </ul> <p><strong>Consider this example:</strong> In a sprint review, instead of manually typing “Dev team to fix API bug by Tuesday,” the AI agent captures it in real time and adds it to Jira, assigned to the right person, with a deadline.</p> <p>The picture below shows how AI turns meeting conversations directly into structured outputs. On the dashboard, <strong>live meeting notes</strong> are captured in real time, with <strong>key action items highlighted</strong> for clarity. These action items are then <strong>automatically logged as tasks</strong>, ensuring that nothing slips through the cracks.</p> <p>This workflow demonstrates how AI eliminates the need for manual note-taking and follow-up tracking, giving project managers a single, reliable source of truth after every meeting.</p> <center><img src="/img/7.2.png" width="600" height="550" alt="Real-time note" title="Real-time note"></center> <h3>Section 2: Auto-generating follow-up tasks</h3> <p>We’ve all been there: someone says, <em>“I’ll check the budget tomorrow,”</em> and then… it’s forgotten. Unless the PM follows up, the task vanishes into thin air.</p> <p><strong>AI agents prevent this by</strong></p> <ul> <li>Converting natural-language commitments into structured tasks.</li> <li>Adding deadlines automatically based on context.</li> <li>Sending reminders so nothing slips.</li> </ul> <p><strong>As an example:</strong> During a vendor negotiation, someone says, <em>“I’ll send the updated proposal by Friday.”</em> The AI agent creates a task, assigns it to the vendor lead, and sends a reminder on Thursday. No PM chasing required.</p> <p>The flow chart below shows how AI can transform casual spoken commitments into actionable tasks. When a team member makes a <strong>verbal commitment</strong>, for example, “I’ll send the report tomorrow” the system’s <strong>AI detection</strong> picks it up. From there, it automatically <strong>creates a task</strong> in the project management system and sets a <strong>reminder</strong>, ensuring the promise doesn’t get lost in conversation.</p> <p>This flow helps project managers capture important actions without relying on manual note-taking or memory, keeping accountability strong and projects moving forward.</p> <center><img src="/img/7.3.png" width="600" height="550" alt="Auto-generating task" title="Auto-generating task"></center> <h3>Sentiment and engagement analysis in meetings</h3> <p>Not all meeting insights are about tasks. Sometimes the tone of the conversation matters more than the words.</p> <p><strong>AI agents analyze sentiment and engagement</strong></p> <ul> <li>Flagging if a team sounds disengaged or stressed.</li> <li>Generating word clouds of recurring themes (e.g., “delays,” “handoffs,” “QA”).</li> <li>Helping PMs spot morale or alignment issues early.</li> </ul> <p><strong>As an example:</strong> In a cross-team sync, AI reports: <em>“Marketing expressed repeated concern about engineering handoffs”</em> and highlights it visually. That insight helps the PM address friction before it festers.</p> <p>The dashboard below shows how AI makes meeting insights more visual and actionable. A <strong>sentiment chart</strong> breaks down team mood into <strong>positive, neutral, and negative tones</strong>, helping leaders quickly sense the overall atmosphere. Alongside it, a <strong>word cloud</strong> highlights the most frequently mentioned terms—like “deadlines,” “QA,” or “communication”—so recurring themes stand out at a glance.</p> <p>Together, these visuals turn meeting discussions into structured intelligence, making it easier to identify concerns, celebrate wins, and take targeted follow-up actions.</p> <center><img src="/img/7.4.png" width="600" height="550" alt="AI meeting" title="AI meeting"></center> <h3>Meeting-free decision-making culture</h3> <p>The real power of AI meeting agents isn’t just in better notes — it’s in <strong>fewer meetings</strong>.</p> <p>When updates, tasks, and risks are captured automatically, you don’t need 10 weekly check-ins. Half of them can be replaced by AI-generated summaries. The remaining meetings can focus on <strong>strategic decisions, not status updates</strong>.</p> <p><strong>As an example</strong>: A software team cuts their weekly syncs from 10 to 5. Instead of rehashing status updates, they review AI-prepared summaries and spend meetings on problem-solving.</p> <p>The illustration below shows how AI meeting summaries can dramatically reduce the time spent in meetings. In the <strong>“Before”</strong> view, project managers sit through around <strong>10 meetings per week</strong>, often just to gather updates. In the <strong>“After”</strong> view, with AI generating clear, reliable summaries, the number of meetings drops to <strong>5 per week</strong>, freeing up time for deeper work while keeping everyone equally (or even better) informed.</p> <p>This comparison makes it clear: <strong>AI doesn’t just make meetings smarter—it helps cut them in half.</strong></p> <center><img src="/img/7.5.png" width="600" height="550" alt="Decision-making culture" title="Decision-making culture"></center> <h3>Conclusion: Less meeting fatigue, more results</h3> <p>Meetings will never disappear, but they don’t have to consume half your workweek. With <strong>Agentic AI as the meeting whisperer</strong>, project managers can:</p> <ul> <li>Spend less time taking notes and chasing action items.</li> <li>Gain richer insights through sentiment and engagement analysis.</li> <li>Cut meeting load without sacrificing alignment.</li> </ul> <p>The result? <strong>Less fatigue, more clarity, and better outcomes.</strong></p> <p><strong>Key takeaway</strong></p> <p><strong>"AI remembers and follows up, so PMs don’t have to -- freeing them to focus on leadership, not logistics."</strong></p> <p>This scene captures how AI ensures that no important decision or responsibility slips through the cracks—turning meeting outcomes into clear, actionable roadmaps without extra effort from the manager.</p> <center><img src="/img/7.6.png" width="600" height="550" alt="Future AI meeting" title="Future AI meeting"></center>https://developer.hpe.com/the-agentic-ai-command-center-for-project-managers/https://developer.hpe.com/the-agentic-ai-command-center-for-project-managers/Mon, 08 Sep 2025 14:27:27 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 6 of our <em><strong>Gen AI for PM series</strong></em></p> <p>Imagine a single screen where every moving part of your project comes together—tasks, risks, resources, and communications—all continuously monitored by AI. That’s the vision behind the Agentic AI command center.</p> <p>Rather than juggling spreadsheets, dashboards, and status meetings, through its use, project managers gain a central hub powered by intelligent agents. Each agent watches a different dimension of the project—tracking schedules, scanning for risks, analyzing sentiment, or monitoring budgets—and feeds insights into one unified view. The result is a real-time command center that helps managers move from reacting to problems to proactively steering projects toward success.</p> <p><strong>PM dashboards vs. intelligent command centers</strong></p> <p>Every project manager knows the struggle fifteen open browser tabs, dashboards from five different tools, Slack buzzing with updates, and a Gantt chart that was outdated yesterday. Traditional dashboards are useful, but they’re static snapshots to show you what happened, not what’s about to happen.</p> <p>Now imagine this instead: A single command center powered by intelligent agents, orchestrating every aspect of your project in real time. Not just a dashboard but a digital control room that listens, predicts, and suggests.</p> <p>That’s the promise of the Agentic AI command center.</p> <p>The side-by-side illustration shows the difference between today’s fragmented workflows and the future of project management with Agentic AI. On the left, a project manager is overwhelmed, juggling multiple dashboards and sticky notes, each tracking only part of the project. On the right, everything comes together in a clean, futuristic AI command center—a single screen that consolidates tasks, risks, resources, and updates in real time. This shift makes project oversight simpler, smarter, and far less stressful.</p> <center><img src="/img/6.1.png" width="600" height="550" alt="AI command center" title="AI command center"></center> <h3>Section 1: Unified view of project health with agent orchestration</h3> <p>Instead of siloed dashboards, an Agentic AI command center unifies everything. Think of it as a <strong>digital war room</strong> where specialized agents handle different dimensions of project health:</p> <ul> <li><strong>Finance agent</strong> → tracks budgets, forecasts, and alerts on overruns.</li> <li><strong>Resource agent</strong> → monitors workloads and reallocates tasks.</li> <li><strong>Risk agent</strong> → scans dependencies and external factors for early warnings.</li> <li><strong>Communication agent</strong> → tailors updates for each stakeholder group.</li> </ul> <p><strong>As an example</strong>: If a supplier misses a milestone, the Risk agent flags it, the Resource agent suggests reallocation, the Finance agent recalculates the budget, and the Communication agent updates stakeholders is all seamlessly orchestrated in the command center.</p> <p>The illustration below shows how <strong>multiple specialized AI</strong> agents work together to support project managers. Each agent—covering <strong>Finance, Resource, Risk, and Communication</strong> monitors its domain in real time and feeds insights into a <strong>central Project Management Dashboard Hub.</strong></p> <p>This circular flow ensures that no critical aspect is overlooked. Finance agents track budgets, resource agents optimize workloads, risk agents flag potential blockers, and communication agents keep stakeholders aligned. By consolidating all of this intelligence into one hub, project managers gain a <strong>360° view of the project</strong>, enabling faster, more confident decisions.</p> <center><img src="/img/6.2.png" width="600" height="550" alt="Agent orchestration" title="Agent orchestration"></center> <h3><strong>Section 2: Natural language queries for real-time insights</strong></h3> <p>Forget clicking through endless filters or waiting for reports. In the AI command center, you simply ask questions in plain language:</p> <ul> <li><em>“Which milestones are most at risk this quarter?”</em></li> <li><em>“Show me the cost impact if Vendor X slips by two weeks.”</em></li> <li><em>“Who’s overbooked for next sprint?”</em></li> </ul> <p>The system parses your question, pulls data from across tools, and gives you instant, visual insights.</p> <p><strong>As an example:</strong> A PM types, “Show me the top 3 risks for Project Alpha.” The command center instantly displays a ranked heatmap with probabilities, impact scores, and suggested mitigations.</p> <p>The illustration below shows how project management becomes more conversational with Agentic AI. A project manager simply types a question into the system, and within seconds the screen displays graphs, risk scores, and tailored recommendations. This natural, question-and-answer interaction allows managers to skip manual data digging and instead get instant, actionable insights—turning complex project analysis into a simple conversation.</p> <center><img src="/img/6.3.png" width="600" height="550" alt="NL queries" title="NL queries"></center> <h3>Section 3: Integrated decision support</h3> <p>Traditional dashboards tell you the “what.” The Agentic AI command center also tells you the <strong>“what next.”</strong></p> <p>It doesn’t just show a risk — it <strong>suggests responses</strong> based on data and simulations:</p> <ul> <li>Adjust timelines.</li> <li>Reallocate resources.</li> <li>Increase budget buffers.</li> <li>Recommend alternate vendors.</li> </ul> <p><strong>Consider this example</strong>: When a critical vendor signals a delay, the command center simulates scenarios:</p> <ul> <li><strong>Option A</strong>: Wait for the vendor → project slips by 10 days.</li> <li><strong>Option B</strong>: Switch to backup vendor → budget rises 5%.</li> <li><strong>Option C</strong>: Reallocate internal resources → timeline preserved, but lower capacity for another workstream.</li> </ul> <p>The PM chooses, but the AI provides the <strong>decision support</strong>.</p> <p>The decision-tree diagram below shows how AI helps project managers evaluate different choices when facing a delay. The decision tree outlines three possible actions:</p> <p>·   <strong>Switch Vendor</strong>, which is predicted to result in <strong>increased costs</strong>.</p> <p>·   <strong>Do nothing and accept the delay</strong>, which leads to a <strong>missed deadline</strong>.</p> <p>·   <strong>Reallocate Resources</strong>, which keeps the <strong>project on track</strong>.</p> <p>By mapping out the likely outcomes of each option, the system gives managers the clarity they need to make faster and more informed decisions.</p> <center><img src="/img/6.4.png" width="600" height="550" alt="AI decision support" title="AI decision support"></center> <h3>Security and audit logs for compliance</h3> <p>In industries like finance, healthcare, and government, compliance is just as important as delivery. The AI command center builds <strong>compliance into the workflow</strong>:</p> <ul> <li>Every decision is logged with a timestamp and rationale.</li> <li>Audit-ready reports are generated automatically.</li> <li>Sensitive data is flagged and access-controlled.</li> </ul> <p><strong>Consider this example</strong>: A central bank project team uses the command center. When budget reallocations happen, the system auto-generates an audit report with “<em>Who approved, When, Why,</em> <em>Impact Analysis.</em>” Compliance is no longer a scramble at the end — it’s baked into daily operations.</p> <p>The illustration below shows how an <strong>AI audit log</strong> keeps track of every action in a project for transparency and accountability. Each entry records the <strong>timestamp</strong>, the <strong>action taken</strong>, who initiated it—whether an <strong>AI agent</strong> or a <strong>project manager</strong>—and the <strong>status</strong> of that action. For example, task approvals and budget changes are neatly logged and marked as <strong>approved</strong>, making it easy to review decisions and maintain trust in AI-assisted project management.</p> <center><img src="/img/6.5.png" width="600" height="550" alt="Audit logs" title="Audit logs"></center> <h2>Conclusion: The “One screen to rule them all” for PMs</h2> <p>For decades, project managers have been stuck hopping between tools, spreadsheets, and dashboards. The Agentic AI command center changes that.</p> <p>It becomes the <strong>one screen to rule them all</strong> -- unifying project health, answering questions in natural language, providing decision support, and ensuring compliance.</p> <p>Instead of drowning in fragmented tools, project managers can finally focus on <strong>strategy, leadership</strong>, <strong>and outcomes.</strong></p> <p><strong>Key takeaway</strong></p> <p>AI command centers replace fragmented tool-hopping — giving project managers foresight, clarity, and control in one place.</p> <center><img src="/img/6.6.png" width="600" height="550" alt="Future AI" title="Future AI"></center>https://developer.hpe.com/agentic-ai-in-agile-smarter-sprints-faster-retros/https://developer.hpe.com/agentic-ai-in-agile-smarter-sprints-faster-retros/Fri, 05 Sep 2025 15:29:28 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 5 of our <em><strong>Gen AI for PM series</strong></em></p> <h2>Intro</h2> <p>Agile teams thrive on speed, collaboration, and adaptability—but even the best processes can get bogged down by manual tracking, missed dependencies, or lengthy retrospectives. This is where <strong>Agentic AI steps in.</strong></p> <p>By acting as a smart co-pilot, Agent AI can automatically <strong>analyze sprint progress, flag blockers in real time, and suggest adjustments</strong> to keep velocity high. When the sprint ends, it can instantly pull together <strong>retrospective insights</strong>—highlighting what worked, what slowed the team down, and how to improve next time. The result: smarter sprints and faster retros, giving teams more time to focus on building rather than chasing updates.</p> <p><strong>Agile is great — but still needs better data flow</strong></p> <p>Agile has transformed how teams build products. Short sprints, daily stand-ups, and constant feedback loops keep teams adaptive and focused. But let’s be honest — even the most disciplined Agile teams still wrestle with messy data flow.</p> <p>Burndown charts lag behind reality because updates are manual. Sprint retrospectives sometimes feel like therapy sessions without data to back insights. Backlogs balloon into chaos, with “top priority” items buried under noise.</p> <p>Agile gives us the right philosophy, but execution still demands better visibility, accuracy, and speed. That’s where Agentic AI steps in — not as a replacement for scrum masters or project manager (PMs), but as a co-pilot that keeps Agile teams lean, data-driven, and one step ahead.</p> <p>The below side-by-side comparison shows the shift from traditional sprint planning to AI-enhanced sprint planning. On the left, sticky notes and a messy board represent the manual, error-prone process that often leaves teams overwhelmed. On the right, a clean digital board powered by Agentic AI provides real-time updates, capacity forecasts, and smarter tracking—helping teams plan with confidence and adapt quickly.</p> <center><img src="/img/5.1.png" width="600" height="550" alt="Sprint Planning" title="Sprint Planning"></center> <h3>Section 1: Sprint planning with Agentic AI forecasting</h3> <p>Sprint planning is part science, part guessing game. Teams estimate velocity, but surprises (like unexpected bugs or absences) can derail the best-laid plans.</p> <p><strong>Agentic AI changes the game</strong> by forecasting sprint outcomes with remarkable accuracy:</p> <ul> <li>Analyzes <strong>historical sprint velocity</strong>.</li> <li>Considers <strong>team availability</strong> (holidays, PTO, workload).</li> <li>Factors in <strong>complexity of tasks</strong> (based on past data).</li> </ul> <p><strong>As an example</strong>: Instead of debating whether the team can handle 40 story points, an AI agent predicts, “Based on the past 6 sprints and current workload, your realistic capacity is 32 points.” Suddenly, sprint planning shifts from guesswork to data-backed decisions.</p> <p>The illustration below shows how Agentic AI can make sprint planning smarter. On the sprint board, the AI overlay highlights the <strong>predicted sprint capacity is 32 points</strong> giving teams a clear forecast of what can realistically be completed. This proactive insight helps prevent overcommitment and ensures more reliable sprint outcomes.</p> <center><img src="/img/5.2.png" width="600" height="550" alt="Sprint board" title="Sprint board"></center> <h3>Section 2: Automated burndown tracking</h3> <p>Burndown charts are critical for visibility — but too often, they’re outdated or inaccurate because they depend on manual updates.</p> <p>With Agentic AI:</p> <ul> <li>Task completions update the burndown chart in real time.</li> <li>Scope creep is flagged instantly when new work sneaks into the sprint.</li> <li>Velocity anomalies trigger alerts before they become blockers.</li> </ul> <p><strong>Consider this example</strong>: Mid-sprint, an AI agent detects that only 20% of tasks are complete halfway through the timeline. It pings the Scrum Master with: “At this rate, sprint completion probability is 60%. Recommend re-scoping backlog.”</p> <p>The chart below shows how AI can make burndown charts more insightful. Instead of only showing completed work against the original plan, the chart includes a red dotted line that projects a likely delay based on current progress. This proactive signal gives teams an early warning so they can adjust resources, scope, or priorities before the milestone is missed.</p> <center><img src="/img/5.3.png" width="600" height="550" alt="Burndown chart" title="Burndown chart"></center> <h3>Section 3: Instant retro summaries with AI sentiment analysis</h3> <p>Sprint retrospectives are gold mines for learning — but they’re also time-consuming, and insights are often anecdotal.</p> <p><strong>Agentic AI applies sentiment analysis and pattern detection</strong> to team feedback:</p> <ul> <li>It scans chat channels, retro boards, and surveys.</li> <li>Identifies recurring blockers (e.g., “deployment delays mentioned 5 times”).</li> <li>Detects team morale trends via tone of feedback.</li> </ul> <p><strong>As an example</strong>: Instead of a retro filled with vague “communication issues,” the AI summarizes:</p> <ul> <li><strong>Top blocker</strong>: Delayed QA environment setup.</li> <li><strong>Sentiment trend</strong>: Developer morale dropped 15% due to unclear requirements.</li> </ul> <p>The illustration below shows how retrospectives become more <strong>data-driven and actionable</strong> with AI support. Instead of relying only on memory or scattered notes, the system generates a <strong>summary dashboard</strong> that highlights key themes through <strong>word clouds</strong>—such as “delays,” “QA,” or “communication”—and presents <strong>sentiment graphs</strong> to show team mood over the sprint. This makes it easier for teams to spot recurring issues, celebrate wins, and agree on concrete next steps for improvement.</p> <center><img src="/img/5.4.png" width="600" height="550" alt="Retro board" title="Retro board"></center> <h3>Section 4: Backlog prioritization with AI recommendations</h3> <p>A bloated backlog is every PM’s nightmare. It often contains hundreds of tickets, each marked as “important,” while the team has only limited sprint capacity to handle them.</p> <p>Agentic AI helps by <strong>ranking backlog items</strong> based on:</p> <ul> <li>Business impact</li> <li>Dependencies</li> <li>Estimated effort</li> <li>Team capacity</li> </ul> <p><strong>As an example</strong>: Instead of arguing over priorities, the AI generates a ranked list:</p> <ol> <li>Fix payment gateway bug (high impact, high urgency)</li> <li>Optimize API response time (medium effort, high impact)</li> <li>Redesign landing page (low effort, medium impact)</li> </ol> <p>The picture below shows how the backlog transforms into a <strong>clear, data-backed roadmap</strong> with the help of Agentic AI. Each item in the backlog is automatically assigned a <strong>priority score,</strong> e.g ., 95/100 or 80/100, based on factors like impact, urgency, and dependencies. This gives teams a transparent way to see what matters most, align on priorities, and focus their effort where it delivers the greatest value.</p> <center><img src="/img/5.5.png" width="600" height="550" alt="Backlog" title="Backlog"></center> <h3><strong>Conclusion: Agile meets intelligence</strong></h3> <p>Agile taught us to work smarter, not harder. But even Agile struggles when data is fragmented or delayed.</p> <p>With <strong>Agentic AI</strong>, Agile becomes <strong>Agile + Intelligence:</strong></p> <ul> <li>Sprint planning shifts from guesswork to forecasting.</li> <li>Burndowns update themselves.</li> <li>Retros gain clarity with sentiment-driven insights.</li> <li>Backlogs transform from chaos into priority pipelines.</li> </ul> <p>This isn’t about replacing Agile roles. Scrum Masters, PMs, and Agile coaches remain essential. But now, they’re equipped with AI-powered allies who free them from busywork and give them clarity at speed.</p> <p><strong>Key takeaway</strong></p> <p>Agentic AI makes every sprint data-driven — helping Agile teams move faster, learn smarter, and deliver better.</p> <center><img src="/img/5.6.png" width="600" height="550" alt="Agile future" title="Agile future"></center>https://developer.hpe.com/2025-sep-04/https://developer.hpe.com/2025-sep-04/Thu, 04 Sep 2025 13:00:00 GMThttps://developer.hpe.com/gen-ai-for-pm-series-part-4-multi-agent-systems-for-multi-stakeholder-projects/https://developer.hpe.com/gen-ai-for-pm-series-part-4-multi-agent-systems-for-multi-stakeholder-projects/Sat, 30 Aug 2025 06:50:31 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 4 of our <em><strong>Gen AI for PM series</strong></em></p> <h2>Intro</h2> <p>Large projects often involve multiple stakeholders, each with different priorities, risks, and information needs. Managing all of these moving parts with a single tool or process quickly becomes overwhelming. This is where <strong>multi-agent systems</strong> come in.</p> <p>By assigning specialized AI agents to handle distinct responsibilities, such as scheduling, compliance, risk management, or communication, projects can scale more smoothly. Each agent works autonomously within its domain but shares insights with others, creating a <strong>coordinated ecosystem</strong> that adapts to stakeholder demands in real time. For project managers, this means fewer blind spots, faster responses, and better alignment across diverse teams.</p> <p><strong>Coordinating across departments is chaotic</strong></p> <p>If you’ve ever managed a cross-department project, you know how messy things can get. Finance is tracking budgets in spreadsheets. Engineering is sprinting toward deadlines in Jira. Marketing is waiting for updates before they can launch campaigns. And operations? They’re buried in vendor calls.</p> <p>What ties it all together? Usually, it’s the project manager(PM) stuck in the middle — drowning in emails, calendar invites, and status requests.</p> <p>This <strong>coordination chaos</strong> is where <strong>multi-agent systems</strong> step in. Instead of a single PM being the communication bottleneck, digital agents act like specialized assistants, each handling a slice of responsibility and keeping everyone aligned.</p> <p>The image compares two scenarios of project management:</p> <ul> <li><strong>Left side (“Without AI agents”)</strong><br> A project manager (PM) is overwhelmed, juggling multiple responsibilities at once Finance, Engineering, Marketing, and Operations. This represents the traditional challenge of keeping all stakeholders aligned manually.</li> <li><strong>Right side (“With Agentic AI”)</strong><br> Instead of juggling, AI agents work together seamlessly. Each ball (Finance, Engineering, Marketing, Operations) is smoothly passed around by specialized AI agents, with the PM overseeing rather than micromanaging.</li> </ul> <center><img src="/img/4.1.png" width="600" height="550" alt="Multiple department projects" title="Multiple department projects"></center> <h2>Section 1: Defining multi-agent collaboration</h2> <p>So, what is multi-agent collaboration?</p> <p>Think of it this way: Instead of one AI tool doing everything, you have multiple AI agents, each with a defined role, working together like a team. They:</p> <ul> <li>Collect data from their respective domains.</li> <li>Communicate findings automatically.</li> <li>Coordinate actions with other agents.</li> <li>Keep the PM in the loop with summaries, not overload.</li> </ul> <p>It’s like moving from a single Swiss Army knife to a team of expert specialists — each focused, each efficient, and each accountable.</p> <p><strong>Consider this example</strong>: In a construction project, a Resource Agent monitors workforce schedules, a Finance Agent tracks cost overruns, and a Compliance Agent ensures safety requirements are met. Together, they keep the project moving without endless manual updates.</p> <p>The <strong>circular diagram</strong> you mentioned is a way to visualize how <strong>Agentic AI creates a connected ecosystem</strong> around a project manager’s dashboard:</p> <ul> <li><strong>Finance Agent</strong> → Feeds budget updates, cost forecasts, and expense alerts directly into the dashboard.</li> <li><strong>Resource Agent</strong> → Monitors workload distribution, capacity, and availability of team members, updating the dashboard in real time.</li> <li><strong>Communication Agent</strong> → Pulls in updates from team channels, emails, or tools like Slack and pushes them to the dashboard as structured insights.</li> </ul> <p>At the center sits the <strong>PM Dashboard</strong> — the single source of truth where the project manager can see <strong>all three streams of intelligence combined</strong>.</p> <center><img src="/img/4.2.png" width="600" height="550" alt="PM dashboard" title="PM dashboard"></center> <h2>Section 2: Role-based agents in action</h2> <p>Let’s break down some of the most useful role-based agents:</p> <ul> <li> <p><strong>Finance agent</strong> – Tracks spending vs. budget in real time, sends alerts if costs exceed thresholds, and shares instant updates with stakeholders.</p> <ul> <li>Scenario: A procurement request pushes spend 5% over budget. The Finance Agent flags it, informs the Resource Agent, and updates the PM automatically.</li> </ul> </li> <li> <p><strong>Resource agent</strong> – Balances workloads, reallocates tasks when someone is overbooked, and ensures skills are matched to priorities.</p> <ul> <li>Scenario: A key engineer is on sick leave. The Resource Agent shifts their tasks to the next available developer, updates Jira, and notifies the team.</li> </ul> </li> <li> <p><strong>Communication agent</strong> – Tailors updates for each stakeholder group without flooding inboxes.</p> <ul> <li>Scenario: Instead of a giant status email, the Communication Agent sends Finance a budget snapshot, Marketing a timeline update, and the PM a summary dashboard.</li> </ul> </li> </ul> <p>This image shows how <strong>different AI agents act as specialized assistants</strong> for various project needs and how their insights flow directly to the right stakeholders:</p> <ul> <li><strong>Finance Agent</strong> → Tracks budgets, expenses, and forecasts. Its updates flow to the <strong>Project Manager</strong> and <strong>Finance Team</strong>, ensuring they always know where the money is going.</li> <li><strong>Resource Agent</strong> → Balances workloads, predicts availability, and flags bottlenecks. Its insights go to the <strong>Engineers</strong> and <strong>PM</strong>, helping optimize who works on what and when.</li> <li><strong>Communication Agent</strong> → Monitors updates from meetings, emails, and chat tools. It delivers structured updates to the <strong>Marketing Team</strong> and other stakeholders, cutting through noise and ensuring everyone stays aligned.</li> </ul> <p>At the center of it all is the <strong>Project Manager</strong>, who no longer has to manually chase updates. Instead, each stakeholder gets the <strong>right information at the right time</strong>, automatically delivered through these AI agents.</p> <center><img src="/img/4.3.png" width="600" height="550" alt="AI agent" title="AI agent"></center> <h2>Section 3: Automatic cross-team updates</h2> <p>One of the biggest pain points in multi-stakeholder projects is <strong>information lag.</strong> By the time Finance updates the budget, Engineering has already made a decision that conflicts with it.</p> <p><strong>Multi-agent systems eliminate this lag</strong>.</p> <ul> <li>Agents share updates across domains instantly.</li> <li>Dependencies update automatically in connected dashboards.</li> <li>Stakeholders see the latest info without needing to ask.</li> </ul> <p><strong>In a global product launch:</strong></p> <ul> <li>Finance updates ad spend forecasts → Finance Agent updates Marketing instantly.</li> <li>Engineering delays a feature by one week → Resource Agent updates the timeline, and Communication Agent informs all stakeholders.</li> </ul> <p>No “Monday catch-up” required — the project updates itself.</p> <p>This flow diagram demonstrates how <strong>Agentic AI streamlines cross-team updates automatically</strong>:</p> <ol> <li><strong>Finance Agent updates the budget</strong> → Whenever there’s a change in cost or resource usage, the Finance Agent instantly logs it.</li> <li><strong>Communication Agent sends a timeline update</strong> → The Communication Agent translates the finance change into a project impact update (e.g., “Timeline adjusted by 2 days”).</li> <li><strong>Marketing notified instantly</strong> → Instead of waiting for a weekly sync or status meeting, the Marketing team gets the update in real time, ensuring they can adjust campaigns or customer communications without delay.</li> </ol> <center><img src="/img/4.4.png" width="600" height="550" alt="Cross team update" title="Cross team update"></center> <h2>Section 4: Transparency without meeting overload</h2> <p>Meetings are the traditional fix for misalignment. But too many meetings kill productivity.</p> <p>With multi-agent systems, <strong>transparency is built-in:</strong></p> <ul> <li>Live dashboards show the current state of play.</li> <li>Agents generate concise, role-specific updates.</li> <li>PMs only call meetings for strategic discussions, not routine updates.</li> </ul> <p><strong>Consider this example:</strong> A weekly 2-hour cross-department sync shrinks into a 20-minute strategic review. Why? Because agents have already updated budgets, tasks, and dependencies in real time.</p> <p>This graphic illustrates the productivity shift that <strong>AI meeting support brings</strong>:</p> <ul> <li><strong>Before:</strong> A typical project meeting lasts two hours, with messy handwritten or scattered digital notes. Action items often get lost, follow-ups are inconsistent, and team members leave with different interpretations of what was decided.</li> <li><strong>After:</strong> With AI agents in place, the same meeting takes just 20 minutes. The AI captures the discussion in real time, generates a clear summary, assigns owners to tasks, and even prioritizes next steps. The result is a <strong>strategic, focused session</strong> where everyone leaves aligned — without the fatigue of long, unstructured meetings.</li> </ul> <center><img src="/img/4.7.png" width="600" height="550" alt="Transparency overload" title="Transparency overload"></center> <h2>Conclusion: Project updates become a living knowledge base</h2> <p>Multi-agent systems don’t just automate tasks — they transform how knowledge flows across a project. Updates are no longer buried in inboxes or trapped in one department’s tool. Instead, they become part of a <strong>living, evolving knowledge base</strong>, accessible to everyone, at any time.</p> <p>For PMs, this means less time chasing updates and more time focusing on leadership, strategy, and outcomes. For stakeholders, it means the right information, in the right format, at the right time.</p> <h3>Key takeaway</h3> <p><strong>Each stakeholder gets the right update, at the right time — thanks to multi-agent systems.</strong></p> <center><img src="/img/4.6.png" width="600" height="550" alt="PM future dashboard" title="PM future dashboard"></center>https://developer.hpe.com/gen-ai-for-pm-series-part-3-agentic-ai-as-the-risk-radar-for-project-managers/https://developer.hpe.com/gen-ai-for-pm-series-part-3-agentic-ai-as-the-risk-radar-for-project-managers/Sat, 30 Aug 2025 06:16:53 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 3 of our <em><strong>Gen AI for PM series</strong></em></p> <h3>Intro</h3> <p>Just as pilots rely on a radar to detect turbulence before it hits, project managers (PMs) can now rely on <strong>Agentic AI as their risk radar</strong>. Instead of waiting for issues to surface in weekly reviews or status meetings, the AI agent continuously scans project data, <strong>predicting risks early and raising alerts</strong>. This proactive approach ensures that project managers (PMs) can address problems before they escalate, keeping projects on course and stakeholders informed.</p> <p><strong>Why risk management is still manual in many orgs</strong></p> <p>Risk management is one of the most critical responsibilities of a project manager — yet in many organizations, it’s still a manual, reactive process.</p> <p>Here’s how it often plays out: A supplier misses a delivery update. A developer is behind on a sprint task. A budget line creeps higher than expected. None of these issues get noticed until the <strong>weekly review meeting</strong>, at which point the project is already veering off track.</p> <p>By then, the PM is in firefighting mode — pulling resources, rearranging timelines, and explaining to stakeholders why things slipped.</p> <p>But what if risks didn’t sneak up on you? What if you had a radar system scanning your project continuously, flagging issues before they turned into crises? That’s exactly what Agentic AI brings to project management.</p> <p>The comparison below highlights the difference between <strong>traditional</strong> and <strong>AI-driven</strong> risk management. On the left, project managers manually review risks during weekly check-ins—often discovering issues only after they’ve already caused delays. On the right, AI systems continuously monitor project data, providing <strong>predictive alerts</strong> so risks are identified early and corrective action can be taken before they escalate.</p> <center><img src="/img/3.1.png" width="600" height="550" alt="Data anomalies" title="Data anomalies"></center> <h2>Section 1: How AI agents scan project data for anomalies</h2> <p>Unlike human PMs who can only process limited information at a time, <strong>AI agents never sleep</strong>. They monitor all project data streams — tasks, budgets, communications, even IoT data in some industries — to spot anomalies.</p> <p><strong>In software development</strong>: Agents monitor sprint velocity, backlog size, and code check-ins. If velocity drops suddenly, they flag it.</p> <ul> <li><strong>In supply chain projects</strong>: Agents track supplier updates and logistics data. If a shipment is late, they predict downstream impact immediately.</li> <li><strong>In construction</strong>: Agents monitor workforce schedules and equipment usage, alerting when resources fall below thresholds.</li> </ul> <p><strong>Consider this example</strong>: A PM running a global product rollout doesn’t need to wait for a weekly sync. If a supplier in Asia fails to confirm shipment, the AI agent notices within hours and alerts the PM that downstream assembly tasks in Europe are at risk.</p> <p>The illustration below shows how the AI dashboard works in real time. Multiple data streams feed into the AI agent, which continuously monitors for irregularities. When an <strong>anomaly is detected</strong>, the system immediately raises a <strong>red alert notification</strong>, ensuring teams can act quickly before small issues turn into bigger problems.</p> <center><img src="/img/3.2.png" width="600" height="550" alt="Multiple data streams" title="Multiple data streams"></center> <h2>Section 2: Predictive alerts for delays or budgetary overruns</h2> <p>Detecting risks is useful. But predicting them? That’s a game-changer.</p> <p>Agentic AI uses <strong>predictive analytics</strong> to forecast delays and overspends before they happen:</p> <ul> <li>If the sprint velocity trend suggests the team won’t finish the backlog, the AI warns the PM early.</li> <li>If spending on cloud infrastructure grows faster than expected, the AI projects budget overrun for the quarter.</li> <li>If resource bottlenecks emerge, the AI highlights the likelihood of milestone delays.</li> </ul> <p><strong>Consider this example</strong>: In a software project, the AI forecasts that the team is likely to miss the sprint deadline by three days. The PM gets the alert a week in advance and reallocates a senior developer to the critical path — preventing a late delivery.</p> <p>The illustration below shows how AI helps anticipate project delays before they happen. The <strong>red dotted line</strong> marks a <strong>predicted delay</strong>, flagged ahead of the actual milestone, giving teams enough time to adjust plans and keep the project on track.</p> <center><img src="/img/3.3.png" width="600" height="550" alt="Milestone" title="Milestone"></center> <h2>Section 3: Scenario simulation &#x26; “What-If” planning</h2> <p>One of the toughest parts of risk management is deciding how to respond. Should you add resources? Cut scope? Delay the milestone?</p> <p>Agentic AI helps by running <strong>scenario simulations</strong>:</p> <ul> <li><strong>What if we delay Task A by two days?</strong> → The AI shows the ripple effect on dependent tasks.</li> <li><strong>What if we reassign 2 people from QA to Dev?</strong> → The AI shows improved velocity but increased bug risk.</li> <li><strong>What if we increase budget by 5%?</strong> → The AI shows how much time can be saved with extra contractors.</li> </ul> <p><strong>As an example</strong>: In a construction project, AI simulates the impact of moving indoor painting tasks earlier to cover for upcoming bad weather. The simulation shows minimal disruption, so the PM confidently approves the adjustment.</p> <p>The chart below shows how AI supports decision-making by mapping out different choices and their likely outcomes. For instance, <strong>choosing Option A may lead to a delay</strong>, while <strong>Option B reallocates resources</strong> and <strong>Option C increases the budget</strong>. By predicting the results of each path, the system helps project managers make more informed decisions before committing to an action.</p> <center><img src="/img/3.4.png" width="600" height="550" alt="Planning" title="Planning"></center> <h2>Section 4: Combining AI alerts with human decision-making</h2> <p>Agentic AI doesn’t replace PMs — it augments them.</p> <ul> <li>The <strong>AI surfaces risks</strong> quickly and accurately.</li> <li>The <strong>PM applies judgment</strong> — weighing culture, stakeholder politics, and long-term strategy.</li> <li>Together, they make <strong>faster, better decisions</strong>.</li> </ul> <p><strong>As an example</strong>: An AI agent alerts that overspending is likely in a marketing campaign. The AI suggests reducing ad spend in underperforming channels. But the PM knows the client values brand visibility over efficiency, so they adjust strategy to meet both the AI’s warning and the client’s preferences.</p> <p>The decision-tree diagram below shows how decision-making is shared between AI and human project managers. The AI first <strong>scans data</strong>, then <strong>predicts outcomes</strong>, and finally <strong>recommends actions</strong>. At that point, the <strong>project manager makes the final decision</strong>, combining AI-driven insights with human judgment. This loop ensures faster, smarter, and more reliable project decisions.</p> <center><img src="/img/3.5.png" width="600" height="550" alt="AI & Human decision making" title="AI & Human decision making"></center> <h2>Conclusion: Risk management shifts from reactive to proactive</h2> <p>For too long, risk management has been about looking backward — filling out risk logs, updating registers, and reacting when things go wrong.</p> <p>Agentic AI flips the model:</p> <ul> <li>Risks are detected early.</li> <li>Delays and overspending are predicted, not discovered too late.</li> <li>Scenarios are tested instantly, helping PMs choose the best response.</li> </ul> <p>Instead of firefighting, PMs become <strong>strategic leaders</strong>, steering their projects with foresight. The ones who embrace AI won’t just manage risks — they’ll <strong>manage confidently, proactively, and with influence.</strong></p> <h3>Key takeaway</h3> <p><strong>Agents give PMs foresight, not just hindsight — transforming risk management into a proactive, always-on discipline.</strong></p> <center><img src="/img/3.6.png" width="600" height="550" alt="Futuristic risk highlight" title="Futuristic risk highlight"></center>https://developer.hpe.com/experimenting-with-the-model-context-protocol-and-chapel/https://developer.hpe.com/experimenting-with-the-model-context-protocol-and-chapel/Thu, 28 Aug 2025 19:01:09 GMT<p>external blog</p>https://developer.hpe.com/part-12-agentos-the-invisible-conductor-of-enterprise-ai/https://developer.hpe.com/part-12-agentos-the-invisible-conductor-of-enterprise-ai/Thu, 28 Aug 2025 09:10:38 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2><strong>Scene one: The orchestra without a conductor</strong></h2> <p>Picture an orchestra hall. The violinist plays beautifully, the percussionist pounds with passion, the flutist adds magic — but there’s no conductor.</p> <p>Instead of harmony, you hear chaos.</p> <p>This is where most enterprises stand today with AI. They have chatbots for customer service, fraud detection systems in finance, diagnostic assistants in healthcare, HR copilots, and IT ticketing bots. Each plays its part, but without coordination, they overlap, conflict, and underperform.</p> <p>Enter <strong>AgentOS</strong>: the conductor that turns fragmented AI tools into a well-orchestrated symphonic enterprise.</p> <h2><strong>Why enterprises need AgentOS</strong></h2> <p>For the past two decades, enterprises have ridden the waves of digital transformation through multiple stages of AI:</p> <ol> <li> <p><strong>Traditional AI</strong> – Predictive, rule-based systems solving narrow problems.</p> </li> <li> <p><strong>Generative AI</strong> – Large language models that produce text, code, and content.</p> </li> <li> <p><strong>Agentic AI</strong> – Autonomous helpers reasoning, planning, and acting. Agentic AI even introduced modes like:</p> <ul> <li><strong>Coordination Mode –</strong> Agents interact to accomplish a shared task.</li> <li><strong>Collaboration Mode –</strong> Agents divide work and merge results.</li> <li><strong>Routing Mode –</strong> Agents pass tasks to the best-suited peer.</li> </ul> </li> </ol> <p>These modes gave enterprises glimpses of what AI teamwork could look like. But in practice, enterprises quickly ran into scaling problems: lack of governance, resource wastage, compliance risks, and integration hurdles.</p> <p><strong>AgentOS was designed to solve this.</strong> It is not just a tool. It is the operating system for agents, enabling enterprises to manage, orchestrate, and scale agent ecosystems responsibly.</p> <p>Just as operating systems defined enterprise computing, and Android/iOS defined enterprise mobility, <strong>AgentOS defines the enterprise era of agentic intelligence.</strong></p> <h2>How AgentOS is superior to agentic AI modes</h2> <p>While Agentic AI modes (coordination, collaboration, routing) describe <em>how</em> agents behave at the task level, they often remain limited to small clusters or localized workflows. <strong>AgentOS operates at the system level</strong>, enabling these modes to run systematically, securely, and at scale across the enterprise. Rather than being an “either/or,” Agentic AI and AgentOS are <strong>complementary layers</strong>: the former provides the micro-level behaviors of agents, while the latter provides the infrastructure, governance, and optimization needed to operationalize those behaviors across departments, business systems, and compliance requirements. The table below contrasts these perspectives—<strong>Agentic AI modes at the agent level vs. AgentOS at the enterprise level</strong>—to highlight how they build on each other.</p> <h1>Agentic AI vs AgentOS Capabilities</h1> <table style="border-collapse: collapse; width: 100%; text-align: left; background: linear-gradient(135deg, #f0f8ff, #e6f7ff); border: 2px solid #0073e6;"> <thead> <tr style="background-color: #0073e6; color: white;"> <th style="padding: 12px; font-weight: bold; text-align: left;">Capability</th> <th style="padding: 12px; font-weight: bold; text-align: left;">Agentic AI Modes (Coordination / Collaboration / Routing)</th> <th style="padding: 12px; font-weight: bold; text-align: left;">AgentOS (Enterprise-Grade)</th> </tr> </thead> <tbody> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Coordination</b></td> <td style="padding: 10px;">Agents communicate on a shared task</td> <td style="padding: 10px;">Orchestration across thousands of agents simultaneously</td> </tr> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Collaboration</b></td> <td style="padding: 10px;">Agents divide work and merge results</td> <td style="padding: 10px;">Workflow automation integrated with ERP, CRM, and APIs</td> </tr> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Routing</b></td> <td style="padding: 10px;">Routes tasks to relevant agents</td> <td style="padding: 10px;">Policy-driven routing with compliance and audit</td> </tr> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Scalability</b></td> <td style="padding: 10px;">Limited to agent clusters or workflows</td> <td style="padding: 10px;">Enterprise-wide scale with cross-departmental coverage</td> </tr> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Resource Management</b></td> <td style="padding: 10px;">Task-level optimization</td> <td style="padding: 10px;">Enterprise-grade compute allocation, GPU scheduling, cost optimization</td> </tr> <tr> <td style="padding: 10px; background-color:#f9f9f9;"><b>Governance</b></td> <td style="padding: 10px;">Minimal logs at agent level</td> <td style="padding: 10px;">Centralized compliance, execution history, and audit trails</td> </tr> </tbody> </table> <p><strong>In short:</strong> Agentic AI lets agents talk. AgentOS ensures they collaborate securely, at scale, under enterprise governance.</p> <h2>Features of AgentOS</h2> <ul> <li><strong>Orchestration beyond coordination –</strong> Manages thousands of agents like a mission control center.</li> <li><strong>Resource management –</strong> Allocates cloud, GPU, or edge resources intelligently.</li> <li><strong>Workflow automation –</strong> Converts ad-hoc agent interactions into repeatable enterprise workflows.</li> <li><strong>Integration layer –</strong> Connects to CRMs, ERPs, ITSM tools, data lakes, and APIs seamlessly.</li> <li><strong>Governance &#x26; Trust –</strong> Provides execution history, compliance enforcement, and role-based access.</li> </ul> <h2>Scene two: The smart hospital of tomorrow</h2> <p>Fast-forward to 2030. A major hospital runs with AI at its core:</p> <ul> <li>A <strong>diagnostic agentic AI</strong> system scans patient X-rays.</li> <li>A <strong>compliance agentic AI</strong> system checks treatment against regulations.</li> <li>A <strong>scheduling agentic AI</strong> system assigns the right doctors and resources.</li> <li>A <strong>finance agentic AI</strong> system validates insurance in real time.</li> </ul> <p>If these were run purely in Agentic AI modes, they might collaborate, but gaps remain: duplicated tasks, inconsistent audit logs, and overlooked compliance.</p> <p>With <strong>AgentOS</strong>, all these agents operate within a single, governed orchestration framework.</p> <ul> <li>Every recommendation is automatically validated for compliance.</li> <li>Compute resources are allocated efficiently.</li> <li>End-to-end records are logged for transparency.</li> </ul> <p>The result: safer patients, faster care, and reduced administrative costs.</p> <h2>Scene three: The corporate boardroom</h2> <p>A future enterprise boardroom operates like this:</p> <ul> <li>The <strong>Sales agentic AI team</strong> negotiates contracts with a client’s AI.</li> <li>The <strong>Finance agentic AI team</strong> calculates profit margins in real-time.</li> <li>The <strong>Legal agentic AI team</strong> ensures terms meet compliance.</li> <li>The <strong>Ops</strong> <strong>agentic AI team</strong> checks global supply chain capacity.</li> </ul> <p>Without AgentOS, each agent works in isolation — outputs fragmented across dashboards.</p> <p>With AgentOS, they function as a team under enterprise governance. A CXO can simply ask: “Can we close this deal profitably within compliance?” AgentOS synthesizes the inputs, runs checks, allocates compute, and produces a clear, enterprise-ready answer.</p> <h2>Enterprise benefits of AgentOS</h2> <h4><strong>1. Operational efficiency at scale</strong></h4> <p>Unifies siloed AI projects across departments, preventing duplication and aligning outcomes.</p> <ul> <li><strong>Example:</strong> A bank saves millions by linking fraud detection, customer support, and compliance agents under one roof.</li> </ul> <h4><strong>2. Production-grade governance</strong></h4> <p>Tracks every decision, logs every interaction, enforces policies, and ensures regulations like GDPR, HIPAA, or SOX are met.</p> <ul> <li><strong>Example:</strong> Healthcare providers ensure every diagnosis recommendation is audit-ready.</li> </ul> <h4>3. Enterprise-wide integration</h4> <p>Plug-and-play compatibility with ERP, CRM, HR, and supply chain systems.</p> <ul> <li><strong>Example:</strong> A manufacturing firm syncs predictive maintenance with ERP procurement automatically.</li> </ul> <h4>4. Resource &#x26; cost optimization</h4> <p>Allocates GPUs, cloud compute, and workloads smartly to reduce costs.</p> <ul> <li><strong>Example:</strong> Retailers balance demand forecasting, supply optimization, and customer recommendations without runaway GPU bills.</li> </ul> <h4>5. Future-proofing investments</h4> <p>Like Windows or Linux, AgentOS is not transient. It is foundational infrastructure that will only advance, not disappear. Enterprises can adopt once and evolve continuously.</p> <h2>Types of AgentOS emerging</h2> <ol> <li><strong>Enterprise AgentOS (PwC)</strong> – AI scaled across departments with compliance.</li> <li><strong>Research AgentOS (SyncIQ)</strong> – Multi-agent R&#x26;D orchestration.</li> <li><strong>Academic projects</strong> – Distributed OS for decentralized, mobile agents.</li> <li><strong>Open-Source AgentOS (Prolog)</strong> – Community-driven frameworks for coding and standards.</li> </ol> <h2>AgentOS in enterprise production – The game changer</h2> <p>AgentOS transforms how organizations operationalize AI by addressing inefficiencies that once slowed adoption. It provides opportunities to unify scattered AI initiatives, reduce infrastructure waste, and embed governance directly into workflows. Consider how things looked before AgentOS—and how they have changed after its introduction.</p> <p><strong>Before AgentOS</strong></p> <ul> <li>Siloed AI projects</li> <li>Expensive GPU wastage</li> <li>Long integration cycles</li> <li>Weak governance</li> </ul> <p><strong>After AgentOS</strong></p> <ul> <li>Unified agent workforce</li> <li>Optimized compute usage</li> <li>Plug-and-play enterprise integration</li> <li>Audit-ready compliance</li> </ul> <h2>Why AgentOS will define the next Era</h2> <p>Every digital revolution had a backbone:</p> <ul> <li>PCs → Operating systems</li> <li>Web → Browsers and cloud platforms</li> <li>Smartphones → Android and iOS</li> <li>AI Agents → AgentOS</li> </ul> <p>AgentOS is the infrastructure that ensures agents aren’t just clever individuals but a <strong>coordinated, compliant, enterprise-ready digital workforce.</strong></p> <h2>Closing thought</h2> <p>The future of enterprise AI is not about isolated copilots or one-off pilots. It is about <strong>agent ecosystems functioning as digital colleagues,</strong> transforming how businesses operate.</p> <p>Agentic AI introduced task-level coordination. AgentOS delivers system-wide orchestration, governance, and scalability.</p> <p>For enterprises, AgentOS is not a passing trend. It is the <strong>technology that will stay,</strong> advancing year after year, as fundamental as the operating systems that run our laptops and servers today.</p> <blockquote> <p>The real question for enterprise leaders is not “Should we adopt AgentOS?” It is <em><strong>“How fast can we adopt AgentOS to stay competitive in the AI-driven future?”</strong></em></p> </blockquote>https://developer.hpe.com/gen-ai-for-pm-series-part-2-your-new-pm-assistant-the-rise-of-agent-ai-in-daily-task-management/https://developer.hpe.com/gen-ai-for-pm-series-part-2-your-new-pm-assistant-the-rise-of-agent-ai-in-daily-task-management/Wed, 27 Aug 2025 17:39:15 GMT <style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 2 of our <em><strong>Gen AI for PM series</strong></em></p> <h1><strong>Intro:</strong></h1> <p>Project managers juggle endless moving parts—status updates, task assignments, shifting priorities, and the constant chase for clarity. Too often, valuable time is wasted just piecing together information, instead of driving decisions forward.</p> <p>This is where <strong>Agentic AI</strong> begins to change the game: by streamlining updates, automating repetitive follow-ups, and surfacing the insights you need, right when you need them.</p> <p><strong>The endless cycle of updates</strong></p> <p>If you’ve ever been a project manager, you know the drill. You open your inbox in the morning, and it’s flooded with updates: “Task delayed,” “Waiting for approval,” “Who’s picking this up?” Then come the Slack pings, calendar invites, and those dreaded status meetings.</p> <p>By lunchtime, you’ve spent more time chasing information than making decisions. And still, you find yourself sending the same email you’ve typed a hundred times before: “Where are we on this</p> <p>This is the reality of traditional project management — too much <strong>manual overhead</strong>, not enough time for leadership. But things are changing. With <strong>Agentic AI</strong>, task management is no longer a reactive grind. It’s becoming <strong>proactive, automated, and intelligent</strong>.</p> <p>An overwhelmed PM surrounded by emails and sticky notes (left) or one who is calm, cool, and collected, as a result of an AI assistant automatically updating tasks(right).</p> <center><img src="/img/ai-1.png" width="600" height="550" alt="Task allocation" title="Task allocation"></center> <h2>Section 1: Agentic AI for task allocation and reminders</h2> <p>Allocating tasks sounds simple, but in practice, it’s a juggling act. You need to balance skills, availability, deadlines, and workload. Traditionally, that means spreadsheets, endless emails, and lots of guesswork.</p> <p><strong>With Agentic AI, this process becomes effortless.</strong></p> <ul> <li><strong>Skill-based allocation</strong> – AI matches tasks to the right team members based on expertise and bandwidth.</li> <li><strong>Load balancing</strong> – If someone is overbooked, the AI reallocates tasks automatically.</li> <li><strong>Smart reminders</strong> – Instead of the PM chasing, AI sends nudges when deadlines are near.</li> </ul> <p><strong>Consider this example</strong>: Sarah, a PM, notices that her developer John already has three critical tasks. Instead of assigning him another bug fix, the AI routes it to Priya — who has the right skills and availability. John stays productive without burnout, and Priya feels trusted with meaningful work.</p> <p>The graphic below shows how tasks move through the AI-driven workflow: They begin with <strong>Task Creation</strong>, are followed by a <strong>Skill Match</strong>, then proceed to <strong>Agent Assignment</strong>. From there, the system ensures follow-through with <strong>Auto Reminders</strong> and concludes with <strong>Status Updates</strong> to keep everything transparent and on track.</p> <center><img src="/img/ai-2.png" width="600" height="550" alt="AI Task allocation" title="AI Task allocation"></center> <h2>Section 2: Natural language inputs can trigger automatic task adjustments</h2> <p>One of the most exciting aspects of Agentic AI is its ability to understand <strong>plain human language.</strong></p> <p><strong>Imagine this:</strong></p> <ul> <li>A developer types in Slack: “Feature A is blocked until the API fix goes live.”</li> <li>The AI picks it up, flags “Feature A” as blocked, shifts dependent tasks, and notifies affected stakeholders.</li> <li>No PM intervention needed.</li> </ul> <p>Instead of PMs manually updating Jira boards or rescheduling, the AI listens, interprets, and acts in real time.</p> <p><strong>Consider this Example</strong>: During a standup, a designer says, “Wireframes are done, just waiting on review.” Instantly, the AI updates the design task as complete, creates a “Review” sub-task, and alerts the approver.</p> <p>The illustration below shows how the system responds when a task becomes <strong>blocked</strong>: the AI agent steps in to automatically adjust the project timeline and trigger <strong>alert notifications</strong>, ensuring the issue is surfaced and addressed without delays.</p> <center><img src="/img/picture-11.png" width="600" height="550" alt="NL Auto task" title="NL Auto task"></center> <h2>Section 3: Reducing human error in task tracking</h2> <p>Manual task tracking is messy. People forget to update boards, mark things “done” that aren’t done, or miss dependencies. These errors snowball into delays, missed deadlines, and finger-pointing.</p> <p>Agentic AI tackles this by:</p> <ul> <li><strong>Auto-syncing across tools</strong> – Updates in Jira, Asana, Trello, or email are reflected everywhere instantly.</li> <li><strong>Error detection</strong> – If a task is marked complete but the code hasn’t been merged, AI raises a flag.</li> <li><strong>Audit trails</strong> – Every update is logged for transparency and compliance.</li> </ul> <p><strong>As an example</strong>: Without AI, a QA engineer closes a ticket marked “passed,” but the bug reappears in staging. AI catches the mismatch and reopens the task, preventing the issue from being reported late in production.</p> <p>To compare and contrast, consider the image below that shows the differences between manual tracking versus AI-powered tracking.</p> <center><img src="/img/picture-12.png" width="600" height="550" alt="Reducing error" title="Reducing error"></center> <h2>Section 4: Case Study – 20% time saved in sprint management</h2> <p>At a mid-sized software company, the PM team adopted an AI task assistant to manage sprint planning and daily updates.</p> <p><strong>After three months, they realized the following results:</strong></p> <ul> <li><strong>20% reduction in sprint management time</strong> (less manual backlog grooming, fewer status check-ins).</li> <li><strong>30% fewer status meetings</strong> (updates auto-logged by AI).</li> <li><strong>Higher developer satisfaction</strong> (less time spent on admin, more on building features).</li> </ul> <p><strong>In this case study</strong>, we observed that instead of three 1-hour sprint syncs per week, the team only needed one. The AI managed the other updates asynchronously, saving the PMs nearly 8 hours per month.</p> <p>The bar graph below illustrates the sprint management time that was saved as a result of the adoption of Agentic AI.</p> <center><img src="/img/ai-5.png" width="600" height="550" alt="Sprint Management" title="Sprint Management"></center> <h2>Focus on leadership, not micromanagement</h2> <p>Here’s the truth: AI isn’t coming to replace project managers. It’s coming to liberate them.</p> <p>By offloading repetitive work like task allocation, reminders, and updates, PMs can finally focus on the work that matters most:</p> <ul> <li>Motivating teams</li> <li>Driving strategy</li> <li>Building relationships</li> <li>Guiding projects toward meaningful outcomes</li> </ul> <p>The next generation of PMs won’t be remembered as task chasers — they’ll be celebrated as <strong>strategic leaders powered by intelligent AI assistants</strong>.</p> <h2>Key takeaway</h2> <p><strong>Agentic AI eliminates the “where are we on this?” emails, replacing them with real-time, self-updating project visibility.</strong></p> <center><img src="/img/ai-6.png" width="600" height="550" alt="Futuristic AI dashboard" title="Futuristic AI dashboard"></center>https://developer.hpe.com/1-from-gantt-charts-to-generative-ai-how-agentic-ai-is-revolutionizing-project-management/https://developer.hpe.com/1-from-gantt-charts-to-generative-ai-how-agentic-ai-is-revolutionizing-project-management/Wed, 27 Aug 2025 15:04:35 GMT<style> li {    font-size: 27px;    line-height: 33px;    max-width: none; } </style> <p>Part 1 of our <em><strong>Gen AI for PM series</strong></em></p> <h2><strong>Intro</strong></h2> <p>For decades, project managers have relied on <strong>Gantt charts</strong> and other static tools to track timelines and dependencies. While these methods provided structure, they often struggled to keep pace with the speed and complexity of modern projects. Enter <strong>Generative AI and Agentic AI</strong>—a leap forward that transforms project management from reactive tracking to <strong>proactive intelligence</strong>.</p> <p>Agentic AI doesn’t just visualize timelines; it continuously analyzes data, predicts risks, reallocates resources, and even recommends next best actions. This shift marks a new era where project managers move beyond manual monitoring and become <strong>strategic leaders</strong>, supported by AI-driven insights that keep projects adaptive, resilient, and future-ready.</p> <p><strong>From static plans to intelligent partners</strong></p> <p>Picture this: You’re a project manager (PM) juggling deadlines, budget limits, and a flood of emails. You’ve got your trusted Gantt chart open, but within hours, it’s already outdated. A supplier delay changes Task A, which cascades into Tasks B and C, and suddenly your neat timeline looks like spaghetti. Sound familiar?</p> <p>For decades, project management has relied on <strong>static tools</strong> — from sticky notes on whiteboards to spreadsheets and Gantt charts. They’ve helped PMs visualize tasks and dependencies, but they’re fundamentally <strong>snapshots in time</strong>. The moment reality shifts, the tools fall behind, and project managers are left manually updating and chasing information.</p> <p>Today, a new paradigm is emerging. <strong>Agentic AI</strong> — intelligent agents that adapt, predict, and act — is turning project management from reactive firefighting into proactive leadership. Think of it not as another tool, but as your <strong>digital co-pilot</strong> who keeps the plan alive, responsive, and self-updating.</p> <p>In this blog post, we'll compare and contrast the differences, as shown in the example below, between traditional project management and project management that is optimized through the use of Agentic AI.</p> <p><strong>Example</strong>: A split-screen illustration showing Traditional Gantt Chart → Manual Updates → Bottlenecked PM vs. Agentic AI Dashboard → Real-Time Updates → Autonomous Adjustments.</p> <center><img src="/img/g1.png" width="600" height="550" alt="Gantt chart" title="Gantt chart"></center> <h2><strong>Section 1: What is Agentic AI?</strong></h2> <p>At its core, <strong>Agentic AI</strong> refers to AI systems that not only respond to queries but also <strong>take initiative</strong>. Unlike chatbots that wait for prompts, Agentic AI can monitor, make decisions, and act autonomously within defined boundaries.</p> <p>In project management, this means:</p> <ul> <li><strong>Always-on Monitoring,</strong> i.e, scanning budgets, tasks, risks, and dependencies in real time.</li> <li><strong>Context-aware Decision-Making</strong>, i.e, adjusting schedules when delays happen.</li> <li><strong>Proactive communication,</strong> i.e, alerting stakeholders before problems escalate.</li> </ul> <p>Think of using Agentic AI as hiring a <strong>junior project manager</strong> who works 24/7, never forgets, and never gets tired.</p> <p><strong>As an example</strong>: If a teammate calls in sick, an <strong>AI scheduling agent</strong> instantly reallocates their tasks based on skills and availability, updates the timeline, and notifies the team — before you even sip your morning coffee.</p> <p>A diagram showing Agents → Scheduling, Risk, Communication → Central PM Dashboard.</p> <center><img src="/img/g2.png" width="600" height="550" alt="Scheduling agent" title="Scheduling agent"></center> <h2>Section 2: How agents automate planning &#x26; scheduling</h2> <p>One of the biggest pain points for PMs is the constant back-and-forth of scheduling.</p> <p><strong>How Agentic AI helps:</strong></p> <ol> <li> <p><strong>Dynamic scheduling</strong> – Agents adjust project timelines automatically when dependencies shift.</p> <ul> <li>Example: A supplier delays material delivery by two days. As a consequence AI shifts downstream tasks, updates the Gantt chart, and sends alerts.</li> </ul> </li> <li> <p><strong>Skill-based allocation</strong> – AI assigns tasks to the most suitable team members based on workload and expertise.</p> <ul> <li>Example: A high-priority bug fix goes to the developer with a proven track record in similar issues.</li> </ul> </li> <li> <p><strong>Automated reminders</strong> – AI sends nudges when deadlines approach, reducing PM’s email load.</p> </li> </ol> <p><strong>The diagram below illustrates how Agentic AI streamlines task management.</strong></p> <p>First, new tasks are <strong>created</strong> and automatically <strong>assigned to the right agent</strong>. The system then performs a <strong>skill match</strong> to ensure the most capable resource is selected. Once assigned, the AI sends <strong>automatic reminders</strong> to keep tasks on track. Finally, it provides <strong>real-time progress tracking</strong>, ensuring transparency and accountability at every stage.</p> <p>All of this is orchestrated under the umbrella of <strong>Automated Task Management</strong> <strong>powered by Agentic AI</strong>, minimizing manual intervention and boosting efficiency.</p> <center><img src="/img/g3.png" width="600" height="550" alt="Automate planning" title="Automate planning"></center> <h2><strong>Section 3: Real-time risk detection &#x26; mitigation</strong></h2> <p>Risk registers are often dusty spreadsheets used by PMs that are updated once a week. By the time risks surface, it’s usually too late.</p> <p><strong>Agentic AI acts as a real-time risk radar that offers:</strong></p> <ul> <li><strong>Continuous scanning</strong>: Agents monitor project velocity, costs, and communications for anomalies.</li> <li><strong>Predictive alerts</strong>: AI forecasts likely delays or overspending before they occur.</li> <li><strong>Suggested mitigations</strong>: Offers solutions, like shifting resources or adjusting scope.</li> </ul> <p> <strong>As an example</strong>: In a construction project, AI detects upcoming storms. It reschedules indoor tasks for those days, preventing downtime and cost overruns.</p> <p>In the graphic below, we compare traditional risk management with that which is AI-driven. When compared side-by-side, you can see how traditional risk management is manual and reactive, whilst AI-driven risk management is predictive and proactive.</p> <center><img src="/img/g4.png" width="600" height="550" alt="Real-time risk" title="Real-time risk"></center> <h2>Section 4: Examples in complex, multi-team projects</h2> <p>The larger and more complex the project, the more powerful Agentic AI becomes.</p> <ul> <li><strong>Global product launch -</strong> Marketing, supply chain, and engineering must align across time zones. AI agents track dependencies, flag misalignments, and ensure simultaneous readiness.</li> <li><strong>Software development at scale -</strong> Agents monitor sprint velocity, bug queues, and release schedules. If delays in one module impact another, the AI re-prioritizes tasks automatically.</li> <li><strong>Healthcare facility rollout -</strong> Compliance checks, vendor schedules, and training sessions are intertwined. AI ensures compliance tasks are met and automates approval escalations.</li> </ul> <p>The diagram below illustrates this ecosystem, showing how multiple specialized agents—Scheduling, Risk, and Communication—work together under one intelligent system to create a seamless, AI-driven project management experience.</p> <center><img src="/img/g5.png" width="600" height="550" alt="Multi team project chart" title="Multi team project chart"></center> <h2>Conclusion: AI PM as the next career superpower</h2> <p>Will Agentic AI replace project managers? Absolutely not. Just like spreadsheets didn’t replace accountants, Agentic AI won’t replace PMs. Instead, it <strong>augments their role</strong> — handling the repetitive grunt work so PMs can focus on <strong>strategy, leadership, and creativity</strong>.</p> <p>In fact, PMs who embrace Agentic AI are positioning themselves as the <strong>next generation of strategic leaders.</strong> With less time spent on “status chasing” and more on stakeholder engagement, they’ll be the ones driving organizational impact.</p> <h2>Key takeaway</h2> <p><strong>Agentic AI turns project plans into living, self-updating systems — giving PMs foresight, agility, and the freedom to lead.</strong></p> <p>The visualization below captures this idea: a <strong>futuristic project manager at a unified AI-powered dashboard</strong>, with supporting agents feeding their insights in real time. It symbolizes the partnership between <strong>human leadership</strong> and <strong>AI execution</strong>, where the manager focuses on strategic decisions while the agents handle executional intelligence.</p> <center><img src="/img/picture-1.png" width="600" height="550" alt="Futuristic PM" title="Futuristic PM"></center>https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-5-productivity-and-magic-compilers/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-5-productivity-and-magic-compilers/Thu, 21 Aug 2025 00:19:53 GMT<p>external blog</p>https://developer.hpe.com/sivabala-model-context-protocol-mcp-the-universal-connector-for-ai-applications/https://developer.hpe.com/sivabala-model-context-protocol-mcp-the-universal-connector-for-ai-applications/Tue, 12 Aug 2025 02:00:38 GMT<h2>Introduction</h2> <p>In today's AI-driven landscape, large language models (LLMs) are transforming how we approach writing, research, and problem-solving. Yet even the most sophisticated models face a fundamental limitation: they're only as effective as the context they operate within. This is where the Model Context Protocol (MCP) emerges as a revolutionary standard, bridging the gap between AI applications and real-world data ecosystems.</p> <h2>The challenge: AI applications in isolation</h2> <p>Picture this scenario: Your organization has deployed cutting-edge AI tools that can generate brilliant insights, write compelling content, and solve complex problems. Yet these same tools remain blind to your company's real-time data, can't access your proprietary systems, and operate without understanding your current operational context.</p> <p>This is the reality for most enterprises today. Traditional LLMs operate in silos, disconnected from live systems and dynamic data sources. This isolation severely constrains their utility in practical, enterprise-grade scenarios. Organizations find themselves with powerful AI tools that can't access their proprietary databases, integrate with existing APIs, or leverage real-time information streams.</p> <p>The result? AI applications that feel impressive in demonstrations but fall short in production environments where contextual awareness is paramount. It's like having a brilliant consultant who knows everything about general business practices but nothing about your specific company, industry challenges, or current operational state.</p> <h2>Enter MCP - the game changer</h2> <p>Model Context Protocol addresses this challenge head-on by functioning as the "USB-C port" for AI applications. Just as USB-C provides a universal standard for connecting diverse devices and peripherals, MCP provides a standardized interface that allows AI systems to connect seamlessly with external tools, APIs, and data sources.</p> <p>As content strategist Gary Vaynerchuk observed, "Content is king, but context is god." MCP ensures that AI models transcend mere intelligence to achieve deep contextual understanding of your specific environment, data, and operational needs.</p> <h2>How MCP works - architecture deep dive</h2> <p>Understanding MCP requires grasping its elegant three-component architecture that works in perfect harmony:</p> <h3>The three pillars</h3> <p><strong>Host Application</strong><br> Think of this as your AI command center – applications like Claude Desktop, Cursor, or any AI-powered tool that serves as the primary interface for users. The host houses the MCP Client functionality and acts as the orchestrator of AI-human interactions.</p> <p><strong>MCP Client</strong><br> This is the sophisticated translator and communication bridge. It facilitates seamless data exchange between your AI application and the various MCP servers, handling protocol negotiations and data formatting while ensuring that information flows smoothly.</p> <p><strong>MCP Server</strong><br> These are specialized gateways that expose your organization's tools, structured data, and computational capabilities to client applications through standardized interfaces. Each server can represent different aspects of your infrastructure – from databases to APIs to monitoring systems.</p> <h3>Three interfaces that make the magic happen</h3> <p>The protocol operates through three critical interfaces that transform static AI into dynamic, actionable intelligence:</p> <p><strong>Tools interface</strong><br> This transforms passive AI models into active, agentic AI systems capable of real-world interactions. Unlike traditional AI that merely generates recommendations, agentic AI powered by MCP can autonomously execute actions based on context and user intent. Through the Tools Interface, AI agents can trigger deployments, create incident tickets, send notifications, schedule maintenance windows, or execute complex multi-step workflows – all while maintaining appropriate guardrails and approval processes. This represents a fundamental shift from AI as a consultative tool to AI as an operational participant in your infrastructure.</p> <p><strong>Resources interface</strong><br> This provides structured access to your organization's data repositories, content management systems, and information databases. Your AI models operate with current, relevant information rather than stale training data, ensuring responses are both intelligent and immediately applicable.</p> <p><strong>Prompts interface</strong><br> This facilitates the creation and deployment of reusable templates and workflows, enabling consistent AI behavior patterns across different use cases. Think of it as creating "muscle memory" for your AI applications.</p> <h2>The communication dance</h2> <p>The MCP communication process follows an elegant handshake protocol that ensures robust, secure interactions:</p> <ol> <li> <p><strong>Capability discovery</strong>: The client initiates contact by querying servers for available tools, prompts, and resources – essentially asking "What can you help me with?"</p> </li> <li> <p><strong>Capability acknowledgment</strong>: Servers respond with comprehensive inventories of their offerings, like a detailed service menu</p> </li> <li> <p><strong>Message exchange</strong>: Once capabilities are established, ongoing communication enables dynamic interaction between AI models and server resources</p> </li> </ol> <p>Let me illustrate with a practical example: Imagine a Weather API server that exposes forecasting tools and meteorological data templates. When an AI application needs weather information, the MCP client can leverage these resources to generate contextually rich, location-specific weather insights that go far beyond generic forecasts.</p> <h2>The universal value proposition</h2> <p>MCP delivers transformative advantages across the entire AI ecosystem:</p> <h3>For AI developers: Acceleration through standardization</h3> <ul> <li><strong>Universal compatibility</strong>: Connect your applications to any MCP-compliant server without building custom integrations</li> <li><strong>Reduced development overhead</strong>: Focus on core AI functionality rather than wrestling with bespoke connectors</li> <li><strong>Scalable architecture</strong>: Easily expand application capabilities by integrating additional MCP servers as your needs grow</li> </ul> <h3>For tool and API providers: Build once, impact everywhere</h3> <ul> <li><strong>Universal reach</strong>: Create a single MCP server that works across all compatible AI applications</li> <li><strong>Expanded market access</strong>: Make your tools and data accessible to the entire growing MCP ecosystem</li> <li><strong>Standardized Interface</strong>: Reduce documentation burden and support overhead through consistent protocols</li> </ul> <h3>For end users: Intelligence meets reality</h3> <ul> <li><strong>Enhanced AI capabilities</strong>: Access AI applications that truly understand and interact with your real-world data</li> <li><strong>Seamless workflows</strong>: Experience fluid integration between AI tools and your existing systems</li> <li><strong>Context-aware responses</strong>: Receive more relevant, actionable insights that reflect your current situation</li> </ul> <h3>For enterprise organizations: governance meets innovation</h3> <ul> <li><strong>Clear separation of concerns</strong>: Maintain distinct boundaries between AI development and product teams</li> <li><strong>Security and governance</strong>: Implement consistent access controls and data governance across AI integrations</li> <li><strong>Scalable AI strategy</strong>: Deploy AI capabilities across your organization without architectural complexity</li> </ul> <h2>Market momentum - the numbers tell the story</h2> <p>MCP is rapidly establishing itself as the foundational protocol for AI integration, with adoption metrics that demonstrate serious industry traction:</p> <ul> <li><strong>Over 5,000 active MCP servers</strong> deployed as of May 2025</li> <li><strong>Major AI platforms</strong> including OpenAI integrated MCP support within one week of release</li> <li><strong>Enterprise adoption</strong> accelerating across technology, healthcare, and financial services sectors</li> <li><strong>Developer community</strong> growing exponentially with contributions from leading tech companies</li> </ul> <p>This momentum positions MCP as the "HTTP protocol" of AI integration – a fundamental standard that enables the next generation of intelligent applications.</p> <h2>Real-world impact that goes beyond theory</h2> <p>Organizations implementing MCP are seeing immediate, measurable benefits:</p> <p><strong>DevOps teams</strong> can now ask AI assistants to "Check the health of our production environment and create tickets for any critical alerts" – with the AI actually accessing monitoring systems and ticketing platforms.</p> <p><strong>Sales teams</strong> leverage AI that understands current pipeline data, customer interaction history, and market conditions to provide genuinely useful recommendations rather than generic advice.</p> <p><strong>Support organizations</strong> deploy AI that can access knowledge bases, ticket systems, and customer data to provide contextual, actionable support guidance.</p> <h2>The path forward</h2> <p>The Model Context Protocol represents more than a technical specification – it embodies a fundamental shift in AI application architecture. By standardizing context exchange mechanisms, MCP enables a new class of AI applications that are:</p> <ul> <li><strong>Deeply integrated</strong>: Connected to real-world data and systems</li> <li><strong>Contextually aware</strong>: Operating with current, relevant information</li> <li><strong>Dynamically capable</strong>: Able to perform actions beyond text generation</li> <li><strong>Enterprise ready</strong>: Designed for production deployment with appropriate security and governance</li> </ul> <h2>Getting started on your MCP journey</h2> <p>Organizations considering MCP adoption should focus on these strategic steps:</p> <ol> <li><strong>Identify high-value use cases</strong>: Determine where contextual AI can deliver immediate business impact in your environment</li> <li><strong>Assess integration points</strong>: Catalog existing APIs and data sources that would benefit from AI accessibility</li> <li><strong>Plan your pilot</strong>: Start with a focused MCP server implementation that addresses specific organizational needs</li> <li><strong>Engage the ecosystem</strong>: Connect with the growing MCP community for best practices and shared learnings</li> </ol> <h2>What's next?</h2> <p>In my next article, I'll dive deep into a practical implementation story: Building HPE OpsRamp's MCP server. I'll explore the technical architecture decisions, implementation challenges, and real-world benefits that emerge when monitoring and operations data becomes truly AI-accessible.</p> <p>The future of AI isn't just about smarter models – it's about smarter connections between those models and the real world where business happens. MCP is making that future possible, one standardized connection at a time.</p> <hr> <p><em>Part 2 of this series will explore the practical implementation of OpsRamp's MCP server, showcasing how these concepts translate into real-world monitoring and operations intelligence. Stay tuned for deep technical insights and practical guidance.</em></p>https://developer.hpe.com/part-11-agentic-ai-vs-ai-agent/https://developer.hpe.com/part-11-agentic-ai-vs-ai-agent/Mon, 11 Aug 2025 08:48:18 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>Introduction</h2> <p>Artificial Intelligence (AI) is evolving rapidly, and two terms — Agentic AI and AI agent — are increasingly appearing in business strategy documents, technical roadmaps, and boardroom discussions. While they sound similar, they represent distinct concepts with different implications for enterprise strategy, operations, and innovation.</p> <p>For business leaders and senior managers, understanding the distinction is not just academic — it can determine whether an AI initiative scales effectively, integrates seamlessly into your operations, and delivers measurable Return on investment (ROI).</p> <h3>This article breaks down Agentic AI vs AI agent with:</h3> <ul> <li>Clear definitions and conceptual differences</li> <li>Technical underpinnings</li> <li>Business use cases</li> <li>Strategic considerations for adoption</li> <li>Risks and governance</li> <li>Future trends</li> <li>References for deeper exploration</li> </ul> <h2>1. Defining the terms</h2> <h3>1.1 AI agent</h3> <p>An <strong>AI agent is a single, autonomous software program</strong> that perceives an environment, makes decisions, and takes actions toward a defined goal, often within a narrow domain.</p> <p><strong>Key characteristics:</strong></p> <ul> <li>Operates <strong>within a predefined scope</strong></li> <li>Uses <strong>rules, heuristics, or ML models</strong> for decision-making</li> <li>Limited ability to adapt beyond programmed or trained boundaries</li> <li>Often embedded into <strong>applications or workflows</strong> for a specific function</li> </ul> <p><strong>Examples:</strong></p> <ul> <li>A chatbot that answers HR policy questions</li> <li>A recommendation engine for an e-commerce site</li> <li>An autonomous trading bot</li> </ul> <h3>1.2 Agentic AI</h3> <p><strong>Agentic AI</strong> is a system of multiple AI agents orchestrated to work collaboratively, often with dynamic planning, self-reflection, and multi-step reasoning capabilities. It moves beyond isolated automation toward goal-oriented, adaptive, and multi-role AI-driven ecosystems.</p> <p><strong>Key characteristics:</strong></p> <ul> <li><strong>Multi-agent orchestration</strong>: Different specialized agents work together</li> <li><strong>Autonomy in task decomposition</strong>: Breaks high-level goals into sub-tasks</li> <li><strong>Reasoning loops</strong>: Self-reflects, evaluates outcomes, retries or adjusts</li> <li><strong>Tool integration</strong>: Uses APIs, databases, and other systems dynamically</li> <li><strong>Adaptability</strong>: Learns and optimizes over time</li> </ul> <p><strong>Examples:</strong></p> <ul> <li> <p>An AI-powered compliance team where:</p> <ul> <li>Agent A scans documents</li> <li>Agent B applies regulatory rules</li> <li>Agent C drafts compliance reports</li> <li>Orchestrator Agent manages workflows and escalations</li> </ul> </li> <li> <p>An industrial repair assistant that autonomously diagnoses, orders parts, and schedules technicians.</p> </li> </ul> <p><strong>Quick analogy:</strong></p> <p>Let me offer a simple analogy to bring clarity to the difference between an AI agent and Agentic AI.</p> <ul> <li><strong>AI Agent</strong> = A skilled individual employee</li> <li><strong>Agentic AI =</strong> A <strong>self-managed, multi-skilled team</strong> with a project manager, analysts, and doers — all AI-driven</li> </ul> <h2>2. Technical architecture differences</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Feature</th> <th>AI Agent</th> <th>Agentic AI</th> </tr> </thead> <tbody> <tr> <td><strong>Scope</strong></td> <td>Narrow, task-specific</td> <td>Broad, multi-task, goal-oriented</td> </tr> <tr> <td><strong>Architecture</strong></td> <td>Single process or microservice</td> <td>Multi-agent framework with orchestration layer</td> </tr> <tr> <td><strong>Decision-making</strong></td> <td>Rule-based or model-based within fixed scope</td> <td>Multi-step reasoning, task decomposition</td> </tr> <tr> <td><strong>Adaptability</strong></td> <td>Limited</td> <td>High (dynamic adaptation to changing contexts)</td> </tr> <tr> <td><strong>Integration</strong></td> <td>Usually integrates with one system</td> <td>Connects to multiple tools, APIs, data sources</td> </tr> <tr> <td><strong>Examples of frameworks</strong></td> <td>Rasa, Botpress, Dialogflow</td> <td>LangChain Agents, AutoGPT, BabyAGI, AGNO framework</td> </tr> </tbody> </table> <h2>3. Business use cases</h2> <h3>3.1 AI agent use cases</h3> <ul> <li><strong>Customer support bots</strong> – Provide FAQs and simple troubleshooting</li> <li><strong>Automated trading systems</strong> – Execute trades based on pre-defined signals</li> <li><strong>HR chatbots</strong> – Answer leave policy questions</li> </ul> <p>**Business Impact:**Quick to deploy, lower cost, but limited in complexity and scope.</p> <h3>3.2 Agentic AI use cases</h3> <ul> <li><strong>Regulatory compliance automation</strong> – Multiple agents scan, analyze, summarize, and report</li> <li><strong>Healthcare assistants</strong> – Agents for symptoms checking, scheduling, and generating discharge summaries</li> <li><strong>Complex industrial troubleshooting</strong> – Agents for diagnostics, parts ordering, repair instructions</li> </ul> <p><strong>Business impact:</strong> Higher complexity but greater ROI potential through process automation at scale.</p> <h2>4. Strategic considerations for business leaders</h2> <h3>4.1 When to use an AI agent</h3> <ul> <li>You have a <strong>clear, narrow task</strong></li> <li>The process is <strong>repeatable with predictable inputs/outputs</strong></li> <li>ROI needs to be realized quickly with low implementation risk</li> </ul> <h3>4.2 When to use Agentic AI</h3> <ul> <li>Multiple complex workflows need <strong>coordination</strong></li> <li>There is <strong>uncertainty and variability</strong> in the environment</li> <li>Long-term scalability and adaptability are priorities</li> </ul> <p><strong>Case example:</strong></p> <p>A bank could deploy:</p> <ul> <li><strong>AI agent:</strong> To answer customer queries about loan status</li> <li><strong>Agentic AI:</strong> To orchestrate fraud detection, compliance checks, and customer communication in an integrated way</li> </ul> <h2>5. Risks, challenges, and governance</h2> <h3>5.1 AI agent risks</h3> <ul> <li>Overfitting to narrow tasks</li> <li>Limited scalability</li> <li>Vulnerable to changing business requirements</li> </ul> <h3>5.2 Agentic AI risks</h3> <ul> <li>Complexity in orchestration</li> <li>Higher cost of development and maintenance</li> <li>AI hallucinations amplified if orchestration lacks guardrails</li> <li>Governance challenges (data security, compliance, ethics)</li> </ul> <p><strong>Mitigation strategies:</strong></p> <p>To reduce the potential risks and challenges identified, the following strategies can be implemented. Each one is aimed at ensuring operational resilience, improving outcomes, and minimizing negative impacts</p> <ul> <li><strong>Guardrails</strong>: NeMo Guardrails, policy frameworks</li> <li><strong>Auditability</strong>: Maintain decision logs</li> <li><strong>Ethics</strong>: Align with corporate AI principles</li> <li><strong>Testing</strong>:Continuous evaluation under real-world conditions</li> </ul> <h2>6. Technology enablers</h2> <ul> <li> <p><strong>For AI agents:</strong></p> <p>When deploying AI Agents, the following mitigation strategies help ensure reliable, safe, and efficient performance within their defined scope</p> <ul> <li>Rasa, Dialogflow, Botpress</li> <li>Domain-specific ML models</li> </ul> </li> <li> <p><strong>For Agentic AI:</strong></p> <p>Agentic AI systems operate with greater autonomy and complexity, so their mitigation strategies should account for adaptability, multi-step reasoning, and integration across multiple systems:</p> <ul> <li>LangChain multi-agent orchestration</li> <li>AutoGPT &#x26; BabyAGI architectures</li> <li>AGNO Framework (for enterprise-grade agent teams)</li> <li>Vector databases (Qdrant, Milvus)</li> <li>LLMs (GPT-4, Claude, LLaMA variants)</li> </ul> </li> </ul> <h2>7. Future trends</h2> <ul> <li><strong>Hybrid Systems:</strong> AI agents enhanced with Agentic AI orchestration</li> <li><strong>Industry-Specific Agent Ecosystems:</strong> Pre-built for finance, healthcare, logistics</li> <li><strong>Agent Marketplaces:</strong> Plug-and-play agents that integrate into orchestrators</li> <li><strong>Integration with IoT &#x26; Edge AI:</strong> Enabling real-time decision-making in physical environments</li> </ul> <h2>8. Decision framework for leaders</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Question</th> <th>If “Yes” →</th> <th>Answer</th> </tr> </thead> <tbody> <tr> <td>Is the task narrow &amp; predictable?</td> <td>AI Agent</td> <td>✅</td> </tr> <tr> <td>Does it require multi-step reasoning?</td> <td>Agentic AI</td> <td>✅</td> </tr> <tr> <td>Will it integrate with one system only?</td> <td>AI Agent</td> <td>✅</td> </tr> <tr> <td>Do you need adaptability to changing inputs?</td> <td>Agentic AI</td> <td>✅</td> </tr> <tr> <td>Is speed-to-market the top priority?</td> <td>AI Agent</td> <td>✅</td> </tr> <tr> <td>Is scalability across processes the goal?</td> <td>Agentic AI</td> <td>✅</td> </tr> </tbody> </table> <h2>9. Conclusion</h2> <p>The choice between <strong>AI agent</strong> and <strong>Agentic AI</strong> is not binary — many enterprises will deploy both. The key is <strong>understanding the maturity of your AI roadmap</strong>, your operational complexity, and your scalability ambitions.</p> <ul> <li><strong>AI agents</strong> are quick wins for automation</li> <li><strong>Agentic AI</strong> is a long-term strategic play for transformation</li> </ul> <p>By aligning your choice with business strategy and technical capability, you position your organization to move from isolated AI successes to enterprise-wide AI transformation.</p> <h2>References</h2> <ol> <li>Russell, S., &#x26; Norvig, P. (2021). Artificial Intelligence: A Modern Approach. Pearson. <a href="https://www.pearson.com/en-us/subject-catalog/p/artificial-intelligence%E2%80%91a%E2%80%91modern%E2%80%91approach/P200000003500/9780137505135">https://www.pearson.com/en-us/subject-catalog/p/artificial-intelligence‑a‑modern‑approach/P200000003500/9780137505135</a></li> <li>LangChain Documentation – <a href="https://docs.langchain.com">https://docs.langchain.com</a></li> <li>Auto-GPT – <a href="https://github.com/Torantulino/Auto-GPT%5B%5D(https://github.com/Torantulino/Auto-GPT)">https://github.com/Torantulino/Auto-GPT[](https://github.com/Torantulino/Auto-GPT)</a></li> <li>AGNO Framework – <a href="https://docs.agno.com/introduction">https://docs.agno.com/introduction</a></li> </ol>https://developer.hpe.com/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/provisioning-mks-clusters-in-hpe-greenlake-for-private-cloud-enterprise/Fri, 08 Aug 2025 08:09:07 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> now includes the Morpheus Kubernetes Service (MKS) feature, allowing users to deploy and manage Kubernetes (K8s) clusters directly through <a href="https://www.hpe.com/us/en/morpheus-enterprise-software.html">HPE Morpheus Enterprise software</a>. With HPE Private Cloud Enterprise, now in its <em>Beta</em> phase with MKS feature, customers can take advantage of streamlined MKS cluster provisioning using predefined cluster layouts, making it easier to launch and manage their containerized workloads.</p> <p>In this blog post, I will guide you through the process of provisioning an MKS cluster in HPE Private Cloud Enterprise, followed by key post-deployment tasks. These include downloading the <em>kubeconfig</em> file, scaling the cluster by adding worker nodes, upgrading the K8s cluster version, deploying applications via running workflows, and finally, deleting the MKS cluster when it's no longer needed.</p> <h2>Overview</h2> <p><a href="https://www.hpe.com/us/en/hpe-private-cloud-enterprise.html">HPE Private Cloud Enterprise</a> is a fully managed <em>Infrastructure as a Service</em> (IaaS) offering that brings a modern, cloud-like experience to on-premises environments. It combines the flexibility of hybrid cloud with the enterprise-grade control and security required by enterprise IT.</p> <p>Through the integration with <a href="https://www.hpe.com/us/en/morpheus-enterprise-software.html">HPE Morpheus Enterprise</a>, which serves as the cloud management and orchestration layer, HPE Private Cloud Enterprise delivers a unified self-service interface for provisioning virtual machines (VMs), creating containers, and deploying applications, all governed by role-based access control (RBAC). This integration now enables support for the Morpheus Kubernetes Service (MKS) feature, allowing users to deploy and manage K8s clusters with built-in automation and observability capabilities.</p> <p>HPE Morpheus Enterprise provides a set of prebuilt MKS cluster layouts that support a variety of K8s versions and cluster types. These cluster layouts provision MKS clusters using the native K8s distribution, streamlining and accelerating deployment. This blog post walks through the process of creating an MKS cluster using one of these preconfigured cluster layouts.</p> <h2>Prerequisites</h2> <p>Ensure that the following prerequisites are fulfilled:</p> <ul> <li>Access to an HPE Private Cloud Enterprise workspace with the '<em>Private Cloud Tenant Owner'</em> role, allowing administrative actions in the <em><strong>Virtual Machines</strong></em> service.</li> <li>The group named <em>'Customer Department B'</em> and the network <em>'Green-Segment'</em> have already been created.</li> <li>HPE Morpheus Enterprise running version 8.0.5 or higher.</li> <li>The MKS feature is enabled in HPE Private Cloud Enterprise. You can confirm the presence of the <em>Clusters</em> menu from <em><strong>Infrastructure</strong></em> tab.</li> </ul> <h2>Provisioning an MKS cluster</h2> <ol> <li>Log in to HPE GreenLake Cloud at <em><a href="https://common.cloud.hpe.com/">https://common.cloud.hpe.com/</a></em>.</li> <li>Locate an HPE Private Cloud Enterprise workspace and click <em><strong>Go to Workspace</strong></em>.</li> </ol> <p><img src="/img/workspace.png" alt=""></p> <ol start="3"> <li>From the <em>Getting Started</em> screen, click <em><strong>Find Services</strong></em>. (If you've already launched HPE GreenLake Flex Solutions, the service will appear under <em>Recent Services</em>, from which you can click <em><strong>Launch</strong></em>, then skip to the step <em>6</em> below.)</li> </ol> <p><img src="/img/get-started.png" alt=""></p> <ol start="4"> <li>From the <em>Services Catalog</em>, enter <em>'HPE GreenLake Flex Solutions'</em>. Click the <em><strong>HPE GreenLake Flex Solutions</strong></em> Workloads result.</li> </ol> <p><img src="/img/service-catalog.png" alt=""></p> <ol start="5"> <li>From the Workloads <em><strong>Overview</strong></em> tab, click <em><strong>Launch</strong></em> to open the HPE GreenLake Flex Solutions.</li> </ol> <p><img src="/img/launch-glc.png" alt=""></p> <ol start="6"> <li>From the Cloud Services <em><strong>Dashboard</strong></em>, locate the <em>Private Cloud Services</em> card and ensure that the correct location is selected from the drop-down list.</li> </ol> <p><img src="/img/launch-morpheus.png" alt=""></p> <ol start="7"> <li>Click <em><strong>Launch HPE Morpheus Enterprise</strong></em>. The Morpheus Dashboard screen (<strong>Operations</strong> > <strong>Dashboard</strong>) displays.</li> </ol> <p><img src="/img/morpheus-dashboard.png" alt=""></p> <ol start="8"> <li>In the <em>Service Console</em>, click <em><strong>Infrastructure</strong></em> and select <em>Clusters</em>.</li> </ol> <p><img src="/img/mks-feature.png" alt=""></p> <ol start="9"> <li>From the clusters screen, click <em><strong>+Add Cluster</strong></em> to initiate MKS cluster provisioning.</li> </ol> <p><img src="/img/add-cluster.png" alt=""></p> <p>9.1 <strong>Choose cluster type</strong></p> <p>In the <em>CREATE CLUSTER</em> panel, select <em>'KUBERNETES CLUSTER'</em> as the <em>Cluster Type</em> and click <em><strong>Next</strong></em>.</p> <p><img src="/img/cluster-type.png" alt=""></p> <p>9.2 <strong>Select a group</strong></p> <p>Select a group, for example, <em>'Customer Department B'</em> and click <em><strong>Next</strong></em>.</p> <p><img src="/img/cluster-group.png" alt=""></p> <p>9.3 <strong>Specify the cluster name</strong></p> <p>Enter <em>CLUSTER NAME</em> as <em>'mks-demo'</em>, and optionally specify <em>RESOURCE NAME</em>, <em>DESCRIPTION</em>, and <em>LABELS</em>. Click <em><strong>Next</strong></em>.</p> <p><img src="/img/cluster-name.png" alt=""></p> <p>9.4 <strong>Select cluster layout &#x26; configure master</strong></p> <p>Select <em>LAYOUT</em> and <em>PLAN</em>, configure <em>VOLUMES</em>, and select the <em>NETWORKS</em>, such as <em>'Green-Segment'</em>. Click <em><strong>Next</strong></em>.</p> <p><img src="/img/cluster-config.png" alt=""></p> <p>NOTE: For demonstration purpose, the <em>'MKS Kubernetes 1.31 Cluster on Ubuntu 22.04'</em> is selected. This cluster layout provisons an MKS cluster using a single master with K8s version <em>1.31</em>.</p> <p><strong>Important:</strong> The Classless Inter-Domain Routing (CIDR) for the <em>POD</em> and <em>SERVICE</em> define the internal IP ranges by routers to K8s <em>Pods</em> and <em>Services</em>. To prevent conflicts, ensure these CIDRs are distinct from the network settings.</p> <p>9.5 <strong>Configure worker</strong></p> <p>Specify <em>NUMBER OF WORKERS</em>, along with <em>PLAN</em>, <em>VOLUMES</em>, and <em>NETWORKS</em>. You may retain the default settings or reuse the values previously configured for the master. Click <em><strong>Next</strong></em>.</p> <p><img src="/img/cluster-worker.png" alt=""></p> <p>9.6 <strong>Review cluster details</strong></p> <p>Skip this step for <strong>Automation</strong> settings. The cluster review screen displays.</p> <p><img src="/img/cluster-review.png" alt=""></p> <p>Click <em><strong>Complete</strong></em>. The MKS cluster provisioning process initiates.</p> <p><img src="/img/cluster-provisioning.png" alt=""></p> <h2>Verify MKS cluster</h2> <p>Approximately after a few minutes, the cluster <em>'mks-demo'</em> is created using the specified cluster layout: <em>MKS Kubernetes 1.31 Cluster on Ubuntu 22.04</em>.</p> <p><img src="/img/cluster-status.png" alt=""></p> <h2>Access MKS cluster</h2> <p>Click the <em>'mks-demo'</em> cluster to view its details from the <em><strong>Summary</strong></em> tab.</p> <p><img src="/img/cluster-details.png" alt=""></p> <p>Navigate to the <em><strong>Control</strong></em> tab and run the command <em>'kubectl get nodes'</em> to view the cluster's node information.</p> <p><img src="/img/cluster-nodes.png" alt=""></p> <p>In line with the cluster type, <em>MKS Kubernetes 1.31 Cluster on Ubuntu 22.04</em>, the <em>'mks-demo'</em> cluster consists of one master and three workers.</p> <h2>Run daily cluster operations</h2> <p>From the provisioned MKS cluster, the <em><strong>Actions</strong></em> menu provides a curated set of supported operations that simplify and streamline day-to-day cluster management. From downloading kubeconfig to scaling the cluster and performing upgrade, these built-in actions help automate key cluster operations, making cluster administration faster, easier, and more consistent.</p> <p><img src="/img/cluster-actions.png" alt=""></p> <p>The following sections explore how to perform these day-to-day cluster operations.</p> <ul> <li><strong>View and download kube config</strong></li> </ul> <p>From the <em>'mks-demo'</em> cluster, click <em><strong>Actions</strong></em>.</p> <p><img src="/img/view-kubeconfig-menu.png" alt=""></p> <p>Click <em><strong>View Kube Config</strong></em> to view the Kube config of the cluster.</p> <p><img src="/img/view-kubeconfig.png" alt=""></p> <p>Save the Kube config content to a file. For example, <em>'mks-demo.kubeconfig'</em> on a <em>Linux</em> client as shown in the following output. This file can then be used with the <em>kubectl</em> CLI or the <em>Helm</em> tool to access the MKS cluster and deploy applications using either YAML manifests or <em>Helm</em> charts.</p> <p><img src="/img/kubectl-console.png" alt=""></p> <ul> <li><strong>Add addtional worker</strong></li> </ul> <p>In K8s, adding new workers to an existing cluster involves setting up self-registration, provisioning VM instances, and integrating them into the cluster. This process can be time-consuming and often demands custom automation scripts to streamline the workflow.</p> <p>From the MKS screen, navigate to a cluster and click <em><strong>Actions</strong></em>.</p> <p><img src="/img/add-worker.png" alt=""></p> <p>Click <em><strong>Add VMware Kubernetes Worker</strong></em> to initiate adding new worker to the cluster.</p> <ol> <li>Enter <em>NAME</em> and an optional brief <em>DESCRIPTION</em>.</li> </ol> <p><img src="/img/cluster-add-worker-name.png" alt=""></p> <ol start="2"> <li>Select <em>PLAN</em> and configure <em>VOLUMES</em> and <em>NETWORKS</em>.</li> </ol> <p><img src="/img/cluster-add-worker-config.png" alt=""></p> <p>To add multiple workers at once, set the desired value in the <em>NUMBER OF WORKERS</em> field.</p> <ol start="3"> <li>Review worker details.</li> </ol> <p>Skip this step for <strong>Automation</strong> settings. The worker review screen displays.</p> <p><img src="/img/cluster-add-worker-review.png" alt=""></p> <p>Click <em><strong>Complete</strong></em> to initiate provisioning new VM instances and adding them to the cluster as new workers.</p> <p><img src="/img/cluster-new-worker-adding.png" alt=""></p> <ol start="4"> <li>Verify new worker.</li> </ol> <p>Navigate to the <em><strong>Nodes</strong></em> tab to check the new worker <em>'new-mks-worker'</em> is listed in the node list.</p> <p><img src="/img/cluster-new-worker.png" alt=""></p> <ul> <li><strong>Upgrade MKS cluster</strong></li> </ul> <p>K8s follows a frequent release cycle, every 4 months on average, to ensure stability, innovation, and timely security updates. While upgrading a K8s cluster is crucial for maintaining security, performance, and access to the latest features, it remains a complex and demanding task. The upgrade process presents significant challenges, including managing scale and complexity, minimizing downtime risks, and handling substantial operational overhead that can span weeks and require coordination across multiple teams.</p> <p>From the MKS screen, navigate to a cluster and click <em><strong>Actions</strong></em>.</p> <p><img src="/img/upgrade-cluster-menu.png" alt=""></p> <p>Click <em><strong>Upgrade Cluster</strong></em>. The <em>UPGRADE CLUSTER</em> screen displays the list of supported versions available for upgrade.</p> <p><img src="/img/upgrade-cluster.png" alt=""></p> <p>Select version <em>1.32.7</em> and click <em><strong>Apply</strong></em> to initiate upgrading the cluster.</p> <p><img src="/img/upgrading-cluster.png" alt=""></p> <p>Approximately after a few minutes, the cluster status updates to <em><strong>Ok</strong></em> and displays its cluster layout as <em>'MKS Kubernetes 1.32 Cluster on Ubuntu 22.04'</em>.</p> <p><img src="/img/upgraded-cluster.png" alt=""></p> <h2>Deploy applications</h2> <p>After downloading the kubeconfig file as outlined earlier, you can easily deploy applications to the MKS cluster using the <em>kubectl</em> CLI or <em>Helm</em>. This section includes steps to deploy applications using the built-in features available from the provisioned MKS cluster in the <em>Service Console</em>.</p> <ul> <li><strong>Deploy applications from the cluster's <em>Control</em> tab</strong></li> </ul> <p>Under <em><strong>Control</strong></em> tab of the <em>'mks-demo'</em> cluster, deploy a sample application <em>'nginx-demo'</em> to the namespace <em>'app-demo'</em> by running the commands <em>'kubectl create namespace app-demo'</em> and <em>'kubectl create deployment nginx-demo --image=nginx --namespace app-demo'</em>.</p> <p><img src="/img/kubectl-create-deploy.png" alt=""></p> <ul> <li><strong>Deploy applications from the cluster's <em>Actions</em> menu</strong></li> </ul> <p>From the <em>'mks-demo'</em> cluster screen, select <strong>Actions</strong> > <strong>Run Workload</strong>.</p> <p><img src="/img/run-workload-menu.png" alt=""></p> <p>Define the following <em>Deployment</em> YAML in <em>CUSTOM SPEC</em> and click <em><strong>Apply</strong></em>. It will deploy the <em>'nginx-demo'</em> application to the <em>'default'</em> namespace.</p> <pre><code class="language-shell">apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-demo spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx </code></pre> <p><img src="/img/run-workload.png" alt=""></p> <p>Run the following command from the cluster's <em><strong>Control</strong></em> tab to confirm the <em>'nginx-demo'</em> has been deployed successfully to the <em>default</em> namespace.</p> <pre><code class="language-shell">kubectl get all -n default </code></pre> <p><img src="/img/get-workload.png" alt=""></p> <h2>Conclusion</h2> <p>This blog post provided a step-by-step guide to provisioning an MKS cluster using the MKS Beta feature within the HPE Private Cloud Enterprise environment. By selecting from a list of preconfigured MKS cluster layouts, you can quickly deploy an MKS cluster with your preferred cluster type and K8s version. Once provisioned, adding more workers is as simple as clicking the button from the cluster's <em>Actions</em> menu. Cluster upgrading to a newer K8s version follows the same streamlined process. This makes cluster administration more efficient, consistent, and user-friendly.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/2025-aug-05/https://developer.hpe.com/2025-aug-05/Tue, 05 Aug 2025 13:00:00 GMThttps://developer.hpe.com/configure-isut-and-ams-for-vlcm-based-firmware-updates-on-gen-12-servers/https://developer.hpe.com/configure-isut-and-ams-for-vlcm-based-firmware-updates-on-gen-12-servers/Mon, 04 Aug 2025 10:04:54 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>HPE Gen12 servers introduce enhanced security by supporting only High Security modes (SecureStandard, CNSA, FIPS). This impacts how you configure Intelligent System Update Tool <strong>(iSUT)</strong> and Agentless Management Service <strong>(AMS)</strong> for vSphere Lifecycle Manager <strong>(vLCM)</strong> based firmware updates.</p> <p>Unlike previous generations, configuration through the HPE OneView for VMware vCenter <strong>(OV4VC)</strong> and HPE Compute Ops Management plug-in for VMware vCenter <strong>(COM4VC)</strong> vLCM Pre-Check page is not available in these modes, as iLO credentials are now required. Instead, you must manually configure AMS and iSUT by creating an application account and providing valid HPE iLO credentials.</p> <p>In this blog post, I’ll show you how to configure iSUT and AMS to enable vLCM-based firmware updates on HPE Gen12 servers.</p> <hr> <h2><strong>Prerequisites</strong></h2> <ul> <li>HPE Gen12 server with iLO 7</li> <li>vSphere environment with vLCM enabled</li> <li>iLO credentials with sufficient privileges</li> <li>Access to server CLI (SSH or local console)</li> </ul> <hr> <h2><strong>Step 1: Create an Application Account on iLO 7</strong></h2> <p>Application accounts are service accounts in iLO 7, used by host applications (like iSUT and AMS) to securely authenticate and communicate with iLO.</p> <p><strong>To create an application account using CLI:</strong></p> <pre><code class="language-shell">sut appaccount create -u &#x3C;ilo_username> -p &#x3C;ilo_password> </code></pre> <p><strong>Alternatively, to proceed without creating an application account, provide the iLO credentials using the following CLI command:</strong></p> <pre><code class="language-shell">sut -set ilousername=&#x3C;ilo_username> ilopassword=&#x3C;ilo_password> </code></pre> <hr> <h2><strong>Step 2: Set iSUT Mode to AutoDeploy</strong></h2> <p>Set the iSUT mode to <code>AutoDeploy</code> to enable automated firmware updates:</p> <pre><code class="language-shell">sut -set mode=AutoDeploy </code></pre> <hr> <h2><strong>Step 3: Configure AMS Application Account (for VMware)</strong></h2> <p>For VMware environments, create the AMS application account:</p> <pre><code class="language-shell">/opt/amsdv/bin/amsdCli appaccount create -u &#x3C;iLO_username> -p &#x3C;iLO_password> </code></pre> <hr> <h2><strong>Step 4: Verify Application Account in iLO</strong></h2> <ol> <li>Open the <strong>iLO GUI</strong></li> <li>Navigate to <strong>iLO Settings</strong> > <strong>User Management</strong> > <strong>Users</strong></li> <li>Select <strong>Application Account</strong></li> <li>Confirm the application account details are present</li> </ol> <hr> <h2><strong>Step 5: Check AMS status in iLO GUI</strong></h2> <ul> <li>Ensure AMS status is reported as <strong>Available</strong> in the iLO GUI.</li> </ul> <hr> <h2><strong>Step 6: Verify iSUT and AMS status in vSphere</strong></h2> <ol> <li>Log in to <strong>VMware vSphere</strong>.</li> <li>Select the required <strong>cluster</strong> and click the <strong>Configure</strong> tab.</li> <li>In the left panel, go to <strong>Cluster > Configure > HPE Server Hardware</strong>.</li> <li>On the <strong>vLCM Pre-Check</strong> panel, check the <strong>iSUT mode</strong> and <strong>AMS state</strong>.</li> <li>Refresh the page and confirm both statuses are <strong>green</strong>.</li> </ol> <hr> <h2><strong>Conclusion</strong></h2> <p>With AMS and iSUT properly configured, you are ready to proceed with vLCM-based firmware updates on HPE Gen12 servers, including both <strong>ProLiant</strong> and <strong>Synergy</strong> models. This ensures secure, automated, and compliant lifecycle management in high-security environments.</p> <hr> <p><strong>Tip:</strong> Always refer to the latest HPE and VMware documentation for updates on security practices and supported configurations.</p>https://developer.hpe.com/the-digital-layover-what-happens-before-you’re-allowed-online-at-the-airport/https://developer.hpe.com/the-digital-layover-what-happens-before-you’re-allowed-online-at-the-airport/Fri, 01 Aug 2025 05:35:23 GMT<p>Picture this: You're at Gate 23. Your flight’s delayed again and all you want is to unwind with a Netflix binge or clear out your inbox. You connect to the “Airport_Free_WiFi,” open your browser, and BANG, you get redirected to a page asking you to accept terms and conditions, enter your email, or maybe even watch a short ad. Annoying? Perhaps. Necessary? Absolutely! And sometimes, oddly enough, it just works with no login page, no interaction whatsoever. So what’s really going on behind the scenes?</p> <p>Welcome to the world of captive portals, the digital gatekeepers of public Wi-Fi. In this post, I'm going to pull back the curtain on how airport Wi-Fi actually works, why you’re asked to “sign in” sometimes, and how some devices skip that step altogether like VIPs at a club.</p> <p><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXcunUUNAp1SO7Fyx6GtdrnVcr15KEATXWXv_LSzcxRC7j3fybLlYLo2KB7dAixFhopQpfGHqZOUFY3tugNHJOeT7A8eFJ5qB4I_8qezNq0C-nbjRXlKySRoigTfIeDwtcPYWzjN?key=_tjnUcJBKaxknXG-TWFLfw" alt="Image of a person trying to connect to the Airport WiFi"></p> <h2>What is a captive portal?</h2> <p>Think of a captive portal like a hotel’s front desk. You've stepped into the building, you’re technically inside, but before you can head to your room or enjoy the amenities, you have to stop at the reception. There, you're asked to provide some details, show identification, sign a few forms, or maybe even put down a deposit. Only after completing that interaction are you given your room key and granted full access.</p> <p>In the same way, a captive portal is a special web page that appears automatically when you connect to a configured Wi-Fi network. While your device is physically connected to the network, your access to the internet is temporarily blocked or “captured” until you interact with the portal. The network essentially puts you in a “digital lobby”. Just like different hotels have different check-in policies depending on their brand or pricing tier, captive portals vary based on how the network is configured. Some are simple and quick, others require more steps. Common requests include:</p> <ul> <li>Accepting terms of service, like a digital waiver</li> <li>Providing your email address or flight number</li> <li>Entering a voucher code, boarding pass, or access key</li> <li>Watching a sponsored video or promotional message</li> <li>Paying for access, especially for higher-speed or premium tiers</li> </ul> <p>The word "captive" might sound harsh, but that's literally what's happening. Your web browser is temporarily being captured and redirected until you complete the required action. Once that’s done, the network releases the restrictions and lets you roam the internet freely.</p> <h2>What's really happening in the background? </h2> <p>When you connect to public Wi-Fi, the process happening behind the scenes is a little more involved than just tapping “connect.” First, your device sends out a request to join the wireless network. This request is picked up by a wireless access point, which is basically a specialized router that handles connections from lots of people at once. If everything checks out, the network lets your device in. But even though you’re now connected to the network, you’re not yet "connected" to the internet.</p> <p>Next, your device needs to get an IP address, which acts like a temporary digital “home address” for your session on the network. It requests this from the airport’s system using something called Dynamic Host Configuration Protocol (DHCP). Once your device receives this address, it’s officially on the local network, but it still can’t browse the web just yet.</p> <p>At this point, you might open your browser or launch an app that tries to connect to the internet, like checking your email or visiting Google. But instead of taking you directly to your destination, the network steps in and intercepts your request. <strong>Why?</strong> Because it needs you to "check in" first, just like a hotel front desk <strong>that won't</strong> let you use the pool until you've signed in.</p> <p>So, instead of loading the site you asked for, the network reroutes you to the captive portal configured. This is where you might be asked to accept terms and conditions, provide your email, or enter a flight number. Until you complete this step, the network won’t let your traffic reach the wider internet. Once you’ve fulfilled the requirement, the network clears you for full access.</p> <p>All of this is quietly managed behind the scenes by something called an access controller. You can think of it like a digital traffic cop. It monitors all the devices trying to connect and decides who gets through immediately, who needs to stop at the portal, and who might not be allowed on at all. It helps keep the network secure, fair, and manageable, especially in busy places like airports where thousands of people are trying to get online at the same time.</p> <h2>The VIP lane: MAC address authentication caching</h2> <p>Every single device that can connect to a network, your smartphone, laptop, tablet, or even a smartwatch, has an unique identifier called a Media Access Control (MAC) address. This isn't like your phone number or email; it's a permanent, hardware-level address, often described as your device's digital fingerprint. Unlike an IP address, which can change frequently, your MAC address is hard-coded into your device's network card and stays the same. Airport networks, or any public Wi-Fi network for that matter, can leverage MAC addresses to streamline the connection process for specific users or devices. </p> <p>Here's how it generally works: Network administrators maintain a list, which is a pre-approved list of trusted MAC addresses. When your device tries to connect to the Wi-Fi network, the access controller immediately checks its MAC address. If your device's MAC address is on that list, perhaps because you've successfully logged in during a previous visit or because you're authorized airport staff, the network instantly recognizes you. This means you skip the captive portal entirely, gaining immediate and seamless internet access without needing to re-enter details or watch ads. This clever use of MAC addresses makes for a much smoother and faster experience, especially for frequent travelers or those whose devices have been previously authenticated on the network.</p> <h2>Step-by-Step: What happens when you connect to airport Wi-Fi</h2> <p>Whether you're leveraging a pre-registered device or connecting for the first time, your device undergoes a series of well-defined steps. Here's how it goes:</p> <p><strong>Step 1: 802.11 association and IP assignment</strong></p> <p>When you choose "Airport_Free_WiFi," your device reaches out to the airport's Wi-Fi network. They complete a quick DHCP handshake, and then your device is assigned a temporary IP address. At this point, you're connected to their internal network, but an access controller is still preventing you from reaching the wider internet.</p> <p><strong>Step 2: Access controller decision point – MAC address authentication vs. captive portal redirection</strong></p> <p>Upon IP address assignment, the access controller steps in as a policy enforcer. It immediately intercepts all initial traffic coming from devices that haven't yet been authenticated. The access controller then performs a MAC Address Lookup: it inspects your device's unique MAC address, which it learned when your device first connected. This MAC address is checked against a list, which is a database of pre-approved or authorized MAC addresses, often stored internally. If your device's MAC address is recognized on this list, the access controller instantly applies a policy that grants direct internet access for your device's specific MAC and IP address combination. However, if the MAC address is unknown or not authorized, the access controller then forces a redirection of your device's web traffic to the captive portal.</p> <p><strong>Step 3A: MAC address authentication flow</strong></p> <p>If your device's unique identifier (MAC address) is on the network's special "approved" list, the access controller immediately opens the gate, allowing your device's internet requests to go straight out to the internet without any detours. This means when you type in a website address, the network doesn't mess with it; it goes directly to find that website. So, all your normal internet activities like browsing, using apps, and checking secure sites work instantly and without any hassle.</p> <p><strong>Step 3B: Captive portal flow</strong></p> <p>When a device that hasn’t yet been authenticated tries to access a website, say <a href=""></a><em><a href="http://www.google.com">www.google.com</a></em>, the access controller intervenes using a technique known as DNS interception. Instead of resolving the domain to the real IP address of Google, the access controller returns the IP address of the captive portal. This causes the device to resolve the Google domain to the captive portal’s IP address. Once the device sends an HTTP or HTTPS request to that spoofed address, the captive portal responds with an HTTP 302 redirect, a standard mechanism used to point browsers to a different URL. This redirect guides the browser away from the originally requested site and instead delivers the user to the captive portal.</p> <p><strong>Step 4: Authentication/Terms agreement via captive portal</strong></p> <p>The user then interacts with the captive portal, a process that typically involves submitting information like an email address, flight number, or an acceptance of terms through an HTTP POST request to the captive portal server. This server performs backend validation, checking the provided data against its own internal databases, external authentication servers or even simply logging the acceptance of terms. Upon successful validation, the captive portal server sends a request to the access controller, saying "my job is done here".</p> <p><strong>Step 5: Authorization and internet access granted</strong></p> <p>Once the user has completed what the captive portal asked for, the access controller gets the message that you're good to go. It then quickly changes its internal rules, essentially lifting the blockade it had on your device. This opens up the internet for you, allowing all your web requests and other online activities to flow freely out to the rest of the internet without any further restrictions.</p> <h2>How devices detect captive portals automatically</h2> <p>Ever notice how your phone automatically shows a login screen the moment you connect to a network? This automatic detection is achieved by what's commonly known as a captive portal detection mechanism. Your device isn't just passively waiting for a redirect; it's actively checking to see if it's being held behind a portal. Here's how it generally works:</p> <p>Every modern operating system (OS), be it iOS, Android, Windows, macOS, or Linux, has a designated trusted URL that it tries to reach shortly after connecting to a new Wi-Fi network. These URLs are specifically designed to return a very simple, predictable response when they are successfully accessed.</p> <ul> <li>For Apple devices, the operating system attempts to load <em><a href="http://captive.apple.com/hotspot-detect.html">http://captive.apple.com/hotspot-detect.html</a>.</em></li> <li>Android devices use a similar mechanism, often checking a URL from Google's servers like <em><a href="http://connectivitycheck.gstatic.com/generate_204">http://connectivitycheck.gstatic.com/generate_204</a>.</em></li> <li>Windows devices try to reach this URL: <em><a href="http://www.msftconnecttest.com/connecttest.txt">http://www.msftconnecttest.com/connecttest.txt</a>.</em></li> </ul> <p>The key here is what happens next. If your device successfully loads this dedicated URL and gets the expected response (HTTP 200 OK), the OS concludes that there's no captive portal blocking access, and it allows normal internet traffic to flow.</p> <p>However, if your device tries to reach those URLs and it doesn’t get the HTTP 200 OK response, it probably means the access controller has intercepted the request and redirected it to the captive portal's IP address. The OS immediately understands that it's behind a captive portal. In this scenario, your operating system, acting like a helpful guide, automatically opens a mini-browser window or a notification that directly leads you to the captive portal's login page. This pre-emptive action saves you the hassle of opening a browser yourself and trying to navigate to a website only to be redirected. It's a seamless user experience, making it feel as if your device has a built-in travel agent that instinctively knows where to take you when you need to "check in" to the network.</p> <h2>Why do airports use these systems?</h2> <p>It might seem like a hassle, but there are smart reasons behind these digital gatekeepers:</p> <h3>1. Legal compliance and liability protection</h3> <p>Requiring users to agree to terms of service helps limit the airport’s liability for how the network is used. It sets boundaries and expectations just like a waiver at a gym.</p> <h3>2. Bandwidth management</h3> <p>Airports handle thousands of simultaneous users. Captive portals allow administrators to throttle bandwidth, enforce usage limits, or offer paid tiers, ensuring that everyone gets a fair shot at connectivity.</p> <h3>3. Revenue generation</h3> <p>Captive portals often serve up ads or upsell faster internet access. It’s one way airports offset the cost of offering Wi-Fi for free.</p> <h3>4. Security and monitoring</h3> <p>By requiring logins or recognizing MAC addresses, airports can keep an eye on network usage and respond more quickly to unusual behavior or security threats.</p> <p>So the next time you’re stuck at an airport, waiting for your boarding call and connecting to public Wi-Fi, remember this invisible dance of digital infrastructure working behind the scenes. Whether you’re redirected through a captive portal or glide through with MAC authentication, it’s not just a matter of convenience, it’s a carefully engineered system balancing security and usability. These systems ensure millions of travelers each day can access the internet in a way that’s fast, safe, and fair. And now that you know what’s really happening behind the login screen, you might just appreciate that airport Wi-Fi a little more, buffering and all!</p>https://developer.hpe.com/7-questions-for-tiago-carneiro-and-guillaume-helbecque-combinatorial-optimization-in-chapel/https://developer.hpe.com/7-questions-for-tiago-carneiro-and-guillaume-helbecque-combinatorial-optimization-in-chapel/Thu, 31 Jul 2025 01:59:08 GMT<p>external blog</p>https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-4-syntax-matters/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-4-syntax-matters/Wed, 23 Jul 2025 17:44:42 GMT<p>external blog</p>https://developer.hpe.com/part-10-agentic-ai-serving-—-hosting-agents-like-llms-with-agno-playground/https://developer.hpe.com/part-10-agentic-ai-serving-—-hosting-agents-like-llms-with-agno-playground/Mon, 21 Jul 2025 11:54:31 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>We’re all familiar with model serving — deploying LLMs like GPT, LLaMA, or Mistral behind APIs. But what if you could serve not just a model — but a <strong>complete AI agent with memory, tools, goals, and personality</strong>?</p> <p>This is the essence of Agentic AI Serving using the AGNO Framework — a next-gen architecture where agents are hosted, monitored, and interacted with like full applications.</p> <blockquote> <p>You can learn more about it by reading <a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-10-agentic-ai-33a7ff1dd010">My post on Medium.</a></p> </blockquote> <h4>This guide covers:</h4> <ul> <li>What Agentic AI Serving is</li> <li>How to serve agents via the AGNO Playground</li> <li>Hosting single or multi-agent systems</li> <li>Capturing session data for compliance and observability</li> </ul> <h2>What is Agentic AI Serving?</h2> <h3>In traditional GenAI:</h3> <p>LLMs are stateless tools. You orchestrate logic around them.</p> <h3>In Agentic AI:</h3> <p>The agent contains the logic — tools, reasoning, goals, and memory — and can be served as an interactive, stateful application.</p> <h3>Serving means:</h3> <ul> <li>Hosting the agent as an interactive app</li> <li>Enabling real-time communication and control</li> <li>Monitoring its behavior, tools, and outputs</li> <li>Running agents like microservices — locally or in the cloud</li> </ul> <h3>AGNO enables:</h3> <ul> <li>Agent-as-a-Service deployment</li> <li>A Browser-based chat interface</li> <li>Support for OpenAI, Claude, Ollama, DeepSeek, and more</li> <li>Full session monitoring and conversation logging</li> </ul> <h2>Getting practical: Serving your first agent</h2> <h3>1. Set up AGNO Playground</h3> <p>Ensure you're using an AGNO-compatible environment such as autogen_py_3_11_11, and that Ollama or OpenAI is accessible.</p> <h3>2. Sample playground.py script</h3> <pre><code class="language-python">import os from agno.agent import Agent from agno.models.ollama import Ollama from agno.playground import Playground, serve_playground_app # Set environment variables os.environ['AGNO_API_KEY'] = 'ag-\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\**-U' os.environ['AGNO_MONITOR'] = 'true'  # Enable session tracking </code></pre> <h3>Define a news reporter agent</h3> <pre><code class="language-python">agent = Agent( model=Ollama(id="llama3.2", provider="Ollama"), description="You are an enthusiastic news reporter with a flair for storytelling!", markdown=True ) </code></pre> <h3>(Optional) Test agent locally before serving</h3> <pre><code class="language-python">agent.print_response("Tell me about a breaking news story from New York.", stream=True) </code></pre> <h4>Create playground app</h4> <pre><code class="language-python">app = Playground(agents=agent).get_app() </code></pre> <h3>Launch local playground</h3> <pre><code class="language-python">if __name__ == "__main__": serve_playground_app("playground:app", reload=True) </code></pre> <p>Access your hosted agent</p> <pre><code class="language-python">#After running: python playground.py https://app.agno.com/playground/chat?endpoint=localhost:7777&#x26;agent=&#x3C;your-agent-id> </code></pre> <p>You now have a fully interactive Agentic AI service — complete with memory, tools, and autonomy — accessible via browser.</p> <h3>Bonus: Team agent serving</h3> <p>Want to host multiple agents with different skills, roles, or tools?</p> <pre><code class="language-python"> app = Playground(agents=team.members).get_app() </code></pre> <p><strong>Each agent maintains:</strong></p> <ul> <li>Its own LLM backend (OpenAI, Ollama, Claude, etc.)</li> <li>Independent tools and reasoning logic</li> <li>A unique personality or domain focus</li> <li>Access to shared memory or state if configured</li> </ul> <p>Perfect for multi-agent collaboration, delegation, or workflows.</p> <h3>Session logging &#x26; monitoring</h3> <h4>AGNO includes built-in monitoring:</h4> <pre><code class="language-python">os.environ\['AGNO_MONITOR'] = 'true' </code></pre> <p>This activates:</p> <ul> <li>Session logs</li> <li>Tool call traces</li> <li>Execution monitoring</li> <li>Replay/debug capabilities</li> </ul> <p>These capabilities are essential for enterprise use cases requiring reproducibility, auditing, or compliance.</p> <h2>Summary</h2> <p>As I've illustrated in this post, the AGNO Playground offers multiple tools to build and serve AI agents with ease.</p> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Component</th> <th>Purpose</th> </tr> </thead> <tbody> <tr> <td><code>Playground()</code></td> <td>Initializes the app interface</td> </tr> <tr> <td><code>agents=agent</code></td> <td>Serve a single agent instance</td> </tr> <tr> <td><code>agents=team.members</code></td> <td>Serve a multi-agent team</td> </tr> <tr> <td><code>AGNO_MONITOR=true</code></td> <td>Enables observability and logs</td> </tr> <tr> <td><code>AGNO_API_KEY</code></td> <td>Authenticates with AGNO cloud if needed</td> </tr> <tr> <td><code>serve_playground_app()</code></td> <td>Boots the local or hosted serving app</td> </tr> </tbody> </table> <h2>Pro tips</h2> <p>You can deploy AGNO agents:</p> <ul> <li>Locally, using models from Ollama</li> <li>Remotely, using cloud LLM APIs</li> <li>In production, with full-stack hosting</li> <li>As teams, where each agent plays a defined role</li> </ul> <h2>Final thoughts</h2> <p>Agentic AI Serving is the bridge between prompt engineering and software deployment. You’re not just sending prompts — you’re hosting intelligent, tool-using entities with goals and context.</p> <p>When agents are deployed, monitored, and refined, GenAI evolves into true AI Systems.</p>https://developer.hpe.com/post-9-agentic-ai-with-agno-ollama-and-local-llama3/https://developer.hpe.com/post-9-agentic-ai-with-agno-ollama-and-local-llama3/Mon, 21 Jul 2025 11:12:57 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>As Agentic AI evolves, the need for local, private, and flexible inference becomes critical. Frameworks like AGNO provide orchestration, but the ability to plug in LLMs running locally is what sets the next-gen agentic stack apart.</p> <blockquote> <p>You can learn more about it by reading <a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-9-agentic-ai-agno-74d74cd0d9f3">My post on Medium.</a></p> </blockquote> <h3>In this walkthrough, I explore:</h3> <ul> <li>What is Ollama and how it powers local LLMs</li> <li>Running LLaMA 3.2 locally with minimal setup</li> <li>Connecting Ollama with AGNO framework</li> <li>Building an offline agent pipeline using only Python</li> <li>How this empowers fully private, offline, and customizable AI deployments</li> </ul> <h2>AGNO Meets local LLMs via Ollama</h2> <p>One of AGNO’s core strengths is modularity — it can interface with any LLM provider, including:</p> <ul> <li>OpenAI (GPT4o, GPT-3.5)</li> <li>Claude (3.5 Sonnet, Haiku)</li> <li>DeepSeek</li> <li>Mistral</li> <li>Ollama (local)</li> </ul> <p>This makes it possible to define agents using LLaMA 3, Mistral, or Gemma without cloud dependencies — while maintaining the full Agentic AI loop: <strong>Think → Plan → Act → Reflect</strong></p> <h2>What is Ollama?</h2> <p>Ollama is a local inference server that can run transformer models on a CPU or GPU. It supports major open-source LLMs like LLaMA, Mistral, DeepSeek, QWEN, and Gemma.</p> <p>Once running, it exposes a REST API at <code>http://localhost:11434</code>, compatible with OpenAI-style inference.</p> <p>You can find more information on this at Ollama's official site: <a href="https://ollama.com">ollama.com</a></p> <pre><code class="language-python">Install and Run Ollama # Mac brew install ollama # Linux curl -fsSL https://ollama.com/install.sh | sh # Run a model ollama run llama3 </code></pre> <p>Running the code above downloads and launches LLaMA 3.2 locally. Once active, Ollama exposes endpoints that work with both synchronous and streaming chat.</p> <h2>AGNO + Ollama: An example of how to use them together to build an agent</h2> <p>Let’s build a storytelling agent using AGNO connected to Ollama.</p> <pre><code class="language-python">from agno.agent import Agent from agno.models.ollama import Ollama agent = Agent( model=Ollama(id="llama3.2", provider="Ollama"), description="You are an enthusiastic news reporter with a flair for storytelling!", markdown=True ) agent.print_response("Tell me about a breaking news story from New York.", stream=True)\ </code></pre> <h2>Parameters</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Parameter</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>model=Ollama(...)</code></td> <td>Connects to a local LLaMA model via Ollama</td> </tr> <tr> <td><code>description</code></td> <td>Agent personality and behavior</td> </tr> <tr> <td><code>markdown=True</code></td> <td>Outputs markdown-formatted content</td> </tr> </tbody> </table> <h2>No framework? No problem.</h2> <p>Frameworks like AGNO offer orchestration, but what if you're running in:</p> <ul> <li>Air-gapped networks</li> <li>Lightweight environments</li> <li>Custom experimental setups?</li> </ul> <p>Here’s how to build a raw agent pipeline using just:</p> <ul> <li>Ollama for LLM</li> <li>DuckDuckGo search for tool use</li> <li>Custom prompt logic</li> </ul> <h2>Full Python agent pipeline</h2> <pre><code class="language-python">from ollama import Client from duckduckgo_search import DDGS ollama_client = Client(host='http://localhost:11434') ### Tool: Web search via DuckDuckGo def duckduckgo_search(query, max_results=5): with DDGS() as ddgs:      return \[r for r in ddgs.text(query, region="wt-wt", safesearch="off", max_results=max_results)] </code></pre> <h3>Pipeline logic</h3> <pre><code class="language-python">def agent_pipeline(user_input): if "what" in user_input.lower() or "happening" in user_input.lower():      print("\[Tool] Searching DuckDuckGo...")      search_results = duckduckgo_search(user_input)      summary = "\n".join(\[f"- {r['title']}: {r\['href']}" for r in search_results]) else:      summary = "" if summary:      system_prompt = (          "You are an assistant. I found these recent search results:\n"          f"{summary}\n"          "Now generate a helpful answer for the user question below.\n"          f"User question: {user_input}"      ) else:      system_prompt = f"You are a helpful assistant. Answer this: {user_input}" response = ollama_client.chat(      model='llama3.2',      messages=\[{"role": "user", "content": system_prompt}] ) print("\n\[Agent Response]:") print(response\['message']\['content']) </code></pre> <h3>Example outputs</h3> <pre><code class="language-python">agent_pipeline("What's happening in New York?") agent_pipeline("Tell me a joke.") </code></pre> <h2>Breakdown of components</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Component</th> <th>Role</th> </tr> </thead> <tbody> <tr> <td><code>LLM Backend</code></td> <td>Ollama running LLaMA 3.2 locally</td> </tr> <tr> <td><code>Tool</code></td> <td>DuckDuckGo search for real-time info</td> </tr> <tr> <td><code>Orchestration</code></td> <td>Custom Python logic</td> </tr> <tr> <td><code>Agentic Behavior</code></td> <td>Manual Think → Act → Reflect implementation</td> </tr> </tbody> </table> <p>This pattern gives you full control with zero cloud dependency — and forms the base for private AI workflows.</p> <h2>Pro tip</h2> <p>Looking for multi-agent orchestration with Ollama?</p> <p><strong>Check out Langmanus — a framework for:</strong></p> <ul> <li>Graph-based agent orchestration</li> <li>Streaming LLM outputs</li> <li>Agent-task dependencies and coordination</li> </ul> <h2>Final thoughts</h2> <p>By combining AGNO, Ollama, and LLaMA3, developers can build fully private Agentic AI systems that:</p> <ul> <li>Work offline</li> <li>Are modular and extensible</li> <li>Use both tools and models interchangeably</li> <li>Scale from simple scripts to complex workflows</li> </ul> <p>This stack represents the future of <strong>agent design — grounded, capable, and locally operable.</strong></p>https://developer.hpe.com/part-8-agentic-ai-and-qdrant-building-semantic-memory-with-mcp-protocol/https://developer.hpe.com/part-8-agentic-ai-and-qdrant-building-semantic-memory-with-mcp-protocol/Mon, 21 Jul 2025 10:50:25 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>As <strong>Agentic AI</strong> systems evolve from reactive language models into structured thinkers, a new challenge emerges: <strong>how do we give these agents memory?</strong> Not just basic logs or static files, but real, <strong>searchable memory</strong> that understands and adapts to context over time.</p> <p>This is where tools like <strong>Qdrant</strong> and the <strong>Model Context Protocol (MCP)</strong> come in—a modular pairing that brings semantic search and long-term knowledge storage into agent workflows. Together, they enable agents to not only recall relevant information but to reason across past experiences, making <strong>Agentic AI</strong> systems more intelligent, adaptive, and human-like in their decision-making.</p> <p><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-8-agentic-ai-mcp-281567e26838">Inspired by my Medium post</a>, this article explores how <strong>MCP</strong>, the <strong>Model Context Protocol</strong>—a kind of connective tissue between LLMs and external tools or data sources—<strong>standardizes interactions</strong> between intelligent agents and vector databases like <strong>Qdrant</strong>. By enabling seamless storage and retrieval of embeddings, agents can now “remember” useful information and leverage it in future reasoning.</p> <p>Let’s walk through the full architecture and code implementation of this cutting-edge combination.</p> <h2>LLMs + MCP + Database = Thoughtful Agentic AI</h2> <p>In Agentic AI, a language model doesn’t just generate — it thinks, acts, and reflects using external tools. That’s where MCP comes in.</p> <p>Think of MCP as a “USB interface” for AI — it lets agents plug into tools like Qdrant, APIs, or structured databases using a consistent protocol.</p> <p>Qdrant itself is a high-performance vector database — capable of powering semantic search, knowledge retrieval, and acting as long-term memory for AI agents. However, direct integration with agents can be messy and non-standardized.</p> <p>This is solved by wrapping Qdrant inside an MCP server, giving agents a semantic API they can call like a function.</p> <h3>Architecture overview</h3> <pre><code class="language-cwl">[LLM Agent] | |-- [MCP Client] [MCP Protocol] | |-- [Qdrant MCP Server] |   |-- Tool: qdrant-store |   |-- Tool: qdrant-find | [Qdrant Vector DB] </code></pre> <h3>Use case: Support ticket memory for AI assistants</h3> <p>Imagine an AI assistant answering support queries.</p> <ul> <li>It doesn't have all answers built-in.</li> <li>But it has semantic memory from prior support logs stored in Qdrant.</li> <li>It uses qdrant-find to semantically retrieve similar issues .</li> <li>It then formulates a contextual response.</li> </ul> <h2>Step-by-step implementation</h2> <h3>Step 1: Launch Qdrant MCP Server</h3> <pre><code class="language-cwl">export COLLECTION_NAME="support-tickets" export QDRANT_LOCAL_PATH="./qdrant_local_db" export EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2" </code></pre> <pre><code class="language-cwl">uvx mcp-server-qdrant --transport sse </code></pre> <h2>Key parameters:</h2> <ul> <li>COLLECTION_NAME: Name of the Qdrant collection</li> <li>QDRANT_LOCAL_PATH: Local vector DB storage path</li> <li>EMBEDDING_MODEL: Embedding model for vectorization</li> </ul> <h3>Step 2: Connect the MCP Client</h3> <pre><code class="language-python">from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client async def main(): server_params = StdioServerParameters(      command="uvx",      args=\["mcp-server-qdrant"],      env={          "QDRANT_LOCAL_PATH": "./qdrant_local_db",          "COLLECTION_NAME": "support-tickets",          "EMBEDDING_MODEL": "sentence-transformers/all-MiniLM-L6-v2"      } ) </code></pre> <pre><code class="language-python">async with stdio_client(server_params) as (read, write):      async with ClientSession(read, write) as session:          await session.initialize()          tools = await session.list_tools()          print(tools) </code></pre> <pre><code class="language-cwl">Expected Output: Lists tools like qdrant-store, qdrant-find </code></pre> <h3>Step 3: Ingest a new memory</h3> <pre><code class="language-python">ticket_info = "Order #1234 was delayed due to heavy rainfall in transit zone." result = await session.call_tool("qdrant-store", arguments={ "information": ticket_info, "metadata": {"order_id": 1234} }) </code></pre> <p>This stores an embedded version of the text in Qdrant.</p> <h3>Step 4: Perform a semantic search</h3> <pre><code class="language-python">query = "Why was order 1234 delayed?" search_response = await session.call_tool("qdrant-find", arguments={ "query": "order 1234 delay" }) </code></pre> <h3>Example output:</h3> <pre><code class="language-cwl">[   { "content": "Order #1234 was delayed due to heavy rainfall in transit zone.", "metadata": {"order_id": 1234}   } ] </code></pre> <h3>Step 5: Use with LLM</h3> <pre><code class="language-python">import openai context = "\n".join(\[r["content"] for r in search_response]) prompt = f""" You are a helpful assistant. Use this context to answer: """ {context} """ Question: Why was order #1234 delayed? """ response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}] ) print(response["choices"][0]["message"]["content"]) </code></pre> <h3>Final answer:</h3> <pre><code class="language-cwl">"Order #1234 was delayed due to heavy rainfall in the transit zone." </code></pre> <h2>Parameter reference</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Tool</th> <th>Parameter</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>qdrant-store</code></td> <td><code>information</code></td> <td>Raw string to embed</td> </tr> <tr> <td></td> <td><code>metadata</code></td> <td>Optional metadata for filtering</td> </tr> <tr> <td><code>qdrant-find</code></td> <td><code>query</code></td> <td>Natural language query</td> </tr> <tr> <td><code>env var</code></td> <td><code>EMBEDDING_MODEL</code></td> <td>Model used to create embeddings</td> </tr> <tr> <td><code>env var</code></td> <td><code>COLLECTION_NAME</code></td> <td>Qdrant vector collection name</td> </tr> </tbody> </table> <h2>Pro tip: Chain MCP servers</h2> <p>You can deploy multiple MCP servers for different tools and plug them into agent workflows:</p> <ul> <li>qdrant-find for memory</li> <li>google-search for web data</li> <li>postgres-query for structured facts</li> </ul> <p>Then, orchestrate it all using Agentic AI Teams to perform high-level, multi-tool reasoning.</p> <h2>Final thoughts</h2> <p>By pairing Qdrant with MCP, Agentic AI gains powerful, semantic memory — a critical enabler of contextual understanding and long-term knowledge retention. This pattern abstracts the complexity of vector DBs behind a unified protocol, empowering agents to think, recall, and act without manual data plumbing.</p> <p>As the AI stack modularizes further, approaches like this will form the backbone of scalable, pluggable, and intelligent multi-agent ecosystems.</p>https://developer.hpe.com/part-7-how-collaborative-teams-of-agents-unlock-new-intelligence/https://developer.hpe.com/part-7-how-collaborative-teams-of-agents-unlock-new-intelligence/Mon, 21 Jul 2025 10:34:35 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>The rapid shift from <strong>Generative AI to Agentic AI</strong> marks more than a technical milestone—it represents a philosophical change in how machines reason, collaborate, and solve problems. Instead of relying on a single, all-purpose model, <strong>Agentic AI</strong> introduces a dynamic ecosystem of specialized agents that work together like human teams, each offering a distinct capability or perspective.</p> <p>One of the most transformative configurations in this space is <strong>Collaborate Mode</strong>, where multiple agents contribute—either asynchronously or in parallel—to achieve a unified outcome. This mode enables more nuanced problem-solving, especially in complex workflows where different types of reasoning, tools, or perspectives must come together seamlessly.</p> <p><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-7-agentic-ai-a-13ee0b43bc42">Inspired by my Medium post,</a> this blog breaks down the architecture, purpose, and code implementation of this mode using the AGNO framework, making the power of distributed machine collaboration more approachable and actionable.</p> <center><img src="/img/screenshot-2025-07-21-at-4.06.15 pm.png" width="600" height="550" alt="LLM Mode" title="LLM Mode"></center> <h2>What is Collaborate Mode?</h2> <p>Collaborate Mode is an agent orchestration strategy where multiple intelligent agents receive the same task, operate independently, and deliver unique insights that are then synthesized by a coordinator. This design mirrors how effective human teams operate—through parallel expertise, independent judgment, and collaborative synthesis.</p> <h3>Ideal use cases:</h3> <ul> <li>Brainstorming across different domains</li> <li>Aggregating cross-platform knowledge</li> <li>Speeding up research through parallelism</li> <li>Building consensus across diverse information sources</li> </ul> <h2>How it works visually</h2> <p>Imagine each agent as a researcher assigned to a unique platform:</p> <ul> <li>Reddit Agent gathers opinions from communities</li> <li>HackerNews Agent scans developer insights</li> <li>Twitter Agent captures trending conversations</li> <li>Academic Agent retrieves scholarly context</li> </ul> <p>Each one returns findings from its ecosystem, which the coordinator blends into a single, meaningful response.</p> <h2>AGNO framework code implementation</h2> <h3>1. Import modules &#x26; tools</h3> <pre><code class="language-python">from textwrap import dedent from agno.agent import Agent from agno.models.openai import OpenAIChat from agno.team.team import Team from agno.tools.arxiv import ArxivTools from agno.tools.duckduckgo import DuckDuckGoTools from agno.tools.googlesearch import GoogleSearchTools from agno.tools.hackernews import HackerNewsTools </code></pre> <h3>2. Define specialized agents</h3> <p>Each agent is built for platform-specific intelligence gathering.</p> <h3>Reddit Agent</h3> <pre><code class="language-python">reddit_researcher = Agent( name="Reddit Researcher", role="Research a topic on Reddit", model=OpenAIChat(id="gpt-4o"), tools=\[DuckDuckGoTools()], add_name_to_instructions=True, instructions=dedent("""You are a Reddit researcher..."""), ) </code></pre> <h3>HackerNews Agent</h3> <pre><code class="language-python">hackernews_researcher = Agent( name="HackerNews Researcher", model=OpenAIChat("gpt-4o"), role="Research a topic on HackerNews.", tools=\[HackerNewsTools()], add_name_to_instructions=True, instructions=dedent("""You are a HackerNews researcher..."""), ) </code></pre> <h3>Academic Agent</h3> <pre><code class="language-python">academic_paper_researcher = Agent( name="Academic Paper Researcher", model=OpenAIChat("gpt-4o"), role="Research academic papers...", tools=\[GoogleSearchTools(), ArxivTools()], add_name_to_instructions=True, instructions=dedent("""You are an academic researcher..."""), ) </code></pre> <h2>Twitter - X Agent</h2> <pre><code class="language-python">twitter_researcher = Agent( name="Twitter Researcher", model=OpenAIChat("gpt-4o"), role="Research Twitter/X topics", tools=[DuckDuckGoTools()], add_name_to_instructions=True, instructions=dedent("""You are a Twitter researcher..."""), ) </code></pre> <h3>3. Define the team</h3> <pre><code class="language-python">agent_team = Team( name="Discussion Team", mode="collaborate", model=OpenAIChat("gpt-4o"), members=[      reddit_researcher,      hackernews_researcher,      academic_paper_researcher,      twitter_researcher, ], instructions=[      "You are a discussion master.",      "You must conclude the discussion once consensus is reached.", ], success_criteria="The team has reached a consensus.", update_team_context=True, send_team_context_to_members=True, show_tool_calls=True, markdown=True, show_members_responses=True, ) </code></pre> <h3>4. Running the discussion</h3> <pre><code class="language-python">if __name__ == "__main__": asyncio.run(      agent_team.print_response(          message="Start the discussion on the topic: 'What is the best way to learn to code?'",          stream=True,          stream_intermediate_steps=True,      ) ) </code></pre> <h3>Example output</h3> <pre><code class="language-python">* Reddit: Focus on project-building and freeCodeCamp * HackerNews: Start with Python and open-source * Academia: Reinforce with spaced repetition and mentorship * Twitter/X: Emphasize consistency and public learning * Team Consensus: Use beginner-friendly languages, build real-world projects, and immerse yourself in learning communities. </code></pre> <center><img src="/img/screenshot-2025-07-21-at-4.06.02 pm.png" width="600" height="550" alt="Agent Parameters" title="Agent Parameters"></center> <h2>Pro tip: Run agents in parallel</h2> <pre><code class="language-python">asyncio.run( agent_team.print_response(      message="Start the discussion on the topic: 'How should we improve remote team collaboration?'",      stream=True,      stream_intermediate_steps=True, )) </code></pre> <p>Using asyncio ensures agents work simultaneously, which dramatically boosts speed and output quality—especially in research-heavy or time-sensitive use cases.</p> <h2>Final thoughts</h2> <p>Collaborate Mode is more than a clever orchestration pattern—it’s the embodiment of distributed intelligence. By mimicking the structure of human brainstorming, it allows AI to perform with greater breadth, depth, and creativity. With frameworks like AGNO making implementation seamless, the age of intelligent, agent-led collaboration is no longer speculative—it’s operational.</p> <blockquote> <p><em>As we continue evolving from single-shot prompts to structured autonomy, Collaborate Mode stands out as a key innovation for scalable, multi-perspective problem-solving in AI systems.</em></p> </blockquote>https://developer.hpe.com/part-6-agentic-ai-teams-in-router-mode-multilingual-routing-with-agno/https://developer.hpe.com/part-6-agentic-ai-teams-in-router-mode-multilingual-routing-with-agno/Mon, 21 Jul 2025 10:04:38 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>One of the most powerful capabilities in Agentic AI is orchestrating multiple agents to work together—each with its own specialization. In this segment, we explore the <strong>Router Mode</strong> pattern, a configuration where a central team detects <strong>context</strong> (like language or domain) and routes queries to the right agent accordingly.</p> <blockquote> <p><strong>What exactly is "context"?</strong> In Agentic AI, <em>context</em> refers to the key details in a user's input that help the system understand how to respond appropriately. Think of it like clues that tell the AI what kind of help is needed. For example, if a user submits a query in Hindi, the language itself becomes part of the context. Similarly, if the query mentions "insurance claims" or "server configuration," that reveals the domain or topic the user is focused on.</p> <p><strong>Simple Analogy:</strong> Imagine walking into a large helpdesk at an international airport. You speak to a receptionist and ask for assistance. Based on your language, destination, or issue (lost luggage vs. visa questions), they direct you to the right expert—someone who speaks your language or understands your problem. That receptionist is acting like the router agent. They’re not solving the issue themselves but are smart enough to know <em>who</em> should help you based on <em>context</em>. That’s exactly what the Router Mode does in Agentic AI.</p> </blockquote> <p>This method is especially effective in scenarios requiring multilingual or domain-specific support. Using the <strong>AGNO framework</strong>, we’ll see how to construct a language-routing team that handles diverse user inputs with precision and fallback logic—making it especially friendly for no-code or low-code setups.</p> <p><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-6-agentic-ai-a-39714050857b">Inspired by my Medium post.</a></p> <h2>Router Mode: What it is?</h2> <p>In Router Mode, the team acts like a switchboard, rather than executing tasks itself. Its core responsibility is to:</p> <ul> <li>Analyze user input</li> <li>Detect the appropriate context (e.g., language)</li> <li>Route the request to a specialized agent</li> <li>Handle unsupported inputs gracefully</li> </ul> <center><img src="/img/screenshot-2025-07-21-at-3.38.34 pm.png" width="600" height="550" alt="Route Mode" title="Route Mode"></center> <h3>Use case: Multilingual chat support</h3> <p>Imagine a chatbot that receives queries in different languages. Router Mode enables:</p> <ul> <li>Language detection</li> <li>Delegation to language-specific agents (e.g., Japanese, French, German)</li> <li>Fallback messages for unsupported languages</li> </ul> <h3>Implementation: AGNO framework setup</h3> <p>We’ll define a set of language-specific agents and create a routing team that delegates accordingly.</p> <h4>Step 1: Define language agents</h4> <pre><code class="language-python">from agno.agent import Agent from agno.models.anthropic import Claude from agno.models.deepseek import DeepSeek from agno.models.mistral.mistral import MistralChat from agno.models.openai import OpenAIChat english_agent = Agent( name="English Agent", role="You can only answer in English", model=OpenAIChat(id="gpt-4.5-preview"), instructions=["You must only respond in English"], ) japanese_agent = Agent( name="Japanese Agent", role="You can only answer in Japanese", model=DeepSeek(id="deepseek-chat"), instructions=["You must only respond in Japanese"], ) chinese_agent = Agent( name="Chinese Agent", role="You can only answer in Chinese", model=DeepSeek(id="deepseek-chat"), instructions=["You must only respond in Chinese"], ) spanish_agent = Agent( name="Spanish Agent", role="You can only answer in Spanish", model=OpenAIChat(id="gpt-4.5-preview"), instructions=["You must only respond in Spanish"], ) french_agent = Agent( name="French Agent", role="You can only answer in French", model=MistralChat(id="mistral-large-latest"), instructions=["You must only respond in French"], ) german_agent = Agent( name="German Agent", role="You can only answer in German", model=Claude("claude-3-5-sonnet-20241022"), instructions=["You must only respond in German"], ) </code></pre> <h3>Step 2: Create the Router team</h3> <pre><code class="language-python">from agno.team.team import Team multi_language_team = Team( name="Multi Language Team", mode="route", model=OpenAIChat("gpt-4.5-preview"), members=[ english_agent, spanish_agent, japanese_agent, french_agent, german_agent, chinese_agent, ], show_tool_calls=True, markdown=True, show_members_responses=True, instructions=[ "You are a language router that directs questions to the appropriate language agent.", "If the user asks in a language whose agent is not a team member, respond in English with:", "'I can only answer in the following languages: English, Spanish, Japanese, French and German. Please ask your question in one of these languages.'", "Always check the language of the user's input before routing to an agent.", "For unsupported languages like Italian, respond in English with the above message.", ] ) </code></pre> <h3>Step 3: Run multilingual examples</h3> <pre><code class="language-python">multi_language_team.print_response("How are you?", stream=True) # English multi_language_team.print_response("你好吗？", stream=True) # Chinese multi_language_team.print_response("お元気ですか?", stream=True) # Japanese multi_language_team.print_response("Comment allez-vous?", stream=True) # French </code></pre> <h3>Output:</h3> <pre><code class="language-python">[English Agent]: I'm doing great! How can I help you today? [Chinese Agent]: 我很好，谢谢你的关心。你呢？ [Japanese Agent]: 元気です。あなたはお元気ですか？ [French Agent]: Je vais bien, merci. Et vous ? </code></pre> <center><img src="/img/screenshot-2025-07-21-at-3.38.47 pm.png" width="600" height="550" alt="Agent Mode Parameters" title="Agent Mode Parameters"></center> <h2>Key parameters explained</h2> <table border="1" cellpadding="8" cellspacing="0" style="border-collapse: collapse; width: 100%;"> <thead style="background-color:#f2f2f2"> <tr> <th>Parameter</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>mode="route"</code></td> <td>Instructs the team to act as a switchboard, not an executor</td> </tr> <tr> <td><code>show_members_responses=True</code></td> <td>Displays individual agent replies for traceability</td> </tr> <tr> <td><code>instructions\\\[]</code></td> <td>Core router logic: detect language, enforce exclusivity, manage fallbacks</td> </tr> <tr> <td><code>model=OpenAIChat(...)</code></td> <td>Backbone LLM used by the router team for input analysis</td> </tr> </tbody> </table> <h2>Why Router Mode works</h2> <ul> <li>Context-awareness: Inputs are analyzed for language, not just keywords</li> <li>Agent exclusivity: Each agent strictly operates in its assigned language</li> <li>Fallback resilience: Unsupported queries are met with a clear, unified message</li> <li>Modularity: Each language agent is replaceable or extendable</li> </ul> <h2>Conclusion</h2> <p>Router Mode in Agentic AI introduces scalable intelligence by structuring agents like a multilingual team. Rather than overwhelming a single agent, you delegate responsibility across specialists and keep interactions clean, accurate, and context-driven.</p> <p>With the AGNO framework, creating such intelligent, language-aware teams becomes seamless — and your agents become not just reactive, but well-organized and self-aware of their boundaries.</p> <p>A structured team, strict instructions, and intelligent routing — that’s the future of responsive AI.</p>https://developer.hpe.com/part-5-agentic-ai-team-coordination-mode-in-action/https://developer.hpe.com/part-5-agentic-ai-team-coordination-mode-in-action/Mon, 21 Jul 2025 07:24:24 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>One of the most transformative patterns in Agentic AI is team-based orchestration — a collaborative approach where specialized <strong>agents work together to fulfill complex goals</strong>. In this edition, we explore coordinate mode using the AGNO framework — a design where a team manager delegates, supervises, and integrates the contributions of each agent.</p> <p><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-5-agentic-ai-a-2d6651c9cc5c">Inspired by my Medium post.</a></p> <center><img src="/img/screenshot-2025-07-21-at-12.57.22 pm.png" width="600" height="550" alt="LLM Mode" title="LLM Mode"></center> <h2>What are agentic AI teams?</h2> <p>An agentic team is a structured collection of AI agents, each performing a specific role with autonomy and tool access. Teams can include roles like:</p> <ul> <li>Researcher: Finds and filters relevant data</li> <li>Writer: Synthesizes content with tone and structure</li> <li>Translator: Converts content across languages</li> <li>Planner: Organizes execution based on goals</li> </ul> <h3>In Coordinate Mode:</h3> <ul> <li>A team manager Agent directs the flow of tasks</li> <li>Individual agents handle sub-tasks independently</li> <li>Final results are reviewed, refined, and unified by the manager</li> </ul> <h2>AGNO Framework: Coordinating a multi-agent content team</h2> <p>Let’s examine a professional-grade configuration of a New York Times-style editorial team, where search, writing, and editorial review are handled by distinct agents.</p> <h3>Imports</h3> <pre><code class="language-python">from agno.agent import Agent from agno.models.openai import OpenAIChat from agno.team.team import Team from agno.tools.search import DuckDuckGoTools from agno.tools.read import Newspaper4kTools </code></pre> <h3>Searcher agent</h3> <pre><code class="language-python">searcher = Agent( name="Searcher", role="Searches the top URLs for a topic", instructions=[ "Generate 3 search terms for a topic.", "Search the web and return 10 high-quality, relevant URLs.", "Prioritize credible sources, suitable for the New York Times." ], tools=[DuckDuckGoTools()], add_datetime_to_instructions=True, ) </code></pre> <h3>Writer agent</h3> <pre><code class="language-python">writer = Agent( name="Writer", role="Writes a high-quality article", description="Senior NYT writer tasked with long-form editorial content.", instructions=[ "Read all articles using `read_article`.", "Write a structured, engaging article of at least 15 paragraphs.", "Support arguments with factual citations and ensure clarity.", "Never fabricate facts or plagiarize content." ], tools=[Newspaper4kTools()], add_datetime_to_instructions=True, ) </code></pre> <h3>Editor team (Manager agent in Coordinate Mode)</h3> <pre><code class="language-python">editor = Team( name="Editor", mode="coordinate", model=OpenAIChat("gpt-4o"), members=[searcher, writer], description="You are a senior NYT editor coordinating the team.", instructions=[ "Delegate research to the search agent.", "Delegate drafting to the writer.", "Review, proofread, and enhance the final article.", "Maintain NYT-level quality, structure, and tone." ], add_datetime_to_instructions=True, send_team_context_to_members=True, show_members_responses=True, markdown=True, ) </code></pre> <h3>Running the team</h3> <pre><code class="language-python">Method 1: Print output directly editor.print_response("Write an article about latest developments in AI.") Method 2: Get raw result response = editor.run("Write an article about latest developments in AI.") </code></pre> <h3>Key parameters explained</h3> <table> <thead style="background-color:#f2f2f2"> <tr> <th>Parameter</th> <th>Purpose</th> </tr> </thead> <tbody> <tr> <td><code>mode="coordinate"</code></td> <td>Enables structured delegation and task flow</td> </tr> <tr> <td><code>members=\\\\\[...]</code></td> <td>Assigns role-specific agents</td> </tr> <tr> <td><code>send_team_context_to_members</code></td> <td>Shares global task context with all agents</td> </tr> <tr> <td><code>show_members_responses=True</code></td> <td>Displays each member's intermediate output</td> </tr> <tr> <td><code>add_datetime_to_instructions</code></td> <td>Contextualizes outputs with current date/time</td> </tr> </tbody> </table> <h2>Pro tip: Define success criteria</h2> <p>Adding success criteria helps agents align their efforts with measurable outcomes.</p> <pre><code class="language-python">strategy_team = Team( members=[market_analyst, competitive_analyst, strategic_planner], mode="coordinate", name="Strategy Team", description="A team that develops strategic recommendations", success_criteria="Produce actionable strategic recommendations supported by market and competitive analysis", ) response = strategy_team.run( "Develop a market entry strategy for our new AI-powered healthcare product" ) </code></pre> <p>This ensures agents not only act — but act with strategic purpose and direction.</p> <center><img src="/img/screenshot-2025-07-21-at-12.57.44 pm.png" width="600" height="550" alt="Agentic AI Parameters" title="Agentic AI Parameters"></center> <h2>Conclusion</h2> <p>Coordinate Mode in Agentic AI exemplifies intelligent task distribution, where specialized agents work under centralized leadership to deliver complex, high-quality outputs. The AGNO framework simplifies this orchestration through agent roles, tool integration, and goal alignment <strong>—</strong> <strong>enabling scalable, auditable AI workflows.</strong></p> <p>From editorial pipelines to business strategy engines, multi-agent coordination is redefining how work gets done <strong>— autonomously, intelligently, and collaboratively.</strong></p>https://developer.hpe.com/part-4-the-rise-of-agentic-ai-and-the-power-of-the-agno-framework/https://developer.hpe.com/part-4-the-rise-of-agentic-ai-and-the-power-of-the-agno-framework/Mon, 21 Jul 2025 07:02:23 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>As artificial intelligence continues its rapid evolution, a new frontier has emerged — Agentic AI. This paradigm moves us beyond passive, prompt-based LLMs and into an era where AI doesn’t just respond — <strong>it thinks, plans, acts, and collaborates.</strong></p> <p>Building on insights inspired by <a href="https://dineshr1493.medium.com/agentic-ai-framework-a4df29a8fc62">my post on Medium,</a> this guide explores what Agentic AI truly is, why it matters, and how modern frameworks like AGNO (formerly Phidata) are enabling intelligent agent-based systems that work autonomously in real-world settings.</p> <p>Let’s step into the mechanics of intelligent agents and discover how they’re transforming how work gets done.</p> <h2>What is Agentic AI?</h2> <p><strong>Agentic AI</strong> refers to AI systems designed not just to generate content, but to <strong>autonomously reason, decide, and execute tasks —</strong> often in coordination with external tools or other agents.</p> <p>Unlike basic LLMs or traditional “LLM + tool” stacks, Agentic AI systems can:</p> <ul> <li><strong>Deconstruct complex goals into sub-tasks</strong></li> <li><strong>Delegate and execute those sub-tasks via specialized agents</strong></li> <li><strong>Integrate tools, APIs, and live data sources to take meaningful actions</strong></li> <li><strong>Reflect on their outputs and improve over time</strong></li> </ul> <p>This evolution from reactive chatbots to proactive agents is redefining automation and digital intelligence.</p> <center><img src="/img/screenshot-2025-07-21-at-12.45.30 pm.png" width="600" height="550" alt="LLM Evolution" title="LLM Evolution"></center> <h2>The AGNO framework (Previously Phidata)</h2> <p><strong>AGNO</strong> is an open-source framework purpose-built to create modular, autonomous AI agents that <strong>think, plan, act, and adapt</strong>. It’s one of the most advanced and flexible toolkits for building <em><strong>real-world Agentic AI systems.</strong></em></p> <p><strong>Core capabilities:</strong></p> <ul> <li><strong>Contextual reasoning</strong> through logic chains</li> <li><strong>Task planning and delegation</strong></li> <li><strong>Tool invocation</strong> (APIs, databases, automation systems)</li> <li><strong>Result reflection</strong> for improved decisions</li> <li><strong>Multi-agent orchestration</strong> at scale</li> <li><strong>Streaming support</strong> using protocols like Model Context Protocol (MCP)</li> <li><strong>Workflow visualization</strong> and agent team configurations</li> </ul> <p>🔗 GitHub: <a href="https://github.com/agno-agi/agno">AGNO Framework</a></p> <h2><strong>Agents, tools, and teams — The building blocks</strong></h2> <h3><strong>1. Agents</strong></h3> <p>An <strong>agent</strong> is a self-contained AI module designed to handle a specific task or role.</p> <ul> <li>Operates autonomously or as part of a team</li> <li>Can invoke tools, fetch data, or generate content</li> <li>Uses reasoning and memory to complete goals</li> </ul> <h3><strong>2. Tools</strong></h3> <p>Agents in AGNO use <strong>tools</strong> to interact with the real world. These can be:</p> <ul> <li>APIs (e.g., Google Search, Slack, Salesforce)</li> <li>Databases (e.g., Postgres, MongoDB, Qdrant)</li> <li>Custom internal services (e.g., CRMs, file systems)</li> <li>Processing modules (e.g., calculators, formatters)</li> </ul> <h3><strong>3. Teams</strong></h3> <p>Agents can collaborate through structured <strong>team modes</strong> for complex, multi-faceted workflows.</p> <h2><strong>Modes of teamwork in AGNO</strong></h2> <p>Modes are the means by which agents communicate with each other I will be walking you through few common modes of agents communication.</p> <center><img src="/img/screenshot-2025-07-21-at-12.45.57 pm.png" width="600" height="550" alt="Modes of Teamwork in AGNO" title="Modes of Teamwork in AGNO"></center> <h3><strong>Coordinator Mode</strong></h3> <p>In <strong>Coordinator Mode</strong>, a central agent takes charge of assigning and managing sub-tasks across a network of specialized agents. Think of it like a project manager in a team—delegating responsibilities, tracking progress, and assembling the final output.</p> <ul> <li><strong>Acts as an orchestrator</strong>, breaking down complex goals into manageable parts</li> <li><strong>Delegates tasks</strong> to the most capable agents based on their expertise</li> <li><strong>Aggregates results</strong> and presents a unified final outcome</li> <li><strong>Excels in hierarchical workflows</strong>, such as multi-step reasoning, multi-stage content generation, or structured decision-making pipelines</li> </ul> <p>This mode becomes particularly powerful when tasks require sequencing, prioritization, or dependency handling across multiple agents.</p> <blockquote> <p><em><a href="https://developer.hpe.com/blog/part-5-agentic-ai-team-coordination-mode-in-action/">Will be explored in depth in Part 5.</a></em></p> </blockquote> <h3><strong>Router Mode</strong></h3> <p>In <strong>Router Mode</strong>, tasks are automatically routed to the most appropriate agent based on the type, language, or domain of the query—without requiring manual intervention.</p> <ul> <li><strong>Lightweight and fast</strong>: It doesn’t require the central agent to deeply understand or process the query itself. Instead, it acts like a traffic controller—quickly identifying what the query is about and directing it to the right specialized agent. This makes it highly efficient, especially in high-volume environments.</li> <li><strong>Common in chatbots, support desks, and multi-skilled assistants</strong>: For example, in a multilingual support bot, Router Mode can detect the language of a user query and route it to an agent that handles that language. Or it might detect whether a question is about billing, tech support, or product features and send it to the corresponding expert agent.</li> </ul> <blockquote> <p><em><a href="https://developer.hpe.com/blog/part-6-agentic-ai-teams-in-router-mode-multilingual-routing-with-agno/">Detailed breakdown coming in Part 6.</a></em></p> </blockquote> <h3><strong>Collaborator Mode</strong></h3> <p>In <strong>Collaborator Mode</strong>, agents work together dynamically—<strong>sharing knowledge, negotiating decisions, and contributing their perspectives</strong>—to reach a common goal. Unlike Router or Coordinator modes, this pattern embraces simultaneous or iterative agent interactions that mirror how real-world teams brainstorm, refine ideas, or co-develop solutions.</p> <ul> <li><strong>Best for consensus-driven tasks</strong>, where multiple viewpoints or skills need to be considered</li> <li><strong>Ideal for creative and collective output</strong>, such as writing, strategy development, or decision support</li> <li><strong>Common in research, design, and system planning</strong>, where exploration, feedback, and iteration are essential</li> </ul> <blockquote> <p><em><a href="https://developer.hpe.com/blog/part-7-how-collaborative-teams-of-agents-unlock-new-intelligence/">Deep dive ahead in Part 7.</a></em></p> </blockquote> <center><img src="/img/screenshot-2025-07-21-at-12.46.13 pm.png" width="600" height="550" alt="Pro Insight: Langmanus — A Complementary Framework" title="Pro Insight: Langmanus — A Complementary Framework"></center> <h2><strong>Pro Insight: Langmanus — A Complementary Framework</strong></h2> <p>For developers seeking visual workflows and advanced task orchestration, Langmanus on GitHub offers:</p> <ul> <li>Workflow graphs and dashboards</li> <li>Real-time task delegation</li> <li>Progress tracking across agent teams</li> </ul> <p>Its system architecture includes:</p> <ul> <li><strong>Coordinator</strong> — Routes initial queries</li> <li><strong>Planner</strong> — Builds strategies</li> <li><strong>Supervisor</strong> — Oversees agents</li> <li><strong>Researcher</strong> — Gathers info</li> <li><strong>Coder</strong> — Handles code tasks</li> <li><strong>Browser</strong> — Performs online searches</li> <li><strong>Reporter</strong> — Summarizes outcomes</li> </ul> <p>GitHub: <a href="https://github.com/langmanus/langmanus">Langmanus Repository</a></p> <h2>Conclusion</h2> <p><strong>Agentic AI represents a turning point</strong> in artificial intelligence — a shift from passive, text-based outputs to autonomous, context-aware action systems. With frameworks like AGNO, developers can create agents that plan, reason, and act just like humans would in complex workflows.</p> <p>These agents aren’t just smarter — they’re collaborative, modular, and capable of evolving with the task at hand. As more organizations adopt these systems, the future of automation will belong not to static scripts, but to dynamic agents working in harmony.</p> <blockquote> <p><strong><a href="https://developer.hpe.com/blog/part-5-agentic-ai-team-coordination-mode-in-action/">Up next in Part 5:</a></strong> We’ll dive deep into <strong>Coordinator Mode</strong> and how AGNO orchestrates multi-agent task flows like a seasoned project manager.</p> </blockquote>https://developer.hpe.com/model-context-protocol-mcp-the-protocol-that-powers-ai-agents/https://developer.hpe.com/model-context-protocol-mcp-the-protocol-that-powers-ai-agents/Fri, 18 Jul 2025 14:23:55 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>As AI agents grow beyond text generation into autonomous problem-solvers, a new challenge emerges — communication. Not between humans and AI, but between AI and the vast world of services, APIs, databases, and tools. That’s where <strong>Model Context Protocol (MCP)</strong> steps in.</p> <p>Inspired by <a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-3-mcp-model-context-f026578ff0dd">my post on medium</a>, this blog post demystifies the MCP standard — reinterpreted with clarity, depth, and real-world relevance to help you understand how AI agents actually get things done. If LLMs are the brains, MCP is the nervous system connecting them to the real world. Let’s unpack how this protocol makes agentic AI functional, contextual, and enterprise-ready.</p> <center><img src="/img/mcp1.png" width="600" height="550" alt="MCP Arch" title="MCP Arch"></center> <h2>What is MCP, and why does it matter?</h2> <p>At its core, MCP is a standardized way for AI agents to communicate with external services. Instead of treating each tool or database as a black box, MCP defines a consistent interface — allowing the agent to send structured requests and receive contextual responses.</p> <p>Imagine an agent saying:</p> <p>“Here’s the context, here’s what I need — now act smartly based on it.”</p> <p>That’s the essence of MCP. It removes ambiguity, reduces dependency on ad hoc code, and enables agents to <strong>perform tasks with understanding, not just commands.</strong></p> <center><img src="/img/mcp2.png" width="600" height="550" alt="MCP Flow" title="MCP Flow"></center> <h2>The building blocks of MCP</h2> <p>MCP is comprised of three major components:</p> <ul> <li> <p>MCP client: Resides inside the AI agent and is responsible for making requests.</p> </li> <li> <p>MCP server: Wraps around external tools or services and handles incoming requests.</p> </li> <li> <p>MCP protocol: Uses JSON-RPC over transport layers like:</p> <ul> <li>Standard IO for local service calls</li> <li>Server-Sent Events (SSE) for remote or network-based integrations.</li> </ul> </li> </ul> <center><img src="/img/mcp3.png" width="600" height="550" alt="MCP Working" title="MCP Working"></center> <h2>How MCP works — The flow</h2> <p>Here’s a simplified view of the interaction:</p> <ol> <li> <p>The agent asks its MCP client to perform a task.</p> </li> <li> <p>The MCP client sends a well-formed JSON-RPC request to the MCP server.</p> </li> <li> <p>The MCP server either:</p> <ul> <li>Executes a tool (e.g., semantic_search)</li> <li>Fetches data (e.g., a file or DB record)</li> <li>Returns a structured prompt (e.g., a Q&#x26;A template)</li> </ul> </li> <li> <p>The MCP server streams back results or updates.</p> </li> <li> <p>The agent uses this data to reflect, re-plan, or execute the next step.</p> </li> </ol> <p>This architecture ensures that AI agents don’t just interact with data — they do so with awareness and strategy.</p> <h2>MCP + Reflection + Meta-Context = Smarter AI</h2> <p>What separates MCP from basic APIs is its inclusion of <strong>meta-context and reflection:</strong></p> <ul> <li><strong>Meta-Context:</strong> Includes user role, session history, intent, and environment details.</li> <li><strong>Reflection:</strong> Agents can evaluate responses. If a query fails, they can retry with a better approach.</li> <li><strong>Context-Aware Tools:</strong> MCP servers can use meta-data to dynamically tailor responses.</li> <li><strong>Tool Discovery:</strong> Agents can ask, “What tools are available right now?” and adjust plans accordingly.</li> </ul> <p>This turns the agent into a <strong>situationally aware operator</strong>, not just a command runner.</p> <h2>The race of MCP</h2> <p>Curious about the groundbreaking <em><strong>startups racing to develop the next wave of MCP</strong></em> (Model Context Protocol) servers? In this roundup, we highlight the most innovative players redefining how AI agents access, interact with, and orchestrate information across tools, databases, financial platforms, and more. For each startup, you’ll find a brief overview of their core technology, real-world use cases, and direct links to explore their solutions further.</p> <p>Whether you're an AI developer, tech enthusiast, or enterprise looking to supercharge your workflows, discover how these emerging MCP platforms are shaping the future of AI-driven connectivity—unlocking seamless integrations and unprecedented automation across industries.</p> <table> <thead style="background-color:#f2f2f2"> <tr> <th>Startup</th> <th>Description</th> <th>Tech Focus</th> <th>Use Case</th> <th>Website</th> </tr> </thead> <tbody> <tr> <td><strong>Anthropic</strong></td> <td>Creators of MCP and Claude AI</td> <td>AI Research & Safety</td> <td>Secure tool access via MCP for Claude AI</td> <td><a href="https://anthropic.com">anthropic.com</a></td> </tr> <tr> <td><strong>Replit</strong></td> <td>Cloud IDE with AI capabilities</td> <td>Developer Tools & AI Agents</td> <td>MCP-powered code assistant in their IDE</td> <td><a href="https://replit.com">replit.com</a></td> </tr> <tr> <td><strong>Sourcegraph</strong></td> <td>Code intelligence & search platform</td> <td>Developer Productivity</td> <td>MCP to connect AI to codebases & tickets</td> <td><a href="https://sourcegraph.com">sourcegraph.com</a></td> </tr> <tr> <td><strong>Qdrant</strong></td> <td>Open-source vector database</td> <td>AI Infrastructure (RAG)</td> <td>MCP server for semantic memory in agents</td> <td><a href="https://qdrant.tech">qdrant.tech</a></td> </tr> <tr> <td><strong>Neon</strong></td> <td>Serverless Postgres provider</td> <td>Databases (Postgres Cloud)</td> <td>MCP for AI-driven Postgres analytics & ops</td> <td><a href="https://neon.tech">neon.tech</a></td> </tr> </tbody> </table> <h2>Real-world applications of MCP</h2> <ol> <li><strong>Faster integrations</strong> Instead of hard-coding APIs, developers can plug agents into pre-wrapped MCP servers. This dramatically shortens time-to-integration.</li> <li><strong>Live data access</strong> Agents can now access up-to-date information from production-grade systems — avoiding stale, hallucinated responses.</li> <li><strong>Enterprise control</strong> MCP enables governance: every action is logged, controlled, and auditable — essential for security-conscious environments.</li> <li><strong>Cross-agent compatibility</strong> Build a tool once, and any MCP-compliant agent can use it. No more agent-specific wrappers.</li> </ol> <h3><strong>Case study: Qdrant with MCP</strong></h3> <p><strong>Qdrant</strong> is a vector database used for semantic search. Here’s how it operates under MCP:</p> <ol> <li>MCP server exposes a tool like semantic_search(query: str).</li> <li>Agent calls: semantic_search("incident policy").</li> <li>Qdrant streams back relevant documents in real-time.</li> <li>The agent uses those documents as dynamic context to reason or response.</li> </ol> <p>This is vector search integrated into an agentic loop — not just storage, but intelligence.</p> <h3>Case study: PostgreSQL with MCP</h3> <p>A <strong>Postgres MCP Server</strong> might expose methods such as:</p> <ul> <li>get_sales(region: str, quarter: str).</li> <li>run_query(sql: str).</li> </ul> <p>An agent could now answer a prompt like:</p> <p>“What were APAC sales in Q4?”</p> <p>The Postgres MCP Server abstracts the SQL, safely executes it, and returns clean, structured results — instantly usable by the agent.</p> <p><strong>Leading startups driving MCP adoption</strong></p> <p>While <strong>Part 8</strong> will go deeper into startup ecosystems, here are some notable names in the industry who are building or supporting MCP infrastructure:</p> <ul> <li>Qdrant</li> <li>LangChain</li> <li>AutoGen by Microsoft</li> <li>OpenDevin</li> <li>Auto-GPT (community forks)</li> </ul> <p>These players are shaping a plug-and-play AI world where tools and agents speak a common protocol.</p> <h2>Conclusion</h2> <p>MCP is more than a technical standard — it's a <strong>philosophy of interoperability</strong> for the agentic era. It shifts AI from being a passive responder to an active participant in real-world systems. With MCP, agents don’t just have the ability to talk — they gain the <strong>power to think, act, adapt, and connect</strong> meaningfully.</p> <p>As we continue this series, <a href="https://developer.hpe.com/blog/part-4-the-rise-of-agentic-ai-and-the-power-of-the-agno-framework/">the next chapter</a> will spotlight a top Agentic AI framework and reveal how it uses MCP to orchestrate intelligent, autonomous workflows across environments.</p> <blockquote> <p><em>If you’re building with AI — or planning to — MCP is the connective tissue you can’t afford to ignore.</em></p> </blockquote>https://developer.hpe.com/how-to-use-dscc-api-and-ansible-to-collect-the-storage-configuration/https://developer.hpe.com/how-to-use-dscc-api-and-ansible-to-collect-the-storage-configuration/Tue, 15 Jul 2025 12:18:34 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Capturing the current storage configuration to verify it against best practices or configuration rules is something that customer request regularly. If the customer uses Ansible as their automation platform, the <a href="https://github.com/HewlettPackard/hpe3par_ansible_module?tab=readme-ov-file">HPE 3PAR Ansible module</a> can be used to create and delete hosts, volumes etc., but it is not really a solution for gathering the complete configuration.</p> <p>Furthermore, this module uses the WSAPI of individual Alletra storage systems. The HPE Data Services Cloud Console (DSCC) would be the better option to collect storage configuration data of multiple systems, even those that might be distributed across multiple sites. Through a single location, the DSCC would be able to get the data of all storage systems.</p> <p><a href="https://developer.hpe.com/blog/automating-operations-on-dscc-using-ansible-playbooks/">Ansible playbooks for the DSCC</a> were discussed in one of the previous HPE Developer Community blogs. The playbooks offer fact gatherings for storage systems, hosts and volumes. However, once you dig into the details, you will find that the modules have not been updated for more than two years, and do not support the HPE Alletra MP B10000 storage array. In this blog post, I will discuss a possible approach for DSCC data gathering using Ansible built-in functionality to overcome the lack of continuous playbook development.</p> <h1>Capture the storage system configuration</h1> <p>Upon learning that the playbooks for the DSCC were not well maintained, I looked for a different way to capture the configuration data of all arrays of the HPE Customer Technology Center in Böblingen. The <a href="https://github.com/HewlettPackard/hpe3par_ansible_module"></a> <a href="https://github.com/HewlettPackard/hpe3par_ansible_module?tab=readme-ov-file">HPE 3PAR Ansible module</a> requires one to connect to each array individually and does not provide a complete capture of the array configuration. Hence it is not a solution to the problem. A way forward would be to use the HPE Data Services Cloud Console and the corresponding Data Services REST API (the basics are already discussed in previous posts on the HPE Developer Community blog: <a href="https://developer.hpe.com/greenlake/data-services-on-the-hpe-greenlake-platform/home/">Data Services on the HPE GreenLake platform | HPE Developer Portal )</a>. The Data Services REST API offers a complete list of commands that can be issued on the DSCC.</p> <p>The configuration of a storage system generally includes the configuration data of the storage system itself, the details of the configured volumes of a storage array, the host group and the host details. The first step of gathering the configuration information would be to get a list of storage arrays connected to the Data Services Cloud Console. Once you have the list, you can go and gather details of each storage array. The <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">Data Services REST API</a> is supporting the data gathering by supplying with every array a list of associated links, that refer to controller, disk etc. information of this array. An example of REST API call response is given below:</p> <p><img src="/img/getstoragesystems.png" alt="" title="Get Storage System API call"></p> <p>In order to be independent of any Python library (or the lack of updates to a Python library), I have decided to use Ansible's built-in functionality to create the DSCC capture playbooks. The basic tasks that are used by the playbooks are the DSCC REST API call using the ansible.builtin.uri function and as a special call variant, the retrieval of the DSCC access token (which is special in terms of the URI used to get the access token).</p> <h1>Basic tasks</h1> <h2>Retrieving a DSCC access token</h2> <p>The steps to first generate the client ID and the client secret used to access the DSCC REST API was already described in a post on the HPE Developer Community blog: <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Using HPE GreenLake Console's API Gateway for Data Services Cloud Console</a>.</p> <p>Once you have your client ID and client secret, you can generate an access token that is valid for two hours. This access token will allow you to issue REST API calls to the Data Services Cloud Console, as it identifies you as the user that is linked with the client ID and secret to create the access token. Hence, it is best practice to store the client ID and secret in a secure place.</p> <p>The below code example had the client credentials stored in the credentials.yml file, that was encrypted using ansible-vault. The current Ansible playbook stores the access token in a file that grants access only to the current user (hence, the access mode 600 for this file) to avoid misuse of the retrieved access token.</p> <pre><code class="language-yaml">— name: Include encrypted vars include_vars: credentials.yml — name: Get Access Token ansible.builtin.uri: url: "{{ sso_url }}" headers: Content-Type: "application/x-www-form-urlencoded" Authorization: "Basic {{ (dscc_id + ':' + dscc_secret) | b64encode }}" method: POST body: "grant_type=client_credentials" validate_certs: false register: oauth_response — name: Define header ansible.builtin.set_fact: token: "Bearer {{ oauth_response.json.access_token }}" — name: Store Token ansible.builtin.copy: content: "{{ token }}" dest: 'vars/token.txt' mode: "0600" </code></pre> <h2>DSCC REST API call</h2> <p>A DSCC REST API call can be with and without a request body and can have multiple responses depending on the actual API call. Nevertheless, it is good practice to build a modular code approach that uses a generalized REST API call to access the Data Services Cloud Console. The generalized DSCC REST API call, shown in the following code block, has its parameters:</p> <ul> <li>requestUri (as mentioned in <a href="https://developer.hpe.com/greenlake/data-services-on-the-hpe-greenlake-platform/home/"></a><a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">Data Services REST API</a>)</li> <li>request method (Get, Post, Delete, Put)</li> <li>request body (optional)</li> </ul> <pre><code class="language-yaml">- name: Include encrypted vars include_vars: vars/credentials.yml - name: Get Access Token ansible.builtin.set_fact: token: "{{ lookup('file', 'vars/token.txt') }}" - name: Check the Methood ansible.builtin.fail: msg: "DSCC-API-CALL: RestAPI Method is not defined!" when: method is not defined - name: Check for the request Uri ansible.builtin.fail: msg: "DSCC-API-Call: Request URI is not defined!" when: request_uri is not defined - name: DSCC Command - {{request_uri}} ansible.builtin.uri: url: "{{ base_url }}{{ request_uri }}" headers: Authorization: "{{ token }}" Content-Type: "application/json" method: "{{ method }}" validate_certs: false status_code: [200, 201, 202, 401, 404] register: result when: body is not defined - name: Set result status ansible.builtin.set_fact: status: "{{ result.status }}" tmpres: "{{ result }}" when: body is not defined - name: DSCC Command with body {{request_uri}} ansible.builtin.uri: url: "{{ base_url }}{{ request_uri }}" headers: Authorization: "{{ token }}" Content-Type: "application/json" method: "{{ method }}" body_format: json body: "{{ body | to_json }}" validate_certs: false status_code: [200, 201, 202, 400, 401, 404] register: result2 when: body is defined - name: Set result status ansible.builtin.set_fact: status: "{{ result2.status }}" tmpres: "{{ result2 }}" when: body is defined - name: Set response when status in [200, 201, 202, 401] ansible.builtin.set_fact: response: "{{ tmpres }}" when: status in ['200', '201', '202','401'] - name: Undefine Response when status not in [200...] ansible.builtin.set_fact: response: "" when: status not in ['200', '201', '202','401'] </code></pre> <p>You can see, that it first retrieves the stored access token, then checks that the method and the request URI is available. Next it issues the API call either with or without a call body before the possible response status is checked and the call response is set.</p> <h1>System configuration capture</h1> <p>The complete workflow of the DSCC data capture is shown in the following flow diagram. First, the list of connected storage arrays is compiled and stored in a dictionary. Next, the playbook will loop through the storage array dictionary in order to capture the details of each connected storage array (this includes looping through all associated links of a storage array and the gathering of all volumes that are defined on the storage array). Afterwards, the host group and host details are also captured and stored.</p> <p><img src="/img/capturestorage-flowdiagram.png" alt="" title="Capture Storage System Flow Diagram"></p> <p>This system configuration capture flow chart can now be implemented using the above mentioned basic task in combination with the correct request URIs and request bodies. You can see in the example below that the playbook first gets the list of storage arrays (request uri: /api/v1/storage-systems). If the command returns a status code of 401 (i.e. unauthorized access), it repeats the same call after retrieving a refreshed access token (this is the difference between the DSCC-API-Call.yaml and the DSCC-API-401.yaml playbook). After successfully retrieving the system list, a system dictionary is populated first, followed by looping through the dictionary (Loop-Systems.yml playbook) and storing the system configuration information. Afterwards, the host group and hosts details are retrieved and stored.</p> <pre><code class="language-yaml"> hosts: localhost vars: method: "GET" tasks: - name: DSCC API Call GET storage systems vars: request_uri: "/api/v1/storage-systems" ansible.builtin.include_tasks: file: DSCC-API-Call.yaml - name: Retry the command if status 401 vars: request_uri: "/api/v1/storage-systems" ansible.builtin.include_tasks: file: DSCC-API-401.yaml when: status == '401' - name: Set Systems ansible.builtin.set_fact: systems: "{{ response.json['items'] }}" when: status in ['200', '201'] - name: Initialize Storage system dictionary if not defined ansible.builtin.set_fact: storage_systems: "{{ storage_systems | default({}) }}" - name: Create StorageSystems Dictionary ansible.builtin.set_fact: storage_systems: "{{ storage_systems | combine({item.name: {'id': item.id, 'resourceUri': item.resourceUri}}) }}" with_items: "{{ systems }}" - name: Loop Systems vars: ansible.builtin.include_tasks: file: Loop-Systems.yaml with_dict: "{{storage_systems}}" loop_control: loop_var: my_system - name: Get HostGroups vars: request_uri: "/api/v1/host-initiator-groups" ansible.builtin.include_tasks: file: DSCC-API-Call.yaml - name: Store the HostGroups ansible.builtin.copy: content: "{{ response.json | to_nice_json }}" dest: "../Outputs/hostGroups.json" mode: '0600' when: response.json is defined - name: Get Hosts ansible.builtin.include_tasks: file: GetAllHosts.yaml - name: Store the Hosts ansible.builtin.copy: content: "{{ response.json | to_nice_json }}" dest: "../Outputs/hosts.json" mode: '0600' when: response.json is defined </code></pre> <p>The Loop-Systems.yaml playbook retrieves the storage system details and loops for each system through all the associated links of this array, providing you with a complete capture of the storage array configuration. The captured data is stored in multiple files with the naming structure: <strong>SystemName.associatedLink-Keyname.json.</strong></p> <p>The Ansible playbooks used to capture the system configuration are:</p> <ul> <li> <p>Capture-Systems.yaml</p> <ul> <li> <p>DSCC-API-Call.yaml</p> </li> <li> <p>DSCC-API-401.yaml</p> </li> <li> <p>Loop-Systems.yaml</p> <ul> <li>Loop-Links.yaml</li> <li>GetAllSystemVolumes.yaml</li> </ul> </li> <li> <p>GetAllHosts.yaml</p> </li> </ul> </li> </ul> <p>In order to keep this blog readable and not code overloaded only a few of the playbooks used are shown, but all playbooks (and even some more) can be retrieved on Github at: <a href="https://github.com/tbeha/DSCC-Ansible">https://github.com/tbeha/DSCC-Ansible.</a></p> <h1>Conclusion</h1> <p>It is possible to use Ansible playbooks to capture the storage array configuration using the HPE Data Services Cloud Console REST API and built-in Ansible functions. Having the storage array captured in one or multiple JSON-files is leading to an obvious next step: use the captured configuration information to automate the redeployment of a storage array. This is one of my planned next activities. Stay tuned to the <a href="https://developer.hpe.com/">HPE Developer Community blog</a> for more.</p>https://developer.hpe.com/build-your-first-ai-chatbot-on-hpe-private-cloud-ai-using-flowise-and-hpe-mlis/https://developer.hpe.com/build-your-first-ai-chatbot-on-hpe-private-cloud-ai-using-flowise-and-hpe-mlis/Fri, 11 Jul 2025 13:38:06 GMT<p>In today’s AI-driven landscape, conversational interfaces are transforming how organizations interact with users and automate workflows. Building a secure, scalable, and customizable chatbot solution requires robust infrastructure and flexible AI tooling. HPE Private Cloud AI provides a powerful platform for deploying and managing AI workloads, while Flowise and HPE Machine Learning Inference Software offer the tools to rapidly build, deploy, and manage chatbots powered by large language models (LLMs).</p> <p>This blog post walks you through deploying FlowiseAI on HPE PCAI to build a modern chatbot solution. By leveraging these technologies, organizations can accelerate chatbot development, ensure data privacy, and maintain full control over their AI lifecycle.</p> <h2>HPE Private Cloud AI</h2> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (HPE PCAI)</a> offers a comprehensive, turnkey AI solution designed to address key enterprise challenges, from selecting the appropriate LLMs to efficiently hosting and deploying them. Beyond these core functions, HPE Private Cloud AI empowers organizations to take full control of their AI adoption journey by offering a curated set of pre-integrated <em>NVIDIA Inference Microservices (NIM)</em> LLMs, along with a powerful suite of AI tools and frameworks for data engineering, analytics, and data science.</p> <p>HPE Machine Learning Inference Software is a user-friendly solution designed to simplify and control the deployment, management, and monitoring of machine learning (ML) models, including LLMs, at any scale.</p> <p>HPE Private Cloud AI has pre-integrated NVIDIA NIM LLMs, a suite of AI tools (including HPE Machine Learning Inference Software), and a flexible <em>Import Framework</em> that enables organizations to deploy their own applications or third-party solutions, like FlowiseAI.</p> <p><img src="/img/importframework.jpg" alt=""></p> <h2>What is Flowise?</h2> <p><a href="https://flowiseai.com/">Flowise</a> is an open source generative AI development platform for building AI Agents and LLM workflows. It provides a visual interface for designing conversational flows, integrating data sources, and connecting to various LLM endpoints. Flowise provides modular building blocks for you to build any agentic systems, from simple compositional workflows to autonomous agents.</p> <h2>Deploying Flowise via import framework</h2> <h3>1. Prepare the Helm charts</h3> <p>Obtain the Helm chart for Flowise v5.1.1 from <a href="https://artifacthub.io/packages/helm/cowboysysop/flowise">artifacthub.io</a>. Following changes to the Helm chart are needed to deploy it on HPE Private Cloud AI.</p> <p>Add the following YAML manifest files to <em>templates/ezua/</em> directory:</p> <ul> <li><em>virtualService.yaml</em>: Defines an Istio <em>VirtualService</em> to configure routing rules for incoming requests.</li> <li><em>kyverno.yaml</em>: A Kyverno <em>ClusterPolicy</em> that automatically adds required labels to the deployment.</li> </ul> <p>Updates to <em>values.yaml</em> file</p> <ul> <li>Set resource request/limits.</li> <li>Update the PVC size</li> <li>Add the following <em>'ezua'</em> section to configure the <em>Istio Gateway</em> and expose the endpoint.</li> </ul> <pre><code>ezua: virtualService: endpoint: "flowise.${DOMAIN_NAME}" istioGateway: "istio-system/ezaf-gateway" </code></pre> <p>Here's the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie18hen_us&#x26;page=ManageClusters/importing-applications.html">reference document</a> for the import framework prerequisites.</p> <p>These updates are implemented in the revised Flowise Helm charts, and are available in the GitHub repository <a href="https://github.com/ai-solution-eng/frameworks/tree/main/flowise">ai-solution-eng/frameworks. </a>With these customizations, <em>Flowise</em> can now be deployed on HPE Private Cloud AI using <em>Import Framework.</em></p> <h3>2. Deploy Flowise via the import framework</h3> <p>Use the import framework in HPE Private Cloud AI to deploy Flowise.</p> <p><img src="/img/flowise-deploy-1.jpg" alt=""></p> <p><img src="/img/flowise-deploy-2.jpg" alt=""></p> <p><img src="/img/flowise-deploy-3.jpg" alt=""></p> <p><img src="/img/flowise-deploy-4.jpg" alt=""></p> <h3>3. Access Flowise UI via its endpoint</h3> <p>After deployment, Flowise will appear as a tile under <em>Tools &#x26; Frameworks / Data Engineering</em> tab.</p> <p><img src="/img/flowsie-deployed.jpg" alt=""></p> <p>Click the <em>Open</em> button on the <em>Flowise</em> Tile, or click on the <em>Endpoint</em> URL to launch the Flowise login page. Setup the credentials and login.</p> <p><img src="/img/flowise-home-7-11-2025.jpg" alt=""></p> <hr> <h2>Deploy a LLM in HPE MLIS</h2> <p>HPE MLIS is accessed by clicking on <em>HPE MLIS</em> tile in <em>Tools &#x26; Frameworks / Data Engineering</em> tab.</p> <p><img src="/img/mlis.jpg" alt=""></p> <p>To deploy a pre-packaged LLM (Meta/Llama3-8b-instruct) in HPE MLIS, you need to know how to add a registry, a packaged model, and how to create deployments.</p> <h3>1. Adding a registry</h3> <p>You'll first want to add a new registry called "NGC", which refers to NVIDIA GPU Cloud. This can be used to access pre-packaged LLMs.</p> <p><img src="/img/mlis-registry.jpg" alt=""></p> <h3>2. Adding a packaged model</h3> <p>Create a new packaged model by clicking the <em>Add New Model</em> tab. Fill in the details as shown in the below screen shots.</p> <p><img src="/img/package-model-1.jpg" alt=""></p> <p>Choose the registry created in the previous step and select 'meta/llama-3.1-8b-instruct' for the <em>NGC Supported Models</em></p> <p><img src="/img/package-model-2.jpg" alt=""></p> <p>Set the right resources required for the model. Do this by choosing from either a built-in template or "custom" in the <em>Resource Template</em> section.</p> <p><img src="/img/package-model-3.jpg" alt=""></p> <p><img src="/img/package-model-4.jpg" alt=""></p> <p>Newly created packaged model appears in the UI.</p> <p><img src="/img/package-model-final.jpg" alt=""></p> <h3>3. Creating deployments</h3> <p>Using the packaged model created in the previous step, create a new deployment by clicking on <em>Create new deployment.</em></p> <p><img src="/img/deployment-1.jpg" alt=""></p> <p>Give a name to the deployment and choose the packaged model created in the previous step.</p> <p><img src="/img/deployment-2.jpg" alt=""></p> <p><img src="/img/deployment-3.jpg" alt=""></p> <p>Set auto scaling as required. In this example, we have used 'fixed-1' template.</p> <p><img src="/img/deployment-4.jpg" alt=""></p> <p><img src="/img/deployment-5.jpg" alt=""></p> <p>The LLM is now deployed and can be accessed using the endpoint and corresponding API token.</p> <p><img src="/img/deployment-6.jpg" alt=""></p> <h2>Create AI Chatbot in Flowise</h2> <p>Use Flowise's drag-and-drop interface to design your chatbot’s conversational flow. Integrate with HPE MLIS by adding an LLM node and configuring it to use the MLIS inference endpoint.</p> <ul> <li><strong>Add New Chatflow:</strong></li> </ul> <p><img src="/img/chatflow-1.jpg" alt=""></p> <p>Save the Chatflow using the name "AI Chatbot" and add the following nodes, making the connections shown in the screenshot.</p> <ul> <li><strong>Chat Models (Chat NVIDIA NIM):</strong> Set Deployment 'Endpoint' from HPE MLIS as 'Base Path', corresponding 'Model Name' and 'API Key' from HPE MLIS for 'Connect Credential'.</li> <li><strong>Memory (Buffer Window Memory):</strong> Set appropriate 'Size'.</li> <li><strong>Chains (Conversation Chain):</strong> Connect 'Chat NVIDIA NIM' and 'Buffer Window Memory' nodes as shown.</li> </ul> <p><img src="/img/chatflow-2.jpg" alt=""></p> <p>Your new AI Chatbot is now ready! You may quickly test it by clicking the chat icon on the top right corner of the screen.</p> <p><img src="/img/chatflow-3.jpg" alt=""></p> <h3>Accessing AI Chatbot from external applications</h3> <p>Flowise provides an API endpoint for the chatbot, with multiple ways of integrating it with your applications. Also, you may explore multiple configurations that are available to enhance the chatbot.</p> <p><img src="/img/chatflow-4.jpg" alt=""></p> <h2>Conclusion</h2> <p>By combining Flowise’s intuitive chatbot builder with HPE MLIS’s robust model management, HPE Private Cloud AI empowers organizations to rapidly develop, deploy, and govern conversational AI solutions. This integrated approach ensures data privacy, operational control, and scalability for enterprise chatbot deployments.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more guides and best practices on leveraging HPE Private Cloud AI for your AI.</p>https://developer.hpe.com/from-generative-to-agentic-ai-—-part-2-what-makes-ai-agents-truly-intelligent/https://developer.hpe.com/from-generative-to-agentic-ai-—-part-2-what-makes-ai-agents-truly-intelligent/Wed, 09 Jul 2025 04:15:26 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In the <a href="https://developer.hpe.com/blog/from-generative-to-agentic-ai-tracing-the-leap-from-words-to-actions/">first part of this series</a>, I discussed the shift from passive large language models to more capable, action-oriented AI. Now, I will provide a closer look at what actually powers this transformation — the concept of the AI agent. Far from being just an advanced chatbot, an agent is a structured system that can understand, plan, execute, and respond — much like a real-world assistant, only faster and smarter.</p> <p>Inspired by <a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-part-2-agentic-ai-74dcf045aff0">my post on Medium</a>, this post builds upon the original work with added clarity, practical examples, and a more conversational tone to help you truly grasp how agentic AI is reshaping automation across industries.</p> <center><img src="/img/screenshot-2025-07-18-at-4.10.42 pm.png" width="500" height="542" alt="AI agnet Framework" title="AI agnet Framework"></center> <h2>What are AI agents?</h2> <p>An AI agent is not just something that responds to prompts — it’s something that takes initiative. Unlike traditional LLMs, which generate output only when asked, agents can independently decide what actions to take, how to take them, and when to stop.</p> <p>Here’s what makes an agent different:</p> <ul> <li>It can decide how to solve a problem based on context.</li> <li>It can use tools such as APIs, search engines, or databases.</li> <li>It can take real actions, like analyzing data, sending emails, or making reservations.</li> <li>It can break down large tasks into smaller, manageable steps and complete them autonomously.</li> </ul> <p>In essence, an AI agent behaves more like a virtual assistant capable of doing actual work — not just holding a conversation.</p> <h2>The agent workflow: Think → Plan → Act → Respond</h2> <p>The heart of an agentic system lies in this continuous loop:</p> <ul> <li><strong>Think :</strong> It starts with understanding the objective or problem at hand.</li> <li><strong>Plan :</strong> Based on that understanding, it creates a strategy — often a sequence of steps.</li> <li><strong>Act :</strong> It then begins executing the plan, calling tools, retrieving data, or initiating actions.</li> <li><strong>Respond :</strong> Finally, it summarizes or communicates the results — or loops back to continue solving.</li> </ul> <p>This cycle allows agents to operate with minimal human intervention, even on complex, multi-step workflows.</p> <h2>Real-world impact: How agents are already changing industries</h2> <p><strong>Healthcare:</strong> AI agents can retrieve patient history, summarize medical notes, and monitor for critical conditions, aiding doctors in real-time.</p> <p><strong>Finance:</strong> Agents can analyze markets, detect fraud, and automate reporting — operating at speeds no human team could match.</p> <p><strong>Customer support</strong>: Instead of generic replies, agents can pull data from CRM systems, open service tickets, or resolve technical issues directly.</p> <p><strong>IT and DevOps:</strong> Agents are now monitoring systems, fixing bugs, deploying updates — all without waiting on a human operator.</p> <p>These are not theoretical applications — they are happening right now, streamlining operations and reducing bottlenecks across the board.</p> <h2>Agents + Tools = Real-world superpowers</h2> <p>What truly empowers agents is their ability to interface with tools. Think of APIs, web services, internal databases, scripts, and even IoT systems. These integrations allow agents to interact with the real world, not just the digital one.</p> <p>For example:</p> <ul> <li>Google Maps or Search APIs.</li> <li>CRM and ERP databases.</li> <li>Automation scripts for cloud platforms or internal workflows.</li> </ul> <p>When agents are connected to tools, they don’t just think — they execute.</p> <h2>Need for AI agents</h2> <p>The value of AI agents lies in their ability to scale thinking into action:</p> <ul> <li>They automate entire workflows — not just single responses.</li> <li>They handle decision-making on the fly.</li> <li>They adapt to changing inputs and data.</li> <li>They reduce repetitive manual work across industries.</li> <li>They can collaborate as multi-agent teams to solve broader, interconnected problems.</li> </ul> <h2>Top 5 frameworks to build AI agents</h2> <p>If you're ready to build with agents, here are the top frameworks that developers swear by:</p> <ul> <li>Auto-GPT</li> <li>BabyAGI</li> <li>AGNO</li> <li>LangChain</li> <li>Autogen by Microsoft</li> </ul> <table> <thead> <tr> <th>Framework</th> <th>What it does</th> <th>GitHub Link</th> <th>⭐ Stars</th> </tr> </thead> <tbody> <tr> <td><strong>AGNO (Phidata)</strong></td> <td>Build fast, multi-modal agents (text + images)</td> <td><a href="https://github.com/phidatahq/phidata">AGNO GitHub</a></td> <td>⭐ 21.5k</td> </tr> <tr> <td><strong>Auto-GPT</strong></td> <td>The first viral agent that automates tasks</td> <td><a href="https://github.com/Torantulino/Auto-GPT">Auto-GPT GitHub</a></td> <td>⭐ 174k</td> </tr> <tr> <td><strong>BabyAGI</strong></td> <td>Agents that manage task lists autonomously</td> <td><a href="https://github.com/yoheinakajima/babyagi">BabyAGI GitHub</a></td> <td>⭐ 21.2k</td> </tr> <tr> <td><strong>LangChain Agents</strong></td> <td>Powerful and flexible agent toolkit for developers</td> <td><a href="https://github.com/langchain-ai/langchain">LangChain GitHub</a></td> <td>⭐ 104k</td> </tr> <tr> <td><strong>Microsoft Autogen</strong></td> <td>Build multi-agent systems that work together</td> <td><a href="https://github.com/microsoft/autogen">Autogen GitHub</a></td> <td>⭐ 42k</td> </tr> </tbody> </table> <p>Each offers a different approach — some focus on chaining tasks, others on autonomy and memory. Together, they make it easier than ever to bring agentic AI to life.</p> <h2><strong>Conclusion</strong></h2> <p>AI agents are no longer a futuristic idea — they’re here, and they’re transforming how work gets done. By combining decision-making, planning, and tool usage, agents represent the leap from intelligent text generation to intelligent action. They’re bridging the gap between knowing what needs to be done and actually doing it.</p> <p><a href="https://developer.hpe.com/blog/model-context-protocol-mcp-the-protocol-that-powers-ai-agents/">In Part 3 of this series,</a> I'll dig deeper into the architecture behind agentic systems — what components make them tick, how memory and feedback loops work, and how they can scale. If you're building the future or just trying to understand it, you're in the right place.</p> <p>Until then, keep watching the space where AI stops being a helper… and becomes a doer</p>https://developer.hpe.com/2025-july-07/https://developer.hpe.com/2025-july-07/Mon, 07 Jul 2025 13:00:00 GMThttps://developer.hpe.com/from-generative-to-agentic-ai-tracing-the-leap-from-words-to-actions/https://developer.hpe.com/from-generative-to-agentic-ai-tracing-the-leap-from-words-to-actions/Thu, 03 Jul 2025 10:33:21 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>AI has come a long way from simply finishing our sentences. Today, it’s not just generating content — it’s actively solving problems, making decisions, and executing complex tasks. This blog post kicks off a 10-part series where I'II trace that incredible journey — from basic generative models to fully autonomous agents. Along the way, I’ll unpack the key shifts, architectures, and mindsets that shaped this evolution.</p> <p>Inspired by a <a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-65de72254a86">my post on </a><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-65de72254a86">Medium</a><a href="https://dineshr1493.medium.com/all-you-need-to-know-about-the-evolution-of-generative-ai-to-agentic-ai-65de72254a86">,</a> this piece reimagines and expands on the original with a human-first lens and practical clarity.</p> <p>Whether you're an AI developer, tech leader, or just curious about where all this is headed — welcome. Let’s dive in.</p> <h2>Phase 1: LLMs — The linguistic powerhouse</h2> <p>Large Language Models (LLMs) like GPT, DeepSeek, QWEN, and LLaMA burst onto the scene with one incredible skill — understanding and generating human language. These models are trained on massive datasets and excel at:</p> <ul> <li>Multilingual conversations</li> <li>Summarization, classification, and text generation</li> <li>Contextual prediction based on vast patterns</li> </ul> <p>But here’s the catch:</p> <p>LLMs are great at “saying” things… but they don’t do anything.</p> <p>On their own, LLMs are like brilliant thinkers without hands — capable of deep analysis, but unable to act in the real world.</p> <center><img src="/img/llms.png" width="600" height="550" alt="LLM Evolution" title="LLM Evolution"></center> <h2>Phase 2: LLMs + Tools — Giving the brain some hands</h2> <p>The next leap came when developers began connecting LLMs with external tools — APIs, plugins, databases, and custom workflows. This simple but powerful integration gave models the ability to:</p> <ul> <li>Search the web (like Perplexity AI)</li> <li>Execute code and commands</li> <li>Fetch real-time or contextual information</li> </ul> <p>This expanded what AI could do. Suddenly, the models weren’t just conversational — they became useful assistants.</p> <p><strong>But there was still a problem:</strong></p> <p>Tool-based systems are fragile. APIs break, schemas change, and workflows can become unreliable.</p> <p>Think of it like giving a brain a set of hands — but the hands don’t always listen, or worse, they change shape every other week.</p> <h2>Phase 3: LLMs + Agents — The rise of agentic AI</h2> <p>This is where things get truly exciting.</p> <p>Agentic AI introduces a new layer of intelligence: autonomy. Instead of the model responding directly to every input, agentic systems:</p> <ul> <li>Set goals</li> <li>Break them into tasks</li> <li>Select and operate tools</li> <li>Make iterative decisions</li> <li>Learn from outcomes</li> </ul> <p>In essence, AI stops being reactive and starts becoming proactive. These agents operate like digital coordinators — orchestrating actions, delegating responsibilities, and adjusting course as needed. They move beyond simple tasks and begin solving complex workflows.</p> <p>This isn’t just a better assistant — it’s the early form of AI co-workers.</p> <p><strong>TL;DR Breakdown</strong></p> <ul> <li>LLMs = Great with words, but passive</li> <li>LLMs + Tools = Adds capabilities, but brittle and manual</li> <li>LLMs + Agents = Autonomous systems that think, plan, and act</li> </ul> <p><strong>We’ve moved from “talking AI” to “doing AI.”</strong></p> <h2>Conclusion</h2> <p>The shift from generative to agentic AI is more than just a technical upgrade — it’s a philosophical turning point in how we think about artificial intelligence. We’re no longer training machines to just converse with us; we’re teaching them to collaborate, adapt, and even take initiative. Agentic AI is the foundation for everything from self-operating software agents to autonomous business logic.</p> <p>In the next part of this series, I’ll peel back the curtain on how agentic architectures actually work — the brains behind the autonomy. Until then, consider this: the next time you interact with an AI, it may not just be listening… it may already be planning your next move.</p>https://developer.hpe.com/setting-up-harbor-as-a-local-container-registry-in-hpe-private-cloud-ai/https://developer.hpe.com/setting-up-harbor-as-a-local-container-registry-in-hpe-private-cloud-ai/Thu, 03 Jul 2025 07:21:44 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>A container registry serves as a centralized system for storing and managing container images. In today’s fast-paced containerized application development landscape, speed, security and control over container workflows using a robust container registry are critical. While both cloud-based container registries, such as Google Container Registry (<em>GCR</em>), Azure Container Registry (<em>ACR</em>), and Amazon Elastic Container Registry (<em>ECR</em>), and third-party services like <em>DockerHub</em>, <em>GitHub</em> / <em>GitLab</em> Container Registry, and <em>JFrog</em> Container Registry, offer convenience, organizations often face challenges with latency, external dependencies, and security compliance constraints.</p> <p>This blog post describes the process of deploying <em>Harbor</em> and setting it up as a local container registry within <em>HPE Private Cloud AI</em>. By using <em>Harbor</em> as a local container registry, organizations gain faster image access, reduced dependence on external networks, improved security, and a tailored registry environment that aligns with internal compliance and governance needs.</p> <h2>HPE Private Cloud AI</h2> <p><a href="https://developer.hpe.com/platform/hpe-private-cloud-ai/home/">HPE Private Cloud AI (HPE PCAI)</a> offers a comprehensive, turnkey AI solution designed to address key enterprise challenges, from selecting the appropriate large language models (LLMs) to efficiently hosting and deploying them. Beyond these core functions, HPE PCAI empowers organizations to take full control of their AI adoption journey by offering a curated set of pre-integrated <em>NVIDIA NIM</em> LLMs, along with a powerful suite of AI tools and frameworks for <em>Data Engineering</em>, <em>Analytics</em>, and <em>Data Science</em>.</p> <p>The <em>Import Framework</em> in HPE PCAI further enhances flexibility by enabling organizations to integrate their own applications or third-party solutions alongside pre-installed components, accommodating a wide range of enterprise-specific use cases.</p> <p><img src="/img/pcai-import-framework.png" alt=""></p> <p>This blog post guides you through the step-by-step process of deploying the open-source <em>Harbor</em> into HPE PCAI using the <em>Import Framework</em>. Once deployed and configured, <em>Harbor</em> can serve as a local container registry within HPE PCAI. With key features such as policy management, role-based access control (RBAC), security scanning, and image signing, <em>Harbor</em> strengthens container lifecycle security and governance.</p> <h2>Prerequisites</h2> <p>Before starting, make sure that <a href="https://docs.docker.com/engine/install/">Docker Engine</a>, version <em>28.1.1</em> or later, is installed, including the default <em>docker</em> CLI, which will be used for building and pushing images.</p> <p>The following sections show application deployment details using the <em>kubectl</em> CLI and <em>kubeconfig</em> to access the HPE PCAI Kubernetes (K8s) cluster. However, direct cluster access via <em>kubectl</em> is generally not required.</p> <h2>Harbor</h2> <p><em>Harbor</em> is an open-source container registry designed for cloud-native environments like K8s. It securely stores and manages container images with policies and RBAC, ensures images are scanned and free from vulnerabilities, and signs images as trusted.</p> <p>The following sections describe in detail how to deploy <em>Harbor</em> into HPE PCAI using the <em>Import Framework</em>. You will learn how to create a private project, create users and assign them with specific role permissions, and push images using <em>Harbor</em> credentials. Used as a local image registry within HPE PCAI, <em>Harbor</em> helps ensure your container images remain secure and well governed.</p> <h3>Harbor deployment via HPE PCAI <em>Import Framework</em></h3> <p>Based on the latest Helm charts from the official <a href="https://helm.goharbor.io/harbor-1.17.0.tgz"><em>Harbor</em> site</a>, the following YAML manifest files have been added under <em>templates/ezua/</em> directory:</p> <ul> <li><em>virtualService.yaml</em>: Defines an Istio <em>VirtualService</em> to configure routing rules for incoming requests.</li> <li><em>kyverno-cluster-policy</em>: A Kyverno <em>ClusterPolicy</em> that automatically adds required labels to the deployment.</li> </ul> <p>Additionally, the default <em>values.yaml</em> file has been modified with the following updates:</p> <ul> <li>introduced an <em>ezua</em> section to configure the <em>Istio Gateway</em> and expose the endpoint:</li> </ul> <pre><code class="language-bash"> ezua: virtualService: endpoint: "harbor.${DOMAIN_NAME}" istioGateway: "istio-system/ezaf-gateway" </code></pre> <ul> <li>expanded <em>Harbor</em> registry storage from the default <em>5G</em> to <em>500G</em>:</li> </ul> <pre><code class="language-bash"> persistence.persistentVolumeClaim.registry.size = 500G </code></pre> <p>These updates are implemented in the revised <em>Harbor</em> Helm charts, available in the <em>GitHub</em> repository <a href="https://github.com/GuopingJia/pcai-helm-examples/tree/main/harbor"><em>pcai-helm-examples</em></a>. With these customizations, <em>Harbor</em> can be easily deployed into HPE PCAI using the <em>Import Framework</em>:</p> <p><img src="/img/import-harbor.png" alt=""></p> <h3>Harbor UI access via its endpoint</h3> <p>After <em>Harbor</em> is deployed via the HPE PCAI <em>Import Framework</em>, an <strong>Imported</strong> <em>Harbor</em> tile appears under <em>Tools &#x26; Frameworks</em> on the <em>Data Science</em> tab. A service endpoint, e.g., <em><a href="https://harbor.ingress.pcai0104.ld7.hpecolo.net">https://harbor.ingress.pcai0104.ld7.hpecolo.net</a></em>, is automatically configured and exposed, providing access to <em>Harbor</em>.</p> <p><img src="/img/harbor-deployment.png" alt=""></p> <p>Click the <em>Open</em> button, or paste the endpoint URL into your browser, to launch the <em>Harbor</em> login page:</p> <p><img src="/img/harbor-login.png" alt=""></p> <p>From there, you can log into <em>Harbor</em> projects page using the default <em>admin</em> user credentials:</p> <p><img src="/img/harbor-ui.png" alt=""></p> <h3>Harbor project and user creation</h3> <p><em>Harbor</em> manages container images through projects, each of which hosts the image repositories for your application. Before pushing images to <em>Harbor</em>, a project must first be created. A default public project named <em>library</em> is pre-created, but new projects can be created by clicking <em>+ NEW PROJECT</em>:</p> <p><img src="/img/create-project.png" alt=""></p> <p>To enhance security, it's recommended to create <em>private</em> projects in <em>Harbor</em> to prevent unauthorized images pulls. In this blog post, a private project named <em>demo</em> is created with an unlimited quota (<strong>-1</strong>). For production environments, applying a defined quota, e.g., <em>200G</em>, can help manage registry storage capacity more effectively.</p> <p>After creating the project, users can be created and assigned to projects using role-based access control (RBAC). In this blog post, two sample users, <em>pcai-developer</em>, &#x26; <em>pcai-admin</em>, are created:</p> <p><img src="/img/two-users-harbor.png" alt=""></p> <p>These users, along with the default <em>admin</em> user, are added to the project <em>demo</em> with distinct roles:</p> <ul> <li><em>pcai-developer</em> has the <strong>Developer</strong> role (with read/write access to project)</li> <li><em>pcai-admin</em> is assigned the <strong>Maintainer</strong> role, with extended privileges including image scanning, replication job visibility and image deletion</li> </ul> <p><img src="/img/project-member.png" alt=""></p> <p>For a detailed breakdown of each role's capabilities, refer to the official <a href="https://goharbor.io/docs/2.13.0/administration/managing-users/">Harbor Managing Users page</a>. As a best practice, production deployments should enforce role separation to maintain security and operational clarity in <em>Harbor</em>.</p> <h3>Pushing images to Harbor registry</h3> <p>With the project and users set up, you're ready to push the container images to <em>Harbor</em> by following these steps:</p> <ul> <li><em>Log in to Harbor registry</em></li> </ul> <p>Use the Docker client to authenticate with the <em>Harbor</em> registry using the <em>pcai-admin</em> user credentials by running the following command :</p> <pre><code class="language-shell">$ docker login harbor.ingress.pcai0104.ld7.hpecolo.net Username: pcai-admin Password: WARNING! Your credentials are stored unencrypted in '/home/guoping/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded </code></pre> <p>If you get any certificate error when logging in from a Linux client, update the file <em>/etc/docker/daemon.json</em> by adding the following entry, replacing the <em>Harbor</em> registry URL with your own:</p> <pre><code class="language-shell">{ "insecure-registries" : [" harbor.ingress.pcai0104.ld7.hpecolo.net "] } </code></pre> <p>After making this change, reload the daemon and restart the Docker service:</p> <pre><code class="language-shell">$ sudo systemctl daemon-reload $ sudo systemctl restart docker </code></pre> <ul> <li><em>Tag an existing image</em></li> </ul> <p>Rather than building a Docker image from a Dockerfile, pull the sample CFE Nginx image, <em>'pcaidemo/cfe-nginx'</em>, from <em>DockerHub</em> and tag it with the <em>Harbor</em> registry URL and project name:</p> <pre><code class="language-shell">$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE pcaidemo/cfe-nginx v0.1.0 1e5f3c5b981a 2 months ago 192MB $ docker tag pcaidemo/cfe-nginx:v0.1.0 harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE pcaidemo/cfe-nginx v0.1.0 1e5f3c5b981a 2 months ago 192MB harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx v0.1.0 1e5f3c5b981a 2 months ago 192MB </code></pre> <ul> <li><em>Push the image to Harbor registry</em></li> </ul> <p>Push the image to the <em>Harbor</em> registry by running the following command:</p> <pre><code class="language-shell">$ docker push harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0 The push refers to repository [harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx] 7e893c1b6ee8: Pushed 463308bed0c9: Pushed 4197a611afec: Pushed 3e96162769d5: Pushed 892e805f6f4f: Pushed 626ab8a5d57b: Pushed 7fb72a7d1a8e: Pushed v0.1.0: digest: sha256:114dff0fc8ee3d0200c3a12c60e3e2b79d0920dd953175ecb78a0b157425b25e size: 1778 </code></pre> <ul> <li><em>Verify the image from Harbor registry</em></li> </ul> <p>From the <em>Harbor</em> UI, the image <em>cfe-nginx</em> appears under <em>Repositories</em> tab of the <em>demo</em> project:</p> <p><img src="/img/demo-project.png" alt=""></p> <p>From the Docker client, log in to the <em>Harbor</em> registry as the <em>pcai-developer</em> user, then pull the image <em>cfe-nginx</em> from the registry. The image should download successfully, confirming that the user has appropriate access and the <em>Harbor</em> registry is functioning as expected.</p> <pre><code class="language-shell">$ docker login harbor.ingress.pcai0104.ld7.hpecolo.net Username: pcai-developer Password: WARNING! Your credentials are stored unencrypted in '/home/guoping/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE $ docker pull harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0 v0.1.0: Pulling from demo/cfe-nginx b895f377d09e: Already exists 3b00567da964: Pull complete 56b81cfa547d: Pull complete 1bc5dc8b475d: Pull complete 979e6233a40a: Pull complete d2a7ba8dbfee: Pull complete 32e44235e1d5: Pull complete Digest: sha256:114dff0fc8ee3d0200c3a12c60e3e2b79d0920dd953175ecb78a0b157425b25e Status: Downloaded newer image for harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0 harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx v0.1.0 1e5f3c5b981a 2 months ago 192MB </code></pre> <h2>Application deployment using Harbor registry</h2> <p>With the container images pushed to the <em>Harbor</em> registry, the next step is to deploy the application to HPE PCAI using the same <em>Import Framework</em>. It will demonstrate how to pull images from <em>Harbor</em>.</p> <p>The Helm charts of the sample <em>CFE Nginx</em> application are available from <em>GitHub</em> repository <a href="https://github.com/GuopingJia/pcai-helm-examples/tree/main/nginx-chart">pcai-helm-examples</a>. Alongside the required <em>virtualService</em> and Kyverno <em>ClusterPolicy</em> YAML files, the <em>values.yaml</em> file includes the <em>imageCredentials</em> section that specifies the <em>Harbor</em> access credentials for the <em>pcai-developer</em> user. It also references the <em>imagePullSecrets</em> field that uses the Secret resource <em>harbor</em>, which is created during deployment, to securely pull container images from the <em>Harbor</em> registry.</p> <pre><code class="language-shell">image: repository: harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. tag: "v0.1.0" imagePullSecrets: - name: harbor imageCredentials: registry: harbor.ingress.pcai0104.ld7.hpecolo.net username: pcai-developer password: PCAIDev12345 email: glcs.cfe@hpe.com </code></pre> <p>Using the provided sample Helm charts, the <em>CFE Nginx</em> application can be easily deployed to HPE PCAI via the <em>Import Framework</em>. After deployment, an <strong>Imported</strong> <em>Nginx</em> tile appears under <em>Tools &#x26; Framework</em>, along with its configured service endpoint:</p> <p><img src="/img/nginx-deployment.png" alt=""></p> <p>Clicking the <em>Open</em> button launches the <em>CFE Nginx</em> main page:</p> <p><img src="/img/nginx-ui.png" alt=""></p> <p>The <em>CFE Nginx</em> application is deployed to the namespace <em>'nginx'</em> in the K8s cluster. If you have access to the cluster, you can verify the deployment by running the following command:</p> <pre><code class="language-shell"># kubectl get all -n nginx NAME READY STATUS RESTARTS AGE pod/nginx-chart-546476cd99-2nqzz 1/1 Running 0 6s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-chart ClusterIP 10.99.78.114 &#x3C;none> 80/TCP 6s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-chart 1/1 1 1 6s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-chart-546476cd99 1 1 1 6s </code></pre> <p>Within the <em>'nginx'</em> namespace , a Secret named <em>'harbor'</em>, of type <em>dockerconfigjson</em>, is created. This secret is used to authenticate and pull images from the <em>Harbor</em> registry during the deployment of the <em>CFE Nginx</em> application:</p> <pre><code class="language-shell"># kubectl get secret harbor -n nginx NAME TYPE DATA AGE harbor kubernetes.io/dockerconfigjson 1 3m41s </code></pre> <p>Type the following command to observe the <em>cfe-nginx</em> image, tagged <em>v0.1.0</em>, being pulled from the <em>demo</em> private project in <em>Harbor</em> registry:</p> <pre><code class="language-shell">[root@ez-master01 ~]# k describe pod/nginx-chart-546476cd99-2nqzz -n nginx Name: nginx-chart-546476cd99-2nqzz Namespace: nginx ... ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m18s scheduler-plugins-scheduler Successfully assigned nginx/nginx-chart-546476cd99-2nqzz to scs04.pcai0104.ld7.hpecolo.net Normal Pulling 2m18s kubelet Pulling image "busybox" Normal Pulled 2m17s kubelet Successfully pulled image "busybox" in 860ms (860ms including waiting) Normal Created 2m17s kubelet Created container web-content Normal Started 2m17s kubelet Started container web-content Normal Pulling 2m16s kubelet Pulling image "harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0" Normal Pulled 2m16s kubelet Successfully pulled image "harbor.ingress.pcai0104.ld7.hpecolo.net/demo/cfe-nginx:v0.1.0" in 377ms (377ms including waiting) Normal Created 2m16s kubelet Created container nginx-chart Normal Started 2m16s kubelet Started container nginx-chart </code></pre> <p>The <em>Logs</em> page of the <em>Harbor</em> UI provides a comprehensive audit trail, capturing key activities such as project and user creation, as well as image push and pull operations:</p> <p><img src="/img/harbor-audit.png" alt=""></p> <h2>Conclusion</h2> <p>In this blog post, I explored how to deploy <em>Harbor</em> to HPE Private Cloud AI and configure it as a local container registry. By setting up a private <em>Harbor</em> project and assigning user roles, organizations can securely manage, push and pull container images tailored to their application needs.</p> <p>More than just a container registry, <em>Harbor</em> strengthens security with built-in vulnerability scanning, image signing, and content trust features, ensuring only verified, compliant images are used across deployments. With <em>Harbor</em> integrated into HPE PCAI, organizations can confidently host container images internally, eliminating the need for external registries. The local container registry offers greater control over image provenance and aligns more effectively with organization security policies and regulatory requirements.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE Private Cloud AI and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-3-new-languages-vs-language-extensions/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-3-new-languages-vs-language-extensions/Thu, 26 Jun 2025 22:20:33 GMT<p>external blog</p>https://developer.hpe.com/new-features-in-hpe-morpheus-vm-essentials-software-8-0-5/https://developer.hpe.com/new-features-in-hpe-morpheus-vm-essentials-software-8-0-5/Tue, 24 Jun 2025 15:33:52 GMT<p>external blog post</p>https://developer.hpe.com/integrating-hpe-greenlake-webhooks-with-splunk/https://developer.hpe.com/integrating-hpe-greenlake-webhooks-with-splunk/Tue, 24 Jun 2025 11:34:04 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>Overview</h2> <p>This guide shows you how to connect HPE GreenLake webhooks with <a href="https://www.splunk.com/">Splunk</a>. Splunk is a data platform that collects, indexes, and analyzes machine-generated data to provide insights for various purposes, including security monitoring, IT operations, and business analytics. When the two are connected, you will be able to see your HPE GreenLake events through Splunk for improved data monitoring and analysis.</p> <h2>What you’ll learn</h2> <ul> <li>How to set up Splunk to receive data from HPE GreenLake</li> <li>How to handle HPE GreenLake's security requirements</li> <li>How to build a helper app that makes everything work together</li> <li>How to test and monitor your setup</li> </ul> <h2>Overview of Splunk HTTP Event Collector (HEC)</h2> <p>The <a href="https://dev.splunk.com/enterprise/docs/devtools/httpeventcollector/">HTTP Event Collector (HEC)</a> is a Splunk feature that lets you send data and application events to a Splunk deployment over the HTTP and Secure HTTP (HTTPS) protocols. HEC uses a token-based authentication model. You can generate a token and then configure a logging library or HTTP client with the token to send data to HEC in a specific format.</p> <h3>Key features of HEC</h3> <ul> <li>Token-based authentication: Each token has a unique value, which is a 128-bit number that is represented as a 32-character globally unique identifier (GUID)</li> <li>Secure communication: Supports both HTTP and HTTPS protocols for data transmission</li> <li>API key support: Provides secure authentication mechanisms that align perfectly with HPE GreenLake's security requirements</li> <li>Flexible data formats: Accepts both JSON-formatted events and raw text data</li> </ul> <h2>Overview of HPE GreenLake webhooks</h2> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/event/public/webhooks/">HPE GreenLake webhooks</a> facilitate automated, real-time communication between HPE GreenLake cloud services and an external service of your choosing. For example, a webhook could notify your IT Operation Management platform when a new audit log is created, or when subscriptions are about to expire. A getting started guide to HPE GreenLake webhooks is available <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">here</a>.</p> <h3>HPE GreenLake webhook security features:</h3> <p>HPE GreenLake implements robust security measures to ensure webhook authenticity:</p> <ul> <li>Challenge Request Validation: After registering a webhook, a verification challenge is sent to the destination (the webhook URL). The event type is <code>hpe.greenlake.events.v1beta1.webhooks.verification</code>. The challenge request includes a unique, random string generated by the server that is sent in the body as a payload.</li> <li>HMAC SHA-256 Signature Verification: HPE GreenLake webhooks use a verification challenge process to ensure that webhook connections are legitimate and secure. HPE GreenLake secures webhook notifications through HMAC (Hash-based Message Authentication Code) with SHA-256, a cryptographic hashing algorithm.</li> <li>Shared Secret Management: The platform supports dual secret rotation for zero-downtime security updates.</li> </ul> <p>Challenge Request Example:</p> <pre><code class="language-json">{ "specversion" : "1.0", "type" : "hpe.greenlake.events.v1beta1.webhooks.verification", "source" : "//global.api.greenlake.hpe.com/events", "id" : "C234-1234-1234", "time" : "2018-04-05T17:31:00Z", "datacontenttype" : "application/json", "data" : { "challengeRequest" : "&#x3C;TOKEN>" } } </code></pre> <h2>Challenges and solutions</h2> <p>The primary challenge in integrating HPE GreenLake webhooks with Splunk HEC lies in the webhook verification process. The destination must read the value from the challengeRequest field and create an HMAC SHA-256 hash, using the webhook secret as salt and the challengeRequest value as a string to hash. When successful, the destination responds with a JSON object with the format <code>{"verification": "CREATED_HASH"}</code> and a HTTP 200 OK status.</p> <h3>A challenge</h3> <p>Splunk's HEC endpoint is designed for data ingestion and doesn't natively support the  challenge-response mechanism required by HPE GreenLake webhooks. HEC expects to receive event data directly and cannot handle the initial verification handshake.</p> <h3>The solution</h3> <p>This is where <a href="https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/">Splunk's custom REST endpoints</a> capability becomes invaluable. A custom REST endpoint is a developer-defined endpoint and associated handler that lets you build out the Splunk REST API to meet your specific needs. We can create a custom endpoint handler that:</p> <ol> <li>Intercepts the initial challenge request from HPE GreenLake</li> <li>Validates the challenge using HMAC SHA-256</li> <li>Responds appropriately to complete the verification</li> <li>Forwards validated event data to HEC for ingestion</li> </ol> <h2>Custom REST endpoints in Splunk</h2> <p>Splunk's custom REST endpoints provide powerful extensibility for scenarios exactly like ours. You use a custom endpoint to add a special feature that Splunk doesn't have built-in, like, in our case, handling the unique secret handshake from HPE GreenLake.</p> <p>Key benefits of our integration:</p> <ul> <li>Flexible request handling: Can process both challenge requests and event data</li> <li>Custom logic implementation: Handler code implements HPE GreenLake's specific validation requirements</li> <li>Centralized management: Provides a single endpoint for webhook management</li> </ul> <h2>Where to configure the endpoint handler: Splunk Enterprise vs Splunk Cloud</h2> <p>Splunk Enterprise is the self-hosted version that an organization deploys and manages on its own infrastructure, either on-premises (on-prem) or in a private cloud.</p> <p>Splunk Cloud Platform is the Software as a Service (SaaS) offering, where the Splunk platform is hosted, managed, and maintained by Splunk.</p> <h3>For Splunk Enterprise</h3> <p>You can install and configure the endpoint handler directly on your Splunk Enterprise instance by placing it in the etc/apps/ directory and following the steps in this guide. Splunk Enterprise supports custom REST endpoints out of the box.</p> <h3>For Splunk Cloud</h3> <p><a href="https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud">Splunk Cloud has extra security controls</a>, so you might need to take additional steps to allow your helper to communicate with the Splunk REST API.</p> <h2>Generate an Splunk REST API key</h2> <p>In order to use the custom REST endpoint in Splunk, you need to get an API key which HPE GreenLake will use to authenticate against Splunk when initially setting up the webhook and sending events. You can use the following cURL command to generate a key.</p> <pre><code class="language-shell">curl -k https://your-splunk-instance:8089/services/auth/login \ --data-urlencode username=YOUR_USERNAME \ --data-urlencode 'password=YOUR_PASSWORD’ </code></pre> <p>The response will be:</p> <pre><code class="language-json">&#x3C;response> &#x3C;sessionKey>YOUR_SESSION_KEY&#x3C;/sessionKey> &#x3C;messages> &#x3C;msg code="">&#x3C;/msg> &#x3C;/messages> &#x3C;/response> </code></pre> <h2>Sample Python handler</h2> <p>Let's create a custom REST endpoint handler in Python to handle the HPE GreenLake webhook validation and forwards events to Splunk HEC, once validated.</p> <h3>Directory structure</h3> <pre><code class="language-markdown">splunk_hpe_webhook_app/ ├── bin/ │   └── hpe_webhook_handler.py ├── default/ │   ├── restmap.conf │   └── web.conf └── metadata/ └── default.meta </code></pre> <h3>Python handler (bin/hpe_webhook_handler.py)</h3> <pre><code class="language-python">import os import sys import json import hmac import hashlib import urllib.request import urllib.parse from splunk.rest import BaseRestHandler class HPEWebhookHandler(BaseRestHandler): def __init__(self, command_line, command_arg): super(HPEWebhookHandler, self).__init__(command_line, command_arg) # Configure your HEC endpoint and token self.hec_url = "&#x3C;https://your-splunk-instance:8088/services/collector/event>" self.hec_token = "YOUR_HEC_TOKEN_HERE" self.webhook_secret = "YOUR_WEBHOOK_SECRET_HERE" def handle_POST(self): """Handle POST requests from HPE GreenLake webhooks""" try: # Parse the incoming request body request_body = json.loads(self.request['payload']) event_type = request_body.get('type', '') # Check if this is a verification challenge if event_type == 'hpe.greenlake.events.v1beta1.webhooks.verification': return self._handle_challenge(request_body) else: # Validate signature for regular events if self._validate_signature(): return self._forward_to_hec(request_body) else: return self._return_error("Invalid signature", 401) except Exception as e: return self._return_error(f"Error processing request: {str(e)}", 500) def _handle_challenge(self, request_body): """Handle HPE GreenLake webhook verification challenge""" try: # Extract challenge request token challenge_request = request_body['data']['challengeRequest'] # Create HMAC SHA-256 hash hmac_hash = hmac.new( key=self.webhook_secret.encode('utf-8'), msg=challenge_request.encode('utf-8'), digestmod=hashlib.sha256 ) # Create verification response verification_response = { "verification": hmac_hash.hexdigest() } # Return successful verification self.response.setHeader('content-type', 'application/json') self.response.write(json.dumps(verification_response)) return except Exception as e: return self._return_error(f"Challenge validation failed: {str(e)}", 400) def _validate_signature(self): """Validate HMAC signature for regular webhook events""" try: # Get signature from headers signature_header = self.request.get('headers', {}).get('hpe-webhook-signature', '') if not signature_header.startswith('sha256='): return False expected_signature = signature_header[7:] # Remove 'sha256=' prefix # Calculate expected signature payload = self.request['payload'] calculated_signature = hmac.new( key=self.webhook_secret.encode('utf-8'), msg=payload.encode('utf-8'), digestmod=hashlib.sha256 ).hexdigest() # Compare signatures return hmac.compare_digest(expected_signature, calculated_signature) except Exception: return False def _forward_to_hec(self, event_data): """Forward validated event data to Splunk HEC""" try: # Prepare HEC request hec_data = { "event": event_data, "sourcetype": "hpe:greenlake:webhook", "source": "hpe_greenlake", "index": "main" # Configure as needed } # Create HTTP request to HEC req = urllib.request.Request( url=self.hec_url, data=json.dumps(hec_data).encode('utf-8'), headers={ 'Authorization': f'Splunk {self.hec_token}', 'Content-Type': 'application/json' } ) # Send to HEC with urllib.request.urlopen(req) as response: if response.status == 200: self.response.setHeader('content-type', 'application/json') self.response.write(json.dumps({"status": "success"})) return else: return self._return_error("Failed to forward to HEC", 500) except Exception as e: return self._return_error(f"HEC forwarding failed: {str(e)}", 500) def _return_error(self, message, status_code): """Return error response""" self.response.setStatus(status_code) self.response.setHeader('content-type', 'application/json') self.response.write(json.dumps({"error": message})) </code></pre> <h3>Configuration files</h3> <h4>default/restmap.conf</h4> <pre><code class="language-markdown">[script:hpe_webhook_handler] match = /hpe/webhook script = hpe_webhook_handler.py scripttype = persist handler = hpe_webhook_handler.HPEWebhookHandler requireAuthentication = false output_modes = json passPayload = true passHttpHeaders = true passHttpCookies = false </code></pre> <h4>default/web.conf</h4> <pre><code class="language-markdown">[expose:hpe_webhook_handler] pattern = hpe/webhook methods = POST` </code></pre> <h4>metadata/default.meta</h4> <pre><code class="language-markdown">[restmap/hpe_webhook_handler] export = system [views] export = system` </code></pre> <h2>Configuring Splunk HTTP Event Collector (HEC)</h2> <p>You need to create an API token to use HEC via its API. You can do this from:</p> <p>1.    <strong>Settings > Data Inputs > HTTP Event Collector</strong></p> <p>2.    Select <strong>New token.</strong> Use this token to update the Python handler script line</p> <p><img src="/img/dccf7431-b83b-4799-a795-25bccc7637db.png" alt="Data inputs settings " title="Data inputs settings "></p> <p>Your final configuration should look like this:</p> <p><img src="/img/03af2f59-ab50-443b-ae93-90d274738317.png" alt="HEC final configuration " title="HEC final configuration "></p> <p>Verify your global settings so that they match the following:</p> <p><img src="/img/310bb0be-5363-428a-99a2-5f64bef2ca8a.png" alt="Global settings" title="Global settings"></p> <p>This allows you to get your HEC endpoint, which is used in the Python handler to create an incident based on the HPE GreenLake event received via the webhook. The URL of the endpoint should look like this:</p> <p><code>&#x3C;https://&#x3C;splunk-host>>:8088/services/collector/event</code></p> <h2>Final integration flow</h2> <p>The complete integration flow works as follows:</p> <ol> <li> <p>Initial setup</p> <ul> <li> <p>Deploy the custom Splunk endpoint handler using the above HPE webhook handler Python script.</p> </li> <li> <p>Make sure to set the HEC endpoint in the Python script (line 13)</p> </li> <li> <p>Make sure to set the Splunk HEC API token in the Python script (line 14)</p> </li> <li> <p>Make sure to set the HPE GreenLake webhook secret in the Python script (line 15)</p> </li> <li> <p>Register the webhook handler URL with HPE GreenLake:</p> <ul> <li>URL of handler: <a href="https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook"><code>https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook</code></a></li> <li>Set the same secret key as we setup in the Python handler (line 15)</li> <li>Use API as Authentication type and set the API key to the Splunk REST API Key generated in the section above.</li> </ul> </li> </ul> </li> </ol> <blockquote> <p>Note: See <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">this blog</a> to learn how to register a new webhook handler in HPE GreenLake</p> </blockquote> <ol start="2"> <li> <p>Webhook handler verification process</p> <ul> <li>HPE GreenLake sends a verification challenge to your custom endpoint.</li> <li>The custom REST handler receives the challenge and validates it using HMAC SHA-256.</li> <li>The handler responds with the computed verification hash.</li> <li>HPE GreenLake confirms the webhook and marks it as active.</li> </ul> </li> <li> <p>Event processing flow</p> <ul> <li>HPE GreenLake sends event data to the custom handler endpoint.</li> <li>The custom REST handler validates the HMAC signature.</li> <li>The handler forwards validated events to HEC.</li> <li>Splunk HEC ingests the data for analysis and visualization.</li> </ul> </li> <li> <p>Data flow diagram</p> </li> </ol> <p><img src="/img/diagram.jpg" alt="Data flow diagram" title="Data flow diagram"></p> <h2>Benefits of this architecture</h2> <p>Security: The custom endpoint handler ensures only validated, authentic events reach your Splunk environment.</p> <p>Reliability: If there are more than 20 failures in a 12-hour period, the webhook enters the critical state in HPE GreenLake. If there are no new failures in 12 hours, the webhook returns to the active state. The custom handler can implement robust error handling to maintain webhook health.</p> <p>Scalability: The solution can handle multiple webhook types and route them to different HEC endpoints or indexes as needed.</p> <p>Monitoring: All webhook interactions are logged within Splunk for troubleshooting and monitoring.</p> <h2>Testing and deployment</h2> <p>Testing the integration</p> <ol> <li>Verify custom endpoint: Test your custom REST endpoint using curl:</li> </ol> <pre><code class="language-shell">`curl -X POST`[`https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook`](https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook)`\` `-H "Content-Type: application/json" \` `-d '{"type": "test.event", "data": {"message": "Hello Splunk"}}'` </code></pre> <ol start="2"> <li>Webhook registration: Register your webhook with HPE GreenLake using the custom endpoint URL.</li> <li>Challenge validation: Monitor Splunk logs to ensure the challenge request is handled correctly.</li> <li>Event flow testing: Trigger test events from HPE GreenLake and verify they appear in your Splunk index.</li> </ol> <h2>Conclusion</h2> <p>Integrating HPE GreenLake webhooks with Splunk via HTTP Event Collector presents unique challenges due to the webhook verification requirements, but Splunk's custom REST endpoints capabilities provide an elegant solution. Such integration offers several key benefits:</p> <p>Enhanced security: The custom REST endpoint handler ensures that only validated, authentic events from HPE GreenLake reach your Splunk environment, maintaining the security standards required by both platforms.</p> <p>Seamless event flow: Once configured, events flow automatically from HPE GreenLake to Splunk, enabling real-time monitoring and analysis of your cloud infrastructure.</p> <p>Extensible architecture: The custom REST handler can be extended to support multiple webhook types, different routing logic, and additional validation mechanisms as your requirements evolve.</p> <p>Whether you're monitoring subscription changes, or audit events, this integration ensures that your HPE GreenLake data becomes a valuable part of your Splunk analytics ecosystem, empowering your organization with comprehensive, real-time insights into your cloud infrastructure.</p>https://developer.hpe.com/open-sourcing-workshops-on-demand-part-5-deploying-and-managing-api-db-server/https://developer.hpe.com/open-sourcing-workshops-on-demand-part-5-deploying-and-managing-api-db-server/Tue, 24 Jun 2025 09:52:09 GMT<p>In previous articles of this series dedicated to the <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">open sourcing of our Workshops-on-Demand project</a>, I covered the reasons why we open sourced the project and how we did it. I also explained in details how you could install your own Workshops-on-Demand backend server. I also took the time to detail the automation that was hosted on this backend server. I also described to you the management of this backend server. This is what is often referred to as Day2 operations. I plan now to explain how to deploy and manage the API-DB server.</p> <p>The api-db server provides an open API 3.0 based api used to manage the Workshops-on-Demand project. it also provides a Database hosting the different status of participants, workshops, students.</p> <p>The following image is describing the different interactions existing between the different components of the wod architecture.</p> <p><img src="/img/howto-wod-1.png" alt="" title="WOD Architecture"></p> <h2>How to deploy your own api-db server...</h2> <p>As explained in the previous <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">article</a>, the project is split into multiple repositories from the architectural and public / private aspects. The architecture is divided between the frontend and backend. The project admins will need to decide whether they are willing to develop and propose public-only content to the participants or add any proprietary and private content.</p> <p>I will start with the simpliest scenario: A public-only approach. Then we will dive into the specificities related the private approach.</p> <h3>Public-only Deployment: No private backend nor private workshops</h3> <p><strong>Important Note:</strong></p> <p><strong>This part is compulsory for any type of deployment. Public only or public + private.</strong></p> <p>First, you need a repository to clone. The Workshops-on-Demand GitHub projects can be found <a href="https://github.com/Workshops-on-Demand/">here</a>. We have packaged the solution in several Github repos. Each repository handles a specific role in the overall architecture. We recently introduced a wod-install repository.</p> <p>This repository is the most important one when it comes to deploying the wod infrastructure. It contains all the installation scripts for every part of the solution.</p> <p>Here's a quick look at what can be found in each:</p> <p><img src="/img/wod-blogserie2-2repos.png" alt="" title="WOD repositories"></p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-notebooks">wod-notebooks</a>:</strong> Public Workshops-on-Demand based on Jupyter Notebooks.</p> <ul> <li>You can test them live at <a href="https://hackshack.hpedev.io/workshops">https://hackshack.hpedev.io/workshops</a></li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-install">wod-install</a>:</strong> Installer part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-backend">wod-backend</a>:</strong> Back-end part of our Workshops-on-Demand setup.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend">wod-frontend</a>:</strong> Frontend part of the Workshops-on-Demand project.</p> <ul> <li>Based on NGINX and NodeJS technologies, it provides the participtants' Registration Portal used to enable booking of the workshops.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db">wod-api-db</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>Open API 3.0 based api used to manage the Workshops-on-Demand project. It also provides a database hosting the different status of participants, workshops, and students.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-private">wod-private</a>:</strong> Example Private configuration for Workshops-on-Demand (WoD).</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend-private">wod-frontend-private</a>:</strong> Private Frontend part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db-private">wod-api-db-private</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>This provide examples for creating your own cutomization layer on top of the public standard WoD Backend / wod Notebooks content. Do not put any confidential data here as this is a public repository!</li> </ul> <p><strong>Note</strong>: There are now 7 repositories available for now.</p> <p>It provides:</p> <ul> <li> <p>An Installer allowing you to install either Backend, Api-DB server, or Frontend using a single line of command.</p> </li> <li> <p>A complete JupyterHub server with some addons (additional Jupyterhub kernels, Ansible galaxies, and PowerShell libraries) on your system, ready to host Workshops-on-Demand that you can find here.</p> </li> <li> <p>An api-db Server to host Workshops data and provide an API server to retrieve relevant workshops data</p> </li> <li> <p>A frontend server to provide registration process</p> </li> <li> <p>A postfix server used for the procmail API</p> </li> <li> <p>An Ansible engine to allow automation</p> </li> <li> <p>A fail2ban service</p> </li> <li> <p>An Admin user to manage everything</p> </li> <li> <p>A set of scripts to handle different tasks such as:</p> <ul> <li>Notebooks deployment</li> <li>Jupyterhub compliancy</li> <li>Users compliancy</li> <li>Security Management</li> </ul> </li> </ul> <h4>API-DB server preparation:</h4> <p>Before cloning the install repository, you will need to prepare the server that will host the backend features. When ready, you will proceed with the cloning and then the installation process.</p> <h5>Prerequesites:</h5> <p>In order to setup the api-db server, you will need:</p> <ul> <li>A fresh OS install on physical / virtualized server running Ubuntu 24.04 or Centos 7.9 leveraging any deployment mechanism of your choice.(e.g. iLO, vagrant, etc.). You may even use this vagrant file to automatically generate a complete setup leveraging vagrant, libvirt and QEMU/KVM.</li> <li>A Linux account with sudo priviledges on your Linux distro. Name it <code>install</code></li> <li>Git installed on the machine</li> </ul> <p>Our current setup leverages the following specs for our api-db server</p> <ul> <li>2 cpus or more machine</li> <li>16 GB of RAM</li> <li>80 GB of storage</li> </ul> <p>We are currently using a virtual machine on AWS for our different production sites.</p> <p>When done with OS installation and preparation</p> <ul> <li>From the WoD-api-db server, as the install user, you will need to clone the wod-install repo first.</li> </ul> <pre><code>install$ git clone https://github.com/Workshops-on-Demand/wod-install.git install$ cd wod-install/install </code></pre> <ul> <li>The installation is based on a common install script <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/install/install.sh">install.sh </a>that allows the deployment of the different parts of the solution. It can be called as follows:</li> </ul> <pre><code>install$ install.sh [-h][-t type][-g groupname][-b backend][-f frontend][-a api-db][-e external][-u user][-s sender] </code></pre> <p><code>-h</code> provides the help</p> <p><code>install.sh</code> performs the following tasks depending on the type</p> <ul> <li> <p>Calls the <code>install-system-&#x3C;&#x3C; distribution name >>.sh</code> script</p> </li> <li> <p>Installs minimal required (<code>ansible, git, jq, openssh server, npm</code>)</p> </li> <li> <p>Creates an admin user as defined upper (default is <code>wodadmin</code>) with sudo rights</p> </li> <li> <p>Calls the <code>install-system-common.sh</code> script that performs the following tasks:</p> <ul> <li>Cleanup</li> <li>Github repos cloning (leveraging install.repo file) : public Backend and public Private repos</li> <li>Create ssh keys for wodadmin</li> <li>Creates GROUPNAME variables</li> <li>Creates Ansible inventory files</li> </ul> </li> <li> <p>Calls the <code>install_system.sh</code> script with the type (api-db, backend, or frontend) that performs the following tasks:</p> <ul> <li>Install the necessary stack based on selected type</li> <li>Create a <code>wod.sh</code> script in <code>wod-backend</code> directory to be used by all other scripts</li> <li>Source the <code>wod.sh</code> file</li> <li>Setup Ansible-galaxies (<code>community.general</code> and <code>posix</code>)</li> <li>Setup Ansible and call the playbook <code>install_&#x3C;type>.yml</code> followed by the <code>ansible\_check\_&#x3C;type>.yml</code></li> </ul> </li> </ul> <p>At the end of the api-db installation process:</p> <ul> <li>you will have a postgres database running in a docker container and populated with the data coming from the different workshops yaml files.</li> <li>You will have a postgres adminer running</li> <li>You will get a api server running along with his swagger description.</li> </ul> <p>Please note that this setup phase can be concurrent with the public setup phase. Indeed, the install script should detect the presence of the private repository owing to the presence of the install.priv file. It will automatically adjust the different scripts and variables to add the relevant content. It will actually overload some of the variables with private ones.</p> <p>You now have a working Workshops-on-Demand api-db server in place.</p> <p>Congratulations! The next article in the series will help you better understand the lifecycle of the backend server. How does a workshop registration work from the backend server 's side? How do you manage this server on a daily basis? How and when do you need to update it ? All these questions will be answered in the next article. And from there, we will move to the frontend side of things and finally to a workshop's creation process.</p> <p>If you need support for this installation process, use our dedicated <a href="https://hpedev.slack.com/archives/C01B60X8SSD">slack channel</a>.</p> <p>Please be sure to check back <a href="https://developer.hpe.com/blog">HPE Developer blog site</a> to read all the articles in this series. Also, check out the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a> <a href="https://developer.hpe.com/hackshack/replays/42">Data Visualization 101</a> is now available! Stay tuned for additional Workshops-on-Demand in our catalog.</p> <h2>How to manage your own api-db server...</h2> <p>The main component of the api-db server is database. there are two ways of managing the data content :</p> <p>Postgres adminer:</p> <p>in order to access the postgres adminer console: browse to either the internal or external (if any) IP Address on port 8083</p> <p><img src="/img/screenshot-2025-06-24-at-15.35.58.png" alt="" title="Postgres Adminer console login"></p> <p>Using the console, the admin can update the relevant tables manually.</p> <p><img src="/img/screenshot-2025-06-24-at-15.38.11.png" alt="" title="Postgres DB Workshops table"></p> <p>However, we now recommand you leverage our scripts that will take care of updating the relevant data automatically.</p> <p>Let's start with a simple example:</p> <h3>You want to add a new workshop to your catalog. How should you proceed?</h3> <ul> <li>Start by developing a new workshop (a future blog will help you understand how to achieve this).</li> <li>As part of the workshop development, you will have to create a simple yaml file that will describe the workshop.</li> <li>Once the workshop content is ready, you will create a pull request to the wod-notebooks repository to update its content.</li> <li>From the api-db server, as the install user, launch now the seeders script to update the database. The script will parse the different yaml files and proceed with the necessary updates to the database.</li> </ul> <p>Take a look at the folowing example:</p> <pre><code>%YAML 1.1 # Meta data for the API101 Workshop to populate seeder --- name: 'API 101 - API basics and the value they provide' description: 'You may know that application programming interfaces (APIs) allow applications to talk to other apps, but have you ever used them? Today, APIs are available for most products and solutions. You can take advantage of them when writing automation scripts, integrating code, or defining infrastructure-as-code, as long as you understand the mechanisms used to consume an API. In this hands-on workshop, we’ll review all the jargon and technology used by REST APIs.' active: true capacity: 20 priority: 1 range: [1,6] reset: false ldap: false replayId: 9 varpass: false compile: false workshopImg: 'https://us-central1-grommet-designer.cloudfunctions.net/images/jay-giang-hpe-com/WOD_Opensource-API_101_REST_API_basics_and_the_value_they_provide.jpg' badgeImg: 'https://us-central1-grommet-designer.cloudfunctions.net/images/jay-giang-hpe-com/Opensource-API_101_REST_API_basics_and_the_value_they_provide.jpg' beta: false category: ['Open Source'] duration: 2 presenter: 'Didier Lalli' role: 'Distinguished Technologist' avatar: '/img/wod/SpeakerImages/Didier.png' replayLink: 'https://youtu.be/T57L-6LfUgw' </code></pre> <p>As you can see, it contains many information, from the name of the workshop, its description, whether it should be active in the databse, its capacity and much more...</p> <p>Every single filed in this file with be leverage to add a workshop in the workshop table form the database. Using a seeding script, every single yaml file present in each workshop folder of the wod-notebook repositories (public and private) get imported in the databse as part of the api-db server install process.</p> <p>Leveraging the very same mechanism, one can add, update a workshop.</p> <h3>You want to add a new filed in the workshop table. How should you proceed?</h3> <p>In order to achieve this, you will need to update the workshops.js file in the model directory within the wod-api-db folder on the api-db server.</p>https://developer.hpe.com/announcing-chapel-2-5/https://developer.hpe.com/announcing-chapel-2-5/Fri, 13 Jun 2025 08:44:29 GMT<p>external blog</p>https://developer.hpe.com/hpe-greenlake-cloud-integration-with-servicenow-using-webhooks/https://developer.hpe.com/hpe-greenlake-cloud-integration-with-servicenow-using-webhooks/Fri, 06 Jun 2025 14:45:12 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><a href="https://servicenow.com/">ServiceNow</a>, as a third-party platform, provides robust capabilities to develop custom webhook handlers that can directly receive and process external events in real time. These webhook handlers can be easily integrated with HPE GreenLake webhooks, allowing for smooth communication between systems. This flexibility enables teams to automate a wide range of workflows and trigger specific actions based on customer-defined requirements. By enabling seamless integration with other tools and services, this feature significantly enhances operational efficiency, reduces manual intervention, and ensures faster response to critical business events. Below are the steps to set up ServiceNow, develop a webhook handler, and integrate with the HPE GreenLake unified events framework to receive real-time events.</p> <h2>Optional step 1: Login and setup of a ServiceNow instance</h2> <blockquote> <p><em>Note: You can ignore Step 1 if you already have a ServicesNow instance.</em></p> </blockquote> <ol> <li>Create a ServiceNow developer instance in ServiceNow using the following link: <a href="https://signon.service-now.com/">https://signon.service-now.com/</a></li> <li>After signing in, click on the Request Instance found at the top right corner of the screen.</li> </ol> <p><img src="/img/servicenow-webhook-blog-eceaeba3-a264-4fa7-af6b-434a1d26832d.jpeg" alt="Request ServiceNow instance" title="Request ServiceNow instance"></p> <ol start="3"> <li>Once requested, it will take a minute to provision the instance, click on the profile to check what the status is. The status should be Online.</li> </ol> <p><img src="/img/servicenow-webhook-blog-a8242a8f-67d7-474e-8f06-60985ab1719a.jpeg" alt="Manage ServiceNow instance" title="Manage ServiceNow instance"></p> <ol start="4"> <li>Click on Manage instance password from the INSTANCE ACTION list, as shown above. Here, you will find the details about your ServiceNow instance such as its URL, username, and password. Log in to your instance URL with the provided credentials.</li> </ol> <p><img src="/img/servicenow-webhook-blog-0017ecb8-a01f-4ab8-9261-4d626d983491.jpeg" alt="ServiceNow instance details" title="ServiceNow instance details"></p> <h2>Step 2: Create a webhook endpoint</h2> <ol> <li>Once logged into your ServiceNow instance, click on All, search, and click on scripted rest api.</li> <li>Click on New in the top-right corner and fill in the Name. Click outside the box. The App Id will automatically pop up. Then, click on submit.</li> <li>Once it has been successfully submitted, you can search in the scripted REST API using the same name you created (ex: unified events)., Click on that record, scroll to bottom, and click on New.</li> <li>Now, fill in the details: i.e. give it the appropriate Name and relative path, select HTTP method to POST, uncheck Require authentication, and click on Submit.</li> </ol> <p><img src="/img/servicenow-webhook-blog-cfb0ec5a-f2f3-4982-bf02-361f5efe9d73.jpeg" alt="Registering a scripted REST API handler" title="Registering a scripted REST API handler"></p> <ol start="5"> <li>Paste the code snippet below in the Script section of the above created record (Change the secret in line 144).</li> </ol> <blockquote> <p><em>Note: To better understand the code below, which implements the verification challenge sent to the ServiceNow handler by HPE GreenLake cloud, refer to the documentation page <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/event/public/webhooks/#webhook-verification">here</a> or the following <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">blog post</a>.</em></p> </blockquote> <pre><code class="language-python">// Code Snippet for GreenLake Webhook handler (function process( /*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) { var requestBody = request.body.data; var setAuth = gs.getProperty('setAuth'); if (setAuth == "true") { var reqHeaderAuthorization = request.getHeader("Authorization"); if (reqHeaderAuthorization == null || reqHeaderAuthorization == "") { response.setStatus(401); response.setHeader("Content-Type", "application/json"); var respWriter = response.getStreamWriter(); respWriter.writeString(JSON.stringify({ httpStatusCode: "401", error: "Authorization header is missing" })); respWriter.writeString("\n"); return; } else { var includesPrefixBearer = reqHeaderAuthorization.includes("Bearer"); gs.info("includesPrefixBearer " + includesPrefixBearer); if (includesPrefixBearer) { // oauth validation var token = reqHeaderAuthorization.split('Bearer')[1]; token = token ? token.trim() : ''; var validatedToken = validateToken(token); if (!validatedToken) { response.setStatus(401); response.setHeader("Content-Type", "application/json"); var respWriter2 = response.getStreamWriter(); respWriter2.writeString(JSON.stringify({ httpStatusCode: "401", error: "Authorization failed" })); respWriter2.writeString("\n"); return; } } else { // apikey validation var validatedResponse = checkAPIKEYValidation(reqHeaderAuthorization); if (!validatedResponse) { response.setStatus(401); response.setHeader("Content-Type", "application/json"); var respWriter4 = response.getStreamWriter(); respWriter4.writeString(JSON.stringify({ httpStatusCode: "401", error: "Authorization failed" })); respWriter4.writeString("\n"); return response; } } } } eventType = requestBody.type; gs.info("event type " + eventType); eventData = requestBody.data; if (eventType.includes("verification")) { gs.info("Recieved code verification request"); codeChallengeStr = requestBody.data.challengeRequest; gs.info("codeChallenege Request :" + codeChallengeStr); var generated; var resp; generated, resp = getHMACSecretFromLambda(codeChallengeStr); if (generated == false) { response.setStatus(500); response.setContentType("application/json"); var writer = response.getStreamWriter(); writer.writeString(JSON.stringify({ httpStatusCode: "500", error: JSON.stringify(resp) })); writer.writeString("\n"); return; } response.setStatus(201); response.setHeader("Content-Type", "application/json"); var writerResp2 = response.getStreamWriter(); writerResp2.writeString(JSON.stringify({ httpStatusCode: "201", verification: resp })); writerResp2.writeString("\n"); pushEventToIncidentTable(JSON.stringify(eventData), eventType); return; } else if (eventType.includes("com.hpe")) { gs.info("Recieved event with event type : " + requestBody.type); hpeWebhookSignature = request.getHeader("HPE-Webhook-Signature"); eventData = requestBody; var generatedSignature; generated, resp = getHMACSecretFromLambda(JSON.stringify(eventData)); generatedSignature = "sha256=" + resp; if (generatedSignature == hpeWebhookSignature) { gs.info("Received webhook signature are matching"); } else { gs.info("webhook signature are not same, Recieved: " + hpeWebhookSignature + " generated : " + resp); response.setStatus(412); response.setHeader("Content-Type", "application/json"); var writerResp4 = response.getStreamWriter(); writerResp4.writeString(JSON.stringify({ httpStatusCode: "412", error: "hmac signature failed to match" })); writerResp4.writeString("\n"); return; } } else { gs.warn("Not a valid request body doesn't contains type in req body"); response.setStatus(400); response.setHeader("Content-Type", "application/json"); var respWriter3 = response.getStreamWriter(); respWriter3.writeString(JSON.stringify({ httpStatusCode: "400", error: "Not a valid request body, type field is missing" })); respWriter3.writeString("\n"); return; } // push incident to incident table pushEventToIncidentTable(JSON.stringify(eventData), eventType); response.setStatus(201); response.setHeader("Content-Type", "application/json"); var writerResp3 = response.getStreamWriter(); writerResp3.writeString(JSON.stringify({ httpStatusCode: "200", })); writerResp3.writeString("\n"); })(request, response); function getHMACSecretFromLambda(codechallengeReq) { function encodeToHex(byteArray) { var hexChars = "0123456789abcdef"; var hexStr = ""; for (var i = 0; i &#x3C; byteArray.length; i++) { var bt = byteArray[i] &#x26; 0xFF; hexStr += hexChars[(bt >> 4) &#x26; 0x0F]; hexStr += hexChars[bt &#x26; 0x0F]; } return hexStr; } key = "your secret"; var mac = new GlideCertificateEncryption(); var base64Key = GlideStringUtil.base64Encode(key); var base64HMAC = mac.generateMac(base64Key, "HmacSHA256", codechallengeReq); var byteArray = GlideStringUtil.base64DecodeAsBytes(base64HMAC); var hexHMAC = encodeToHex(byteArray); gs.info("hmac generated sig " + hexHMAC); return true, hexHMAC; } function pushEventToIncidentTable(description, shortDescription) { try { var gr = new GlideRecord('incident'); gr.initialize(); gr.short_description = "HPE GLCP: " + shortDescription; gr.description = description; gr.assigned_to = "admin"; gr.assignment_group = "HPE GLCP"; gr.caller_id = "System Administrator"; gr.urgency = 1; gr.priority = 1; gr.insert(); gs.info("✅ Pushed event to Incident table: " + gr.number); } catch (ex) { gs.warn("Error in pushing event to Incident table: " + ex.message); } } function checkAPIKEYValidation(apikey) { var gr = new GlideRecord("u_unified_events_api_key"); gr.query(); var response = sn_ws_int.RESTAPIResponse var apiKeyValid = false; while (gr.next()) { var now = new GlideDateTime(); var expiryTime = new GlideDateTime(gr.u_expiry_time); gs.info("expiryTime.after(now) " + expiryTime.after(now)); if (apikey == gr.u_api_key &#x26;&#x26; expiryTime.after(now)) { gs.info("API key is valid"); apiKeyValid = true; break; } } if (!apiKeyValid) { gs.info(" the requested api key is not a valid key or might had expired: " + apikey); return false; } return true; } function decodeBase64Url(base64Url) { var base64 = base64Url.replace(/-/g, '+').replace(/_/g, '/'); while (base64.length % 4) base64 += '='; return GlideStringUtil.base64Decode(base64); } function validateToken(token) { var parts = token.split('.'); if (parts.length === 3) { var header = decodeBase64Url(parts[0]); var payload = decodeBase64Url(parts[1]); gs.info("JWT Header: " + header); gs.info("JWT Payload: " + payload); } else { gs.error("Invalid JWT token format."); return false; } var payloadObj = JSON.parse(payload); gs.info("Expires At (epoch): " + payloadObj.exp); var now = new Date().getTime() / 1000; if (payloadObj.exp &#x26;&#x26; payloadObj.exp &#x3C; now) { gs.error("Token is expired."); } else { gs.info("Token is still valid"); return false; } return true; } </code></pre> <h2>Step 3: Set up the API key</h2> <ol> <li>You can set up an API Key as an authentication method, To do this, you need to create a new table.</li> <li>Search for tables in All (header bar), look for System Definitions → Tables, and click on it, Click on New (top-right corner), fill in the details for its label name. It will auto-populate by clicking outside, Click submit. (ex: unified-events-api-key)</li> <li>Now, search in tables with above-created name, making sure to select for text as a filter. Click on the table with matched result.</li> </ol> <p><img src="/img/servicenow-webhook-blog-4d7ccf04-4626-45a9-998f-35d9a03acf6f.jpeg" alt="Setting up API key" title="Setting up API key"></p> <ol start="4"> <li>Now, add 2 more columnar fields to the table api_key and expiry_time, as shown below, with proper types by clicking on insert new rows.</li> </ol> <p><img src="/img/servicenow-webhook-blog-f470575f-6d7a-48d1-a98c-ef161a54a827.jpeg" alt="Adding api_key and expiry_time fields" title="Adding api_key and expiry_time fields"></p> <ol start="5"> <li>Next, search this table for unified-events-api-key in All using the search bar. Click on new to add api-keys by generating new alphanumeric key, and select expiry time by calendar pick. One can add more API keys with other expiry time for easy rotation.(<a href="https://generate-random.org/api-key-generator?count=1&#x26;length=128&#x26;type=mixed-numbers&#x26;prefix=">https://generate-random.org/api-key-generator?count=1&#x26;length=128&#x26;type=mixed-numbers&#x26;prefix=</a>.)</li> </ol> <h2>Step 4: Configure the incidents table to get notifications in ServiceNow.</h2> <p>This step will allow you to trigger notifications in ServiceNow so one can easily verify the triggered events.</p> <ol> <li>In All section search for Business rules (System definitions → Business rules), click on and search Abort changes, and select one, as shown below.</li> </ol> <p><img src="/img/servicenow-webhook-blog-b59f8ece-6b27-4b70-af46-2bc83a062a3a.jpeg" alt="Configuring the incidents table" title="Configuring the incidents table"></p> <ol start="2"> <li>Once you click on the abort changes on a group, uncheck the Insert, as shown below.</li> </ol> <p><img src="/img/servicenow-webhook-blog-eaabb59e-9fbb-4ddc-907b-8796ecd3c5e6.jpeg" alt="Modifying abort change on group" title="Modifying abort change on group"></p> <ol start="3"> <li>Navigate to Business rules again and click on new to create a new rule, name it, (ex: trigger notification), select table Incident, enable checks for Active, Insert, Update, and add a filter condition (“Short description starts with HPE”), as shown below. Click on Submit.</li> </ol> <p><img src="/img/servicenow-webhook-blog-c6a22d4d-56b9-41cb-8dfa-a8fdac99ab6d.jpeg" alt="Adding description for event" title="Adding description for event"></p> <h2>Step 5: Call the webhook and test it with ServiceNow</h2> <ol> <li>Get the webhook endpoint. To get the Resource path, navigate to Scripted Rest API then search with created REST API name.</li> </ol> <p>It should look like:</p> <p><img src="/img/servicenoew-webhook-note1.jpg" alt="Finding the endpoint URL" title="Finding the endpoint URL"></p> <ol start="2"> <li>Once all the steps have been completed, call the webhook from Postman or copy the below CURL and paste it in a terminal. Make sure to change the --location url of your instance.</li> </ol> <pre><code class="language-shell">curl --location 'https://&#x3C;dev instance>.service-now.com/api/1762185/unified_events/webhook1' \ --header 'Content-Type: application/json' \ --data '{ "name": "Grills", "status": "SOLD", "description": "Grills is a Beer", "photo_url": "http://baseurl/resourceId?action=confirm", "data":{"challengeRequest":"e37dc544569d43c73d088"}, "type":"com.hpe.verification" }' </code></pre> <ol start="3"> <li>Now you will receive events in ServiceNow. Search for Incidents in All using the search bar, click on Assigned to me, and you should see the events triggered in table.</li> </ol> <p><img src="/img/servicenow-webhook-blog-e980f286-a395-48b7-9f8e-8596f0ab008b.jpeg" alt="Receiving events from GreenLake" title="Receiving events from GreenLake"></p> <ol start="4"> <li>Also, you will get an event notification in ServiceNow web console, as shown below.</li> </ol> <p><img src="/img/servicenow-webhook-blog-2743dab1-f57e-4e38-9878-b9d78860ad1b.jpeg" alt="Event notification in web console" title="Event notification in web console"></p> <ol start="5"> <li>One can debug from the logs in case of any failures. To do so, search for System Log from All in the search bar, and click on All in System Log to see all levels of logs.</li> </ol> <p><img src="/img/servicenow-webhook-blog-5c9b0e45-419b-4cb2-ab5d-62cd1d753712.jpeg" alt="events coming in" title="events coming in"></p> <ol start="6"> <li>Now, one can register the ServiceNow webhook using the secret that had been set in the script with Unified events framework, and will start receiving events in ServiceNow.</li> </ol> <h2>Conclusion</h2> <p>Leveraging ServiceNow’s webhook handling capabilities allows us to create highly responsive, automated workflows tailored to customer needs. By integrating seamlessly with existing systems, it not only streamlines operations but also enhances agility and scalability in managing real-time events across platforms.</p>https://developer.hpe.com/2025-june-04/https://developer.hpe.com/2025-june-04/Wed, 04 Jun 2025 13:00:00 GMThttps://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-2-past-failures-and-future-attempts/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-2-past-failures-and-future-attempts/Mon, 02 Jun 2025 09:07:43 GMThttps://developer.hpe.com/hybrid-observability-service-–-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/https://developer.hpe.com/hybrid-observability-service-–-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/Tue, 27 May 2025 15:36:05 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">my previous blog post</a>, I introduced the steps to install and configure an agentless SSH-enabled integration module and associated monitoring templates. This enables you to discover and monitor a Linux-based operating system in the hybrid observability service powered by HPE OpsRamp Software.</p> <p>Continuing from the third part of this series, I’ll dive into the observability of a physical HPE ProLiant compute server with an Integrated Lights-Out (iLO) 5 and above, out-of-band management interface, and an HPE storage array. I’ll also explore creating and importing a dashboard to visualize the collected metrics in charts.</p> <h2>Discovering and monitoring of HPE physical servers via the Redfish API</h2> <p>The <a href="https://glp.docs.opsramp.com/integrations/compute/server-hardware-monitoring-redfish/redfish-server/">Redfish - Server integration module</a> monitors and manages physical servers via the Redfish API, a modern, standardized interface for out-of-band hardware management.</p> <p>Redfish is a RESTful API designed by the Distributed Management Task Force (DMTF) for managing servers, storage, and other hardware components over the network. It’s supported by most modern server vendors like Dell iDRAC, HPE iLO, and NVIDIA.</p> <p>The Redfish Server integration module enables the <strong>discovery</strong> of physical server hardware and its components (CPU, memory, storage, power supplies, fans, temperature, and so on). Once discovered, Redfish Server monitors are <strong>automatically</strong> applied to the resource via predefined Global Monitoring Templates and Global Device Management Policies to manage the health and status of the server hardware components. You can customize the Global Monitoring Templates by cloning the templates as appropriate and applying the cloned monitoring template to your resource. Refer to <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">Part 3 of the series</a> to learn how to clone and apply a monitoring template to a resource.</p> <h3>Installing the Redfish – Server integration module</h3> <p>The Redfish Server integration module installation process for an HPE compute server is as follows:</p> <ol> <li> <p>Select the <strong>client account</strong>.</p> </li> <li> <p>From the navigation bar go to <strong>Setup > Account > Integrations</strong>.</p> </li> <li> <p>Click <strong>+ADD</strong> to search for <em>Redfish</em>.</p> </li> <li> <p>Choose to install <strong>Redfish – Server</strong> by selecting <strong>ADD</strong>.</p> </li> <li> <p>Select the application version and select <strong>+ADD</strong> to input basic account information, such as the <strong>Name</strong> for the configuration and the <strong>IP address/Host name</strong> of iLO for the HPE compute server that should be accessible from the gateway collector appliance. Specify the <strong>Port</strong> (typically port 443) and check the <strong>Is Secure</strong> checkbox.</p> </li> <li> <p>You must assign credentials for iLO interface. You can either:</p> <ul> <li>Select existing credentials for access to the iLO interface.</li> <li>Create new credentials. Click <strong>+ADD</strong> and specify the credentials name, a description, the username and password to access the iLO interface of the compute server.</li> </ul> </li> <li> <p>Although optional, it is recommended to select:</p> <ul> <li><strong>App Failure Notifications</strong> to be notified when the hybrid observability service can’t collect or process data from the Redfish-enabled server.</li> <li><strong>Event Polling/Alert Configuration</strong> for fetching events and alerts from the Integrated Management Log (IML) from the iLO. You can also select <strong>Alert On Root Resource</strong> for alerts to appear on the main server resource in the hybrid observability service.</li> </ul> </li> </ol> <blockquote> <p><strong>Note:</strong> Do not check the option <strong>CLI Credentials</strong> because it is only valid for HPE Edgeline servers.</p> </blockquote> <ol start="8"> <li> <p>In <strong>RESOURCE TYPE</strong>, check <strong>ALL</strong> or specify the hardware components to be discovered.</p> </li> <li> <p>Define a <strong>Discovery Schedule</strong>. You can select minutes, hourly, daily, weekly, or monthly schedule.</p> </li> <li> <p>Select <strong>ADD</strong> to save the configuration of the integration module.</p> </li> </ol> <blockquote> <p><strong>Note:</strong> In the <strong>ADVANCED SETTINGS</strong>, notice the <em><strong>Bypass Resource Reconciliation</strong></em> option is checked by default. It controls how HPE OpsRamp handles previously discovered resources by another integration module when it runs discovery again.</p> </blockquote> <ol start="11"> <li> <p>Click <strong>NEXT</strong> and <strong>select</strong> a collector profile. That is, the gateway collector appliance used to install the integration module and discover the Redfish – Server hardware resource.</p> </li> <li> <p>Click <strong>FINISH</strong> to complete the installation of the integration module and initiate the resource discovery and monitoring.</p> </li> <li> <p>The gateway collector appliance displays with the status <strong>CONFIGURED</strong> for the Redfish – Server integration module. After a few minutes, the status of the gateway collector appliance will change to <strong>RUNNING</strong>.</p> </li> </ol> <h3>Visualizing the components of the Redfish - Server</h3> <p>After a few iterations of discovery and monitoring of the newly installed Redfish Server integration module:</p> <ol> <li> <p>Navigate to <strong>Infrastructure > Search</strong> and select <strong>COMPUTE > Redfish – Server</strong>.</p> </li> <li> <p>Select either <strong>MANAGER</strong> or <strong>SYSTEM</strong> or <strong>LINUX</strong> in <strong>MORE</strong> option to view the sub-components of the compute device and the monitoring status (color indication is green which means the elements are now monitored):</p> </li> </ol> <p><img src="/img/integration-redfish-img7.png" alt="Visualizing the components of Redfish - Server" title="Visualizing the components of Redfish - Server"></p> <p>When the hybrid observability service connects to a Redfish-enabled system, it typically discovers and maps multiple components as separate resources:</p> <ul> <li>The <strong>Manager</strong> represents the baseboard management controller (BMC) — that is the iLO management controller of the HPE ProLiant server — used for out-of-band monitoring. The Hybrid observability service connects to the controller for inventory, control, and monitoring of the server hardware.</li> <li>The <strong>physical server resource</strong> to monitor system-level hardware such as the processor, memory, and network interfaces.</li> <li>The <strong>server with the operating system</strong> (Linux in our example) detected via the agentless SSH integration module previously installed.</li> </ul> <h3>Visualizing the metrics and monitoring templates</h3> <ol> <li> <p>From the <strong>Infrastructure > Search > COMPUTE > Redfish – Server</strong>, select a sub-component of the Redfish - Server to open the resource details page. For example, select the <strong>Compute System</strong> or the <strong>Manager</strong>. Click on the component. You will see multiple tabs with detailed information about the Redfish - Server sub-component, such as Overview, Metrics, Attributes, Components, and Inventory.</p> </li> <li> <p>Click on the <strong>Metrics</strong> tab to view the metric graphs. A graph is plotted for each metric that is enabled in the Monitoring Template automatically assigned to the sub-component.</p> </li> <li> <p>On the upper right side of the Metrics tab, click the <strong>Settings</strong> icon to <em>View Monitoring Configuration</em>, such as the monitored metrics, the thresholds, and the availability monitor.</p> </li> <li> <p>Navigate to <strong>ASSIGNED TEMPLATES</strong> to see the global monitoring template(s) assigned to the sub-component of the Redfish – Server.</p> </li> </ol> <p><img src="/img/integration-redfish-img7-bis2.png" alt="Visualizing the metrics and monitoring templates" title="Visualizing the metrics and monitoring templates"></p> <p>Based on your IT operational requirements, you can clone and customize the Global Monitoring Templates, then apply them to the Redfish - Server components as appropriate. Refer to <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">Part 3 of the series</a> to learn how to clone and apply a monitoring template to a resource.</p> <h3>Getting a real-time view of the Redfish – Server components</h3> <p>You can also explore <strong>Topology Maps</strong>, a visual representation of relationships and dependencies between your infrastructure resources:</p> <ul> <li>Go to <strong>Infrastructure > Topology Maps</strong> and select your Redfish - Server.</li> <li>You can also select a resource for a sub-component of the Redfish - Server from <strong>Infrastructure > Search > COMPUTE > Redfish – Server</strong>. Click the ellipsis (…) at the top right and select <strong>View Topology</strong>.</li> </ul> <p><img src="/img/integration-redfish-img8-bis.png" alt="Topology Maps for Redfish - Server" title="Topology Maps for Redfish - Server"></p> <p>When viewing <strong>Topology Maps</strong>, you can customize how much detail is shown and how deep the relationships go using <strong>View Settings</strong> (gear icon) on the top right and set the <strong>Depth</strong> and <strong>Layout</strong> options.</p> <p><img src="/img/integration-redfish-img9.png" alt="Depth and Layout of the Topology Map" title="Depth and Layout of the Topology Map"></p> <p>The <strong>depth</strong> setting controls how many levels of relationships you see in the topology view, and the <strong>Layout</strong> defines how resources and their relationships are visually arranged on the screen:</p> <p><img src="/img/integration-redfish-img10.png" alt="Topology Map with depth and layout" title="Topology Map with depth and layout"></p> <h3>Visualizing monitoring data in a dashboard</h3> <p>You can use the <strong>dashboards</strong> (classic dashboard and modern dashboard 2.0). A dashboard is a collection of widgets and tiles that present visualizations of your environment’s monitoring data, helping you gain a greater understanding of the state of the infrastructure.</p> <h2>Discovering and monitoring of HPE Storage Array</h2> <p>HPE OpsRamp provides integration modules to discover and monitor storage arrays from many vendors, including HPE, Dell, Hitachi, NetApp, Vast and more.</p> <p>The <a href="https://glp.docs.opsramp.com/integrations/storage/hpe/hpe-alletra/">HPE Alletra MP/9000 integration module</a> is used for discovering, monitoring and managing HPE Alletra MP/9000 storage systems. This module helps you pull detailed health, performance, and inventory data from the storage arrays (disks, controllers, fans, capacity utilization, input/output operations per second, latency, throughput and so on).</p> <p>Once discovered, storage monitors are <strong>automatically</strong> applied to the resource via predefined Global Monitoring Templates and Global Device Management Policies, to manage the health and status of the storage system components.</p> <h3>Installing the storage array integration module</h3> <p>The storage array integration module installation process is very similar to the Redfish – Server integration module:</p> <ol> <li> <p>Select the <strong>client account</strong>.</p> </li> <li> <p>In the navigation bar, go to <strong>Setup > Account > Integrations</strong> and click <strong>+ADD</strong> to search for <strong>Storage</strong> category.</p> </li> <li> <p>Choose to install <strong>HPE Alletra MP/9000 Series</strong> by selecting <strong>ADD</strong>.</p> </li> <li> <p>Select the <strong>application version</strong> and select <strong>+ADD</strong> to input basic account information such as:</p> <ul> <li>The <strong>Name</strong> for the configuration.</li> <li>The storage array type (Alletra MP or Alletra 9000).</li> <li>Check the <strong>Is Secure</strong> checkbox.</li> <li>Specify the <strong>IP address/Host name</strong> of the storage array that should be accessible from the gateway collector appliance.</li> <li>Specify the <strong>WSAPI port</strong> (port 443) and <strong>SSH port</strong> (port 22).</li> </ul> </li> <li> <p>Select existing credentials that apply to the storage array or click <strong>+ADD</strong> to create the Credentials for the storage array account. Specify the credentials name, a description, the username, and password.</p> </li> <li> <p>Although optional, it is recommended to select:</p> <ul> <li><strong>App Failure Notification</strong> to be notified by an event or an alert when availability is impacted due to an issue within the HPE Alletra MP/9000 storage environment.</li> <li><strong>Alert Polling</strong> that allows HPE OpsRamp to check for and collect alerts generated by the storage array.</li> <li><strong>API Timeouts</strong> to define the maximum amount of time HPE OpsRamp will wait for a response from the storage array when making an API call to fetch data such as alerts, events, metrics, or device information.</li> </ul> </li> <li> <p>In <strong>RESOURCE TYPE</strong>, check <strong>ALL</strong> or specify the hardware components to be discovered.</p> </li> <li> <p>The last step is to define a <strong>Discovery Schedule</strong>. You can select a minute, hourly, daily, weekly, or monthly schedule.</p> </li> <li> <p>Select <strong>ADD</strong> to save the configuration of the integration module.</p> </li> <li> <p>Click <strong>NEXT</strong> and <strong>select</strong> a collector profile. That is, the gateway collector appliance that will be used to install the integration module and discover the storage array resource.</p> </li> <li> <p>Click <strong>FINISH</strong> to complete the installation of the integration module and initiate the resource discovery.</p> </li> <li> <p>The gateway collector appliance is displayed with the status <strong>CONFIGURED</strong> for the storage array integration module. After a few minutes, the status of the gateway collector appliance will change to <strong>RUNNING</strong>.</p> </li> </ol> <h3>Visualizing the components of the storage array</h3> <p>After a few iterations of discovery and monitoring of the newly installed storage array integration module:</p> <p>Navigate to <strong>Infrastructure > Search</strong> and select <strong>STORAGE > HPE Alletra MP/9000 Series</strong> to check the monitoring status (color indication is green) of the storage array:</p> <p><img src="/img/integration-alletramp-img5.png" alt="Visualizing the components of the storage array" title="Visualizing the components of the storage array"></p> <p>The <strong>HPE Alletra Storage System</strong> is the top-level resource being discovered and monitored. It refers to the entire managed storage array instance — a physical and logical collection of components that operate together to deliver enterprise-grade storage services. Components include controllers, disks, virtual storage pool, volumes, network interfaces, chassis, cache modules, and so on.</p> <p>You can select each component of the storage array to visualize its monitoring status, attributes, and metrics.</p> <h3>Checking the Monitoring Templates assigned to the storage array</h3> <ol> <li> <p>Select a component of the storage array to open the resource details page.</p> </li> <li> <p>Click on the <strong>Metrics</strong> tab to view the metric graphs.</p> </li> <li> <p>On the upper right side of the <strong>Metrics</strong> tab, click the <strong>Settings</strong> icon to view monitoring configurations such as the monitored metrics, the thresholds, and the availability monitor.</p> </li> <li> <p>Navigate to <strong>ASSIGNED TEMPLATES</strong> to see the global monitoring templates assigned to the component of the storage array.</p> </li> </ol> <p>Based on your IT operational requirements, you can clone the Global Monitoring Templates and customize them, then apply them to the storage array components as appropriate.</p> <blockquote> <p><strong>Important Note:</strong> For the HPE Alletra MP/9000 storage, although a Global Monitoring Template is automatically applied to the <strong>HPE Alletra Storage System</strong> component, <strong>no availability monitor</strong> is assigned. This feature will be made available in an upcoming HPE OpsRamp release. Meanwhile, to enable monitoring of the storage system component, you need to clone the monitoring template assigned to the <strong>HPE Alletra Storage System</strong>. Then you must customize it to enable one or two availability monitors for the metrics that are more important for the resource. Refer to <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">part 3 of the series</a> to learn how to clone and apply a monitoring template to a resource.</p> </blockquote> <h3>Getting a real-time view of the storage array components</h3> <p>You can explore <strong>Topology Maps</strong> to get a graphical, real-time view of the storage array components, their relationships, and dependencies.</p> <p>Navigate to <strong>Infrastructure > Search</strong> and select the storage system, click the ellipsis (…) at the top right, and select <strong>View Topology</strong>. Use the <strong>View Settings</strong> (gear icon) on the top right to set the <strong>Depth</strong> and <strong>Layout</strong> of the graphical representation.</p> <p>Finally, you can use the dashboards (classic dashboard and modern dashboard 2.0) that represent monitoring data of your environment.</p> <h2>Dashboard</h2> <p>Once monitoring tools have collected metrics, the next step in creating actionable insights is to visualize them in a dashboard. A dashboard is a collection of charts based on tiles and widgets for visualizing metrics data measured over intervals of time. Hybrid observability service powered by HPE OpsRamp leverages the open-source query language, Prometheus Query Language (PromQL), for dashboard creation.</p> <p>The purpose of using and creating customizable dashboards is to easily identify anomalies so team members can identify them and troubleshoot issues quickly.</p> <p>You can choose from the classic dashboard or dashboard 2.0. HPE <strong>recommends leveraging the modern dashboard 2.0</strong> for its advanced capabilities.</p> <p>Dashboard 2.0 allows you to:</p> <ul> <li>Create a dashboard with common chart types for observability such as line charts, bar charts, value charts, list charts, gauge charts, honeycomb charts, or pie charts.</li> <li>Leverage a set of predefined, <strong>curated</strong> dashboards. They are pre-built, use-case-specific dashboards that give IT teams immediate visibility into key areas of IT operations without needing to build dashboards.</li> <li>Group related dashboards by function, project, or team into the default collection (<strong>My Dashboards</strong>) or create a new collection.</li> <li>Copy a curated dashboard and customize it according to your operational requirements.</li> <li>Designate a dashboard as the default dashboard.</li> <li>Export a dashboard as a JSON file and import it into another environment or client account for your partner domain.</li> </ul> <p>To learn more about dashboards, see the <a href="https://glp.docs.opsramp.com/platform-features/feature-guides/dashboards/">HPE OpsRamp Dashboards documentation</a>.</p> <h3>Creating a dashboard</h3> <p>As a <em>partner administrator</em> you can create and manage dashboards. In this example, I’ll show you the sequence of steps to create a new dashboard 2.0 in the default collection for the client account <em>DreamCompany</em> to monitor and manage the discovered Redfish - Server.</p> <ol> <li> <p>Select the <strong>client account</strong>, navigate to <strong>Dashboards</strong>, and click <strong>Dashboard</strong>.</p> </li> <li> <p>Click the menu (<strong>three stacked horizontal lines</strong>), also referred to as <em>hamburger icon</em>, on the left side of the dashboard.</p> </li> <li> <p>Click <strong>+CREATE DASHBOARD</strong>.</p> </li> <li> <p>Specify a dashboard name for the default collection <strong>My Dashboards</strong>. You can specify a new collection name.</p> </li> <li> <p>Click <strong>CREATE</strong> to start designing the dashboard by adding tiles and widgets.</p> </li> </ol> <p><img src="/img/create-dashboard-img2.png" alt="Creating a dashboard 2.0" title="Creating a dashboard 2.0"></p> <ol start="6"> <li> <p>Click <strong>CREATE TILE</strong> or <strong>+</strong> on the toolbar.</p> </li> <li> <p>Select <strong>Text &#x26; Images</strong> tile to create a header.</p> <ul> <li>Select <strong>Header</strong> tab.</li> <li>You can set the font, the size, and the background color of the header.</li> <li>Click <strong>ADD TILE</strong>.</li> </ul> </li> <li> <p>Click <strong>+</strong> in the toolbar to add another tile. Here, I will add a <strong>Metric</strong> tile to visualize metrics coming from the Redfish - Server:</p> <ul> <li>Select <strong>Build my own</strong> tab.</li> <li>In the <strong>DATA</strong> section, click <strong>+QUERY</strong> and select the metric that is important for you. For example, I selected the <strong>ComputeSystem Average CPU</strong> and the <strong>Line/Bar</strong> chart. You can also select the duration over which the selected metric data is displayed and calculated, for example, last 1 hour or last 24 hours.</li> <li>Optionally, you can define <strong>filters and operations</strong> to help refine what data is shown and how it’s calculated.</li> <li>Click the optional <strong>Legend</strong> icon and enter double curly brackets <strong>{{</strong> in the <strong>Query a Legend</strong> field to see a list of options. For example, I specify the <em>name</em> and <em>rootResourceIp</em> of the resource from the list of options to identify the resource in the chart. Click <strong>DONE.</strong></li> </ul> </li> </ol> <p><img src="/img/create-dashboard-img7.png" alt="Metric tile" title="Metric tile"></p> <p><img src="/img/create-dashboard-img8.png" alt="Add a legend" title="Add a legend"></p> <ol start="9"> <li>In the <strong>VISUALIZATION</strong> section, you can specify a header for the tile. For example, <em>Avg CPU Utilization</em>. You can select <strong>Line</strong> or <strong>Bar</strong> as the graph type and its color.</li> </ol> <p><img src="/img/create-dashboard-img9.png" alt="Select header and graph for the metric tile" title="Select header and graph for the metric tile"></p> <ol start="10"> <li>You can also define <strong>axis labels</strong> for the chart and <strong>thresholds</strong>. Thresholds define <strong>value ranges</strong> for a metric and assign them a <strong>color/status</strong> (OK, Warning, Critical) to reflect performance or health. They help visually flag performance degradation or anomalies in the metrics and take corrective actions.</li> </ol> <p><img src="/img/create-dashboard-img10.png" alt="Define axis labels and thresholds" title="Define axis labels and thresholds"></p> <ol start="11"> <li> <p>Click <strong>SAVE</strong> or <strong>CREATE</strong> to save the tile.</p> </li> <li> <p>To resize a tile, hover the mouse over the bottom-right corner of the tile in the dashboard. You will see a resize handle shown by a diagonal arrow. Click and drag the corner to increase or decrease the tile size both horizontally and vertically.</p> </li> <li> <p>To edit a tile, hover the mouse to the top right of the tile, click the ellipsis (…), and select <strong>Edit</strong>.</p> </li> <li> <p>To move a tile, hover the mouse over the top of the tile, and position the tile on your dashboard.</p> </li> <li> <p>You can add other tiles to the dashboard for additional metrics, resource tile, alert tile, and so on.</p> </li> </ol> <p>Here is an example of a basic dashboard for monitoring the status of the Redfish - Server for the client account <em>DreamCompany</em>:</p> <p><img src="/img/create-dashboard-img14.png" alt="Example of dashboard to monitor Redfish - Server" title="Example of dashboard to monitor Redfish - Server"></p> <h3>Importing a dashboard</h3> <p>The <strong>Import Dashboard</strong> feature allows you to upload and reuse dashboard configurations, usually in JSON format. This helps replicate dashboards, share dashboards between teams, and quickly deploy standard monitoring views.</p> <ol> <li> <p>Select the <strong>client account</strong>, navigate to <strong>Dashboards</strong> and click <strong>Dashboard</strong>.</p> </li> <li> <p>Click the menu icon (three stacked horizontal lines) on the left side.</p> </li> <li> <p>Click <strong>IMPORT DASHBOARD</strong> and upload a JSON dashboard file for the default collection of <strong>My Dashboards</strong>. Click <strong>IMPORT</strong> to import the dashboard JSON file.</p> </li> <li> <p>The imported dashboard appears in the collection, and you can edit each tile to adjust the data to be collected the same way you created a new dashboard. You can also add, remove tiles, move tiles, and adjust the size of the tiles.</p> </li> </ol> <h2>Summary</h2> <p>The hybrid observability solution, powered by HPE OpsRamp Software, discovers the compute, networks, and storage infrastructure, the applications, and workloads they host, and their dependencies. It then observes and monitors the health, performance, and capacity events, metrics, logs, traces, and network flows providing customers with true end-to-end visibility.</p> <p>This blog series walked you through the sequence of steps to provision and activate the hybrid observability service in HPE GreenLake workspace, and set up the service to discover, monitor, and observe the health, performance, and availability of agentless SSH-enabled systems, and physical infrastructure devices included in the HPE GreenLake Flex Solutions contract.</p> <p>There’s also a series of video tutorials on the <a href="https://developer.hpe.com/greenlake/hybrid-observability-flex-solutions/home/">Hybrid observability in HPE GreenLake Flex Solutions landing page</a> that walks you through the contents described in this blog series for readers who prefer an audio-visual learning experience.</p> <p>To resolve issues with HPE GreenLake Flex Solutions or HPE hybrid observability service powered by HPE OpsRamp, contact the support team. While logged in to your HPE GreenLake workspace:</p> <ol> <li>Click <strong>Help &#x26; Support</strong>.</li> <li>Under <strong>Help</strong>, select <strong>OpsRamp</strong> or <strong>HPE GreenLake Flex Solutions</strong>.</li> <li>Click <strong>Create New Case</strong>.</li> </ol>https://developer.hpe.com/hybrid-observability-service-–-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/https://developer.hpe.com/hybrid-observability-service-–-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/Mon, 26 May 2025 15:35:32 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/">my previous blog post</a>, I introduced the steps to install and configure a gateway collector appliance for a specific tenant client account in the hybrid observability service powered by HPE OpsRamp Software in HPE GreenLake Flex Solutions.</p> <p>The gateway collector appliance is a prerequisite to enable the discovery of infrastructure devices in HPE GreenLake Flex Solutions before they can be monitored.</p> <p>Continuing from the second part of this series, I’ll now explore:</p> <ul> <li>How to deploy an integration module to discover an agentless SSH-enabled server running Linux as the operating system.</li> <li>How to apply a monitoring template to <em>non SDK resources</em> such as the agentless SSH resource to monitor it.</li> </ul> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">the last Part</a> of the blog series, I’ll explore the observability of physical devices and their monitoring using dashboards to visualize the collected metrics in charts.</p> <h2>Applying integration modules and monitoring configuration to discover and monitor resources</h2> <h3>Integration modules</h3> <p>Once the gateway collector appliance is installed and configured to communicate with the hybrid observability service in the cloud, the next step is to install <strong>Integration modules</strong> to discover IT infrastructure devices. For example, compute, storage, and network equipment. Further, integration modules enable data exchange between the service in the cloud and IT infrastructure devices included in the HPE GreenLake Flex Solutions.</p> <p>Once a resource is discovered, you can apply a <strong>Monitoring Template</strong> to periodically track the resource's performance, availability, and health metrics, based on configured metrics.</p> <h3>Monitoring Templates</h3> <p>Monitoring is a method for periodically querying IT resources and forwarding resource metrics to the service for processing and analysis. <a href="https://glp.docs.opsramp.com/solutions/monitoring/template/">Monitoring Templates</a> are used to apply standardized monitoring configurations to resources. These templates define the <strong>metrics</strong> and <strong>thresholds</strong> that will be monitored.</p> <p>The hybrid observability service powered by HPE OpsRamp comes with a library of pre-configured <strong>Global Monitoring Templates</strong> that cover popular resource types. These built-in Global Monitoring Templates apply <strong>automatically</strong> through <strong>Device Management Policies</strong> to a wide range of resource types such as Operating System (Linux, Windows), compute servers, network devices, storage arrays, applications, virtualization and databases.</p> <h3>Device Management Policies</h3> <p><a href="https://glp.docs.opsramp.com/platform-features/feature-guides/resource-management/">Device Management Policies</a> are rules that <strong>automate</strong> managing and monitoring resources after discovery.</p> <p>There are predefined <strong>Global Device Management Policies</strong> available for common platforms and device types. These built-in policies are designed to help you <strong>automatically</strong> bind the Monitoring Templates to discovered resources based on matching criteria (for example, <em>Native Type = Linux</em>, <em>Operating System contains CentOS</em>, <em>Make = HPE</em>).</p> <p>There might be some scenarios where you need to define a Monitoring Template and apply it either <strong>manually</strong> to a specific resource or <strong>automatically</strong> via a Device Management Policy to monitor a set of resources of the same type. For example, discovered agentless resources (also referred to as Non SDK resources) require a Monitoring Template to be defined and applied manually or via a Device Management Policy to a single resource or a set of resources.</p> <blockquote> <p><strong>Note:</strong> You can navigate to <strong>Setup > Setup > Clients</strong> to <strong>edit</strong> the <em>client account</em> and check the <strong>Enable Global Policies</strong> checkbox to automatically apply a Global Device Monitoring Policy, <strong>if one exists</strong>, for <strong>Non SDK resources</strong> such as the SSH-enabled system. In this post, I have <strong>not</strong> enabled this option for the <em>client account</em> to explore how to clone a built-in Global Monitoring Template, customize it, and assign it to the Linux system to enable its monitoring.</p> </blockquote> <h2>Agentless discovery and monitoring of SSH-enabled devices</h2> <p>Let us start by deploying an Integration module and creating a Monitoring Template to discover and monitor an <a href="https://glp.docs.opsramp.com/integrations/os/linux-os-agentless-discovery/">agentless SSH-enabled system</a>.</p> <p>The hybrid observability service supports agentless monitoring. Agentless monitors use the <strong>gateway collector appliance</strong> to discover resources via SSH and monitor IT infrastructure agentless resources to track their health, performance, and availability.</p> <p>The SSH Agentless Integration module discovers Linux/Unix-based systems <strong>without installing an agent</strong>, by securely connecting to the device over SSH via the gateway collector appliance. The Monitoring Template then needs to be assigned to the target agentless system for monitoring.</p> <h3>Requirements:</h3> <ul> <li>A gateway collector appliance is installed and registered.</li> <li>The gateway collector appliance has network access (through SSH protocol – typically on port 22) to connect to the target Linux server.</li> <li>IP address of the target SSH-based machine(s).</li> <li>SSH credentials (username and password) of the target machine.</li> <li>A Monitoring Template and Device Management Policy to monitor the target resource through the gateway collector appliance and collect data for monitoring the metrics and resource availability.</li> </ul> <h3>Install the Linux agentless integration module in a client account</h3> <p>As Partner Administrator, let’s install an <strong>agentless SSH Integration module</strong> for the <em>client account</em> to discover an HPE ProLiant server added in the HPE GreenLake workspace devices inventory. This compute server runs the Linux operating system. The Linux OS agentless SSH Integration module facilitates agentless discovery of SSH-enabled devices using the gateway collector appliance. Here is the procedure:</p> <ol> <li> <p>To discover a compute resource, select the <strong>client account</strong>, then select <strong>Setup</strong> in the navigation bar, and select <strong>Account</strong> in the drop-down menu to navigate to the account details page.</p> </li> <li> <p>Select the <strong>Integrations</strong> tile to access the Integrations App store and click <strong>+ADD</strong> to display all the integration applications available for the <em>client account</em>. If this is your first integration module you want to install, click <strong>+ADD</strong> directly from the Integrations tile.</p> </li> <li> <p>Filter the list of available integrations by a variety of <strong>Categories</strong> or use the <strong>Search</strong> feature. For agentless SSH-enabled devices, you can filter through category <strong>OS</strong> or in the Search field, type <strong>SSH</strong>.</p> </li> </ol> <p><img src="/img/agentless-ssh-integration-img2.png" alt="Filter by category or Search" title="Filter by category or Search"></p> <ol start="4"> <li> <p>Choose the integration module to install the <strong>Linux OS – Agentless (SSH)</strong> by selecting <strong>ADD</strong>.</p> </li> <li> <p>Navigate to the agentless-SSH configuration page. The first step in the process is to input <strong>Basic Account Information</strong>. Select <strong>+ADD</strong> at the right of the <strong>Configuration</strong> tab, then enter a <strong>Name</strong> for the configuration. For example, <em>Name = SSH Compute integration</em>.</p> </li> <li> <p>In the <strong>Host Name/IP Address</strong> box, enter an IP address, or the hostname of the target agentless system.</p> </li> <li> <p>Select the <strong>SSH Credential</strong> box and click <strong>+ADD</strong> to create new credentials to discover the agentless server. Specify:</p> <ul> <li>The credential name and a brief description.</li> <li>The authentication type (typically Password) and appropriate credentials (username and password).</li> <li>The SSH port (typically port 22).</li> <li>Check the box <strong>Secure</strong>.</li> <li>Finally, click <strong>ADD</strong> to create the SSH credential.</li> </ul> </li> </ol> <blockquote> <p><strong>Note:</strong> You can also select existing SSH credentials if one already exists and that applies to your resource.</p> </blockquote> <p><img src="/img/agentless-ssh-integration-img3.png" alt="SSH Credential" title="SSH Credential"></p> <ol start="8"> <li> <p>In <strong>PERFORM ACTIONS</strong>, select the <strong>Manage Device</strong> checkbox.</p> </li> <li> <p>Next, define a <strong>Discovery Schedule</strong>. The schedule ensures that all resources and components inventory are in sync between the hybrid observability service and the agentless (SSH) server. You can select hourly, daily, weekly, or monthly schedule.</p> </li> <li> <p>Now that all the configuration details are entered, select <strong>ADD</strong> to save the configuration of the integration module.</p> </li> <li> <p>Click <strong>NEXT</strong> and <strong>select</strong> a collector profile, the gateway collector appliance that will be used to install the integration module and discover the agentless resource.</p> </li> <li> <p>Click <strong>FINISH</strong> to complete the installation of the integration module and initiate the resource discovery.</p> </li> <li> <p>Select the Integrations tile, and click on the <strong>Linux OS – Agentless (SSH)</strong> tile to check the status of the agentless SSH integration module:</p> <ul> <li>The gateway collector appliance displays with the status <strong>RUNNING</strong>.</li> <li>Select the gateway collector appliance to check the status of the agentless SSH integration module. The discovery status should be <strong>Completed</strong>.</li> <li>Based on your discovery schedule, the value for the devices might be set to <strong>0</strong>. In this case, select the ellipsis (...) on the right and select <strong>Discover</strong> to discover the device manually.</li> <li>Click <strong>DONE</strong>.</li> <li>Connect again to the SSH integration module and select the gateway collector appliance. You should see now a device discovered in <strong>Devices</strong> tab. If a device is not discovered, ensure the prerequisites are fulfilled: the gateway collector appliance is installed and registered successfully, and it can connect to the target agentless SSH-enabled server.</li> <li>Click on the <strong>1</strong> for the devices to visualize the resource information for the Linux server. Notice the status is <strong>UNDEFINED</strong> (color indication brown). The reason is that in our case, <strong>no</strong> default Global Monitoring Template is automatically assigned to an agentless SSH-enabled system. In the next step, you will define a monitoring template and assign it to the agentless device.</li> </ul> </li> </ol> <p><img src="/img/agentless-ssh-integration-img6.png" alt="Resource information" title="Resource information"></p> <ol start="14"> <li>Select the server's name to collect attributes of the Linux server device such as the <em>Operating System (CentOS)</em>, <em>Native type (Linux)</em> and <em>Make (HPE)</em>. You will need this information in the next step to configure the filter criteria for the monitoring policy of the agentless system.</li> </ol> <h2>Define a monitoring template and assign it to agentless devices</h2> <p>When the option <strong>Enable Global Policies</strong> for non SDK resources is not enabled for the <em>client account</em>, and an agentless system is discovered through the gateway collector appliance using the appropriate Integration module, you need to define and apply a <strong>Monitoring Template</strong> for the gateway collector appliance to monitor the device for a specific client account.</p> <p>The built-in Global Monitoring Templates are <strong>not editable</strong>. <strong>Copy</strong> an appropriate built-in Global Monitoring Template and customize the <em>metrics</em>, <em>thresholds</em>, and <em>resource availability monitor</em> for the most important metric(s) for the resource to meet the IT operational needs for the client account.</p> <blockquote> <p><strong>Note:</strong> The <a href="https://glp.docs.opsramp.com/solutions/availability/resource-availability/">Resource Availability documentation</a> refers to the uptime and operational status of a monitored resource. It tracks whether the resource is running or experiencing downtime based on the metrics with the availability monitor applied.</p> </blockquote> <p>You can then <strong>manually</strong> assign the customized Monitoring Template to a particular resource, or via a <strong>Device Management Policy</strong> to automatically bind your Monitoring Template to the discovered agentless devices.</p> <p>Let’s see how to define a Monitoring Template and the methods to assign it to the discovered agentless Linux device(s).</p> <p>The gateway collector appliance discovered the agentless SSH system. It will monitor the agentless SSH system using a Monitoring Template. Therefore, let’s select a Global Monitoring Template with a name that contains the word <em>gateway</em> and <strong>clone</strong> this template:</p> <ol> <li> <p>Select a <strong>client account</strong>.</p> </li> <li> <p>Go to <strong>Setup > Setup > Monitoring > Templates</strong> and click <strong>Advanced</strong>.</p> </li> </ol> <blockquote> <p><strong>Note:</strong> This feature will be made available under <strong>Setup > Account > Monitoring</strong> in an upcoming HPE OpsRamp releases.</p> </blockquote> <ol start="3"> <li>Search for Scope <em>Global Templates</em> and enter <em>gateway</em> or <em>gateway - linux</em> as the Template Name. Then click <strong>Search</strong> at the bottom of the <strong>Advanced</strong> search.</li> </ol> <p><img src="/img/agentless-ssh-monitoring-img1.png" alt="Search Global Monitoring Template for gateway - linux" title="Search Global Monitoring Template for gateway - linux"></p> <ol start="4"> <li>Select the <strong>latest version</strong> of the Global Monitoring Template <strong>Gateway - Linux OS Performance Remote Monitoring</strong> available. You can click the arrow on the left of the template to visualize the metrics and thresholds.</li> </ol> <p><img src="/img/agentless-ssh-monitoring-img2.png" alt="Metrics and thresholds of the monitoring template" title="Metrics and thresholds of the monitoring template"></p> <ol start="5"> <li> <p>Select the <strong>Template</strong> checkbox. Click <strong>Copy</strong> to clone the template and click <strong>Yes</strong> to duplicate the template.</p> </li> <li> <p>Select the <strong>client account</strong>, give a <strong>Name</strong> to your template. Use the <em>client account</em> name in the <em>Template Name</em> to identify the template for a particular client account easily. For example, <em>DreamCompany-Agentless-Linux-Performance</em>. Click <strong>Save</strong>.</p> </li> <li> <p>To list the Monitoring Templates for the <em>client account</em>, go to <strong>Monitoring > Templates > Advanced Search</strong>, select <strong>Client Templates</strong> for the <em>Scope</em>. You can now customize the template to adjust some parameters such as the <em>tags</em>, metrics such as the <strong>warning</strong> or <strong>critical thresholds</strong>, and the <strong>availability monitor</strong> per your IT operational requirements.</p> </li> <li> <p>To monitor the resource, enable an <strong>Availability Monitor</strong> for a metric (for example, <em>os.uptime</em>).</p> <ul> <li>Select the cloned Monitoring Template and click <strong>Edit</strong> at the top right of the template.</li> <li>Click <strong>Next</strong> at bottom of the page and select the metric that is more important for the resource by clicking on the pencil to the right of the metric.</li> <li>Check the <strong>Apply Availability Monitor</strong> checkbox and click <strong>Update</strong>.</li> <li>You can also modify the warning and critical thresholds of any metric per your operational requirements.</li> </ul> </li> </ol> <p><img src="/img/agentless-ssh-monitoring-img4.png" alt="Apply Availability Monitor" title="Apply Availability Monitor"></p> <ol start="9"> <li>Finally, click <strong>Save</strong> to save the Monitoring Template.</li> </ol> <p>You can now assign the Monitoring Template to the agentless resource either manually or via a Device Management Policy.</p> <p>You may want to assign the Monitoring Template manually if you have a single resource. If you are planning to monitor a set of resources of the same type, it is recommended that you apply the Monitoring Template via a <strong>Device Management Policy</strong>. The policy will automatically apply the resource policy to the resources based on a set of filter criteria you specify in the policy.</p> <p>Let’s walk you through the two methods to apply the Monitoring Template to a resource:</p> <p><strong>Method 1 — Manual assignment to a particular resource</strong></p> <ol> <li> <p>Select the <strong>client account</strong> and go to <strong>Infrastructure > Search</strong>.</p> </li> <li> <p>Select your Linux server.</p> </li> <li> <p>Check the checkbox of your server, click <strong>ACTIONS</strong> on the right, and select <strong>Assign Templates</strong>.</p> </li> <li> <p>Search for the <strong>Monitoring Template</strong> you have just created, check the box for the Monitoring Template, and click <strong>ASSIGN TEMPLATE</strong> button.</p> </li> </ol> <p><strong>Method 2 — Automatic assignment to a set of resources via a Device Management Policy</strong></p> <ol> <li>Select the <strong>client account</strong> and from the navigation bar go to <strong>Setup > Setup > Resources > Device Management Policies</strong>.</li> </ol> <blockquote> <p><strong>Note:</strong> This feature will be made available under <strong>Setup > Account > Monitoring</strong> in an upcoming HPE OpsRamp releases.</p> </blockquote> <ol start="2"> <li> <p>If it is the first time you want to create a Device Management Policy, click <strong>Create New</strong>. If you have already created policies, click the option <strong>+Add</strong> to create a new Device Management Policy for the agentless SSH system. The Device Management Policy page is displayed.</p> </li> <li> <p>Ensure the <strong>Client</strong> option is selected for the <strong>Scope</strong> and the appropriate <strong>client account</strong> has been selected. Give a policy <strong>Name</strong> (for example <em>Agentless Linux Monitoring Policy</em>) and ensure the <strong>Add members automatically using filter criteria</strong> checkbox is checked.</p> </li> <li> <p>Select one or more <strong>Filter Criteria</strong> that apply to your agentless device(s) and click <strong>Show Matching Members</strong> to ensure the resources matching the filter criteria are the expected devices. For example, you can filter on <em>Native Type</em> and <em>Make</em> of the device as shown here:</p> </li> </ol> <p><img src="/img/agentless-ssh-monitoring-img7.png" alt="Filter criteria for the device management policy" title="Filter criteria for the device management policy"></p> <ol start="5"> <li> <p>In the <strong>Perform Actions</strong> section, select the <strong>Assign Monitoring Templates</strong> checkbox. A list of available monitoring templates is listed. Select the Monitoring Template you previously created and click the right arrow to move it to the <strong>Assigned Monitoring Templates</strong> column on the right.</p> </li> <li> <p>Click <strong>Save &#x26; Run Now</strong> to assign the policy to the resources matching the filter criteria. The policy will apply to all resources that match the filter criteria you specified in the policy.</p> </li> <li> <p>After a few minutes, check the agentless SSH device in <strong>Infrastructure > Search</strong> and select the device to check the status and the metrics. The status should now be <strong>UP</strong> (color indication is green) and you can start visualizing the metrics by navigating to METRICS tab.</p> </li> </ol> <p><img src="/img/agentless-ssh-monitoring-img10.png" alt="Check status and metrics of the SSH agentless device" title="Check status and metrics of the SSH agentless device"></p> <h2>Summary</h2> <p>This blog post walked you through the steps to set up the hybrid observability service powered by HPE OpsRamp for an agentless SSH-enabled server. With this integration service enabled, you can discover, monitor, and observe the health, performance, and availability of an agentless SSH-enabled system.</p> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">my last part</a> of this blog series, I’ll dive into the observability of a physical server and a storage array as we continue the journey to set up the hybrid observability service to discover and monitor, through dashboards, physical IT infrastructure devices included in the HPE GreenLake Flex Solutions contract.</p> <p>There’s also a series of video tutorials on the <a href="https://developer.hpe.com/greenlake/hybrid-observability-flex-solutions/home/">Hybrid observability in HPE GreenLake Flex Solutions landing page</a> that walks you through the contents described in this blog series for readers who prefer an audio-visual learning experience.</p> <p>To resolve issues with HPE GreenLake Flex Solutions or hybrid observability service powered by HPE OpsRamp, contact the support team. While logged in to your HPE GreenLake workspace:</p> <ol> <li>Click <strong>Help &#x26; Support</strong>.</li> <li>Under <strong>Help</strong>, select <strong>OpsRamp</strong> or <strong>HPE GreenLake Flex Solutions</strong>.</li> <li>Click <strong>Create New Case</strong>.</li> </ol>https://developer.hpe.com/hybrid-observability-service-–-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/https://developer.hpe.com/hybrid-observability-service-–-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/Fri, 23 May 2025 15:34:55 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-1-provisioning-and-activation-in-hpe-greenlake-flex-solutions/">my previous blog post</a>, I covered the steps to provision and activate the hybrid observability service powered by HPE OpsRamp Software in a HPE GreenLake cloud workspace as part of the HPE GreenLake Flex Solutions. You must complete this initial step before configuring the hybrid observability service to enable the observability of your IT infrastructure resources.</p> <p>This is Part 2 of the blog series, and it explores the <strong>initial configuration</strong> of the hybrid observability service to <strong>enable the discovery</strong> of resources before they can be monitored, and metrics data collected from the resources. As requirements to enable the discovery of physical resources included in your HPE GreenLake Flex Solutions contract, I’ll start setting up the hybrid observability service by:</p> <ul> <li>Creating a <strong>client account</strong>.</li> <li>Managing user accounts and permissions for your organization.</li> <li>Installing and registering a <strong>gateway collector appliance</strong> in a virtual environment.</li> </ul> <p>In <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">Part 3</a> and <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">Part 4</a> of the blog series, I’ll explore how to enable the observability of IT infrastructure devices included in the HPE GreenLake Flex Solutions by deploying integration modules and assigning monitoring configurations to a set of resources.</p> <h2>Creating a client account</h2> <p>When the hybrid observability service powered by HPE OpsRamp is deployed in HPE GreenLake cloud by the <strong>Workspace Administrator</strong>, the service uses a <strong>multi-tenant model</strong> associated with a <strong>partner account</strong>. A partner account is the <strong>master</strong> tenant, including Clients who each have a <em>client account</em>. A <em>Workspace Administrator</em> is the user who created the workspace and has this role assigned by default, or invited users can get this role assigned by the <em>Workspace Administrator</em>.</p> <p>In a multi-tenant model, a <em>client account</em> is a tenant of a partner account. It represents a monitoring and management instance for an individual organization, a business unit, or a specific customer environment.</p> <blockquote> <p><strong>Note:</strong> By default, a <em>client account</em> is created with the name of the HPE GreenLake workspace. You may want to create additional <em>client accounts</em> for an individual organization, business unit, or customer environment.</p> </blockquote> <p>To start setting up the environment, log in to the service using an account provisioned with the <em>Partner Administrator</em> role. The <em>Workspace Administrator</em> who provisioned the HPE OpsRamp service in the HPE GreenLake cloud has this role assigned by default.</p> <blockquote> <p><strong>Note:</strong> The other users with the <em>OpsRamp Access</em> role granted in the workspace will have the role <em>GLP Invited Partner User</em> assigned in the service.</p> </blockquote> <p>To create a new client account:</p> <ol> <li>From the top navigation menu of HPE OpsRamp user interface, select <strong>Setup > Setup</strong>.</li> </ol> <blockquote> <p><strong>Note:</strong> This feature will be made available under <strong>Setup > Account</strong> in an upcoming HPE OpsRamp release.</p> </blockquote> <ol start="2"> <li> <p>Select <strong>Clients</strong> on the left navigation bar and click <strong>+Add</strong> to add a new client account. If this is the first client of your partner account, click <strong>Create New</strong>.</p> </li> <li> <p>Enter the <em>Client Name</em>, <em>Address</em>, <em>Country</em>, and <em>Time Zone</em>. Contact email is also recommended. If you specify a client state, the length must be 3 characters maximum. Then click <strong>Next</strong>.</p> </li> <li> <p>Keep the default selection for <strong>Product Package</strong> entries for Hybrid Discovery and Monitoring, Event and Incident Management, Remediation and Automation.</p> </li> <li> <p>Click <strong>Finish</strong>.</p> </li> <li> <p>Once the new client account is created, select the new <em>client account</em> from the list of <strong>Clients</strong>.</p> </li> <li> <p>To see the new client and continue configuring the service, click <strong>OpsRamp</strong> icon on the upper left.</p> </li> </ol> <blockquote> <p><strong>Note:</strong> Alternatively, you can log out from the service and log in again from your HPE GreenLake workspace, and then select the <strong>Partner</strong> domain for your organization. You should see the new client account, in our example, <strong>DreamCompany</strong>, on the list of clients. Select the <em>client account</em> to continue with the configuration of the hybrid observability service for the client account.</p> </blockquote> <p><img src="/img/opsramp-client-setup-img-6.png" alt="Client Account" title="Client Account"></p> <ol start="8"> <li> <p>Enabling advanced add-ons for the client account, as per the client’s operational requirements, is recommended:</p> <ul> <li>Select <strong>Setup > Setup > Accounts > Clients</strong>.</li> <li>Select the <strong>client account</strong> from the list of clients.</li> <li>Click <strong>Edit</strong> button for the <strong>ADDONS</strong> section and enable the Add-ons. To learn more about the Add-ons, see the <a href="https://glp.docs.opsramp.com/guides/component-model/#hybrid-discovery-and-monitoring">HPE OpsRamp Hybrid Discovery and Monitoring documentation</a>.</li> </ul> </li> </ol> <blockquote> <p><strong>Note:</strong> To enable client account Add-ons, ensure your HPE GreenLake workspace details (name, address) do not contain any special characters.</p> </blockquote> <h2>Managing user account</h2> <p>The <em>Partner Administrator</em> role is provided to the HPE GreenLake cloud user who provisioned the hybrid observability service in the workspace. Invited HPE GreenLake users with the <em>OpsRamp Access</em> role granted in the workspace will have the role <strong>GLP Invited Partner User</strong> assigned in the service.</p> <ol> <li> <p>When you log in to hybrid observability service user interface powered by HPE OpsRamp, ensure you have selected your <em>partner organization</em> on the top left menu.</p> </li> <li> <p>From the top navigation menu, select <strong>Setup > Account</strong>.</p> </li> <li> <p>Select the <strong>Users and Permissions</strong> tile.</p> </li> <li> <p>Select the <strong>Users</strong> tile.</p> </li> <li> <p>Select an invited user who launched and logged in to the service to visualize the details description of the user and the roles. You will see the default role assigned to the user as <strong>GLP Invited Partner User</strong>. You can assign the role of <em>Partner Administrator</em> to an invited user.</p> </li> </ol> <p>In HPE GreenLake Flex Solutions, as <em>Partner Administrator</em>, you can manage accounts for your partner organization. You can create permissions sets, roles, user groups, and assign them to users who have been granted <em>OpsRamp Access</em> role in your workspace and launched the hybrid observability service.</p> <blockquote> <p><strong>Note:</strong> In HPE GreenLake Flex Solutions, you cannot assign roles to users at the <em>client account</em> level.</p> </blockquote> <p>To learn more about account management, refer to the <a href="https://glp.docs.opsramp.com/platform-features/feature-guides/account-management/">HPE OpsRamp Account Management documentation</a>.</p> <h2>Installing and configuring a gateway collector appliance</h2> <p>Resources need to be discovered before they can be monitored, and metrics collected.</p> <p>With the hybrid observability service, you discover, monitor, and manage infrastructure resources (compute, storage, network) included in the HPE GreenLake Flex Solutions using an <strong>agentless</strong> method with a <strong>gateway collector appliance</strong> installed <strong>within</strong> your firewall environment. This appliance can be a virtual machine or a cloud-native application that runs on your own Kubernetes environment.</p> <p>To learn more about the gateway collector appliance installation and activation procedures, and deployment requirements, refer to the <a href="https://glp.docs.opsramp.com/platform-features/">HPE OpsRamp Platform documentation</a>.</p> <p>There are two types of gateway collectors:</p> <ul> <li><a href="https://glp.docs.opsramp.com/platform-features/gateways/">Classic gateway</a> that can be installed as a Virtual Machine (VM) in a hypervisor.</li> <li><a href="https://glp.docs.opsramp.com/platform-features/nextgen-gateways/">NextGen gateway</a> that runs on a Kubernetes environment. The NextGen gateway is a new generation gateway collector appliance, and it is <strong>HPE’s recommended</strong> gateway collector appliance.</li> </ul> <blockquote> <p><strong>Note:</strong> Agent-based methods can also be used for the discovery of physical devices running Linux and Microsoft Windows Operating Systems. To learn more about agents, refer to the <a href="https://glp.docs.opsramp.com/platform-features/agents/">HPE OpsRamp Agent documentation</a>.</p> </blockquote> <h3>Gateway Collector appliance prerequisites</h3> <ol> <li> <p>Ensure your HPE GreenLake workspace details (name, address) do not contain any special character.</p> </li> <li> <p>A <strong>virtual environment</strong> (a hypervisor or a Kubernetes environment) is required within your firewall environment.</p> </li> <li> <p>The gateway collector appliance needs to be appropriately sized based on the monitored resource count. Refer to the following documentation:</p> <ul> <li><a href="https://glp.docs.opsramp.com/platform-features/gateways/gateway-deployment-requirements/#gateway-appliance">Classic gateway collector appliance deployment sizing</a>.</li> <li><a href="https://glp.docs.opsramp.com/platform-features/nextgen-gateways/installation-nextgen-gateway/#gateway-capacity-parameters">NextGen gateway collector appliance sizing</a>.</li> </ul> </li> <li> <p>Connectivity network protocol requirements:</p> <ul> <li>The outbound communication from the gateway collector appliance to the HPE OpsRamp platform is fully secured using a TLS 1.2 tunnel and TCP port 443 to securely transfer data from the gateway collector appliance to the HPE OpsRamp service.</li> <li>SNMP: port 161 for outbound communication from the gateway collector appliance to the networking devices and on-premises infrastructure, and port 162 for inbound communication from networking devices to the gateway collector appliance. See the <a href="https://glp.docs.opsramp.com/integrations/network/snmp-discovery/">HPE OpsRamp SNMP documentation</a>.</li> <li>TCP port 3128 for a proxy connection: outbound communication from compute servers’ agent (Microsoft Windows OS and Linux OS) to the gateway collector appliance’s embedded proxy (if proxy is enabled in the gateway). See the <a href="https://glp.docs.opsramp.com/platform-features/agents/agent-reference/">HPE OpsRamp agent connectivity requirement documentation</a>.</li> <li>SSH and Windows Management Instrumentation (WMI) protocols are used to discover infrastructure devices.</li> </ul> </li> </ol> <p><img src="/img/network-deployment-requirements.png" alt="Network protocol requirements" title="Network protocol requirements"></p> <h3>Gateway collector appliance installation and activation procedure</h3> <p>You then proceed as follows to install the gateway collector appliance on your HPE OpsRamp service environment:</p> <ol> <li> <p>Select the <strong>client account</strong> you created in the previous step.</p> </li> <li> <p>In the top navigation menu, select <strong>Setup > Account</strong>, and click <strong>Collector Profiles</strong>.</p> </li> <li> <p>Click <strong>+ADD</strong> to create a gateway collector profile. The gateway collector profile page opens.</p> </li> </ol> <p><img src="/img/opsramp-gateway-install-img-2.png" alt="Add the gateway collector profile" title="Add the gateway collector profile"></p> <ol start="4"> <li> <p>Enter a <strong>Profile Name</strong> and a <strong>Description</strong> for your gateway.</p> </li> <li> <p>Select a <em>Classic gateway</em> or a <em>NextGen gateway</em> image based on your environment. In our example, the <strong>recommended NextGen gateway</strong> is deployed. You can use an ISO (International Organization for Standardization) or OVA (Open Virtual Appliance) image to run the NextGen gateway on a VMware virtual machine. If you want to run the NextGen gateway in your <strong>own Kubernetes environment</strong>, select the <strong>Cloud-Native Application (Installer)</strong>.</p> </li> </ol> <p>Here, we install the NextGen gateway collector OVA as a virtual machine on a VMware ESXi server.</p> <p><img src="/img/opsramp-gateway-install-img-4.png" alt="Install NextGen gateway" title="Install NextGen gateway"></p> <blockquote> <p><strong>Note:</strong> You can click the ellipsis (…) of the selected virtual appliance to view the installation and activation instructions.</p> </blockquote> <ol start="6"> <li> <p>Click <strong>Next</strong>. The gateway profile is displayed.</p> </li> <li> <p>Click the arrow to download the virtual appliance image in your environment.</p> <p><img src="/img/opsramp-gateway-install-img-5.png" alt="Download the gateway file" title="Download the gateway file"></p> </li> <li> <p>Follow the <strong>INSTALLATION</strong> instructions to install the gateway collector appliance as a virtual machine on the hypervisor. In our case example, our hypervisor is a VMware ESXi server. The OVA will install Ubuntu Linux Operating System in the virtual machine. Summary of the installation steps:</p> <ul> <li>Log in to vSphere.</li> <li>Deploy the OVF template on your ESXi server or cluster environment. Select the downloaded OVA file.</li> <li>Specify a unique name for the virtual machine and a location in your environment.</li> <li>Select a compute resource such as an ESXi cluster, ESXi host, or a Resource Pool for the virtual machine.</li> <li>Review the detailed information.</li> <li>Select a datastore for the virtual machine and use the <em>Thick Provision Lazy Zeroed</em> disk format.</li> <li>Select a destination Network for the virtual machine.</li> <li>Verify the deployment settings and click <em>FINISH</em> to start creating the virtual machine in the hypervisor.</li> <li>Power on the virtual machine.</li> </ul> </li> <li> <p>Next, once the virtual machine is deployed and powered on, apply the <strong>ACTIVATION</strong> instructions to activate and register the gateway collector appliance on the virtual machine. The activation process takes a few minutes to complete installation of the Kubernetes cluster and activation of the gateway pods in the Kubernetes cluster. The activation process will instruct you to:</p> <ul> <li>Step 1 — Log in to the gateway collector appliance.</li> <li>Step 2 — Set the hostname for the gateway collector appliance and change the default password.</li> <li>Step 3 — Install the K3s Kubernetes cluster on the gateway collector appliance.</li> <li>Step 4 — Register the gateway collector appliance in the HPE OpsRamp platform.</li> <li>Step 5 — Click <em>FINISH</em> when the Activation is completed.</li> </ul> </li> </ol> <blockquote> <p><strong>Important note:</strong> If you want to assign <strong>a static IP address</strong> to the gateway appliance, refer to the <a href="https://glp.docs.opsramp.com/platform-features/nextgen-gateways/installation-nextgen-gateway/iso-based-installation/#step-3-update-hostname-and-install-kubernetes">HPE OpsRamp NextGen gateway installation documentation</a>. Follow these steps before installing the Kubernetes cluster (step 3 above) on the gateway collector appliance.</p> </blockquote> <p>The activation process will install the K3s Kubernetes cluster on the virtual machine with the main pod (<em>nextgen-gw-0</em>) where the gateway collector code runs, and the storage pod (<em>nextgen-gw-redis-master-0</em>).</p> <p>When the gateway collector is installed and registered in your HPE OpsRamp service instance, go to <strong>Setup > Account > Collector Profiles</strong>. The status of the gateway collector appliance should be <strong>CONNECTED</strong>.</p> <p><img src="/img/opsramp-gateway-install-img-8.png" alt="Gateway is connected" title="Gateway is connected"></p> <p>You can also navigate to <strong>Infrastructure</strong> tab and select <strong>Resources > Gateway</strong> or use the <strong>recommended method</strong>: <strong>Infrastructure > Search > OTHERS > OpsRamp Gateway</strong>, to visualize detailed information about the newly installed and registered gateway collector appliance.</p> <p><img src="/img/opsramp-gateway-install-img-9.png" alt="Gateway resources" title="Gateway resources"></p> <p>If you see the main pod (nextgen-gw-0) status as <strong>Running</strong> (color indication <strong>green</strong>), it means the NextGen gateway is running successfully.</p> <p>You can also navigate to <strong>Dashboard > Classic Dashboard</strong> to drill into infrastructure elements of the gateway collector appliance. The dashboard provides an overview of health, performance, and availability of resources and services.</p> <p><img src="/img/opsramp-gateway-install-img-10.png" alt="Classic dashboard gateway resources status" title="Classic dashboard gateway resources status"></p> <blockquote> <p><strong>Note:</strong> The resource status for the <em>NextGen Gateway Cluster</em> and the <em>NextGen Gateway Node</em> are shown as <strong>Undefined</strong> (color indication <strong>brown</strong>). This is expected because no availability monitor is assigned to these two resources. To monitor the NextGen Gateway Node, you will need to assign a <strong>cloned</strong> version of the Global Monitoring Template <strong>Agent G2 – Linux OS Performance Monitoring</strong>. Refer to the next blog post &#x3C;link to blog post part 3> to learn how to assign a Monitoring Template to a resource.</p> </blockquote> <h2>Whitelisting IP addresses to improve network access security</h2> <p>HPE recommends restricting a specific set of IP addresses by whitelisting them within your firewall for outbound connection to the hybrid observability service in the cloud. The whitelist is based on your regional deployment of HPE OpsRamp service in the HPE GreenLake workspace. See the <a href="https://glp.docs.opsramp.com/support/reference/public-ip-addresses/">HPE OpsRamp Collector Whitelisted IP Addresses documentation</a>.</p> <h2>Summary</h2> <p>This blog post helps you get started with the initial configuration of the hybrid observability service powered by HPE OpsRamp Software to enable the discovery of IT infrastructure devices included in HPE GreenLake Flex Solutions.</p> <p>The initial setup consists of:</p> <ol> <li>Creating a tenant (client account) of the Partner account for your organization.</li> <li>Managing HPE GreenLake users’ roles.</li> <li>Installing and registering a gateway collector appliance, a prerequisite to enable the discovery and monitoring of physical devices.</li> </ol> <p>Don’t miss <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-3-enabling-the-monitoring-of-agentless-ssh-enabled-systems-in-hpe-greenlake-flex-solutions/">part 3</a> and <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-4-enabling-the-monitoring-of-physical-devices-in-hpe-greenlake-flex-solutions/">Part 4</a> of this blog series, where I’ll further explore the integration set up activities to discover infrastructure resources, start monitoring resources and viewing metrics and alerts.</p> <p>There’s also a series of video tutorials on the <a href="https://developer.hpe.com/greenlake/hybrid-observability-flex-solutions/home/">Hybrid observability in HPE GreenLake Flex Solutions landing page</a> that walks you through the contents described in this blog series for readers who prefer an audio-visual learning experience.</p> <p>To resolve issues with HPE GreenLake Flex Solutions or hybrid observability service powered by HPE OpsRamp, contact the support team. While logged in to your HPE GreenLake workspace:</p> <ol> <li>Click <strong>Help &#x26; Support</strong>.</li> <li>Under <strong>Help</strong>, select <strong>OpsRamp</strong> or <strong>HPE GreenLake Flex Solutions</strong>.</li> <li>Click <strong>Create New Case</strong>.</li> </ol>https://developer.hpe.com/hybrid-observability-service-–-part-1-provisioning-and-activation-in-hpe-greenlake-flex-solutions/https://developer.hpe.com/hybrid-observability-service-–-part-1-provisioning-and-activation-in-hpe-greenlake-flex-solutions/Thu, 22 May 2025 15:42:16 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>As organizations embrace cloud-native technologies, and adopt hybrid or multi-cloud environments, system architecture is becoming more complex and distributed than ever. With this, IT teams are under constant pressure to detect and resolve issues faster — so they can keep services reliable and business operations running smoothly.</p> <p>To address these challenges, IT teams are turning to observability solutions to manage complex systems and proactively identify and resolve issues. Observability plays a critical role in modern IT Operations Management (ITOM), offering real-time visibility across hybrid and multi-cloud environments. With the observability tools, teams can understand what’s happening across their infrastructure, detect anomalies early, and resolve incidents quickly – keeping systems efficient, stable, and resilient.</p> <p><a href="https://www.hpe.com/us/en/hpe-greenlake-flex-solutions.html">HPE GreenLake Flex Solutions</a> includes a cloud-based hybrid observability service, powered by <a href="https://www.hpe.com/us/en/opsramp.html">HPE OpsRamp Software</a>, an AI-powered command center that simplifies and optimizes IT operations.</p> <p>Hybrid observability in HPE GreenLake Flex Solutions is a SaaS-based ITOM solution and AIOps monitoring and observability service. It is designed to handle the complexity of hybrid multi-cloud environments. By leveraging the hybrid observability service powered by HPE OpsRamp, you will have visibility and control to centrally monitor and manage the <strong>physical</strong> infrastructure resources (servers, storage, network equipment) that comes within the HPE GreenLake Flex Solutions.</p> <p>You can also expand your observability capabilities by purchasing additional subscriptions to provide coverage of <strong>logical resources</strong> such as virtual machines, containers, and workloads, including those running on public clouds or non-HPE infrastructure.</p> <p>Let’s get started with the journey to set up the hybrid observability service powered by HPE OpsRamp Software.</p> <h2>Prerequisites</h2> <p>To enable monitoring and observability of physical devices in HPE GreenLake Flex Solutions, there are some requirements that must be fulfilled:</p> <ol> <li> <p>Create an HPE account, sign in to HPE GreenLake cloud and create a workspace for your organization. The workspace is the fundamental access management control boundary for HPE GreenLake resources. When you create a workspace, you are automatically assigned the role of <em><strong>Workspace Administrator</strong></em>.</p> <blockquote> <p><strong>Note:</strong> If you haven’t already set up your HPE account and workspace, refer to the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-497192AA-FDC2-49C5-B572-0D2F58A23745.html">HPE GreenLake Cloud User Guide</a> to create them.</p> </blockquote> </li> <li> <p>Ensure you receive the hybrid observability subscription key by email as part of the HPE GreenLake Flex Solutions contract.</p> </li> <li> <p>Provision and activate hybrid observability service on the HPE GreenLake workspace.</p> </li> <li> <p>Create a client account.</p> </li> <li> <p>Install a gateway collector appliance in a virtual environment as prerequisite to enable discovery of physical resources before they can be monitored.</p> </li> <li> <p>Install integration modules to discover physical resources and enable monitoring and management of these resources.</p> </li> <li> <p>Apply monitoring configuration to manage and track the health, performance, and availability status of the discovered resources components.</p> </li> </ol> <p>In this blog post it is assumed that you have created an HPE account and a workspace. You will then learn how to:</p> <ul> <li>Provision the hybrid observability service powered by HPE OpsRamp Software in the HPE GreenLake workspace.</li> <li>Activate the associated subscription key.</li> <li>Assign appropriate access role to desired users in the HPE GreenLake workspace.</li> <li>Launch the hybrid observability service in HPE GreenLake Flex Solutions to begin your hybrid observability journey.</li> </ul> <p>In the subsequent blog posts of the series, I will walk you through setting up the hybrid observability service through the HPE OpsRamp user interface to monitor, manage, and control your physical infrastructure resources.</p> <h2>Provisioning the hybrid observability service powered by HPE OpsRamp Software in HPE GreenLake Flex Solutions</h2> <p>To activate the hybrid observability service in HPE GreenLake Flex Solutions and the associated subscription key on your workspace, you need to:</p> <ol> <li> <p>Provision the hybrid observability service powered by HPE OpsRamp Software in a desired region in your workspace using your HPE account with the <em>Workspace Administrator</em> role.</p> </li> <li> <p>Activate the subscription key for your workspace.</p> </li> <li> <p>Grant the <em><strong>OpsRamp Access</strong></em> role to the desired users in your workspace.</p> </li> <li> <p>Launch the HPE OpsRamp Software from the workspace to use and configure the hybrid observability service.</p> </li> </ol> <p><strong>Let’s get started with this sequence of steps!</strong></p> <h3>Step 1 – Provision the HPE OpsRamp Software service in your workspace</h3> <p>Log in to your HPE GreenLake cloud workspace using your <em>Workspace Administrator</em> account. Provision the HPE OpsRamp Software to your workspace and select a deployment region. You can provision the HPE OpsRamp Software before adding the subscription key.</p> <ul> <li>Click <strong>Services > Catalog</strong>.</li> </ul> <p><img src="/img/opsramp-activation-img-2.png" alt="HPE GreenLake Service catalog" title="HPE GreenLake Service catalog"></p> <ul> <li>Under <strong>Management &#x26; Governance</strong>, select <strong>OpsRamp</strong>.</li> <li>Click <strong>Provision</strong>.</li> <li>Select a <strong>Deployment Region</strong> from the drop-down. You can add more regions once the service is installed.</li> <li>Click <strong>Terms of Service</strong> to review the terms, and then select the checkbox to confirm your agreement.</li> <li>Click <strong>Deploy</strong>.</li> </ul> <p><img src="/img/opsramp-activation-img-5.png" alt="Provision HPE OpsRamp" title="Provision HPE OpsRamp"></p> <h3>Step 2 – Activate the subscription key in your workspace</h3> <p>As the <em>Workspace Administrator</em>, add the hybrid observability <strong>subscription key</strong> that you received via email for your workspace. A subscription key is needed to launch the service in your workspace:</p> <ul> <li>In your HPE GreenLake workspace, click <strong>Services</strong> and then <strong>Service Subscriptions</strong>.</li> <li>Click <strong>Add Service Subscription</strong>.</li> <li>Enter the <em>Subscription key</em> you received by email.</li> <li>Click <strong>Add</strong>.</li> <li>Click <strong>Next</strong>.</li> <li>(Optional) You can assign tags to the subscription key or manage tags later. Tags allow you to categorize resources based on purpose, owner, location, or other criteria. Click <strong>Assign</strong> if you add a tag, then click <strong>Next</strong>.</li> <li>Showback rates are optional. Skip this step. Clear the <strong>Add showback rates</strong> checkbox.</li> <li>Click <strong>Next</strong>.</li> <li>Click <strong>Finish</strong>. The subscription key is automatically applied to the workspace and the hybrid observability service powered by HPE OpsRamp.</li> </ul> <p><img src="/img/opsramp-activation-img-12.png" alt="Activate HPE OpsRamp subscription key" title="Activate HPE OpsRamp subscription key"></p> <h3>Step 3 – Assign the OpsRamp Access role to the desired users</h3> <p>After you provisioned the HPE OpsRamp Software to your workspace, you must give the desired users permission to access and run the hybrid observability service. Users must be granted the built-in <em><strong>Opsramp Access</strong></em> role. Once users are assigned this role, they have the authorization they need.</p> <ul> <li>In your workspace, on the <strong>Home</strong> page, select <strong>Manage Workspace</strong> from the <strong>Quick Links</strong>.</li> <li>Click <strong>Workspace identity &#x26; access</strong>.</li> <li>Click <strong>Users</strong>.</li> <li>Select the desired user or select the ellipsis (...) on the right of the user.</li> <li>Click <strong>Assign Role</strong>.</li> <li>Select the <strong>OpsRamp</strong> service.</li> <li>Select the <strong>Opsramp Access</strong> role.</li> <li>Do not slide the <strong>Limit Resource Access</strong> toggle to the right.</li> <li>Click <strong>Assign Role</strong>.</li> <li>Click <strong>Change Role Assignment</strong>.</li> </ul> <p><img src="/img/opsramp-activation-img-19.png" alt="Assign OpsRamp Access role" title="Assign OpsRamp Access role"></p> <h3>Step 4 – Launch the hybrid observability service</h3> <p>With the HPE OpsRamp Software deployed in a region and attached to a valid subscription, users are ready to take advantage of the hybrid observability operations management solution.</p> <ul> <li>Click the <strong>Launch</strong> button to access and use the hybrid observability user interface powered by HPE OpsRamp. The hybrid observability service provides a unified view of observability across multi-cloud and on-premises infrastructure devices, resources, and services.</li> <li>The first time you access the service user interface, you must accept the <strong>End User Agreement</strong>.</li> </ul> <p><img src="/img/opsramp-activation-img-20.png" alt="HPE OpsRamp service user interface" title="HPE OpsRamp service user interface"></p> <p>The service has now launched. You log in to the service as <em>Partner Administrator</em> role.</p> <p>By default, Partner’s advanced monitoring <strong>Add-ons</strong> are not enabled. It is recommended to enable the <strong>Add-ons</strong> for the Product Packages at the Partner organization level. To enable the advanced monitoring Add-ons, select <strong>Setup > Setup > Accounts > Partner details</strong>. Click <strong>Edit</strong> button at the top right, select the <strong>Add Ons</strong> tab and check all the Add-ons, then <strong>Save</strong>. To learn more about the Add-ons, see the <a href="https://glp.docs.opsramp.com/guides/component-model/#hybrid-discovery-and-monitoring">HPE OpsRamp Hybrid Discovery and Monitoring documentation</a>.</p> <blockquote> <p><strong>Note:</strong> This feature will be made available under <strong>Setup > Account</strong> in an upcoming HPE OpsRamp release.</p> <p><strong>Note:</strong> To enable Partner’s Add-ons, ensure your HPE GreenLake workspace details (name, address) do not contain any special character.</p> </blockquote> <p>The hybrid observability in HPE GreenLake Flex Solutions uses a multi-tenant model. As a <em>Partner Administrator</em>, you can manage multiple client accounts, each client account representing an individual organization, business unit, or customer environment. You’ll start setting up your hybrid observability service by:</p> <ul> <li>Defining a client account.</li> <li>Assigning specific access roles to users.</li> <li>Installing a virtual appliance, called a gateway collector appliance, that discovers and manages your on-premises physical devices.</li> <li>Applying integration modules and monitoring configurations to resources.</li> </ul> <p>These steps will be explained in the subsequent blog posts of the series.</p> <h2>Summary</h2> <p>This blog post helps you get started provisioning of the hybrid observability service powered by HPE OpsRamp Software and activating the subscription key in your HPE GreenLake cloud workspace. For more information about the provisioning of the service and subscription key activation, check out the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-9EDAAB42-9182-488D-A06F-6E8CB4BFAB60.html">HPE GreenLake Cloud User Guide</a>.</p> <p>Stay tuned for <a href="https://developer.hpe.com/blog/hybrid-observability-service-%E2%80%93-part-2-initial-configuration-to-enable-the-discovery-of-resources-in-hpe-greenlake-flex-solutions/">my next post of the series</a>, where I’ll delve into the hybrid observability service integration setup activities to <strong>discover</strong> infrastructure resources. I’ll create a tenant, also referred to as client account, for the Partner organization. I’ll also install and configure a gateway collector appliance as a prerequisite to enable the discovery of physical infrastructure devices included in the HPE GreenLake Flex Solutions contract.</p> <p>There’s also a series of video tutorials on the <a href="https://developer.hpe.com/greenlake/hybrid-observability-flex-solutions/home/">Hybrid observability in HPE GreenLake Flex Solutions landing page</a> that walks you through the contents described in this blog series for readers who prefer an audio-visual learning experience.</p> <p>To resolve issues with HPE GreenLake Flex Solutions or hybrid observability service powered by HPE OpsRamp, contact the support team. While logged in to your HPE GreenLake workspace:</p> <ol> <li>Click <strong>Help &#x26; Support</strong>.</li> <li>Under <strong>Help</strong>, select <strong>OpsRamp</strong> or <strong>HPE GreenLake Flex Solutions</strong>.</li> <li>Click <strong>Create New Case</strong>.</li> </ol>https://developer.hpe.com/from-data-centers-to-centers-of-data-navigating-the-pull-of-data-gravity/https://developer.hpe.com/from-data-centers-to-centers-of-data-navigating-the-pull-of-data-gravity/Fri, 09 May 2025 11:31:10 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <blockquote> <p><em><strong>“Son, when I don’t have the data—people die.“</strong></em></p> </blockquote> <p>The room fell silent as a highly decorated Army colonel spoke about supporting frontline combat operations with real-time IT. His point landed hard: in the fog of war, data isn’t just a convenience—it’s a lifeline. Without it, decisions are delayed. And in combat, delays can cost lives.</p> <p>It may sound extreme, but business leaders face a strikingly similar reality. When data is missing, late, or incomplete, decisions suffer. Opportunities are missed. Customers leave. Competitors surge ahead. Whether in warfare or in business, the equation remains simple: <strong>data drives decisions—but only if it can be processed within time to matter</strong>.</p> <p>In my <a href="https://developer.hpe.com/blog/the-rise-of-the-distributed-enterprise/">previous blog post</a>, I described the rise of the distributed enterprise and how businesses are moving away from centralized data centers and embracing a model that brings technology closer to where value is created. In this post, I'll get into details about data gravity and the effect it has had on this shift.</p> <h2>Ever expanding data—and why you can’t centralize your way out</h2> <p>Workloads are getting increasingly complex. The data needed to precisely calibrate a business decision today has grown by several magnitudes. New infrastructure capabilities drive new workloads which in turn expand data mass exponentially. We have witnessed the required data mass more than double every year. This trend has been observable for at least a decade. What used to be a manageable trickle has become a roaring flood. Real-time analytics, new AI models, IoT sensors, and always-on digital services all demand access to more—and faster—data.</p> <p>But here’s the problem: traditional, centralized data center architectures can’t keep up. Trying to move petabytes of data across wide-area networks (WANs) introduces costly delays. Even with WAN speeds improving, they’re simply not accelerating fast enough to match data’s pace. As a result, there is a growing impedance mismatch between data generated at a company’s location and the ability to get it processed at the other end of a network connection before the window of opportunity slams shut.</p> <p>Today’s reality? <strong>We’re generating too much data, too quickly, in too many places</strong>. And that’s forcing us to rethink how, and where, we process it.</p> <p><strong>Enter: Data gravity</strong></p> <p>Data isn’t just growing—it’s anchoring itself, an effect referred to as <em><strong>data gravity</strong></em>. Like a star with enough mass to pull planets into orbit, large datasets tend to attract everything around them: applications, services, even more data. And the larger the dataset, the harder—and more expensive—it becomes to move back to a centralized data center or public cloud availability zone for processing.</p> <p>Instead, the smarter, faster move is to <strong>bring processing power to the data</strong>, not the other way around.</p> <p><img src="https://hybridcloud.hpecorp.net/assets/data-needed-to-make-decisions.png" alt="Data needed to make a business decision" title="Data needed to make a business decision"></p> <blockquote> <p><em>Source: HPE research</em></p> </blockquote> <p>This is especially true at distributed enterprise locations, where data is being created at unprecedented rates: in hospitals, factories, smart cities, autonomous vehicles, and beyond. These environments demand <strong>real-time</strong> decision making—and that means they can’t wait for data to go elsewhere to be processed.</p> <h2>The numbers tell the story</h2> <p>Over the last ten years, we’ve witnessed a staggering transformation in the way data is generated, consumed, and leveraged. This change hasn't just been a byproduct of digital transformation; it's been a foundational force behind the rise of artificial intelligence, particularly large language models (LLMs) and agentic AI systems.</p> <p>Over the past decade, global data volumes have increased by <strong>115% per year</strong>. That’s not just exponential growth—it’s super-exponential. Think about it: a terabyte of data generated today will likely be a petabyte in just a few years given current trends.</p> <p>But as data mass expands, our ability to move that data hasn't kept pace. During this same period of time, wide-area network (WAN) bandwidth has grown at just <strong>69% per year</strong>. While that might sound impressive in isolation, it pales in comparison to the data it's supposed to support.</p> <p>Here’s the stark reality:</p> <ul> <li>Over the last 10 years, the mass of decision-making data has increased <strong>2,250X</strong>.</li> <li>WAN bandwidth? It’s only grown about <strong>200X</strong>.</li> </ul> <p>Let that sink in. We’re facing a dramatic performance drag—at the exact moment when speed is becoming a critical business differentiator. This gap between data growth and bandwidth growth has led to a critical WAN infrastructure strain.</p> <p><img src="https://hybridcloud.hpecorp.net/assets/impedence-mismatch.png" alt="Relative data mass to WAN bandwidth impedance mismatch" title="Relative data mass to WAN bandwidth impedance mismatch"></p> <blockquote> <p><em>Source: HPE research</em></p> </blockquote> <p>The bandwidth shortfall has real consequences. The lag in WAN infrastructure has introduced an additional 11 seconds of delay per every second of decision-making time over the past decade. In high-speed trading, autonomous vehicles, or real-time AI systems, those seconds are unacceptable. They're not just delays—they're the difference between relevance and irrelevance, or in some cases, safety and disaster.</p> <p>Take a quality control system in a manufacturing plant, for example. It might generate 100MB of camera data <strong>every second</strong>. To ascertain product quality, that data needs to be processed instantly. Transmitting it to a central data center requires a 1.2 Gbps link—technically possible, but prohibitively expensive and complex, especially at scale.</p> <p>The smarter move? <strong>Process it on site</strong>. Act on it in real time. Let the factory floor become the “center of data.”</p> <h2>A new equation for the real-time enterprise</h2> <p>Modern AI systems, especially large language models and agentic AI, are data-hungry by nature. Training a state-of-the-art LLM requires ingesting massive datasets that span languages, domains, and formats. But it's not just about training—it's about real-time inference, adaptation, and autonomy.</p> <p>Agentic AI, which refers to systems that can act autonomously and make decisions in complex environments, amplifies this need. These systems rely on vast amounts of localized, real-time data: sensor feeds, user interactions, environmental inputs, and more. Their effectiveness hinges not only on access to data but on how quickly and efficiently that data can be processed.</p> <p>We need a new way to measure value in a data-driven world.</p> <p><strong>Try this:</strong></p> <p><em><strong>Data mass / Processing time = Business value</strong></em></p> <p>It’s not just about how much data you have—it’s about how quickly you can turn that data into decisions. And the infrastructure to support that transformation is finally here.</p> <p>A few years ago, you needed 24 racks to get 768 CPU cores. Today, you can get that same horsepower in <strong>just three 1U servers</strong>. Flash storage like NVMe drives is <strong>35x faster</strong> than traditional disks. 400 Gbps networking between servers is no longer bleeding-edge—it’s becoming standard. But only at a given location.</p> <p>So, location-specific hardware is ready. The question is whether our thinking—and architecture—has caught up. To fully realize the potential of LLMs and agentic AI, we need to rethink how and where AI is deployed. The future isn't about massive, centralized models alone. It's about hybrid architectures that combine the power of the cloud with the immediacy of the edge.</p> <p>We’ve entered a new era—one where <strong>location</strong> is just as important as <strong>speed</strong> or <strong>scale</strong>. As data volumes balloon and decision windows shrink, the old model of hauling everything to a central data center no longer works.</p> <p>The new model? Flip the script.</p> <ul> <li>Build <strong>edge-native, distributed</strong> systems that eliminate bottlenecks.</li> <li>Leverage r<strong>eal-time intelligence</strong> at the point of creation so you can act on data where it lives, and then move it when it grows cold</li> <li>Design <strong>for gravity</strong>—not against it—by aligning compute and storage with the natural flow of data.</li> </ul> <h2>Taking your data’s temperature into account</h2> <p>To be clear, not all data needs to stay at the edge forever. Hot, high-value data—the kind that feeds real-time decisions—should absolutely be processed on site. However, when today’s data cools – or becomes less active – then it should move. What? We were just discussing how IT infrastructure like servers, networks, storage, and orchestration needed to move to the distributed locations and no longer remain in the classic old data center. That still stands. And precisely because IT infrastructure now is distributed, it must only be used for data that drives decisions where operational value is high. But as data cools, it can (and should) be archived or moved to lower-cost, centralized storage.</p> <p>Too often, organizations treat all data the same, hoarding everything on an expensive, high-performance infrastructure. That’s a recipe for inefficiency. Instead, take a smarter approach and implement <strong>data lifecycle management</strong> strategies that balance performance, cost, and compliance.</p> <p>In other words: <strong>keep your fast storage clean. Let cold data go</strong>. Make room for what matters now.</p> <h2>Data Mass / Time to process X Probability of data reuse = Business value</h2> <p>For most data, the probability of data reuse declines exponentially over time. Fresh data generated now has a 100% chance of data use. But the same data set 60 days from now may have fallen to only a ten percent chance of data being accessed. This is a powerful insight.</p> <p><img src="https://hybridcloud.hpecorp.net/assets/data-scew.png" alt="Data skew" title="Data skew"></p> <blockquote> <p><em>Source: Patent number: 9372636 - Dave Zeryck, Denis Vilfort, June 21, 2016</em></p> </blockquote> <p>Since WAN bandwidth is constant, the data set will take the same amount of time to move, but now has 1/10th the impact on operations. Thus, there comes a point in the data lifecycle where the time to move the data outweighs the risk while the savings of freeing up hundreds of GBs of NVMe storage saves considerable cost in purchasing more storage capacity for that location. Moving data promotes good ROI and lowers operating costs by enabling location-specific special re-use. Data movement can be combined further with data de-duplication and data compression for deep archiving of cold data in a central location while recycling location-specific IT infrastructure and making capacities available for new fresh incoming data. Think of this strategy as “hot edge – cold core”.</p> <h2>A call to act on data where it lives – Now is the time to evolve</h2> <p>The last decade of data growth hasn't just enabled AI—it's made it inevitable. But it’s also made clear that our old architectures won’t scale. The math is against us: data growth is outpacing our ability to move it, and data gravity is only increasing.</p> <p>To keep up, we need a compute paradigm that respects the physics of data. That means putting AI where the data is, embracing distributed intelligence, and designing systems that can thrive in a world of fragmented, fast-moving, and increasingly immobile information.</p> <p>Modern infrastructure gives us the tools. High-performance compute, GPU scaling, NVMe flash, and next-gen networking let us act on data where it lives, unlocking insights without dragging data across expensive networks. And since the business value of a given data set decreases over time, we must modify the impact of data gravity for that data set as it cools, moving it away to make room for new insights.</p> <p>If your organization is still operating on a centralized data model, now is the time to evolve. The data gravity challenge isn’t going away—it’s only getting stronger. But with the right architecture, strategy, and partners, it’s also an incredible opportunity.</p> <p>At HPE, we help enterprises rethink their infrastructure to meet this challenge head-on—building <strong>centers of data</strong> that maximize performance, reduce waste, and accelerate decision-making.</p> <p>Don’t wait. Don’t move your data.</p> <p>Act on it**—where it lives**.</p>https://developer.hpe.com/2025-may-05/https://developer.hpe.com/2025-may-05/Mon, 05 May 2025 13:00:00 GMThttps://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-1-productivity-and-performance/https://developer.hpe.com/10-myths-about-scalable-parallel-programming-languages-redux-part-1-productivity-and-performance/Wed, 30 Apr 2025 15:05:02 GMT<p>external blog</p>https://developer.hpe.com/chapel-fortran-interop-in-an-ocean-model-introduction/https://developer.hpe.com/chapel-fortran-interop-in-an-ocean-model-introduction/Fri, 25 Apr 2025 05:56:41 GMT<p>external blog</p>https://developer.hpe.com/migrating-vsphere-virtual-volumes-to-kubevirt-with-hpe-csi-driver-for-kubernetes/https://developer.hpe.com/migrating-vsphere-virtual-volumes-to-kubevirt-with-hpe-csi-driver-for-kubernetes/Tue, 22 Apr 2025 15:04:21 GMT<p>When we talk about moving to the cloud and in the advent of cloud repatriation there’s no denying that data has gravity. Sometimes we refer to this phenomenon as "Hotel California" referencing the American rock band song by the Eagles about being held captive within a hotel. Without a doubt, this concept springs to mind when we think about migrating virtual machines (VMs) within our private cloud between hypervisors. Is your data being held captive by your legacy virtual infrastructure? The most likely answer is "It depends".</p> <p>There are plenty of migration tools and options out there that will assist application owners to either copy data out, restore from a backup, or similar. This works great for small workloads with a handful of gigabytes. The downtime involved while making the transaction isn't very significant and can thus be performed within existing Service Level Agreements (SLA).</p> <p>Consider the scenario where you have a large virtual file server with a complex file structure of tiny files where everyone knows that copying or restoring the file structure alone would take days and incremental updates would take hours to traverse. How do you tackle this challenge?</p> <p><img src="/img/intro.png" alt="" title="Migration workflow"></p> <p>In this blog I’ll revisit the concept of <a href="https://developer.hpe.com/blog/lift-and-transform-apps-with-hpe-csi-driver-for-kubernetes/">Lift and Transform Apps with HPE CSI Driver for Kubernetes</a> and I'll show you how to apply this same methodology in migrating large data volumes from VMware vSphere to KubeVirt using native features of the HPE CSI Driver for Kubernetes with HPE Alletra Storage MP B10000.</p> <p>The intermediary clones of the source volume that is used to test migration before cutover is what makes the solution compelling and very powerful. Iteratively testing the migration workflow and carefully planning the cutover will prevent mishaps and unnecessary downtime that might occur with error prone copy and restore procedures.</p> <p>This blog will go through each relevant step in detail. There’s also a <a href="https://www.youtube.com/embed/5PYROAFVD-A?si=UBBOMb2FXdAgJuKi">short demo on YouTube</a> that walks through the contents described below for readers who prefer a more audio-visual learning experience.</p> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.212424849699396%; position: relative; height: 0; overflow: hidden; margin-bottom: 1.0725rem" > <iframe src="https://www.youtube.com/embed/5PYROAFVD-A?si=UBBOMb2FXdAgJuKi" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen="" style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> <h1>Assumptions</h1> <p>In this scenario we’ll work with a VM that uses a separate data volume for a MariaDB database. There’s one Fedora VM running on vSphere and one Fedora VM running on KubeVirt. A sample test database is being deployed, cloned and migrated where it’s easy to validate the contents of database from an application perspective at each checkpoint.</p> <p>For full disclosure, Ansible playbooks are being provided in this blog post to better describe each step in the migration workflow. Since each application and environment will be different for each step of the way, it’s assumed that automation workflows to conduct the migration will be written according to the application’s best practices.</p> <p>KubeVirt is running on OKD 4.17 (KubeVirt v1.3.1 and Kubernetes v1.30) and has HPE CSI Driver for Kubernetes v2.5.2 installed.</p> <h1>Environment</h1> <p>The Ansible inventory consists of two VMs; one grouped as “kubevirt” and one grouped as “legacy” to distinguish which VM the plays are executed on. Both share a “common” group to allow certain prerequisites to be fulfilled on both VMs.</p> <p>The entire migration workflow can easily be done with Ansible but certain steps are illustrated with the respective platform’s graphical user interfaces for demonstrative purposes.</p> <pre><code class="language-yaml">all: children: legacy: hosts: mariadb-legacy: ansible_user: packer source_host_device: /dev/sdb kubevirt: hosts: mariadb-kubevirt: ansible_user: fedora common: children: legacy: kubevirt: vars: filesystem_label: mariadb-testdb var_lib_mysql: /var/lib/mysql test_db_tgz: https://github.com/datacharmer/test_db/releases/download/v1.0.7/test_db-1.0.7.tar.gz </code></pre> <p>The prerequisites playbook is idempotent and can be run as a sanity checker at any time to declare the VMs primed for either cloning or migration. It’s assumed there’s an empty data disk attached to the source VM residing on a VMware vSphere vVol datastore hosted by an HPE Alletra Storage MP B10000 or earlier system. Note that this workflow works with any HPE storage platform compatible with the HPE CSI Driver for Kubernetes. There are nuances on how cloning is executed and how to find the backing vVol volume name but the principles remain the same.</p> <pre><code class="language-yaml">--- # Attach data disk to VM prior - hosts: legacy become: yes tasks: - name: Assert variables assert: that: - source_host_device - filesystem_label - test_db_tgz - name: Gather device info stat: path: "{{ source_host_device }}" register: source_host_device_exist - name: Make sure device exist assert: that: - source_host_device_exist.stat.isblk is true - name: Create filesystem on device with LABEL filesystem: dev: "{{ source_host_device }}" fstype: ext4 opts: "-L {{ filesystem_label }}" - name: Mount filesystem and persist mount: path: "{{ var_lib_mysql }}" state: mounted src: "LABEL={{ filesystem_label }}" fstype: ext4 # Install MariaDB on both VMs and download Test_DB - hosts: common become: yes tasks: - name: Install prereqs manually command: dnf install -y virt-what mariadb-server - name: Download Test_DB unarchive: src: "{{ test_db_tgz }}" remote_src: yes dest: "{{ ansible_user_dir }}" creates: test_db # Disable/stop MariaDB on destination and remove mountpoint - hosts: kubevirt become: yes tasks: - name: Disable MariaDB service: enabled: no state: stopped name: mariadb - name: Remove default MariaDB datadir file: path: "{{ var_lib_mysql }}" state: directory recurse: true # Start/enable MariaDB and deploy TestDB on source - hosts: legacy become: yes tasks: - name: Set permissions file: path: "{{ var_lib_mysql }}" owner: mysql group: mysql mode: "0755" seuser: system_u serole: object_r setype: mysqld_db_t selevel: s0 - name: Enable MariaDB service: name: mariadb state: started enabled: true - name: Deploy Test_DB shell: mysql &#x3C; employees_partitioned.sql args: creates: "{{ var_lib_mysql }}/employees" chdir: "{{ ansible_user_dir }}/test_db" - name: Test Test_DB shell: mysql -t &#x3C; test_employees_sha.sql &#x26;&#x26; mysql -t &#x3C;&#x3C;&#x3C; 'select @@hostname as 'VM';' args: chdir: "{{ ansible_user_dir }}/test_db" register: test_db_results - name: Show results pause: prompt: "{{ test_db_results.stdout }}" seconds: 10 </code></pre> <p>Another detail in the playbook that might need clarification is that the Test_DB sources is downloaded to the destination VM only because the database queries are needed for validation of the cloning and migration.</p> <p>The last few output lines of the playbook reveal that the database was deployed and tested properly.</p> <pre><code class="language-shell">... +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ +----------------+ | VM | +----------------+ | mariadb-legacy | +----------------+: ok: [mariadb-legacy] PLAY RECAP ************************************************************************* mariadb-kubevirt : ok=6 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 mariadb-legacy : ok=15 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>Once the prerequisites have been installed, you can also inspect the hypervisor platform each VM is running on to ensure what you're doing is actually being executed where expected.</p> <pre><code class="language-shell">$ ansible -m command all -b -a virt-what mariadb-legacy | CHANGED | rc=0 >> vmware mariadb-kubevirt | CHANGED | rc=0 >> redhat kvm </code></pre> <h1>Preparing for the first failure</h1> <p>What is being described in this blog is a perfectly canned example. In reality, the first experiment always fails. Maybe even the second, and third. The point is that it should be non-disruptive and easy to recover from the failures to iteratively improve as the migration is taken across the finish line with as brief downtime as possible. What happens before the cutover may take days and weeks of planning and risk-free test execution.</p> <p>In the first iteration, the vSphere vVol needs to be cloned into a PersistentVolume (PV) using a Persistent Volume Claim (PVC). The PVC is then attached to the KubeVirt VM as a disk.</p> <p>PVC creation may rely on pre-provisioned “static” PVs but more commonly a StorageClass is used to dynamically provision the PV. To allow the PV to clone a pre-existing volume on the storage array, the HPE CSI Driver for Kubernetes needs to be supplied a parameter with the corresponding volume name.</p> <p>The CSI driver supports a construct that allows Kubernetes users to supply their own values to StorageClass parameters called “allowOverrides”. For this exercise, you'll need to craft a StorageClass that allows users to influence the “importVolumeName” (used for migration) and “importVolAsClone” (used for cloning).</p> <pre><code class="language-yaml">apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: legacy-migration provisioner: csi.hpe.com parameters: csi.storage.k8s.io/controller-expand-secret-name: hpe-backend csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage csi.storage.k8s.io/fstype: ext4 description: "Volume cloned or imported by the HPE CSI Driver for Kubernetes" allowOverrides: importVolumeName,importVolAsClone compression: "false" provisioningType: "" reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>Users then call out the parameter and the value they desire in the PVC using an annotation.</p> <pre><code class="language-yaml">apiVersion: v1 kind: PersistentVolumeClaim metadata: name: testdb-cloned namespace: hpe-vmi annotations: csi.hpe.com/importVolAsClone: dat-mariadbl-56480669 spec: volumeMode: Block storageClassName: legacy-migration accessModes: - ReadWriteMany resources: requests: storage: 128Gi </code></pre> <p>So, where does “dat-mariadbl-56480669” come from? It’s the storage array volume name. It’s available to storage administrators of the storage array and it’s also available through the HPE Storage Integration Pack for VMware vCenter (SIP4VC) when used with HPE Alletra Storage MP B10000 and earlier platforms.</p> <p><img src="/img/sip4vc.png" alt="" title="Virtual machine view in SIP4VC"></p> <p>Create the PVC and attach the PVC as a disk to the running KubeVirt VM.</p> <pre><code class="language-shell">$ kubectl create -f pvc-clone.yaml </code></pre> <p>Next, login to OKD.</p> <p><img src="/img/attach-disk.png" alt="" title="Attach disk to VM"></p> <p>Once attached, a “migrate” playbook will mount the disk inside the VM and fire up the database and test it.</p> <pre><code class="language-yaml">--- # With PVC attached to the KubeVirt VM, mount and start MariaDB - hosts: kubevirt become: yes tasks: - name: Mount filesystem and persist mount: path: "{{ var_lib_mysql }}" state: mounted src: "LABEL={{ filesystem_label }}" fstype: ext4 - name: Ensure correct perms and SELinux labels file: path: "{{ var_lib_mysql }}" owner: mysql group: mysql mode: "0755" seuser: system_u serole: object_r setype: mysqld_db_t selevel: s0 recurse: yes - name: Enable MariaDB service: name: mariadb state: started enabled: true - name: Test Test_DB shell: mysql -t &#x3C; test_employees_sha.sql &#x26;&#x26; mysql -t &#x3C;&#x3C;&#x3C; 'select @@hostname as 'VM';' args: chdir: "{{ ansible_user_dir }}/test_db" register: test_db_results - name: Show results pause: prompt: "{{ test_db_results.stdout }}" seconds: 10 </code></pre> <p>Executing the playbook should reveal the test results and you can see the database is actually running on the KubeVirt VM.</p> <pre><code class="language-shell">... +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ +------------------+ | VM | +------------------+ | mariadb-kubevirt | +------------------+: ok: [mariadb-kubevirt] PLAY RECAP ************************************************************************* mariadb-kubevirt : ok=6 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>Success! The cloning process worked. Now, before finalizing with the actual volume. Let’s run a “detach” playbook to stop the database and unmount the filesystem.</p> <pre><code class="language-yaml">--- # With PVC attached to the KubeVirt VM, stop MariaDB and unmount the filesystem - hosts: kubevirt become: yes tasks: - name: Make sure MariaDB is stopped and disabled service: name: mariadb state: stopped enabled: no - name: Remove data disk mounts mount: path: "{{ var_lib_mysql }}" state: absent </code></pre> <p>In the OKD UI, detach the volume and delete the PVC.</p> <p><img src="/img/detach-disk.png" alt="" title="Detach disk from VM"></p> <pre><code class="language-shell">$ kubectl delete -f pvc-clone.yaml </code></pre> <p>Next, let’s migrate the workload permanently.</p> <h1>Finalizing</h1> <p>The last step involves simply shutting down the source vSphere VM to ensure the source volume isn’t accessed. Once the VM is properly shut down, create a PVC with the “importVolumeName” annotation to move the volume out of the vVol datastore and make it into a PV.</p> <pre><code class="language-yaml">apiVersion: v1 kind: PersistentVolumeClaim metadata: name: testdb-migrated namespace: hpe-vmi annotations: csi.hpe.com/importVolumeName: dat-mariadbl-56480669 spec: volumeMode: Block storageClassName: legacy-migration accessModes: - ReadWriteMany resources: requests: storage: 128Gi </code></pre> <p>Create the PVC.</p> <pre><code class="language-shell">$ kubectl create -f pvc-migrate.yaml </code></pre> <p>Now, repeat the step within OKD to attach the PVC as a disk to the running VM. Once attached, run the “migrate” playbook.</p> <pre><code class="language-shell">... +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ +------------------+ | VM | +------------------+ | mariadb-kubevirt | +------------------+: ok: [mariadb-kubevirt] PLAY RECAP ************************************************************************* mariadb-kubevirt : ok=6 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>You should see identical results as during the cloning procedure.</p> <p>Next, delete the source VM in vCenter and plan the next migration.</p> <h1>Learn more</h1> <p>I’ve barely scratched the surface of the HPE CSI Driver for Kubernetes features with HPE Alletra Storage MP B10000. The “allowOverrides” construct along with “importVolAsClone” and “importVolumeName” lends itself to many different use cases where data migration is one of them. Importing a “foreign” volume to Kubernetes enables sophisticated DevOps processes where terabytes of data can be made available to developers, data analysts, and testing frameworks using CI/CD pipelines.</p> <p>Visit <a href="http://scod.hpedev.io">scod.hpedev.io</a> to learn more about the HPE CSI Driver for Kubernetes and neighboring technologies. Sign up to the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack</a> to interact with HPE staff and technologists from customers and partners.</p> <p>New to KubeVirt? Check out the author’s previous blog post on <a href="https://developer.hpe.com/blog/management-paradigms-for-virtual-machines-running-on-kubernetes/">management paradigms for virtual machines running on Kubernetes</a>.</p>https://developer.hpe.com/building-the-intelligent-private-cloud-with-the-triple-o-loop/https://developer.hpe.com/building-the-intelligent-private-cloud-with-the-triple-o-loop/Tue, 22 Apr 2025 10:24:51 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>This is the second installment in our series that explores the evolution of the intelligent private cloud. From the foundation provided through virtualization to AI-driven automation, each blog post will examine a critical aspect of building a modern, efficient, and scalable private cloud infrastructure.</p> <h2>Building an intelligent private cloud</h2> <p>Today's private clouds are highly sophisticated combinations of hardware infrastructure, software, and services. They leverage advanced data observability and AI capabilities to enhance performance, optimize resource utilization, and enable as-a-service consumption models, all while ensuring robust security and compliance.</p> <p>In my <a href="https://developer.hpe.com/blog/the-rise-commoditization-and-integration-of-virtualization-into-the-private-cloud/">previous post</a>, I covered the importance of virtualization in the development of these intelligent private clouds, and how it allows for the breaking down of the infrastructure into modular, abstracted layers using open-source tools like KVM, Open vSwitch, and Ceph. However, virtualization alone doesn’t make a private cloud intelligent. That transformation requires operationalizing the cloud using what I call the Triple-O Loop.</p> <p>In this post, I’ll explore how the Triple-O loop turns a virtualized environment into an adaptive, AI-enhanced platform — delivering today’s Intelligent Private Cloud.</p> <h2>Operationalizing intelligence: Introducing the Triple-O Loop</h2> <p>At the heart of the Intelligent Private Cloud is the Triple-O Loop — a closed, continuous cycle where cloud components work together to Orchestrate, Observe, and Optimize operations. This concept is somewhat akin to the decision-making model developed by United States Air Force Colonel John Boyd, known as the OODA loop. A major difference, however, is that in the Triple-O Loop, orchestration, observation, and optimization are used in a proactive manner to drive system behaviors, as opposed to the more reactive OODA model that starts by observing the field operations environment and then reacts to field events.</p> <center><img src="/img/adly-lakhdar-tiple-o-loop-img1.png" width="410" height="421" alt="Introducing the Triple-O Loop" title="Introducing the Triple-O Loop"></center> <p>In the Triple-O Loop, each element of the loop enables the platform to take action, sense its environment, and adapt intelligently.</p> <ul> <li><strong>Orchestrate:</strong> Executes provisioning, workflows, and policy actions.</li> <li><strong>Observe:</strong> Gathers and correlates telemetry, events, and behavior.</li> <li><strong>Optimize:</strong> Uses AI to tune resources and improve performance, sustainability, and compliance.</li> </ul> <p>These elements operate not as standalone silos, but as a tightly integrated system. Each informs and enhances the others, forming the feedback engine behind a truly intelligent infrastructure.</p> <h2>Aligning elements to the private cloud</h2> <p>To visualize how these elements function within the architecture, imagine the Triple-O Loop converted into horizontal layers, with each mapped to a role in the private cloud stack:</p> <center><img src="/img/adly-lakhdar-aligning-elements-img2.png" width="479" height="382" alt="Elements to the Private Cloud" title="Elements to the Private Cloud"></center> <ul> <li><strong>Top:</strong> Optimize layer – intelligence and control plane with AI and compliance logic.</li> <li><strong>Middle:</strong> Orchestrate layer – automation and policy enforcement.</li> <li><strong>Middle:</strong> Observe layer – telemetry and system awareness.</li> <li><strong>Base:</strong> Hardware and software infrastructure.</li> </ul> <p>These layers correspond directly to the structure of a modern private cloud, from unified control and security at the top to observability and telemetry at the base. Here, the Triple-O Loop transforms the virtualized environment into an adaptive, AI-enhanced platform — delivering today’s Intelligent Private Cloud.</p> <h2>Agentic AI: Intelligence that acts</h2> <p>A defining feature of the Intelligent Private Cloud architecture is the presence of Agentic AI—intelligent agents embedded within each layer of the Triple-O Loop. These agents operate autonomously, performing crucial tasks that enhance the cloud's intelligence and responsiveness.</p> <p>These agents are capable of:</p> <ul> <li><strong>Monitoring</strong> infrastructure and application behavior continuously, ensuring that all components are functioning optimally.</li> <li><strong>Predicting</strong> failures or inefficiencies based on learned patterns, thereby preventing potential issues before they impact operations.</li> <li><strong>Taking</strong> corrective action by enforcing policies, or escalating issues without human intervention, ensuring that the cloud can self-manage and adhere to compliance requirements.</li> </ul> <p>By aligning Triple-O capabilities with architectural layers and empowering them with Agentic AI, the Intelligent Private Cloud transcends being a mere management system. It becomes a self-optimizing, mission-aware platform capable of sustaining operations in real-time, regulated, and mission-critical environments. This transformation is pivotal in creating a cloud environment that not only adapts to current conditions but also anticipates future needs and challenges.</p> <h2>The expanded loop</h2> <p>To further understand the intricacies of the Triple-O Loop, let's delve into each of its components: Orchestrate, Observe, and Optimize.</p> <h3>Orchestrate:</h3> <p>At its core, the orchestration layer executes provisioning, workflows, and policy actions. This involves automating the deployment and management of resources, ensuring that the infrastructure can scale dynamically and efficiently. Tools such as Kubernetes, Ansible and HPE Morpheus are instrumental in this layer, enabling seamless orchestration of containers, configurations, and infrastructure as code.</p> <h3>Observe:</h3> <p>The observation layer is crucial for maintaining system awareness. It gathers and correlates telemetry, events, and behavior from various components of the infrastructure. This data is then analyzed to provide insights into the performance and health of the cloud environment. Tools like Prometheus, ELK Stack, Grafana and HPE OpsRamp play a vital role in providing visibility and actionable insights.</p> <h3>Optimize:</h3> <p>The optimization layer leverages AI to fine-tune resources and improve overall cost, performance, sustainability, and compliance. By continuously analyzing data, AI algorithms can dynamically adjust resource allocation, identify inefficiencies, and implement improvements. This ensures that the cloud environment remains optimized and compliant with regulatory and sustainability standards.</p> <center><img src="/img/adly-lakhdar-agenti-ai-img3.png" width="590" height="668" alt="Agentic AI - Intelligence That Acts" title="Agentic AI - Intelligence That Acts"></center> <p>In a Venn diagram representation, the three elements — <strong>Orchestrate</strong>, <strong>Observe</strong>, and <strong>Optimize</strong> — form layered capabilities that interact continuously across the private cloud stack. Signals from observability trigger automation via orchestration, and insights generated through optimization feed both layers to refine behavior over time.</p> <p>At the center of the Venn diagram, where all three capabilities intersect, lies the autonomous, agentic platform — a zone of continuous feedback and intelligent action. Here, AI agents monitor, decide, and execute in real time, creating a living, learning system that adapts dynamically to workload, policy, and mission demands.</p> <h2>The outcome: The Intelligent Private Cloud</h2> <p>The Intelligent Private Cloud integrates open-source technologies, enterprise-grade orchestration, AI-augmented optimization, and deep observability to deliver a living infrastructure system.</p> <p>Key features include:</p> <ul> <li>A built-in data fabric that ensures seamless data management and integration across various environments.</li> <li>Integrated GPUs for AI/ML/GenAI workloads, enabling high-performance computing capabilities.</li> <li>Seamless module onboarding via orchestration, which simplifies the deployment and management of new services and applications.</li> </ul> <p>Together, these features enable hybrid, edge, and multi-cloud operations with full visibility, intelligence, and control.</p> <h2>A note on organizational alignment</h2> <p>To effectively deliver such a closely integrated private cloud, organizations must rethink their team structures. Moving away from discrete product silos and aligning teams around the three core capabilities — Orchestrate, Observe, and Optimize — can enhance agility, quality, modularity, and efficiency.</p> <p>This concept, known as the reverse Conway law, will be explored in greater detail in a subsequent blog post. Organizations that follow the reverse Conway law intentionally design their team and organizational structures to align with the desired architecture, thereby ensuring that the technology and organizational strategies are in harmony.</p> <h2>Conclusion: A living cloud</h2> <p>The Intelligent Private Cloud represents more than just infrastructure — it is a dynamic system that senses, decides, and acts. Powered by the Triple-O Loop, it revolutionizes IT operations by transitioning from reactive management to proactive, intelligent automation.</p> <p>This transformation is critical for organizations seeking to stay ahead in an increasingly complex and fast-paced digital landscape. By embracing the principles of the Intelligent Private Cloud, enterprises can achieve unprecedented levels of efficiency, agility, and resilience.</p> <p>In my next blog post, I’ll do a deep dive into observability and AI-powered insight and show you the power of OpsRamp as demonstrated within this architecture.</p>https://developer.hpe.com/beyond-generic-ai-achieve-contextual-accuracy-with-hpes-knowledge-bases/https://developer.hpe.com/beyond-generic-ai-achieve-contextual-accuracy-with-hpes-knowledge-bases/Tue, 15 Apr 2025 08:45:24 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>The demand for AI applications that deliver accurate, contextually relevant insights from enterprise data is rapidly increasing. However, the challenge of integrating fragmented, often sensitive, data into generative models remains a significant hurdle.</p> <p>To solve this, Hewlett Packard Enterprise (HPE) has introduced knowledge bases in HPE AI Essentials Software, the software foundation that makes HPE Private Cloud AI a comprehensive, user-friendly platform for enterprises seeking to deploy and scale AI solutions. This new feature provides a fully managed Retrieval Augmented Generation (RAG) experience, enabling secure and efficient connection between foundation models and internal data. This streamlined approach handles everything from vector database setup to sophisticated query handling, allowing data science professionals to create highly customized and accurate AI applications that are tailored to their specific business needs.</p> <p><img src="/img/abhipicture12-img1.png" alt="HPE AI Essentials workflow to automate connecting LLMs to enterprise data" title="HPE AI Essentials workflow to automate connecting LLMs to enterprise data"></p> <p>HPE AI Essentials simplifies the implementation of RAG by automating the critical steps involved with connecting Large Language Modules (LLM) to enterprise data. Users retain control over LLM selection and embedding model choice, while the platform manages the underlying infrastructure. HPE AI Essentials automatically handles the conversion of diverse data formats into vector embeddings and ensures efficient storage and retrieval through a managed vector database. This approach allows developers to focus on application logic, rather than the intricacies of RAG pipeline management.</p> <p>The platform manages the creation, storage, maintenance, and updates of vector embeddings, the numerical representations of semantic textual data. This automation simplifies data synchronization, allowing users to efficiently update source data. HPE AI Essentials Software provides granular control over the RAG pipeline through configurable parameters for chunking, retrieval, and response generation. This enables data science professionals to tailor a model's processing and understanding to specific use cases, resulting in improved retrieval accuracy and response coherence.</p> <p><img src="/img/abhipicture21-img2.png" alt="HPE AI Essentials: Streamlining LLM Application with Intuitive Data Source and Model Management." title="HPE AI Essentials: Streamlining LLM Application with Intuitive Data Source and Model Management."></p> <p>Automated document chunking defaults to 512-word segments, optimized for question-answering tasks. Users can further customize chunk sizes and overlaps, with a recommended 0-20% overlap for accuracy gains, while being aware of the potential for reduced relevancy with excessive overlap.</p> <p>HPE AI Essentials features a playground, a dedicated environment for interactive knowledge base exploration and management across multiple sessions. This tool enables iterative refinement of model behavior through customizable response parameters and prompt templates. Users can inject background data, define user-specific constraints, and implement detailed prompting strategies, providing the flexibility required for advanced AI development and optimization.</p> <p><img src="/img/abhipicture31-img3.png" alt="Starting a playground for knowledge base exploration and management." title="Starting a playground for knowledge base exploration and management."></p> <p><img src="/img/abhipicture42-img4-merged.png" alt="Customization of knowledge base playground components " title="Customization of knowledge base playground components "></p> <p>To support applications requiring sophisticated, data-driven workflows, knowledge bases can be accessed programmatically via dedicated endpoints within HPE AI Essentials. Secure authorization is achieved using long-lived authorization tokens, allowing for sustained interaction with these endpoints. The following code example provides a clear demonstration of endpoint usage.</p> <p><img src="/img/abhipicture51-img5.png" alt="Code to programmatically access knowledge base via endpoints" title="Code to programmatically access knowledge base via endpoints "></p> <h2>Summary</h2> <p>Leverage HPE AI Essentials knowledge bases to streamline the development of data-driven AI applications. The platform automates key RAG pipeline components, including vector embedding management and data synchronization reducing operational overhead. Data scientists can focus on application logic and customization, utilizing programmable endpoints and the interactive playground for efficient development. For implementation details and API integration, refer to the technical resources listed below.</p> <h2>Learn more:</h2> <ul> <li><strong><a href="https://www.hpe.com/psnow/product-documentation?oid=1014847366&#x26;cc=my&#x26;lc=en&#x26;jumpid=in_pdp-psnow-docs">HPE Private Cloud AI Documentation</a></strong></li> <li><strong><a href="https://hpe.com/support/PCAIUserGuide">Administration Guide</a></strong></li> <li><strong><a href="https://www.hpe.com/support/AIEDocs">HPE AI Essentials Software </a></strong></li> <li><strong><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00aie16hen_us&#x26;page=Tutorials/Tutorials/Tutorials.html">Tutorials for HPE AI Essentials on HPE Private Cloud AI</a></strong></li> <li><strong><a href="https://github.com/HPEEzmeral/aie-tutorials/tree/aie-1.7.0">GitHub Tutorials in HPE AI Essentials Software <img src="Github" alt=""></a></strong></li> <li><strong><a href="https://www.brighttalk.com/webcast/19535/640132?utm_source=HPE&#x26;utm_medium=brighttalk&#x26;utm_campaign=640132">Technical Demo Video</a></strong></li> </ul>https://developer.hpe.com/memory-safety-in-chapel/https://developer.hpe.com/memory-safety-in-chapel/Thu, 10 Apr 2025 17:49:51 GMT<p>external blog</p>https://developer.hpe.com/fueling-ai-innovation-with-hpe-data-fabric-software/https://developer.hpe.com/fueling-ai-innovation-with-hpe-data-fabric-software/Wed, 09 Apr 2025 12:00:06 GMT<p>external blog post</p>https://developer.hpe.com/2025-april-04/https://developer.hpe.com/2025-april-04/Thu, 03 Apr 2025 13:00:00 GMThttps://developer.hpe.com/announcing-chapelcon-25/https://developer.hpe.com/announcing-chapelcon-25/Mon, 31 Mar 2025 19:25:45 GMT<p>external blog</p>https://developer.hpe.com/the-rise-commoditization-and-integration-of-virtualization-into-the-private-cloud/https://developer.hpe.com/the-rise-commoditization-and-integration-of-virtualization-into-the-private-cloud/Mon, 24 Mar 2025 16:40:57 GMT<p>This is the first installment in a four-part series exploring the evolution of the intelligent private cloud. From the foundation provided through virtualization to AI-driven automation, each blog post will examine a critical aspect of building a modern, efficient, and scalable private cloud infrastructure.</p> <h2>Virtualization: The foundation of private and hybrid cloud</h2> <p>Virtualization has been a cornerstone of IT infrastructure for decades, evolving from a call for greater mainframe efficiency in the 1960s to the x86-based hypervisors of the 2000s that enabled more scalable and flexible workloads. Initially, businesses relied on costly, inflexible bare-metal servers to handle their computing needs, but virtualization transformed IT infrastructure by allowing multiple virtual machines (VMs) to run on a single system, optimizing resources and paving the way for public cloud adoption.</p> <p>By the 2000s, companies like VMware, Red Hat, and IBM helped accelerate virtualization adoption, enabling resource sharing, multi-tenancy, and on-demand services to set the stage for cloud computing. As a result, businesses moved many workloads to the public cloud, taking advantage of the ease and flexibility of Infrastructure-as-a-Service. But while public cloud services offer greater flexibility, they also introduce cost unpredictability and security concerns, leading enterprises to invest in private clouds for better control. Private clouds allow them to replicate the cloud experience while maintaining governance and compliance.</p> <p>I remember the early days of cloud adoption when developers in my team started bypassing IT to spin up workloads in the public cloud. It was called shadow IT. It was fast and easy but also introduced risks — uncontrolled spending, security blind spots, and compliance challenges. To regain control, we built our first private cloud, mimicking the public cloud experience but with enterprise-grade governance. It was a turning point. The lessons learned from that transition shaped my understanding of how virtualization and other private cloud technologies must work together to balance agility with control.</p> <h2>Virtualization and commoditization</h2> <p>Today, virtualization remains essential, supporting modern hybrid cloud strategies where VMs and containers coexist. More than just a tool for efficiency, virtualization is the backbone of private cloud architectures, ensuring scalability, security, and cost management.</p> <p>Despite the rise of containerization, virtualization remains a $100 billion+ industry, proving its enduring value.</p> <p><img src="/img/virtualization-privatecloud-image1.png" alt="Server Virtualization Software Global Market Report of 2025 from The Business Research Company" title="Server Virtualization Software Global Market Report of 2025 from The Business Research Company"></p> <p>The cloud-first (private and public) shift that virtualization initiated has led to it no longer being considered a standalone product — it is now an assumed layer of infrastructure much like networking and storage. As enterprises transition to infrastructure-as-a-service (IaaS) and platform-as-a-service (PaaS) models, virtualization remains the backbone for their self-service private cloud environments.</p> <h2>Open source as the standard, not an alternative</h2> <p>Virtualization was once considered a premium feature, but open-source innovation has commoditized it. Open-source technology now dominates the private cloud landscape. According to Gartner (2023), 70% of enterprises use open-source virtualization. Forrester (2023) agrees, stating that 80% of new virtualization deployments are open-source. As a result, proprietary virtualization markets are shrinking by 5-8% year-over-year (IDC). According to Gartner (2023), hybrid cloud adoption has reached 75%, driven by the adoption of open-source.</p> <p>HPE uses an open-core business model where the core software is open-source, while the management plane, enterprise-grade indemnification, and support offered are proprietary. This is a business model adopted by many other companies, including IBM, Red Hat, HashiCorp, and Docker. HPE adopts this model for many of its products like HPE AI Essentials, HPE Ezmeral Unified Analytics, and HPE Ezmeral Container Platform.</p> <h2>Cost: Per-socket vs. per-core</h2> <p>Licensing costs have played a significant role in virtualization’s evolution. Traditional per-core pricing penalizes hardware advancements, while per-socket pricing ensures predictable costs and higher return on investment.</p> <p>The Next Platform (2024) found that licensing costs can be reduced by 50% with single-socket, high-core CPUs. High-core CPUs also reduce power per virtual machine by more than 30%. IDC reports that per-core pricing increases costs exponentially as CPUs scale, making per-socket a preferred model.</p> <p><img src="/img/virtualization-privatecloud-image2.png" alt="Comparing per-core versus per-socket pricing models" title="Comparing per-core versus per-socket pricing models"></p> <h2>Examining the layered infrastructure: Bare metal, virtual machines, containers, and orchestration</h2> <p>Let’s explore the HPE model a bit more, where open-core virtualization acts as a building block that bridges Linux kernel technologies with the HPE hybrid cloud orchestration platform. Modern cloud architectures leverage a layered approach where bare metal, virtual machines, and containers coexist. The mixture serves to accommodate many different types of workloads. Bare metal is well suited for high-performance computing, artificial intelligence, machine learning, and low-latency workloads. Virtual machines provide security, multi-tenancy, and flexibility. Containers enable cloud-native applications and DevOps agility.</p> <p>It is worth noting that both virtualization and containerization share foundational similarities, leveraging the Linux kernel’s namespaces, cgroups, SELinux, AppArmor, iptables, systemd, and OverlayFS for isolation, security, and resource management. While virtualization relies on KVM (kernel module) with user-space QEMU, containerization uses Kubernetes, Docker, and Ceph in user space, built on the same Linux primitives.</p> <p><img src="/img/hand-drawn-k8s-larger-2.png" alt="Virtualization and containerization" title="Virtualization and containerization"></p> <p>This approach allows efficient resource allocation and high-density application hosting on high-density servers. These technologies have now been fully commoditized at the platform level, ensuring seamless integration into private cloud environments.</p> <p>With HPE’s new virtualization solution, HPE VME (VM Essentials) is an open-core virtualization building block that simultaneously integrates with the bare-metal Linux OS and the HPE private cloud orchestration platform (through the HPE VM Essentials Manager), enabling seamless workload deployment from edge to cloud in a distributed enterprise. The HPE KVM-based VM Essentials enhances virtualization with enterprise-grade cluster management and capabilities such as high availability, live migration, distributed workload placement, integrated data protection, secure hardening, and external storage support.</p> <p><img src="/img/virtualization-privatecloud-image4.png" alt="HPE VM Essentials architecture" title="HPE VM Essentials architecture"></p> <h2>Next steps</h2> <p>This blog post is the first in a series that will take us on a journey from traditional virtualization to lean, cost-efficient, AI-driven infrastructure that scales; where open-source is the default at its core, efficiency is the priority, and artificial intelligence is the future.</p> <p>My next post will explore orchestration as the engine of the modern cloud. Subsequent posts will cover observability and AI, leading to the intelligent and fully automated private cloud.</p>https://developer.hpe.com/announcing-chapel-2-4/https://developer.hpe.com/announcing-chapel-2-4/Thu, 20 Mar 2025 21:43:27 GMT<p>external blog</p>https://developer.hpe.com/no-code-integration-of-hpe-greenlake-cloud-with-servicenow/https://developer.hpe.com/no-code-integration-of-hpe-greenlake-cloud-with-servicenow/Tue, 18 Mar 2025 07:49:59 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">previous tutorial</a> on webhooks for HPE GreenLake cloud, I used a no-code/low-code platform called <a href="https://www.make.com/">Make.com</a> to implement a webhook handler. The webhook handler’s URL endpoint was then registered with HPE GreenLake cloud to subscribe to audit log events from the platform. In my simplistic use case, I stored these events in a Google Sheet as they arrived.</p> <p>For this handler, I was responsible for taking care of the handshake initiated by HPE GreenLake cloud when the new webhook was registered. This is a security feature that uses a secret key shared by HPE GreenLake cloud and the webhook handler. The following diagram illustrates the mechanics involved in this process.</p> <p><img src="/img/slide-pour-blog-webhooks.jpg" alt="HPE Greenlake cloud event framework" title="HPE Greenlake cloud event framework"></p> <p>As this implementation was working well, I thought I would reuse this low-code handler technique to demonstrate another use case of an integration with an IT service management (ITSM) platform. I decided to use ServiceNow because it’s a very popular ITSM platform and because it provides a very nice way for developers to fire up a <a href="https://devportaluat.service-now.com/dev.do#!/learn/learning-plans/washingtondc/new_to_servicenow/app_store_learnv2_buildmyfirstapp_washingtondc_personal_developer_instances">personal developer instance</a> (PDI) and get their work done.</p> <h2>Problems, incidents and change orders</h2> <p>Most of these ITSM platforms are managed through the lifecycle of different types of tickets (New, In Progress, Resolved). One of the most common ticket types is called <em>Incident</em>, which is the type of ticket that users might open when they face an issue and want to raise it to their IT team for resolution. In most cases, end users use a web portal to do this. However, there is also an API to programmatically do it, and this is exactly what I’m going to use in this blog.</p> <h2>What’s the right API then?</h2> <p>To be fair, I must admit that I initially reached out to ChatGPT to get started, asking it <em>“how do I create an incident in ServiceNow?”</em>. To my surprise it returned a short Python script that, once edited to use my ServiceNow personal developer instance, my username and password, worked immediately. This confirmed my choice of selecting ServiceNow.</p> <p>Here is the Python script returned by ChatGPT:</p> <pre><code class="language-python">import requests import json # Replace these variables with your own information instance_url = "https://your-instance.service-now.com" username = "your-username" password = "your-password" service_id = "service-id" # The Service ID related to the incident # Define the API endpoint for creating an incident url = f"{instance_url}/api/now/table/incident" # Create the headers for the request (set the content-type to application/json) headers = { "Content-Type": "application/json", "Accept": "application/json" } # Define the data to be sent in the request body data = { "short_description": "Issue with email service", "description": "The email service is down and needs urgent attention.", "category": "Inquiry", "impact": "3", # 1=High, 2=Medium, 3=Low (you can adjust accordingly) "urgency": "2", # 1=High, 2=Medium, 3=Low (you can adjust accordingly) "service": service_id # The ID of the service being impacted } # Make the POST request to create the incident response = requests.post(url, auth=(username, password), headers=headers, data=json.dumps(data)) # Check the response if response.status_code == 201: print(f"Incident created successfully! Incident Number: {response.json()['result']['number']}") else: print(f"Failed to create incident. Status Code: {response.status_code}") print(f"Response: {response.text}") </code></pre> <blockquote> <blockquote> <p><strong>Note</strong>: ServiceNow provides the full documentation for their API once you sign up as a ServiceNow developer and it is free.</p> </blockquote> </blockquote> <p>The script it provided was very helpful because it helped me understand:</p> <ul> <li>The API I need to use is the table API and the table name to be used is called incident, thus the API call: POST /api/now/table/incident</li> <li>The (minimal) payload I need to pass in the API call</li> <li>The necessary headers</li> <li>The response code expected if it worked should be 201</li> </ul> <h2>From event to incident</h2> <p>Now that I understand how to create an incident, it would be good to think about what represents an incident within HPE GreenLake cloud. In my <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">previous tutorial</a>, I subscribed to audit log events (workspace loaded by user X, user Y logged out, user Z changed these settings, etc.) but you most likely would not want to open an incident for each of these. First, they are not really incidents, and second, you don’t want to flood your incident table with inappropriate data, as real incidents might get lost. For this use case, I decided to use another event that could be triggered by the <strong>Subscriptions Management</strong> service in HPE GreenLake cloud when a subscription is expiring in 1, 30, 60, and 90 days. The event is called <strong>Expiring Subscriptions</strong>. The full description of this event can be found on the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/subscription-management/public/catalog/subscriptions-unified-events-glcp-v1/paths/Expiring%20Subscriptions/post/">HPE GreenLake Developer Portal</a>.</p> <p>This can be considered an incident and could be added in ServiceNow incident table.</p> <p>The payload sent to the webhook handler when this event is triggered by the Subscriptions Management service has the following format:</p> <pre><code class="language-json">{ "specversion": "1.0", "id": "fb25f344-5e20-4f13-9937-c3ecc0327dc3", "source": "https://global.api.greenlake.hpe.com/subscriptions", "type": "com.hpe.greenlake.subscriptions.v1.expiring-subscriptions", "datacontenttype": "application/json", "time": "2024-10-30T21:09:16.127443806Z", "data": { "key": "OFKIAASTEST168IZ", "expirydate": "10/25/2024", "sku": "S0B80AAE", "licensetier": "ENHANCED PROLIANT", "quantity": 500, "username": "John Doe Inc.", "subscriptionendinsecs": 1729848600, "subscriptiontype": "Compute Subscription", "producttype": "DEVICE", "platformcustomerid": "22e7e552184511ef88e9cacb36fa7032", "devices": [] } } </code></pre> <p>Now that I know what my webhook handler will receive as input and how I can open an incident in ServiceNow, I can get started.</p> <h2>From code to no-code</h2> <p>Let me show you how to translate these learnings in Make.com. In my previous <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">article</a>, I created a scenario in Make.com that looked like this:</p> <p><img src="/img/second-route-taken.jpg" alt="Original scenario in make " title="Original scenario in make "></p> <p>The topmost branch was used to handle the initial security handshake. I will keep that intact. If you need details about this branch, check out my <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">previous tutorial</a>.</p> <p>What I will do now is duplicate this scenario, delete the Google Sheets module, and replace it with another one called <strong>HTTP Make a request</strong>. Then I will link the new module to the Webhook response.</p> <p>The HTTP Make a request module allows you to make any type of REST API call. It comes in very handy for integration use cases.</p> <blockquote> <blockquote> <p><strong>Note</strong>: Make.com also provides a native ServiceNow integration, but you need an Enterprise plan to use it, which I don’t have.</p> </blockquote> </blockquote> <p>My scenario looks like this:</p> <p><img src="/img/webhook-blog-servicenow-picture-2.jpg" alt="New scenario in make" title="New scenario in make"></p> <p>My new scenario includes:</p> <ul> <li>The <strong>Custom webhook</strong> module as a trigger of your webhook scenario with <em>advanced option</em> enabled</li> <li>The <strong>Set variable</strong> module and the <strong>Webhook response</strong> module to validate the verification challenge and ensure the webhook communication between HPE GreenLake cloud and the webhook handler is authenticated</li> <li>A <strong>Router</strong> module in the scenario to branch your flow into several routes and process the data within each route differently, based on the type of event received</li> <li>The <strong>HTTP Make a request</strong> module connected to the bottom branch of the Router module along with the <strong>Webhook response</strong> module connected to the HTTP Make a request module to create the incident in ServiceNow and return a 200 status to HPE GreenLake cloud</li> </ul> <p>For more information, you can refer to the chapter <em>“Getting started with Make”</em> in my first <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">blog post</a>.</p> <p>Next, I’ll dive into the properties of the <strong>HTTP Make a request</strong> module that needs to be configured based on the details collected from the Python script:</p> <ul> <li>The URL endpoint of my ServiceNow instance: <em>https://{your-instance}/api/now/table/incident</em></li> <li>Method: <em>POST</em></li> <li>Two custom headers for Accept and Content-Type both set to <em>application/json</em></li> <li>Body type: <em>Raw</em></li> </ul> <p><img src="/img/webhook-blog-servicenow-picture-3.jpg" alt="Setting up HTTP Make a request properties - part 1" title="Setting up HTTP Make a request properties - part 1"></p> <p>To setup the JSON payload as shown below, I need to first configure the first step of my webhook scenario to use the JSON payload of the <strong>Expiring Subscriptions</strong> event described earlier.</p> <p>I then need to set up the JSON payload (<strong>Request content</strong> as shown below) with ServiceNow information such as:</p> <ul> <li>The <em>short description</em> (for example, the subscription type is expiring soon)</li> <li>The <em>description</em> (for example, License <license type> expires on <date>)</li> <li>The <em>category</em>, and the <em>subcategory</em></li> <li>The level of <em>urgency</em> (1=high, 2=medium, 3=low) and the level of the <em>impact</em> (1=high, 2=medium, 3=low) which will be used to compute <em>priority</em> in ServiceNow</li> <li>The <em>caller_id</em></li> <li>The <em>service</em></li> </ul> <p>Once this is in place, I can build the Request content and drag/drop items from the JSON input payload (shown in red) into the Request content of the HTTP call as shown below:</p> <p><img src="/img/webhook-blog-servicenow-picture-4.jpeg" alt="Setting up HTTP Make a request properties - part 2" title="Setting up HTTP Make a request properties - part 2"></p> <p>It’s important not to forget to specify username and password obtained when creating a ServiceNow personal developer instance.</p> <h2>Putting it all together</h2> <p>Voila! Now the Webhook scenario is in place. I need to subscribe to the event with HPE GreenLake cloud. As noted in my first <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/">blog</a> post, I can apply the same logic, except that this time I will subscribe to <strong>Expiring Subscriptions</strong> event as shown below:</p> <p><img src="/img/webhook-blog-servicenow-picture-5.jpg" alt="Subscribing to expiring subscriptions" title="Subscribing to expiring subscriptions"></p> <p>If everything goes well, I should see the following subscribed event configuration:</p> <p><img src="/img/webhook-blog-servicenow-picture-6.jpg" alt="Webhook armed" title="Webhook armed"></p> <p>First, let’s take a look at the open incidents page of the ServiceNow console:</p> <p><img src="/img/webhook-blog-servicenow-picture-7.jpg" alt="ServiceNow console before event received" title="ServiceNow console before event received"></p> <p>Once a subscription reaches an expiration threshold, HPE GreenLake cloud will send an event to my webhook handler, which will, in turn, open a new incident to track it in ServiceNow, as shown below:</p> <p><img src="/img/webhook-blog-servicenow-picture-8.jpg" alt="ServiceNow console after event received" title="ServiceNow console after event received"></p> <p>The new incident, visible at the top of the list above, was automatically opened by the webhook handler. I can open it to check out the details that were set up by the webhook handler:</p> <p><img src="/img/webhook-blog-servicenow-picture-9.jpg" alt="Detail of incident created" title="Detail of incident created"></p> <p>This HPE GreenLake cloud event is now an incident in ServiceNow. It will continue its lifecycle until its closure in ServiceNow.</p> <h2>Call to action</h2> <p>Webhooks together with the HPE GreenLake cloud events framework provide a great way to integrate with HPE GreenLake cloud using modern technology. They provide great flexibility on where you can run the subscriber code and allow you to choose the language in which to write the webhook code. They also allow you to build a tight integration with an existing platform such as ServiceNow or HPE OpsRamp.</p> <p>Additional benefits I can see from this technique are:</p> <ul> <li>No need to verify that a polling code is still running</li> <li>No risk of API token expiration</li> <li>No loss of events</li> <li>It’s a well-established industry standard mechanism</li> <li>There is a huge choice of implementation methods including low code/no code ones</li> </ul> <blockquote> <blockquote> <p><strong>Note</strong>: While a low-code/no-code approach was taken in this tutorial, the same logic applies to any other programming language selected.</p> </blockquote> </blockquote> <p>You can request access to this feature by signing up to the Automations/Webhook Access Beta Program <a href="https://app.smartsheet.com/b/form/0e61e8c2bd6d48c7829845ab824c11d6">here</a>.</p>https://developer.hpe.com/using-structured-outputs-in-vllm/https://developer.hpe.com/using-structured-outputs-in-vllm/Sun, 16 Mar 2025 19:28:00 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Generating predictable and reliable outputs from large language models (LLMs) can be challenging, especially when those outputs need to integrate seamlessly with downstream systems. Structured outputs solve this problem by enforcing specific formats, such as JSON, regex patterns, or even formal grammars. vLLM, an open source inference and serving engine for LLMs, has supported structured outputs for a while. However, there is little documentation on how to use it. This is why I decided to contribute and write the <a href="https://docs.vllm.ai/en/latest/usage/structured_outputs.html">Structured Outputs documentation page</a>.</p> <p>In this blog post, I'll explain how structured outputs work in vLLM and walk you through how to use them effectively.</p> <h2>Why structured outputs?</h2> <p>LLMs are incredibly powerful, but their outputs can be inconsistent when a specific format is required. Structured outputs address this issue by restricting the model’s generated text to adhere to predefined rules or formats, ensuring:</p> <ol> <li><strong>Reliability:</strong> Outputs are predictable and machine-readable.</li> <li><strong>Compatibility:</strong> Seamless integration with APIs, databases, or other systems.</li> <li><strong>Efficiency:</strong> No need for extensive post-processing to validate or fix outputs.</li> </ol> <p>Imagine there is an external system which receives a JSON object with the all the details to trigger an alert, and you want your LLM-based system to be able to use it. Of course you could try to explain the LLM what should be the output format and that it must be a valid JSON object, but LLMs are not deterministic and thus you may end up with an invalid JSON. Probably, if you have tried to do something like this before, you would have found yourself in this situation.</p> <p>How do these tools work? The idea behind them is to filter a list of possible next tokens to force a valid token to be generated that produces the desired output format, for example, a valid JSON object.</p> <p><img src="/img/structured_outputs_thumbnail.png" alt="Structured outputs in vLLM" title="Structured outputs in vLLM"></p> <h2>What is vLLM?</h2> <p>vLLM is a state-of-the-art, open-source inference and serving engine for LLMs. It’s built for performance and simplicity, offering:</p> <ul> <li><strong>PagedAttention:</strong> An innovative memory management mechanism for efficient attention key-value handling.</li> <li><strong>Continuous batching:</strong> Supports concurrent requests dynamically.</li> <li><strong>Advanced optimizations:</strong> Includes features like quantization, speculative decoding, and CUDA graphs.</li> </ul> <p>These optimizations make vLLM one of the fastest and most versatile engines for production environments.</p> <h2>Structured outputs on vLLM</h2> <p>vLLM extends the OpenAI API with additional parameters to enable structured outputs. These include:</p> <ul> <li><strong>guided_choice:</strong> Restricts output to a set of predefined choices.</li> <li><strong>guided_regex:</strong> Ensures outputs match a given regex pattern.</li> <li><strong>guided_json:</strong> Validates outputs against a JSON schema.</li> <li><strong>guided_grammar:</strong> Enforces structure using context-free grammars.</li> </ul> <p>Here’s how each works, along with example outputs:</p> <h3><strong>1. Guided choice</strong></h3> <p>Guided choice is the simplest form of structured output. It ensures the response is one from of a set of predefined options.</p> <pre><code class="language-python">from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="-") completion = client.chat.completions.create( model="Qwen/Qwen2.5-3B-Instruct", messages=[ {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"} ], extra_body={"guided_choice": ["positive", "negative"]}, ) print(completion.choices[0].message.content) </code></pre> <p><strong>Example output:</strong></p> <pre><code>positive </code></pre> <h3><strong>2. Guided Regex</strong></h3> <p>A guided regex constrains the output to match a regex pattern, which is useful for formats like email addresses.</p> <pre><code class="language-python">completion = client.chat.completions.create( model="Qwen/Qwen2.5-3B-Instruct", messages=[ { "role": "user", "content": "Generate an example email address for Alan Turing at Enigma. End in .com.", } ], extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]}, ) print(completion.choices[0].message.content) </code></pre> <p><strong>Example output:</strong></p> <pre><code>alan.turing@enigma.com </code></pre> <h3><strong>3. Guided JSON</strong></h3> <p>Guided JSON enforces a valid JSON format based on a schema, simplifying integration with other systems.</p> <pre><code class="language-python">from pydantic import BaseModel from enum import Enum class CarType(str, Enum): sedan = "sedan" suv = "SUV" truck = "Truck" coupe = "Coupe" class CarDescription(BaseModel): brand: str model: str car_type: CarType json_schema = CarDescription.model_json_schema() completion = client.chat.completions.create( model="Qwen/Qwen2.5-3B-Instruct", messages=[ {"role": "user", "content": "Generate a JSON for the most iconic car from the 90s."} ], extra_body={"guided_json": json_schema}, ) print(completion.choices[0].message.content) </code></pre> <p><strong>Example output:</strong></p> <pre><code class="language-json">{ "brand": "Toyota", "model": "Supra", "car_type": "coupe" } </code></pre> <h3><strong>4. Guided grammar</strong></h3> <p>Guided grammar uses an Extended Backus-Naur Form (EBNF) grammar syntax to define complex output structures, such as SQL queries.</p> <pre><code class="language-python">completion = client.chat.completions.create( model="Qwen/Qwen2.5-3B-Instruct", messages=[ {"role": "user", "content": "Generate a SQL query to find all users older than 30."} ], extra_body={ "guided_grammar": """ query ::= "SELECT" fields "FROM users WHERE" condition; fields ::= "name, age" | "*"; condition ::= "age >" number; number ::= [0-9]+; """ }, ) print(completion.choices[0].message.content) </code></pre> <p><strong>Example output:</strong></p> <pre><code class="language-sql">SELECT * FROM users WHERE age > 30; </code></pre> <h2><strong>Next steps</strong></h2> <p>To start integrating structured outputs into your projects:</p> <ol> <li><strong>Explore the documentation:</strong> Check out the <a href="https://docs.vllm.ai/en/latest/">official documentation</a> for more examples and detailed explanations.</li> <li><strong>Install vLLM locally:</strong> Set up the inference server on your local machine using the <a href="https://github.com/vllm-project/vllm">vLLM GitHub repository</a>.</li> <li><strong>Experiment with structured outputs:</strong> Try out different formats (choice, regex, JSON, grammar) and observe how they can simplify your workflow.</li> <li><strong>Deploy in production:</strong> Once comfortable, deploy vLLM to your production environment and integrate it with your applications.</li> </ol> <p>Structured outputs make LLMs not only powerful but also practical for real-world applications. Dive in and see what you can build!</p>https://developer.hpe.com/management-paradigms-for-virtual-machines-running-on-kubernetes/https://developer.hpe.com/management-paradigms-for-virtual-machines-running-on-kubernetes/Wed, 12 Mar 2025 20:13:53 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>With the rise of virtual machine containerization, it’s imperative to familiarize ourselves with the different aspects of performing VM management on Kubernetes. From crude CLIs, to declarative GitOps patterns, and further extending to lush UIs where your next VM is just a right-click away, having a handle on each of these disciplines is essential for Kubernetes VM management regardless of which role you're in.</p> <p>Whether you're a classic sysadmin, site reliability engineer (SRE) or have any kind of role in IT operations touching virtualization, the winds of change are catching up. Collectively we need to re-evaluate the VM estate, understand platform requirements for mission-critical applications and look for alternatives with the least amount of friction and resistance to ease migration.</p> <p>KubeVirt, an open source project governed by the Cloud-Native Computing Foundation (CNCF), is an add-on for Kubernetes that allows management of virtual machines alongside containers using a single API endpoint. KubeVirt is where a large chunk of the market is gravitating towards, whether the abstractions are disguised by a glossy frontend or deployed manually on existing Kubernetes clusters, KubeVirt needs to be considered for any new virtualization project.</p> <p>This blog post brushes over the basics in VM management on KubeVirt covering the most common patterns to give you an idea of what tools and processes to adopt in your organization, like declarative CLIs, imperative GUIs or idempotent IT platform automation tools such as Ansible. There are strengths and weaknesses across the different interfaces but understanding how to operate them is fundamental for any VM management journey with KubeVirt.</p> <p>But first, let me give you a brief introduction to KubeVirt.</p> <h1>A KubeVirt crash course</h1> <p>KubeVirt provides abstractions to Kubernetes users for Linux Kernel Virtual Machines (KVM). KVM has been around for about two decades now with several successful commercial hypervisors built around the implementation and is, at this point, considered mature.</p> <p>KubeVirt itself does not have a user interface that most VM administrators are used to. The point of abstraction is through standard Kubernetes tools by manipulating API resources of different <code>Kinds</code> provided by <code>CustomResourceDefinitions</code> (CRDs).</p> <p>The <code>CRDs</code> allow users to manage VM resources through a set of KubeVirt’s controllers.</p> <p><img src="/img/kubevirt.png" alt=""></p> <p>Deploying KubeVirt on upstream Kubernetes and other distributions is straightforward. The <a href="https://kubevirt.io/user-guide/">official documentation</a> walks through the different distributions and platform specific quirks that needs to be considered.</p> <p>The examples below use KubeVirt provided by the KubeVirt HyperConverged Cluster Operator installed on OKD, the community distribution of Kubernetes that powers Red Hat OpenShift.</p> <p>Most VM administrators connect VMs to existing networks that assign IP addresses and DNS names. Having the VM immediately reachable from your desktop computer or other already established infrastructure management tools makes the transition from legacy VM management platforms to KubeVirt much smoother.</p> <p>As a prerequisite for this exercise and examples, the following resources have been created prior:</p> <ul> <li>An SSH public key has been created on the cluster as a <code>Secret</code> to be injected into the VM instance during initialization.</li> <li>A <code>NodeNetworkConfigurationPolicy</code> using the Kubernetes NMState Operator that creates a bridge on a NIC connected to the data center management network.</li> <li>A <code>NetworkAttachmentDefinition</code> in my VM instance <code>Namespace</code> to connect virtual NICs to.</li> </ul> <p>For the sake of completeness, this is what those resources look like:</p> <pre><code class="language-yaml">apiVersion: v1 kind: Namespace metadata: name: hpe-vmi --- apiVersion: v1 kind: Secret metadata: name: desktop namespace: hpe-vmi stringData: key: ssh-rsa &#x3C;public key string> you@yourdesktop --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br0-ens224 spec: nodeSelector: node-role.kubernetes.io/worker: "" maxUnavailable: 3 desiredState: interfaces: - name: br0 type: linux-bridge state: up ipv4: enabled: false bridge: options: stp: enabled: false port: - name: ens224 --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: mgmt namespace: hpe-vmi annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br0 spec: config: | { "cniVersion": "0.3.1", "name": "mgmt", "type": "cnv-bridge", "bridge": "br0", "ipam": {}, "macspoofchk": true, "preserveDefaultVlan": false } </code></pre> <p>Another essential prerequisite is that a <code>StorageClass</code> exists on the cluster that supports KubeVirt. The examples below use the HPE CSI Driver for Kubernetes but that could be swapped out for any vendor or platform supporting the bare minimum requirements for KubeVirt (see the KubeVirt <a href="https://kubevirt.io/user-guide/storage/clone_api/">admin guide</a> for details).</p> <p>Now that the environment is primed, let’s provision a VM and take KubeVirt for a spin.</p> <h1>The command line interface</h1> <p>It is entirely possible to use <code>kubectl</code> out-of-the-box to deploy and manage VM resources. The <code>virtctl</code> CLI features a more rich experience with the ability to upload disk images, connect to the VM console and manage power states more easily. The most important task of <code>virtctl</code> is to render tedious manifests from just a few arguments to deploy new VMs.</p> <p>Installing <code>virtctl</code> varies by platform and KubeVirt distribution. It’s advised at this point to have the same client and server version, which at the time of writing is 1.4.0. If using a Mac with Brew installed, it’s simply:</p> <pre><code class="language-shell">brew install virtctl </code></pre> <p>First we need to inform ourselves what <code>DataSources</code> are available on the cluster. Building new <code>DataSources</code> or importing new ones is out of scope for this blog post. VMs are cloned into new <code>PersistentVolumeClaims</code> (PVCs) from <code>DataSources</code>. List the existing the <code>DataSources</code> on the cluster:</p> <pre><code class="language-shell">$ kubectl get datasources -A NAMESPACE NAME AGE kubevirt-os-images centos-stream8 13h kubevirt-os-images centos-stream9 13h kubevirt-os-images centos6 13h kubevirt-os-images centos7 13h kubevirt-os-images fedora 13h kubevirt-os-images opensuse 13h kubevirt-os-images rhel7 13h kubevirt-os-images rhel8 13h kubevirt-os-images rhel9 13h kubevirt-os-images ubuntu 13h kubevirt-os-images win10 13h kubevirt-os-images win11 13h kubevirt-os-images win2k16 13h kubevirt-os-images win2k19 13h kubevirt-os-images win2k22 13h </code></pre> <p>Not all <code>DataSources</code> are populated by default. On OKD, only “fedora” and “centos-stream9” are available. It can be checked by examining <code>DataImportCrons</code>.</p> <pre><code class="language-shell">$ kubectl get dataimportcrons -A NAMESPACE NAME FORMAT kubevirt-os-images centos-stream9-image-cron pvc kubevirt-os-images fedora-image-cron pvc </code></pre> <p>Let’s create a Fedora VM, assign the SSH public key and connect it to the management LAN, but first create a manifest named “my-network.yaml” to describe the network we want to connect the VM to.</p> <pre><code class="language-yaml">spec: template: spec: domain: devices: interfaces: - bridge: {} model: virtio name: my-vnic-0 networks: - multus: networkName: mgmt name: my-vnic-0 </code></pre> <p>Now, create the VM and attach it to the network:</p> <pre><code class="language-shell">virtctl create vm --name my-vm-0 \ --access-cred type:ssh,src:desktop,user:fedora \ --volume-import=type:ds,src:kubevirt-os-images/fedora,size:64Gi \ | kubectl create -n hpe-vmi -f- &#x26;&#x26; \ kubectl patch vm/my-vm-0 -n hpe-vmi --type=merge --patch-file my-network.yaml </code></pre> <p>Monitor the progress of the VM:</p> <pre><code class="language-shell">$ kubectl get vm -n hpe-vmi -w NAME AGE STATUS READY my-vm-0 13s Provisioning False my-vm-0 29s Starting False my-vm-0 42s Running False my-vm-0 42s Running True </code></pre> <p>Once the VM is running, it’s possible to login with the SSH identity and hostname given to the VM (assuming DHCP registers the hostname in DNS on the management network).</p> <pre><code class="language-shell">$ ssh fedora@my-vm-0 [fedora@my-vm-0 ~]$ </code></pre> <p>So, what does the VM instance actually look like? Let’s install some tools and inspect.</p> <pre><code class="language-shell">$ sudo dnf install -yq fastfetch virt-what $ sudo virt-what redhat kvm $ fastfetch --pipe --localip-show-ipv4 false .',;::::;,'. fedora@my-vm-0 .';:cccccccccccc:;,. -------------- .;cccccccccccccccccccccc;. OS: Fedora Linux 41 (Cloud Edition) x86_64 .:cccccccccccccccccccccccccc:. Host: KubeVirt (RHEL-9.4.0 PC (Q35 + ICH9, 2009)) .;ccccccccccccc;.:dddl:.;ccccccc;. Kernel: Linux 6.11.4-301.fc41.x86_64 .:ccccccccccccc;OWMKOOXMWd;ccccccc:. Uptime: 7 mins .:ccccccccccccc;KMMc;cc;xMMc;ccccccc:. Packages: 550 (rpm) ,cccccccccccccc;MMM.;cc;;WW:;cccccccc, Shell: bash 5.2.32 :cccccccccccccc;MMM.;cccccccccccccccc: Terminal: /dev/pts/0 :ccccccc;oxOOOo;MMM000k.;cccccccccccc: CPU: Intel Core (Haswell, no TSX, IBRS) @ 2.60 GHz cccccc;0MMKxdd:;MMMkddc.;cccccccccccc; GPU: Unknown Device 1111 (VGA compatible) ccccc;XMO';cccc;MMM.;cccccccccccccccc' Memory: 435.27 MiB / 3.80 GiB (11%) ccccc;MMo;ccccc;MMW.;ccccccccccccccc; Swap: 0 B / 3.80 GiB (0%) ccccc;0MNc.ccc.xMMd;ccccccccccccccc; Disk (/): 805.66 MiB / 62.92 GiB (1%) - btrfs cccccc;dNMWXXXWM0:;cccccccccccccc:, Locale: en_US.UTF-8 cccccccc;.:odl:.;cccccccccccccc:,. ccccccccccccccccccccccccccccc:'. :ccccccccccccccccccccccc:;,.. ':cccccccccccccccc::;,. </code></pre> <p>Except for the “Host” hint, this looks like any VM instance on a KVM hypervisor.</p> <p>With <code>virtctl</code> it’s possible to live migrate, pause/unpause, stop/start and restart the VM. Deleting the VM requires <code>kubectl</code>.</p> <pre><code class="language-shell">kubectl delete -n hpe-vmi vm/my-vm-0 </code></pre> <p>This will remove all resources created with <code>virtctl</code>, including <code>PVCs</code>.</p> <h1>User experience with web user interfaces</h1> <p>KubeVirt does not have an official graphical user interface. That is a tall threshold for new users who are familiar with legacy VM management solutions where everything is a right-click away, structured in an intuitive manner. In a way, the KubeVirt project assumes that the user has a fundamental knowledge of KVM and can scrape through by managing Kubernetes resources through the CLI.</p> <p>Fortunately, there are KubeVirt implementations that heavily focus on a graphical user experience and provide a great way to learn and explore the capabilities, very similar to legacy hypervisors.</p> <p>Let's take a closer look at OKD, the upstream Kubernetes distribution of OpenShift, and Harvester, an Hyper Converged Infrastructure (HCI) solution built for VMs on KubeVirt with striking simplicity.</p> <p><img src="/img/screenshot-2025-03-05-at-12.18.46-pm.png" alt="OKD Virtualization landing page." title="OKD Virtualization landing page."></p> <p>OKD is the upstream open source project of Red Hat OpenShift. Enabling virtualization is a two-click operation and considered the gold standard for managing VMs and containers with a unified control plane. KubeVirt has been part of OKD and OpenShift since 2020.</p> <p><img src="/img/screenshot-2025-03-05-at-12.48.03-pm.png" alt="Harvester landing page" title="Harvester landing page"></p> <p>Harvester is an open source HCI solution primarily focused on running a highly opinionated stack of software and tools on Kubernetes designed solely for running VMs. Harvester can be consumed by Rancher to allow Rancher to deploy and manage Kubernetes clusters on Harvester in a symbiotic relationship.</p> <p>Walking through the UIs are out of scope for this blog post but the same outcomes can be accomplished in a few clicks similar to using the CLI with <code>virtctl</code> and <code>kubectl</code>.</p> <h1>Ansible</h1> <p>Using CLIs and graphical UIs are great for exploratory administration and one-offs. They’re usually tedious and error prone when it comes to repeating the same set of tasks indefinitely. This is where Ansible comes it. Idempotent and declarative interfaces allow it to distill very complex tasks across multiple layers of infrastructure to gain full control all the way up to deploying the application. This kind of IT automation lends itself to GitOps and self-service patterns in large scale environments. Write once, delegate and reuse with ease, like cookie cutter templates.</p> <p>Ansible has historically been well integrated with other KVM-based hypervisors such as oVirt/RHEV and provides VM management at scale quite elegantly.</p> <p>Ansible can be installed on your desktop computer in a multitude of ways and will not be covered in this blog. Once Ansible is in place, install the KubeVirt collection:</p> <pre><code class="language-shell">ansible-galaxy collection install kubevirt.core </code></pre> <p>There are a couple of distinct patterns for managing cloud compute instances (VMs on KubeVirt in this case) with Ansible.</p> <ul> <li>Declaratively CRUD (Create, Read, Update Delete) the instances from a pre-rendered inventory, preferable templatized with Ansible, idempotent with desired parameters. Manage the OS and apps with playbooks using the rendered inventory.</li> <li>Imperatively CRUD the instances with some other tooling, either from the cloud provider directly or idempotent with something like OpenTofu. Employ dynamic inventory plugins to manage the OS and apps inside the instances.</li> <li>Imperatively CRUD the instances with Ansible playbooks and using a dynamic inventory plugin to manage OS and apps.</li> </ul> <p>For the sake of simplicity and clarity the examples will imperatively CRUD the instances and showcase the dynamic inventory plugin with KubeVirt. In a production scenario where collaboration among engineers is required, the first option is the more elegant choice.</p> <p>Create a playbook named “create_vm.yaml” or similar.</p> <pre><code class="language-yaml">--- - hosts: localhost connection: local tasks: - name: Ensure VM name assert: that: vm is defined - name: Create a VM kubevirt.core.kubevirt_vm: state: present name: "{{ vm }}" namespace: hpe-vmi labels: app: my-example-label instancetype: name: u1.medium preference: name: fedora data_volume_templates: - metadata: name: "{{ vm }}-0" spec: sourceRef: kind: DataSource name: fedora namespace: kubevirt-os-images storage: resources: requests: storage: 64Gi spec: domain: devices: interfaces: - name: mgmt bridge: {} networks: - name: mgmt multus: networkName: mgmt accessCredentials: - sshPublicKey: propagationMethod: qemuGuestAgent: users: - fedora source: secret: secretName: desktop volumes: - cloudInitConfigDrive: userData: |- #cloud-config # The default username is: fedora runcmd: - [ setsebool, -P, 'virt_qemu_ga_manage_ssh', 'on' ] name: cloudinitdisk - dataVolume: name: "{{ vm }}-0" name: "{{ vm }}-0" wait: yes </code></pre> <p>Many attributes have been hardcoded in this example but it illustrates the similarities of what <code>virtctl</code> outputs based on the parameters provided.</p> <p>Use the playbook to create a VM:</p> <pre><code class="language-shell">ansible-playbook -e vm=my-vm-0 create_vm.yaml </code></pre> <p>It takes a minute or so for the VM to come up. When the prompt comes back, create a file named “hosts.kubevirt.yaml” (the “kubevirt.yaml” part of the filename is mandatory):</p> <pre><code class="language-yaml">plugin: kubevirt.core.kubevirt namespaces: - hpe-vmi host_format: "{name}" network_name: mgmt label_selector: app=my-example-label compose: ansible_user: "'fedora'" </code></pre> <p>It’s now possible to use the KubeVirt inventory plugin to manage the OS and apps in the VM. Let’s see if it connects:</p> <pre><code class="language-shell">ansible -i hosts.kubevirt.yaml -m ping my-vm-0 my-vm-0 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3.13" }, "changed": false, "ping": "pong" } </code></pre> <p>At this point it’s possible to manage the VM like any other host provisioned on any kind of server, hypervisor or cloud platform.</p> <h1>Summary</h1> <p>It doesn’t matter what your distinct VM management workflow looks like, KubeVirt serves all popular patterns. That said, current tools and processes will require an overhaul. Why not switch to idempotent VM management through GitOps while transitioning from your legacy hypervisor in the meantime? That's a topic for another day.</p> <p>Connect with the HPE Developer Community via <a href="https://developer.hpe.com/slack-signup/">Slack</a> or sign up for the <a href="https://developer.hpe.com/campaign/munch-and-learn/">Munch &#x26; Learn Technology Talks</a> to immerse yourself in the latest breakthrough technologies from HPE, customers, and partners.</p>https://developer.hpe.com/sivabala-opsql-opsramps-kalpavriksha-wish-fulfilling-tree/https://developer.hpe.com/sivabala-opsql-opsramps-kalpavriksha-wish-fulfilling-tree/Mon, 10 Mar 2025 21:32:15 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In today's data centers, IT Ops managers face mounting challenges in managing and retrieving data efficiently. They've introduced so many network, storage, and computing devices into their environment that they have become unmanageable. At times performance issues and critical services going down keep happening and they're left with headaches, higher costs, and poor performance. If only they had a Kalpavriksha, a magical wish-fulfilling tree that could answer their needs for quickly searching for any resource from their complex, wide data centre to make all their dreams come true.</p> <p>With OpsRamp, a comprehensive IT operations management platform, they may very well get their wish. With its OpsRamp Query Language (OpsQL) and its powerful API, users can perform complex searches within the OpsRamp platform. It allows users to retrieve specific data based on various attributes and conditions. To quickly access and manipulate data in order to maintain optimal system performance and resolve issues, OpsQL is a Kalpavriksha that truly grants wishes. In this blog post, I will explain the basics of OpsQL and how to use it.</p> <h2>What is OpsQL?</h2> <p>OpsQL is a flexible and powerful query language to search objects within the OpsRamp platform. It allows users to retrieve specific data based on various attributes and conditions. OpsQL is essential for IT administrators to quickly access and manipulate data, in order to maintain optimal system performance and resolve issues. IT Ops Managers, users can run OpsQL in intuitive OpsRamp UI or using OpsQL API.</p> <h2>Basic syntax and structure</h2> <p>Let's say you want to make a query. The general syntax for an OpsQL query is very straightforward:</p> <pre><code class="language-PlainText"> &#x3C;attribute> &#x3C;operator> | &#x3C;coperator> "&#x3C;value>" [[&#x3C;operator> [&#x3C;attribute> | &#x3C;coperator> "&#x3C;value>"[)]] ... ] </code></pre> <p>You can use logical operators <code>AND</code> and <code>OR</code> to refine your search further.</p> <h2>Operators</h2> <p>Operator is the key of the query. It relates the attribute to the value. OpsQL supports a variety of operators to create precise queries.</p> <ol> <li><strong>Equality operators</strong></li> </ol> <p><code>=</code>, <code>!=</code> 2. <strong>String operators</strong> <code>CONTAINS</code>, <code>STARTS WITH</code>, <code>LIKE</code> 3. <strong>Logical operators</strong> <code>AND</code>, <code>OR</code></p> <p>For example, to find all resources with an agent installed and of type "Windows," you would use:</p> <pre><code class="language-PlainText">agentInstalled = "true" AND type = "Windows" </code></pre> <h2>Attributes</h2> <p>Attributes are different types of information available on an object. For instance, a resource might have attributes like <code>make</code>, <code>ipAddress</code>, and <code>agentInstalled</code>, while an alert might have attributes like <code>priority</code>, <code>currentState</code>, and <code>createdTime</code>.</p> <p>For more details, you can refer to the <a href="https://docs.opsramp.com/platform-features/feature-guides/query-language-reference/query-language-ref/">OpsRamp Query Language Reference Documentation.</a></p> <h2>More examples</h2> <p>Here are some OpsQL examples to search resources on the OpsRamp platform.</p> <h3>Search for resources that were discovered by an AWS integration​</h3> <pre><code class="language-PlainText">installedAppName = aws </code></pre> <h3>Search for resources that I tagged in AWS with AWS tag OwnerName :SivaBalaSubramanian</h3> <pre><code class="language-PlainText">installedAppName = "aws" AND tags.name = "OwnerName" and tags.value = "SivaBalaSubramanian"​ </code></pre> <h3>Search for resources with alerts that have been open for the last 2 hours​</h3> <pre><code class="language-PlainText">createdTime > "-7200sec" ​ createdTime > "-120min"​ </code></pre> <h3>Search for open alerts that are critical​</h3> <pre><code class="language-PlainText">currentState = "CRITICAL" AND status = "OPEN" ​ </code></pre> <h3>Search for open and critical alerts that have an open incident​</h3> <pre><code class="language-PlainText">currentState = "CRITICAL" AND status = "OPEN" AND incidentId IS NOT NULL ​ </code></pre> <h3>Search for alerts that have been Open for longer than 2 hours​</h3> <pre><code class="language-PlainText">createdTime &#x3C; "-7200sec" ​ </code></pre> <h3>Search for Open and critical alerts on resources tagged with AWS tag “BU: bu-123”​</h3> <pre><code class="language-PlainText">currentState = "CRITICAL" AND status = "OPEN" AND resource.tags.name = "Team" AND resource.tags.value = "Opsqa"​ </code></pre> <h2>What is OpsQL API?</h2> <p>The OpsQL API is a powerful interface that allows users to execute OpsQL queries programmatically. This API provides the flexibility to filter, search data within the OpsRamp platform, making it an indispensable tool for IT administrators and developers. IT Administrators can invoke OpsQL API using tools such as Postman, cURL, Python.</p> <h4>Key features</h4> <ol> <li><strong>Comprehensive querying</strong>: The OpsQL API supports a wide range of query operations, allowing users to filter data based on various attributes and conditions.</li> <li><strong>Flexibility</strong>: Users can create complex queries using logical operators and a variety of comparison operators.</li> <li><strong>Integration</strong>: The API can be integrated into custom applications, scripts, and workflows, enhancing automation and efficiency.</li> </ol> <h4>Basic syntax and structure</h4> <p>The general structure of an OpsQL API request involves specifying the tenant ID and the query payload.</p> <p>Here’s a basic example:</p> <pre><code>POST /opsql/api/v3/tenants/{tenantId}/queries </code></pre> <p>The request body typically includes:</p> <ul> <li><code>objectType</code>: The type of object to query (e.g., resource, alert, ticket).</li> <li><code>fields</code>: The fields to retrieve.</li> <li><code>filterCriteria</code>: The criteria to filter the objects.</li> </ul> <h2>Common use cases and code samples</h2> <h3>Filtering critical alerts</h3> <p>There are times that you would want to filter only critical alerts. This is how that would be done:</p> <pre><code class="language-python">import requests import json accessToken='valid access token' url = "https://server/opsql/api/v3/tenants/client_id/queries" payload = json.dumps({ "objectType": "alert", "fields": [ "id", "clientId", "component", "currentState" ], "filterCriteria": "currentState=critical" }) headers = { 'Accept': 'application/json', 'Content-Type': 'application/json', 'Authorization': accessToken } response = requests.request("POST", url, headers = headers, data = payload) </code></pre> <h3>Saving OpsQL response as CSV file</h3> <p>There are times that you would want to save an OpsQL response as a CSV file, perhaps for further analysis. This is how that would be done:</p> <pre><code class="language-python">def invoke_opsql() -> None: response = requests.request("POST", url, headers = headers, data = payload) timestr = time.strftime("%Y%m%d-%H%M%S") json_file_name = "siva_aws_resources-" + timestr + ".json" csv_file_name = "siva_aws_resources-" + timestr + ".csv" with open(json_file_name, "wb") as file: file.write(response.content) json_to_csv(json_file_name, csv_file_name) def json_to_csv(resources_json, file_csv) -> None : with open(resources_json) as json_file: data = json.load(json_file) opsql_response = data['results'] data_file = open(file_csv, 'w') csv_writer = csv.writer(data_file) count = 0 for opsql_row in opsql_response: if count == 0: header = opsql_row.keys() csv_writer.writerow(header) count += 1 csv_writer.writerow(opsql_row.values()) data_file.close() </code></pre> <h2>Conclusion</h2> <p>The OpsQL API is a powerful tool that enhances the capabilities of the OpsRamp platform, providing users with the flexibility to perform complex queries and manage data efficiently. By leveraging the OpsQL API, IT administrators and developers can streamline their operations, improve data management, and enhance overall productivity. Thus OpsQL grants the wishes of IT Ops Managers with a single pane of glass and unified searching capability across their wide, complex real estate. For more details, you can refer to the <a href="https://develop.opsramp.com/v3/api/opsql/tenantid-queries/">OpsQL API Documentation</a></p>https://developer.hpe.com/learning-to-manage-ilo-7-with-restful-interface-tool-ilorest-6-0/https://developer.hpe.com/learning-to-manage-ilo-7-with-restful-interface-tool-ilorest-6-0/Mon, 10 Mar 2025 14:05:22 GMT<h2>Introduction</h2> <p>We know you love servers by Hewlett Packard Enterprise (HPE) for their security and ease of management. With the release of HPE iLO 7 on Gen12 servers, HPE has taken security to the next level. Unlike previous generations where Production was the default <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/securityservice/#ilo-security-state" target="_blank">security state</a>, iLO 7 introduces the Secure Standard state by default, enhancing system security.</p> <p>Another major change is in the default login method - Virtual NIC (<a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/vnic/#the-ilo-redfish-host-interface-virtual-nic" target="_blank">VNIC</a>) replaces the <a href="https://servermanagementportal.ext.hpe.com/docs/etc/glossaryterms/" target="_blank">CHIF</a> interface, which was used in iLO 6 and earlier versions.</p> <p>In this article, I'll walk you through how to manage HPE iLO 7 using the iLOrest 6.0 tool via in-band access. We’ll start by installing <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">iLOrest</a> on different operating systems and then dive into logging into iLO 7. Since iLO 7 introduces a new <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/securityservice/#application-accounts" target="_blank">application account</a> login method, I’ll also cover how iLOrest 6.0 fully supports this feature.</p> <p>Let’s get started! 🚀</p> <h2>Installation</h2> <h3>Linux:</h3> <p>On Linux, iLOrest can be installed as an RPM package. If you already have a previous version installed, you can upgrade it using the -Uvh option. For a fresh installation, use:</p> <pre><code class="language-bash">rpm -ivh ilorest-6.0.x86_64.rpm </code></pre> <p>Here is a screenshot of a successful installation.</p> <p><img src="/img/rpm_linux.jpg" alt=""></p> <p><strong>Application account creation during iLOrest installation</strong></p> <p>During the RPM installation of iLOrest 6.0, you might notice that the installer prompts for iLO credentials to create an Application account. While this step is optional, HPE strongly recommends creating the Application account during installation itself. Why?</p> <ul> <li> <p>The Application account provides an additional method for in-band authentication with iLO 7, enhancing security and flexibility.</p> </li> <li> <p>Once created, the Application account allows you to log in without needing traditional credentials every time.</p> </li> </ul> <p>If you choose to skip this step, you can always create the Application account later using the following command:</p> <pre><code class="language-bash">ilorest appaccount create -u ilo-user -p password --self </code></pre> <p>By leveraging this new authentication method, managing iLO 7 becomes even more seamless!!!</p> <h3>Windows:</h3> <p>On Microsoft Windows, iLOrest is installed using an MSI package. During installation, a user interface will appear, prompting you to enter iLO credentials for the Application account creation. Just like on Linux, this step is optional but recommended by HPE, as the Application account allows for in-band authentication with iLO 7 without requiring traditional credentials. Here’s a screenshot of the Application account creation dialog box during installation:</p> <p><img src="/img/windows_msi.jpg" alt=""></p> <h3>All other OSes:</h3> <p>On operating systems like Ubuntu and the macOS, the iLOrest tool can be installed effortlessly via PyPI using the following command:</p> <pre><code class="language-bash">pip install ilorest </code></pre> <p>On VMware ESXi, it is installed using:</p> <pre><code class="language-bash">esxcli software component apply -d ilorest-component.zip </code></pre> <h4>Creating an Application account</h4> <p>For these OSes, you can create an Application account using the following iLOrest command:</p> <pre><code class="language-bash">ilorest appaccount create -u ilo-user -p password --self </code></pre> <h4>Logging into iLO 7</h4> <p>Once the Application account is created, you can perform an inband login with:</p> <pre><code class="language-bash">ilorest login </code></pre> <p>If you prefer to log in without using an Application account, you can opt for credential-based login instead:</p> <pre><code class="language-bash">ilorest login --no_app_account -u ilo-user -p password </code></pre> <h2>Summary</h2> <p>In this guide, I have demonstrated how to install iLOrest 6.0 across different operating systems and leverage the new Application account login method introduced in iLO 7. Get started today! Download <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">iLOrest</a>, explore its exciting new features, and take full control of iLO 7 with ease.</p> <p>For more information on HPE iLO, along with some tips and tricks in working with it, make sure you check out the HPE Developer blog regularly.</p>https://developer.hpe.com/2025-march-04/https://developer.hpe.com/2025-march-04/Tue, 04 Mar 2025 13:00:00 GMThttps://developer.hpe.com/the-rise-of-the-distributed-enterprise/https://developer.hpe.com/the-rise-of-the-distributed-enterprise/Mon, 24 Feb 2025 18:31:55 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>A major shift is underway in IT, and customers are leading the charge. Across industries — retail, logistics, healthcare, and manufacturing — businesses are moving away from centralized data centers and embracing a model that brings technology closer to where value is created. The distributed enterprise isn’t just an emerging trend; it’s a complete rethinking of IT strategy.</p> <p>In this blog series, I’ll explore why this shift matters, how it’s reshaping IT infrastructure, and what it means for organizations looking to stay ahead. The distributed model isn’t just about efficiency — it’s about unlocking new levels of agility, resilience, and innovation.</p> <h2>If the next slide has a cloud, this meeting is over!</h2> <p>In 1997, I found myself in the middle of a wide-area networking pitch to a key customer executive. My slides were packed with cloud diagrams, showing how devices would seamlessly connect. Five slides in, the executive stopped me cold.</p> <p>“If the next slide has a cloud, this meeting is over.”</p> <p>Without missing a beat, I smiled, shut my laptop, and said, “No problem. Let’s talk about what really matters. What are you trying to do?”</p> <p>Long before cloud computing was even a thing, that moment taught me a lesson that has stuck with me for decades: customers bring their own priorities, their own perspectives, and their own challenges to the table. What’s important to me as a vendor might be completely irrelevant to them. Their time is limited. Their goals are non-negotiable. And the most valuable insights don’t come from talking — they come from listening. And what they’re saying today may surprise you.</p> <h2>Fast forward to 2025: IT’s biggest challenge isn’t what you think</h2> <p>At Hewlett Packard Enterprise (HPE), we have some of the most forward-thinking customers in the world. What we’re hearing is that the biggest IT challenge today isn’t simply choosing between cloud, AI, or edge computing. It’s something deeper.</p> <p>Decades of IT decisions — investments in legacy architecture, vendor lock-in, and rigid operating models — have created an enormous weight that enterprises are struggling to escape. Many organizations find themselves not just burdened by technical debt, but by the outdated IT philosophies that shaped those decisions. Even modernization efforts, like shifting workloads to the cloud or navigating compliance challenges, have often resulted in fragmented, complex, and costly environments that are difficult to manage.</p> <p>It’s a frustrating paradox. The technologies meant to drive innovation are now causing bottlenecks. Instead of shaping bold, forward-looking strategies, many IT leaders are spending their time maintaining toolchains and platforms built for a different era.</p> <p>The solution isn’t another incremental upgrade or a slightly better cloud migration strategy. It requires something far more fundamental: a shift in mindset.</p> <h2>Breaking free: Clean-slate thinking</h2> <p>For years, IT has been shaped by a data center-first mindset. The enterprise computing model was built around large, centralized hubs that processed and stored everything, with branch locations, remote offices, and manufacturing sites acting as secondary nodes. That model worked well in an era when businesses were centralized, and network capacity was scarce.</p> <p>But today, enterprises operate as dynamic, distributed ecosystems. Retail chains, healthcare providers, and logistics companies don’t function as a single monolithic entity — they are sprawling, interconnected networks, each with unique operational needs and real-time data demands. IT should reflect that reality.</p> <p>Clean-slate thinking offers a different approach. Instead of trying to optimize an aging model, it asks: if we were starting from scratch today, how would we design IT for a distributed world? The answer isn’t about moving everything to the cloud or holding onto traditional infrastructure — it’s about rethinking IT as a flexible, location-aware, and real-time system that adapts to business needs rather than forcing businesses to adapt to IT limitations.</p> <h2>Cracks in the old model</h2> <p>Real-time applications are pushing IT to the edge — literally — and putting immense pressure on legacy IT infrastructure. AI-driven analytics, high-resolution cameras, and IoT sensors are generating massive amounts of data — data that must be processed instantly to be valuable.</p> <p><img src="/img/the-rise-of-the-distributed-enterprise-image-1.png" alt="Real-time applications are pushing IT to the edge" title="Real-time applications are pushing IT to the edge"></p> <p>Consider modern quality assurance systems in manufacturing. A single AI-powered inspection system can generate <strong>100MB of data per second</strong>, all of which needs to be analyzed immediately. Sending that data over a network to a distant data center introduces <strong>10 to 20 seconds of delay</strong> — an eternity when a pass/fail decision must be made in under a second.</p> <p><img src="/img/the-rise-of-the-distributed-enterprise-image-2.png" alt="Consider modern quality assurance systems in manufacturing" title="Consider modern quality assurance systems in manufacturing"></p> <p>Retailers face similar challenges. AI-driven theft detection systems rely on real-time video analysis. If video feeds must be sent to a centralized cloud or data center before an action can be taken, the system fails in its primary goal: preventing theft in the moment.</p> <p>For industries dependent on instant decision-making, the old model simply doesn’t work. A centralized processing approach turns real-time data into historical data before it can be acted upon. And in these cases, delayed insights are as good as no insights at all.</p> <p>Organizations that cling to a rigid, centralized model are putting themselves at a disadvantage. Those that embrace a distributed approach, where data is processed where it’s generated, will gain an operational edge.</p> <h2>Rethinking IT for 2025 and beyond</h2> <p>Enterprise IT wasn’t always built this way. In the early days of computing, a single <strong>central processing unit (CPU)</strong> ran everything. Early mainframes were massive, expensive machines housed in specialized rooms, accessible to only a handful of people. Adjusted for 2025 dollars, running a single-CPU mainframe in the 1950s would cost upwards of <strong>$200,000 per month</strong> — not including the space and personnel required to operate it.</p> <p>The world moved on. Computing became distributed. Processing power expanded beyond a single CPU to networks of interconnected systems. But despite these advances, enterprise IT remained largely centralized, tied to the idea that data must flow to a core processing hub before it can be useful.</p> <p>That mindset is outdated. Data is no longer just an asset to be stored and analyzed — it’s the lifeblood of a modern enterprise. And just like goods, services, and financial transactions, it needs to flow freely to be valuable.</p> <p>Shifting from a data center-centric approach to a distributed enterprise model requires more than just new technology. It demands a new way of thinking. Today’s use cases require a model where:</p> <ul> <li> <p>Instead of a few massive data centers, IT infrastructure is spread across a network of private clouds, edge locations, and intelligent local processing nodes, using spot instances, GPU-sharing, and dynamic workload placement to maximize efficiency.</p> </li> <li> <p>Instead of relying on centralized security models and rigid VPNs, organizations embrace Zero Trust principles, where access is identity-based and data is protected at every level.</p> </li> <li> <p>Instead of monolithic applications that require extensive backend processing, businesses build microservices that run dynamically in containers and serverless environments.</p> </li> <li> <p>And, instead of constantly pinging data back and forth to the cloud, AI and analytics run where data is generated, with distributed data fabrics and federated learning ensuring that insights are shared efficiently.</p> </li> </ul> <p>This is not just about efficiency. It’s about agility, resilience, and the ability to operate at the speed of business. Enterprises that distribute their compute, storage, and analytics capabilities will be positioned to make faster decisions, reduce costs, and unlock new opportunities for innovation.</p> <h2>What comes next?</h2> <p>The future of enterprise IT isn’t about holding on to outdated models—it’s about unlocking the full potential of real-time data where it’s actually needed. The distributed enterprise is the natural evolution of IT, and companies that embrace it will be at the forefront of the next wave of innovation.</p> <p>Watch for my next posts where I’ll delve into <strong>data gravity</strong> — how it shapes IT strategy and why understanding the lifecycle of data is key to making the distributed enterprise a reality, as well as other aspects of this model.</p> <p>See you then.</p>https://developer.hpe.com/optimizing-data-processing-with-apache-spark-best-practices-and-strategies/https://developer.hpe.com/optimizing-data-processing-with-apache-spark-best-practices-and-strategies/Mon, 24 Feb 2025 13:39:56 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Big Data processing is at the core of modern analytics, and <strong>Apache Spark</strong> has emerged as a leading framework for handling large-scale data workloads. However, optimizing Spark jobs for <strong>efficiency, performance, and scalability</strong> remains a challenge for many data engineers. Traditional data processing systems struggle to keep up with the exponential growth of data, leading to issues like <strong>resource bottlenecks, slow execution, and increased complexity</strong>.</p> <p>This whitepaper explores <strong>best practices and optimization strategies</strong> to enhance Spark’s performance, improve resource utilization, and ensure scalability. With <strong>data collection becoming cheaper and more widespread</strong>, organizations must focus on extracting business value from massive datasets efficiently. <strong>Apache Spark was designed to solve some of the biggest challenges in Big Data</strong>, enabling everything from basic data transformations to advanced machine learning and deep learning workloads.</p> <p><strong>Understanding Apache Spark</strong></p> <p>Apache Spark, an open-source distributed data processing framework, addresses these challenges through its innovative architecture and in-memory computing capabilities, making it significantly faster than traditional data processing systems.</p> <p>Apache Spark was developed to address several limitations and challenges that were present in existing big data processing frameworks, such as Hadoop MapReduce. It supports multiple programming languages, including Python (PySpark), Scala, and Java, and is widely used in ETL, machine learning, and real-time streaming applications.</p> <p>Here are the key reasons why Spark came into existence and what sets it apart from other frameworks in the big data world:</p> <ul> <li><em>In-memory processing</em></li> <li><em>Iterative and interactive processing</em></li> <li><em>Ease of use</em></li> <li><em>Unified framework</em></li> <li><em>Resilient distributed datasets (RDDs)</em></li> <li><em>Lazy evaluation and DAG execution</em></li> <li><em>Interactive analytics</em></li> <li><em>Streaming</em></li> <li><em>Machine learning libraries</em></li> <li><em>Graph processing</em></li> <li><em>Advanced analytics</em></li> </ul> <p><strong>Challenges in Spark optimization</strong></p> <p>While Spark is designed for speed and scalability, several challenges can impact its performance:</p> <ol> <li><strong>Inefficient data partitioning</strong> - Poor partitioning can lead to data skew and uneven workload distribution.</li> <li><strong>High shuffle costs</strong> - Excessive shuffling of data can slow down performance.</li> <li><strong>Improper resource allocation</strong> - Inefficient use of memory and CPU can cause bottlenecks.</li> <li><strong>Slow data reads and writes</strong> - Suboptimal file formats and storage choices can degrade performance.</li> <li><strong>Poorly written code</strong> - Unoptimized transformations and actions can increase execution time.</li> <li><strong>No storage layer</strong> – Spark does not have a built-in storage layer, so it relies on external storage systems for data persistence.</li> </ol> <p><strong>Best practices for Spark optimization</strong></p> <p>To address these challenges, the following best practices should be adopted:</p> <p><strong>1. Optimize data partitioning</strong></p> <ul> <li>Use appropriate partitioning techniques based on data volume and usage patterns.</li> <li>Leverage bucketing and coalescing to manage partition sizes.</li> </ul> <p><strong>2. Reduce shuffle operations</strong></p> <ul> <li>Avoid wide transformations like groupBy() and reduceByKey() when possible.</li> <li>Use broadcast joins for small datasets to minimize shuffling.</li> </ul> <p><strong>3. Efficient memory management</strong></p> <ul> <li>Tune Spark configurations like spark.executor.memory and spark.driver.memory.</li> <li>Optimize garbage collection settings for long-running jobs.</li> </ul> <p><strong>4. Use optimized file formats</strong></p> <ul> <li>Prefer columnar storage formats like Parquet or ORC over CSV and JSON.</li> <li>Enable compression to reduce I/O overhead.</li> </ul> <p><strong>5. Leverage catalyst optimizer and tungsten execution engine</strong></p> <ul> <li>Let Spark’s Catalyst Optimizer handle query optimization.</li> <li>Utilize Tungsten’s bytecode generation and memory management features.</li> </ul> <p><strong>6. Optimize code for performance</strong></p> <ul> <li>Use DataFrame API instead of RDDs for better optimization.</li> <li>Avoid unnecessary collect() and count() operations.</li> <li>Cache and persist intermediate results where necessary.</li> </ul> <p><strong>7. Monitor and debug performance</strong></p> <ul> <li>Use Spark UI and Event Logs to analyze job execution.</li> <li>Employ metrics and monitoring tools like Ganglia and Prometheus.</li> </ul> <p><strong>Conclusion</strong></p> <p>Optimizing Apache Spark requires a strategic approach that combines efficient data handling, resource management, and code optimization. By implementing the best practices outlined in this whitepaper, organizations can enhance performance, reduce costs, and accelerate large-scale data processing.</p> <p>As big data continues to grow, mastering Spark’s fundamentals will empower organizations to unlock its full potential, drive innovation, and make smarter data-driven decisions in the digital era.</p>https://developer.hpe.com/deploying-a-hugging-face-llm-in-hpe-private-cloud-ai/https://developer.hpe.com/deploying-a-hugging-face-llm-in-hpe-private-cloud-ai/Thu, 20 Feb 2025 20:03:50 GMT<p>Deploying new language models for users to interact with can be challenging for beginners. HPE developed Private Cloud AI to help users set up and implement AI solutions quickly and easily.</p> <p>In this post, we will show how to use the HPE Machine Learning Inference Service (MLIS) as a part of HPE Private Cloud AI to add a new packaged model from a Hugging Face repository and create an endpoint to query the model. This is done using a Jupyter Notebook.</p> <h3>Prerequisites</h3> <p>This tutorial uses the <a href="https://www.hpe.com/us/en/private-cloud-ai.html">HPE Private Cloud AI</a> (PCAI) platform. A PCAI system is required for these steps to work. It is assumed that the PCAI system is physically installed, patched and running with user accounts provisioned.</p> <h3>Steps to deploy</h3> <p>First, you will need to choose a model to deploy. In this case, we've chosen a model hosted on Hugging Face called <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct">SmolLM2 1.7B</a>. This is a compact model that can solve a wide range of problems even though it is relatively diminutive at 1.7B parameters.</p> <h3>Launching the interface</h3> <p><img src="/img/mlis.png" alt="Computer screen showing the HPE Private Cloud AI user interface and the HPE MLIS tile is highlighted."></p> <p>Next select "Add new model".</p> <p><img src="/img/new-model.png" alt="Computer screen showing packaged AI models and a selection to add a new model."></p> <p>This brings up the "Add new packaged model" dialog box. Fill in the the name of the model, storage requirements, and resources. We have reduced the default resources, given that this is a small model.</p> <p><img src="/img/define-parameters.png" alt="Dialog box for defining a new packaged model."></p> <p>Once the package is set up, you will receive a confirmation.</p> <p><img src="/img/package-running.png" alt="Shows running packaged model."></p> <p>With the new packaged model complete, you will need to deploy it for use. Select "create new deployment" from the HPE MLIS "Deployments" tab. Select submit when all tabs are filled out as shown below.</p> <p>This will create an endpoint for use in the notebook and provide an API token.</p> <p><img src="/img/new-deployment.png" alt="New deployment for AI model"></p> <p>When the process is complete, an endpoint will be provided.</p> <p><img src="/img/endpoint.png" alt="Endpoint provided by MLIS system"></p> <p>Next up, let's take the now deployed model that's ready for inference and connect to it and interact with it from a Jupyter Notebook.</p> <h3>Building the Jupyter Notebook</h3> <p>First, install <code>openai</code> if you do not already have it and import.</p> <pre><code class="language-python"># vLLM Chat OpenAI # !pip intall openai from openai import OpenAI </code></pre> <p>Then, using the endpoint and key generated by HPE MLIS, enter them into your Jupyter Notebook. Be sure to append /v1 to the URL.</p> <pre><code class="language-python"># Grab endpoint URL and API key from MLIS, remember to include "/v1" for latest version of the OpenAI-compatible API model = "HuggingFaceTB/SmolLM2-1.7B-Instruct" openai_api_base = "https://smollm2-1-7b-vllm-predictor-dave-wright-hpe-1073f7cd.hpepcai-ingress.pcai.hpecic.net/v1" openai_api_key = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE3Mzk5MzgzMzAsImlzcyI6ImFpb2xpQGhwZS5jb20iLCJzdWIiOiI5MjNhM2JhOC1mMGU4LTQxOTQtODNkMS05ZWY4NzNjZGYxOWYiLCJ1c2VyIjoiZGF2ZS53cmlnaHQtaHBlLmNvbSJ9.YwH9gGPxTWxy4RSdjnQA9-U3_u7P0OIcarqw25DV8bOiftU1L4IvvyERHspj2lMGtZWbff1F3uh84wjAePHaHDcDTLoGtq6gJYwo_qRU03xV8Q2lwBetCCLUE4OHqS608gjJ-j1SLyqwxFxlXkqMOtnBY5_nswlAwCzHV28P8u8XxxfWuXFmoJpSA1egCWVVfEoTuK8CTz9kUJJ5opSp6m8qdqJmC2qxH0igcpKmL2H_MZ-62UHfEf240VRtc0DRNlOjeCoDM79aVPs3SjCtGeVkeEHimJwJbfGFIcu3LibX3QjbABUzWb5BPPZjzyEYUVM5ak12_sJ8j1mUW-r0sA" </code></pre> <p>You will now need to create an OpenAI client interface.</p> <pre><code class="language-python"># create OpenAI client interface client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) </code></pre> <p>In order to interact with the model, you will need to create a chat function. For the purposes of our example, let's give it a history feature as well as basic chat.</p> <pre><code class="language-python"># Interactive chat function with message history. def chat(): # Initialize conversation history messages = [] print("Chat with "+model+"! Type 'quit' to exit.") while True: # Get user input user_input = input("\nYou: ").strip() # Check for quit command if user_input.lower() == 'quit': print("Goodbye!") break # Add user message to history messages.append({"role": "user", "content": user_input}) try: # Get model response using chat completion response = client.chat.completions.create( model=model, messages=messages ) # Extract assistant's message assistant_message = response.choices[0].message.content # Add assistant's response to history messages.append({"role": "assistant", "content": assistant_message}) # Print the response print("\nAssistant:", assistant_message) except Exception as e: print(f"\nError: {str(e)}") </code></pre> <p><img src="/img/jupyter.png" alt="Jupyter Notebook showing imported model endpoint and API key."></p> <p>Once this is done, you can interact with the model through a simple chat.</p> <p><img src="/img/chat-interface.png" alt="Interaction with the SmolLM2 Small Language Model in a Jupyter Notebook"></p> <p>You can access <a href="https://www.youtube.com/watch?v=oqjc-2c1Vtk">this link</a> to see a recorded demonstration that shows this process in real time. <a href="https://www.youtube.com/watch?v=oqjc-2c1Vtk"></a></p> <h3>Summary</h3> <p>With HPE Private Cloud AI, loading new models into the system and providing endpoints is just a few simple clicks and easily integrates with popular tools like Jupyter Notebooks. To learn more about HPE Private Cloud AI, please visit: <a href="https://www.hpe.com/us/en/private-cloud-ai.html">https://www.hpe.com/us/en/private-cloud-ai.html</a></p>https://developer.hpe.com/7-questions-for-bill-reus-interactive-supercomputing-with-chapel-for-cybersecurity/https://developer.hpe.com/7-questions-for-bill-reus-interactive-supercomputing-with-chapel-for-cybersecurity/Thu, 13 Feb 2025 02:40:08 GMT<p>external blog</p>https://developer.hpe.com/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/https://developer.hpe.com/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/Thu, 06 Feb 2025 13:53:06 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>Polling API or subscribing to events: That IS the question</h2> <p>In one of my previous blog posts, I used the HPE GreenLake API to query the audit log and, if anything appeared in the audit log over the course of a few minutes, I arranged for it to be displayed on screen. To do this, I had to continuously poll the API at a regular polling interval. While this method works, it is not ideal, since it is not done in real time, and you might get notified of an important event after, at max, your polling interval. A better approach that is often available on software platforms is called events, also referred to as webhooks. HPE GreenLake cloud provides this functionality and, in this post, I will explain how to leverage it.</p> <h2>It's a publisher/subscriber world</h2> <p>HPE GreenLake cloud provides an events framework in which event publishers (any of the HPE GreenLake cloud services) can register event types with the platform and event subscribers can declare what event types they would like to subscribe to. After they establish a security handshake, HPE GreenLake cloud forwards selected events to the subscriber in a close-to-real-time mode. No polling is necessary as the event handler (webhook) will be notified asynchronously.</p> <p>The following diagram illustrates the mechanism by which this works:</p> <p><img src="/img/slide-pour-blog-webhooks.jpg" alt="HPE GreenLake cloud events framework" title="HPE GreenLake cloud events framework"></p> <h2>How do you write webhook handlers?</h2> <p>There are many ways you can write a webhook handler. You could use traditional languages such as Node.js (using Express.js), Python (using Flask) or Ruby (using Sinatra). Also very popular is the use of serverless functions such as AWS Lambda or Google Cloud Functions. You could also use low-code/no-code platforms such as Zapier, Make or Automate.io. Another option is to use an existing platform that supports webhooks such as ServiceNow or HPE OpsRamp.</p> <p>In any case, a webhook handler will do the following:</p> <ol> <li>Wait for an HTTP POST request</li> <li>Verify request validity</li> <li>Parse and process data</li> <li>Return an HTTP status code</li> </ol> <p>Additional requirements for writing a webhook handler for HPE GreenLake include:</p> <ul> <li>The URL for the endpoint needs to be a Fully Qualified Domain Name (FQDN)</li> <li>The endpoint needs to use HTTPS</li> <li>The handler needs to respond to the initial challenge through the use of a shared secret key</li> </ul> <h2>Taking the challenge</h2> <p>From the illustration above, you can see that the webhook handler is a piece of code that runs outside of the HPE GreenLake cloud, possibly posing a security risk. In order to avoid calling a rogue code, HPE GreenLake will establish a trust relationship with the webhook handler by issuing a challenge request and expecting a very specific response. The challenge is an HTTP POST request that is sent to the webhook handler (via its URL) with a payload containing a <strong>challengeRequest</strong> as shown below:</p> <pre><code class="language-json">{ "data": { "challengeRequest": "8cbecfea-d708-4706-bba3-669225084a10" }, "datacontenttype": "application/json", "id": "9d2698f2-0dd4-4870-b026-7e18c755e121", "source": "https://global.api.greenlake.hpe.com/events", "specversion": "1.0", "subject": "3009de2825f211ec8a84fedebcb4a754", "time": "2024-12-12T19:09:05Z", "type": "com.hpe.greenlake.events.v1beta1.webhooks.verification" } </code></pre> <blockquote> <blockquote> <p><em>Example of a challenge payload</em></p> </blockquote> </blockquote> <p>The mission of the challenge handler is to provide the correct answer to the challenge in a timely fashion and with an HTTP response in the following form, where <strong><CHALLENGE-RESPONSE></strong> is the computed <a href="https://en.wikipedia.org/wiki/SHA-2">SHA-256</a> <a href="https://en.wikipedia.org/wiki/HMAC">HMAC</a> (Hash-based Message Authentication Code) of the <strong>challengeRequest</strong> provided in the input payload.</p> <ul> <li>Status code: <strong>200</strong></li> <li>JSON body: <strong>{ "verification" : "<CHALLENGE-RESPONSE>" }</strong></li> <li>Header: <strong>content-type: application/json</strong></li> </ul> <blockquote> <p>Note: There are <a href="https://www.devglan.com/online-tools/hmac-sha256-online">online ways</a> to generate SHA256 MAC encodings and test different options but what you need to remember is that you have to provide a hexadecimal format of the hash and that a secret key is used as a “salt” for the calculation of that hash. That secret key is shared by HPE GreenLake cloud and the webhook handler.</p> </blockquote> <h3>Processing events as they arrive</h3> <p>If the challenge was successful, meaning that the webhook handler has been recognized as a valid target for HPE GreenLake event forwarding, the next task of the webhook handler is to process events as they are received through subsequent POST requests. The payload shown below is an example of an event received from the HPE GreenLake audit log. It describes the fact that a user has logged out of the HPE GreenLake cloud. More details about other event payloads can be found in the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/event/public/events/">HPE GreenLake developer portal</a>.</p> <pre><code class="language-json">{ "specversion": "1.0", "id": "uJoQ5JMB_HAPiq2sSllt", "source": "//global.api.greenlake.hpe.com/audit-log", "type": "com.hpe.greenlake.audit-log.v1.logs.created", "datacontenttype": "application/json", "dataschema": "https://developer.greenlake.hpe.com/docs/greenlake/services/audit-logs/public/catalog/audit-log-event-latest/paths/Audit%20Log%20Created/post/", "time": "2024-12-20T12:34:53.161Z", "data": { "id": "uJoQ5JMB_HAPiq2sSllt", "user": { "username": "john.doe@hpedev.io" }, "workspace": { "id": "3009de2825f211ec8a84fedebcb4a754", "workspace_name": null, "workspace_type": null }, "application": { "id": "00000000-0000-0000-0000-000000000000" }, "category": "user_management", "description": "User john.doe@hpedev.io logged out", "created_at": 1734698093161, "updated_at": 1734698093161, "additional_info": { "ip_address": "194.9.99.5", "account_name": "HPEDEV -GLCP- Hackshack", "ip_address_str": "1.1.1.1" }, "has_details": false } </code></pre> <blockquote> <blockquote> <p><em>Example of an event payload</em></p> </blockquote> </blockquote> <p>Now that you have a clear understanding of what has to be done within a webhook handler, let me show you how to build one.</p> <h2>Using Make.com to create a webhook handler</h2> <p>I decided to use Make (previously known as Integromat , which was a great name as it was a contraction of Integration-Automat and that is exactly what it is) to demonstrate this because:</p> <ul> <li>it doesn’t require any specific programming language knowledge</li> <li>it provides a FQDN of the webhook over HTTPS</li> <li>it’s free for simple prototyping usage</li> </ul> <p>It turned out to be a very elegant and fast way to get the work done.</p> <h3>Getting started with Make</h3> <p>First, you will need to create an account at <a href="https://make.com">https://make.com</a>. Next you will create a new scenario (aka workflow).</p> <p>The first step in the scenario will be a Webhooks module:</p> <p><img src="/img/create-a-webhook.jpg" alt="Create a webhook in Make" title="Create a webhook in Make"></p> <p>Give it a name and leave the rest of the settings as their defaults.</p> <p><img src="/img/create-webhook-2.jpg" alt="Use URL of Webhook" title="Use URL of Webhook"></p> <p>You can already <strong>Copy address to clipboard</strong> to get the URL of your webhook. Save it for later.</p> <p>Select the <strong>Add</strong> button, then click on <strong>Show Advanced Option</strong> to add a data structure using the challenge payload example from the Take the challenge section above. Click <strong>generate,</strong> paste the JSON and save it as <strong>challenge</strong>.</p> <p>Make sure the <strong>challenge</strong> data structure is selected in the advanced settings of the Webhooks module before continuing.</p> <p><img src="/img/select-challenge.jpg" alt="Select challenge data structure " title="Select challenge data structure "></p> <p>Next, add another module to compute the HMAC. Click the plus sign and add a <strong>Tools Set</strong> <strong>variable</strong> module (use search to avoid the long list of modules).</p> <p><img src="/img/add-set-variabke-step.jpg" alt="Add set variable step" title="Add set variable step"></p> <p>Configure the Set variable module with:</p> <ul> <li>Variable name: <strong>hmac</strong></li> <li>Variable value: <strong>sha256( challengeRequest ; ; <SecretKey> )</strong></li> </ul> <blockquote> <p>Note: you can drag the <strong>challengeRequest</strong> property from the <strong>Webhooks</strong> module, and drop it as your first parameter. <strong><SecretKey></strong> is a placeholder which you will replace later, once we know the the real shared secret key.</p> </blockquote> <p><img src="/img/drag-drop-properties.jpg" alt="drag and drop properties" title="drag and drop properties"></p> <p>The final step is to prepare the response of the webhook. For this, you need to add another module after the <strong>Set variable</strong> module, called <strong>Webhook response</strong>.</p> <p><img src="/img/add-webhook-response.jpg" alt="Add webhook response" title="Add webhook response"></p> <p>Set the status to <strong>200</strong> and the body to <strong>{"verification":"hmac"}</strong>. Feel free to drag/drop the <strong>hmac</strong> property from the <strong>Set variable</strong> step in between the double quotes.</p> <p><img src="/img/setwebhook-response.jpg" alt="Set webhook response" title="Set webhook response"></p> <p>In the advanced settings, add a custom header for <strong>content-type: application/json</strong></p> <p>Your overall workflow basically looks like this:</p> <p><img src="/img/step1-final.jpg" alt="Part 1 workflow" title="Part 1 workflow"></p> <p>In the parameter of the scenario (bottom of your editor), make sure the green check <strong>to run Immediately as data arrives</strong> is checked.</p> <p><img src="/img/run-immediatly.jpg" alt="Run immediately as data arrives " title="Run immediately as data arrives "></p> <h3>Debugging webhook using Postman</h3> <p>You can use Postman to test this workflow first before you declare it in HPE GreenLake cloud.</p> <p>For this, create a <strong>POST</strong> request using the URL of the webhook (which you saved earlier) and the challenge example payload in JSON (we used it earlier). Make sure you add a <strong>content-type:application/json</strong> header before clicking <strong>Send</strong>. Check:</p> <ul> <li>That the status code is 200</li> <li>That the response body is the expected <strong>hmac</strong> value</li> </ul> <p><img src="/img/postman-challenge.jpg" alt="Try challenge from Postman" title="Try challenge from Postman"></p> <h3>Handling events in Make</h3> <p>Now that you know how to handle the initial challenge, you need to also take care of the other type of events that will be sent to the webhook handler.</p> <p>To make that distinction, you will use a new module in Make called a <strong>Router</strong> and insert it in between the <strong>Webhooks</strong> and the <strong>Tools Set variable</strong> modules (right-click add a router to <strong>Webhooks</strong>).</p> <p><img src="/img/add-a-router.jpg" alt="Add a router" title="Add a router"></p> <p>Next, you will have to place a filter on this first branch (the challenge case) so that only the challenge call gets routed on the topmost branch.</p> <p>To do this, click on the topmost link and set up a filter to verify that property <strong>challengeRequest</strong> exists in the payload. As earlier, you can use drag/drop to setup the condition of the filter, which is very convenient.</p> <p><img src="/img/setup-filter.jpg" alt="Setup a filter on top most branch" title="Setup a filter on top most branch"></p> <p>Earlier, I showed an example of a payload received when an event is triggered. You can create another structure in the custom webhook (like what was done with the challenge JSON earlier). Call this structure <strong>event</strong> and make sure it’s selected before continuing.</p> <p><img src="/img/select-event-struture.jpg" alt="Select event structure" title="Select event structure"></p> <p>Now you can proceed and add a second branch to the router. This branch will be used to process all other events (when it’s not a challenge) received by the handler. For this blog post, I have decided to keep it simple and simply add a row within a Google Sheet.</p> <p><img src="/img/add-a-google-sheet.jpg" alt="Add a Google Sheet" title="Add a Google Sheet"></p> <p>For this, you must add a Google Sheet module and configure it to fit your needs. As you can see below, I have configured mine with my Google account, and the path to an existing Google Sheet called AuditLog. I also mapped the fields from the data structure that I’d like to record in each new row.</p> <p><img src="/img/build-a-row.jpg" alt="Build a new row in google sheet" title="Build a new row in google sheet"></p> <p>Finally, you also need to add to this branch a <strong>Webhook response</strong> module (like we did in the topmost branch). This one is simpler, as it only returns a status 200 (no body, no header needed).</p> <p><img src="/img/second-webhook-response.jpg" alt="Add a second webhook response" title="Add a second webhook response"></p> <p>The last thing you need to do is to place a fallback filter on the bottom branch, so it is used for all events except the challenge.</p> <p>To do this, select the properties of the bottom branch of the <strong>Router</strong> and select <strong>yes</strong> for <strong>fallback</strong>.</p> <p><img src="/img/set-fallback.jpg" alt="Set a fallback branch" title="Set a fallback branch"></p> <p>Make sure you save your scenario.</p> <h3>Debugging webhook using Postman… again</h3> <p>Create a second Postman <strong>POST</strong> request using the same webhook URL but this time, in the body, use the audit log event payload used earlier.</p> <p><img src="/img/post-event-in-postman.jpg" alt="Post event in Postman" title="Post event in Postman"></p> <p>Click <strong>Send</strong> in Postman, and check in Make (in the scenario’s history) that the second route was taken, as shown below.</p> <p><img src="/img/second-route-taken.jpg" alt="Second route was taken" title="Second route was taken"></p> <p>You can also verify that in the <strong>Google sheets</strong> module, the content of the payload was mapped to different values (A:, B:, C:,…)</p> <p><img src="/img/googlesheet-set.jpg" alt="Google sheet property mapping " title="Google sheet property mapping "></p> <p>Finally, verify that your Google sheet was updated and a new row was added using the payload content, as shown in my Google sheet below.</p> <p><img src="/img/google-row-visible.jpg" alt="Row was added in Google sheet" title="Row was added in Google sheet"></p> <h2>Putting it all together</h2> <p>It’s now time to put it all together. Connect to HPE GreenLake cloud console and select to <strong>Manage Workspace</strong>. From there, select the <strong>Automations</strong> tile and finally the <strong>Webhooks</strong> tile to access the webhooks configuration section:</p> <p><img src="/img/webhooksview.jpg" alt="Webhooks in HPE GreenLake cloud" title="Webhooks in HPE GreenLake cloud"></p> <p>To register a new webhook as shown in the capture above, you need to provide:</p> <ul> <li>A name</li> <li>A description</li> <li>The URL of the webhook</li> <li>The Webhook secret key. (When you set this secret key don’t forget to go back to Make and edit the <strong>Tool Set variable</strong> module to use the same secret key. These keys MUST match exactly!)</li> </ul> <p><img src="/img/same-secret-key.jpg" alt="You must use the same secret key on both ends" title="You must use the same secret key on both ends"></p> <h3>Register your webhook</h3> <p>When you select <strong>Register webhook</strong>, the first thing HPE GreenLake cloud will do is to establish a trust relationship with the webhook pointed by the URL, by sending a challenge payload using an HTTP POST request (as already discussed in the previous section).</p> <p>If the webhook response is the expected one, then the webhook is placed in <strong>Active</strong> state. Otherwise, after a few attempts, the webhook state will be set to <strong>Critical</strong>. At any point in time you can use the <strong>Test</strong> button, to submit a new challenge (after fixing an issue with your webhook, for example).</p> <h3>It’s time to subscribe to events</h3> <p>From the web console, you can now subscribe to events available from the HPE GreenLake services. You can find the list of those already available from the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/event/public/ui/#finding-events-on-hpe-greenlake-developer-portal">HPE GreenLake Developer Portal</a>. The list will grow over time as new services adopt this events framework.</p> <p>Select your webhook from the list and select <strong>Subscribe to event</strong>. Select the source <strong>Service manager</strong> (there is only HPE GreenLake Platform for now), then cut/paste the <strong>Event type</strong> from the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/#event-catalog">event catalog</a>. The following show two examples of event type.</p> <pre><code class="language-markdown">com.hpe.greenlake.audit-log.v1.logs.created com.hpe.greenlake.subscriptions.v1.expiring-subscriptions </code></pre> <p><img src="/img/subscribe.jpg" alt="Subscribing to events" title="Subscribing to events"></p> <p>Once you select <strong>Subscribe to event</strong>, your webhook handler is now registered to receive these types of events.</p> <p><img src="/img/event-subscribed.jpg" alt="Event subscribed" title="Event subscribed"></p> <p>Let’s check what happened in the Google Sheet after a while.</p> <p><img src="/img/events-in-google-docs.jpg" alt="Google Sheet starts being populated with events" title="Google Sheet starts being populated with events"></p> <p>You can see different types of audit log events coming up from HPE GreenLake cloud. They could be processed on the receiving end, for example, to open incident tickets or take an automatic action.</p> <h2>Call to action</h2> <p>Webhooks together with the HPE GreenLake cloud events framework provide a great way to integrate with HPE GreenLake cloud using modern technology. It provides great flexibility on where you can run the subscriber code, and allows you to choose the language in which to write the webhook code. It also allows you to build a tight integration with an existing platform such as HPE OpsRamp or ServiceNow.</p> <p>Additional benefits I can see from this technique are:</p> <ul> <li>no need to verify that a polling code is still running</li> <li>no risk of API token expiration</li> <li>no loss of events</li> <li>it’s well established industry standard mechanism</li> <li>there is a huge choice of implementation methods including low code/no code ones</li> </ul> <p>You can request access to this feature by signing up to the Automations/Webhook Access Beta Program <a href="https://app.smartsheet.com/b/form/0e61e8c2bd6d48c7829845ab824c11d6">here</a>.</p>https://developer.hpe.com/using-the-chapel-compiler-to-develop-language-tooling/https://developer.hpe.com/using-the-chapel-compiler-to-develop-language-tooling/Wed, 05 Feb 2025 04:33:04 GMT<p>external blog</p>https://developer.hpe.com/2025-february-03/https://developer.hpe.com/2025-february-03/Tue, 04 Feb 2025 13:00:00 GMThttps://developer.hpe.com/dragonhpc-featured-at-sc24/https://developer.hpe.com/dragonhpc-featured-at-sc24/Wed, 29 Jan 2025 16:53:37 GMT<p>DragonHPC was featured again at this year's Supercomputing 2024 conference as part of a special workshop on High Performance Python for Science at Scale. DragonHPC's presentation focused on telemetry and visualization for complex HPC and AI workflows. The workshop featured a range of different Python efforts for scientific computing at HPE, Argonne National Lab, Lawrence Berkeley National Lab, and the University of Delaware.</p> <p>Here is a link to this year's conference workshop presentation:</p> <p><a href="https://sc24.supercomputing.org/proceedings/workshops/workshop_pages/ws_hppss101.html">https://sc24.supercomputing.org/proceedings/workshops/workshop_pages/ws_hppss101.html</a></p> <p>A sample DragonHPC telemetry panel image from the workshop is shown below:</p> <p><img src="/img/dragonhpc_telemetry_sc24.png" alt="DragonHPC Telemetry Sample Panel" title="DragonHPC Telemetry Sample Panel"></p> <p>DragonHPC is a composable and highly versatile distributed runtime for HPC &#x26; AI workflows. The DragonHPC distributed runtime is Python native but also extensible to other common programming languages. DragonHPC can help orchestrate distributed processes effectively, access distributed data efficiently, and can be used on heterogeneous hardware.</p> <p>DragonHPC would be of benefit and high interest for HPC &#x26; AI software developers, architects, researchers, and data scientists.</p> <p>You can learn more about DragonHPC by visiting <a href="https://developer.hpe.com/platform/dragonhpc/home/">the landing page on HPE Developer Community portal</a>.</p>https://developer.hpe.com/llm-agentic-tool-mesh-orchestrating-agentic-tools-for-the-next-revolution/https://developer.hpe.com/llm-agentic-tool-mesh-orchestrating-agentic-tools-for-the-next-revolution/Mon, 20 Jan 2025 08:36:14 GMT<style> li { font-size: 27px !important; line-height: 33px !important; max-width: none !important; } </style> <p>In my previous blog posts, I dolve into the <a href="https://developer.hpe.com/blog/ll-mesh-exploring-chat-service-and-factory-design-pattern/">Chat Service</a>, <a href="https://developer.hpe.com/blog/llm-agentic-tool-mesh-harnessing-agent-services-and-multi-agent-ai-for-next-level-gen-ai/">Agent Service</a>, and <a href="https://developer.hpe.com/blog/llm-agentic-tool-mesh-empowering-gen-ai-with-retrieval-augmented-generation-rag/">RAG Service</a> of <a href="https://github.com/HewlettPackard/llmesh">LLM Agentic Tool Mesh open source project</a>. Today, I'll explore the system services of LLM Agentic Tool Mesh, which are essential for managing and orchestrating the mesh of agentic tools.</p> <p>I'll provide insights into these services, showcase an example of a Mesh available in the repository, discuss federated governance, and share our vision for the future evolution of the LLM Agentic Tool Mesh project.</p> <p><img src="/img/mesh.png" alt=""></p> <h1>Understanding the system services</h1> <p>The system services in LLM Agentic Tool Mesh are crucial for the seamless operation and orchestration of agentic tools and web applications. These services ensure consistency, ease of use, and flexibility across the platform. They include:</p> <ol> <li><strong>Tool Client Service</strong></li> <li><strong>Tool Server Service</strong></li> </ol> <p>Let's explore each of these components in detail.</p> <h2>Tool Client Service</h2> <p>The Tool Client Service enables developers to transform any code function into an LLM Agentic Tool Mesh tool by applying a simple decorator. This service abstracts the complexities of tool integration, allowing for quick conversion of functions into reusable tools within the LLM Agentic Tool Mesh ecosystem.</p> <p>Key features:</p> <ul> <li>Decorator-based: Convert functions into tools using the <code>@AthonTool</code> decorator</li> <li>Seamless integration: Decorated functions are fully integrated into the LLM Agentic Tool Mesh platform</li> </ul> <p>Example usage:</p> <pre><code class="language-python">from athon.system import AthonTool, Logger config = { "greetings": "Hello World! Welcome, " } logger = Logger().get_logger() @AthonTool(config, logger) def hello_world(query: str) -> str: """Greets the user.""" greeting_message = f"{config['greetings']} {query}!" return greeting_message.capitalize() </code></pre> <h2>Tool Server Service</h2> <p>The Tool Server Service provides the necessary infrastructure to manage and run LLM Agentic Tool Mesh tools on the platform. It includes capabilities for tool discovery and execution, ensuring that tools are easily accessible and efficiently managed.</p> <p>Key features:</p> <ul> <li>Tool discovery: Automatically discover tools within the platform</li> <li>Execution management: Manage the execution of tools, ensuring efficient operation</li> </ul> <p><img src="/img/tools.png" alt=""></p> <p>Example usage:</p> <pre><code class="language-python">from athon.agents import ToolRepository from athon.system import ToolDiscovery projects_config = [ { "name": "Simple Project", "tools": [ "examples/local_tool", # A local tool path "https://127.0.0.1:5003/", # A remote tool URL ] } ] tools_config = { "type": "LangChainStructured" } def discover_and_load_tools(projects_config, tools_config): tool_repository = ToolRepository.create(tools_config) tool_discovery = ToolDiscovery() tool_id_counter = 1 for project in projects_config: for tool_reference in project["tools"]: tool_info = tool_discovery.discover_tool(tool_reference) if tool_info: tool_metadata = { "id": tool_id_counter, "project": project["name"], "name": tool_info["name"], "interface": tool_info.get("interface") } tool_repository.add_tool(tool_info["tool"], tool_metadata) tool_id_counter += 1 return tool_repository # Run the tool discovery and loading process tool_repository = discover_and_load_tools(projects_config, tools_config) # Display the discovered tools for tool in tool_repository.get_tools().tools: print(f"Discovered tool: {tool['name']} from project: {tool['metadata']['project']}") </code></pre> <h1>Building a mesh of LLM Agentic Tools</h1> <p>We have developed a series of web applications and tools, complete with examples, to demonstrate the capabilities of LLM Agentic Tool Mesh in our <a href="https://github.com/HewlettPackard/llmesh">GitHub repo</a>.</p> <p>Web applications:</p> <ul> <li><strong>Chatbot</strong> (<code>examples/app_chatbot</code>): This is a chatbot capable of reasoning and invoking appropriate LLM tools to perform specific actions. You can configure the chatbot using files that define LLM Agentic Tool Mesh platform services, project settings, toolkits, and memory configurations. The web app orchestrates both local and remote LLM tools, allowing them to define their own HTML interfaces, supporting text, images, and code presentations.</li> <li><strong>Admin panel</strong> (<code>examples/app_backpanel</code>): There's an admin panel that enables the configuration of basic LLM tools to perform actions via LLM calls. It allows you to set the system prompt, select the LLM model, and define the LLM tool interface, simplifying the process of configuring LLM tool interfaces.</li> </ul> <p>Tools:</p> <ul> <li><strong>Basic copywriter</strong> (<code>examples/tool_copywriter</code>): This tool rewrites text, providing explanations for enhancements and changes.</li> <li><strong>Temperature finder</strong> (<code>examples/tool_api</code>): This tool fetches and displays the current temperature for a specified location by utilizing a public API.</li> <li><strong>Temperature analyzer</strong> (<code>examples/tool_analyzer</code>): Another tools generates code, using a language model to analyze historical temperature data and create visual charts for better understanding</li> <li><strong>Telco expert</strong> (<code>examples/tool_rag</code>): The RAG tool provides quick and accurate access to 5G specifications.</li> <li><strong>OpenAPI manager</strong> (<code>examples/tool_agents</code>): This multi-agent tool reads OpenAPI documentation and provides users with relevant information based on their queries.</li> </ul> <h2>Running the examples</h2> <p>You can run the tools and web applications individually or use the provided <code>run_examples.sh</code> script to run them all together. Once everything is started:</p> <ul> <li>Access the chatbot at <code>https://127.0.0.1:5001/</code></li> <li>Access the admin panel at <code>https://127.0.0.1:5011/</code></li> </ul> <p><img src="/img/mesh-apps.png" alt=""></p> <h1>Federated governance and standards</h1> <p>In the LLM Agentic Tool Mesh platform, the management of LLM tools is decentralized, promoting flexibility and innovation. To ensure this decentralization does not compromise the platform's integrity, LLM Agentic Tool Mesh implements a unified framework of governance policies and standards.</p> <p>Key principles:</p> <ul> <li><strong>Interoperable standards</strong>: Ensures all tools and services work together seamlessly while adhering to best practices</li> <li><strong>Ethical compliance</strong>: Emphasizes minimizing biases, ensuring fairness, and upholding ethical principles across all AI tools and models</li> <li><strong>Security and privacy</strong>: Maintains rigorous standards to protect data and ensure compliance with privacy regulations.</li> <li><strong>Continuous improvement</strong>: Encourages feedback and collaboration to refine governance practices</li> <li><strong>Automated governance</strong>: Plans to extend code quality checks to enforce governance policies, ensuring comprehensive compliance across the platform</li> </ul> <p>LLM Agentic Tool Mesh includes a dedicated repository containing text files that outline various policies and standards (<code>federated_governance/</code>). These documents cover essential areas such as LLM model usage, RAG processes, and more.</p> <h1>Vision for LLM Agentic Tool Mesh project evolution</h1> <p>The LLM Agentic Tool Mesh platform is evolving into a comprehensive solution, providing several panel views tailored for various stages of the tool and application lifecycle.</p> <p>When creating a new web app, you can build upon the existing examples. With all services fully parameterized, there is unparalleled flexibility to design diverse user experience panels. For instance, current examples include a chatbot as a user interface and an admin panel for configuring an LLM tool. Additionally, web apps can be developed to support deployment tasks or facilitate experiments aimed at optimizing service parameters for specific objectives.</p> <p>Currently, the platform provides a user panel and a dvelopment panel.</p> <p><strong>User panel</strong>:</p> <ul> <li>Implemented features: This panel focuses on engaging with tools like Chat, RAG, and Agent services. It provides a user-friendly interface for interacting with these capabilities.</li> <li>Future goals: We plan to enrich the existing services, offering an even more seamless and feature-rich experience for end-users.</li> </ul> <p><strong>Development panel</strong>:</p> <ul> <li>Implemented features: This panel has been partially tackled with the backpanel web app example, which allows users to runtime modify the basic copywriter agentic tool and the RAG tool.</li> <li>Future goals: We aim to add more system services to support development, including real-time LLM tuning and configuration.</li> </ul> <p>In the future, we hope to be able to offer a deployment panel and an experiment panel.</p> <p><strong>Deployment panel (future)</strong>:</p> <ul> <li>Purpose: This panel will focus on deploying the LLM Agentic Tool Mesh tools seamlessly across one or more clusters, enabling large-scale and distributed deployments.</li> <li>Planned features: We hope to offer tools for monitoring deployed tools, orchestrating distributed systems, and managing deployment pipelines.</li> </ul> <p><strong>Experiment Panel (Future)</strong>:</p> <ul> <li>Purpose: This panel will be designed to track and manage experiments to optimize LLM tool performance and suitability.</li> <li>Planned features: This panel will allow users to try different configurations and compare outcomes, helping teams evaluate the most effective settings for their use cases.</li> </ul> <p><img src="/img/usage.png" alt=""></p> <h1>Our mission ahead</h1> <p>As LLM Agentic Tool Mesh evolves, we aim to continue enhancing the platform by enriching existing services, especially in the user panel, and expanding the development panel with more robust system services. The addition of the deployment panel and experiment panel will complete the platform's vision, enabling a fully integrated lifecycle from development to deployment and optimization.</p> <p>Stay tuned as we advance toward democratizing Gen AI with a comprehensive, flexible, and user-centric platform!</p>https://developer.hpe.com/using-the-gdb4hpc-debugger-at-scale-with-a-cuda-mpi-hpc-application/https://developer.hpe.com/using-the-gdb4hpc-debugger-at-scale-with-a-cuda-mpi-hpc-application/Fri, 17 Jan 2025 16:37:53 GMT<p>Command line debuggers for Linux have existed for decades. The GNU debugger (gdb) is the most famous and arguably the most powerful. But gdb has a weakness when it comes to high performance computing (HPC) applications - it can only run on one system at a time. A typical HPC application runs tens of thousands of processes on thousands of systems at once! Classic debuggers like gdb were never designed for that.</p> <p>This is where gdb4hpc, the gdb based debugger for HPC, comes in. The gdb4hpc debugger is part of the HPE Cray Programming Environment package. It captures and extends the powerful single-system features of gdb and provides a familiar command line debugging experience for HPC applications running on thousands of nodes.</p> <p>To provide these features, gdb4hpc connects the user to many instances of gdb at once. While connected, it provides tools enabling the user to control all of the instances of gdb simultaneously. Familiar debugging features like breakpoints, backtraces, and printing variables can be applied to all processes simultaneously. Additionally, more advanced debugging features like attaching to a running application and debugging GPU kernels are supported.</p> <p>To help the user manage the inherent complexities of debugging thousands of processes at once, the gdb4hpc debugger aggregates and filters the results from potentially thousands of gdb instances into representations that will comfortably fit on a single terminal screen. Additionally, it provides tools to inspect large amounts of data like the very large arrays that are typical in HPC applications.</p> <center><img src="/img/gdb4hpc-controlling-gdbs.png" width="95%" alt="Illustration showing how gdb4hpc connects to individual gdb instances. A single gdb4hpc instance is run on the login node. On each compute node, multiple application ranks are running. Each application rank has an instance of gdb attached to it. In turn, gdb4hpc remotely attaches to each individual gdb instance." title="gdb4hpc controlling multiple instances of gdb"></center> <p>In this tutorial, you will learn how to debug a multinode MPI/CUDA application with gdb4hpc in the HPE Cray Programming Environment. This tutorial uses a CUDA application and NVIDIA GPUs as examples, but the concepts are applicable to HIP (AMD's equivalent of NVIDIA's CUDA) applications on AMD GPUs as well.</p> <h2>Prerequisites</h2> <p>This tutorial assumes you already have a familiarity with <a href="https://sourceware.org/gdb/">command line debuggers</a>, <a href="https://git-scm.com/">Git</a>, <a href="https://en.wikipedia.org/wiki/Message_Passing_Interface">MPI</a>, and <a href="https://developer.nvidia.com/cuda-toolkit">CUDA</a>.</p> <p>You will need a system with access to Cray C++ compilers and gdb4hpc, which are parts of the HPC Cray Programming Environment. You will also need access to the NVIDIA C++ compiler.</p> <h2>Setup</h2> <p>To set up the debugging session, you will need to obtain the sample application and compile it for your chosen GPU with debug information.</p> <h3>Download the sample code</h3> <p>For this tutorial, you will use the <code>simpleMPI</code> example from <a href="https://github.com/NVIDIA/cuda-samples">NVIDIA's cuda-samples repository</a>. The <code>simpleMPI</code> application does a distributed calculation involving square roots and averages:</p> <ol> <li><code>simpleMPI</code> generates millions of random floats (2,560,000 per node)</li> <li><code>simpleMPI</code> uses MPI to distribute those numbers across multiple compute nodes</li> <li><code>simpleMPI</code> uses uses CUDA to calculate the square root of each number</li> <li><code>simpleMPI</code> uses MPI to collect the resulting square roots and average them</li> </ol> <p>Start by cloning the sample code repository.</p> <pre><code class="language-shell">$ git clone https://github.com/NVIDIA/cuda-samples.git </code></pre> <p>Next, navigate to the directory containing the <code>simpleMPI</code> sample.</p> <pre><code class="language-shell">$ cd cuda-samples/Samples/0_Introduction/simpleMPI </code></pre> <p>The sample comes with some source files, some Visual Studio project files, a Makefile, and a README.</p> <pre><code class="language-shell">$ ls Makefile simpleMPI.cu simpleMPI_vs2017.vcxproj simpleMPI_vs2022.sln README.md simpleMPI.h simpleMPI_vs2019.sln simpleMPI_vs2022.vcxproj simpleMPI.cpp simpleMPI_vs2017.sln simpleMPI_vs2019.vcxproj </code></pre> <p>For this tutorial, you will only need the following source files:</p> <pre><code class="language-shell">simpleMPI.cpp simpleMPI.cu simpleMPI.h </code></pre> <p>Remove the other files, if desired.</p> <pre><code class="language-shell">$ rm *.sln *.vcxproj README.md Makefile </code></pre> <h3>Build the application</h3> <p>To compile the application, load the appropriate compilers and compile the sample application for your chosen GPU architecture with debug information.</p> <h4>Load compilers</h4> <p>The version of the <code>cce</code> and <code>cuda</code> compilers that you should load depend on which GPU architecture you will be compiling for and the version of the CUDA drivers on your system. Find your GPU on the <a href="https://developer.nvidia.com/cuda-gpus">NVIDIA GPU Compute Capability page</a> and choose a version of CUDA that supports the required compute capability. Then load an appropriate CCE version. For information about which CCE versions support which versions of CUDA, see the <a href="https://cpe.ext.hpe.com/docs/latest/release_announcements/index.html">HPE Cray Programming Environment release announcements</a>. If in doubt, the default versions on your system should work just fine.</p> <p>When writing this tutorial, I used CCE 17 and CUDA 12 and ran the application on NVIDIA A100 GPUs, but the debugging process is the same for any compiler combination.</p> <p>After finding which versions you need, use the <code>module</code> command in a terminal to ensure that the Cray <code>CC</code> compiler and the NVIDIA <code>nvcc</code> compiler are available.</p> <pre><code class="language-shell">$ module load cce/17 $ module load cuda/12 </code></pre> <p>You can check that your environment is set up correctly by checking that CC (the Cray CCE C++ compiler) and nvcc (the NVIDIA CUDA application compiler) are available in the terminal.</p> <pre><code class="language-shell">$ CC --version Cray clang version 17.0.1 (5ec9405551a8c8845cf14e81dc28bff7aa3935cb) ... $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Jun__6_02:18:23_PDT_2024 Cuda compilation tools, release 12.5, V12.5.82 ... </code></pre> <h4>Compile the application</h4> <p>The sample application comes with a Makefile, but it's not compatible with the HPE Cray Programming Environment. You will compile the application manually.</p> <h4>Include debug info</h4> <p>When compiling an application, compilers apply optimizations to make the resulting application faster and smaller. These optimizations often rearrange the order of code or completely remove it which makes reasoning about the original code at runtime difficult. When preparing to debug an application, you need to tell the compiler to disable these optimizations and include extra information that the debugger can use to reason about the original code.</p> <h4>Compile the .cu file</h4> <p>The simpleMPI.cu file contains the code that will be run on the GPU. Use nvcc to compile it.</p> <pre><code class="language-shell">$ nvcc -g -G -O0 -c -ccbin=cc -gencode arch=... simpleMPI.cu -o simpleMPI_gpu.o </code></pre> <p>The <code>-g</code> flag tells nvcc to include debugging information for the host code (the code that will run on the CPU) and the <code>-G</code> flag tells nvcc to include debugging information for the device code (the code that will run on the GPU). <code>-O0</code> disables optimizations for the host code. <code>-G</code> implicitly disables optimizations for the device code.</p> <p><code>-c</code> tells the compiler to produce an object file instead of a executable. An object file is a mostly-compiled program that only needs to be linked to create an executable. After you compile the .cpp file in the next section, you will link the object files together to create the final executable.</p> <p>The nvcc compiler does not actually fully compile C/C++. It actually processes the CUDA directives in the .cu file and produces a new C/C++ file where the CUDA directives are replaced with the proper C/C++ code that interacts with the CUDA API to do the work of loading the correct GPU driver etc. <code>-ccbin=cc</code> tells nvcc to use the <code>cc</code> compiler to do the actual compilation of the generated C/C++ code. When the Cray CCE module is loaded, cc is the Cray C/C++ compiler.</p> <p><code>-gencode</code> tells nvcc which GPU architecture to target. The actual value for this argument will vary based on which GPU you are targeting. Refer to the <a href="https://developer.nvidia.com/cuda-gpus">NVIDIA compute capabilities table</a> to find the compute capability of your GPU. For example, the A100 GPU that I am targeting while writing this tutorial has a compute capability of 8.0. Take the compute capability of your GPU and pass it to nvcc via the <code>-gencode</code> flag by removing the decimal point and embedding it in an <code>arch=compute_XX,code=sm_XX</code> pair. For my A100 example, the complete option is <code>-gencode compute_80,code=sm_80</code>.</p> <p><code>simpleMI.cu</code> is the input file and <code>-o simpleMPI_gpu.o</code> tells <code>nvcc</code> to call the output file <code>simpleMPI_gpu.o</code>.</p> <p>My complete nvcc invocation for an A100 GPU looks like this:</p> <pre><code class="language-shell">$ nvcc -g -G -c -ccbin=cc -gencode arch=compute_80,code=sm_80 simpleMPI.cu -o simpleMPI_gpu.o </code></pre> <p>Refer to the <a href="https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html"><code>nvcc</code> user manual</a> and the <a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html#compiling-the-application">compilation section of the cuda-gdb documentation</a> for more information.</p> <h4>Compile the .cpp file</h4> <p>The simpleMPI.cpp file contains code that will be run on the CPU. It is the entry point to the program and contains the MPI operations. Use CC to compile it.</p> <pre><code class="language-shell">$ CC -g -O0 -c simpleMPI.cpp -o simpleMPI_cpu.o </code></pre> <p>The flags have the same meanings as in the nvcc compilation. The Cray CC compiler will automatically include the flags needed for building with MPI.</p> <h4>Link the application and produce an executable</h4> <p>Finally, use CC to link the two object files into the final executable.</p> <pre><code class="language-shell">$ CC simpleMPI_gpu.o simpleMPI_cpu.o -o simpleMPI </code></pre> <h4>Test the application</h4> <p>Test that the application was compiled correctly by running it on multiple nodes. For example, on a Slurm system:</p> <pre><code class="language-shell">$ srun -n8 -N2 --ntasks-per-node 4 --exclusive -p griz256 ./simpleMPI Running on 8 nodes Average of square roots is: 0.667337 PASSED </code></pre> <p>Your srun command's flags may vary depending on your system. You should launch the job on nodes with your targeted GPU with one rank per GPU. See <a href="https://slurm.schedmd.com/srun.html">the <code>srun</code> documentation</a> for how to achieve a correct job launch on your system.</p> <p>In my specific example:</p> <p><code>-n8 -N2 --ntasks-per-node 4</code> tells Slurm to run 8 ranks across 2 nodes, splitting the job evenly by running 4 ranks on each node. In this example I am using nodes with 4 A100s each.</p> <p><code>--exclusive</code> tells Slurm to run on nodes that aren't being used by anyone else, and to not let anyone else use them while running the job. Only one application can use each GPU at a time, so you need to make sure that you aren't given nodes that have GPUs that are already in use.</p> <p><code>-p griz256</code> tells Slurm to use the nodes in the griz256 partition. In my case, the griz256 partition designates all the nodes with A100s. Your partition name will probably be different.</p> <h2>Debugging</h2> <h3>Load gdb4hpc and cuda-gdb</h3> <p>You're ready to start debugging! Load the gdb4hpc debugger.</p> <pre><code class="language-shell">$ module load gdb4hpc $ gdb4hpc --version gdb4hpc-4.16.3. </code></pre> <p>For this tutorial, I am using gdb4hpc 4.16.3, but any version above 4.13.1 will have the commands used in this tutorial.</p> <p>The gdb4hpc debugger will internally use the cuda-gdb debugger for GPU debugging. cuda-gdb is provided by the cuda module that you loaded for building the application. Check that cuda-gdb is available.</p> <pre><code class="language-shell">$ cuda-gdb --version NVIDIA (R) cuda-gdb 12.5 Portions Copyright (C) 2007-2024 NVIDIA Corporation Based on GNU gdb 13.2 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later &#x3C;http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. </code></pre> <h4>Launch the application in gdb4hpc</h4> <p>Start gdb4hpc. You will be dropped into a command line interface. gdb4hpc is operated by issuing debugging commands on the command line. The commands should be familiar to anyone who has used gdb before.</p> <pre><code class="language-shell">$ gdb4hpc gdb4hpc 4.16.3. - Cray Interactive Parallel Debugger With Cray Comparative Debugging Technology. Copyright 2007-2024 Hewlett Packard Enterprise Development LP. Copyright 1996-2016 University of Queensland. All Rights Reserved. Type "help" for a list of commands. Type "help &#x3C;cmd>" for detailed help about a command. dbg all> </code></pre> <p>Use the <code>launch</code> command to start the application in the debugger.</p> <pre><code class="language-shell">dbg all> launch $simpleMPI{8} --launcher-args="-N2 --ntasks-per-node=4 --exclusive -p griz256" --gpu ./simpleMPI Starting application, please wait... Launched application... 0/8 ranks connected... (timeout in 300 seconds) .. 8/8 ranks connected. Created network... Connected to application... Launch complete. simpleMPI{0..7}: Initial breakpoint, main at simpleMPI.cpp:63 </code></pre> <p><code>$simpleMPI{8}</code> tells gdb4hpc to launch a job with 8 ranks and give it a handle called <code>simpleMPI</code>.</p> <p><code>--launcher-args</code> tells gdb4hpc which WLM arguments to use to launch the job correctly. Use the same arguments you used in the <strong>Test The Application</strong> section.</p> <p>The <code>--gpu</code> flag tells gdb4hpc to use the appropriate GPU-aware debugger. Here, that's cuda-gdb.</p> <p>The final argument, <code>./simpleMPI</code>, specifies the binary to launch.</p> <p>After the launch completes, gdb4hpc stops at <code>main</code> and gives you control.</p> <h4>Stop at a breakpoint in a GPU kernel</h4> <p>The sample application contains one CUDA Kernel called <code>simpleMPIKernel</code> defined in simpleMPI.cu. The source code for the kernel looks like this:</p> <pre><code class="language-cpp">52 // Device code 53 // Very simple GPU Kernel that computes square roots of input numbers 54 __global__ void simpleMPIKernel(float *input, float *output) { 55 int tid = blockIdx.x * blockDim.x + threadIdx.x; 56 output[tid] = sqrt(input[tid]); 57 } </code></pre> <p>Set a breakpoint at the first line of the kernel with the <code>break</code> command, just like you would in gdb:</p> <pre><code class="language-shell">dbg all> break simpleMPI.cu:55 simpleMPI{0..7}: Breakpoint 1: file simpleMPI.cu, line 55. </code></pre> <blockquote> <p>You could have also set the breakpoint via the function name by typing <code>break simpleMPIKernel</code>.</p> </blockquote> <p>Now use the <code>continue</code> command to run the application until it reaches the breakpoint in the kernel.</p> <pre><code class="language-shell">dbg all> continue dbg all> &#x3C;$simpleMPI>: Running on 8 nodes simpleMPI{0..7}: Breakpoint 1, at simpleMPI.cu:55 </code></pre> <p>The application has stopped in a GPU kernel. From here you can use the usual debugger commands like <code>print</code>, <code>step</code>, <code>next</code>, etc. You can use <code>print</code> to inspect GPU memory as you would any other memory. Here I print the first 8 items of the <code>input</code> array. Since the program initializes the array with random data, the result is different for each rank, and your data might be different.</p> <pre><code class="language-shell">dbg all> info locals simpleMPI{0..7}: Name:tid Type:@register int simpleMPI{0..7}: Name:input Type:@generic float * @parameter simpleMPI{0..7}: Name:output Type:@generic float * @parameter dbg all> print input[0]@8 # print the first 8 items of the input array simpleMPI{0}: {0.840188,0.394383,0.783099,0.79844,0.911647,0.197551,0.335223,0.76823} simpleMPI{1}: {0.432718,0.407395,0.918358,0.348798,0.421943,0.527366,0.664573,0.899948} simpleMPI{2}: {0.490992,0.530234,0.736292,0.0698217,0.825561,0.719689,0.26409,0.439787} simpleMPI{3}: {0.404152,0.403803,0.990251,0.10481,0.267578,0.685576,0.794025,0.972998} simpleMPI{4}: {0.113738,0.293829,0.443866,0.706545,0.762252,0.451777,0.93063,0.569985} simpleMPI{5}: {0.386505,0.760585,0.354206,0.784775,0.329661,0.25768,0.0700815,0.955718} simpleMPI{6}: {0.959459,0.299068,0.889675,0.0221367,0.809361,0.0220988,0.373968,0.589243} simpleMPI{7}: {0.839492,0.778462,0.195212,0.832727,0.125331,0.211325,0.0624168,0.56848} </code></pre> <h4>Inspect CUDA threads and use CUDA commands</h4> <p>The gdb4hpc debugger supports cuda-gdb's <code>cuda</code> commands.</p> <p>To get info about currently running CUDA threads, use <code>info cuda threads</code>.</p> <pre><code class="language-shell">dbg all> info cuda threads simpleMPI{0}: Showing one thread per location; use -v for full list. simpleMPI{0}: BlockIdx ThreadIdx Count Location simpleMPI{0}: Kernel 0 simpleMPI{0}: (649,0,0) (0,0,0) 7840 simpleMPI.cu 54 simpleMPI{0}: * (0,0,0) (0,0,0) 213344 simpleMPI.cu 55 simpleMPI{1}: Showing one thread per location; use -v for full list. simpleMPI{1}: BlockIdx ThreadIdx Count Location simpleMPI{1}: Kernel 0 simpleMPI{1}: (649,0,0) (0,0,0) 7936 simpleMPI.cu 54 simpleMPI{1}: * (0,0,0) (0,0,0) 213248 simpleMPI.cu 55 simpleMPI{2}: Showing one thread per location; use -v for full list. simpleMPI{2}: BlockIdx ThreadIdx Count Location simpleMPI{2}: Kernel 0 simpleMPI{2}: (651,0,0) (0,0,0) 8896 simpleMPI.cu 54 simpleMPI{2}: * (0,0,0) (0,0,0) 212288 simpleMPI.cu 55 ... </code></pre> <p>You can see that each rank's GPU threads are all stopped in the <code>simpleMPIKernel</code> in <code>simpleMPI.cu</code> lines 54 or 55. The ranks have different locations because gdb4hpc debugs CUDA applications in <a href="https://sourceware.org/gdb/current/onlinedocs/gdb.html/All_002dStop-Mode.html">all-stop mode</a>. All-stop mode means that once one thread stops, all other threads are stopped immediately. This means that every thread was stopped as soon as the first thread hit the breakpoint, resulting in all threads halting wherever they happened to be at the time. Because of this, your locations might be slightly different.</p> <p>The current thread (the rank that any gdb4hpc commands will apply to) is indicated with a <code>*</code> symbol. You can switch the current block or thread (or warp or lane) by using the <code>cuda [block x] thread y</code> command. Since this application uses a one dimensional grid of one dimensional thread blocks, you can derive block and thread IDs by simply using the first number in the block's/thread's (x, 0, 0) tuple.</p> <p>The example above shows block 649 on rank 0 as being in a different location than block 0 thread 0. Your exact block and thread numbers might be different. Use <code>cuda block x thread y</code> to switch to a different block and thread. You can confirm the switch by printing the CUDA kernel built in variables <code>blockIdx</code> and <code>threadIdx</code>.</p> <pre><code class="language-shell">dbg all> cuda block 649 thread 0 simpleMPI{0}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 2, sm 0, warp 59, lane 0] simpleMPI{0}: 0x00007fcef725da10 54 in simpleMPI.cu simpleMPI{1}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 0, sm 0, warp 59, lane 0] simpleMPI{1}: 0x00007f282725da20 54 in simpleMPI.cu simpleMPI{2}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 3, sm 0, warp 59, lane 0] simpleMPI{2}: 55 in simpleMPI.cu simpleMPI{3}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 1, sm 0, warp 59, lane 0] simpleMPI{3}: 0x00007f9dc725da20 54 in simpleMPI.cu simpleMPI{4}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 2, sm 0, warp 59, lane 0] simpleMPI{4}: 0x00007ff3d725da20 54 in simpleMPI.cu simpleMPI{5}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 0, sm 0, warp 59, lane 0] simpleMPI{5}: 0x00007f79c325da20 54 in simpleMPI.cu simpleMPI{6}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 3, sm 0, warp 59, lane 0] simpleMPI{6}: 0x00007f410525da20 54 in simpleMPI.cu simpleMPI{7}: [Switching focus to CUDA kernel 0, grid 1, block (649,0,0), thread (0,0,0), device 1, sm 4, warp 50, lane 0] simpleMPI{7}: 55 in simpleMPI.cu dbg all> print blockIdx simpleMPI{0..7}: {x = 649, y = 0, z = 0} dbg all> print threadIdx simpleMPI{0..7}: {x = 0, y = 0, z = 0} </code></pre> <h4>Use more CUDA commands</h4> <p>To see what other <code>info</code> commands are available, use <code>cuda info help</code>:</p> <pre><code class="language-shell">dbg all> cuda help info simpleMPI{0..7}: Print informations about the current CUDA activities. Available options: simpleMPI{0..7}: devices : information about all the devices simpleMPI{0..7}: sms : information about all the SMs in the current device simpleMPI{0..7}: warps : information about all the warps in the current SM simpleMPI{0..7}: lanes : information about all the lanes in the current warp simpleMPI{0..7}: kernels : information about all the active kernels simpleMPI{0..7}: contexts : information about all the contexts simpleMPI{0..7}: blocks : information about all the active blocks in the current kernel simpleMPI{0..7}: threads : information about all the active threads in the current kernel simpleMPI{0..7}: launch trace : information about the parent kernels of the kernel in focus simpleMPI{0..7}: launch children : information about the kernels launched by the kernels in focus simpleMPI{0..7}: managed : information about global managed variables simpleMPI{0..7}: line : information about the filename and linenumber for a given $pc simpleMPI{0..7}: </code></pre> <p>There are also other top level <code>cuda</code> commands other than <code>info</code>:</p> <pre><code class="language-shell">dbg all> cuda help simpleMPI{0..7}: Print or select the CUDA focus. simpleMPI{0..7}: simpleMPI{0..7}: List of cuda subcommands: simpleMPI{0..7}: simpleMPI{0..7}: cuda block -- Print or select the current CUDA block. simpleMPI{0..7}: cuda device -- Print or select the current CUDA device. simpleMPI{0..7}: cuda grid -- Print or select the current CUDA grid. simpleMPI{0..7}: cuda kernel -- Print or select the current CUDA kernel. simpleMPI{0..7}: cuda lane -- Print or select the current CUDA lane. simpleMPI{0..7}: cuda sm -- Print or select the current CUDA SM. simpleMPI{0..7}: cuda thread -- Print or select the current CUDA thread. simpleMPI{0..7}: cuda warp -- Print or select the current CUDA warp. simpleMPI{0..7}: simpleMPI{0..7}: Type "help cuda" followed by cuda subcommand name for full documentation. simpleMPI{0..7}: Type "apropos word" to search for commands related to "word". simpleMPI{0..7}: Type "apropos -v word" for full documentation of commands related to "word". simpleMPI{0..7}: Command name abbreviations are allowed if unambiguous. </code></pre> <p>See <a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html">the cuda-gdb documentation</a> for more information.</p> <h4>Single step in a GPU kernel</h4> <p>The usual <code>step</code> and <code>next</code> commands are supported. In a GPU kernel, stepping has the granularity of a warp. A warp is a group of 32 GPU threads and is the granularity the GPU scheduler works at. See <a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html#single-stepping">the cuda-gdb documentation</a> for more info.</p> <pre><code class="language-shell">simpleMPI{0..7}: Breakpoint 1, at simpleMPI.cu:55 dbg all> list 55 simpleMPI{0..7}: 50 } simpleMPI{0..7}: 51 simpleMPI{0..7}: 52 // Device code simpleMPI{0..7}: 53 // Very simple GPU Kernel that computes square roots of input numbers simpleMPI{0..7}: 54 __global__ void simpleMPIKernel(float *input, float *output) { simpleMPI{0..7}: 55 int tid = blockIdx.x * blockDim.x + threadIdx.x; simpleMPI{0..7}: 56 output[tid] = sqrt(input[tid]); simpleMPI{0..7}: 57 } simpleMPI{0..7}: 58 simpleMPI{0..7}: 59 // Initialize an array with random data (between 0 and 1) dbg all> next # set tid simpleMPI{0..7}: 56 output[tid] = sqrt(input[tid]); dbg all> p tid simpleMPI{0..7}: 64 dbg all> p output[tid] simpleMPI{0..7}: 0 dbg all> n simpleMPI{0..7}: 57 } dbg all> p output[tid] simpleMPI{0}: 0.516397 simpleMPI{1}: 0.392356 simpleMPI{2}: 0.883166 simpleMPI{3}: 0.806327 simpleMPI{4}: 0.458106 simpleMPI{5}: 0.999046 simpleMPI{6}: 0.366925 simpleMPI{7}: 0.646477 </code></pre> <h2>Next steps</h2> <p>You now know how to start debugging a multinode MPI/CUDA application with gdb4hpc in the HPE Cray Programming Environment! You can now start tracking down bugs in HPC applications with thousands of processes just as easily as you would in a single process application, saving you time and sanity.</p> <p>In general, gdb4hpc's GPU debugging capabilities are that of the underlying GPU debugger. See the <a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html">cuda-gdb documentation</a> for more capabilities.</p> <p>For more information about gdb4hpc, see <a href="https://cpe.ext.hpe.com/docs/latest/debugging-tools/index.html#gdb4hpc">the gdb4hpc documentation</a>.</p> <p>Happy bug hunting!</p>https://developer.hpe.com/using-the-hpegl-hpe-greenlake-and-morpheus-terraform-providers-in-concert-to-create-provision-and-manage-hpe-pce-private-cloud-enterprise-vmaas-instances/https://developer.hpe.com/using-the-hpegl-hpe-greenlake-and-morpheus-terraform-providers-in-concert-to-create-provision-and-manage-hpe-pce-private-cloud-enterprise-vmaas-instances/Wed, 15 Jan 2025 14:58:20 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h1><strong>Introduction</strong></h1> <p>Many customers of HPE GreenLake for Private Cloud Enterprise want to use <a href="https://www.terraform.io/">Terraform</a> or the open-source alternative <a href="https://opentofu.org/">OpenTofu</a> to create, provision and manage VMaaS (Virtual Machines as a Service, a service offering in HPE GreenLake for Private Cloud Enterprise) VMs. The cloud-based VMaaS service interacts with <a href="https://morpheusdata.com/">Morpheus</a> (recently acquired by HPE) running on-premise which provides the core Virtual Machine functionality. There is a HPE GreenLake Terraform provider called <em><a href="https://registry.terraform.io/providers/HPE/hpegl/latest">hpegl</a></em> (HPE GreenLake) and a separate <a href="https://registry.terraform.io/providers/gomorpheus/morpheus/latest">Morpheus</a> provider. Both providers complement each other and when used in concert offer a rich set of IaC capabilities.</p> <p>In this blog post we will demonstrate how to use the two providers in concert to create, provision and manage VMaaS VM instances. Central to this is a hpegl data-source which can be used to retrieve an Access Token and URL for the on-premise Morpheus instance which are then passed to the Morpheus provider.</p> <p>Note that we plan to write and release a new Terraform provider which will combine the functionality of the hpegl and Morpheus providers into a single provider and replace both. This will simplify the process of creating, provisioning and managing VM instances. It will also extend the provider functionality. This new provider will be released at a future date and will be the subject of a follow-on blog post.</p> <h1><strong>IaC and HPE GreenLake for Private Cloud Enterprise</strong></h1> <p><a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">IaC</a> (Infrastructure as Code) is a well-established approach to configuring, creating, and managing resources. Terraform and its OpenSource alternative OpenTofu, along with per-service providers, are especially popular. For HPE GreenLake for Private Cloud Enterprise's VMaaS (VM as a Service) service, there are two relevant providers: </p> <ul> <li>hpegl (HPE GreenLake), which interacts with the HPE GreenLake for Private Cloud Enterprise Services such as Identity and Access Management (IAM) and VMaaS. </li> <li>Morpheus, which interacts with the on-premise Morpheus instance or instances that are associated with the VMaaS service.  Each Morpheus instance is a separate on-premise installation of Morpheus. </li> </ul> <p>Note that we plan to write and release a new unified provider which will replace these providers and offer a richer set of functionality.</p> <p>These two current providers (hpegl and Morpheus) complement each other: </p> <ul> <li>hpegl deals with HPE GreenLake IAM and manages VMaaS resources, including VM instances. </li> <li>Morpheus deals with underlying Morpheus resources on which the VMaaS resources – including VM instances – depend. This covers resources such as Clouds, VM images, Node Types, Groups, Instance Types, and Instance Layouts. </li> </ul> <p>The glue between these two providers is a hpegl <em>data source</em> called <em>hpegl_vmaas_morpheus_details</em>. For a specific VMaaS <em>location</em>, this data source will output: </p> <ul> <li>A Morpheus access token. </li> <li>The URL of the Morpheus instance. </li> <li>The time to expiry of the access token (in seconds). </li> </ul> <p>The first two outputs can then be passed into a Morpheus provider stanza. See the following example where there is just one <em>location</em> for a VMaaS instance:</p> <pre><code class="language-hcl">provider "hpegl" { vmaas { location = var.location space_name = var.space } } data "hpegl_vmaas_morpheus_details" "morpheus_details" {} provider "morpheus" { url = data.hpegl_vmaas_morpheus_details.morpheus_details.url access_token = data.hpegl_vmaas_morpheus_details.morpheus_details.access_token } </code></pre> <p>For VMaaS instances with multiple locations <em>provider aliasing</em> can be used to tie a specific VMaaS <em>location</em> to the corresponding Morpheus instance:</p> <pre><code class="language-hcl"># Location 1 provider "hpegl" { vmaas { location = var.location_1 space_name = var.space_1 } alias = "location_1" } data "hpegl_vmaas_morpheus_details" "location_1" { provider = hpegl.location_1 } provider "morpheus" { url = data.hpegl_vmaas_morpheus_details.location_1.url access_token = data.hpegl_vmaas_morpheus_details.location_1.access_token alias = "morpheus_location_1" } # Location 2 provider "hpegl" { vmaas { location = var.location_2 space_name = var.space_2 } alias = "location_2" } data "hpegl_vmaas_morpheus_details" "location_2" { provider = hpegl.location_2 } provider "morpheus" { url = data.hpegl_vmaas_morpheus_details.location_2.url access_token = data.hpegl_vmaas_morpheus_details.location_2.access_token alias = "morpheus_location_2" } </code></pre> <h1><strong>Complete examples for one VMaaS location</strong></h1> <p>In the next two sections we present complete HCL examples for two configurations each with one VMaaS location: </p> <ul> <li>One VMaaS cloud, with one VMaaS instance. </li> <li>Two VMaaS clouds, each with one VMaaS instance for a total of two VMaaS instances. </li> </ul> <p> The clouds are VMware vSphere clouds. In both cases: </p> <ul> <li> <p>hpegl is used to: </p> <ul> <li>Define the VMaaS location and space. </li> <li>Retrieve the relevant Morpheus token and URL. </li> <li>Retrieve details for the VMaaS Datastore, Network, Resource Pool, Plan, Environment, and Cloud Folder. </li> <li>Create a VMaaS instance. </li> </ul> </li> <li> <p>Morpheus is used to: </p> <ul> <li>Retrieve details for the VMaaS Cloud. </li> <li>Create a Group for the above cloud. </li> <li>Create an Instance Type. </li> <li>Retrieve details for a VM Image. </li> <li>Create a Node Type for the VM Image. </li> <li>Create an Instance Layout for the above Node Type and Instance Type.</li> </ul> </li> </ul> <h2><strong>One location, one VMaaS cloud, one VMaaS instance example</strong></h2> <p><img src="/img/hpegl-and-morpheus-terraform-blog-post-frame-1-1-.jpg" alt=""></p> <h3>HCL</h3> <p><a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Terraform/HPEGL-Morpheus-PCE-VMaaS/One-Location/One-Cloud-One-Instance/1_location_1_cloud">View the example HCL</a></p> <h3>The output from a Terraform (v1.5.7) run to create the VMaaS instance</h3> <pre><code class="language-shellsession">➜ 1_location_1_cloud git:(blog1) ✗ terraform apply -var-file variables_tenant.tfvars -auto-approve data.hpegl_vmaas_environment.env: Reading... data.hpegl_vmaas_network.blue_segment: Reading... data.hpegl_vmaas_plan.g1_large: Reading... data.hpegl_vmaas_morpheus_details.morpheus_details: Reading... data.hpegl_vmaas_plan.g1_large: Read complete after 0s [id=407] data.hpegl_vmaas_network.blue_segment: Read complete after 0s [id=14] data.hpegl_vmaas_environment.env: Read complete after 1s [id=1] data.hpegl_vmaas_morpheus_details.morpheus_details: Read complete after 2s [id=94de0616-db52-4737-8afe-aeef618db00b] data.morpheus_cloud.cloud: Reading... data.morpheus_virtual_image.example_virtual_image: Reading... data.morpheus_cloud.cloud: Read complete after 2s [name=HPE GreenLake VMaaS Cloud-Trial3] data.hpegl_vmaas_datastore.c_3par: Reading... data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... data.hpegl_vmaas_cloud_folder.compute_folder: Reading... data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 0s [id=3] data.hpegl_vmaas_datastore.c_3par: Read complete after 0s [id=1] data.morpheus_virtual_image.example_virtual_image: Read complete after 2s [name=ubuntu-20] data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 0s [id=6] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 1 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = "cfe_tf_example_instance" + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 407 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v41" + no_agent = false + resource_pool_id = 3 } + network { + id = 14 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "1" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 1, ] + code = "CFE TF Group" + id = (known after apply) + location = "galway" + name = "CFE TF Group" } # morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = "cfe_todo_app_frontend" + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = "cfe_tf_example_instance" + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = "cfe_tf_example_instance" + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = "cfe_tf_example_node_type_1" + script_template_ids = (known after apply) + short_name = "tfexamplenodetype1" + technology = "vmware" + version = "2.0" + virtual_image_id = 967 } # random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + max = 50000 + min = 1 + result = (known after apply) } Plan: 6 to add, 0 to change, 0 to destroy. morpheus_group.tf_example_group: Creating... morpheus_node_type.tf_example_node: Creating... morpheus_instance_type.tf_example_instance_type: Creating... random_integer.random: Creating... random_integer.random: Creation complete after 0s [id=34008] morpheus_node_type.tf_example_node: Creation complete after 1s [id=1968] morpheus_instance_type.tf_example_instance_type: Creation complete after 1s [id=207] morpheus_instance_layout.tf_example_instance_layout: Creating... morpheus_group.tf_example_group: Creation complete after 2s [id=83] morpheus_instance_layout.tf_example_instance_layout: Creation complete after 2s [id=1409] hpegl_vmaas_instance.sample_vm[0]: Creating... hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m40s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m50s elapsed] hpegl_vmaas_instance.sample_vm[0]: Creation complete after 1m51s [id=52688] Apply complete! Resources: 6 added, 0 changed, 0 destroyed. </code></pre> <h3>The output from an OpenTofu (v1.6.2) run to create the VMaaS instance</h3> <pre><code class="language-shellsession">➜ 1_location_1_cloud git:(blog1) ✗ tofu apply -var-file variables_tenant.tfvars -auto-approve data.hpegl_vmaas_network.blue_segment: Reading... data.hpegl_vmaas_morpheus_details.morpheus_details: Reading... data.hpegl_vmaas_plan.g1_large: Reading... data.hpegl_vmaas_environment.env: Reading... data.hpegl_vmaas_environment.env: Read complete after 0s [id=1] data.hpegl_vmaas_plan.g1_large: Read complete after 0s [id=407] data.hpegl_vmaas_network.blue_segment: Read complete after 1s [id=14] data.hpegl_vmaas_morpheus_details.morpheus_details: Read complete after 2s [id=94de0616-db52-4737-8afe-aeef618db00b] data.morpheus_cloud.cloud: Reading... data.morpheus_virtual_image.example_virtual_image: Reading... data.morpheus_cloud.cloud: Read complete after 1s [name=HPE GreenLake VMaaS Cloud-Trial3] data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... data.hpegl_vmaas_datastore.c_3par: Reading... data.hpegl_vmaas_cloud_folder.compute_folder: Reading... data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 1s [id=3] data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 1s [id=6] data.morpheus_virtual_image.example_virtual_image: Read complete after 2s [name=ubuntu-20] data.hpegl_vmaas_datastore.c_3par: Read complete after 1s [id=1] OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create OpenTofu will perform the following actions: # hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 1 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = "cfe_tf_example_instance" + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 407 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v41" + no_agent = false + resource_pool_id = 3 } + network { + id = 14 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "1" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 1, ] + code = "CFE TF Group" + id = (known after apply) + location = "galway" + name = "CFE TF Group" } # morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = "cfe_todo_app_frontend" + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = "cfe_tf_example_instance" + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = "cfe_tf_example_instance" + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = "cfe_tf_example_node_type_1" + script_template_ids = (known after apply) + short_name = "tfexamplenodetype1" + technology = "vmware" + version = "2.0" + virtual_image_id = 967 } # random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + max = 50000 + min = 1 + result = (known after apply) } Plan: 6 to add, 0 to change, 0 to destroy. morpheus_node_type.tf_example_node: Creating... morpheus_instance_type.tf_example_instance_type: Creating... morpheus_group.tf_example_group: Creating... random_integer.random: Creating... random_integer.random: Creation complete after 0s [id=38926] morpheus_instance_type.tf_example_instance_type: Creation complete after 2s [id=208] morpheus_node_type.tf_example_node: Creation complete after 2s [id=1969] morpheus_instance_layout.tf_example_instance_layout: Creating... morpheus_group.tf_example_group: Creation complete after 2s [id=84] morpheus_instance_layout.tf_example_instance_layout: Creation complete after 2s [id=1410] hpegl_vmaas_instance.sample_vm[0]: Creating... hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] hpegl_vmaas_instance.sample_vm[0]: Creation complete after 1m34s [id=52689] Apply complete! Resources: 6 added, 0 changed, 0 destroyed. </code></pre> <h2>One location, two VMaaS clouds, two VMaaS instances example</h2> <p><img src="/img/hpegl-and-morpheus-terraform-blog-post-frame-2-1-.jpg" alt=""></p> <p>The example HCL used here is based on the HCL for the first example.  However we have created two separate <em>Modules</em> based on that HCL: </p> <p><em>morpheus_artefacts</em> does the following. </p> <ul> <li> <p>Takes as input: </p> <ul> <li>A VMaaS cloud name </li> <li>A group name </li> <li>An image name </li> </ul> </li> <li> <p>Creates: </p> <ul> <li>A group </li> <li>An instance type </li> <li>An instance layout </li> <li>A node type </li> </ul> </li> <li> <p>Outputs: </p> <ul> <li>Instance type details </li> <li>Instance layout details </li> <li>Node type details </li> <li>Group details </li> <li>Cloud ID</li> </ul> </li> </ul> <p><em>vmaas_instance</em> does the following. </p> <ul> <li> <p>Takes inputs from an instance of the <em>morpheus_artefacts</em> module </p> </li> <li> <p>Retrieves details for the following VMaaS resources: </p> <ul> <li>VMaaS datastore </li> <li>VMaaS network </li> <li>VMaaS resource pool </li> <li>VMaaS plan </li> <li>VMaaS environment </li> <li>VMaaS cloud folder </li> </ul> </li> <li> <p>Creates a VMaaS instance </p> </li> </ul> <p>These modules can be combined in different ways. In our specific case we: </p> <ul> <li>Make two calls to <em>morpheus_artefacts</em>. </li> <li>Make two calls to <em>vmaas_instance</em> each using outputs from one of the calls to <em>morpheus_artefacts</em>.</li> </ul> <p>In this way we create two VMaaS instances each in a separate VMaaS cloud based on the one underlying on-premise Morpheus instance.</p> <h3>HCL</h3> <p><a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Terraform/HPEGL-Morpheus-PCE-VMaaS/One-Location/One-Cloud-One-Instance/1_location_1_cloud">View the example HCL</a></p> <h3>The output from a Terraform (v1.5.7) run to create the two VMaaS instances</h3> <pre><code class="language-shellsession">➜ 2_clouds_module_nest git:(blog1) ✗ terraform apply -var-file variables_tenant.tfvars -auto-approve data.hpegl_vmaas_morpheus_details.morpheus_details: Reading... module.instance_cloud_1.data.hpegl_vmaas_environment.env: Reading... module.instance_cloud_1.data.hpegl_vmaas_network.blue_segment: Reading... module.instance_cloud_2.data.hpegl_vmaas_plan.g1_large: Reading... module.instance_cloud_1.data.hpegl_vmaas_plan.g1_large: Reading... module.instance_cloud_2.data.hpegl_vmaas_environment.env: Reading... module.instance_cloud_2.data.hpegl_vmaas_network.blue_segment: Reading... module.instance_cloud_1.data.hpegl_vmaas_network.blue_segment: Read complete after 2s [id=22] module.instance_cloud_1.data.hpegl_vmaas_environment.env: Read complete after 2s [id=1] module.instance_cloud_2.data.hpegl_vmaas_network.blue_segment: Read complete after 3s [id=41] module.instance_cloud_2.data.hpegl_vmaas_plan.g1_large: Read complete after 3s [id=877] module.instance_cloud_1.data.hpegl_vmaas_plan.g1_large: Read complete after 3s [id=877] module.instance_cloud_2.data.hpegl_vmaas_environment.env: Read complete after 3s [id=1] data.hpegl_vmaas_morpheus_details.morpheus_details: Read complete after 5s [id=4c4fcdd1-e283-4d9e-8a43-d050a3d201f9] module.morpheus_artefacts_1.data.morpheus_cloud.cloud: Reading... module.morpheus_artefacts_1.data.morpheus_virtual_image.example_virtual_image: Reading... module.morpheus_artefacts_2.data.morpheus_cloud.cloud: Reading... module.morpheus_artefacts_2.data.morpheus_virtual_image.example_virtual_image: Reading... module.morpheus_artefacts_1.data.morpheus_cloud.cloud: Read complete after 5s [name=HPE GreenLake VMaaS Cloud] module.morpheus_artefacts_2.data.morpheus_cloud.cloud: Read complete after 5s [name=ST01] module.instance_cloud_1.data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... module.instance_cloud_1.data.hpegl_vmaas_datastore.c_3par: Reading... module.instance_cloud_1.data.hpegl_vmaas_cloud_folder.compute_folder: Reading... module.instance_cloud_2.data.hpegl_vmaas_cloud_folder.compute_folder: Reading... module.instance_cloud_2.data.hpegl_vmaas_datastore.c_3par: Reading... module.instance_cloud_2.data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... module.morpheus_artefacts_1.data.morpheus_virtual_image.example_virtual_image: Read complete after 5s [name=ubuntu20] module.morpheus_artefacts_2.data.morpheus_virtual_image.example_virtual_image: Read complete after 5s [name=ubuntu-20] module.instance_cloud_1.data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 2s [id=1] module.instance_cloud_1.data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 2s [id=4] module.instance_cloud_1.data.hpegl_vmaas_datastore.c_3par: Read complete after 2s [id=4] module.instance_cloud_2.data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 2s [id=8] module.instance_cloud_2.data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 2s [id=11] module.instance_cloud_2.data.hpegl_vmaas_datastore.c_3par: Read complete after 2s [id=11] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 1 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = (known after apply) + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 877 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v1009" + no_agent = false + resource_pool_id = 1 } + network { + id = 22 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "4" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # module.instance_cloud_1.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "1" } + max = 50000 + min = 1 + result = (known after apply) } # module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 5 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = (known after apply) + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 877 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v68" + no_agent = false + resource_pool_id = 8 } + network { + id = 41 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "11" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # module.instance_cloud_2.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "5" } + max = 50000 + min = 1 + result = (known after apply) } # module.morpheus_artefacts_1.morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 1, ] + code = (known after apply) + id = (known after apply) + location = "galway" + name = (known after apply) } # module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = (known after apply) + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = (known after apply) + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # module.morpheus_artefacts_1.morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + script_template_ids = (known after apply) + short_name = "tfexamplenodetype-ubuntu20" + technology = "vmware" + version = "2.0" + virtual_image_id = 3065 } # module.morpheus_artefacts_1.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "HPE GreenLake VMaaS Cloud" } + max = 50000 + min = 1 + result = (known after apply) } # module.morpheus_artefacts_2.morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 5, ] + code = (known after apply) + id = (known after apply) + location = "galway" + name = (known after apply) } # module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = (known after apply) + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = (known after apply) + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # module.morpheus_artefacts_2.morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + script_template_ids = (known after apply) + short_name = "tfexamplenodetype-ubuntu-20" + technology = "vmware" + version = "2.0" + virtual_image_id = 3056 } # module.morpheus_artefacts_2.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "ST01" } + max = 50000 + min = 1 + result = (known after apply) } Plan: 14 to add, 0 to change, 0 to destroy. Changes to Outputs: + instance_ids_cloud_1 = (known after apply) + instance_ids_cloud_2 = (known after apply) + instance_layout_id_cloud_1 = (known after apply) + instance_layout_id_cloud_2 = (known after apply) + instance_layout_name_cloud_1 = (known after apply) + instance_layout_name_cloud_2 = (known after apply) + instance_names_cloud_1 = (known after apply) + instance_names_cloud_2 = (known after apply) + node_type_id_cloud_1 = (known after apply) + node_type_id_cloud_2 = (known after apply) + node_type_name_cloud_1 = (known after apply) + node_type_name_cloud_2 = (known after apply) module.morpheus_artefacts_2.random_integer.random: Creating... module.morpheus_artefacts_1.random_integer.random: Creating... module.morpheus_artefacts_2.random_integer.random: Creation complete after 0s [id=29896] module.morpheus_artefacts_1.random_integer.random: Creation complete after 0s [id=17891] module.morpheus_artefacts_1.morpheus_group.tf_example_group: Creating... module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type: Creating... module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type: Creating... module.morpheus_artefacts_1.morpheus_node_type.tf_example_node: Creating... module.instance_cloud_1.random_integer.random: Creating... module.instance_cloud_2.random_integer.random: Creating... module.morpheus_artefacts_2.morpheus_group.tf_example_group: Creating... module.morpheus_artefacts_2.morpheus_node_type.tf_example_node: Creating... module.instance_cloud_1.random_integer.random: Creation complete after 0s [id=39454] module.instance_cloud_2.random_integer.random: Creation complete after 0s [id=48337] module.morpheus_artefacts_1.morpheus_node_type.tf_example_node: Creation complete after 5s [id=5104] module.morpheus_artefacts_2.morpheus_node_type.tf_example_node: Creation complete after 5s [id=5107] module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type: Creation complete after 5s [id=328] module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout: Creating... module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type: Creation complete after 6s [id=325] module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout: Creating... module.morpheus_artefacts_2.morpheus_group.tf_example_group: Creation complete after 7s [id=88] module.morpheus_artefacts_1.morpheus_group.tf_example_group: Creation complete after 8s [id=85] module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout: Creation complete after 6s [id=3556] module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout: Creation complete after 5s [id=3553] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Creating... module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Creating... module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Creation complete after 4m18s [id=628] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Creation complete after 4m35s [id=631] Apply complete! Resources: 14 added, 0 changed, 0 destroyed. Outputs: instance_ids_cloud_1 = "628" instance_ids_cloud_2 = "631" instance_layout_id_cloud_1 = "3556" instance_layout_id_cloud_2 = "3553" instance_layout_name_cloud_1 = "cfe_todo_app_frontend--17891" instance_layout_name_cloud_2 = "cfe_todo_app_frontend--29896" instance_names_cloud_1 = "CFE morpheus-vmaas sample VM 1-39454-0" instance_names_cloud_2 = "CFE morpheus-vmaas sample VM 1-48337-0" node_type_id_cloud_1 = "5104" node_type_id_cloud_2 = "5107" node_type_name_cloud_1 = "cfe_tf_example_node_type--17891" node_type_name_cloud_2 = "cfe_tf_example_node_type--29896" </code></pre> <h3>The output from an OpenTofu (v1.6.2) run to create the two VMaaS instances</h3> <pre><code class="language-shellsession">➜ 2_clouds_module_nest git:(blog1) ✗ tofu apply -var-file variables_tenant.tfvars -auto-approve module.instance_cloud_1.data.hpegl_vmaas_environment.env: Reading... module.instance_cloud_2.data.hpegl_vmaas_environment.env: Reading... module.instance_cloud_1.data.hpegl_vmaas_network.blue_segment: Reading... module.instance_cloud_2.data.hpegl_vmaas_plan.g1_large: Reading... module.instance_cloud_1.data.hpegl_vmaas_plan.g1_large: Reading... module.instance_cloud_2.data.hpegl_vmaas_network.blue_segment: Reading... data.hpegl_vmaas_morpheus_details.morpheus_details: Reading... module.instance_cloud_2.data.hpegl_vmaas_environment.env: Read complete after 2s [id=1] module.instance_cloud_2.data.hpegl_vmaas_network.blue_segment: Read complete after 2s [id=41] module.instance_cloud_1.data.hpegl_vmaas_environment.env: Read complete after 3s [id=1] module.instance_cloud_1.data.hpegl_vmaas_network.blue_segment: Read complete after 3s [id=22] module.instance_cloud_1.data.hpegl_vmaas_plan.g1_large: Read complete after 3s [id=877] module.instance_cloud_2.data.hpegl_vmaas_plan.g1_large: Read complete after 3s [id=877] data.hpegl_vmaas_morpheus_details.morpheus_details: Read complete after 5s [id=4c4fcdd1-e283-4d9e-8a43-d050a3d201f9] module.morpheus_artefacts_2.data.morpheus_cloud.cloud: Reading... module.morpheus_artefacts_2.data.morpheus_virtual_image.example_virtual_image: Reading... module.morpheus_artefacts_1.data.morpheus_cloud.cloud: Reading... module.morpheus_artefacts_1.data.morpheus_virtual_image.example_virtual_image: Reading... module.morpheus_artefacts_2.data.morpheus_cloud.cloud: Read complete after 5s [name=ST01] module.instance_cloud_2.data.hpegl_vmaas_cloud_folder.compute_folder: Reading... module.instance_cloud_2.data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... module.instance_cloud_2.data.hpegl_vmaas_datastore.c_3par: Reading... module.morpheus_artefacts_2.data.morpheus_virtual_image.example_virtual_image: Read complete after 5s [name=ubuntu-20] module.morpheus_artefacts_1.data.morpheus_virtual_image.example_virtual_image: Read complete after 5s [name=ubuntu20] module.morpheus_artefacts_1.data.morpheus_cloud.cloud: Read complete after 5s [name=HPE GreenLake VMaaS Cloud] module.instance_cloud_1.data.hpegl_vmaas_cloud_folder.compute_folder: Reading... module.instance_cloud_1.data.hpegl_vmaas_resource_pool.cl_resource_pool: Reading... module.instance_cloud_1.data.hpegl_vmaas_datastore.c_3par: Reading... module.instance_cloud_2.data.hpegl_vmaas_datastore.c_3par: Read complete after 2s [id=11] module.instance_cloud_2.data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 2s [id=11] module.instance_cloud_2.data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 3s [id=8] module.instance_cloud_1.data.hpegl_vmaas_resource_pool.cl_resource_pool: Read complete after 3s [id=1] module.instance_cloud_1.data.hpegl_vmaas_cloud_folder.compute_folder: Read complete after 3s [id=4] module.instance_cloud_1.data.hpegl_vmaas_datastore.c_3par: Read complete after 3s [id=4] OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create OpenTofu will perform the following actions: # module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 1 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = (known after apply) + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 877 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v1009" + no_agent = false + resource_pool_id = 1 } + network { + id = 22 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "4" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # module.instance_cloud_1.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "1" } + max = 50000 + min = 1 + result = (known after apply) } # module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0] will be created + resource "hpegl_vmaas_instance" "sample_vm" { + cloud_id = 5 + containers = (known after apply) + environment_code = "dev" + group_id = (known after apply) + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = (known after apply) + labels = [ + "sample_vm", ] + layout_id = (known after apply) + name = (known after apply) + plan_id = 877 + server_id = (known after apply) + status = (known after apply) + tags = { + "APPLICATION" = "Custom Ubuntu" + "BillingCostCenter" = "999" + "Division" = "AUK" + "PatchGroup" = "None" + "ResourceContact" = "john.lenihan@hpe.com" + "ResourcePurpose" = "CFE" + "purpose" = "sample" } + config { + asset_tag = "vm_tag" + create_user = true + folder_code = "group-v68" + no_agent = false + resource_pool_id = 8 } + network { + id = 41 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "11" + id = (known after apply) + name = "root_vol" + root = true + size = 30 } } # module.instance_cloud_2.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "5" } + max = 50000 + min = 1 + result = (known after apply) } # module.morpheus_artefacts_1.morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 1, ] + code = (known after apply) + id = (known after apply) + location = "galway" + name = (known after apply) } # module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = (known after apply) + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = (known after apply) + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # module.morpheus_artefacts_1.morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + script_template_ids = (known after apply) + short_name = "tfexamplenodetype-ubuntu20" + technology = "vmware" + version = "2.0" + virtual_image_id = 3065 } # module.morpheus_artefacts_1.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "HPE GreenLake VMaaS Cloud" } + max = 50000 + min = 1 + result = (known after apply) } # module.morpheus_artefacts_2.morpheus_group.tf_example_group will be created + resource "morpheus_group" "tf_example_group" { + cloud_ids = [ + 5, ] + code = (known after apply) + id = (known after apply) + location = "galway" + name = (known after apply) } # module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout will be created + resource "morpheus_instance_layout" "tf_example_instance_layout" { + creatable = true + description = (known after apply) + id = (known after apply) + instance_type_id = (known after apply) + labels = [ + "demo", + "layout", + "terraform", ] + minimum_memory = (known after apply) + name = (known after apply) + node_type_ids = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + spec_template_ids = (known after apply) + support_convert_to_managed = (known after apply) + technology = "vmware" + version = "1.0" + workflow_id = (known after apply) } # module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type will be created + resource "morpheus_instance_type" "tf_example_instance_type" { + category = "web" + code = (known after apply) + description = "Terraform Example Instance Type" + enable_deployments = (known after apply) + enable_scaling = (known after apply) + enable_settings = (known after apply) + environment_prefix = (known after apply) + featured = true + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + option_type_ids = (known after apply) + price_set_ids = (known after apply) + visibility = "public" } # module.morpheus_artefacts_2.morpheus_node_type.tf_example_node will be created + resource "morpheus_node_type" "tf_example_node" { + category = "tfexample" + file_template_ids = (known after apply) + id = (known after apply) + labels = [ + "demo", + "instance", + "terraform", ] + name = (known after apply) + script_template_ids = (known after apply) + short_name = "tfexamplenodetype-ubuntu-20" + technology = "vmware" + version = "2.0" + virtual_image_id = 3056 } # module.morpheus_artefacts_2.random_integer.random will be created + resource "random_integer" "random" { + id = (known after apply) + keepers = { + "cloud" = "ST01" } + max = 50000 + min = 1 + result = (known after apply) } Plan: 14 to add, 0 to change, 0 to destroy. Changes to Outputs: + instance_ids_cloud_1 = (known after apply) + instance_ids_cloud_2 = (known after apply) + instance_layout_id_cloud_1 = (known after apply) + instance_layout_id_cloud_2 = (known after apply) + instance_layout_name_cloud_1 = (known after apply) + instance_layout_name_cloud_2 = (known after apply) + instance_names_cloud_1 = (known after apply) + instance_names_cloud_2 = (known after apply) + node_type_id_cloud_1 = (known after apply) + node_type_id_cloud_2 = (known after apply) + node_type_name_cloud_1 = (known after apply) + node_type_name_cloud_2 = (known after apply) module.morpheus_artefacts_2.random_integer.random: Creating... module.instance_cloud_2.random_integer.random: Creating... module.instance_cloud_1.random_integer.random: Creating... module.morpheus_artefacts_1.random_integer.random: Creating... module.instance_cloud_2.random_integer.random: Creation complete after 0s [id=10027] module.morpheus_artefacts_1.random_integer.random: Creation complete after 0s [id=40272] module.morpheus_artefacts_2.random_integer.random: Creation complete after 0s [id=1661] module.instance_cloud_1.random_integer.random: Creation complete after 0s [id=36444] module.morpheus_artefacts_2.morpheus_group.tf_example_group: Creating... module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type: Creating... module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type: Creating... module.morpheus_artefacts_1.morpheus_group.tf_example_group: Creating... module.morpheus_artefacts_2.morpheus_node_type.tf_example_node: Creating... module.morpheus_artefacts_1.morpheus_node_type.tf_example_node: Creating... module.morpheus_artefacts_2.morpheus_instance_type.tf_example_instance_type: Creation complete after 5s [id=334] module.morpheus_artefacts_1.morpheus_node_type.tf_example_node: Creation complete after 5s [id=5113] module.morpheus_artefacts_2.morpheus_node_type.tf_example_node: Creation complete after 5s [id=5110] module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout: Creating... module.morpheus_artefacts_1.morpheus_instance_type.tf_example_instance_type: Creation complete after 5s [id=331] module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout: Creating... module.morpheus_artefacts_1.morpheus_group.tf_example_group: Creation complete after 7s [id=91] module.morpheus_artefacts_2.morpheus_group.tf_example_group: Creation complete after 9s [id=94] module.morpheus_artefacts_2.morpheus_instance_layout.tf_example_instance_layout: Creation complete after 5s [id=3559] module.morpheus_artefacts_1.morpheus_instance_layout.tf_example_instance_layout: Creation complete after 5s [id=3562] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Creating... module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Creating... module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [1m50s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m30s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [2m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m0s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m10s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m20s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m30s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m40s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m40s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [3m50s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m0s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m10s elapsed] module.instance_cloud_1.hpegl_vmaas_instance.sample_vm[0]: Creation complete after 4m20s [id=637] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Still creating... [4m20s elapsed] module.instance_cloud_2.hpegl_vmaas_instance.sample_vm[0]: Creation complete after 4m21s [id=634] Apply complete! Resources: 14 added, 0 changed, 0 destroyed. Outputs: instance_ids_cloud_1 = "637" instance_ids_cloud_2 = "634" instance_layout_id_cloud_1 = "3562" instance_layout_id_cloud_2 = "3559" instance_layout_name_cloud_1 = "cfe_todo_app_frontend--40272" instance_layout_name_cloud_2 = "cfe_todo_app_frontend--1661" instance_names_cloud_1 = "CFE morpheus-vmaas sample VM 1-36444-0" instance_names_cloud_2 = "CFE morpheus-vmaas sample VM 1-10027-0" node_type_id_cloud_1 = "5113" node_type_id_cloud_2 = "5110" node_type_name_cloud_1 = "cfe_tf_example_node_type--40272" node_type_name_cloud_2 = "cfe_tf_example_node_type--1661" </code></pre> <h1>Summary </h1> <p>When using Terraform (or OpenTofu) to interact with HPE GreenLake for Private Cloud Enterprise VMaaS there are two complementary providers: </p> <ul> <li><a href="https://registry.terraform.io/providers/HPE/hpegl/latest">hpegl</a></li> <li><a href="https://registry.terraform.io/providers/gomorpheus/morpheus/latest">Morpheus</a></li> </ul> <p>When used in concert, these two providers enable a rich IaC functionality. The hpegl provider has a <em>data source</em> named <em>hpegl_vmaas_morpheus_details</em>, which can be used to retrieve an access token and URL for the on-premise Morpheus instance associated with the specific VMaaS <em>location</em>. These can be used as inputs to the provider stanza for the Morpheus provider. </p> <p>We have included example HCL for two separate VMaaS configurations: </p> <ul> <li>One location, one VMaaS cloud with one VMaaS instance </li> <li>One location, two VMaaS clouds each with one VMaaS instance </li> </ul> <p>The HCL for the second configuration is built around a pair of HCL modules. These modules can be combined in various ways to work with a number of different VMaaS configurations, including configurations with more than one on-premise Morpheus instance. These modules are:</p> <ul> <li><a href="https://github.com/eamonnotoole/GLP-API-Tooling/tree/main/Terraform/HPEGL-Morpheus-PCE-VMaaS/One-Location/Two-Clouds-Two-Instances/morpheus_artefacts"><em>morpheus_artefacts</em></a> uses the Morpheus provider to create a group, instance type, instance layout, and node type in the target VMaaS cloud.</li> <li><a href="https://github.com/eamonnotoole/GLP-API-Tooling/tree/main/Terraform/HPEGL-Morpheus-PCE-VMaaS/One-Location/Two-Clouds-Two-Instances/vmaas_instance"><em>vmaas_instance</em></a> takes inputs from morpheus_artefacts and uses the hpegl provider to retrieve details about VMaaS resources and creates and launches a VMaaS VM Instance.</li> </ul> <p>HPE GreenLake for Private Cloud Enterprise customers are encouraged to use these example HCL bundles as a starting point for their own investigations into using hpegl and Morpheus for IaC with their environments.</p>https://developer.hpe.com/why-private-ai/https://developer.hpe.com/why-private-ai/Fri, 10 Jan 2025 13:21:41 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Everybody’s getting into AI. The visceral experience of using ChatGPT in professional or personal lives can raise a lot of concern. Concerns about the future of work, creativity... even civilization's lifespan.</p> <p>One big concern that often comes up is about <strong>control</strong>. How can an organization or individual maintain a certain amount of control over their experience when using AI?</p> <p>Not everyone cares about this. But for those who do, it’s critical.</p> <p>I think there are three major reasons why organizations want control over their AI-powered applications, and I’ll discuss them in this post: <strong>Data</strong>, <strong>Model Performance</strong>, and <strong>Cost</strong>.</p> <p>The only way to address these concerns is to run a model in a private, protected environment - one in which you have full control.</p> <h2>1. Data</h2> <p>Having control over your data means what you think it does; that you don't let anyone else see it.</p> <h3>Privacy and security guarantees</h3> <p>When you run your own large language model (LLM) endpoint, all data is processed locally, on your network. This allows you to minimize the risk of exposure in two ways: when the data is in transit, and when the data is stored in the LLM endpoint’s logs.</p> <p>When you depend on a service that is hosted externally to your organization, there is always a form of <a href="https://www.investopedia.com/terms/c/counterpartyrisk.asp">counterparty risk</a>. Public services can fall victim to scalability issues, power outages, ransomware attacks, or other Force Majeure. Also, counterparties can choose to update or change models without telling you. And you can't forget the cost of API calls.</p> <p>Processing data locally or in controlled environments minimizes these risks. Not because you’re any better at cybersecurity or running a datacenter than these counterparties… just because you’re <strong>already exposed</strong> to issues on your side. Why increase the surface area? Why trust someone with your tokens if you don’t have to?</p> <p>In addition, if you are using a public LLM endpoint that provides a streaming response, even encrypted data in transmit makes it possible to recover the plain text from the encrypted network traffic. More detail is available in <a href="https://cdn.arstechnica.net/wp-content/uploads/2024/03/LLM-Side-Channel.pdf">this paper</a>, which I hope public endpoint providers have seen and addressed.</p> <p><img src="https://jordannanos.github.io/images/2024-11-15-streaming-response.png" alt=""></p> <h3>Compliance with regulations</h3> <p>Beyond reputational risk, regulations like HIPAA and PCI force counterparties in healthcare and finance to store data for years, while the EU’s GDPR and India’s DPDPA force PII to be stored within their jurisdiction (at least, that’s how companies are interpreting the laws). Ensuring compliance with these regulations is important, but it’s also important in conjunction with the security risks described earlier. Why rely on a third-party who is forced to store every prompt and response for six years?</p> <h3>IP and copyright protection</h3> <p>The final example of losing control of your data is when some engineer goes and uploads a document describing trade secrets, debugs some code with a proprietary algorithm inside, or signs a waiver to download a confidential document, forgets about the waiver, and then uploads it to get a quick summary. Looking at you <a href="https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak">Samsung</a>…</p> <h2>2. Model Performance</h2> <p>I referred to this <a href="https://developer.hpe.com/blog/how-to-pick-a-large-language-model-for-private-ai/">in a previous blog</a>: there are two ways to measure model performance: the <strong>speed</strong> of the output, and the <strong>quality</strong> of the output.</p> <h3>Improving Model Speed</h3> <p>Primarily, there are two key metrics that impact user experience: latency and throughput. Latency is generally considered to be “time to first token” or TTFT. This is constrained by how fast the model can process the input (i.e. the prompt) measured in tokens-per-second (tok/sec). Throughput is generally considered to be “time per output token” or TPOT. Throughput can also be measured by inter-token latency and generally, it is represented in tokens-per-second (tok/sec).</p> <blockquote> <p><strong>Note:</strong> Tokens are the basic units of input and output in a large language model. Tokens typically represent words, sub-words, or characters. They are the smallest units of meaning in a text that can be processed by a large language model.</p> </blockquote> <p>When considering a minimum tok/sec of performance for language models, most people jump to a comparison that includes reading speed. The reasoning goes something like: proficient readers are around 300 words per minute. Considering the vocabulary size of current LLM tokenizers to be 1.5 tokens per word, that is 450 tokens per minute, or 7.5 tokens per second.</p> <p>However, a few comments:</p> <ul> <li><strong>5 tok/sec feels slow to me</strong>. 10 tok/sec feels good. 20+ tok/sec feels fast.</li> <li><strong>Many LLM’s are used for things beyond just chatting back and forth</strong>, where the user doesn’t read every line that was generated line by line. For example, when I am editing a blog or generating an image, I am just waiting for the LLM to finish generating before I ctrl+C, ctrl+V. Or, when generating, refactoring, and adding comments to code, I am going to immediately insert it and move on.</li> <li><strong>Many “online” or “interactive” LLM-powered applications contain multiple prompts and responses</strong>. For example, RAG involves an embedding model and a chat model. Text-to-SQL involves a prompt/response to generate the query, running the query, and an optional prompt/response for synthesis of the response. Tool use, agentic workflows, reflection, and chain-of-thought prompting is all growing in popularity. These approaches require multiple prompt/response turns before. Anything that involves audio (text-to-speech or speech-to-text) has a minimum speed requirement (people can speak English at around 150 words per minute, about half as fast as they read).</li> <li><strong>Many “offline” or “batch” use cases also exist</strong> (i.e. summarize, classify, translate or transcribe 1000 files)</li> </ul> <p>In just about every one of these cases, faster is better (or cheaper). Having full control over the hardware that your model runs on is really the only way to improve this.</p> <h3>Improving Model Quality</h3> <p>Three ways to improve model quality is via prompt-tuning (or p-tuning), retrieval augmented-generation (RAG), or fine-tuning the model (i.e. changing some of the weights).</p> <p>In all of these cases, it is clear that the developer of the LLM-powered application would like to control things like the system prompt (kind of available via ChatGPT or Claude), the weights of the model (not really available for public endpoints), and most importantly, use <strong>private data</strong> during RAG.</p> <p>As discussed earlier, the quality of your AI-powered application depends on the quality of your data. If you want to integrate private datasets without taking on risk, you have to host the endpoint(s) for yourself.</p> <h2>3. Cost</h2> <p>The biggest thing that bothers me about using public APIs, is paying per token. It seems like cloud has gone too far. First, it was CPU cycles by the hour. Then, it was functions as-a-Service. Now, if the model is too chatty, I’m getting hit with a bill.</p> <p>This isn’t fantasy: many use cases for large context windows are popping up. The first Harry Potter book costs around 100k tokens, and so do a lot of my product spec sheets and user guides. Anthropic prices Claude 3.5 Sonnet at $3/M tokens. OpenAI charges $5/M tokens for GPT-4o. Google offers Gemini 1.5 Pro for $1.25/M tokens.</p> <p>So, (over simplifying and avoiding the pricing improvements of context caching), if I have 10 questions about the first Harry Potter book, it’s going to cost me between $1 and $5. And, I have a few more questions than that.</p> <h2>Conclusion: The world will be hybrid</h2> <p>It seems to me that Private AI will experience a similar evolution as found with Private Cloud. Public experiences will win hearts, minds, and popular culture. However, many companies will feel the need to try and get the same experience in private due to specific requirements for data privacy, model performance, and cost. The result will be a mix of both, resulting in a hybrid AI experience.</p> <p>Hunter and I discussed this in Episode 3 of <em>Things We Read This Week</em>. You can watch it <a href="https://www.youtube.com/watch?v=Byjlr0xplNI">here on YouTube</a>.</p>https://developer.hpe.com/llm-agentic-tool-mesh-empowering-gen-ai-with-retrieval-augmented-generation-rag/https://developer.hpe.com/llm-agentic-tool-mesh-empowering-gen-ai-with-retrieval-augmented-generation-rag/Wed, 08 Jan 2025 11:46:13 GMT<style> li { font-size: 27px !important; line-height: 33px !important; max-width: none !important; } </style> <p>In our previous blog posts, we explored the <a href="https://developer.hpe.com/blog/ll-mesh-exploring-chat-service-and-factory-design-pattern/">Chat Service</a> and the <a href="https://developer.hpe.com/blog/llm-agentic-tool-mesh-harnessing-agent-services-and-multi-agent-ai-for-next-level-gen-ai/">Agents Service</a> of <a href="https://developer.hpe.com/blog/ll-mesh-democratizing-gen-ai-through-open-source-innovation-1/">LLM Agentic Tool Mesh</a>, highlighting how they simplify the integration of Generative AI (Gen AI) into applications.</p> <p>Today, we'll dive into another pivotal feature of LLM Agentic Tool Mesh: <strong>Retrieval-Augmented Generation (RAG) Service</strong>. We'll explain what RAG is, how LLM Agentic Tool Mesh handles it, delve into the RAG services, and showcase an example of an agentic tool using RAG.</p> <h1>Understanding Retrieval-Augmented Generation (RAG)</h1> <p>RAG is a technique that enhances the capabilities of language models by providing them with access to external knowledge sources. Instead of relying solely on the information contained within the model's parameters, RAG allows models to retrieve and utilize relevant data from external documents or databases.</p> <p>This approach improves the accuracy and relevance of generated responses, especially in domains requiring up-to-date or specialized information. The key benefits of RAG are:</p> <ul> <li><strong>Enhanced accuracy</strong>: Provides more precise and factual responses by accessing more data available externally</li> <li><strong>Domain specialization</strong>: Enables models to handle specialized topics by leveraging domain-specific documents</li> <li><strong>Reduced hallucinations</strong>: Minimizes the generation of incorrect or nonsensical information by grounding responses in real data</li> </ul> <h1>RAG in LLM Agentic Tool Mesh</h1> <p>In the LLM Agentic Tool Mesh platform, RAG is a <strong>crucial tool</strong> that <strong>can enhance</strong> <strong>a language model's</strong> capabilities by integrating external knowledge. LLM Agentic Tool implements RAG through two main stages:</p> <ul> <li><strong>Injection</strong></li> <li><strong>Retrieval</strong></li> </ul> <p>Each stage is designed to standardize and optimize data use, ensuring generated content is both relevant and accurate.</p> <h2>The injection process</h2> <p>The injection process involves preparing and integrating data into a storage system where it can be efficiently retrieved when content is being generated.</p> <p>This process is abstracted into three key steps:</p> <ol> <li><strong>Extraction</strong></li> <li><strong>Transformation</strong></li> <li><strong>Loading</strong></li> </ol> <p><img src="/img/ingestion.png" alt=""></p> <h3>Extraction</h3> <p>As part of the extraction phase, data gathering involves collecting information from various sources, such as DOCX, PDF, or other formats, and converting it into a common format, typically JSON, to ensure consistency.</p> <p>Example usage</p> <pre><code class="language-python"># Configuration for the Data Extractor EXTRACTOR_CONFIG = { 'type': 'UnstructuredSections', 'document_type': 'Pdf', 'cache_elements_to_file': True, 'extract_text': True, 'exclude_header': True, 'exclude_footer': True, 'extract_image': False, 'image_output_folder': './images' } # Initialize the Data Extractor data_extractor = DataExtractor.create(EXTRACTOR_CONFIG) # Parse a document file file_path = 'example_document.pdf' result = data_extractor.parse(file_path) # Handle the extraction result if result.status == "success": print(f"EXTRACTED ELEMENTS:\n{result.elements}") else: print(f"ERROR:\n{result.error_message}") </code></pre> <h3>Transformation</h3> <p>As part of the transformation phase, the process involves cleaning the data by removing irrelevant or redundant information, enriching it with metadata to enhance searchability during retrieval, and transforming the cleaned data using LLMs to generate summaries, question-and-answer pairs, or other structured outputs.</p> <p>Example usage</p> <pre><code class="language-python">from athon.rag import DataTransformer # Configuration for the Data Transformer TRANSFORMER_CONFIG = { 'type': 'CteActionRunner', 'clean': { 'headers_to_remove': ['Confidential', 'Draft'], 'min_section_length': 100 }, 'transform': { 'llm_config': { 'type': 'LangChainChatOpenAI', 'api_key': 'your-api-key-here', 'model_name': 'gpt-4o' }, 'system_prompt': 'Summarize the following content.', 'transform_delimeters': ['```', '```json'] }, 'enrich': { 'metadata': { 'source': 'LLM Agentic Tool Mesh Platform', 'processed_by': 'CteActionRunner' } } } # Initialize the Data Transformer data_transformer = DataTransformer.create(TRANSFORMER_CONFIG) # List of extracted elements to be transformed extracted_elements = [ {"text": "Confidential Report on AI Development", "metadata": {"type": "Header"}}, {"text": "AI is transforming industries worldwide...", "metadata": {"type": "Paragraph"}} ] # Define the actions to be performed actions = ['RemoveSectionsByHeader', 'TransformInSummary', 'EnrichMetadata'] # Process the elements result = data_transformer.process(actions, extracted_elements) # Handle the transformation result if result.status == "success": print(f"TRANSFORMED ELEMENTS:\n{result.elements}") else: print(f"ERROR:\n{result.error_message}") </code></pre> <h3>Loading</h3> <p>As part of the loading phase, the process includes injecting the transformed data into the chosen storage solution, such as a vector database, and further adapting the data as needed, such as chunking it into smaller pieces for efficient retrieval.</p> <p>Example usage</p> <pre><code class="language-python">from athon.rag import DataLoader # Configuration for the Data Loader LOADER_CONFIG = { 'type': 'ChromaForSentences' } # Initialize the Data Loader data_loader = DataLoader.create(LOADER_CONFIG) # Example collection (retrieved from a DataStorage instance) collection = data_storage.get_collection().collection # List of elements to be inserted elements = [ {"text": "Generative AI is transforming industries.", "metadata": {"category": "AI", "importance": "high"}}, {"text": "This document discusses the impact of AI.", "metadata": {"category": "AI", "importance": "medium"}} ] # Insert the elements into the collection result = data_loader.insert(collection, elements) # Handle the insertion result if result.status == "success": print("Data successfully inserted into the collection.") else: print(f"ERROR:\n{result.error_message}") </code></pre> <h2>The retrieval process</h2> <p>Once the data has been injected and is ready for use, the retrieval process focuses on fetching the most relevant information based on a given input query. This ensures that the language model has access to the right data to generate accurate and contextually relevant outputs.</p> <ol> <li><strong>Data retrieval</strong>: Uses various methods, such as dense or sparse retrieval, to fetch the most relevant data from storage</li> <li><strong>Metadata filtering</strong>: Applies metadata filters to narrow down search results, ensuring the retrieved data matches the specific needs of the query</li> <li><strong>Chunk Expansion</strong>: Expands the retrieved data chunks to provide comprehensive information for the language model</li> </ol> <p><img src="/img/retrieve.png" alt=""></p> <p>Example usage</p> <pre><code class="language-python">from athon.rag import DataRetriever # Configuration for the Data Retriever RETRIEVER_CONFIG = { 'type': 'ChromaForSentences', 'expansion_type': 'Section', 'sentence_window': 3, 'n_results': 10, 'include': ['documents', 'metadatas'] } # Initialize the Data Retriever data_retriever = DataRetriever.create(RETRIEVER_CONFIG) # Example collection (retrieved from a DataStorage instance) collection = data_storage.get_collection().collection # Query to search within the collection query = "What is the impact of Generative AI on industries?" # Retrieve relevant data based on the query result = data_retriever.select(collection, query) # Handle the retrieval result if result.status == "success": for element in result.elements: print(f"TEXT:\n{element['text']}\nMETADATA:\n{element['metadata']}\n") else: print(f"ERROR:\n{result.error_message}") </code></pre> <h1>LLM Agentic Tool Mesh in action: Agentic tool using RAG</h1> <p>In the <a href="https://github.com/HewlettPackard/llmesh">LLM Agentic Tool Mesh GitHub</a>, there is an example of a RAG-based tool that provides quick and accurate access to 5G specifications: the <strong>telco expert</strong> (inside folder <code>examples/tool_rag</code>).</p> <p>This agentic tool leverages the RAG services in LLM Agentic Tool Mesh to read telco standards, build or use a vector store from them, and then uses a query engine to find and return relevant information based on user queries.</p> <p>For enhanced observability, the telco expert not only provides the answer but also displays the retrieved chunks used to formulate the response. This includes both the text of the chunks and their associated metadata, such as the document source, date, and other relevant details. This feature allows users to verify the origin of the information and gain deeper insights into the data supporting the answer.</p> <p>Tool code snippet</p> <pre><code class="language-python">@AthonTool(config, logger) def telco_expert(query: str) -> str: """ This function reads the telco standards, builds or uses a vector store from them, and then uses a query engine to find and return relevant information to the input question. """ collection = _get_collection() if LOAD: _load_files_into_db(collection) augment_query = _augment_query_generated(query) rag_results = _retrieve_from_collection(collection, augment_query) ordered_rag_results = _rerank_answers(augment_query, rag_results) summary_answer = _summary_answer(augment_query, ordered_rag_results) chunk_answer = _create_chunk_string(ordered_rag_results) return summary_answer + "\n\n" + chunk_answer </code></pre> <p>The functionalities shown are:</p> <ul> <li><strong>Data injection</strong>: Loads telco standards into a vector store</li> <li><strong>Query augmentation</strong>: Enhances the user's query for better retrieval</li> <li><strong>Data retrieval</strong>: Retrieves relevant chunks from the vector store</li> <li><strong>Answer generation</strong>: Summarizes and formats the retrieved information to provide a comprehensive answer</li> </ul> <p><img src="/img/rag_tool.png" alt=""></p> <h1>Conclusion</h1> <p>The RAG Service in LLM Agentic Tool Mesh exemplifies how advanced design principles and innovative engineering simplify and enhance the adoption of Gen AI. By abstracting complexities and providing versatile examples, LLM Agentic Tool Mesh enables developers and users alike to unlock the transformative potential of Gen AI in various domains.</p> <p>Stay tuned for our next post, where we'll explore the System Service of LLM Agentic Tool Mesh, essential for creating and managing a mesh of tools, as we continue our journey to democratize Gen AI!</p>https://developer.hpe.com/2025-january-07/https://developer.hpe.com/2025-january-07/Tue, 07 Jan 2025 13:00:00 GMThttps://developer.hpe.com/sc24-from-the-chapel-language-perspective/https://developer.hpe.com/sc24-from-the-chapel-language-perspective/Wed, 18 Dec 2024 23:25:56 GMT<p>external blog</p>https://developer.hpe.com/announcing-chapel-2-3/https://developer.hpe.com/announcing-chapel-2-3/Thu, 12 Dec 2024 21:25:00 GMT<p>external blog</p>https://developer.hpe.com/llm-agentic-tool-mesh-harnessing-agent-services-and-multi-agent-ai-for-next-level-gen-ai/https://developer.hpe.com/llm-agentic-tool-mesh-harnessing-agent-services-and-multi-agent-ai-for-next-level-gen-ai/Thu, 12 Dec 2024 17:08:46 GMT<style> li { font-size: 27px !important; line-height: 33px !important; max-width: none !important; } </style> <p>In my previous blog post, I explored the <a href="https://developer.hpe.com/blog/ll-mesh-exploring-chat-service-and-factory-design-pattern/">Chat Service</a> of <a href="https://developer.hpe.com/blog/ll-mesh-democratizing-gen-ai-through-open-source-innovation-1/">LLM Agentic Tool Mesh</a>, an <a href="https://github.com/HewlettPackard/llmesh">open-source project</a> aimed at democratizing Generative AI (Gen AI).</p> <p>Today, I'll delve into another core feature: the <strong>Agent Service</strong>. I'll discuss what agents are, explain the LLM Agentic Tool Mesh related services, and showcase examples from its repository.</p> <h2>Understanding LLM agents</h2> <p>In the context of Large Language Models (LLMs), an agent is an autonomous entity capable of:</p> <ul> <li><strong>Perceiving its environment</strong>: Agents can gather and interpret information from their surroundings.</li> <li><strong>Making decisions</strong>: Based on the perceived information, agents decide on the best course of action.</li> <li><strong>Acting on decisions</strong>: Agents execute actions to achieve specific objectives.</li> </ul> <p>These agents can operate independently or interact with one another to optimize their collective performance depending on the complexity of the task. In fact, multi-agent AI involves coordinating multiple agents, each specialized in a specific domain or function, to collaborate and achieve a common goal. These agents handle:</p> <ul> <li><strong>Task division</strong>: Dividing complex tasks into manageable parts.</li> <li><strong>Specialization</strong>: Each agent specializes in a particular function, such as information retrieval or decision-making.</li> <li><strong>Collaboration</strong>: Agents communicate and share information for effective and efficient task execution.</li> </ul> <p><img src="/img/multiagents.png" alt=""></p> <p>Managing such agents typically requires advanced coding and deep knowledge of agent-based systems. However, LLM Agentic Tool Mesh simplifies this process by providing high-level abstractions through intuitive prompts and configuration files. Users can focus on defining tasks and desired outcomes while LLM Agentic Tool Mesh handles the coordination, task distribution, and result aggregation behind the scenes.</p> <h2>LLM Agentic Tool Mesh Agent Service</h2> <p>LLM Agentic Tool Mesh provides all the necessary tools to build a powerful agentic system by handling:</p> <ol> <li>The <strong>tool repository</strong></li> <li>The <strong>reasoning engine</strong></li> <li>A <strong>multi-agent task force</strong></li> </ol> <h3>Tool repository</h3> <p>Agents in LLM Agentic Tool Mesh rely on tools to perform specialized tasks like information retrieval, document summarization, or data analysis. These tools extend the agents' capabilities, allowing them to efficiently complete complex operations. The <strong>tool repository</strong> service in LLM Agentic Tool Mesh simplifies and automates the storage, management, and retrieval of these tools.</p> <p>Key features:</p> <ul> <li><strong>Dynamic tool storage</strong>: Add tools with associated metadata, including tool name, description, function, and usage parameters.</li> <li><strong>Tool retrieval</strong>: Flexible search and retrieval functionality, enabling agents to access tools based on specific criteria.</li> <li><strong>Metadata management</strong>: Store relevant metadata for each tool, aiding in decision-making for task assignments.</li> </ul> <p>Example usage:</p> <pre><code class="language-python">from athon.agents import ToolRepository # Configuration for the Tool Repository REPO_CONFIG = { 'type': 'LangChainStructured' } # Initialize the Tool Repository tool_repository = ToolRepository.create(REPO_CONFIG) Adding a tool to the repository: from langchain.tools import tool @tool def text_summarizer(text: str) -> str: """A simple text summarizer function""" return text[:50] metadata = { 'category': 'NLP', 'version': '1.0', 'author': 'John Doe' } # Add the tool to the repository add_result = tool_repository.add_tool(text_summarizer, metadata) if add_result.status == "success": print("Tool added successfully.") else: print(f"ERROR:\n{add_result.error_message}") # Retrieve tools with a metadata filter metadata_filter = {'category': 'NLP'} get_result = tool_repository.get_tools(metadata_filter) if get_result.status == "success": print(f"RETRIEVED TOOLS:\n{get_result.tools}") else: print(f"ERROR:\n{get_result.error_message}") </code></pre> <h3>Reasoning engine</h3> <p>The <strong>reasoning engine</strong> orchestrates interactions between the LLM and various tools, enabling agents to seamlessly combine decision-making capabilities with tool-based actions. It extends the chat capabilities by managing the dynamic integration of tools with the LLM, allowing for real-time decision-making and task execution.</p> <p>Key features:</p> <ul> <li><strong>Tool orchestration</strong>: Coordinates between the LLM and tools, deciding which tools to invoke based on context and user input.</li> <li><strong>Memory management</strong>: Handles storage and retrieval of relevant memory for ongoing tasks or conversations.</li> <li><strong>Dynamic configuration</strong>: Allows users to adjust the reasoning engine's behavior dynamically, tailoring interactions between LLMs and tools.</li> </ul> <p><img src="/img/reasoning.png" alt=""></p> <h3>Task force</h3> <p>The <strong>multi-agents task force</strong> service enables the orchestration of complex tasks through a network of specialized agents. This service allows users to define a structured workflow where each agent is assigned a specific task, executed in sequence or parallel.</p> <p>Key features:</p> <ul> <li><strong>LLM-driven planning</strong>: Integrates with an LLM to plan task sequences, ensuring intelligent coordination.</li> <li><strong>Agent specialization</strong>: Each agent specializes in a particular task, tailored through prompts defining their role, backstory, and goals.</li> <li><strong>Task-oriented workflow</strong>: Supports both sequential and parallel task execution, configurable through prompts and configuration files.</li> <li><strong>Tool integration</strong>: Agents utilize a suite of tools to complete their tasks, dynamically loaded and executed during task completion.</li> </ul> <p>Example usage:</p> <pre><code class="language-python">from athon.agents import TaskForce from custom_tools import DataFetcher, SalesSummarizer, PresentationBuilder # Example configuration for the Task Force Multi-Agents TASK_FORCE_CONFIG = { 'type': 'CrewAIMultiAgent', 'plan_type': 'Sequential', 'tasks': [ { 'description': 'Analyze the recent sales data.', 'expected_output': 'A summary report of sales trends.', 'agent': { 'role': 'Data Analyst', 'goal': 'Summarize sales data', 'backstory': 'Experienced in sales data analysis', 'tools': ['DataFetcher', 'SalesSummarizer'] } }, { 'description': 'Prepare a presentation based on the report.', 'expected_output': 'A presentation deck summarizing the sales report.', 'agent': { 'role': 'Presentation Specialist', 'goal': 'Create a presentation', 'backstory': 'Expert in creating engaging presentations', 'tools': ['PresentationBuilder'] } } ], 'llm': { 'type': 'LangChainChatOpenAI', 'api_key': 'your-api-key-here', 'model_name': 'gpt-4o-mini' }, 'verbose': True, 'memory': False } # Initialize the Task Force with the provided configuration task_force = TaskForce.create(TASK_FORCE_CONFIG) # Run the task force with an input message input_message = "Generate a sales analysis report and prepare a presentation." result = task_force.run(input_message) # Handle the response if result.status == "success": print(f"COMPLETION:\n{result.completion}") else: print(f"ERROR:\n{result.error_message}") </code></pre> <h2>LLM Agentic Tool Mesh in action: Examples from the repository</h2> <p>The LLM Agentic Tool Mesh GitHub repository includes several examples demonstrating the versatility and capabilities of the agent services.</p> <h3>Chatbot application</h3> <p>This chatbot (examples/app_chatbot) is capable of reasoning and invoking appropriate LLM tools to perform specific actions. You can configure the chatbot using files that define LLM Agentic Tool Mesh platform services, project settings, toolkits, and memory configurations. The web app orchestrates both local and remote LLM tools, allowing them to define their own HTML interfaces, supporting text, images, and code presentations.</p> <p>Configuration example:</p> <pre><code class="language-yaml">projects: - name: "Personal Chat" memory: type: LangChainBuffer memory_key: chat_history return_messages: true tools: - "https://127.0.0.1:5002/" # Basic Copywriter - name: "Project 5G Network" memory: type: LangChainRemote memory_key: chat_history return_messages: true base_url: "https://127.0.0.1:5010/" timeout: 100 cert_verify: false tools: - "https://127.0.0.1:5005/" # OpenAPI Manager - "https://127.0.0.1:5006/" # IMS Expert - name: "Project Meteo" memory: type: LangChainBuffer memory_key: chat_history return_messages: true tools: - "https://127.0.0.1:5003/" # Temperature Finder - "https://127.0.0.1:5004/" # Temperature Analyzer </code></pre> <h3>OpenAPI manager</h3> <p>The OpenAPI manager (examples/tool_agents) is a multi-agent tool that reads OpenAPI documentation and provides users with relevant information based on their queries. It uses the Task Force service to answer questions related to 5G APIs.</p> <p>Capabilities:</p> <ul> <li><strong>ListOpenApis</strong>: Lists all OpenAPI specifications present in the system.</li> <li><strong>SelectOpenApi</strong>: Selects a specific OpenAPI specification.</li> <li><strong>GetOpenApiVersion</strong>: Returns the OpenAPI version of the selected specification.</li> <li><strong>GetInfo</strong>: Returns the information dictionary of the selected specification.</li> <li><strong>GetMethodsByTag</strong>: Lists all methods of the selected specification for a specific tag.</li> <li><strong>GetMethodById</strong>: Returns detailed information about a method selected by ID.</li> <li><strong>GetRequestBody</strong>: Returns the request body schema of the selected specification.</li> <li><strong>GetResponse</strong>: Returns the response schema of the selected specification.</li> </ul> <p><img src="/img/mesh.png" alt=""></p> <h2>Conclusion</h2> <p>The <strong>LLM Agentic Tool Mesh Agent Service</strong> exemplifies how advanced design principles and innovative prompt engineering simplify and enhance the adoption of Gen AI. By abstracting complexities and providing versatile examples, LLM Agentic Tool Mesh enables developers and users alike to unlock the transformative potential of Gen AI in various domains.</p> <p>Stay tuned for our next post, where we'll explore another key service of LLM Agentic Tool Mesh and continue our journey to democratize Gen AI!</p>https://developer.hpe.com/how-to-transform-it-operations-with-ai-infused-full-stack-observability/https://developer.hpe.com/how-to-transform-it-operations-with-ai-infused-full-stack-observability/Mon, 02 Dec 2024 19:21:18 GMT<p>External blog post</p>https://developer.hpe.com/hpe-opsramp-continues-to-push-autonomous-it-operations-forward/https://developer.hpe.com/hpe-opsramp-continues-to-push-autonomous-it-operations-forward/Mon, 02 Dec 2024 19:14:19 GMT<p>External blog post</p>https://developer.hpe.com/the-schrödinger’s-cat-challenge-of-observing-cloud-native-applications/https://developer.hpe.com/the-schrödinger’s-cat-challenge-of-observing-cloud-native-applications/Mon, 02 Dec 2024 19:12:19 GMT<p>External blog post</p>https://developer.hpe.com/2024-december-02/https://developer.hpe.com/2024-december-02/Mon, 02 Dec 2024 13:00:00 GMThttps://developer.hpe.com/ll-mesh-exploring-chat-service-and-factory-design-pattern/https://developer.hpe.com/ll-mesh-exploring-chat-service-and-factory-design-pattern/Thu, 21 Nov 2024 12:21:15 GMT<style> li { font-size: 27px !important; line-height: 33px !important; max-width: none !important; } </style> <p>In our previous blog post, we introduced <strong><a href="https://developer.hpe.com/blog/ll-mesh-democratizing-gen-ai-through-open-source-innovation-1/">LLM Agentic Tool Mesh</a></strong>, an <strong><a href="https://github.com/HewlettPackard/llmesh">open-source project</a></strong> aimed at democratizing Generative AI (Gen AI). The initiative addresses both the technical complexity and organizational challenges of adopting Gen AI. The vision behind LLM Agentic Tool Mesh is to make Gen AI accessible and beneficial to a broader audience, empowering users from diverse backgrounds to leverage cutting-edge AI technologies effortlessly.</p> <p>This blog dives deeper into one of LLM Agentic Tool Mesh's core features: the <strong>Chat Service</strong>. The LLM Agentic Tool Mesh platform provides a robust foundation for creating chat applications using Large Language Models (LLMs). First, we'll detail its key services. Then, we’ll explore some code that illustrates how the <strong>Factory Design Pattern</strong> empowers it to handle diverse LLM integrations seamlessly. Finally, we’ll highlight some examples in the LLM Agentic Tool Mesh repository that utilize the Chat Service, such as the <strong>chatbot</strong> application and <strong>agentic tools</strong>, showcasing the flexibility and practical applications of these services.</p> <h2>Key components of LLM Agentic Tool Mesh's Chat Service</h2> <p>Let’s explore the key components and how they enable seamless integration of Gen AI into chat applications.</p> <ol> <li><strong>Prompt rendering</strong>: This service simplifies the creation and management of prompts, which are the backbone of effective LLM interactions. It supports rendering prompts from templates or files and even saving custom prompts to the file system.</li> <li><strong>Model management</strong>: It manages the initialization and utilization of different LLMs based on configuration parameters.</li> <li><strong>Memory management</strong>: Maintaining context in a chat is essential for meaningful interactions. This service manages storage and retrieval of conversation history.</li> <li><strong>Message processing</strong>: This service handles the serialization and deserialization of messages, converting between different data formats to maintain compatibility across components</li> </ol> <p><img src="/img/chat-chat.png" alt=""></p> <h2>The power of the Factory Design Pattern</h2> <p>All the previous services are implemented using the <strong>Factory Design Pattern</strong>. This is a creational design approach that provides a flexible interface for object creation. In LLM Agentic Tool Mesh, this pattern ensures that the platform can handle multiple service types dynamically based on configuration parameters.</p> <p>One of LLM Agentic Tool Mesh's Chat Service features is the <strong><code>ChatModel</code></strong> factory, which simplifies the creation of specific chat models. Here's an example of how it works:</p> <pre><code class="language-python">class ChatModel: _models = { 'LangChainChatOpenAI': LangChainChatOpenAIModel, 'LangChainAzureChatOpenAI': LangChainAzureChatOpenAIModel, 'LangChainChatGoogleGenAI': LangChainChatGoogleGenAIModel, # Additional models... } @staticmethod def create(config): model_type = config.get('type') if not model_type: raise ValueError("Configuration must include 'type'.") model_class = ChatModel._models.get(model_type) if not model_class: raise ValueError(f"Unsupported model type: {model_type}") return model_class(config) </code></pre> <p>Definitions:</p> <ol> <li><strong>Model registry</strong>: The <code>_models</code> dictionary maps model types (e.g., <code>'LangChainChatOpenAI'</code>) to their corresponding classes.</li> <li><strong>Dynamic model selection</strong>: The <code>create</code> method retrieves the desired model class based on the <code>type</code> in the provided <code>config</code>. If <code>type</code> is missing or unsupported, it raises an error.</li> <li><strong>Instance creation</strong>: The method initializes and returns the appropriate model class with the given configuration.</li> </ol> <p>The main benefits of this approach are:</p> <ul> <li><strong>Encapsulation</strong>: Hides the complexity of model initialization.</li> <li><strong>Flexibility</strong>: Switching models only requires updating the <code>type</code> in <code>config</code>.</li> <li><strong>Extensibility</strong>: New models can be added by updating the <code>_models</code> dictionary without modifying the logic.</li> <li><strong>Error handling</strong>: By validating the configuration and supported model types, the design prevents runtime errors, ensuring the system is robust and user-friendly.</li> </ul> <p>Here’s how a developer might use LLM Agentic Tool Mesh's <strong><code>ChatModel</code></strong> factory:</p> <pre><code class="language-python">from athon.chat import ChatModel from langchain.schema import HumanMessage, SystemMessage # Configuration for the Chat Model LLM_CONFIG = { 'type': 'LangChainAzureChatOpenAI', 'api_key': 'your-api-key-here', 'azure_deployment': 'your-deployment-name', 'endpoint': 'your-endpoint-url', 'api_version': 'your-api-version', 'model_name': ‘gpt-4o’, 'temperature': 0.7 } # Initialize the Chat Model chat = ChatModel.create(LLM_CONFIG) # Define prompts prompts = [ SystemMessage(content="Convert the message to pirate language"), HumanMessage(content="Today is a sunny day and the sky is blue") ] # Invoke the model result = chat.invoke(prompts) # Process the response if result.status == "success": print(f"COMPLETION:\n{result.content}") else: print(f"ERROR:\n{result.error_message}") </code></pre> <p>Explanation:</p> <ol> <li>The <code>ChatModel.create()</code> method dynamically selects and initializes the appropriate chat model based on the configuration.</li> <li>Prompts are processed through the <code>invoke()</code> method, which interacts with the LLM to generate responses.</li> <li>Developers handle responses easily, focusing on application logic rather than the complexities of model interaction.</li> </ol> <p>The modular design of LLM Agentic Tool Mesh ensures scalability and maintainability. Here’s a glimpse into its structure:</p> <pre><code class="language-plaintext">self_serve_platform/ │ └── chat/ ├── model.py # Factory method for creating chat model instances └── models/ ├── base.py # Abstract base class for chat models ├── langchain_chat_openai.py # OpenAI chat model implementation ├── langchain_azure_chat_openai.py # Azure chat model implementation └── ... # Additional implementations </code></pre> <h2>Examples of LLM Agentic Tool Mesh in action</h2> <p>In the <a href="https://github.com/HewlettPackard/llmesh">LLM Agentic Tool Mesh GitHub</a>, there are a series of web applications and tools, complete with examples, to showcase the versatility and capabilities of LLM Agentic Tool Mesh. These examples are designed to demonstrate how the platform's services can be leveraged for real-world applications. Notably, the repository includes:</p> <ul> <li> <p><strong>Chatbot application</strong> (<code>examples/app_chatbot</code>): A web-based chatbot built using the chat service, offering a hands-on example of how to integrate LLM-powered conversational agents.</p> </li> <li> <p><strong>Agentic tools</strong>:</p> <ul> <li><strong>Basic copywriter</strong> (<code>examples/tool_copywriter</code>): A tool designed to rewrite and improve text, providing explanations for suggested enhancements. For instance, it can refine content while offering insights into the reasoning behind the changes.</li> <li><strong>Temperature analyzer</strong> (<code>examples/tool_analyzer</code>): A tool that generates Python code using an LLM to analyze historical temperature data. It creates visual charts, enabling users to gain a deeper understanding of temperature trends.</li> </ul> </li> </ul> <p>You can run these tools and applications individually or use the <code>run_examples.sh</code> script to launch them all at once. Once initialized:</p> <ol> <li>Access the <strong>chatbot application</strong> via <code>https://127.0.0.1:5001/</code> to start chatting or explore its additional features like the <strong>personal chat</strong>. Within Personal Chat, users can enhance text using the copywriter tool by interacting directly in chat or clicking interface buttons.</li> <li>The <strong>basic copywriter</strong> also features a configurable backpanel app accessible at <code>https://127.0.0.1:5011/</code>. Without needing to code, users can modify settings such as system prompts, LLM type, or even interface behavior—for example, transforming a formal copywriter into one with a pirate's flair, where one might start by saying "Welcome to our sale" and it would be transformed into "Ahoy, matey, grab the loot!".</li> <li>The <strong>temperature analyzer</strong> enriches its prompts by incorporating historical temperature datasets. It then uses the chat service to generate code as instructed in the system prompt, executes the analysis, and visualizes the results in a chart.</li> </ol> <p><img src="/img/tool-chat.png" alt=""></p> <p>These examples demonstrate the flexibility and potential of LLM Agentic Tool Mesh services, highlighting the power of LLM models and advanced prompt engineering in solving diverse problems.</p> <h3>Conclusion</h3> <p>The <strong>LLM Agentic Tool Mesh Chat Service</strong> and accompanying tools exemplify how advanced design principles like the Factory Design Pattern, combined with innovative prompt engineering, simplify and enhance the adoption of Gen AI. By abstracting complexities and providing versatile examples, LLM Agentic Tool Mesh enables developers and users alike to unlock the transformative potential of Gen AI in a variety of domains.</p> <p>Stay tuned for our next post, where we’ll delve into another key service of LLM Agentic Tool Mesh and explore how it continues to democratize Gen AI!</p>https://developer.hpe.com/use-redfish-and-ipv6-to-find-lost-servers/https://developer.hpe.com/use-redfish-and-ipv6-to-find-lost-servers/Wed, 20 Nov 2024 16:32:49 GMT<style> li { font-size: 27px; line-height: 35px; max-width: none; } </style> <style> figcaption {font-style: italic; font-size: 15px; line-height: 33px; max-width: none;} </style> <h2>Introduction</h2> <p>Have you ever moved one or more servers from a Local Area Network (LAN) to another without configuring the target network detail before the move ? Perhaps some servers in the corporate network <code>10.1.2.0/24</code> have been moved to the lab network <code>19.168.1.0/22</code>. I am sure the answer is yes. As a result, the servers that have been moved are unreachable and could just as well be expensive bricks!</p> <p>A similar situation happens on LANs without a DHCP server when you initiate a <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/securesystemerase/#initiating-secure-erase-through-redfish">secure erase</a> operation on systems using a static iLO IP address. Among other operations, the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=sd00002007en_us&#x26;docLocale=en_US&#x26;page=GUID-8A7C60B7-A3A1-42C6-BEA5-0252F945B7C9.html#ariaid-title2">One-button Secure Erase</a> procedure resets the iLO to its factory defaults which includes the use of DHCP for its network configuration. As a consequence, the iLOs are not easily reachable anymore.</p> <p>This blog post presents a procedure to recover from these unpleasant situations with the help of the <a href="https://datatracker.ietf.org/doc/html/rfc4291#section-2.5.6" target="_blank">IPv6 link local address</a> of the affected iLO network ports. I will also explain what an IPv6 link local is.</p> <h2>Prerequisites</h2> <p>The <a href="#recovery-process">recovery process</a> presented below is suitable for both the dedicated and the shared iLO network ports of lost iLOs. In this post, it is assumed that access to a system connected to the same LAN as the affected iLOs is possible. This system is called the "brick-saver" throughout of this blog post. During my tests, I used a Linux brick-saver, but I am convinced that Microsoft Windows or Apple macOS could have used it as well.</p> <p>In this post, it is also assumed that it is not possible to change the brick-saver network configuration to match the LAN settings of lost iLOs. If that is not the case, this article is not relevant.</p> <p>When the Media Access Control (MAC) address of impacted iLOs is unknown, an operator must be able to access and manipulate iLO cables and power buttons of the affected servers.</p> <p>Then, in order to re-program the network configuration of lost iLOs, you must have the credentials of a privileged user in each of them. This user could be the factory administrator account and associated password mentioned on the "toe-tag" present on the front panel of HPE servers.</p> <blockquote> <p>TIP:<br> If you know that the Link Layer Discovery Protocol (<a href="https://standards.ieee.org/standard/802_1AB-2005.html" target="_blank">LLDP</a>) is enabled in lost iLOs (<a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_changelog/#schema-updates" target="_blank">iLO 6</a> and later) and you have a mean to collect LLDP information, save their <code>ManagementAddressIPv6</code> property in a list. It contains their link local address. With this list, the recovery process is shorter. The next screenshot shows the content of an LLDP transmit object sent over the network by an HPE iLO 6.</p> </blockquote> <p><img src="/img/fig1-lldp-transmit.png" alt="Figure 1: Link local address in LLDP transmit object" title="Figure 1: Link local address in LLDP transmit object"></p> <figcaption>Figure 1: Link local address in LLDP transmit object</figcaption> <h2>IPv6 basics</h2> <p>If you are already familiar with IPv6, link local, and multicast pre-defined addresses like <code>FF02::1%ZoneId</code>, you can skip this paragraph and go directly to the <a href="#recovery-process">Recovery process</a> section.</p> <p>The basic IPv6 architecture is described in Request For Comments (RFC) <a href="https://datatracker.ietf.org/doc/html/rfc4291" target="_blank">4291</a>. Briefly, IPv6 addresses are 128 bit long with the colon (":") character separating eight chunks of sixteen bits. Each chunk is represented by four hexadecimal digits (i.e. <code>2001:1890:1109:4110:9618:82FF:FE71:D01E</code>).</p> <p>Contiguous chunks of zeros can be compressed and represented with "::". Moreover, leading zeros can be omitted. As an example, <code>2001:0DB8:0000:0000:0008:0800:200C:417A</code> can be represented as <code>2001:DB8::8:800:200C:417A</code>.</p> <p>IPv6 addresses are made of two parts: a subnet prefix and an identifier on the subnet. The text representation of subnet prefixes has the form <code>IPv6-Address/subnet-prefix-length</code>, which is similar to the Classless Inter-Domain Routing <a href="https://datatracker.ietf.org/doc/html/rfc1519" target="_blank">(CIDR)</a> representation of IPv4 addresses.</p> <p>With this in mind, <code>2001:1890:1109:4110:9618:82FF:FE71:D01E/64</code> is the text representation of subnet prefix <code>2001:1890:1109:4110::/64</code> and identifier <code>9618:82FF:FE71:D01E</code>.</p> <h3>IPv6 link local addresses</h3> <p>The recovery process presented in this blog post is entirely based on IPv6 link local addresses. Hence, it is important to fully understand what they are and how to use them.</p> <p>IPv6 link local addresses are also defined in <a href="https://datatracker.ietf.org/doc/html/rfc4291#section-2.5.6" target="_blank">RFC 4291</a>. They are called "link local" because packets coming from or going to such addresses cannot be forwarded to external networks. They must stay on the same physical link.</p> <p>When IPv6 is enabled in an Operating System (OS), network interfaces autoconfigure using the well known prefix <code>FE80::/10</code> and the MAC address, as explained in the <a href="https://datatracker.ietf.org/doc/html/rfc4862#section-5.3" target="_blank">IPv6 Stateless Address Autoconfiguration</a> (SLAAC) RFC.</p> <p>Other link local autoconfiguration methods exist. <a href="https://datatracker.ietf.org/doc/html/rfc8981" target="_blank">RFC 8981</a> describes a temporary address creation that is used by Microsoft Windows on personal computers to make them harder to find. This RFC is not implemented in HPE iLOs.</p> <p>Then, <a href="https://datatracker.ietf.org/doc/html/rfc7217" target="_blank">RFC 7217</a> proposes a method for generating opaque interface identifiers. This RFC is implemented in <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_161/ilo6_network_resourcedefns161/#oemhpeipv6" target="_blank">modern versions</a> of iLO 6 and later, but disabled by default.</p> <p>IPv6 is enabled in the HPE iLO OS kernel and cannot be disabled. This is by design. Hence, the configured management interface (dedicated or shared) autoconfigures as soon as it is physically connected to a network. By default, the link local autoconfiguration process transforms the MAC address into an <a href="https://standards.ieee.org/faqs/regauth/" target="_blank">IEEE EUI-64</a> address. Simply explained, this process consists of stretching the MAC address from 48 bits to 64 bits by inserting <code>FF:FE</code> in one place and inverting the universal/local bit of the MAC address. Refer to Appendix A of <a href="https://datatracker.ietf.org/doc/html/rfc4291" target="_blank">RFC 4291</a> for more detail.</p> <p>The following two examples retrieve the EUI-64 link local address of a dedicated iLO 6 network port, its MAC address and the value of the <code>RFC7217Enabled</code> property. Notice the <code>FF:FE</code> string in the link local address and the similarity between the identifier and the MAC address.</p> <pre><code class="language-shell"># iLOrest example ilorest login ilo-lio365g11-1 -u &#x3C;username> -p password ilorest get IPv6Addresses/Address MACAddress Oem/Hpe/IPv6/RFC7217Enabled \ --select EthernetInterface. \ --filter Name="Manager Dedicated*" --json { "IPv6Addresses": [ { "Address": "FE80::5EED:8CFF:FE01:D7C" } ], "MACAddress": "5C:ED:8C:01:0D:7C", "Oem": { "Hpe": { "IPv6": { "RFC7217Enabled": false } } } } ilorest logout </code></pre> <pre><code class="language-shell"># Same example with cURL curl --noproxy \* --insecure --silent --location --user &#x3C;ilo-user>:password \ https://ilo-lio365g11-1/redfish/v1/Managers/1/EthernetInterfaces/1 | \ jq '{"MACAddress":.MACAddress, "IPv6Address":.IPv6Addresses[].Address, "RFC7217Enabled": .Oem.Hpe.IPv6.RFC7217Enabled}' { "MACAddress": "5C:ED:8C:01:0D:7C", "IPv6Address": "FE80::5EED:8CFF:FE01:D7C", "RFC7217Enabled": false } </code></pre> <p>The following example retrieves the same properties as in the previous example, but against an iLO with the <code>RFC7217Enabled</code> property set to <code>true</code>. In this example, the link local address does not contain the <code>FF:FE</code> string and the identifier has no similarities with the MAC address.</p> <pre><code class="language-shell">ilorest login ilo-lio365g11-2 -u &#x3C;username> -p password ilorest get IPv6Addresses/Address MACAddress Oem/Hpe/IPv6/RFC7217Enabled \ --select EthernetInterface. \ --filter Name="Manager Dedicated*" --json { "IPv6Addresses": [ { "Address": "FE80::8751:5A51:F30F:F676" } ], "MACAddress": "5C:ED:8C:01:0E:BE", "Oem": { "Hpe": { "IPv6": { "RFC7217Enabled": true } } } } ilorest logout </code></pre> <h3>How to completely disable IPv6 in iLO?</h3> <p>This question is a bit aside of the core subject of this blog post, but, as it is a recurrent question and there is enough information to answer it clearly in this blog post, I thought I would answer it here.</p> <p>The answer is: <em>Disable DHCPv6 as well as SLAAC</em> as shown in Figure 1 below. I can hear you reply that you did that, but you still can see the link local address like in Figure 1 below. This is expected.</p> <p>As stated previously, SLAAC is always enabled in the iLO OS kernel, and cannot be disabled. As a consequence, the IPv6 link local SLAAC address is always present in both the iLO Graphical User Interface (GUI), and Redfish responses. However, you will not find any SLAAC addresses resulting of an <a href="https://radvd.litech.org/" target="_blank">advertised prefix</a> or a DHCP server.</p> <p><img src="/img/fig2-ipv6-link-local.png" alt="Figure 2: SLAAC link local visible when SLAAC disabled" title="Figure 2: SLAAC link local visible when SLAAC disabled"></p> <figcaption>Figure 2: SLAAC link local visible when SLAAC disabled</figcaption> <h3>How to compute an EUI-64 link local</h3> <p>The easiest method to compute the EUI-64 link local address using a MAC address is with the <code>ipv6calc</code> <a href="https://www.deepspace6.net/projects/ipv6calc.html" target="_blank">utility</a>. Use your favorite package installer on modern Linux systems or compile the sources on Microsoft Windows to make it work.</p> <p>The following block of code transforms a MAC address into its EUI-64 equivalent. By prepending <code>FE80::</code> to the output of the <code>ipv6calc</code> command, you will get the complete SLAAC link-local address (second command of next example).</p> <pre><code class="language-shell">ipv6calc --quiet --in mac 94:18:82:71:A0:7A --out eui64 9618:82ff:fe71:a07a echo -n "fe80::" ; ipv6calc --quiet --in mac 94:18:82:71:A0:7A --out eui64 fe80::9618:82ff:fe71:a07a </code></pre> <h3>Zone indexes</h3> <p>From your brick-saver, when you want to initiate a communication with a link-local IPv6 address, you will have to tell the operating system the network interface to use to send the packets. The reason is because addresses starting with the <code>FE80::/10</code> prefix are present on all the networks directly connected to the system.</p> <p>You can retrieve the zone indexes (also called interface indexes) used by your brick-saver with the following Windows PowerShell command: <code>Get-NetAdapter | Where-Object { $_.Status -eq 'Up' } | Select-Object Name, InterfaceIndex</code>. The output is generally an integer.</p> <p>The equivalent on Linux is: <code>ip link list | awk -F: '/LOWER_UP/ {print $2}'</code>. The output is generally a string like <code>eth0</code>, <code>eno2</code> or <code>enp1s0f4u4</code>.</p> <p>Once you have identified the interface to go through, the complete URL representation of a lost iLO IPv6 target is: <code>https://[ip:v6::address%IZoneIndex]:PortNumber</code></p> <p>Although this syntax is standard, most browsers (Chrome, Brave, Firefox, Edge) don't support it. They don't consider this string as a valid URL that they should follow, but as a string to search on the Internet.</p> <p>However, command line tools like <code>cURL</code>, <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">HPE iLOrest</a>, the <code>Invoke-WebRequest</code> and <code>Invoke-RestMethod</code> PowerShell Cmdlets support it. Here are some examples that retrieve the IPv4 configuration of the iLO dedicated network port using the IPv6 link local address and zone index.</p> <pre><code class="language-shell">curl --noproxy \* --insecure --silent --location \ --user username:password \ https://[FE80::9618:82FF:FE71:A07B%eno1]/redfish/v1/Managers/1/EthernetInterfaces/1 | \ jq '.IPv4Addresses[]' { "Address": "192.168.1.45", "AddressOrigin": "Static", "Gateway": "192.168.1.1", "SubnetMask": "255.255.252.0" } </code></pre> <pre><code class="language-shell">ilorest login https://[FE80::9618:82FF:FE71:A07B%eno1] -u &#x3C;username> -p password ilorest select EthernetInterface. ilorest get Ipv4Addresses --filter Name="Manager Dedicated*" --json { "IPv4Addresses": [ { "Address": "192.168.1.45", "AddressOrigin": "Static", "Gateway": "192.168.1.1", "SubnetMask": "255.255.252.0" } ] } ilorest logout </code></pre> <pre><code class="language-shell"># Use of PowerShell core (7) and Redfish basic authentication $Credentials = "username:password" $base64EncodedCreds = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credentials)) $Headers = @{} $Headers["Content-Type"] = "application/json" $Headers["Authorization"] = "Basic $base64EncodedCreds" $Ipv6Address="FE80::9618:82FF:FE71:A07B%22" $Uri="https://[$Ipv6Address]/redfish/v1/Managers/1/EthernetInterfaces/1" $response = Invoke-WebRequest -Uri $Uri -Method GET -Headers $Headers -SkipCertificateCheck $JsonResponse = $response.Content | ConvertFrom-Json $JsonResponse.IPv4Addresses Address AddressOrigin Gateway SubnetMask ------- ------------- ------- ---------- 192.168.1.47 Static 192.168.1.1 255.255.252.0 </code></pre> <h3>Multicast addresses</h3> <p>An important concept in IPv6 is the absence of broadcast addresses. Instead, IPv6 uses multicast addresses. The exhaustive list of the pre-defined multicast addresses is present in <a href="https://datatracker.ietf.org/doc/html/rfc4291#section-2.7.1" target="_blank">RFC 4291</a>. Among them is the all nodes <code>FF02::1</code> address.</p> <p>From you brick-saver, when you issue a <code>ping FF02::1%Index</code> command, each and every IPv6 interfaces present on the LAN must reply with their link local address. By doing so, you can build the table of all your IPv6 neighbors. You will find more detail on how to use this table in the second scenario below.</p> <h2>Recovery process</h2> <p>The recovery process presented in this blog post relies entirely on EUI-64 IPv6 link local addresses, and thus on MAC addresses. If you don't have the list of the affected iLO MAC addresses, you need to go through the next paragraph.</p> <p>You can skip the next paragraph and study the second scenario if already have that list and your systems are :</p> <ul> <li>iLO 5 based</li> <li>iLO 6 (or later) based with &#x3C;a href="<a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_161/ilo6_network_resourcedefns161/#oemhpeipv6">https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_161/ilo6_network_resourcedefns161/#oemhpeipv6</a>"</li> </ul> <p>target="_blank">RFC7217</a> turned off.</p> <h3>MAC address discovery</h3> <p>The lost iLOs MAC address discovery is performed with the IPv6 neighbor table maintained in your brick-saver.</p> <p>The overall method used is the following:</p> <ol> <li> <p>Make sure affected servers are powered on. This step can only be done by a human being present in the computer room. Then, hide their iLO port from the network by physically disconnecting the network cable. You could unplug the server power cables to achieve this goal, but this would also hide OS interfaces connected to the management LAN. Those interfaces would then re-appear in step 5 below, although they are not iLO interfaces and alter the discovered MAC address list.</p> </li> <li> <p>Connect to your brick-saver, flush the neighbor table, solicitate IPv6 neighbors and save the table in a file.</p> <pre><code class="language-shell"># Interface `eno1` is connected to the management network ip -6 neigh flush dev eno1 # Flush interface eno1's neighbor table ping -6 -c 3 ff02::1%eno1 # Start neighbor solicitation sleep 15 # Wait for solicitation process to finish ip -6 neigh show dev eno1 | \ sort -u > File1.txt # Sort neighbor table and save it </code></pre> </li> <li> <p>Reconnect affected iLO cables.</p> </li> <li> <p>Repeat step 2.</p> <pre><code class="language-shell"># Interface `eno1` is connected to the management network ip -6 neigh flush dev eno1 # Flush interface eno1 neighbor table ping -6 -c 3 ff02::1%eno1 # Start neighbor solicitation sleep 15 # Wait for neighbor solicitation to finish ip -6 neigh show dev eno1 | \ sort -u > File2.txt # Sort neighbor table and save it </code></pre> </li> <li> <p>The list of affected iLO link local and MAC addresses is obtained by computing the difference of the files saved in steps 2 and 4.</p> <pre><code class="language-shell"># The following command stores discovered link local IPv6 addresses # in variable LL_LIST LL_LIST=$(diff File1.txt File2.txt | awk '/^>/ {print $2}') echo $LL_LIST fe80::5eed:8cff:fe01:d7c fe80::4645:c007:6bc2:bac6 fe80::5eed:8cff:fe6b:617 </code></pre> </li> <li> <p>You should verify that the list collected above contains Redfish conformant addresses and only Redfish conformant addresses. This can be done by performing a GET request toward each IPv6 link-local and extract their <code>RedfishVersion</code> property. If the list contains a non-redfish conformant address, the following <code>cURL/jq</code> commands will report an error.</p> <pre><code class="language-shell">for ll in $LL_LIST ; do curl --proxy \* --insecure --silent https://[${ll}%eno1]/redfish/v1 | \ jq '.RedfishVersion' done "1.20.0" parse error: Invalid numeric literal at line 1, column 10 "1.13.0" </code></pre> </li> </ol> <p>If your brick-saver is running Microsoft Windows, here are some hints to perform the above procedure.</p> <pre><code class="language-shell"># Start a privileged PowerShell session and retrieve Zone indexes Get-NetAdapter | Where-Object { $_.Status -eq 'Up' } | Select-Object Name, InterfaceIndex # Show neighbors: netsh interface ipv6 show neighbors interface=22 > File.txt # Delete neighbor table netsh interface ipv6 delete neighbors interface=22 # Populate neighbor table Test-Connection -Count 1 -ComputerName ff02::1%22 Compare-Object (Get-Content File1.txt) (Get-Content File2.txt) | Where-Object { $_.SideIndicator -eq '=>' } | Select-Object InputObject </code></pre> <h3>Known MAC addresses and RFC 7217 disabled</h3> <p>If you have the list of lost MAC addresses and you know that <a href="https://datatracker.ietf.org/doc/html/rfc7217" target="_blank">RFC 7217</a> is set to disabled, you just have to compute the corresponding IPv6 link-local addresses with the <code>ipv6calc</code> tool and verify their accuracy.</p> <blockquote> <p>NOTE: RFC 7217 is disabled by default as explained <a href="#ipv6-link-local-addresses">above</a>.</p> </blockquote> <p>The following example computes and stores IPv6 link local addresses in a variable. Then it tests whether they correspond to a Redfish conformant (iLO) address by requesting the property <code>RedfishVersion</code> from the <code>ServiceRoot</code> URI.</p> <pre><code class="language-shell">MacList="5C:ED:8C:01:0D:7C 5C:ED:8C:6B:06:17" LL_LIST="$(for mac in $MacList ; do echo -n "fe80::" ; ipv6calc --quiet --in mac $mac --out eui64 done )" for ll in $LL_LIST ; do curl --noproxy \* --insecure --silent https://[${ll}%eno1]/redfish/v1 | \ jq '.RedfishVersion' done "1.20.0" "1.13.0" </code></pre> <p>You can use iLOrest to verify that the computed list of link local addresses corresponds to iLO addresses. However, you will need to provide credentials, even for accessing the Redfish root URI.</p> <pre><code class="language-shell">u="ilo-user" p="ilo-password" for ll in $LL_LIST ; do ilorest --nologo login [${ll}%eno1] -u $u -p $p >/dev/null ilorest --nologo get RedfishVersion --select ServiceRoot ilorest --nologo logout >/dev/null done RedfishVersion=1.20.0 RedfishVersion=1.13.0 </code></pre> <h3>Configuration recovery</h3> <p>The last step to completely recover from the lost iLOs is to re-configure their network parameters. From the brick-saver, the following example shows a generic PATCH request and its associated payload to configure a minimal static IPv4 configuration of a lost iLO.</p> <p>With the previous examples, it should be easy for you to adapt it for iLOrest, cURL or PowerShell. Other examples for configuring iLO network parameters are presented in the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishclients/ilorest-userguide/examplecommandsscripts/#configure-ilo-ip-addresses" target="_blank">documentation</a>.</p> <pre><code class="language-text">PATCH request URL https://[FE80::9618:82FF:FE71:A07B%eno1]/redfish/v1/Managers/1/EthernetInterfaces/1/ Payload: { "DHCPv4": { "DHCPEnabled": false, "UseDNSServers": false, "UseDomainName": false, "UseGateway": false, "UseNTPServers": false, "UseStaticRoutes": false }, "IPv4StaticAddresses": [ { "Address": "192.168.1.44", "Gateway": "192.168.1.1", "SubnetMask": "255.255.252.0" } ] } </code></pre> <h2>Conclusion</h2> <p>This article presented a method with simple PowerShell and Linux commands to find lost servers after a move from one network to another without any configuration preparation prior to the move.</p> <p>As this method uses IPv6, many of its concepts have been introduced, including its autoconfiguration process. If you like more detail on autoconfiguration, I would suggest to read the corresponding section I wrote some time ago in the <a href="https://ipj.dreamhosters.com/wp-content/uploads/issues/2004/ipj07-2.pdf" target="_blank">Internet Protocol Journal</a>.</p> <p>And don't forget to check out some of my other <a href="https://developer.hpe.com/search/?term=donze" target="_blank">blog posts</a> on the HPE Developer Community portal to learn more about Redfish tips and tricks.</p>https://developer.hpe.com/vlan-versus-vxlan/https://developer.hpe.com/vlan-versus-vxlan/Mon, 18 Nov 2024 14:57:37 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In a modern data center or large-scale network environment, <strong>network segmentation</strong> and isolation are key to optimizing both <strong>performance and security</strong>. <strong>Two popular technologies</strong> that address this need are <strong>VLANs and VXLANs</strong>, but they serve slightly different purposes and have distinct configurations. Understanding the nuances of both is essential for network engineers, systems administrators, and architects when deciding how to structure their networks.</p> <p>While the two technologies appear very similar from a high level, they <strong>operate at different layers</strong> in the networking stack and differ in capabilities, scalability, and use cases. In this post, I'll define what a VLAN and VXLAN is, provide details showing their headers, show how they are configured and benefits of using VXLAN over VLAN.</p> <h3>What is VLAN?</h3> <p>A <strong>VLAN (Virtual Local Area Network)</strong>, operates at <strong>Layer 2</strong> and segments a physical network into multiple isolated networks, through the use of <strong>VLAN-ID</strong>, which can range from <strong>&#x3C;1-4096> (12-bit)</strong>. Different ports on a networking device can be assigned to different VLAN-IDs. The networking device ensures that the incoming or outgoing traffic is forwarded based on the VLAN-ID that is present in the ethernet frames.</p> <p>So, if the physical network is defined as Layer 1, the segmented physical network using VLANs is the Layer 2 data link layer.</p> <h3>What is VXLAN?</h3> <p>A <strong>VXLAN (Virtual Extensible Local Area Networks)</strong>, operates as an encapsulation protocol designed to <strong>extend Layer 2 networks over Layer 3</strong> infrastructure. Due to this encapsulation, VXLAN is also called IP tunneling protocol to carry Layer 2 traffic over Layer 3 network for <strong>wide area deployments</strong>.</p> <p>VXLAN segment is identified by <strong>VNI (Virtual Network Identifier)</strong> which is 24 bits. The Layer 2 Ethernet frames are encapsulated by <strong>VNI outer headers</strong>. The VXLAN encapsulation is done by <strong>VTEP (VXLAN tunnel end point)</strong>. A networking device which is configured as VTEP adds the VXLAN header to the Ethernet frame.</p> <h4>How VXLAN encapsulation works?</h4> <p>The VTEP must be configured on a networking device which encapsulate and decapsulates the ethernet frames into VXLAN packets. The intermediate networking devices receiving these VXLAN packets are routed as any other Layer 3 Packet.</p> <p>The VXLAN encapsulated packet or the VNI outer headers contains:</p> <p><strong>1. Other Ethernet Header (Layer 2)</strong></p> <ul> <li><strong>Destination MAC Address:</strong> The MAC address of the next-hop device.</li> <li><strong>Source MAC address:</strong> The MAC address of the VTEP device sending the encapsulated frame.</li> <li><strong>Ether Type:</strong> Typically, <code>0x0800</code> for IPv4 or <code>0x86DD</code> for IPv6.</li> </ul> <p><strong>2. Outer IP Header (Layer 3)</strong></p> <ul> <li><strong>Source IP Address</strong>: The IP address of the VXLAN tunnel endpoint (VTEP) originating the packet.</li> <li><strong>Destination IP Address</strong>: The IP address of the remote VTEP, where the VXLAN frame is being sent.</li> <li><strong>Protocol</strong>: Usually, this is set to <code>0x11</code>, which signifies UDP (User Datagram Protocol).</li> </ul> <p><strong>3. Outer UDP Header (Layer 4)</strong></p> <ul> <li><strong>Source Port</strong>: Typically randomized by the sending VTEP.</li> <li><strong>Destination Port</strong>: Fixed at <code>4789</code>, which is the well-known port for VXLAN.</li> <li><strong>Length</strong>: Specifies the length of the UDP datagram, including the VXLAN header.</li> </ul> <p><strong>4. VXLAN Header (Layer 4)</strong></p> <ul> <li>The VXLAN header is inserted within the UDP payload.</li> <li><strong>VXLAN Network Identifier (VNI, 24 bits)</strong>: A 24-bit field used to identify the VXLAN segment or virtual network.</li> </ul> <p><strong>5. Original Ethernet Frame (Layer 2)</strong></p> <ul> <li>This actual Ethernet frame including the originated Source MAC address, Destination MAC Address, Ether Type and it may contain the VLAN Header, IP Header, UDP Header and Application data.</li> </ul> <h4>Why do we need VXLAN?</h4> <p>With VLANs being used for segmentation across and within data centers, the Spanning Tree Protocol (STP) is used to build a loop-free logical topology for Ethernet networks. The basic function of STP is to prevent bridge loops and the broadcast radiation that results from them. <strong>STP disables links</strong> that are not part of the spanning tree, leaving a single active path between two network nodes. This can result in a <strong>large number of disabled links</strong>, an issue that can be <strong>resolved by using a Layer 3 virtual tunnel over the Layer 2 network</strong>.</p> <p>Another issue that <strong>VXLANs resolve</strong> is that of <strong>scalability</strong>. A 12-bit VLAN ID is used in the Ethernet data frames to divide the larger Layer 2 network into multiple broadcast domains. As such, VLANs can only be used for data centers that require fewer than 4094 VLANs, limiting their scalability.</p> <p>Consider a situation where there are 24 or 48 servers connected to a Top-of-the-Rack (ToR) switch. Instead of just one MAC address per server link, the <strong>switch must learn all the MAC addresses of the virtual machines (VMs)</strong> communicating across the servers. This <strong>inadequate MAC address table issue</strong> would be <strong>addressed with the Layer 3 tunnel over the Layer 2 frames</strong>.</p> <h3>VLAN Header in Ethernet Frame</h3> <p>This image illustrates the structure of an <strong>Ethernet frame with a VLAN tag</strong>. As you can see, the VLAN header is inserted after the source MAC address. VLAN header contains a <strong>4-byte VLAN tag field used to identify the VLAN-ID</strong> from which the packet originated. The format of VLAN-tagged frames is defined in IEEE 802.1Q standard.</p> <p><img src="/img/picture1.png" alt="VLAN Header" title="VLAN Header"></p> <ul> <li> <p>The <strong>VLAN tag</strong> displays the following fields:</p> <ul> <li><strong>TPID</strong> – By Default VLAN Tagged packets has 0x8100.</li> <li><strong>Priority</strong> – To indicate the 802.1p priority of the frame</li> <li><strong>CFI(Canonical Format Indicator)</strong> – By default it is 0. 0 indicates MAC addresses are in standard format. 1 indicated non-standard format.</li> <li><strong>VLAN-ID</strong> – 12 Bit value to identify the VLAN.</li> </ul> </li> </ul> <h3>VXLAN Header in Ethernet Frame</h3> <p>This image illustrates the structure of VXLAN encapsulated Ethernet frame including the VXLAN header.</p> <p><img src="/img/picture2.png" alt="VXLAN Header" title="VXLAN Header"></p> <ul> <li> <p><strong>VXLAN Header</strong> contains the following fields:</p> <ul> <li><strong>Flags</strong> – The <code>"I"</code> bit would be set to 1 for a valid VXLAN Network ID (VNI). The other reserved 7 bits would be set to zero.</li> <li><strong>VXLAN-ID</strong> – This is the 24 bit value used to designate the VXLAN overlay network.</li> </ul> </li> </ul> <h3>VLAN and VXLAN Configuration:</h3> <p>To understand VXLAN further, let us go through the <strong>configurations of VLAN and VXLAN, VNI and VTEP.</strong> Each VNI would be mapped with the VLAN it is carrying.</p> <p><img src="/img/picture3.png" alt="VLAN, VXLAN Configuration" title="VLAN, VXLAN Configuration"></p> <h3>FAQs</h3> <ol> <li> <p>Will VXLAN replace VLAN standard?</p> <ul> <li>VLAN standard is specified in the IEEE 802.1Q standard by IEEE. VXLAN standard is specified in the RFC 7438 by IETF. VXLAN encapsulates the VLAN tagged packet with four outer headers to transmit the packet in IP based network. It does not replace the VLAN standard.</li> </ul> </li> <li> <p>Does VXLAN always have an outer VLAN header?</p> <ul> <li>The outer VLAN header in VXLAN headers is optional. It is based on the deployment scenario.</li> </ul> </li> <li> <p>Does VXLAN only solve the scalability issue?</p> <p>Apart from scalability issues, VXLAN addresses the issues below.</p> <ul> <li>VXLAN encapsulated packet contains IP header. So, the VXLAN technology is used for having IP based traffic in core network.</li> <li>The VNI (VXLAN Network Identifier) serves as a unique identifier for a Layer 2 segment (or virtual network) within a VXLAN environment. VXLAN based network segmentation which solves two major issues mentioned below:</li> <li>This allows the MAC addresses of devices (such as virtual machines or VMs) within different VXLAN segments to overlap without causing any conflict or risk of traffic "crossover" between virtual networks.</li> <li>This allows multiple VMs to exist within the same physical Layer 3 network but remain logically isolated in their own VXLAN segment, even if they have overlapping IP addresses and MAC addresses.</li> </ul> </li> <li> <p>What’s the difference between QinQ and VXLAN?</p> <ul> <li>QinQ (also known as 802.1ad or stacked VLANs) is a Layer 2 technology that adds an outer VLAN tag to the original Ethernet frame, allowing for a hierarchical VLAN structure. This technology is used in service provider environments, where multiple customers need to share the same physical network while maintaining their own isolated virtual networks.VXLAN is primarily used in data centers and cloud networks to create large-scale virtualized Layer 2 networks over a Layer 3 network.</li> <li>VXLAN is primarily used in data centers and cloud networks to create large-scale virtualized Layer 2 networks over a Layer 3 network.</li> </ul> </li> </ol> <h4>Summary</h4> <p>To summarize, both <strong>VLAN and VXLAN</strong> are valuable tools for <strong>network segmentation</strong>, but they serve different needs. VLANs remain a simple and effective choice for smaller, less complex networks, while VXLAN provides the <strong>scalability and flexibility</strong> needed for modern, distributed, and virtualized environments. Understanding the technical differences in configuration and their respective use cases will help you choose the right solution for your network architecture, ensuring optimal performance, security, and scalability. Both VLAN and VXLAN are used together to solve scalability issues and to <strong>overcome the limitations of VLAN</strong>.</p> <p>To learn more about VLANs and VXLANs, please check out this documentation here (<a href="https://www.arubanetworks.com/techdocs/AOS-CX/10.12/PDF/vxlan.pdf">https://www.arubanetworks.com/techdocs/AOS-CX/10.12/PDF/vxlan.pdf)</a></p>https://developer.hpe.com/developer-assistance-for-those-looking-to-work-with-ampere-arm-processors/https://developer.hpe.com/developer-assistance-for-those-looking-to-work-with-ampere-arm-processors/Mon, 18 Nov 2024 11:37:14 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>The HPE ProLiant RL300 Gen11 is renowned as a best-in-class compute platform for cloud-native workloads. Part of what makes it so special is its use of the Ampere® Arm processors. Many of today’s progressive, open-source-centric customers are quickly pivoting to these AArch64 processors for the predictable performance they deliver at the scale required for cloud processing. Others are still looking for assistance on how to make the transition from x86.</p> <p>We recently published a couple of blog posts on the HPE Developer Community portal that provide some assistance. The first, <a href="https://developer.hpe.com/blog/simplifying-code-migration-the-benefits-of-the-new-ampere-porting-advisor-for-x86-to-arm64/">Simplifying code migration: The benefits of the new Ampere Porting Advisor for x86 to ARM64</a>, describes a new open-source software porting advisor that promises to simplify the porting process. Another post introduces the <a href="https://amperecomputing.com/blogs/ampere-performance-toolkit-announcement">Ampere Performance Toolkit</a>, which provides an automated way to run and benchmark important application data.</p> <p>In our research on how to best assist developers in dealing with these processors, we came across the <a href="https://amperecomputing.com/blogs/ampere-launches-developer-program-for-the-cloud-computing-community">Ampere Developer Program for the Cloud Computing Community</a>. On this site, you’ll find:</p> <ul> <li>Porting and tuning utilities</li> <li>Developer outreach technical documents and resources</li> <li>Customer quick start guides</li> <li>And resources for every stage of development</li> </ul> <p>We understand that this site will continue to expand to include things like:</p> <ul> <li>Workload briefs</li> <li>Transition and tuning guides</li> <li>Reference architectures</li> <li>Developer blog posts</li> <li>Tutorials</li> <li>And case studies</li> </ul> <p>If you’re developing on any HPE systems that are using the Ampere ARM processor, you may wish to consult this Ampere Developer Program for more resources.</p>https://developer.hpe.com/navier-stokes-in-chapel-—-distributed-cavity-flow-solver/https://developer.hpe.com/navier-stokes-in-chapel-—-distributed-cavity-flow-solver/Thu, 14 Nov 2024 19:46:35 GMT<p>external blog</p>https://developer.hpe.com/getting-started-with-retrieval-augmented-generation-rag/https://developer.hpe.com/getting-started-with-retrieval-augmented-generation-rag/Thu, 14 Nov 2024 07:42:05 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Keeping up with AI technology and understanding more about Generative AI and LLM (Large Language Models) is quickly becoming an imperative. If you are like me, curious but not that much of a data scientist or a mathematician, you don't really want to deep dive into the gory details of a model and, instead, would prefer to consider them as interchangeable black boxes with specific properties and application ranges. Still, it’s important to understand how to consume some of these foundation models in order to build something for your own business. Let’s say, for example, that you want to use AI to help customers on your commercial web site, something often referred to as a Chatbot.  </p> <p>This blog post will show you how to build a simple questions-answering generative AI application using the LangChain orchestration framework powered by a large language model with a private data source to augment the knowledge of the model and answer the question more effectively. </p> <h2>Using a foundation model </h2> <p>LLMs fall into a category called foundation models (FMs). They are large deep learning neural networks trained on massive datasets. You’ve probably heard about, and even used, foundation models like OpenAI’s GPT, Anthropic’s Claude, or Meta’s Llama. Models like these have been trained using such a large dataset that it can answer all kinds of question from users. You might wonder, why can’t I just use such a foundation model, like GPT or Llama3, on my website to create custom generative AI model? Unfortunately, due to the fact that the model was not trained on your private data, i.e. details of the products you sell on your website, there is little chance that the model would be able to provide a valid response to your customer inquiries. </p> <p>At this point, you have two choices:  </p> <ol> <li>You can invest time and money into finetuning (training) a large language model that includes your product information or private data. It can be complex, expensive and resource intensive, especially if you finetune your model each time you have new data you want the model to know. Finetuning an LLM can be complicated as it requires the right skill set to do the work correctly. It can also be costly as it requires a lot of compute resources including CPU, GPU, power, and cooling. </li> <li>You can use the most appropriate pre-trained foundation model that’s available and augment its knowledge with your own data, all without the need to train the model. This is what we call Retrieval Augmented Generation (RAG). </li> </ol> <h2>RAG to the rescue </h2> <p>RAG is a combination of a natural language processing (NLP) technique combined with a large language model (LLM). A RAG system augments the knowledge that an LLM provides thanks to its training with its own knowledge base. This allows the model to provide responses based on more specific or private content, which would not be found in the public data used to train the foundation model.</p> <h2>The components of the RAG system </h2> <p>In order to build a RAG system, you would need the following: </p> <ul> <li>Access to an LLM (locally or in the cloud) such as Meta’s Llama3, OpenAI’s GPT, or many others. </li> <li>The documents you want to put in the RAG database. </li> <li>A Text Embedding model such as NVIDIA NeMo Retriever Text Embedding NIM to convert the documents into numerical vectors. </li> <li>A vector database such as open-source ones including Milvus, Qdrant, or Chroma. A vector database is a collection of data stored as mathematical representations. This is used to store the private data in the form of embeddings, also known as vectors. </li> <li>A chain server such as LangChain or LlamaIndex. This is responsible to coordinating dialogs between the user, the vector database, and the LLM by connecting the LLM to external data sources. </li> <li>You will also need a way for users to enter their prompt and read the responses from the system along with a mechanism for populating your private data into the vector database. The LangChain framework implements the prompt using a prompt template. </li> </ul> <h2>Seeing is believing </h2> <p>NVIDIA provides a good way to build this and get your hands (a little) dirty. You can find it on <a href="https://github.com/NVIDIA/GenerativeAIExamples">this GitHub repository</a>.</p> <p>Once the repository is cloned, you can select <a href="https://github.com/NVIDIA/GenerativeAIExamples/tree/main/RAG/examples/basic_rag/langchain">the basic RAG using LangChain</a> example.   </p> <p>As explained in the <a href="https://github.com/NVIDIA/GenerativeAIExamples/blob/main/RAG/examples/basic_rag/langchain/README.md">README.md</a>, the example uses NeMo Retriever Text Embedding NIM as text embedding model, Milvus for its vector database, LangChain for its chain server, and meta/llama3-70b-instruct for its LLM. </p> <p><img src="https://github.com/NVIDIA/GenerativeAIExamples/raw/main/docs/images/basic_rag_langchain_arch.png" alt="Architecture of the RAG example" title="Architecture of the RAG example"></p> <p><strong>Prerequisites</strong>: To get this working you will need the following: </p> <ul> <li>A machine with Ubuntu, Docker and Docker Compose (a GPU is not mandatory) </li> <li>An API key from <a href="https://build.nvidia.com/explore/discover">https://build.nvidia.com/explore/discover</a> </li> <li>An NGC API key, which you can get by joining NVIDIA developer program. Refer to these instructions to <a href="https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#generating-api-key">generate an NGC API key</a>. </li> </ul> <p>Once you have this ready, you can use Docker Compose to build the solution and start it. Docker Compose will download from the NVIDIA container registry, build, and run the following:</p> <pre><code class="language-markdown">$ docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}" CONTAINER ID NAMES STATUS 436a1e0e1f2a milvus-standalone Up 18 hours c3a5bf578654 rag-playground Up 18 hours 78be16cb2ef4 milvus-minio Up 18 hours (healthy) dc3bcc2c7b16 chain-server Up 18 hours af87e14ff763 milvus-etcd Up 18 hours (healthy) </code></pre> <p>Let me quickly run through what each container is all about: </p> <ul> <li>milvus-standalone, milvus-etcd and milvus-minio: the vector database used as a knowledge base for private data </li> <li>rag-playground: the GUI to capture the user prompt and display the response from the LLM. It also has the GUI for selecting documents to add into the knowledge base  </li> <li>chain-server: the LangChain based chain server </li> </ul> <p>As you can see from the port configuration of the rag-playground container, it's listening to port 8090. </p> <pre><code class="language-markdown">docker port rag-playground 8090/tcp -> 0.0.0.0:8090 8090/tcp -> [::]:8090 </code></pre> <p>So, let’s open a browser on this URL to see the RAG in action.</p> <p><img src="/img/ragblog-1.jpg" alt="Connect to web UI" title="Connect to web UI"></p> <h3>Step 1 - Query using a foundation LLM </h3> <p>You can see that the model used is meta/llama3-70b-instruct, so first try only using the LLM. To do this, enter some text in the prompt window. For the sake of the example here, I'll pretend to be a mountain bike dealer, carrying the Swiss brand of mountain bikes called Flyer. Imagine that I am looking for details of one model called the <em>Uproc</em>. I ask the model: <em>What is a Flyer Uproc?</em>.</p> <p><img src="/img/ragblog2.jpg" alt="Asking what is a flyer uproc without context" title="Asking what is a flyer uproc without context"></p> <p>As you can see from the response, there is a problem, as the LLM was not trained with the information about the Flyer brand. The model did find something about <em>Flyer Uproc</em>, but clearly this is not what my customer would expect to find on my web site. </p> <h3>Step 2 - Loading data into the knowledge base </h3> <p>To address this issue, use a PDF that describes the technical specifications of the Flyer Uproc in great detail. In the UI, select <strong>Knowledge base</strong> from the upper right corner, then select the PDF (any PDF, TXT or markdown files can work, too). The system will check the content of the file, then build a series of embeddings to describe it, and then store these embeddings in the vector database. It only takes a few seconds to process the details contained in the PDF.</p> <p><img src="/img/ragblog3.jpg" alt="Adding a PDF in the knowledge base" title="Adding a PDF in the knowledge base"></p> <h3>Step 3 - Query using RAG </h3> <p>Now that the knowledge base includes details on the <em>Flyer Uproc</em> product line, try the same query, but this time, make sure to check the "use knowledge base" checkbox. </p> <p><img src="/img/ragblog4.jpg" alt="Asking what is a flyer uproc with context" title="Asking what is a flyer uproc with context"></p> <p>As you can see, the context, provided by the augmented knowledge base, provides additional knowledge to the LLM, and it is now able to provide a much better result, delivering a lot more value to my customers. The RAG system has retrieved the most relevant information from the vector database and passed the information to the LLM. The LLM then used the information to answer the question more effectively. </p> <p>Other examples of customer questions could be:</p> <p><img src="/img/ragblog5.jpg" alt="Asking more questions to system" title="Asking more questions to system"></p> <p><img src="/img/ragblog6.jpg" alt="Asking more questions to system" title="Asking more questions to system"></p> <p>I think you get the point. If this was a Chatbot on my website, it would clearly be very efficient and helpful to my customers.</p> <h2>Next steps </h2> <p>This lab exercise is a great way to put rapidly a RAG system into play. Remember, however, that it calls the NVIDIA API to get to the Llama3 model, which has a cost associated with it. NVIDIA provides a number of credits to get you started, but these credits won’t last forever. </p> <p>The next step from here might be to run this entirely on premises, in your datacenter, on GPU-equipped hardware. There is another version of this lab dedicated to doing this <a href="https://github.com/NVIDIA/GenerativeAIExamples/tree/main/RAG/examples/local_deploy">here</a>. </p> <p>If you are looking for the right platform to run an application like this, please check <a href="https://www.hpe.com/us/en/private-cloud-ai.html">HPE Private Cloud for AI,</a> which has been developed in collaboration with NVIDIA. It is a platform that has been developed specifically for running these types of workloads and can be sized in accordance with your specific needs.</p> <p><img src="/img/pcai.webp" alt="HPE Private Cloud for AI" title="HPE Private Cloud for AI"></p>https://developer.hpe.com/how-to-pick-a-large-language-model-for-private-ai/https://developer.hpe.com/how-to-pick-a-large-language-model-for-private-ai/Wed, 13 Nov 2024 16:48:01 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><em>Note: this blog is based on a video and slide deck available <a href="https://www.youtube.com/watch?v=sNRqJOEKkCw">here on YouTube</a>.</em></p> <p>As organizations continue to explore the potential of generative AI, choosing the right Large Language Model (LLM) is crucial for performance, scalability, and integration with existing systems. This post explores key considerations for selecting an LLM, including understanding different classes of models, evaluating performance, and planning for hardware requirements.</p> <h2>Classes of Language Models: What Does “Large” Mean?</h2> <p>Language models vary significantly in scale, with "large" models generally characterized by the number of parameters they contain. However, "large" has become a misnomer, with many models capable of running on mobile devices like laptops and cell phones. The size of the model will correspond with two types of "performance" -- quality and speed. Understanding these tradeoffs is important when considering the models feasibility for certain applications. <br> <br> In the chart below I describe five T-Shirt sizes of language models.</p> <p><img src="/img/capture1.png" alt=""></p> <h2>What Can These AI Models Do?</h2> <p>LLMs are remarkably versatile, but capabilities can still differ between models. Understanding these distinctions can help narrow down which model class best suits an organization's goals and budget. Performance on given task that is meaningful to your business may depend on the model's architecture, training data, and parameter count.</p> <p>Let us define three classes of problems that LLMs can perform: Chat, Code, and everything else.</p> <h3>Chat</h3> <p>In the domain of Chat, models take in text and output text. It's been the killer app in NLP since ChatGPT was released, and there is no reason to think this trend is slowing down. Seems simple, however there are endless ways in which this can be useful. Here is a list of a few that I have worked on:</p> <ul> <li>Summarize a long email thread and help me write a response</li> <li>Translate a pdf in a foreign language, such as an RFP or product specs and help answer some questions</li> <li>Create a study guide, flashcards, and a multiple-choice practice quiz based on a chapter of a textbook</li> <li>Classify the audio transcription of a patient's visit with a medical professional to determine who the follow up appointment would be with</li> <li>Determine if an auto insurance claim is likely to be fraudulent or not, and explain the reasoning why to a care agent</li> <li>Ask questions about the contents of a database, such as local restaurant reviews or the performance of a business over the past month</li> </ul> <p>I may have even used one to help write this blog post 😁</p> <p>Moving on...</p> <h3>Code</h3> <p>The second most popular LLM-powered application today seems to be GitHub Copilot. It's already spawned open-source projects (with much more flexibility) such as <a href="https://www.cursor.com/">Cursor</a> and <a href="https://continue.dev/">Continue</a>. I'm a big Continue user myself.</p> <p>There are three main things that I do with Cursor: Generate, Edit, and Explain.</p> <ul> <li><strong>Generate</strong> is text in, code out. You write a description of the code you want to see, and the model generates it from scratch. For example, a new page on a website, an API call, or a new function in a library. Personally, I've been writing a lot of yaml for k8s recently (is config code?). Also, I don't hit the docs to figure out slurm and linux commands as much anymore (is bash code? alas...)</li> <li><strong>Edit</strong> is code + text in, code out. Editing is similar to generating code, but you're including some existing code in-context for the model to use. For example, you might want to write a unit test, create a function, refactor something, or add comments. The model is reasoning based on existing code.</li> <li><strong>Explain</strong> is code in, text out. For example, you might be moving onto a new project and trying to understand the existing code base. Or you forgot what you were trying to do the night before. Explanations can be helpful.</li> </ul> <h3>Everything Else</h3> <p>Models are improving over time. With the introduction of "multimodal" LLMs, we now have both "Large" and "Language" being misnomers in this acronym. Anyway, this is what I'll call the domain of Everything Else:</p> <ul> <li>Analyze the contents of an image (say, diagnose an X-Ray for a patient)</li> <li>Generate an image (text-to-image)</li> <li>Transcribe audio (speech-to-text)</li> <li>Generate audio (text-to-speech)</li> <li>Generate a video (text-to-video)</li> </ul> <p>You can see that in general, text is still usually the modality of input or output. However, transformer models are being used in unique ways (audio-to-audio, image-to-video, etc.) and research continues. How long until this blog is out of date?</p> <h2>Types of Performance: Quality vs. Speed</h2> <p>Evaluating LLM performance involves two key metrics:</p> <ol> <li><strong>Quality</strong>: Refers to the accuracy or relevance of the model’s outputs. Quality can be assessed through various benchmarks like the <a href="https://lmarena.ai/?leaderboard">LMSys Chatbot Arena</a>, <a href="https://github.com/hendrycks/test">MMLU,</a> <a href="https://github.com/openai/human-eval">HumanEval</a>, and <a href="https://github.com/openai/grade-school-math">GSM8K</a>, which measure models on complex reasoning and specific problem-solving tasks.</li> <li><strong>Speed</strong>: Involves both throughput and latency. Throughput is the volume of outputs the model can generate per unit of time (request/sec or tokens/sec), while latency (time-to-first-token, or TTFT) indicates the delay before generating the first part of a response.</li> </ol> <h2>Comparing Quality</h2> <p>In terms of measuring quality, it's frankly just really hard to understand the objective quality of an LLM. Open source benchmarks exist, but they're usually not representative of your business objectives. To understand how a model performs on the specific downstream task that your organization will be performing, there is really no substitute for user feedback and custom evals. I'm going to leave this topic for a future blog.</p> <p>Test your models, and talk to your users.</p> <h2>Comparing Speed</h2> <h3>Throughput</h3> <p>Throughput metrics offer insights into how well models handle large volumes of data. High throughput is especially valuable for batch processing tasks, where efficiency directly translates into time and cost savings.</p> <h3>Latency (TTFT)</h3> <p>Latency reflects the model’s responsiveness, which is critical for real-time applications. Lower latency means faster initial responses, which is crucial for user-facing applications, like chatbots or customer support systems.</p> <h2>Continuous Improvement in Runtimes</h2> <p>The field of LLMs is rapidly advancing, with regular improvements in model efficiency and runtime performance. For instance, in a span of just over three months, throughput for certain models nearly doubled thanks to improvements in the vLLM runtime from v0.5.0 (released in June 2024) to v0.6.2 (released in September 2024).</p> <p>Whether you prefer <a href="https://ollama.com/">ollama</a>, <a href="https://docs.vllm.ai/en/latest/">vLLM</a>, <a href="https://huggingface.co/docs/text-generation-inference/en/index">tgi</a>, <a href="https://www.nvidia.com/en-us/ai/">NIMs</a>, or something else, staying updated with these advancements is important to keep deployments cost-effective over time.</p> <h2>Conclusion: Infrastructure Sizing</h2> <p>Determining the right infrastructure to support an LLM is critical to deployment success. Hardware requirements are influenced by model size and complexity, since larger models generally require more powerful infrastructure, as well as desired performance, since the need for high throughput or low latency impacts the choice of hardware and the amount of it.</p> <p>The obvious conclusion to draw here is that there is a tradeoff:</p> <ul> <li>bigger models provide higher quality output, but are slower (or more expensive to run)</li> <li>smaller models provide lower quality output, but are faster (or less expensive to run)</li> </ul> <p>Understanding this tradeoff is critical to ensure that you have control over the cost and performance (quality and speed) of your LLM-powered application.</p> <hr> <p>Watch this video for more detail, and live demo comparing latency and throughput for three different sizes (8B, 70B, 405B) of the Llama 3.1 Instruct model: <a href="https://www.youtube.com/watch?v=sNRqJOEKkCw">https://www.youtube.com/watch?v=sNRqJOEKkCw</a></p>https://developer.hpe.com/ll-mesh-democratizing-gen-ai-through-open-source-innovation-1/https://developer.hpe.com/ll-mesh-democratizing-gen-ai-through-open-source-innovation-1/Mon, 11 Nov 2024 07:58:18 GMT<style> li { font-size: 27px !important; line-height: 33px !important; max-width: none !important; } </style> <p>Generative AI (Gen AI) is revolutionizing technology, transforming industries, and creating unprecedented opportunities. At the heart of this revolution are Large Language Models (LLMs), sophisticated Gen AI models designed to understand and generate human-like text. Trained on vast datasets of billions of words, LLMs grasp the patterns and nuances of language, enabling them to perform tasks ranging from answering questions and summarizing text to generating creative content, translating languages, and even coding.</p> <p>By enabling machines to understand and generate human-like language—a significant leap from traditional data storage—LLMs uniquely contribute to digital transformation by not only storing information but also understanding its context and reasoning. However, these innovations bring significant challenges, including the complexity of adoption, ethical considerations, and the need for scalable, user-friendly solutions. Specifically, adopting Gen AI presents two main challenges:</p> <ol> <li><strong>Technical complexity</strong>: Gen AI tools are powerful but often require expertise in coding and machine learning, making it difficult for companies to use them effectively without specialized skills.</li> <li><strong>Organizational challenge</strong>s: Simply adding a Gen AI team isn't enough. The real value comes from leveraging the knowledge of your existing teams, especially those who may not be tech experts. If not managed properly, integrating Gen AI can impact team dynamics. It's crucial to find ways to use Gen AI that enhance collaboration and maximize everyone's expertise.</li> </ol> <h2>Embrace change: Leverage existing skills with Gen AI</h2> <p>To turn these disruptive challenges into constructive opportunities, we must embrace change. This recalls the story of Michelangelo and the Sistine Chapel. In 1508, although renowned as a sculptor, Michelangelo was commissioned to paint the Sistine Chapel—a monumental task outside his comfort zone. Initially reluctant and feeling inadequate, he faced immense pressure, including working at great heights and competing with Raffaello, the era's leading painter.</p> <p>However, Michelangelo adapted by applying his sculptural techniques to fresco painting, bringing a lifelike, three-dimensional quality to his work. This transformation highlights the power of leveraging existing skills to overcome new challenges, ultimately resulting in one of history's greatest masterpieces.</p> <p><img src="/img/vision_1.png" alt=""></p> <p>Inspired by Michelangelo's adaptation, Gen AI is as an opportunity to craft our own modern masterpieces. Just as he embraced change to overcome challenges, businesses can leverage Gen AI to catalyze this disruptive revolution, unlocking new competitive advantages by leveraging their existing skills and unique differentiators. Embracing Gen AI can unlock new potentials and drive innovation across all parts of the organization. It's more than a technical shift; it's a cultural transformation that requires embracing digital changes organization-wide.</p> <p>To support this digital transformation, HPE Athonet developed LLM Agentic Tool Mesh, an open-source platform. This pioneering initiative aims at democratizing Gen AI and tackling both the technical complexity and the organizational challenges. The vision is to make Gen AI accessible and beneficial to a broader audience, enabling users from various backgrounds to leverage cutting-edge Gen AI technology effortlessly. LLM Agentic Tool Mesh was demonstrated at the beginning of 2024, already demonstrating its potential at the Mobile World Congress in Barcelona, where it stood out for its innovative approach and significant impact on simplifying Gen AI adoption.</p> <p><img src="/img/athon_ux_tools.png" alt=""></p> <h2>Simplify complexity with LLM Agentic Tool Mesh</h2> <p>LLM Agentic Tool Mesh empowers users to create tools and web applications using Gen AI with low or no coding. This approach addresses technical challenges by simplifying the integration process.</p> <p>The Pareto principle, also known as the 80/20 rule, states that roughly 80% of outcomes come from 20% of causes. Leveraging this principle, LLM Agentic Tool Mesh focuses on the 20% of features that cover 80% of user needs. It achieves this by abstracting complex, low-level libraries into easy-to-understand, user-friendly configurations, effectively hiding the underlying complexity.</p> <p>For example, tasks that would traditionally require extensive coding and a deep understanding of machine learning models, such as implementing a chat feature with LLM or utilizing your own data through Retrieval-Augmented Generation (RAG), can now be accomplished through simple, descriptive configurations within LLM Agentic Tool Mesh. These features are particularly focused on delivering high value to the end user.</p> <p>This simplicity not only helps technical teams but also empowers non-technical teams to develop tools tailored to their specific domain of expertise, effectively addressing organizational challenges. For instance, marketing and sales teams can create custom AI assistants without coding, enhancing customer engagement and streamlining internal workflows.</p> <h2>Core principles of LLM Agentic Tool Mesh</h2> <p>The solution is built on three foundational principles, as described in previous blog posts:</p> <ol> <li><strong>Team-centric collaborative framework</strong>: it prioritizes collaborative efforts over individual achievements to enhance team synergy and productivity (see <a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-first-pillar-from-personal-assistant-to-collaborative-corporate-tool/">HPE Athonet LLM platform: From personal assistant to collaborative corporate tool</a>).</li> <li><strong>User experience</strong>: the interface is designed to minimize distractions and support users in achieving deep concentration, or "Flow," thereby enhancing focus and facilitating efficient workflows (see <a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-driving-users-towards-peak-flow-efficiency/">HPE Athonet LLM platform: Driving users toward peak flow efficiency</a>).</li> <li><strong>Versatile architecture</strong>: By incorporating Data Mesh and Microservices principles, the system is engineered for scalability and efficiency, supporting a wide range of operations from data management to service-oriented tasks (see <a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-divide-et-impera-designing-the-ll-mesh/">HPE Athonet LLM platform: Divide et impera, designing the LLM Agentic Tool Mesh</a>).</li> </ol> <p>LLM Agentic Tool Mesh realizes these principles through a robust self-service infrastructure, which empowers teams to independently develop, deploy, and manage their Gen AI-based web applications and tools. This simplicity enables both technical and non-technical teams to create tools tailored to their specific knowledge, fostering a <strong>decentralized ownership</strong> model that is fundamental to LLM Agentic Tool Mesh's philosophy and effectively addresses organizational challenges.</p> <p>Aligning with <strong>Conway's Law</strong>—which states that products mirror the communication structures of the organizations that create them—LLM Agentic Tool Mesh allows each department to leverage its unique knowledge and insights directly into tool development. This approach enhances efficiency and value by eliminating the need for separate teams dedicated solely to Gen AI products, which may overlook crucial domain-specific information.</p> <p><img src="/img/vision_2.png" alt=""></p> <p>Importantly, we're <strong>not reinventing the wheel</strong>. LLM Agentic Tool Mesh builds upon existing libraries to hide complexity, standardize interfaces, and convert coding logic into more descriptive configurations. This enables low-code or no-code development, making advanced Gen AI capabilities accessible to users without deep programming expertise. Each function in the LLM Agentic Tool Mesh library is implemented using the <strong>Factory Design Pattern</strong>, which abstracts functionality and standardizes interfaces through domain-specific, high-level functions.</p> <p>This design pattern allows for the creation of objects without specifying the exact class of the object to be created, ensuring consistency and usability across various tools and applications. For readers unfamiliar with design patterns, this method improves code maintainability and scalability, making it easier to manage complex systems as they grow. Unlike many open-source libraries that prioritize coding flexibility, our user-centric approach emphasizes <strong>ease of use and accessibility</strong>, focusing on enabling even non-technical users to create tools in a descriptive manner.</p> <p><img src="/img/vision_3.png" alt=""></p> <p>The platform enables the creation of a <strong>'Mesh'</strong> of these Gen AI tools, providing powerful orchestration capabilities through an agentic <strong>Reasoning Engine</strong> based on LLMs. A Reasoning Engine is an AI system that mimics human-like decision-making and problem-solving by applying rules, data, and logic. It emulates different types of reasoning, such as deductive, inductive, and abductive, allowing the LLM to not only generate content but also understand context, draw logical inferences, and connect information to solve complex problems.</p> <p>Additionally, each tool developed through LLM Agentic Tool Mesh can be deployed as a standalone web application with a manifest that clearly exposes its information and endpoints, allowing for seamless integration and orchestration across a cohesive network of services. This orchestration ensures that all tools work together seamlessly, enhancing overall functionality and efficiency across the organization.</p> <p><img src="/img/vision_4.png" alt=""></p> <p>This seamless integration is possible because, while the management of LLM tools operates in a decentralized manner, LLM Agentic Tool Mesh implements a unified framework of governance policies and standards to maintain consistency, ethical integrity, and overall quality across the platform. <strong>Federated Governance</strong> ensures a careful balance between fostering innovation and upholding rigorous standards for security, privacy, and compliance. This framework includes setting interoperable standards, enforcing ethical compliance, and utilizing automated governance tools to consistently apply policies. Continuous collaboration and feedback within LLM Agentic Tool Mesh are crucial for adapting and refining governance practices, promoting ongoing improvement and ensuring alignment with emerging technologies and challenges (these principles are inspired by Data Mesh architecture).</p> <h2>Join the LLM Agentic Tool Mesh Community</h2> <p>LLM Agentic Tool Mesh is now open source; you can download it from our <a href="https://github.com/HewlettPackard/llmesh">GitHub repository</a> and explore its <a href="https://github.com/HewlettPackard/llmesh/wiki">Wiki documentation</a>. In the coming weeks, we will publish additional blog posts providing more details about the services offered by LLM Agentic Tool Mesh, moving from theory to practical examples. We will also release a Workshop -on-Demand and host a Meetup to give you hands-on experience and share the latest updates and vision for the project.</p> <p>We invite you to join us on this journey to harness the transformative power of Gen AI. Let's collaborate to craft our own masterpieces and drive innovation forward. Engage with LLM Agentic Tool Mesh, contribute to the open-source project, and share your experiences with the community. Together, we can shape the future of Gen AI.</p>https://developer.hpe.com/7-questions-for-david-bader-graph-analytics-at-scale-with-arkouda-and-chapel/https://developer.hpe.com/7-questions-for-david-bader-graph-analytics-at-scale-with-arkouda-and-chapel/Thu, 07 Nov 2024 04:55:50 GMT<p>external blog</p>https://developer.hpe.com/morpheus-terraform-profiles/https://developer.hpe.com/morpheus-terraform-profiles/Wed, 06 Nov 2024 16:25:49 GMT<p>External blog</p>https://developer.hpe.com/morpheus-extensibility-with-plugins/https://developer.hpe.com/morpheus-extensibility-with-plugins/Wed, 06 Nov 2024 16:24:19 GMT<p>External blog</p>https://developer.hpe.com/managing-morpheus-plugins-via-rest-api/https://developer.hpe.com/managing-morpheus-plugins-via-rest-api/Wed, 06 Nov 2024 15:44:27 GMT<p>External blog</p>https://developer.hpe.com/2024-november-04/https://developer.hpe.com/2024-november-04/Mon, 04 Nov 2024 13:00:00 GMThttps://developer.hpe.com/implementing-your-ai-breakthroughs-effectively-–-the-infrastructure-to-your-a-i/https://developer.hpe.com/implementing-your-ai-breakthroughs-effectively-–-the-infrastructure-to-your-a-i/Thu, 31 Oct 2024 18:58:58 GMT<p>We’ve been here before. Think about the seemingly obscure and ever-evolving infrastructure technologies that have been introduced over the years that only few interact with, learn, and even see, but are always expected to work and are foundational to our digitalized world.</p> <p>In my early days as a sales engineer, one of my favorite opening lines with friends and family was “the cloud really is a place.” Just because we store our data out of sight and out of mind doesn’t mean it’s vanished from the planet. That data is stored on a well-thought-out infrastructure in a well-planned data center facility that makes it accessible anywhere. And just like “the cloud”, AI runs in a real location, on a fully-configured infrastructure, where it’s deployed and secured by people. Well, many people.</p> <p>So, you see, we really have been here before. In this new AI Jam series, we always aim to “keep it real with AI”. And despite the artificial and surreal vocabulary used today, AI workloads, just like the cloud, still need specialized hardware, software, and networking to make it a reality.</p> <p>Join us for our next talk about <a href="https://developer.hpe.com/campaign/get-real-with-ai-%E2%80%93-jam-series/">Implementing your AI Breakthrough Effectively - The infrastructure to you’re AI</a>. We’ll dive into the spectrum of use cases and infrastructure considerations that are challenging businesses today. As you know, different use cases and stages in a development cycle require different configurations. Infrastructure configurations and capacity for fine-tuning LLMs are not the same as what’s needed for inference, retrieval augmented generation (RAG), or even small language models.</p> <p>So how best to get started? Do you need access to large training clusters which could be very challenging for anybody to deploy? And what type of budget do you need, as the common statement you hear about AI projects is that it’s going to be expensive!</p> <p>Fortunately, this is not always the case. In fact, at HPE we help customers get started every day with AI by removing complexity. Large-sized clusters for training large language models are not the only entry point to enabling AI in your organization. If you are stuck between a use case and how to get started, this interactive session will help you navigate how operational improvements can help you get started today.</p> <p>We’ll also discuss how to show value to your organization today that will help enable a successful transformational AI adoption over time. So, Data Engineers, bring your IT Ops Manager, grab some coffee, and join us for this AI Jam session, where we discuss different AI infrastructure environments and software options to help get your use case started.</p>https://developer.hpe.com/navier-stokes-in-chapel-—-distributed-poisson-solver/https://developer.hpe.com/navier-stokes-in-chapel-—-distributed-poisson-solver/Mon, 28 Oct 2024 23:20:39 GMT<p>external blog</p>https://developer.hpe.com/7-questions-for-nelson-luís-dias-atmospheric-turbulence-in-chapel/https://developer.hpe.com/7-questions-for-nelson-luís-dias-atmospheric-turbulence-in-chapel/Tue, 15 Oct 2024 19:01:13 GMT<p>external blog</p>https://developer.hpe.com/distributed-tuning-in-chapel-with-a-hyperparameter-optimization-example/https://developer.hpe.com/distributed-tuning-in-chapel-with-a-hyperparameter-optimization-example/Wed, 09 Oct 2024 06:28:40 GMT<p>external blog</p>https://developer.hpe.com/small-language-models-the-next-frontier-with-private-cloud-ai/https://developer.hpe.com/small-language-models-the-next-frontier-with-private-cloud-ai/Thu, 03 Oct 2024 17:13:41 GMT<p>Text</p>https://developer.hpe.com/7-questions-for-scott-bachman-analyzing-coral-reefs-with-chapel/https://developer.hpe.com/7-questions-for-scott-bachman-analyzing-coral-reefs-with-chapel/Tue, 01 Oct 2024 19:15:34 GMT<p>external blog</p>https://developer.hpe.com/2024-october-01/https://developer.hpe.com/2024-october-01/Tue, 01 Oct 2024 13:00:00 GMThttps://developer.hpe.com/introducing-the-ampere®-performance-toolkit-to-optimize-software/https://developer.hpe.com/introducing-the-ampere®-performance-toolkit-to-optimize-software/Fri, 27 Sep 2024 12:05:40 GMT<p>External Blog</p>https://developer.hpe.com/announcing-chapel-2-2/https://developer.hpe.com/announcing-chapel-2-2/Thu, 26 Sep 2024 21:05:26 GMT<p>external blog</p>https://developer.hpe.com/keeping-it-real-with-a-new-ai-jam-series/https://developer.hpe.com/keeping-it-real-with-a-new-ai-jam-series/Thu, 26 Sep 2024 13:54:21 GMT<p><img src="/img/hpe_story_548_1600_0_72_rgb.jpg" alt=""></p> <p>It wasn’t that long ago we used to say “Artificial Intelligence". Now we just say A.I. These words are so pervasive in conversation that we are now down to the acronym. But what I have been hearing is that companies are completely scattered on the topic. Some are launching experiments on the talent and technologies they have been maturing for years; others are scrambling to get in the game. Despite this, everyone is asking "how do I get started" or "how do I move quickly" right now. That’s why the HPE Developer Community is sponsoring a new series of Skill Up meetings called “<a href="https://developer.hpe.com/campaign/get-real-with-ai-jam-series/">Get real with AI</a>” to help you. We want to get past the hype and bring you technical and business value that meets you where you are today. So, let’s just say “keep calm and get real with AI.”</p> <h2>Responding to requests for more help</h2> <p>Earlier this year, we held a workshop that focused on the basics companies need to successfully host and deploy AI projects. The success of this one workshop resulted in a clamor for more sessions on this same topic. We quickly came to realize that we needed a new series of interactive sessions that were a lot more basic in nature. And many of the subject matter experts we spoke with wanted to share more of their ideas and experiences about where and how to get started.</p> <p>This resulted in the premier of our first AI Jam session titled Exploring Transformative AI Use Cases Across Industries. The discussion brought together thought leaders, industry experts, and technologists who talked through transformative AI use cases across diverse sectors, including healthcare, finance, manufacturing, and retail.</p> <p>It turned out to be a very interactive and useful session with panelists sharing their insights on the critical factors for AI adoption, including data management, ethical considerations, and integration with existing systems. The discussion also highlighted emerging trends, such as the use of AI for predictive analytics, personalization, automation, and decision-making. If you missed this session you can catch the <a href="https://developer.hpe.com/campaign/get-real-with-ai-jam-series/">replay here</a>.</p> <h2>Keeping it real in our next session</h2> <p>It was a good start. Going forward, we decided it was best to continue keeping it well aligned and focused on “keeping it real with AI” basics. To do that, we needed to change up the next session to focus on – “Cool AI Use Case, Now What?”.</p> <p>I hear a lot about people coming up with great ideas where AI technologies could help their business. But as quickly as an idea is floated, all eyes go towards the technologist, the data scientists, and the IT operations teams, and leadership comes back saying “sounds good, now show me something.” At that point it can get complicated, as many people don’t know where to get started or what’s really needed to show success. That’s what we’re going to focus on in the next AI jam session. We’ll be discussing best practices on getting started with AI and illustrate shortcuts on how to get AI projects from data experiments to AI deployments. If this is something you might be interested in, don’t forget to register for the October 2nd AI jam session at 10.00 (CST).</p> <p>And yes, there’s more to come, so please be sure to check in on the new upcoming <a href="https://developer.hpe.com/campaign/get-real-with-ai-jam-series/">AI jam series</a> that we will be running in November and December. See you there!</p>https://developer.hpe.com/7-questions-for-éric-laurendeau-computing-aircraft-aerodynamics-in-chapel/https://developer.hpe.com/7-questions-for-éric-laurendeau-computing-aircraft-aerodynamics-in-chapel/Tue, 17 Sep 2024 20:44:07 GMT<p>external blog</p>https://developer.hpe.com/finding-the-best-lora-parameters/https://developer.hpe.com/finding-the-best-lora-parameters/Wed, 11 Sep 2024 18:44:16 GMT<p>External blog post</p>https://developer.hpe.com/2024-september-05/https://developer.hpe.com/2024-september-05/Thu, 05 Sep 2024 13:00:00 GMThttps://developer.hpe.com/what’s-new-with-chapel-nine-questions-for-the-development-team/https://developer.hpe.com/what’s-new-with-chapel-nine-questions-for-the-development-team/Wed, 04 Sep 2024 16:54:10 GMT<p>external blog</p>https://developer.hpe.com/setting-up-hierarchical-namespaces-in-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/setting-up-hierarchical-namespaces-in-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/Thu, 22 Aug 2024 16:16:41 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><strong>Multi-tenancy</strong> for Kubernetes (K8s) cluster requires sophisticated namespace management to enable robust tenant isolation and organization. <strong>Hierarchical namespaces</strong> allow you to model namespaces according to your own organizational hierarchy and allocate capabilities inside a single K8s cluster. This eliminates the need for a new cluster for each organizational unit. The utilization of hierarchical namespaces can result in a more streamlined namespace management and improved security in the complex K8s production environment.</p> <p>This blog post provides a step-by-step guide on how to set up hierarchical namespaces in K8s in HPE GreenLake for Private Cloud Enterprise. The simplicity of handling relationships between K8s namespaces, propagating configurations and resource constraints, and applying access control policies using hierarchical namespaces in K8s is demonstrated here.</p> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>A K8s <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/"><em>namespace</em></a> is a fundamental API object within the K8s architecture. It provides a mechanism for grouping and isolating K8s resources within a single cluster. Each namespace has its own set of resources, policies and constraints. Cluster administrators can create roles using role-based access control (RBAC) to define permissions within a specific namespace. They can aslo enforce resource limits and quotas to control the consumption of resources by objects in a namespace. K8s namespaces facilitate the separation of resources and allow multiple users, teams, or projects to share a cluster within an organization. This approach is more cost-effective than creating a new cluster for each organizational unit. With the application of various K8s configurations and policies to namespaces, cluster administrators can ensure safe and fair resource isolation and cluster sharing.</p> <p>K8s namespaces created within a cluster are peers, each fully isolated from the others. K8s namespaces do not align well with organizational structures. Cluster administrators must define roles and create policies and constraints for each individual namespace. At scale, managing numerous namespaces can become challenging, leading to potential management issues. The administrative overhead can make managing namespaces within the cluster tedious and prone to errors.</p> <p>In 2020, K8s upstream introduced a K8s extension known as the <a href="https://github.com/kubernetes-sigs/hierarchical-namespaces#the-hierarchical-namespace-controller-hnc"><em>Hierarchical Namespace Controller</em> (HNC)</a>. HNC supports hierarchical namespaces and helps cluster administrators manage the security and capabilities of namespaces with less effort than the flat, peer-to-peer namespace model. Using HNC, administrators can organize namespaces according to an organizational hierarchy and allocate capabilities using a list of K8s resources, such as <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole"><em>Role</em></a>/<a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding"><em>RoleBinding</em></a>, <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/"><em>ResourceQuota</em></a> and <a href="https://kubernetes.io/docs/concepts/configuration/secret/"><em>Secret</em></a>.</p> <h3>Prerequisites</h3> <ul> <li>A K8s cluster provisioned using <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">HPE GreenLake Terraform provider</a> for example, in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em> CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li><a href="https://curl.se/">cURL</a> CLI tool</li> </ul> <h3>Set up hierarchical namespaces</h3> <p>K8s does not come with hierarchical namespace support by default. There are two components, the <em>HNC manager</em> and the optional kubectl plugin <em>kubectl-hns</em> that must be installed to support hierarchical namespaces in K8s.</p> <h4>Install the HNC manager</h4> <p>Type the following commands to install the HNC manager within the control plane of the K8s cluster:</p> <pre><code class="language-shell">$ HNC_VERSION=v1.1.0 $ HNC_VARIANT=default $ kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/${HNC_VERSION}/${HNC_VARIANT}.yaml namespace/hnc-system created customresourcedefinition.apiextensions.k8s.io/hierarchicalresourcequotas.hnc.x-k8s.io created customresourcedefinition.apiextensions.k8s.io/hierarchyconfigurations.hnc.x-k8s.io created customresourcedefinition.apiextensions.k8s.io/hncconfigurations.hnc.x-k8s.io created customresourcedefinition.apiextensions.k8s.io/subnamespaceanchors.hnc.x-k8s.io created role.rbac.authorization.k8s.io/hnc-leader-election-role created clusterrole.rbac.authorization.k8s.io/hnc-admin-role created clusterrole.rbac.authorization.k8s.io/hnc-manager-role created rolebinding.rbac.authorization.k8s.io/hnc-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/hnc-manager-rolebinding created secret/hnc-webhook-server-cert created service/hnc-controller-manager-metrics-service created service/hnc-webhook-service created deployment.apps/hnc-controller-manager created mutatingwebhookconfiguration.admissionregistration.k8s.io/hnc-mutating-webhook-configuration created validatingwebhookconfiguration.admissionregistration.k8s.io/hnc-validating-webhook-configuration created </code></pre> <p>The above commands install the latest <a href="https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/tag/v1.1.0">HNC v1.1.0</a> to the namespace <em>'hnc-system'</em> in the cluster.</p> <p>Type the following command to check the HNC manager installation:</p> <pre><code class="language-shell">$ kubectl get all -n hnc-system NAME READY STATUS RESTARTS AGE pod/hnc-controller-manager-9b5dbcd48-2268c 1/1 Running 0 29s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/hnc-controller-manager-metrics-service ClusterIP 10.96.255.60 &#x3C;none> 8080/TCP 32s service/hnc-webhook-service ClusterIP 10.96.73.3 &#x3C;none> 443/TCP 32s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hnc-controller-manager 1/1 1 1 31s NAME DESIRED CURRENT READY AGE replicaset.apps/hnc-controller-manager-9b5dbcd48 1 1 1 32s </code></pre> <h4>Install the <em>kubectl-hns</em> plugin</h4> <p>After installing the HNC manager, you can set up hierarchical namespaces directly using the kubectl CLI tool, in conjunction with a list of HNC custom resource definitions (CRDs), such as <em>HierarchyConfiguration</em> and <em>SubnamespaceAnchor</em>. You can install the kubectl plugin <em>kubectl-hns</em> in your client environment. This kubectl plugin works together with the kubectl CLI tool and simplifies many hierarchical namespace operations. This section shows the process to install the <em>kubectl-hns</em> plugin to Linux workstation in a local environment.</p> <p>Type the following commands to install the <em>kubectl-hns</em> plugin using <em>cURL</em>:</p> <pre><code class="language-shell"> $ HNC_VERSION=v1.1.0 $ HNC_PLATFORM=linux_amd64 $ curl -L https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/$HNC_VERSION/kubectl-hns_$HNC_PLATFORM -o ./kubectl-hns % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 62.2M 100 62.2M 0 0 2340k 0 0:00:27 0:00:27 --:--:-- 2336k $ chmod +x kubectl-hns $ sudo mv kubectl-hns /usr/local/bin/kubectl-hns </code></pre> <p>Type the following command to verify the <em>kubectl-hns</em> plugin is installed correctly and works with kubectl:</p> <pre><code class="language-shell"> $ kubectl hns Manipulates hierarchical namespaces provided by HNC Usage: kubectl-hns [command] Available Commands: completion generate the autocompletion script for the specified shell config Manipulates the HNC configuration create Creates a subnamespace under the given parent. describe Displays information about the hierarchy configuration help Help about any command hrq Display one or more HierarchicalResourceQuota set Sets hierarchical properties of the given namespace tree Display one or more hierarchy trees version Show version of HNC plugin Flags: --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --as-uid string UID to impersonate for the operation. --cache-dir string Default cache directory (default "/home/guoping/.kube/cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for kubectl-hns --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") -s, --server string The address and port of the Kubernetes API server --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use -v, --version version for kubectl-hns Use "kubectl-hns [command] --help" for more information about a command. </code></pre> <h3>Create hierarchical namespaces</h3> <p>After installing both the HNC manager and the <em>kubectl-hns</em> plugin, you can start creating hierarchical namespaces. This section sets up an imaginary hierarchical namespace structure, in which an organization, named <strong>cfe-pce</strong>, consists of two teams, <em>team-caas</em> &#x26; <em>team-vmaas</em>, each team runs its <em>devops</em> and <em>iac</em> projects:</p> <pre><code class="language-shell">cfe-pce: - team-caas - caas-devops - caas-iac - team-vmaas - vmaas-devops - vmaas-iac </code></pre> <p>A hierarchical namespace can be created using the following command:</p> <pre><code class="language-shell"> kubectl hns create &#x3C;ns_child> -n &#x3C;ns_parent> </code></pre> <p>where, <em>&#x3C;ns_child></em> is the child namespace you want to create, while <em>&#x3C;ns_parent></em> is the parent namespace that is created using the command <em>kubectl create ns &#x3C;ns_parent></em>.</p> <p>In case both <em>&#x3C;ns_child></em> and <em>&#x3C;ns_parent></em> already exist, you can still create a hierarchical structure by running below command:</p> <pre><code class="language-shell"> kubectl hns set &#x3C;ns_child> --parent &#x3C;ns_parent> </code></pre> <p>Type the command below to create the root namespace <em>'cfe-pce'</em> representing the organization:</p> <pre><code class="language-shell"> $ kubectl create ns cfe-pce namespace/cfe-pce created </code></pre> <p>Then run the following commands to create the subnamespaces under the organization:</p> <pre><code class="language-shell">$ kubectl hns create team-vmaas -n cfe-pce Successfully created "team-vmaas" subnamespace anchor in "cfe-pce" namespace $ kubectl hns create vmaas-devops -n team-vmaas Successfully created "vmaas-devops" subnamespace anchor in "team-vmaas" namespace $ kubectl hns create vmaas-iac -n team-vmaas Successfully created "vmaas-iac" subnamespace anchor in "team-vmaas" namespace $ kubectl hns create team-caas -n cfe-pce Successfully created "team-caas" subnamespace anchor in "cfe-pce" namespace $ kubectl hns create caas-devops -n team-caas Successfully created "caas-devops" subnamespace anchor in "team-caas" namespace $ kubectl hns create caas-iac -n team-caas Successfully created "caas-iac" subnamespace anchor in "team-caas" namespace </code></pre> <p>Type the command <em>'kubectl hns tree'</em> to check the hierarchical namespaces:</p> <pre><code class="language-shell">$ kubectl hns tree cfe-pce cfe-pce ├── team-caas │ ├── caas-devops │ └── caas-iac └── team-vmaas ├── vmaas-devops └── vmaas-iac </code></pre> <h3>Create resources and apply propagating capabilities</h3> <p>After creating the hierarchical namespace structure, you can add roles and rolebindings, using RBAC, to enforce access control within namespaces. You can set up resource quotas and secrets to ensure safe cluster sharing. Hierarchical namespaces allow for propagation of configurations and resources, enforcing access control policies across namespaces.</p> <p>You can refer to the <a href="https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/master/docs/user-guide/README.md">HNC user guide</a> for setting up other K8s resources using HNC.</p> <h4>Cascade roles and rolebindings</h4> <p><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC</a> is commonly used in K8s to limit access to the appropirate namespaces. You must ensure that each user or workload has the correct access to only their designated namespaces. K8s <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole"><em>Role</em></a> and <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding"><em>RoleBinding</em></a> are two API objects used at the namespace level to enforce access control.</p> <p>Type the following commands to create an admin role <em>'pce-admin'</em> across the whole organization <em>cfe-pce</em> and two site reliability engineer (SRE) roles, <em>'vmaas-sre'</em> and <em>'caas-sre'</em>, for <em>team-vmaas</em> and <em>team-caas</em>:</p> <pre><code class="language-shell"> $ kubectl -n cfe-pce create role pce-admin --verb=* --resource=pod role.rbac.authorization.k8s.io/pce-admin created $ kubectl -n team-vmaas create role vmaas-sre --verb=update --resource=pod role.rbac.authorization.k8s.io/vmaas-sre created $ kubectl -n team-caas create role caas-sre --verb=update --resource=pod role.rbac.authorization.k8s.io/caas-sre created </code></pre> <p>Type the following commands to create the rolebindings:</p> <pre><code class="language-shell"> $ kubectl -n cfe-pce create rolebinding pce-admins --role pce-admin --serviceaccount=cfe-pce:default rolebinding.rbac.authorization.k8s.io/pce-admins created $ kubectl -n team-caas create rolebinding caas-sres --role caas-sre --serviceaccount=team-caas:default rolebinding.rbac.authorization.k8s.io/caas-sres created $ kubectl -n team-vmaas create rolebinding vmaas-sres --role vmaas-sre --serviceaccount=team-vmaas:default rolebinding.rbac.authorization.k8s.io/vmaas-sres created </code></pre> <p>Creating roles and rolebindings in the parent namespaces are propagated to all associated subnamespaces at team and project levels:</p> <pre><code class="language-shell">$ kubectl get role -n cfe-pce pce-admin NAME CREATED AT pce-admin 2024-06-27T12:45:51Z $ kubectl get rolebindings -n cfe-pce pce-admins NAME ROLE AGE pce-admins Role/pce-admin 44s $ kubectl get role -n team-caas NAME CREATED AT ... caas-sre 2024-06-27T12:45:53Z pce-admin 2024-06-27T12:45:51Z $ kubectl get rolebindings -n team-caas NAME ROLE AGE ... caas-sres Role/caas-sre 44s pce-admins Role/pce-admin 63s $ kubectl get role -n caas-devops NAME CREATED AT ... caas-sre 2024-06-27T12:45:53Z pce-admin 2024-06-27T12:45:51Z $ kubectl get rolebindings -n caas-devops NAME ROLE AGE ... caas-sres Role/caas-sre 53s pce-admins Role/pce-admin 63s $ kubectl get role -n caas-iac NAME CREATED AT ... caas-sre 2024-07-29T12:10:59Z pce-admin 2024-08-08T08:23:42Z $ kubectl get rolebindings -n caas-iac NAME ROLE AGE ... caas-sres Role/caas-sre 53s pce-admins Role/pce-admin 63s $ kubectl get role -n team-vmaas NAME CREATED AT ... pce-admin 2024-06-27T12:45:51Z vmaas-sre 2024-06-27T12:45:52Z $ kubectl get rolebindings -n team-vmaas NAME ROLE AGE ... pce-admins Role/pce-admin 63s vmaas-sres Role/vmaas-sre 43s $ kubectl get role -n vmaas-devops NAME CREATED AT ... pce-admin 2024-08-03T14:08:45Z vmaas-sre 2024-07-29T12:10:59Z $ kubectl get rolebindings -n vmaas-devops NAME ROLE AGE pce-admins Role/pce-admin 63s vmaas-sres Role/vmaas-sre 43s $ kubectl get role -n vmaas-iac NAME CREATED AT ... pce-admin 2024-06-27T12:45:51Z vmaas-sre 2024-06-27T12:45:52Z $ k get rolebindings -n vmaas-iac NAME ROLE AGE ... pce-admins Role/pce-admin 65s vmaas-sres Role/vmaas-sre 45s </code></pre> <h4>Cascade resource quotas</h4> <p>K8s provides <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/">ResourceQuota</a>, an API object that allows the cluster administrators to define resource quotas and limit ranges per namespace. Resource quota tracks aggregate usage of resources in the namespace and allow cluster operators to define hard resource usage limits that a namespace may consume. A limit range defines minimum and maximum constraints on the amount of resources a single entity can consume in a namespace. It's useful to make sure resource usage is staying with certain bounds. This section shows the process to set up resource quotas using <em>ResourceQuota</em>.</p> <p>Type the following commands to apply two resource quotas, <em>'team-vmaas-quota'</em> &#x26; <em>'team-caas-quota'</em>, to the team namespaces <em>team-vmaas</em> and <em>team-caas</em>, respectively:</p> <pre><code class="language-shell"> $ cat team-vmaas-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: team-vmaas-quota spec: hard: cpu: "4" memory: 20Gi pods: "10" persistentvolumeclaims: "10" services.nodeports: "2" requests.storage: 20Gi $ kubectl -n team-vmaas apply -f team-vmaas-quota.yaml resourcequota/team-vmaas-quota created $ cat team-caas-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: team-caas-quota spec: hard: cpu: "8" memory: 60Gi pods: "30" persistentvolumeclaims: "30" services.nodeports: "6" requests.storage: 60Gi $ kubectl -n team-caas apply -f team-caas-quota.yaml resourcequota/team-caas-quota created </code></pre> <p>If you check the resouce quotas using the following commands, you will find they are only created in each team namespace. The quota resources are not propagated to any projects under the team namespaces:</p> <pre><code class="language-shell">$ kubectl get resourcequota -n team-caas NAME AGE REQUEST LIMIT team-caas-quota 67s cpu: 0/8, memory: 0/60Gi, persistentvolumeclaims: 0/30, pods: 0/30, requests.storage: 0/60Gi, services.nodeports: 0/6 $ kubectl get resourcequota -n team-vmaas NAME AGE REQUEST LIMIT team-vmaas-quota 74s cpu: 0/4, memory: 0/20Gi, persistentvolumeclaims: 0/10, pods: 0/10, requests.storage: 0/20Gi, services.nodeports: 0/2 $ kubectl get resourcequota -n vmaas-iac No resources found in vmaas-iac namespace. $ kubectl get resourcequota -n caas-iac No resources found in caas-iac namespace. $ kubectl get resourcequota -n caas-devops No resources found in caas-devops namespace. $ kubectl get resourcequota -n vmaas-devops No resources found in vmaas-devops namespace. </code></pre> <p>The lack of propagation of resource quotas to project namespaces is due to the default HNC configuration. The HNC configuration propagates RBAC objects, specifically, <em>roles</em> and <em>rolebindings</em>, by default. You have to update the HNC configuration to propagate other K8s resources.</p> <pre><code class="language-shell"> $ kubectl hns config describe Synchronized resources: * Propagating: rolebindings (rbac.authorization.k8s.io/v1) * Propagating: roles (rbac.authorization.k8s.io/v1) Conditions: </code></pre> <p>Type the following command to update the HNC configuration to propagate the K8s <em>ResourceQuota</em> resources:</p> <pre><code class="language-shell"> $ kubectl hns config set-resource resourcequotas --mode Propagate $ kubectl hns config describe Synchronized resources: * Propagating: resourcequotas (/v1) * Propagating: rolebindings (rbac.authorization.k8s.io/v1) * Propagating: roles (rbac.authorization.k8s.io/v1) Conditions: </code></pre> <p>If you check the resouce quotas again, you can notice the quota resources propagated to all projects under each team namespace:</p> <pre><code class="language-shell">$ kubectl get resourcequota -n team-caas NAME AGE REQUEST LIMIT team-caas-quota 3m39s cpu: 0/8, memory: 0/60Gi, persistentvolumeclaims: 0/30, pods: 0/30, requests.storage: 0/60Gi, services.nodeports: 0/6 $ kubectl get resourcequota -n caas-iac NAME AGE REQUEST LIMIT team-caas-quota 45s cpu: 0/8, memory: 0/60Gi, persistentvolumeclaims: 0/30, pods: 0/30, requests.storage: 0/60Gi, services.nodeports: 0/6 $ kubectl get resourcequota -n caas-devops NAME AGE REQUEST LIMIT team-caas-quota 60s cpu: 0/8, memory: 0/60Gi, persistentvolumeclaims: 0/30, pods: 0/30, requests.storage: 0/60Gi, services.nodeports: 0/6 $ kubectl get resourcequota -n team-vmaas NAME AGE REQUEST LIMIT team-vmaas-quota 4m10s cpu: 0/4, memory: 0/20Gi, persistentvolumeclaims: 0/10, pods: 0/10, requests.storage: 0/20Gi, services.nodeports: 0/2 $ kubectl get resourcequota -n vmaas-iac NAME AGE REQUEST LIMIT team-vmaas-quota 75s cpu: 0/4, memory: 0/20Gi, persistentvolumeclaims: 0/10, pods: 0/10, requests.storage: 0/20Gi, services.nodeports: 0/2 $ kubectl get resourcequota -n vmaas-devops NAME AGE REQUEST LIMIT team-vmaas-quota 79s cpu: 0/4, memory: 0/20Gi, persistentvolumeclaims: 0/10, pods: 0/10, requests.storage: 0/20Gi, services.nodeports: 0/2 </code></pre> <h4>Cascade secrets</h4> <p>This section demonstrates the process to configure sensitive data to be propagated through the namespace hierarchy. It uses the K8s <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Secret</a> API object.</p> <p>First, type the following command to update the HNC configuration to propagate the K8s <em>Secret</em> resources:</p> <pre><code class="language-shell">$ kubectl hns config set-resource secrets --mode Propagate $ kubectl hns config describe Synchronized resources: * Propagating: resourcequotas (/v1) * Propagating: secrets (/v1) * Propagating: rolebindings (rbac.authorization.k8s.io/v1) * Propagating: roles (rbac.authorization.k8s.io/v1) Conditions: </code></pre> <p>Then execute the following commands to generate two K8s secrets of the <em>docker-registry</em> type, <em>'team-caas-regcrd'</em> &#x26; <em>'team-vmaas-regcrd'</em>, within the <em>team-caas</em> and <em>team-vmaas</em> namespaces, respectively. These secrets can be used by the corresponding teams and their projects, enabling them to authenticate against their respective team Docker registries and retrieve private images necessary for the deployment of their applications.</p> <pre><code class="language-shell"> $ kubectl -n team-caas create secret generic team-caas-regcrd --from-file=.dockerconfigjson=/home/guoping/.docker/config-team-caas.json --type=kubernetes.io/dockerconfigjson secret/team-caas-regcrd created $ kubectl -n team-vmaas create secret generic team-vmaas-regcrd --from-file=.dockerconfigjson=/home/guoping/.docker/config-team-vmaas.json --type=kubernetes.io/dockerconfigjson secret/team-vmaas-regcrd created </code></pre> <p>The secrets created within the team namespaces are automatically propagated to every project within each team namespace:</p> <pre><code class="language-shell"> $ kubectl get secret -n team-caas team-caas-regcrd NAME TYPE DATA AGE team-caas-regcrd kubernetes.io/dockerconfigjson 1 49s $ kubectl get secret -n caas-devops team-caas-regcrd NAME TYPE DATA AGE team-caas-regcrd kubernetes.io/dockerconfigjson 1 99s $ kubectl get secret -n caas-iac team-caas-regcrd NAME TYPE DATA AGE team-caas-regcrd kubernetes.io/dockerconfigjson 1 104s $ kubectl get secret -n team-vmaas team-vmaas-regcrd NAME TYPE DATA AGE team-vmaas-regcrd kubernetes.io/dockerconfigjson 1 40s $ kubectl get secret -n vmaas-devops team-vmaas-regcrd NAME TYPE DATA AGE team-vmaas-regcrd kubernetes.io/dockerconfigjson 1 59s $ kubectl get secret -n vmaas-iac team-vmaas-regcrd NAME TYPE DATA AGE team-vmaas-regcrd kubernetes.io/dockerconfigjson 1 65s </code></pre> <h3>Conclusion</h3> <p>This blog post provides a detailed guide on how to set up hierarchical namespaces in K8s in the HPE GreenLake for Private Cloud Enterprise. It delves into the capabilities of hierarchical namespaces and their impact on managing K8s namespaces. The support of hierarchical namespaces simplifies K8s management and addresses the complexities of administering large-scale namespaces.</p> <p>Keep visiting the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/2024-august-08/https://developer.hpe.com/2024-august-08/Thu, 08 Aug 2024 13:00:00 GMThttps://developer.hpe.com/simplifying-code-migration-the-benefits-of-the-new-ampere-porting-advisor-for-x86-to-arm64/https://developer.hpe.com/simplifying-code-migration-the-benefits-of-the-new-ampere-porting-advisor-for-x86-to-arm64/Mon, 05 Aug 2024 12:34:13 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>The demand for efficient software porting solutions is increasing. With the transition from legacy x86 to AArch64, and with Ampere processors gaining adoption momentum, developers are looking for ways to expedite the migration for existing codebases. Built for sustainable cloud computing, Ampere’s Cloud Native Processors (CNPs) deliver predictable high performance, platform scalability, and power efficiency unprecedented in the industry.</p> <p>Today, we are announcing the Ampere Porting Advisor (APA), a new open-source software porting advisor via our <a href="https://github.com/AmpereComputing/ampere-porting-advisor">GitHub</a> page promising to simplify this process. The introduction of the new Ampere Porting Advisor provides a significant advancement in simplifying the migration of x86 code to the AArch64 architecture. By streamlining the migration process, reducing development costs, and enabling access to a wider ecosystem, the advisor empowers developers to embrace the benefits of the AArch64 architecture more quickly and effectively.</p> <p>The AArch64 architecture has gained significant traction across various software packages. By leveraging the APA, developers can tap into this expanding ecosystem and take advantage of the benefits offered by AArch64-based platforms. Migrating code from x86 to the AArch64 architecture does not have to be an intimidating process.</p> <p>The Ampere Porting Advisor tool:</p> <ul> <li>Reduces time-to-market by minimizing the need for manual intervention. Developers can allocate their time and resources to other critical aspects of the project.</li> <li>Reduces development costs by automating various tasks involved in the migration.</li> <li>Minimizes risk of post-migration issues by eliminating the need for costly debugging and re-work.</li> </ul> <p>The Ampere Porting Advisor offers a streamlined migration process, allowing developers to save time and effort. It automates many of the manual steps involved in porting code, reducing the risk of errors, and ensuring consistency throughout the migration. The APA performs a comprehensive analysis and output recommendations of the source code, providing detailed insights into changes required, highlighting potential pitfalls, and recommending optimal modifications. This guidance enables developers to navigate the intricacies of transitioning between architectures more efficiently, accelerating the overall migration process.</p> <p>The APA includes the following features:</p> <ul> <li><strong>Language support:</strong> Python 3+, Java 8+, Go 1.11+, C, C++, Fortran</li> <li><strong>Architecture specific code detection:</strong> i.e. missing corresponding AAarch64 assembly, architecture specific instructions, architecture specific flags in make files.</li> <li><strong>Dependency checks:</strong> for versioning, JAR scanning, and dependency files.</li> <li><strong>Simple to run:</strong> via python script, binary, or containers.</li> <li><strong>Multiple output formats:</strong> terminal for quick checks, html for easy distribution, and CSV for post-processing.</li> </ul> <p>The Ampere Porting Advisor will not make code modifications, nor API level recommendations, and it will not send data back to Ampere. The APA is a static command line tool that analyzes the make environment and source code for known code patterns and dependency libraries and generates a report with incompatibilities and recommendations.</p> <p>Download and try the Ampere Porting Advisor from <a href="https://github.com/AmpereComputing/ampere-porting-advisor">Ampere’s GitHub</a>.</p> <p>To learn more about the developer efforts, find best practices, insights, you are invited to join the conversation at: <a href="https://developer.amperecomputing.com">https://developer.amperecomputing.com</a>, and <a href="https://community.amperecomputing.com/">https://community.amperecomputing.com/</a>.</p>https://developer.hpe.com/greenlake-api-how-to-generate-aceess-token-in-python/https://developer.hpe.com/greenlake-api-how-to-generate-aceess-token-in-python/Fri, 26 Jul 2024 02:57:15 GMT<p>Accessing and organizing audit logs is crucial for keeping your system secure, meeting compliance requirements, and running operations smoothly. HPE GreenLake’s audit logs are essential for tracking activities and spotting any unusual behavior that might indicate a security threat.</p> <p>In this guide, we'll show you how to set up a Python script to automatically fetch and process HPE GreenLake audit logs. This will save you time and make it easier to keep an eye on your system.</p> <p>If you’re new to HPE GreenLake’s APIs, take a look at this <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">blog post</a> to get started with the HPE GreenLake platform API Audit Logs.</p> <h2>Use cases for fetching audit logs</h2> <p>Audit logs are extremely useful in several scenarios:</p> <ul> <li><strong>Security Monitoring</strong>: Keeping an eye on who accessed the system and what they did helps in spotting unusual activities that might be security threats.</li> <li><strong>Compliance Auditing</strong>: Many industries have rules about keeping records of system activities. Audit logs are perfect for this and can be shown as proof during audits.</li> <li><strong>Operational Analysis</strong>: By looking at the audit logs, companies can see patterns in how their systems are used, helping them to make better decisions about resources and planning.</li> </ul> <h2>Detailed steps to set up the automation</h2> <p>Here’s a step-by-step guide to getting your audit logs automatically using Python:</p> <h3>Step 1: Set up OAuth2 authentication</h3> <p>HPE GreenLake uses OAuth2 for security, which means you first need to get an access token.</p> <ol> <li><strong>Store Your Credentials</strong>: Keep your client ID and secret safe but accessible to your script. A good practice is to store them in a text file. In this case, I did save the client_Id in first line and client_secret in second line.</li> <li><strong>Request a Token</strong>: Your script needs to read these credentials and use them to request an access token. This token proves to HPE GreenLake that your script has permission to access the logs.</li> </ol> <pre><code class="language-python">def generate_and_save_token(): with open('credentials.txt', 'r') as file: client_id, client_secret = [line.strip() for line in file.readlines()] client = BackendApplicationClient(client_id=client_id) oauth = OAuth2Session(client=client) token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=(client_id, client_secret)) return token['access_token'] </code></pre> <h3>Step 2: Fetch the audit logs</h3> <p>Now that you have a token, you can fetch the logs from HPE GreenLake.</p> <ul> <li><strong>Make the Request</strong>: Use the token to make a secure request to the audit logs endpoint. Handle any errors and ensure you process the logs only if the fetch was successful.</li> </ul> <pre><code class="language-python">def fetch_audit_logs(token): headers = {'Authorization': f'Bearer {token}'} response = requests.get('https://global.api.greenlake.hpe.com/audit-log/v1beta1/logs', headers=headers) return response.json() if response.status_code == 200 else None </code></pre> <h3>Step 3: Process and save the data</h3> <p>After getting the logs, you'll want to convert them into a format that’s easy to read and use.</p> <ul> <li><strong>Process the JSON</strong>: Convert the JSON data into a pandas DataFrame. This makes it easier to work with in Python.</li> <li><strong>Save to Excel</strong>: Export this DataFrame to an Excel file. This file will be your audit log record that you can open, share, and analyze without needing specialised software.</li> </ul> <h2>What does it look like in excel?</h2> <p><img src="/img/excel-output.png" alt=""></p> <p>I have used excel in this script just for testing and can be used with any of the databases including sqlite, postgres and oracle. Python can also do it, but the results vary depending on the use case, especially when it comes to real-time data monitoring or handling very large volumes of logs.</p> <p>For these scenarios, other tools and platforms can be more suitable:</p> <ul> <li><strong>Splunk</strong>: This powerful tool specializes in searching, monitoring, and analyzing machine-generated data via a Web-style interface.</li> <li><strong>ELK Stack (Elasticsearch, Logstash, Kibana)</strong>: This group of three open-source tools works together to allow users to collect logs from different sources, process them, and then visualize and analyze these data in real time. Elasticsearch acts as a search and analytics engine, Logstash processes and prepares your data, while Kibana lets you visualize the data with charts and graphs.</li> <li><strong>Tableau</strong>: Known for its advanced visualization capabilities, Tableau can connect directly to nearly any database or log management solution to create complex dashboards and reports.</li> </ul> <h2>Full python script</h2> <pre><code class="language-python">import requests from oauthlib.oauth2 import BackendApplicationClient from requests.auth import HTTPBasicAuth from requests_oauthlib import OAuth2Session import pandas as pd import json # Constants BASE_PATH = '&#x3C;Directory in your PC>' CREDENTIALS_FILE = f'{BASE_PATH}credentials.txt' TOKEN_FILE_PATH = f'{BASE_PATH}access_token.txt' AUDIT_LOGS_URL = 'https://global.api.greenlake.hpe.com/audit-log/v1beta1/logs' EXCEL_FILE_PATH = f'{BASE_PATH}audit_logs.xlsx' def generate_and_save_token(): print("Reading credentials from file...") with open(CREDENTIALS_FILE, 'r') as file: lines = file.readlines() client_id = lines[0].strip() client_secret = lines[1].strip() print("Initializing OAuth client...") client = BackendApplicationClient(client_id=client_id) oauth = OAuth2Session(client=client) auth = HTTPBasicAuth(client_id, client_secret) print("Fetching access token...") token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth) access_token = token['access_token'] print("Saving access token to file...") with open(TOKEN_FILE_PATH, 'w') as token_file: token_file.write(access_token) print("Token fetched and saved successfully.") return access_token def fetch_audit_logs(): print("Generating and saving access token...") access_token = generate_and_save_token() headers = {'Authorization': f'Bearer {access_token}'} print("Fetching audit logs from API...") response = requests.get(AUDIT_LOGS_URL, headers=headers) if response.status_code == 200: print("Audit logs fetched successfully.") return response.json() else: print(f"Failed to fetch audit logs: HTTP status code {response.status_code}") return None def process_and_export_to_excel(data): print("Processing audit log data...") df = pd.json_normalize(data['items']) df = df.reindex(columns=[ # Reordering and selecting columns 'id', 'type', 'user.username', 'workspace.id', 'workspace.workspaceName', 'category', 'application.id', 'application.applicationName', 'description', 'generation', 'createdAt', 'updatedAt', 'additionalInfo.ipAddress', 'hasDetails' ]) df.rename(columns={ # Renaming for clarity 'user.username': 'Username', 'workspace.id': 'Workspace ID', 'workspace.workspaceName': 'Workspace Name', 'category': 'Category', 'application.id': 'Application ID', 'application.applicationName': 'Application Name', 'description': 'Description', 'generation': 'Generation', 'createdAt': 'Created At', 'updatedAt': 'Updated At', 'additionalInfo.ipAddress': 'IP Address', 'hasDetails': 'Has Details' }, inplace=True) print("Exporting data to Excel...") df.to_excel(EXCEL_FILE_PATH, index=False) print(f"Audit logs have been successfully exported to {EXCEL_FILE_PATH}.") def main(): print("Starting audit log processing...") audit_logs_data = fetch_audit_logs() if audit_logs_data: process_and_export_to_excel(audit_logs_data) print("Process completed.") if __name__ == "__main__": main() </code></pre> <h2>Summary</h2> <p>This post showed how to access and organize HPE GreenLake's audit logs using Python. It explained how to set up OAuth2 to get an access token, fetch the audit logs, and turn them into an easy-to-read Excel file. This makes it easier to keep your system secure, follow rules, and run things smoothly.</p> <p>In the next article, we will explore how to get additional detail of an audit log and all audit logs of a particular application or user.</p> <p>If you still have any questions regarding the HPE GreenLake Audit Log APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/configuring-azure-ad-with-long-term-token-for-scim-provisiong/https://developer.hpe.com/configuring-azure-ad-with-long-term-token-for-scim-provisiong/Tue, 16 Jul 2024 14:33:05 GMT<p>Azure Active Directory (Azure AD) is Microsoft's cloud-based identity and access management service, designed to simplify user authentication and authorization across various applications and platforms. It offers a centralized solution for managing user identities, enforcing security policies, and facilitating seamless access to cloud-based resources. Azure AD automatic user provisioning simplifies the creation, maintenance, and removal of user identities in SaaS applications based on business rules.</p> <p>The Azure AD provisioning service provisions users to the HPE GreenLake portal by connecting to the user management API endpoints provided by HPE GreenLake Identity and Access Management (IAM). These user management API endpoints allow Azure AD to programmatically create, update, and remove users and groups. The Azure AD provisioning service uses an HPE GreenLake tenant API token to provision users and groups to the HPE GreenLake IAM. The HPE tenant API tokens are only valid for fifteen minutes. Because Azure AD cannot automatically renew the token, long-term tokens are required.</p> <p>In this blog post, I'll explain the process for configuring Azure AD to use a long-term token for user and group provisioning.</p> <h2>Steps to configure long-term tokens in Azure AD</h2> <ol> <li>Connect to the HPE GreenLake portal and assign roles required for System for cross-domain identity management (SCIM)</li> <li>Get a personal access token</li> <li>Create a SCIM proxy token</li> <li>Update the SCIM proxy token and the tenant URL in Azure AD Enterprise Application</li> <li>Update the attribute mappings of Users and Groups</li> <li>User/Group Provisioning</li> </ol> <h2>Step 1: Connect to HPE GreenLake portal and assign roles required for System for Cross-domain Identity Management (SCIM)</h2> <p>Assign "SCIM Proxy Token Contributor" role to the user or user group that will create the long-term token</p> <ul> <li>Connect to the HPE GreenLake portal (<a href="https://common.cloud.hpe.com/">https://common.cloud.hpe.com</a>).</li> <li>Cross-launch "HPE GreenLake Flex Solutions" service.</li> <li>Click the "User Management" icon on the top-right corner.</li> <li>Select the user/user group that will generate the SCIM proxy token.</li> <li>Select "Actions" and then "Create Assignment".</li> <li>Select "SCIM Proxy Token Contributor" role.</li> <li>Select "All Resources" space and "greenlake.service.user" scope.</li> <li>Enable "I confirm that I want to create the assignments listed above".</li> <li>Click "Create Assignment" button.</li> </ul> <h2>Step 2: Get a personal access token</h2> <p>An API token issued by the HPE GreenLake Flex Solutions platform must be used as the Bearer token in the Authorization header of HPE GreenLake Flex Solutions REST API requests. Perform the following steps to get API access token from HPE GreenLake Flex Solutions portal:</p> <ul> <li>Log into HPE GreenLake Flex Solutions.</li> <li>Click the profile icon on the top-right corner.</li> <li>Select API Access.</li> <li>Copy the API access token.</li> <li>Save it for use with cURL or an other REST API client.</li> <li>For example: export BEARER_TOKEN=<paste token value></li> </ul> <p><strong>Note</strong>: This token is valid for 15 minutes after generation.</p> <h2>Step 3: Create a SCIM proxy token</h2> <p>A SCIM Proxy Token is required for the SCIM integration to work. Run the following cURL command to generate the SCIM Proxy token:</p> <p><code>curl -H "Authorization: bearer $BEARER_TOKEN" -X POST https://sps.us1.greenlake-hpe.com/v1alpha1/proxytoken</code></p> <p><strong>Note</strong>: This step must be performed once during the initial setup and every time a token is deleted.</p> <h2>Step 4: Update the SCIM proxy token and the tenant URL in Azure AD Enterprise Application</h2> <p>The generated SCIM Proxy Token should be copied and applied in the Azure AD Enterprise Application.</p> <ul> <li> In Azure AD, go to the “Enterprise applications”.</li> <li> Click the “SSO-Integration” application.</li> <li> Click the “Provisioning” on the left navigation window.</li> <li> Click the “Get started.</li> </ul> <p><img src="/img/scim-page1.png" alt="" title="Application provisioning in Azure AD"></p> <ul> <li>Select "Provisioning Mode" to "Automatic" </li> <li>Click the “Admin Credentials”.</li> <li> Update the generated token in the “Secret Token” field.</li> <li> Update the URL <a href="https://sps.us1.greenlake-hpe.com/v1alpha1/scimproxy">https://sps.us1.greenlake-hpe.com/v1alpha1/scimproxy</a> in the “Tenant URL” field.</li> <li>Test connection - Connection should HPE GreenLake platform should succeed.</li> <li>Save the configuration.</li> </ul> <p><img src="/img/scim-page2.png" alt="" title="Updating the tenant URL and Token"></p> <h2>Step 5: Update the attribute mappings of users and groups</h2> <p>Before provisioning the users/groups to HPE GreenLake platform, edit the attribute mappings:</p> <ul> <li>Update the attribute mapping of Users</li> <li>Unselect the update options under "Target Object Actions"</li> <li>customappsso attribute should have below attributes configured<br> userName<br> displayName<br> name.givenName<br> name.familyName</li> </ul> <p><img src="/img/scim-page3.png" alt="" title="Attribute Mapping of user"></p> <ul> <li>Update the attribute mapping of groups</li> <li>customappsso attribute should have the below attributes configured<br> displayName<br> externalid<br> members</li> </ul> <p><img src="/img/scim-page4.png" alt="" title="Attribute Mapping of Group"></p> <p>Save the configuration and enable the provisioning status from "OFF" to "ON"</p> <p>![](/img/scim-page6.png "Enabling the Provisioning status to \\"ON\\"")</p> <ul> <li>Assign the Azure AD group to the Enterprise application</li> <li>Note: This step is very important to give access to subset of groups and users who need access to HPE GreenLake platform from large enterprise groups from Azure AD.</li> </ul> <p><img src="/img/scim-page5.png" alt="" title="Assign the Azure AD group to the Enterprise application"></p> <h2>Step 6: User/Group Provisioning</h2> <p>All set to provision the groups/users to "HPE GreenLake platform".</p> <ul> <li>Click "Start Provisioning" to start.</li> <li>Upon successful provisioning verify the users and groups are pushed to "HPE GreenLake platform"</li> <li>Click "Stop Provisioning" to stop.</li> <li>Click "View Provisioning" logs to view the failures.</li> </ul> <p><img src="/img/scim-page7.png" alt=""></p> <p>Users can rotate a long-lived token before its expiration date using the following API:</p> <p><code>curl -H "Authorization: bearer $BEARER_TOKEN" -X POST https://sps.us1.greenlake-hpe.com/v1alpha1/proxytoken/rotate?remove-current=true</code></p> <p>When the "remove_current" flag is enabled, it replaces the current token with a new one. During this process, there might be a temporary disruption in user and group provisioning, which will automatically resolve itself in the subsequent provisioning cycle. Alternatively, if the "remove_current" flag is disabled, the current token is replaced only after the new token takes effect, ensuring uninterrupted user experience without any provisioning failures.</p> <p>I hope this blog post answers any questions you may have regarding configuration of SCIM with HPE GreenLake platform. Please return to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more tips and tricks on working with the HPE GreenLake platform.</p>https://developer.hpe.com/parallel-processing-of-a-billion-rows-of-data-in-chapel/https://developer.hpe.com/parallel-processing-of-a-billion-rows-of-data-in-chapel/Fri, 12 Jul 2024 19:10:04 GMT<p>External blog</p>https://developer.hpe.com/tensor-parallelism-in-three-levels-of-difficulty/https://developer.hpe.com/tensor-parallelism-in-three-levels-of-difficulty/Wed, 10 Jul 2024 13:44:29 GMT<p>External blog post</p>https://developer.hpe.com/navier-stokes-in-chapel-—-2d-simulations-and-performance/https://developer.hpe.com/navier-stokes-in-chapel-—-2d-simulations-and-performance/Tue, 09 Jul 2024 22:34:53 GMT<p>External blog</p>https://developer.hpe.com/integrating-k8sgpt-to-empower-kubernetes-with-ai-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/integrating-k8sgpt-to-empower-kubernetes-with-ai-in-hpe-greenlake-for-private-cloud-enterprise/Tue, 09 Jul 2024 11:48:04 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>This blog post describes the process to integrate <a href="https://github.com/k8sgpt-ai/k8sgpt">K8sGPT</a> serving a local large language model (LLM) as an artificial intelligence (AI) backend to Kubernetes (K8s) in HPE GreenLake for Private Cloud Enterprise. It explores the convergence of K8s and AI for diagnosing and triaging issues in K8s clusters and providing actionable insights and recommendations from AI for K8s management.</p> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>K8s is celebrated for its scalability, self-healing capabilities and compatibility, which make it a preferred choice for developers and organizations. Though K8s dramatically simplifies application deployment in containers, it adds a new set of complexities for managing, securing and troubleshooting applications. Operating K8s clusters and managing the deployed applications can present challenges.</p> <p>AI is a technological innovation that equips computers and machines with the ability to mimic human intelligence and problem-solving skills. These AI systems are trained on vast volumes of data, enabling them to recognize patterns and perform tasks such as solving problems in a human-like manner. The evolution of AI, especially the advent of extensive LLM models, has expanded possibilities in many sectors, including K8s. The application of AI in enhancing diagnostics and troubleshooting workflows in K8s clusters has grown considerably.</p> <p>This blog post explores the convergence of K8s and AI through K8sGPT, a tool for scanning the K8s cluster, diagnosing and triaging K8s issues using AI. It describes the detailed process to integrate K8sGPT with local LLM models to empower K8s in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li> <p>A K8s cluster being provisioned, using <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">HPE GreenLake Terraform provider</a> for example, in HPE GreenLake for Private Cloud Enterprise</p> </li> <li> <p>The <em>kubectl</em> CLI tool, together with the kubeconfig file for accessing the K8s cluster</p> </li> <li> <p><a href="https://www.python.org/downloads/">Python 3.8 or higher</a>, and <em>pip</em> that’s included by default in Python</p> </li> </ul> <h3>K8sGPT and LocalAI</h3> <p><a href="https://github.com/k8sgpt-ai/k8sgpt">K8sGPT</a> is an open source project designed to address common and complex issues within K8s cluster using AI. It leverages large language models (LLMs) to enhance troubleshooting, streamline processes, and improve K8s management. K8sGPT supports various <a href="https://docs.k8sgpt.ai/reference/providers/backend/">AI backends</a> (also called providers), including <a href="https://openai.com/">OpenAI</a>, <a href="https://aws.amazon.com/bedrock/">Amazon Bedrock</a>, <a href="https://azure.microsoft.com/en-us/products/cognitive-services/openai-service">Azure OpenAI</a>, <a href="https://ai.google.dev/docs/gemini_api_overview">Google Gemini</a> as well as <a href="https://github.com/mudler/LocalAI">LocalAI</a>.</p> <p>LocalAI is an open source project that provides an alternative to OpenAI’s offerings for local inferencing. It does not require a GPU and can run on consumer-grade hardware without high-end computing resources. By deploying AI solutions within the local infrastructure and keeping all processes in-house, it avoids the costs associated with external AI services and ensures better data sovereignty and privacy.</p> <p>The following sections describe the process to deploy K8sGPT using LocalAI as its backend to empower K8s cluster with AI capabilities in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Install K8sGPT</h3> <p>There are two options to install K8sGPT, as a CLI tool or as a K8s Operator in the K8s cluster. This section takes the CLI option to install K8sGPT to the Linux workstation in my local environment that's used to manage the K8s clusters. You can refer to the <a href="https://github.com/k8sgpt-ai/k8sgpt">K8sGPT Github</a> for additional information to install the K8sGPT as a K8s Operator or as a CLI tool on <em>Windows</em> or <em>Mac</em>. With the CLI option, the K8sGPT setup will be independent of any specific K8s cluster and it can be used for working on any existing K8s clusters. This is helpful, especially when multiple K8s clusters are created and managed in your environment. They can all work with the same K8sGPT installation.</p> <p>Follow these instructions to install K8sGPT as a CLI tool on the Linux workstation:</p> <pre><code class="language-shell">$ curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.24/k8sgpt_386.deb $ sudo dpkg -i k8sgpt_386.deb </code></pre> <p>Type the following command to verify the K8sGPT installation:</p> <pre><code class="language-shell">$ k8sgpt version k8sgpt: 0.3.24 (eac9f07), built at: unknown $ k8sgpt -h Kubernetes debugging powered by AI Usage: k8sgpt [command] Available Commands: analyze This command will find problems within your Kubernetes cluster auth Authenticate with your chosen backend cache For working with the cache the results of an analysis completion Generate the autocompletion script for the specified shell filters Manage filters for analyzing Kubernetes resources generate Generate Key for your chosen backend (opens browser) help Help about any command integration Integrate another tool into K8sGPT serve Runs k8sgpt as a server version Print the version number of k8sgpt Flags: --config string Default config file (/home/guoping/.config/k8sgpt/k8sgpt.yaml) -h, --help help for k8sgpt --kubeconfig string Path to a kubeconfig. Only required if out-of-cluster. --kubecontext string Kubernetes context to use. Only required if out-of-cluster. Use "k8sgpt [command] --help" for more information about a command. </code></pre> <p>K8sGPT comes with a list of <a href="https://github.com/k8sgpt-ai/k8sgpt?tab=readme-ov-file#analyzers">built-in analyzers</a>, which is essentially a series of rules/checks that can be used to diagnose issues in various K8s API resources, such as <em>Pod</em> crashes, <em>Service</em> failures, etc.</p> <p>K8sGPT provides a list of <em>filters</em> that can be used together with K8sGPT analyzer to scan issues for any specific K8s API resources:</p> <pre><code class="language-shell">$ k8sgpt filters list Active: > Node > Pod > Deployment > ReplicaSet > PersistentVolumeClaim > Service > Ingress > StatefulSet > ValidatingWebhookConfiguration > MutatingWebhookConfiguration > CronJob Unused: > HorizontalPodAutoScaler > PodDisruptionBudget > NetworkPolicy > Log > GatewayClass > Gateway > HTTPRoute </code></pre> <p>You can add any other <em>unused</em> filters, e.g., <em>HorizontalPodAutoScaler</em> or <em>NetworkPolicy</em>, by typing the command <em>'k8sgpt filter add [filter(s)]'</em>.</p> <h3>Set up LocalAI backend</h3> <p>This section will focus on setting up and utilizing LocalAI with a supported LLM model in the local environment. This LocalAI setup in the workstation will be integrated with K8sGPT.</p> <h4>Download a LLM model</h4> <p>LocalAI supports a list of LLM models, such as <em>LLama</em>, <em>GPT4ALL</em>, <em>Alpaca</em> and <em>koala</em>, etc. In this blog post, the LLM model <em><code>Llama-2–13b-chat-hf</code></em> from <a href="https://huggingface.co/">Hugging Face</a> will be downloaded and used as the local AI backend for K8sGPT.</p> <p>After you create an account in <em>Hugging Face</em>, log in to the site and share your contact information. You will then be granted access to this model.</p> <p><img src="/img/hf-llama.png" alt=""></p> <p>Type the following command to clone the LLM model. Make sure you have <a href="https://git-lfs.github.com"><em>git-lfs</em></a> installed.</p> <pre><code class="language-shell">$ git-lfs clone https://huggingface.co/meta-llama/Llama-2-13b-chat-hf $ tree Llama-2-13b-chat-hf/ Llama-2-13b-chat-hf/ ├── config.json ├── generation_config.json ├── LICENSE.txt ├── model-00001-of-00003.safetensors ├── model-00002-of-00003.safetensors ├── model-00003-of-00003.safetensors ├── model.safetensors.index.json ├── pytorch_model-00001-of-00003.bin ├── pytorch_model-00002-of-00003.bin ├── pytorch_model-00003-of-00003.bin ├── pytorch_model.bin.index.json ├── README.md ├── Responsible-Use-Guide.pdf ├── special_tokens_map.json ├── tokenizer_config.json ├── tokenizer.json ├── tokenizer.model └── USE_POLICY.md 0 directories, 18 files </code></pre> <h4>Install <em>Text Generation Web UI</em></h4> <p>There are many LLM serving framework and tools that support various LLM models and provide a way to interact with those models with OpenAI compatible API endpoints.</p> <p>This section introduces the <a href="https://github.com/oobabooga/text-generation-webui">Text Generation Web UI (TGW)</a> tool and shows how to set up this tool to support the locally downloaded LLM model as the AI backend. The TGW is a <a href="https://github.com/gradio-app/gradio/">Gradio</a> based tool that builds a web UI and supports many large language model formats with OpenAI compatible APIs.</p> <p>Type the following commands to clone the TGW repo and install it in the workstation:</p> <pre><code class="language-shell">git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui pip install -r requirements.txt </code></pre> <h4>Serve local LLM model</h4> <p>Type the following command to start serving the downloaded LLM model in your workstation. The option <em>'--extensions openai'</em> specifies to use the OpenAI extension to provide OpenAI format API. The options <em>'--cpu'</em> &#x26; <em>'--load-in-4bit'</em> to load model in 4-bit precision and use the model performed on a CPU to make predictions. This is more cost-effective for inference, and it’s helpful in case you don’t have GPU's installed in your environment.</p> <pre><code class="language-shell">$ python3 server.py --listen-host 0.0.0.0 --listen --model Llama-2-13b-chat-hf --model-dir /home/guoping/CFE/AI/models --extensions openai --cpu --load-in-4bit 20:32:39-883617 INFO Starting Text generation web UI 20:32:39-887086 WARNING You are potentially exposing the web UI to the entire internet without any access password. You can create one with the "--gradio-auth" flag like this: --gradio-auth username:password Make sure to replace username:password with your own. 20:32:39-889820 INFO Loading "Llama-2-13b-chat-hf" 20:32:39-923551 INFO TRANSFORMERS_PARAMS= {'low_cpu_mem_usage': True, 'torch_dtype': torch.float32} Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:47&#x3C;00:00, 15.91s/it] 20:33:27-988083 INFO LOADER: "Transformers" 20:33:27-989519 INFO TRUNCATION LENGTH: 4096 20:33:27-990434 INFO INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)" 20:33:27-991414 INFO Loaded the model in 48.10 seconds. 20:33:27-992350 INFO Loading the extension "openai" 20:33:28-164441 INFO OpenAI-compatible API URL: http://0.0.0.0:5000 Running on local URL: http://0.0.0.0:7860 </code></pre> <p>The OpenAI compatible API endpoint is hosted on <em>'<a href="http://0.0.0.0.5000">http://0.0.0.0.5000</a>'</em>.</p> <p>Apart from this API endpoint URL, the TGW starts running another local web URL, i.e., at <em>'<a href="http://0.0.0.0:7860">http://0.0.0.0:7860</a>'</em>, providing 3 interface modes, <em>chat</em>, <em>default</em> &#x26; <em>notebook</em>, for text generation with the local LLM model backend.</p> <p>You can start the browser and type this local URL <em>'<a href="http://0.0.0.0:7860">http://0.0.0.0:7860</a>'</em>. You then land on the <em>Chat</em> page. Ask a question in the <em>Chat</em> page by typing some text and clicking <strong>Generate</strong> button. You may notice it’s a bit slower if you serve the model using CPU inference. But everything should work and you will get the response to your question by <em>AI</em>.</p> <p><img src="/img/local-llm.png" alt=""></p> <h3>Integrate K8sGPT with LocalAI</h3> <p>Type the following command to configure K8sGPT to use the local API endpoint. Don’t forget to add the <em>“/v1”</em> to the API URL.</p> <pre><code class="language-shell">$ k8sgpt auth add --backend localai --model Llama-2-13b-chat-hf --baseurl http://localhost:5000/v1 localai added to the AI backend provider list $ k8sgpt auth list Default: > openai Active: > localai Unused: > openai > azureopenai > noopai > cohere > amazonbedrock > amazonsagemaker </code></pre> <p>In case the LocalAI API endpoint has been already added to K8sGPT, type the command below to remove it. Then re-run the <em>k8sgpt auth add</em> to add it again with the new LLM model and its API endpoint.</p> <pre><code class="language-shell">$ k8sgpt auth remove --backends localai </code></pre> <h3>Detect K8s issues using K8sGPT</h3> <p>With all components being installed and configured, it's time to detect K8s issues in the K8s cluster.</p> <h4>Deploy a sample application</h4> <p>This section will deploy a sample application using <em>kubectl</em> CLI.</p> <p>Type below commands to deploy such a sample application to the namespace <em>'cfe-apps'</em>:</p> <pre><code class="language-shell">$ kubectl create ns cfe-apps namespace/cfe-apps created $ kubectl create deployment app-with-no-image --image=cfe-image-not-exist -n cfe-apps deployment.apps/app-with-no-image created </code></pre> <p>Check the deployed application using the following command:</p> <pre><code class="language-shell">$ kubectl get all -n cfe-apps NAME READY STATUS RESTARTS AGE pod/app-with-no-image-7ff65f5484-9bt4z 0/1 ImagePullBackOff 0 2m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/app-with-no-image 0/1 1 0 2m NAME DESIRED CURRENT READY AGE replicaset.apps/app-with-no-image-7ff65f5484 1 1 0 2m </code></pre> <p>The output shows some issues in the K8s <em>Pod</em> <em>'app-with-no-image-7ff65f5484-9bt4z'</em>. The K8s Deployment keeps showing as <em>'0/1 READY'</em>.</p> <h4>Run <em>k8sgpt analyze</em></h4> <p>To start K8s issue detection, type the following <em>k8sgpt analyze</em> command:</p> <pre><code class="language-shell">$ k8sgpt analyze --backend localai --filter=Pod --namespace cfe-apps --no-cache AI Provider: localai 0 cfe-apps/app-with-no-image-7ff65f5484-9bt4z(Deployment/app-with-no-image) - Error: Back-off pulling image "cfe-image-not-exist" </code></pre> <p>Instead of running a scan with all the default analyzers, the above command uses the filter <em>Pod</em> to detect K8s Pod issue in the namespace <em>'cfe-apps'</em>. It detects the error of back-off pulling image "cfe-image-not-exist" in the Pod.</p> <p>Execute again the K8sGPT analyze with the option <em>'--explain'</em> this time:</p> <pre><code class="language-shell">$ k8sgpt analyze --backend localai --filter=Pod --namespace cfe-apps --explain --no-cache 100% |█████████████████████████████████████████████████████████████████████████████████████████████████| (1/1, 25 it/hr) AI Provider: localai 0 cfe-apps/app-with-no-image-7ff65f5484-9bt4z(Deployment/app-with-no-image) - Error: Back-off pulling image "cfe-image-not-exist" Error: Back-off pulling image "cfe-image-not-exist" Solution: 1. Check the image name and ensure it exists in the registry. 2. Verify the image tag and ensure it matches the expected tag. 3. Try pulling the image again using the correct name and tag. 4. If the image still does not exist, check the spelling and ensure the name is correct. 5. If all else fails, try using the "docker pull" command to download the image locally and then push it to the registry. </code></pre> <p>The option <em>'--explain'</em> in the above command enables the AI backend. The analyze establishes a connection to the AI backend and provides it with the error message it detected in K8s. When the option <em>'--explain'</em> is used, K8sGPT not only executes the requested action to detect the K8s issue but also provides the step-by-step solutions. The solutions provided by K8sGPT can be used as actionable insights to fix the K8s issues.</p> <p>Apart from the default English, K8sGPT supports getting the response in other languages. Here is an example of getting a response in K8sGPT in French:</p> <pre><code class="language-shell">$ k8sgpt analyze --backend localai --filter=Pod --namespace cfe-apps --language french --explain --no-cache 100% |█████████████████████████████████████████████████████████████████████████████████████████████████| (1/1, 16 it/hr) AI Provider: localai 0 cfe-apps/app-with-no-image-7ff65f5484-9bt4z(Deployment/app-with-no-image) - Error: Back-off pulling image "cfe-image-not-exist" Sure! Here's the simplified error message and solution in French, within the 280 character limit: Error: Impossible de tirer l'image "cfe-image-not-exist". Solution: 1. Vérifiez si l'image est disponible dans le registre de conteneurs. 2. Si l'image n'existe pas, créez-la manuellement en utilisant le commandes `docker pull` ou `docker build`. 3. Essayez à nouveau de tirer l'image en utilisant la commande `kubectl apply`. </code></pre> <h3>Conclusion</h3> <p>This blog post explores the integration of K8s with AI to address the complexities for managing, securing and troubleshooting K8s cluster and its applications. By integrating K8sGPT with LocalAI to K8s in HPE GreenLake for Private Cloud Enterprise, it eliminates the need for external AI services, leading to cost reductions and enhanced data sovereignty and privacy by preventing data transmission to external AI providers.</p> <p>K8sGPT proves to be a valuable asset in diagnosing issues in K8s cluster and enhancing operational efficiency. This tool can be further integrated with other organizational tools like <em>Slack</em> and <em>Microsoft Teams</em> to dispatch notifications and alerts, accompanied by suggested solutions as remediation steps. This futher amplifies the practical value of K8sGPT within the organization.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/2024-july-08/https://developer.hpe.com/2024-july-08/Mon, 08 Jul 2024 13:00:00 GMThttps://developer.hpe.com/the-sleuth-and-the-storyteller-the-dynamic-duo-behind-rag/https://developer.hpe.com/the-sleuth-and-the-storyteller-the-dynamic-duo-behind-rag/Thu, 04 Jul 2024 10:16:28 GMT<p>External blog</p>https://developer.hpe.com/redfish-interoperability-profiles/https://developer.hpe.com/redfish-interoperability-profiles/Wed, 03 Jul 2024 10:58:04 GMT<style> li { font-size: 27px; line-height: 35px; max-width: none; } </style> <style> figcaption {font-style: italic; font-size: 15px; line-height: 33px; max-width: none;} </style> <h2>Introduction</h2> <p>When I explain to an audience that the <a href="https://redfish.dmtf.org" target="_blank">Redfish® standard</a> requires the implementation of only a subset of the properties mentioned in the entire <a href="https://developer.hpe.com/blog/why-is-redfish%C2%AE-different-from-other-rest-apis-part-1/#data-model" target="_blank">data model</a>, I can see people looking at me puzzled and asking themselves:</p> <p>"<em>What? Are you telling me that there is a potential that some BMCs in my data center do not implement the <code>FirmwareVersion</code> property and yet they are considered to be compliant with the standard?</em>".</p> <p>The answer is yes. Those Redfish services that are not implementing properties not labeled "required" are still considered to be compliant.</p> <p>Although they are considered to be compliant, there are instances where this can be problematic. In this blog post, I'll elaborate and provide examples of use cases where it can be a problem. Then I'll introduce the <a href="https://www.dmtf.org/dsp/DSP0272" target="_blank"> Redfish interoperability profiles specification</a> that has been created by the Distributed Management Task Force (<a href="https://dmtf.org" target="_blank">DMTF</a>) to address those use cases.</p> <p>The Redfish interoperability profiles specification constitutes another Redfish specificity that could be added to the list presented in <a href="https://developer.hpe.com/blog/why-is-redfish%C2%AE-different-from-other-rest-apis-part-1/" target="_blank">part 1</a> and <a href="https://developer.hpe.com/blog/why-is-redfish%C2%AE-different-from-other-rest-apis-part-2/" target="_blank">part 2</a> of the <em>Why is Redfish different from other REST APIs</em> blog posts.</p> <h3>Redfish services can omit defined properties</h3> <p>The DSP0266 standard document states in its <a href="https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.20.1.html#properties" target="_blank">Properties overview</a> paragraph:</p> <ul> <li><em>Required properties shall always be returned in a response.</em></li> </ul> <p><img src="/img/fig1-dmtfpropertiesoverview.png" alt="Figure 1: DMTF Properties overview paragraph from DSP0266" title="Figure 1: DMTF Properties overview paragraph from DSP0266"></p> <figcaption>Figure 1: DMTF Properties overview paragraph from DSP0266</figcaption> <p>This assertion suggests that some properties are not "required" in the implementation of the service. As an example, in the data model of the <a href="https://redfish.dmtf.org/schemas/v1/Manager.v1_19_1.json" target="_blank">Baseboard Manager Controller</a> (BMC) (Figure 2), the only required properties are: <code>@odata.id</code>, <code>@odata.type</code>, <code>Id</code> and <code>Name</code>. Since the <code>FirmwareVersion</code> property is not listed in this normative document, its implementation is not required in Redfish services.</p> <p><img src="/img/fig2-managerrequiredproperties.png" alt="Manager required properties" title="Manager required properties"></p> <figcaption>Figure 2: Manager required properties</figcaption> <p>Requiring the implementation of only a small number properties <em>provide[s] significant flexibility, and allow conforming implementations on a wide variety of products</em> as mentioned in the abstract of the <a href="https://www.dmtf.org/sites/default/files/standards/documents/DSP0272_1.8.0.pdf" target="_blank">standard document</a>.</p> <h3>Who cares?</h3> <p>Flexibility is great, but too much flexibility can become an issue for standard organizations, software projects, or end users willing to move away from the Intelligent Platform Management Interface (<a href="https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-home.html" target="_blank">IPMI</a>) to Redfish for hardware management.</p> <p>Standards organizations and software projects, like the <a href="https://www.opencompute.org/about" target="_blank">Open Compute Project®</a> (OCP) and the <a href="https://wiki.openstack.org/wiki/Main_Page" target="_blank">OpenStack</a> projects can only adopt Redfish as their preferred management protocol if they can easily define some sort of "baseline" containing which property must, should, or could be implemented in their managed nodes.</p> <p>Specific to OCP, the charter of the <a href="https://www.opencompute.org/projects/hardware-management" target="_blank">Hardware Management Project</a> mentions: "<em>The hardware management specification incorporates [...] tools and best practices [...] for remote machine management</em>". This means that any server compliant to this specification must implement the network protocol(s) mentioned in the baseline.</p> <p>Systems supported by the OpenStack <a href="https://wiki.openstack.org/wiki/Ironic" target="_blank">Ironic</a> (bare metal machine provisioning) and the <a href="https://wiki.openstack.org/wiki/Valence" target="_blank">Valence</a> projects (system lifecycle management) must implement a baseline of features containing at least the possibility to be powered on and off remotely as well.</p> <p>Redfish clients designed for managing multi-vendor systems also need a list of mandatory and recommended features. For example, if a system cannot return its BMC's firmware version, the client will have difficulties performing firmware updates.</p> <h2>Redfish interoperability profiles</h2> <p>To address the baseline issue mentioned above, the DMTF created the <a href="https://www.dmtf.org/dsp/DSP0272" target="_blank">DSP0272</a> specification document that defines interoperability profiles. A Redfish interoperability profile (or profile) is a JSON document enumerating resources and properties that must, should, or could be implemented in a Redfish service.</p> <h3>Didactic minimal profile example</h3> <p>The following example is a minimal profile created for didactic purpose. It is not relevant for use in a proper production context.</p> <p>A summary of this example could be the following: "To be compliant to this profile, Redfish services must model at least one manager (BMC) and must be able to return the manager's <code>FirmwareVersion</code> value".</p> <blockquote> <p>NOTE: The content of interoperability profiles is described in a versioned schema file. All the profile versioned schema files are grouped in compressed bundles (<code>.zip</code>) and can be <a href="https://www.dmtf.org/dsp/DSP8013" target="_blank">downloaded from the DMTF</a>. The following example is compliant with version 1.8.0 as specified in the <code>SchemaDefinition</code> key of the following example (first line).</p> </blockquote> <pre><code class="language-json">{ "SchemaDefinition": "RedfishInteroperabilityProfile.v1_8_0", "ProfileName": "FDZ's minimal profile", "ProfileVersion": "1.0.0", "Purpose": "This is a minimal educational Redfish interoperability profile.", "OwningEntity": "Koulapic", "ContributedBy": "FDZ", "License": "CC BY-SA", "ContactInfo": "fdz@koulapic.com", "Protocol": { "MinVersion": "1.6" }, "Resources": { "ManagerCollection": { "Purpose": "Every implementation must have at least one BMC.", "PropertyRequirements": { "Members": { "MinCount": 1 } } }, "Manager": { "Purpose": "Make sure Manager is conformant to schema 1.5.1 or later and implements the `FirmwareVersion` property", "MinVersion": "1.5.1", "PropertyRequirements": { "FirmwareVersion": { "ReadRequirement": "Mandatory" } } } } } </code></pre> <p>Profiles contain administrative "self explanatory" keys like <code>ProfileName</code>, <code>ContactInfo</code>, or <code>Purpose</code>. The normative definition of those properties is in the schema mentioned in the above note. For this example I extracted in the next JSON block, the description of the <code>Protocol/MinVersion</code> property, which mentions that it is to be compared to the <code>ServiceRoot/RedfishVersion</code> of the evaluated Redfish service.</p> <pre><code class="language-json">{ "Protocol": { "type": "object", "description": "Requirements related to the Redfish protocol outside of the JSON resources.", "additionalProperties": false, "properties": { "MinVersion": { "$ref": "#/definitions/MinVersion", "description": "Indicates the minimum version of the Redfish Specification protocol support required by this profile. This version shall be reported by the Redfish service in the `ServiceRoot` resource property `RedfishVersion`. The version shall be represented using a `&#x3C;major>.&#x3C;minor>.&#x3C;errata>` format, including an optional errata version. If this property is absent, the minimum value shall be `1.0.0`." } } } } </code></pre> <p>The <code>Resources{}</code> object of the above profile contains two members: <code>ManagerCollection{}</code> and <code>Manager{}</code>. The first one requires at least one BMC modeled in the evaluated Redfish service (<code>MinCount = 1</code>). The second requires the implementation of the <code>Manager/FirmwareVersion</code> (<code>ReadRequirement = Mandatory</code>).</p> <p>The other possible values for the <code>ReadRequirement</code> property are listed and described in the profile schema. I pasted them in the following JSON block. It is interesting to notice that, in addition to obvious values, like <code>Mandatory</code> or <code>Recommended</code>, others (i.e. <code>Conditional</code>, <code>IfImplemented</code>) need more attention.</p> <pre><code class="language-json">{ "ReadRequirement": { "type": "string", "description": "The read (HTTP GET) requirements for this property. The default value, or if `ReadRequirement` is not present, is `Mandatory`. For object properties, requirements of the embedded properties will apply only if the object is present.", "enum": [ "Mandatory", "Supported", "Recommended", "IfImplemented", "IfPopulated", "Conditional", "None" ], "enumDescriptions": { "Mandatory": "This property is required in all instances of this resource. For array properties, the property is required in all non-null array items. If `Values` is defined, at least one instance of each enumeration value is required among instances of this property.", "Supported": "This property is required to be supported by the service, but may not appear in all instances of this resource. The requirement is met if the property appears in at least one instance of this resource.", "Recommended": "It is recommended, but not required, that this property be supported.", "IfImplemented": "This property is required if the underlying functionality is implemented. For object properties, requirements on embedded properties within the object will only apply if the object is present.", "IfPopulated": "For property-level requirements, this property is required if the `State` property within the `Status` object for the object or resource does not contain `Absent`. This value is useful for properties within absent resources where empty slots, sockets, or bays are rendered with minimal properties until they are populated by a device. For resource-level requirements, this value indicates that the resource is required, but may not be present (populated) in the service at all times.", "Conditional": "This property is only required if `ConditionalRequirements` items apply to this instance of the resource.", "None": "This property is not required by this profile. It is listed here for clarity." } } } </code></pre> <h3>Stacking profiles</h3> <p>An interesting feature of Redfish interoperability profiles is that you can extend existing profiles with your own definitions at will. To create a profile that extends the <a href="https://github.com/openstack/ironic/tree/master/redfish-interop-profiles" target="_blank">Ironic profile</a>, just use the <code>RequiredProfile{}</code> object as shown in the next example.</p> <p>This example specifies the URL and minimum version of the Ironic profile and yours.</p> <pre><code class="language-json"> "RequiredProfiles": { "Ironic": { "Repository": "https://github.com/openstack/ironic/tree/master/redfish-interop-profiles", "MinVersion": "1.0.0" }, "MyRequiredProfile": { "Repository": "https://koulapic.com/MyInteropProfiles", "MinVersion": "1.0.0" } } </code></pre> <p>Now that you understand the basic architecture of Redfish interoperability profiles, I encourage you to study the <a href="https://github.com/opencomputeproject/HWMgmt-OCP-Profiles" target="_blank">OCP</a> and OpenStack public <a href="https://github.com/openstack/ironic/tree/master/redfish-interop-profiles" target="_blank">profiles</a>. Don't forget to refer to the <a href="https://www.dmtf.org/dsp/DSP8013" target="_blank">profile schemas</a> in case you have problem understanding some properties, directives or objects.</p> <h2>How to validate profiles?</h2> <p>Profile conformance can be easily performed with the <a href="https://github.com/DMTF/Redfish-Interop-Validator" target="_blank">interoperability validator</a> provided by the DMTF. It is a Python script that takes as input a configuration file and a profile. The main output is an HTML report.</p> <p>The following code block clones the validator GitHub repository and asks to create a configuration file and a profile. Then it launches the validator with those two files as input.</p> <pre><code class="language-shell">git clone https://github.com/DMTF/Redfish-Interop-Validator.git cd Redfish-Interop-Validator # Create configuration file and profile python RedfishInteropValidator.py -c config/ilo-scott380g11-1.ini FdzMiniProfile.v1_0_0.json </code></pre> <p>The profile used in the above code example is the minimal profile mentioned <a href="#didactic-minimal-profile-example">earlier</a>. This profile requests at least one manager in the manager <a href="https://servermanagementportal.ext.hpe.com/docs/concepts/datatypesandcollections/#resource-collections" target="_blank">collection</a> and a <code>FirmwareVersion</code> property in the <code>Manager</code> resource. To be sure the validator verifies those requirements, the configuration file (next code block) specifies <code>payload = tree /redfish/v1/Managers</code>. This line tells the validator to verify the profile directives at <code>/redfish/v1/Managers</code> and then follow recursively each and every link it finds and process them. The exact crawling algorithm is explained in the <a href="https://github.com/DMTF/Redfish-Interop-Validator/blob/main/README.md" target="_blank">validator GitHub README.md</a> file.</p> <pre><code class="language-shell"># Profile Validator configuration file. # # Parameter description at: # https://github.com/DMTF/Redfish-Interop-Validator/blob/main/README.md [Tool] Version = 2 Copyright = Redfish DMTF (c) 2021 verbose = [Host] ip = https://ilo-scott380g11-1 username = username password = password description = iLO 6 forceauth = False authtype = Session token = [Validator] # The following `tree` keyword tells the Validator # to crawl the Redfish tree starting at the following # starting point. # # An alternative is `single` to only validate the # starting point. payload = tree /redfish/v1/Managers logdir = ./logs oemcheck = False online_profiles = False required_profiles_dir = debugging = False collectionlimit = </code></pre> <p>The validator outputs two files in the directory specified in the configuration file (<code>logdir</code>): a report in HTML format and a text file containing the different steps of the validation process.</p> <p>Figure 3 below is a portion of the validator report showing three successful verification. The first one (<code>Service level requirements</code>) requires the existence of the <code>ManagerCollection</code> and <code>Manager</code> end points. It has been automatically added by the validator as a mandatory condition before proceeding the verification of the other requirements mentioned in the profile.</p> <p>In the figure below, you can also view the results of the two verifications required in the profile.</p> <p><img src="/img/fig3-conformancetestreport.png" alt="Figure 3: Redfish conformance test report" title="Redfish conformance test report"></p> <figcaption>Figure 3: Redfish conformance test report</figcaption> <h2>Leveraging Redfish interoperability profiles</h2> <p>Although present for a long time (January 2018) and despite an introduction <a href="https://www.youtube.com/watch?v=iVAYSEPwmV8" target="_blank">video</a>, the Redfish interoperability profile specification is not very well known. It could be better leveraged by Redfish client programmers supervising heterogeneous data centers. As this standard, along with the interoperability validator, highlight the differences between Redfish implementations, it can help them produce a more efficient code quickly. If you already know a property is absent in a Redfish service, you can anticipate and adapt your code early in the development process.</p> <p>The OCP hardware management team is very active relatively to Redfish interoperability profiles. Several profiles are published in their <a href="https://github.com/opencomputeproject/HWMgmt-OCP-Profiles" target="_blank">GitHub repository</a> that you can use "as-is" or extend as mentioned in the <a href="#stacking-profiles">Stacking profile</a> paragraph. You can also help OCP further refine them by submitting ideas and feedback.</p> <p>With this blog post, I hope you discovered enough information about this Redfish standard to eventually use it or present it to friends or colleagues.</p> <p>And don't forget to check out some of my other <a href="https://developer.hpe.com/search/?term=donze" target="_blank">blog posts</a> on the HPE Developer portal to learn more about Redfish tips and tricks.</p>https://developer.hpe.com/rag-for-an-llm-powered-hologram/https://developer.hpe.com/rag-for-an-llm-powered-hologram/Tue, 02 Jul 2024 11:56:03 GMT<p>External blog post</p>https://developer.hpe.com/reflections-on-chapelcon-24-a-community-growing-together/https://developer.hpe.com/reflections-on-chapelcon-24-a-community-growing-together/Mon, 01 Jul 2024 22:18:55 GMT<p>External blog</p>https://developer.hpe.com/announcing-chapel-2-1/https://developer.hpe.com/announcing-chapel-2-1/Fri, 28 Jun 2024 06:27:51 GMT<p>external blog</p>https://developer.hpe.com/announcements-at-hpe-discover-2024-accelerate-ai-adoption-and-offer-optionality/https://developer.hpe.com/announcements-at-hpe-discover-2024-accelerate-ai-adoption-and-offer-optionality/Tue, 25 Jun 2024 14:22:51 GMT<p>Ever since OpenAI released ChatGPT in November 2022, there’s been no question that we have entered into the Age of AI. The possibilities Generative AI affords to accelerating innovation is too compelling to ignore. But ever since then, enterprises have struggled with how to implement AI solutions. Their expense and complexity have proven to be significant barriers to entry.</p> <p>Hewlett Packard Enterprise (HPE) and NVIDIA took a bold step forward at HPE Discover 2024, making several important announcements that aim to enable enterprise AI adoption. With purpose-built AI solutions delivered through <a href="https://www.hpe.com/us/en/greenlake.html">HPE GreenLake cloud</a> that combine the technical prowess of these two companies, they will introduce to the market a simplicity of experience that just may jump start mass corporate AI adoption.</p> <p>HPE GreenLake delivers a portfolio of cloud services and as-a-service solutions across locations, from the edge to the data center and to the cloud, with simple, agile, and secure hybrid multicloud management. It serves as the base for these solutions. Additional announcements related to the platform focused on offering customers new options for virtualization and providing cloud-type operations for highly regulated environments.</p> <h2>Leading the enterprise charge in AI</h2> <h3>Solving for humanity with humanity</h3> <p>Focusing an important corporate event on artificial intelligence can be risky. Fear of the unknown is bound to be in the hearts and minds of attendees. But, as so many pointed out during the event, we have already entered the Age of AI. Isn’t it better that we now take the lead and focus on using AI for the betterment of humanity, setting appropriate guardrails in place, and helping drive its adoption in a positive way?</p> <p>This was the bold leadership stance taken by the two leaders as HPE CEO Antoino Neri and NVIDEO founder and CEO Jensen Huang took to the stage in the Sphere at Las Vegas to open Discover 2024. Delivering the first <a href="https://www.youtube.com/watch?v=p28lHtjWn5k">keynote</a> ever in this amazing venue, Antonio spoke of the hope and limitless possibilities AI could bring for renewable energy, medical breakthroughs, and revolutionized agriculture.</p> <p>Accompanied by video emphasizing these points, Antonio broke the barrier and started the hard conversation. Imbuing hope, he did not shy away from mentioning the risks involved and was quick to point out how HPE has been preparing for AI for a very long time, developing the technology and guardrails to help make AI a positive force for our planet.</p> <h3>Accelerating enterprise GenAI adoption</h3> <p>During the keynote, Antonio and Jensen jointly announced the launch of NVIDIA AI Computing by HPE, a comprehensive portfolio of co-developed AI solutions designed to accelerate the adoption of generative AI in enterprises. The portfolio includes HPE Private Cloud AI, a turnkey solution that includes a complete infrastructure, software portfolio, and AI model library managed via <a href="https://www.hpe.com/us/en/greenlake.html">HPE GreenLake</a>. Delivering a “private AI in a box”, it offers enterprises a solution to accelerate their AI project deployment while ensuring their data remains under their control.</p> <p>The foundation of the AI and data software stack integrated into this solution starts with the <a href="https://www.nvidia.com/en-us/data-center/products/ai-enterprise/">NVIDIA AI Enterprise software</a> platform, which includes <a href="https://www.nvidia.com/en-us/ai/#referrer=ai-subdomain?ncid=pa-srch-goog-772333&#x26;_bt=697697685508&#x26;_bk=nvidia%20nim&#x26;_bm=e&#x26;_bn=g&#x26;_bg=165151891361&#x26;gad_source=1&#x26;gclid=EAIaIQobChMIlJ2kiNS4hgMVyi2tBh0XRw5KEAAYASAAEgI3ivD_BwE">NVIDIA NIM™ inference microservices</a>. HPE AI Essentials software adds a ready-to-run set of curated AI and data foundation tools with a unified control plane. The solution includes a fully integrated AI infrastructure stack with <a href="https://www.nvidia.com/en-us/networking/spectrumx/">NVIDIA Spectrum-X™ Ethernet networking</a>, HPE GreenLake for File Storage, and HPE ProLiant servers with support for <a href="https://www.nvidia.com/en-us/data-center/l40s/">NVIDIA L40S</a>, <a href="https://www.nvidia.com/en-us/data-center/h100/">NVIDIA H100 NVL Tensor Core GPUs</a> and the <a href="https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/">NVIDIA GH200 NVL2 platform</a>. In addition, <a href="https://www.hpe.com/us/en/opsramp.html?jumpid=ps_tvmvd4zkn_aid-521080156&#x26;ef_id=CjwKCAjw1K-zBhBIEiwAWeCOF0lwzLIS76IvYTcWyzQOw935SmbZgRafDYLekOCalqkzT29SVp0WOxoC_FIQAvD_BwE:G:s&#x26;s_kwcid=AL!13472!3!702006004480!e!!g!!hpe%20opsramp!21375753832!169113463891&#x26;gad_source=1&#x26;gclid=CjwKCAjw1K-zBhBIEiwAWeCOF0lwzLIS76IvYTcWyzQOw935SmbZgRafDYLekOCalqkzT29SVp0WOxoC_FIQAvD_BwE">OpsRamp's</a> IT operations are integrated with HPE GreenLake cloud, HPE’s hybrid cloud platform designed to help organizations unlock the power of data and AI, with OpsRamp delivering observability and AIOps to all HPE products and services. <a href="https://www.hpe.com/us/en/newsroom/press-release/2024/06/hewlett-packard-enterprise-and-nvidia-announce-nvidia-ai-computing-by-hpe-to-accelerate-generative-ai-industrial-revolution.html">Get additional details about the solution configurations in the press release</a>.</p> <p>The solution takes into account the gravity of data and its impact on AI and how private clouds can best serve enterprise AI solutions. Read more about the case for private clouds in AI deployment in this blog post, <a href="https://community.hpe.com/t5/ai-unlocked/hpe-private-cloud-ai-a-strategic-necessity-for-enterprises/ba-p/7217830">HPE Private Cloud AI: A strategic necessity for enterprises</a>.</p> <p>For more information on HPE Private Cloud AI and NVIDIA AI computing by HPE, please visit the HPE.com <a href="https://www.hpe.com/us/en/private-cloud-ai.html">HPE Private Cloud AI home page</a>.</p> <h2>Offering more choice</h2> <h3>New KVM offered by HPE</h3> <p>As part of HPE’s desire to offer its customers “optionality”, HPE also announced its decision to enter the hypervisor market with its own open-source virtualization capability. Executive Vice President and CTO Fidelma Russo explained that the decision to release one was in response to partner and customer demand. Providing the new HPE KVM hypervisor virtualization capability provides customers and partners with a choice in the virtualization market. The market response has been quite positive, with numerous industry outlets picking up on the news, such as <a href="https://www.crn.com/news/virtualization/2024/cto-fidelma-russo-on-customer-choice-trust-and-why-hpe-now-has-its-own-virtualization-capability">this article</a> found in CRN.</p> <h3>Addressing regulated environments</h3> <p>HPE also announced that a significant effort being made to address the needs of highly secure and regulated environments with its new dedicated platform. For customers who have data residency and sovereignty requirements and are prohibited from connecting to the public internet or external networks, HPE GreenLake will be offering a fully disconnected/isolated private cloud management platform, allowing customers to consume HPE services in a cloud operating model. You can find more details about this offering in this blog post, <a href="https://community.hpe.com/t5/the-cloud-experience-everywhere/empowering-regulated-environments-with-hpe-greenlake-cloud/ba-p/7217339">Empowering regulated environments with HPE GreenLake cloud</a>. More can be found on this solution, as well as other enhancements to HPE GreenLake cloud, in this article, <a href="https://community.hpe.com/t5/the-cloud-experience-everywhere/what-s-new-expanded-platform-capabilities-of-the-hpe-greenlake/ba-p/7217343">What's new: Expanded platform capabilities of the HPE GreenLake cloud</a>.</p> <h2>Summary</h2> <p>HPE Discover 2024 was truly an event unlike any other. Whether it was being in one of the 17,000 seats at the Sphere to take in Antonio’s keynote, walking the floor through HPE’s Innovation neighborhood, or interacting with an <a href="https://x.com/hpe/status/1803965843707170980?s=46">AI avatar of Antonio Neri</a> himself, the experience was indeed memorable. To catch any of the replays of keynote sessions or interviews with the Cube, be sure to check out the <a href="https://www.youtube.com/watch?v=-FEzi51gs9g">YouTube video library</a>.</p>https://developer.hpe.com/hpe-sustainability-insight-center-api-at-a-glance/https://developer.hpe.com/hpe-sustainability-insight-center-api-at-a-glance/Mon, 24 Jun 2024 12:44:18 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>HPE GreenLake provides a new service called HPE Sustainability Insight Center that can assist you in obtaining detailed information about carbon emissions, energy consumption and the cost of the infrastructure that is managed by HPE GreenLake. In this blog post, I will explain how to extract data from this service in a programmatic way using cURL and the HPE Sustainability Insight Center API.</p> <h2>What is HPE Sustainability Insight Center?</h2> <p>HPE Sustainability Insight Center is a service that runs on HPE GreenLake. You can add it into my workspace from the HPE GreenLake catalog under the Management &#x26; Governance category.</p> <p><img src="/img/sic-blog-1.jpg" alt="HPE GreenLake catalogue" title="HPE GreenLake catalogue"></p> <p>Once deployed, the service provides a dashboard, which can be used to monitor carbon emissions, energy consumption, and energy cost. You can get more details about HPE Sustainability Insight Center from the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-64E2035A-138E-44E8-8A04-7968A272A97E.html">User Guide</a>.</p> <p><img src="/img/sic-blog-2.jpg" alt="HPE Sustainability Insight Center Console" title="HPE Sustainability Insight Center Console"></p> <p>In addition to the dashboard, HPE Sustainability Insight Center provides an API, to enable programmatically accessing this same data.</p> <h2>What can I do with the HPE Sustainability Insight Center API?</h2> <p>Important use cases covered by the HPE Sustainability Insight Center API include:</p> <ul> <li><strong>Achieving sustainability goals</strong> — Use data retrieved from the HPE Sustainability Insight Center API to measure your organization's power consumption and carbon footprint. With this data, your organization can make data-infused, informed decisions to reduce its climate impact and ensure its IT assets operate in a way that meets regulatory and business environmental sustainability goals.</li> <li><strong>Monitoring IT costs</strong> — Use data available from the HPE Sustainability Insight Center API to rationalize the power consumption of IT assets operations for cost efficiency. This will free up budget for innovative and growth-orientated investments.</li> <li><strong>Integrating into reporting workflows</strong> — Use data retrieved from the HPE Sustainability Insight Center to incorporate into existing analytics, reporting, dashboards, and forecasting workflows to give your organization a robust understanding of its IT operations.</li> </ul> <h2>Where is the HPE Sustainability Insight Center API documented?</h2> <p>The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/sustainability/public/">API specifications</a> are found on the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake Developer Portal</a> along with other HPE GreenLake APIs. For HPE Sustainability Insight Center, there are 3 API calls available:</p> <ul> <li><strong>usageByEntity</strong>: Retrieves an aggregated energy usage list grouped by individual entities over a defined time frame.</li> <li><strong>usageTotals</strong>: Returns the total aggregated power cost, power consumption, and carbon emissions over a defined time frame and supports filtering by entities.</li> <li><strong>usageSeries</strong>: Retrieves aggregated energy usage statistics grouped by time bucket over a defined time frame and supports filtering by entities.</li> </ul> <blockquote> <p>Note: You can download the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/sustainability/public/openapi/sustainability-insight-ctr-latest/overview/">OpenAPI specs</a> from the <a href="https://developer.greenlake.hpe.com/">HPE GreenLake developer portal</a>, if you want to use the API from a tool such as Postman.</p> </blockquote> <h2>Be careful with tokens!</h2> <p>A little word of advice about tokens. Because the HPE Sustainability Insight Center is a service, you will need to create a dedicated API client credentials for it in your workspace (under Manage Workspace/API). You cannot use API client credentials created for the platform, as it will result in a 403-error code. Also, you’ll need to be careful, as API client credentials are region specific. When creating an API client credentials, say for Europe (EU Central), make sure to use it to call the API endpoint for that region, as shown below (Connectivity Endpoint).</p> <p><img src="/img/sic-blog-3.jpg" alt="HPE GreenLake platform API client credentials" title="HPE GreenLake platform API client credentials"></p> <h2>Let's give it a try!</h2> <p>Let's start with the first of the 3 calls, <strong>usageByEntity</strong>. From the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/sustainability/public/openapi/sustainability-insight-ctr-latest/operation/getUsageByEntity">documentation</a>, note that only 2 parameters are mandatory, <em>start-time</em> and <em>end-time</em>.</p> <blockquote> <p>Note: the format of the date used by the API, which is ISO 8601 of the form: YYYY-MM-DDTHH:MM:SS.ss-/+FF:ff. For example: '2023-07-24T04:21:22.00Z' for 4:21AM on the 24th of July, 2023 in UTC (Z=Zero Meridian)</p> </blockquote> <p>So, you could use the following command in Bash:</p> <pre><code class="language-bash">$ curl -s -X GET 'https://eu-central.api.greenlake.hpe.com/sustainability-insight-ctr/v1beta1/usage-by-entity?end-time=2024-06-11T08%3A00%3A00Z&#x26;start-time=2024-06-11T08%3A00%3A00Z'-H 'Authorization: Bearer &#x3C;my-token>' -H "Accept:application/json" | jq { "items": [ { "id": "COMPUTE867962-B21CZJ93402YV", "type": "sustainability-insight-ctr/entities", "entityId": "COMPUTE867962-B21CZJ93402YV", "entityMake": "HPE", "entityModel": "ProLiant DL360 Gen10", "entityType": "COMPUTE", "entitySerialNum": "CZJ93402YV", "entityProductId": "867962-B21", "entityManufactureTimestamp": "2024-05-25T06:43:32.934Z", "locationName": "Sophia Antipolis", "locationId": "d1386776-e49d-4333-9ccc-c4bfc2029f40", "locationCity": "Mougins", "locationState": "PACA", "locationCountry": "France", "name": "centos82rf2", "costUsd": 3.1437287, "co2eMetricTon": 0.0015487487, "kwh": 23.115652 }, { "id": "IAPVariousVarious", "type": "sustainability-insight-ctr/entities", "entityId": "IAPVariousVarious", "entityMake": "HPE", "entityModel": "Aggregated HPE Aruba access points", "entityType": "IAP", "entitySerialNum": "Various", "entityProductId": "Various", "entityManufactureTimestamp": "2024-04-25T09:32:06.996Z", "locationName": null, "locationId": null, "locationCity": null, "locationState": null, "locationCountry": null, "name": "", "costUsd": 0.77483493, "co2eMetricTon": 0.0021369294, "kwh": 4.842718 } ], "count": 2, "total": 2, "offset": 0 } </code></pre> <p>Here, you can see from the JSON response obtained for energy cost, CO2 emission and kWh consumption for an HPE ProLiant DL360 and a group of Aruba access points over the selected period.</p> <blockquote> <p>Note: You can also try this from the HPE GreenLake Developer Portal by using the <strong>Try it</strong> function:</p> </blockquote> <p><img src="/img/sic-blog-4.jpg" alt="HPE GreenLake Developer Portal" title="HPE GreenLake Developer Portal"></p> <p>The next call to try out is <strong>usage-totals</strong>, which is documented <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/sustainability/public/openapi/sustainability-insight-ctr-latest/operation/getUsageTotals">here</a>. The parameters of the call are the same as in the previous call, so let's give it a try:</p> <pre><code class="language-bash">$ curl -s -X GET 'https://eu-central.api.greenlake.hpe.com/sustainability-insight-ctr/v1beta1/usage-totals?end-time=2024-06-11T08%3A00%3A00Z&#x26;start-time=2024-06-01T00%3A00%3A00Z' -H 'Authorization: Bearer &#x3C;my-token>' -H "Accept:application/json" | jq { "items": [ { "type": "sustainability-insight-ctr/totals", "costUsd": 3.9185636, "co2eMetricTon": 0.0036856781, "kwh": 27.95837 } ], "count": 1 } </code></pre> <p>This call returns the totals Cost, CO2, and kWh for the complete environment. You can apply filters, such as:</p> <ul> <li><code>filter=locationCountry eq 'France'</code> to reduce scope to only devices located in France</li> <li><code>filter=entityType eq 'COMPUTE'</code> to reduce scope to only COMPUTE devices</li> </ul> <p>Let’s try these two:</p> <pre><code class="language-bash">$ curl -s -X GET 'https://eu-central.api.greenlake.hpe.com/sustainability-insight-ctr/v1beta1/usage-totals?end-time=2024-06-11T08%3A00%3A00Z&#x26;start-time=2024-06-01T00%3A00%3A00Z&#x26;filter=locationCountry%20eq%20%27France%27' -H 'Authorization: Bearer &#x3C;my-token>' -H "Accept:application/json" | jq { "items": [ { "type": "sustainability-insight-ctr/totals", "costUsd": 3.1437287, "co2eMetricTon": 0.0015487487, "kwh": 23.115652 } ], "count": 1 } $ curl -s -X GET 'https://eu-central.api.greenlake.hpe.com/sustainability-insight-ctr/v1beta1/usage-totals?end-time=2024-06-11T08%3A00%3A00Z&#x26;start-time=2024-06-01T00%3A00%3A00Z&#x26;filter=entityType%20eq%20%27COMPUTE%27' -H 'Authorization: Bearer &#x3C;my-token>' -H "Accept:application/json" | jq { "items": [ { "type": "sustainability-insight-ctr/totals", "costUsd": 3.1437287, "co2eMetricTon": 0.0015487487, "kwh": 23.115652 } ], "count": 1 } </code></pre> <p>Finally, give the last call, <strong>usageSeries</strong>, a try. You can see from the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/sustainability/public/openapi/sustainability-insight-ctr-latest/operation/getUsageBySeries">documentation</a> that, in addition to <em>start-time</em> and <em>end-time</em>, there is an additional required parameter called <em>interval</em>. <em>interval</em> should be formatted as an integer value followed by a unit string. Units can be one of: day, hour, week, month, or year.</p> <p>Let's give this a try:</p> <pre><code class="language-bash">$ curl -s -X GET 'https://eu-central.api.greenlake.hpe.com/sustainability-insight-ctr/v1beta1/usage-series?end-time=2024-06-11T08%3A00%3A00Z&#x26;start-time=2024-06-01T00%3A00%3A00Z&#x26;interval=1%20day' -H 'Authorization: Bearer &#x3C;my-token>' -H "Accept:application/json" | jq { "items": [ { "id": "2024-06-01T00:00:00.000Z",: "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-01T00:00:00.000Z", "costUsd": 0.403164, "co2eMetricTon": 0.00041689575, "kwh": 2.858163 }, { "id": "2024-06-02T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-02T00:00:00.000Z", "costUsd": 0.40312916, "co2eMetricTon": 0.0004167917, "kwh": 2.8579495 }, { "id": "2024-06-03T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-03T00:00:00.000Z", "costUsd": 0.4038577, "co2eMetricTon": 0.00041880895, "kwh": 2.8624988 }, { "id": "2024-06-04T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-04T00:00:00.000Z", "costUsd": 0.40462032, "co2eMetricTon": 0.00042090417, "kwh": 2.8672693 }, { "id": "2024-06-05T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-05T00:00:00.000Z", "costUsd": 0.40372863, "co2eMetricTon": 0.00041844492, "kwh": 2.861696 }, { "id": "2024-06-06T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-06T00:00:00.000Z", "costUsd": 0.4039479, "co2eMetricTon": 0.00041905773, "kwh": 2.8630626 }, { "id": "2024-06-07T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-07T00:00:00.000Z", "costUsd": 0.30680534, "co2eMetricTon": 0.00015114676, "kwh": 2.2559216 }, { "id": "2024-06-08T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-08T00:00:00.000Z", "costUsd": 0.4034695, "co2eMetricTon": 0.00041773028, "kwh": 2.8600764 }, { "id": "2024-06-09T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-09T00:00:00.000Z", "costUsd": 0.40351078, "co2eMetricTon": 0.00041754392, "kwh": 2.8604808 }, { "id": "2024-06-09T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-10T00:00:00.000Z", "costUsd": 0.40351078, "co2eMetricTon": 0.00041754392, "kwh": 2.8604808 }, { "id": "2024-06-10T00:00:00.000Z", "type": "sustainability-insight-ctr/timeseries", "timeBucket": "2024-06-11T00:00:00.000Z", "costUsd": 0.3068089, "co2eMetricTon": 0.0001511485, "kwh": 2.2559478 }, ], "count": 11 } </code></pre> <p>You can see that I get one measure per day from June 1 at 00:00 until June 11th.</p> <h2>Using this data externally</h2> <p>You can use another set of data collected every hour, over a period of two days, within a tool such as Excel:</p> <p><img src="/img/sic-blog-5.jpg" alt="Data imported in Excel" title="Data imported in Excel"></p> <p>And start providing graphs according to your needs:</p> <p><img src="/img/sic-blog-graphs.jpg" alt="Excel graphs" title="Excel graphs"></p> <p>You could even go a step further and feed this time-series data into <a href="https://www.elastic.co/elasticsearch">Elasticsearch</a> or <a href="https://www.influxdata.com/">influxdb</a> to build more advanced dashboards with <a href="https://www.elastic.co/kibana">Kibana</a> or <a href="https://grafana.com/">Grafana</a>. The capture below shows an example of influxdb dashboard showing kWh, CO2 and cost over time.</p> <p><img src="/img/sic-blog-9.jpg" alt=""></p> <h2>Next step</h2> <p>In this article, I described how to retrieve carbon emission, energy consumption, and cost of an Infrastructure managed with HPE GreenLake using the API for the HPE Sustainability Insight Center. I used Bash and cURL, but you could do the same using PowerShell or Python. For more details about how to use these two languages with HPE GreenLake, please check out my previous blog posts:</p> <ul> <li><a href="https://developer.hpe.com/blog/hpe-greenlake-edge-to-cloud-platform-scripting-fundamentals/">HPE GreenLake edge-to-cloud platform scripting fundamentals</a></li> <li><a href="https://developer.hpe.com/blog/bulk-onboarding-of-users-in-hpe-greenlake-edge-to-cloud-platform/">Bulk onboarding of users in HPE GreenLake edge-to-cloud platform</a></li> </ul> <p>If you’re interested in trying out the HPE GreenLake API, you might first want to check out one of our hands-on Workshops-on-Demand. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested in and go! It’s as simple as that.</p> <p>If you still have any questions regarding the HPE GreenLake APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-mastering-cloud-migration-with-the-6rs-approach/https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-mastering-cloud-migration-with-the-6rs-approach/Thu, 13 Jun 2024 17:36:32 GMT<p>In today's digital landscape, organizations are increasingly turning to the cloud to drive innovation, agility, and efficiency. However, the journey to the cloud is fraught with challenges, including complex legacy systems, data security concerns, and the need for uninterrupted operations. HPE GreenLake for Private Cloud Enterprise addresses these hurdles with its comprehensive 6Rs approach to cloud migration and management.</p> <p>The 6Rs—<strong>Rehost</strong>, <strong>Replatform</strong>, <strong>Refactor</strong>, <strong>Repurchase</strong>, <strong>Retire</strong>, and <strong>Retain</strong>—provide a structured framework that ensures a seamless transition to the cloud. This blog post offers an in-depth look into each R's methodology, outlining a step-by-step guide that helps organizations optimize their cloud migration strategies and enhance their cloud management capabilities.</p> <p>This post is the first part of a multipart series covering migrations in depth. Below, I will delve into each of the 6Rs, exploring their unique roles and the value they bring to the migration process.</p> <h2>Industry need and challenges in cloud migration </h2> <p>As organizations strive to stay competitive, the shift towards cloud adoption becomes imperative. The need for scalable infrastructure, cost optimization, and enhanced operational efficiency drives this transformation. However, the migration journey is not without its obstacles. Developers and IT teams often grapple with:</p> <ul> <li>Complex legacy systems: Modernizing outdated and intricate legacy systems without disrupting business operations.</li> <li>Data security concerns: Ensuring robust security measures to protect sensitive data during and after migration.</li> <li>Operational continuity: Maintaining uninterrupted service delivery throughout the migration process.</li> </ul> <h2>How HPE GreenLake addresses these challenges</h2> <p>HPE GreenLake for Private Cloud Enterprise is designed to tackle these challenges effectively. By leveraging the 6Rs framework, HPE GreenLake offers:</p> <ul> <li>Tailored migration strategies: Customized solutions that align with organizational goals and existing IT landscapes.</li> <li>Enhanced security: Advanced security protocols and compliance measures to safeguard data integrity.</li> <li>Seamless integration: Tools and services that ensure a smooth transition with minimal downtime, preserving business continuity.</li> </ul> <h2>Let’s take a look at each of the 6Rs individually and how HPE GreenLake for Private Cloud Enterprise can help</h2> <p><strong>1. Rehosting</strong></p> <ul> <li><strong>Goal:</strong> Efficiently migrate applications to the cloud without making changes.</li> <li><strong>Strategy</strong>: Use HPE GreenLake for Private Cloud Enterprise's integration capabilities for a comprehensive application inventory and assessment to identify suitable candidates for migration.</li> <li><strong>Outcome:</strong> Achieve faster migration, immediate cost reduction, and enhanced scalability by relocating applications to a more flexible and scalable cloud environment.</li> </ul> <p><strong>2. Replatforming</strong></p> <ul> <li><strong>Goal</strong>: Ensure cloud compatibility with minimal application adjustments.</li> <li><strong>Strategy</strong>: Utilize HPE GreenLake for Private Cloud Enterprise's adaptable cloud environment to make minor alterations to applications, ensuring they benefit from cloud features.</li> <li><strong>Outcome:</strong> Attain improved cloud performance and long-term operational efficiency by ensuring applications operate optimally in a cloud environment.</li> </ul> <p><strong>3. Rearchitecting</strong></p> <ul> <li><strong>Goal:</strong> Revamp applications to maximize cloud functionality.</li> <li><strong>Strategy</strong>: Utilize HPE GreenLake for Private Cloud Enterprise's tools and resources to redesign and reconstruct applications, integrating cloud-native features for optimal performance.</li> <li><strong>Outcome:</strong> Realize enhanced application functionality, scalability, and superior cloud performance by leveraging tailored, cloud-centric designs.</li> </ul> <p><strong>4. Repurchasing</strong></p> <ul> <li><strong>Goal:</strong> Transition to cloud-optimized applications.</li> <li><strong>Strategy</strong>: Evaluate existing applications and select from HPE GreenLake for Private Cloud Enterprise's cloud service offerings for better cloud compatibility and efficiency.</li> <li><strong>Outcome</strong>: Streamline operations and bolster cloud compatibility and performance by adopting applications designed for the cloud environment.</li> </ul> <p><strong>5. Retire</strong></p> <ul> <li><strong>Goal:</strong> Eliminate redundant or underutilized applications.</li> <li><strong>Strategy:</strong> Use HPE GreenLake for Private Cloud Enterprise's tools for an exhaustive application assessment to identify and safely retire unnecessary applications.</li> <li><strong>Outcome:</strong> Minimize costs and enhance overall efficiency by discontinuing the use of unneeded applications, freeing up resources for critical operations.</li> </ul> <p><strong>6. Retain</strong></p> <ul> <li><strong>Goal:</strong> Identify and maintain applications that are not suitable for cloud migration.</li> <li><strong>Strategy:</strong> Utilize HPE GreenLake for Private Cloud Enterprise's tools for an in-depth risk and performance assessment to identify applications best kept off-cloud.</li> <li><strong>Outcome:</strong> Ensure sustained application performance and security by making informed decisions on which applications to keep in their current environment.</li> </ul> <h2>Pilots/POC</h2> <p>We recommend that you test out some of these to get a sense as to how HPE GreenLake for Private Cloud Enterprise can assist you in modernizing your environment through a pilot or proof-of-concept (POC) approach. A recommended methodology would be as follows:</p> <ul> <li><strong>Approach:</strong> Execute 2-3 pilot projects or proofs of concept for each R. Begin with <strong>Rehosting</strong> and <strong>Replatforming</strong> to assess and refine the migration process.</li> <li><strong>Outcome:</strong> Gather valuable metrics and insights for evaluating migration success and failure, paving the way for continuous improvement in migration strategies.</li> </ul> <h2>Conclusion</h2> <p>By embracing the structured approach of the HPE GreenLake for Private Cloud Enterprise 6Rs Migration Strategy, organizations can ensure a smooth and effective cloud migration journey. HPE GreenLake for Private Cloud Enterprise's approach delivers enhanced performance, scalability, and cost benefits, contributing to the overall success and growth of the business.</p> <p> For more insights and to stay updated on the remaining parts of our blog series, connect back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community portal</a>. Don't miss out on the in-depth discussions and expert advice that will help you navigate your cloud migration journey successfully.</p>https://developer.hpe.com/activation-memory-a-deep-dive-using-pytorch/https://developer.hpe.com/activation-memory-a-deep-dive-using-pytorch/Wed, 12 Jun 2024 15:15:13 GMT<p>External blog post</p>https://developer.hpe.com/2024-june-06/https://developer.hpe.com/2024-june-06/Thu, 06 Jun 2024 13:00:00 GMThttps://developer.hpe.com/glc-scim-api-integration-with-okta-scim-adapter-1/https://developer.hpe.com/glc-scim-api-integration-with-okta-scim-adapter-1/Mon, 27 May 2024 17:30:08 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>I am excited to announce that HPE have developed and promoted a way to bring users and groups from Okta to HPE GreenLake Flex Solutions. Enterprises will be now able to sync and bring all users and groups from Okta into HPE GreenLake Flex Solutions.</p> <p>In this blog post, I will walk you through the process of configuring Okta SCIM adapter to sync users and groups over to HPE GreenLake Flex Solutions.</p> <h2>Okta (SCIM) Adapter</h2> <p>You can synchronize users and groups from your Okta identity management service to HPE GreenLake Flex Solutions using the Okta System for Cross-domain Identity Management (SCIM) adapter.</p> <p>The Okta SCIM adapter application can be installed from the Okta Integration Network (OIN) into your Okta implementation to allow for integration with a SCIM-compliant API. Any user that needs to be synchronized to HPE GreenLake Flex Solutions must be assigned to the Okta SCIM adapter application in your Okta implementation. Groups whose memberships need to be synced to HPE GreenLake Flex Solutions must be added as a Push Group in the application. Users can be assigned to the application using the same groups that are synchronized to HPE GreenLake Flex Solutions.</p> <h1>Configuring a SCIM application in Okta.</h1> <ul> <li> <p><strong>Step 1:</strong> In the Okta Admin Console, deploy an application from the app catalog:<br> a. Go to <strong>Applications</strong> > <strong>Browse App Catalog</strong>.<br> b. In the search bar type SCIM 2.0, and find the app called: SCIM 2.0 Test App (OAuth Bearer Token).<br> c. Select the application and then click <strong>Add Integration</strong>.</p> </li> <li> <p><strong>Step 2:</strong> In the Add Scim2.0 Test App page, do the following:<br> a. Change the application label name if you want and make sure <strong>Automatically log in when user lands on login page</strong> is checked.</p> </li> </ul> <p><img src="/img/scimgeneral.png" alt=""></p> <br /> <p>b. Click <strong>Next</strong>. On the following page, click <strong>Done</strong>.</p> <p><strong>Step 3:</strong> After the application is created, configure the integration:<br> a. Click the <strong>Provisioning</strong> tab, then select <strong>Configure API Integration</strong>.<br> b. Select the <strong>Enable API Integration</strong> check box.<br> c. In the SCIM 2.0 Base Url field, enter: <a href="https://sps.us1.greenlake-hpe.com/v1alpha1/scimproxy">https://sps.us1.greenlake-hpe.com/v1alpha1/scimproxy</a>.<br> d. In the OAuth Bearer Token field: to create long-lived tokens for user provisioning, see step 2 and step 3 of the blog post <a href="https://developer.hpe.com/blog/configuring-azure-ad-with-long-term-token-for-scim-provisiong/">Configuring Azure Active Directory with long-lived tokens for user provisioning</a>.<br> e. Uncheck the box for the Import Groups option.<br> f. Test that the URL and token are valid by clicking <strong>Test API Credentials</strong>, then click <strong>Save</strong>. If everything is correct, the following message is shown:</p> <p><img src="/img/scimtest.png" alt=""></p> <ul> <li> <p><strong>Step 4:</strong> Configure the synchronization settings:<br> a. Under the <strong>Provisioning</strong> tab > <strong>To App</strong> section, enable these settings:</p> <ul> <li>Create Users</li> <li>Deactivate Users</li> </ul> </li> </ul> <p><img src="/img/scim2app.png" alt=""></p> <p>b. Select the six attributes shown in the following screenshot and discard the rest.</p> <p><img src="/img/attributes.png" alt=""></p> <p>c. Assign the group you want to synchronize to HPE GreenLake Flex Solutions to the SCIM application under the <strong>Application</strong> > <strong>Assignments</strong> tab and add it as a push group in the <strong>Push Groups</strong> tab.</p> <p><strong>Assignments</strong> tab:</p> <p><img src="/img/scim-group.png" alt="" title="Assignments tab"></p> <p><strong>Push Groups</strong> tab:</p> <p><img src="/img/scim-push.png" alt="" title="Push Groups tab:"></p> <p><strong>Please note:</strong> Adding the Group Everyone group to the SCIM application could have unintended effects on all users.</p> <p>These are all the steps required to configure a SCIM 2.0 application. Remember that users must be members of a group that is assigned to the SCIM application and that group must be included in a push group. Now all configured groups can be pushed into HPE GreenLake Flex Solutions via the Okta SCIM Adapter.</p> <p>Please return to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> for more tips and tricks on working with the HPE GreenLake platform.</p>https://developer.hpe.com/converting-hpe-greenlake-api-specifications-in-oas-3-1-using-openapi-tools/https://developer.hpe.com/converting-hpe-greenlake-api-specifications-in-oas-3-1-using-openapi-tools/Mon, 27 May 2024 09:42:19 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>What are the HPE GreenLake APIs for Data Services on the HPE GreenLake Platform?</h2> <p>These HPE GreenLake APIs are multiple sets of APIs that enable client applications to manipulate the REST API resources available as part of Data Services on HPE GreenLake edge-to-cloud platform. For more information about Data Services on HPE GreenLake platform, see the <a href="https://developer.hpe.com/greenlake/data-services-on-the-hpe-greenlake-platform/home/">Data Services landing page</a> in the HPE Developer Forum. The landing page provides <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003533en_us&#x26;page=ps_api_dscc.html">information on accessing the documentation</a> for each set of HPE GreenLake APIs related to the data services on the HPE GreenLake platform.</p> <h2>How can I use the HPE GreenLake APIs documentation for my automation projects?</h2> <p>All HPE GreenLake APIs provide example code snippets from different programming or scripting languages in the interactive website. All adopters can make use of the code snippets to create any client application or to create any client library for their development purposes. For early adopters, this method is the quickest and the easiest way to get started with their automation script. As the time goes on, you will see more <a href="https://developer.hpe.com/blog/">blog posts</a> in HPE Developer forum that provide comprehensive code samples to address common use.</p> <p><img src="/img/examples-provided-in-the-documentation.png" alt="Code snippets in multiple languages including JavaScript, Python, GO, and cURL"></p> <p><em>The previous figure displays the request and response samples pane that is present on every API documentation page in the <a href="https://developer.greenlake.hpe.com">HPE GreenLake Developer website</a>. The pane contains the examples of the particular API using the code in cURL, JavaScript, Python, or GO.</em></p> <h2>Do I have an alternative to use these REST APIs for my development?</h2> <p>From the documentation for each set of HPE GreenLake APIs, you will recognize that these APIs were documented using OpenAPI specification files in either JSON or YAML format. However, you will also notice that a single set of the HPE GreenLake APIs, called Data Services Cloud Console, is based on the OpenAPI Standard 3.0. However, the rest of the HPE GreenLake APIs are based on the OpenAPI Standard 3.1.</p> <p>If you decide to use a tool to perform the conversion of these OpenAPI spec files into a client Library for particular programming language or scripting, there are a couple of blog posts that discuss how to convert these files into <a href="https://developer.hpe.com/blog/introducing-python-sdk-for-dscc/">Python</a> or <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-powershell-sdk/">PowerShell</a> client libraries.</p> <p>This method enables an agile adoption of the HPE GreenLake APIs because many of these HPE GreenLake APIs, as of this blog post, are ongoing development cycle. So there is expectation that the existing APIs will be updated, deprecated, or new resources will be introduced. This method opens the opportunity to deploy code development automation by pushing the generated client Library into the GitHub as the new spec is made available.</p> <p>For more information about the versioning for the APIs based on the OpenAPI Standard 3.1, see this <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/public/standards/versioning_basics/">link</a>.</p> <h2>First tool of the day: a converter from OpenAPI Standard 3.1 to the OpenAPI Standard 3.0</h2> <p>The OpenAPI initiative provides a framework to describe any APIs so that these APIs can be consumed by different organizations for documentation, client side, server-side mocks, and many other opportunities. This framework has evolved from standard version 3.0 to version 3.1 with all the benefits as described in this <a href="https://www.youtube.com/live/Sflpzh_cAcA?si=zkAKqGNYQz-5C6oe">video</a>.</p> <p>As of this blog post, the HPE Developer forum uses <a href="https://openapi-generator.tech/">an open source tool</a> to generate client libraries for blog posts. However, the challenge of using this tool is that it only allows the Open API standard version 3.0 specification as the input. However, the majority of the HPE GreenLake APIs were documented using the OpenAPI Standard version 3.1.</p> <p>Let me introduce a tool to convert the spec file in OpenAPI standard 3.1 to a spec file in OpenAPI standard 3.0 to enable conversion using the <a href="https://www.npmjs.com/package/@openapitools/openapi-generator-cli">openapi-generator-cli</a>. The open-source tool is named <strong>apiture/openapi-down-convert</strong> by David Biesack and Mike Ralphson. It is documented in this GitHub <a href="https://github.com/apiture/openapi-down-convert">site</a>, and shown in the following figure.</p> <p><img src="/img/github-to-openapi-down-convert.png" alt="The openapi-down-convert GitHub website"></p> <p><em>The above figure shows the GitHub website for the documentation of the openapi-down-convert.</em></p> <p>To install this tool, follow the instructions on the GitHub <a href="https://github.com/apiture/openapi-down-convert">README</a> using the JavaScript package manager called the <strong>npm</strong>.</p> <p>A lot of the tools used in this part of the blog post are easier to use with the npm (JavaScript) software deployment method. I used the following steps to deploy the npm for JavaScript package management into a Microsoft Windows desktop environment. After these steps, you can deploy the openapi-down-convert tool.</p> <p>Before I started this, I made sure that my Microsoft Windows desktop had access to the internet and can connect to the websites that I provided in below steps. In some cases, you may need to define the internet proxy so that you can connect to these websites.</p> <p>Additionally, prior to deploying this OpenJDK, I removed any older version of the Java SDK or JRE from my Microsoft Windows environment. This ensures that the correct Java version was used for all the following steps.</p> <ol> <li>First, I deployed the Java SDK library into my Microsoft Windows desktop. My recommendation is to deploy the Microsoft Build version of the OpenJDK available from the Microsoft <a href="https://learn.microsoft.com/en-us/java/openjdk/download">website</a> using the instructions provided.</li> <li>Afterward, I set the <code>JAVA_HOME</code> system variable to <code>C:\Program Files\Eclipse Adoptium\jdk-21.0.1.12-hotspot</code> using the following <a href="https://confluence.atlassian.com/doc/setting-the-java_home-variable-in-windows-8895.html">instruction</a>. This step ensured that any applications that require Java would use the OpenJDK by default.</li> <li>Following the steps on the NodeJS <a href="https://nodejs.org/en">website</a>, I deployed the NodeJS package into my Microsoft Windows environment downloaded from the nodejs.org website. I also ensured that I included the <strong>npm package manager</strong> option as shown in one of the steps from the NodeJS installation wizard as shown in figure below.</li> </ol> <p><img src="/img/nodejs-deployment-ensure-npm-is-available.png" alt="The wizard section to deploy the npm package manager"></p> <ol start="4"> <li>After the deployment of NodeJS completed, I was able to use the npm CLI to install the <code>openapi-down-convert</code> in the Microsoft Windows CLI as follows.</li> </ol> <pre><code class="language-shell">C:\>npm I -g @apiture/openapi-down-convert added 5 packages in 4s npm notice npm notice New minor version of npm available! 10.5.0 -> 10.7.0 npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.7.0 npm notice Run npm install -g npm@10.7.0 to update! npm notice C:\> </code></pre> <ol start="5"> <li>After the preceding steps, I had the tool to convert any HPE GreenLake API spec files from OpenAPI Standard 3.1 to OpenAPI Standard 3.0.</li> </ol> <h2>Second tool of the day: deploying the openapi-generator-cli using the npm JavaScript</h2> <p>This time, I want to introduce a version of the openapi-generator that can be executed like any other CLI, <a href="https://www.npmjs.com/package/@openapitools/openapi-generator-cli">@openapitools/openapi-generator-cli</a> shared by the OpenAPI Initiative team. Because we have already deployed the npm package deployment tool, as described previously, I can proceed to deploy this tool quickly. These are steps that I took to deploy openapi-generator-cli:</p> <ol> <li>I opened a Microsoft Windows command line interface and issued the following npm CLI command:</li> </ol> <pre><code class="language-shell">C:\Users\Administrator>npm install -g @openapitools/openapi-generator-cli added 116 packages in 36s 23 packages are looking for funding run `npm fund` for details C:\Users\Administrator> </code></pre> <ol start="2"> <li>Afterward, I tested the deployed openapi-generator-cli to validate the version of the generator that was used. The openapi-generator-cli would use the latest published version of the openapi-generator-cli engine as default. At the time of the publication of this blog post, the latest published version which was 7.5.0 as shown below:</li> </ol> <pre><code class="language-shell"> C:>openapi-generator-cli Download 7.5.0 ... Downloaded 7.5.0 Did set selected version to 7.5.0 Usage: openapi-generator-cli &#x3C;command> [&#x3C;args>] Options: --openapitools &#x3C;openapitools.json> Use the specified openapi-generator-cli configuration file --custom-generator &#x3C;generator> Custom generator jar Commands: version-manager Manage used / installed generator version author Utilities for authoring generators or customizing templates. batch Generate code in batch via external configs. config-help Config help for chosen lang generate \[options] Generate code with the specified generator. help Display help information about openapi-generator list Lists the available generators meta MetaGenerator. Generator for creating a new template set and configuration for Codegen. The output will be based on the language you specify, and includes default templates to include. validate Validate specification version Show version information used in tooling C:> </code></pre> <ol start="3"> <li>With that, I had the tool to convert any of the HPE GreenLake APIs in OpenAPI Standard 3.0 to a client library for the popular scripting or programming languages listed in the <a href="https://openapi-generator.tech/docs/generators">website</a>.</li> </ol> <h2>But wait! What about converting those OpenAPI Standard 3.1 spec files to a client library?</h2> <p>Don't worry because I am not going to leave you stranded! 😊</p> <p>Now we have the required tools to create a pipeline to convert from the OpenAPI Standard 3.1 spec files to the client library for the scripting language of choice. An overview of the process:</p> <ol> <li>Download the OpenAPI spec file from the HPE GreenLake developer website.</li> <li>Convert the spec file to OpenAPI Standard 3.0 spec file.</li> <li>Convert the 3.0 spec file to the client library.</li> </ol> <p>Let me give you an example of converting the HPE GreenLake API for Data Services spec file to a PowerShell client library.</p> <ol> <li>I downloaded the HPE GreenLake API for Data Services OpenAPI Standard 3.1 spec file from the HPE GreenLake Developer website using the UI:</li> </ol> <p><img src="/img/download-the-openapi-3.1-spec-data-services-from-developer-website.png" alt="Download OpenAPI 3.1 JSON spec file from HP GreenLake Data Services"></p> <ol start="2"> <li>Afterward, I changed the name of the downloaded file from <code>swagger.json</code> to <code>GL-dataservices-31.json</code> so that I could recognize this OpenAPI Standard 3.1 spec file from HPE GreenLake APIs for Data Services.</li> <li>After I renamed the JSON file, I followed by using the <code>openapi-down-convert</code> tool to convert the OpenAPI Standard 3.1 to OpenAPI Standard 3.0 using the following command shown below. I named the converted file as <code>GL-dataservices-30.json</code>.</li> </ol> <pre><code class="language-shell"> PS C:\Users\Administrator\Downloads> move .\swagger.json ..\Scripting\GL-dataservices-31.json PS C:\Users\Administrator\Downloads> cd ..\Scripting\ PS C:\Users\Administrator\Scripting> dir Mode LastWriteTime Length Name - - - -a---- 5/2/2024 2:46 PM 171264 GL-dataservices-31.json PS C:\Users\Administrator\Scripting> openapi-down-convert -i .\GL-dataservices-31.json -o .\GL-dataservices-30.json PS C:\Users\Administrator\Scripting> </code></pre> <ol start="4"> <li>Now it’s the time to convert this OpenAPI Standard 3.0 spec file to the PowerShell client library using the <code>openapi-generator tool</code>. Additionally, I also changed the name of the generated package from the standard name <code>OpenAPITools</code> to the specific name <code>GLdataservices</code> using the special arguments.</li> </ol> <pre><code class="language-shell"> PS C:\Users\Administrator\Scripting> openapi-generator-cli generate -g powershell --additional-properties="packageName"="GLdataservices" -i .\GL-dataservices-30.json -o Posh-GL-dataservices </code></pre> <ol start="5"> <li>After the conversion was completed, I found a new folder named <code>Posh-GL-dataservices</code> with some files in the folder as shown below. It looks like that I have a Markdown file called <code>README.md</code>. I can use my favorite development editor, Microsoft Visual Studio Code, to investigate this generated PowerShell module.</li> </ol> <blockquote> <p><strong>Note:</strong> The information on how to install Microsoft Visual Studio Code is available at the Visual Studio Code <a href="https://code.visualstudio.com/">website</a>.</p> </blockquote> <pre><code class="language-shell"> PS C:\Users\Administrator\Scripting> cd .\Posh-GL-dataservices\ PS C:\Users\Administrator\Scripting\Posh-GL-dataservices> dir Mode LastWriteTime Length Name - - - d----- 5/2/2024 5:00 PM .openapi-generator d----- 5/2/2024 5:00 PM docs d----- 5/2/2024 5:40 PM src d----- 5/2/2024 5:00 PM tests -a---- 5/2/2024 5:00 PM 1040 .openapi-generator-ignore -a---- 5/2/2024 5:40 PM 1224 appveyor.yml -a---- 5/2/2024 5:40 PM 2100 Build.ps1 -a---- 5/2/2024 5:40 PM 12794 README.md PS C:\Users\Administrator\Scripting\Posh-GL-dataservices> </code></pre> <ol start="6"> <li>Using the Microsoft Visual Studio Code, I opened the folder where the modules were located. I could see the list of the files that were generated by the openapi-generator. The first file that I opened was the README.md, and using the keyboard command of <code>"CTRL + Left Shift + v"</code>, I was able to convert the README.md into a readable format. Further, I found out the instructions on installation, uninstallation, and detailed information on how to use this PowerShell module.</li> </ol> <p><img src="/img/hpe-gl-dataservices-posh-readme-md.png" alt="HPE GreenLake Data Services POSH README.md"></p> <blockquote> <p>By following the guides in README.md, I knew to click <code>documentation for API Endpoints</code> to display the information on how to use the API. For example, the <code>Invoke-ListAsyncOperations</code> that I decided to use for this blog post.</p> </blockquote> <p><img src="/img/hp-gl-data-services-help-for-async-events-list.png" alt="HPE GreenLake Data Services list of async operations"></p> <ol start="7"> <li>To use this PowerShell client library so that I can call <code>Invoke-ListAsyncOperations</code>, I followed the instructions to install the module into a PowerShell workstation as described in the <code>README.md</code> file.</li> </ol> <pre><code class="language-shell">To install from the source, run the following command to build and install the PowerShell module locally: C:> Build.ps1 C:> Import-Module -Name '.\src\GLdataservices' -Verbose </code></pre> <ol start="8"> <li>Once this module was loaded, I created a short script based on PowerShell to call <code>Invoke-ListAsyncOperations</code> and used that API to display a list of completed tasks.</li> </ol> <pre><code class="language-shell">$fileName = "..\myCredential-rrd1.json" $secretFile = Get-Content -Path $fileName | ConvertFrom-Json # general setting of the PowerShell module, e.g. base URL, authentication, etc $Configuration = Get-Configuration $Configuration.BaseUrl = "https://us1.data.cloud.hpe.com" $Configuration.Username = $secretFile | select-object -ExpandProperty myId $Configuration.Password = $secretFile | select-object -ExpandProperty mySecret $token_url = "https://sso.common.cloud.hpe.com/as/token.oauth2" $AuthenticationResult = Invoke-WebRequest $token_url -Method Post -Body @{ grant_type = "client_credentials" client_id = $Configuration.Username client_secret = $Configuration.Password } $Configuration.AccessToken = $AuthenticationResult.Content | ConvertFrom-Json | select-object -ExpandProperty access_token # general setting of the PowerShell module, e.g. base URL, authentication, etc $Filter = "'backup-and-recovery' in services" # String | The UUID of the object $Select = 'associatedResources,services,displayName,logMessages' # String | A list of properties to include in the response. (optional) # Returns details of a specific async-operation try { $Result = Invoke-ListAsyncOperations -Offset 0 -Limit 10 -Filter $Filter -Select $Select } catch { Write-Host ("Exception occurred when calling Invoke-ListAsyncOperations: {0}" -f ($_.ErrorDetails | ConvertFrom-Json)) Write-Host ("Response headers: {0}" -f ($_.Exception.Response.Headers | ConvertTo-Json)) } $Result.items | ConvertTo-Json </code></pre> <blockquote> <p><strong>Note:</strong> The preceding script read a separate <code>client-credentials</code> file so that I could gain authorization to my HPE GreenLake workspace. That way, I didn't have to include my <code>client-secrets</code> and <code>client-id</code> into this script file to ensure proper secure coding. This file called <code>myCredentials-rrd1.json</code> which contains the JSON structure shown below. For more information on providing this client-credentials information, please see the HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/#authentication">website</a>. There is also a blog <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">post</a> in HPE Developer Forum website that describes the process as well.</p> </blockquote> <pre><code class="language-JSON">{ "myWorkspace": "xxxxxxxxxx-yyyy-yyyy-yyyy-zzzzzzzzzzzz", "myId": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "mySecret": "06cffff699deeeee9f1d92f7gggggggg" } </code></pre> <blockquote> <p>As expected, the execution of the above script completed successfully. The API response returned the list of tasks based on the filtering and selection properties from the example. These were task lists from my workspace created by the backup-and-recovery services in my workspace.</p> </blockquote> <pre><code class="language-JSON">[ { "associatedResources": [], "services": [ "backup-and-recovery" ], "displayName": "Create_individual_file_backup", "logMessages": [ "@{message=Performing incremental copy.; timestamp=04/27/2024 09:01:22}", "@{message=Resources acquired, volume copy preparation done.; timestamp=04/27/2024 09:01:23}", "@{message=Submitted all copy jobs to data mover.; timestamp=04/27/2024 09:02:31}", "@{message=Initiated Vmdk copy operation.; timestamp=04/27/2024 09:02:33}", "@{message=Failed with retryable error code: -3403. Retrying after 1 minutes.; timestamp=04/27/2024 09:07:17}", "@{message=Performing optimized copy since Invalid parent backup: Parent backup with id 61c86d49-2b7c-4e0a-9619-dc9bda08586d is in Reading state.; timestamp=04/27/2024 09:08:18}", "@{message=Resetting retries as data movement successfully progressed after previous failure.; timestamp=04/27/2024 09:21:55}", "@{message=Failed with retryable error code: -3403. Retrying after 15 minutes.; timestamp=04/27/2024 09:30:26}", "@{message=Allocation map collection took 0.002743 seconds with 1 threads.; timestamp=04/27/2024 10:58:21}", "@{message=Successfully completed copy operation; timestamp=04/27/2024 10:58:22}", "@{message=Backup operation completed.; timestamp=04/27/2024 10:58:23}" ] }, { "associatedResources": [ "@{name=0-VM-01-VVOL-DS; resourceUri=/hybrid-cloud/v1beta1/virtual-machines/1d1438c2-3ae2-52e0-b5a1-fa643903f526; type=hybrid-cloud/virtual-machine}" ], "services": [ "backup-and-recovery" ], "displayName": "Delete backup [0-VM-01-VVOL-DS - 13/04/2024 03:48:59]", "logMessages": [ "@{message=Job execution started.; timestamp=04/16/2024 08:03:26}", "@{message=Deleting local backup.; timestamp=04/16/2024 08:03:28}", "@{message=Deleting backup successful.; timestamp=04/16/2024 08:03:30}", "@{message=Job execution completed.; timestamp=04/16/2024 08:03:31}" ] }, { "associatedResources": [], "services": [ "backup-and-recovery" ], "displayName": "Delete MSSQL snapshot [Array_Snapshot_2024-04-27-21:35:19-K1cqhLAg - 2024-04-28T01:36:25.000Z]", "logMessages": [ "@{message=Updating snapshot state; timestamp=04/30/2024 02:01:05}", "@{message=Deleting volume(s) snapshot.; timestamp=04/30/2024 02:01:06}", "@{message=Deleting snapshot; timestamp=04/30/2024 02:01:18}", "@{message=Job execution completed.; timestamp=04/30/2024 02:01:18}" ] }, </code></pre> <h2>Conclusion</h2> <p>With the introduction of new HPE GreenLake APIs on March 2024, HPE GreenLake APIs embraces OpenAPI Standard 3.1 for the specification files. Examples of the APIs released based on the OpenAPI Standard 3.1 include:</p> <ul> <li>Data Services</li> <li>Virtualization</li> <li>Backup and Recovery</li> <li>Private Cloud Business Edition</li> </ul> <p>This blog post introduces the tools to enable any adopter to use the open-source software based openapi-generator to convert these spec files after the conversion from the OpenAPI Standard 3.1 to OpenAPI Standard 3.0. In this blog post, I provided an example on how to convert the HPE GreenLake API for Data Services into a PowerShell client library. Furthermore, I also provided an example on how to use this client library to display the list of the tasks from HPE GreenLake for Backup and Recovery from my workspace.</p> <blockquote> <p><strong>Note:</strong> that this conversion will not be required once the open-source tool such openapi-generator-cli eventually support OpenAPI 3.1 Standard spec in the future. For more information on the supported SDK conversion from OpenAPI Standard 3.1 using this open-source tool, please follow this update to <a href="https://github.com/OpenAPITools/openapi-generator/issues/9083">issues</a>. If you are using other (or commercial) OpenAPI SDK generator tool such as listed <a href="https://openapi.tools/#sdk">here</a>, you may not be required to use the conversion tool as discussed in this blog.</p> </blockquote> <p>Please don’t hesitate to explore the APIs for Data Services on the HPE GreenLake platform and see how you can improve your agility in managing your data. Any questions on HPE GreenLake Data Services Cloud Console API? Please join the HPE Developer Community Slack <a href="https://developer.hpe.com/slack-signup/">Workspace</a>, and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services</a> Slack channel.</p>https://developer.hpe.com/do-not-miss-the-innovation-campus-at-discover-2024/https://developer.hpe.com/do-not-miss-the-innovation-campus-at-discover-2024/Tue, 21 May 2024 14:09:57 GMT<p><img src="https://www.hpe.com/content/dam/hpe/shared-publishing/images-norend/discover/2024/backgrounds/Discover-2024-Sphere-Customer-Story-2-9-6-1.jpg.hpetransform/bounded-resize:width=800/image.webp" alt=""></p> <p>We all recognize in today’s tech world; things are moving faster than ever. And in this uber-accelerated tech landscape, innovation cannot be just a buzzword – it must be the driving force behind growth, competitiveness, and relevance.   </p> <p>At HPE, innovation is in our DNA.  We understand that staying ahead means embracing change, pushing boundaries and continually evolving, iterating and refining. Innovation permeates throughout HPE and that’s why the HPE Developer Community is excited to announce our evolved presence at <a href="https://www.hpe.com/us/en/discover.html">HPE Discover Las Vegas 2024</a>; a space that is dedicated to highlighting our company-wide focus and commitment to innovation – The Innovation Campus.  </p> <p>Differing from our traditional Hack Shack, the Innovation Campus will span a new neighbourhood on the event floor, providing an immersive experience that is designed to demonstrate our focus on innovation. The Campus will walk attendees through the various stages of the innovation process, inviting them to embark on their own journeys of discovery, advancement and transformation alongside us. </p> <p>While the format is different, the focus on collaboration and moving your business ahead remains the same.  From inspiring ideas to empowering individuals, each element of the Innovation Campus will offer a unique glimpse into the different facets of innovation:  </p> <h2>Ignite</h2> <h4>Investing in Technology Futures – Hewlett Packard Labs </h4> <p>While innovation happens across HPE, this is the epicenter that leads how we innovate and ensures we are making an impact. Our approach includes technology and industry horizon scanning, technology invention and customer co-creation. At this booth, attendees can explore technologies beyond the horizon to ignite their technology future. </p> <h2>Engage</h2> <h4>Co-creating Innovation – HPE Customer Innovation Centers </h4> <p>Our global customer innovation centers explore edge-to-cloud opportunities in a tailored engagement. Customer Engagement teams customize customer visits to bring your initiatives to life and advance your digital journey. This is a great opportunity for attendees to immerse themselves in emerging technologies and explore how to apply them to their innovation roadmaps. </p> <h2>Explore</h2> <h4>Uncovering What’s Next – Innovation Exploration Booth  </h4> <p>Building a culture of innovation ensures that every employee understands your innovation priorities. At HPE we empower intelligent and curated search and navigation of innovation engineering projects, ideas, and the startup / open-source ecosystem. We also disseminate innovation maps and gap analysis, providing valuable insights. At this booth, attendees will see a demonstration of how we brought this ecosystem to life and explore what it takes to build a tool to manage the beginning funnel for innovation. </p> <h2>Pioneer</h2> <h4>Innovating Through Partners – The Start-Up Innovators Pavilion </h4> <p>The next stage of discovering what’s next is forming partnerships with companies that are pioneering in your industry. In our Startup Pavilion, attendees will get ideas on how to build their partnership programs to contribute to their innovation initiatives. </p> <h2>Inspire</h2> <h4>See our Innovations in Action – The Innovation Theater </h4> <p>In the Innovation Theater, throughout each day we will have lightning rounds that take a deep dive into some of our technology explorations.  </p> <h2>Empower</h2> <h4>Investing in Incubation – The Innovation Studio </h4> <p>In the Innovation Studio, attendees can try out some of HPE innovation methodologies including customer empathy interviews and co-design, as well as explore AI innovation for their business with our subject matter experts.  </p> <h2>Transform</h2> <h4>Innovation Proof of Value – Ask the Experts and Digital Transformation Advisors </h4> <p>Our Transformation Advisors will be available at the Innovation Campus to assist attendees in advancing their digital transformation journey. How can you assess the impact of innovation to demonstrate its value to your organization? The key is to manage your innovation efforts as a portfolio of investments. Just like any well-managed portfolio, the decisions your company makes about what not to invest in are as crucial as those it does. Our advisors are here to help you consider and make those strategic innovation-led decisions. </p> <p>Through these interactive experiences, we hope to inspire, educate and empower attendees to embrace innovation and to either start, or continue their journey as innovators themselves.  We will be inviting all visitors to the Innovation Campus to become part of our Innovator Community, by booking into dedicated online Innovation Group sessions and in-person workshops at our HPE Customer Innovation Centers.  These workshops are dedicated to actively listening to our customers and collaboratively exploring the realm of possibilities. They’ll be devoted to undertaking a journey of evolution alongside HPE, characterized by exploration, advancement and solution-driven progress. </p> <p>For anyone attending <a href="https://www.hpe.com/us/en/discover.html">HPE Discover Las Vegas 2024</a>, we invite you to stop by the Innovation Campus and embark on this journey of discovery and transformation with us!</p> <p>Last but not least, you can meet two members of the HPE Developer Community team in the hands-on-Lab section, right next to the Innovation Campus. They will be welcoming you to take one of the multiple hands-on-labs available during the week, including two on the HPE GreenLake APIs.</p>https://developer.hpe.com/title-demystifying-machine-learning-at-scale-hpe-ezmeral-unified-analytics-in-action/https://developer.hpe.com/title-demystifying-machine-learning-at-scale-hpe-ezmeral-unified-analytics-in-action/Mon, 20 May 2024 20:24:58 GMT<p>In 2024, businesses are scuffling to seek innovative ways to leverage Generative AI to elevate customer experiences and streamline operational workflows. Traditional machine learning workflows, tools and trades are the bedrock on which the data-driven world is built – and no matter how loud the hype, that will not change anytime soon.</p> <p>Whilst generative AI (such as Large Language Models and ChatGPT) can be leveraged to deliver very human-like experiences to end customers, they quite often miss the mark on delivering actual, specific business value. Smaller, custom models that analyze, predict, and deliver based on relevant, tailored data will always win out when counted on to deliver invaluable insights and real, helpful customer experiences that surprise and delight. The challenge isn't just having the skills to wrangle data and develop models, but also setting up the necessary tools and infrastructure, which becomes especially difficult when dealing with large amounts of data.</p> <p>Enter HPE Ezmeral Unified Analytics – a single-touch, scalable deployment of the best-in-class open-source data, analytics, and machine learning tools on top of any infrastructure.</p> <p>In this blog post I'll take you on a journey using a smart retail experience that showcases the capabilities of HPE Ezmeral Unified Analytics to gather, process, and interpret retail data from stores across several European countries. This journey will show you how easy it is to work with data and models using HPE Ezmeral Unified Analytics and how you can leverage these tools to create cutting-edge customer experiences.</p> <p><img src="/img/demystifying-ml-unifiedanalytics-img1.png" alt="This screenshot shows the built-in ecosystem of open source tools segmented by data engineering, analytics, and data science personas. " title="HPE managed ecosystem of open source tools built into HPE Ezmeral Unified Analytics."></p> <p><strong>Embarking on the journey: Harnessing Apache Spark for customer insights</strong></p> <p>The journey commences with a fundamental need: understanding diverse customer shopping experiences across European regions utilizing Apache Spark on HPE Ezmeral Unified Analytics. The objective is clear yet ambitious: to analyze extensive sales data and derive actionable insights. <strong>Apache Spark</strong>, renowned for its agility and scalability in handling voluminous datasets, emerges as the natural choice. Seamlessly integrated within HPE Ezmeral Unified Analytics, it furnishes a scalable ecosystem for data processing, eliminating the complexities of manual setups.</p> <p><img src="/img/demystifying-ml-spark-img2.png" alt="This screenshot show the ability to create a spark session from the Unified Analytics user interface. " title="Simplified Apache Spark management from HPE Ezmeral Unified Analytics. "></p> <p>In the Smart Retail Experience, a Spark Interactive session powered by <strong>Apache Livy</strong> generates synthetic sales data reflecting diverse currencies, store locations and stores in delta tables for enhanced data reliability and managed version control.</p> <p>This initial phase of data prep-work lays the foundation for informed decision-making. Insights into trends, such as regional product preferences and seasonal sales peaks can now be unearthed, setting the stage for advanced, data-driven strategies.</p> <p><strong>Advancing with Apache Presto to streamline data connectivity</strong></p> <p>The journey progresses with <strong>Apache Presto</strong>, an integral component of HPE Ezmeral Software that amplifies Presto's capabilities, enabling the connection to and querying of various data sources seamlessly within a unified environment. With Presto, data from different stores can be amalgamated into a single view that you can easily manipulate and analyze. This seamless integration empowers real-time access to insights from data lakes, allowing for federated queries on data without worrying about the underlying complexity of tying each application to each data source individually.</p> <p><img src="/img/demystifying-ml-presto-img3.png" alt="This screenshot how HPE Ezmeral Software amplifies Presto&#x27;s capabilities by connecting to and querying of various data sources within a unified environment. " title="From a unified interface, quickly connect to and query multiple data sources then manipulate and analyze. "></p> <p>What's really impressive about Presto is its integration within the HPE Ezmeral Unified Analytics ecosystem, allowing not just for querying but directly connecting processed outputs for further analysis or visualization in tools like Apache Superset. The convenience of managing data sources, running queries, and directly utilizing this data across various stages of the MLOps pipeline underlines the unified approach of HPE Ezmeral Unified Analytics, streamlining workflows and eliminating the need for disjointed tool management.</p> <p><strong>Visualizing Insights: Unveiling data using Apache Superset</strong></p> <p>Next, visualization with <strong>Apache Superset</strong> illustrates the ease of creating engaging, insightful dashboards that outline pivotal business metrics, like the top-selling fruits, variation in sales over different years, or customer purchasing behavior. Clear visual storytelling delivers powerful insights, informing both store operations and strategic decision-making.</p> <p><img src="/img/demystifying-ml-superset-img4.png" alt="This shows a sample of visualization capabilities within HPE Ezmeral Unified Analytics. " title="Quick generate visualizations within HPE Ezmeral Unified Analytics to reduce redundant data preparation steps. "></p> <p>The ability to generate these visualizations from Cached Assets within the HPE Ezmeral Unified Analytics platform adds layers of efficiency—data that is queried and transformed through Presto can directly feed into Superset without redundant steps of data preparation.</p> <p><strong>Towards enhanced checkout experiences: Harnessing machine learning</strong></p> <p>Armed with substantial insights, attention turns to enhancing the customer checkout experience through machine learning. A MobileNetV2 object recognition model is trained using <strong>TensorFlow</strong> to recognize fresh produce via vision-based machine learning, presenting a transformative solution for streamlining checkout processes, particularly in self-service scenarios.</p> <p><img src="/img/demystifying-ml-mlflow-img5.png" alt="This screenshot highlights integration between MLflow and HPE Ezmeral Unified Analytics for managing model training, tracking, and version control. " title="Effortless integration between MLflow and HPE Ezmeral Unified analytics reduces the complexity of managing the machine learning lifecycle. "></p> <p>In the subsequent exercises, we delve into the model-building phase using TensorFlow, followed by MLflow for managing the machine learning lifecycle, including model training, tracking, and version control.</p> <p>HPE Ezmeral Unified Analytics seamless integration with <strong>MLflow</strong> via internally hosted Jupyter notebooks removes all the pain of managing the machine learning lifecycle, from model training and tracking to version control and storing. This integrated workflow not only simplifies the complex process of machine learning model development but also ensures models are scalable, reproducible, and ready for deployment.</p> <p><strong>Seamless model deployment with KServe</strong></p> <p>With the model primed for deployment, the focus shifts to deployment logistics. KServe, a model serving framework on HPE Ezmeral, emerges as the solution, enabling efficient deployment and management of machine learning models. Noteworthy is its capability to auto-scale based on demand and seamlessly integrate with existing Kubernetes infrastructure, all under the umbrella of HPE Ezmeral Unified Analytics.</p> <p><img src="/img/demystifying-ml-kserve-img6.png" alt="This screenshot depicts KServe and HPE Ezmeral work together to efficiently deploy and manage machine learning models. " title="Flawless integration between KServe and HPE Ezmeral Software simplifies ML model deployment, management, and auto-scaling with existing Kubernetes infrastructure. "></p> <p><strong>Bringing it all together</strong></p> <p>The Smart Retail Experience culminates with the transition from background operations to front-end application, where a custom-built smart retail application leverages the <a href="">served</a> model to identify fresh produce through a clean user interface and a connected webcam.</p> <p>Just like a self-checkout lane at a store, this application demonstrates how HPE Ezmeral Unified Analytics lets you easily deploy machine learning models you build and use.</p> <p><img src="/img/alex-blog-visual-ai-orange.png" alt="This pictures shows a human presenting an orange to a checkout webcam. " title="Leveraging the served model, the smart retail application can identify fresh produce through a connected webcam and clean user interface. "></p> <p><img src="/img/retail2-with-pricing-.png" alt="This pictures shows the end result of the ML model and webcam simplifying a retail checkout process by identifying and placing the cost of an orange in the customer&#x27;s virtual basket. " title="The end result of this blog shows the model identifying the orange and charging the customer in the checkout basket. "></p> <p><strong>Summary</strong></p> <p>See how easy it is to deliver data analytics and machine learning projects with HPE Ezmeral Unified Analytics in our technical demonstration. By leveraging powerful open-source tools, businesses can create unique and impactful customer experiences.</p> <p>Thank you for accompanying us on this detailed exploration of data analytics, machine learning, and application deployment within the retail sector using HPE Ezmeral Unified Analytics. We trust this tutorial informed and inspired, propelling your business towards transformative possibilities.</p> <p>Don't miss future updates and tutorials from Hewlett Packard Enterprise as we explore how AI can revolutionize various industries.</p>https://developer.hpe.com/activation-memory-what-is-it/https://developer.hpe.com/activation-memory-what-is-it/Wed, 15 May 2024 17:54:38 GMT<p>External blog post</p>https://developer.hpe.com/ilorest-as-a-pypi-package/https://developer.hpe.com/ilorest-as-a-pypi-package/Wed, 15 May 2024 07:52:27 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Good news!!! The <a href="https://servermanagementportal.ext.hpe.com/docs/redfishclients/ilorest-userguide/" target="_blank">HPE iLOrest tool</a> has been repackaged into both source and binary distributions and is now available on <a href="https://pypi.org/project/ilorest/">PyPI</a>. This means it can be easily utilized on any operating system that has Python 3 installed. The intention is for the PyPI package to replace the existing builds for macOS, Debian and Ubuntu distributions of HPE iLOrest.</p> <p>Here are the steps to install HPE iLOrest from <a href="https://pypi.org/project/ilorest/">PyPI</a>:</p> <ol> <li> <p>Ensure that <a href="https://www.python.org/downloads/">Python 3</a> is installed on your operating system.</p> </li> <li> <p>Check if pip3 is installed. If not, on Ubuntu or Debian, you can run the following command:</p> <pre><code class="language-shell">$ sudo apt install python3-pip </code></pre> <p>On Linux, you can use:</p> <pre><code class="language-shell">$ wget https://bootstrap.pypa.io/get-pip.py </code></pre> </li> <li> <p>In most cases, pip3 will already be available on macOS and Microsoft Windows.</p> </li> <li> <p>Once <code>pip3</code> is installed, and before installing HPE iLOrest, make sure the DMTF Redfish Library is not installed, as mentioned in the <a href="(https://servermanagementportal.ext.hpe.com/docs/redfishclients/python-redfish-library/installationguide/#pip-install" target="_blank">HPE iLOrest user guide</a> and then install the HPE iLOrest PyPi package using the following commands:</p> <pre><code class="language-shell">$ pip3 uninstall redfish $ pip3 install ilorest </code></pre> </li> <li> <p>With the PyPi package installation, <a href="https://developer.hpe.com/blog/chif-driver-not-found/">ilorest_chif.dll/.so</a> will also be installed in site-packages.</p> </li> <li> <p>If you installed the HPE iLOrest PyPI package on an iLO based server you can verify the local (in-band) login by running:</p> <pre><code class="language-shell">$ iLOrest -v login </code></pre> </li> <li> <p>You can check the location of iLOrest using the command:</p> <pre><code class="language-shell">$ find / -name iLOrest </code></pre> </li> </ol> <br /> <h2>Notes:</h2> <ul> <li> <p>PyPI package can also be used for ARM-based operating systems if <a href="https://www.python.org/downloads/">Python 3</a> is present.</p> </li> <li> <p>Additionally, the HPE iLOrest PyPI package can be utilized on RHEL, SLES and Microsoft Windows platforms as long as <a href="https://www.python.org/downloads/">Python 3</a> is installed (preferably version > 3.8).</p> </li> <li> <p>DMTF's <a href="https://pypi.org/project/redfish/">Redfish</a> library can not coexist with HPE <a href="https://pypi.org/project/python-ilorest-library/">Python ilorest library</a> which is a dependency for the HPE iLOrest PyPI package. So, make sure to remove any <a href="https://pypi.org/project/redfish/">Redfish</a> library you may have installed prior to installing the HPE Python iLOrest library using the command shown below:</p> <pre><code class="language-shell">$ pip3 uninstall redfish </code></pre> </li> <li> <p>In an air-gapped environment, the HPE iLOrest PyPI package can be downloaded from the <a href="https://pypi.org/project/ilorest/" target="_blank">PyPI repository</a> and installed using the following command. Dependencies may need to be installed separately.</p> <pre><code class="language-shell">$ pip3 install &#x3C;path to the downloaded PyPI package> </code></pre> </li> <li> <p>To perform a clean removal of the HPE iLOrest PyPI package, don't forget to uninstall the associated Python library as mentioned in the following command:</p> <pre><code class="language-shell">$ pip3 uninstall --yes python-ilorest-library ilorest </code></pre> </li> </ul> <h2>Summary:</h2> <p>By following the instructions I've outlined above, you can simplify the installation of and start using the HPE iLOrest tool on PyPI rather than using rpm or msi. It's really quick and easy to get started. For more information on HPE iLO, along with some tips and tricks in working with it, make sure you check out the HPE <a href="https://developer.hpe.com/blog/">Developer</a> blog regularly.</p>https://developer.hpe.com/generic-linear-multistep-method-evaluator-using-chapel/https://developer.hpe.com/generic-linear-multistep-method-evaluator-using-chapel/Tue, 14 May 2024 00:29:12 GMT<p>external blog</p>https://developer.hpe.com/getting-started-with-private-cloud-business-edition-apis/https://developer.hpe.com/getting-started-with-private-cloud-business-edition-apis/Mon, 13 May 2024 13:19:00 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>HPE GreenLake for Private Cloud Business Edition</h2> <p>Deploy an agile, self-service private cloud wherever you need it. Simplify VM management across on-prem and public clouds as you leverage HPE’s modern hyperconverged infrastructure to build a workload-optimized private cloud.</p> <p>Every organization wants to unleash the power of its data to drive digital transformation. But fragmented infrastructure complexity, manual processes across diverse environments across hybrid cloud, and infrastructure silos spanning edge to cloud are impeding data-driven innovation and agility and creating business risk. Customers are looking for a radically simplified experience for data infrastructure at scale across the lifecycle — from streamlined device deployment to virtual machine (VM) provisioning, to a global VM and infrastructure management. They are looking for a cloud-native control plane that scales with infrastructure, so managing hundreds of systems across geographies is as simple as one.</p> <h2>Simplify infrastructure operations</h2> <p>With cloud agility Hewlett Packard Enterprise provides HPE GreenLake for Private Cloud Business Edition delivering VMs on-demand across hybrid cloud. With this offering, customers can build their self-service cloud on demand when they need it and where they need it in data centers, at the edge, and beyond. HPE GreenLake for Private Cloud Business Edition changes the way customers procure and manage VMs with a cloud operational experience. HPE GreenLake for Private Cloud Business Edition helps eliminate complexity with a unified cloud service for VM to infrastructure management, including public cloud VMs. The solution, delivered through Data Services Cloud Console, enables global unified management and monitoring of VMs and infrastructure deployed at any location, It also supports one-click, multisite upgrades. This enables IT to reduce operating costs while helping optimize resource utilization, move to a generalist model, and shift from managing infrastructure to managing VMs and data, thereby refocusing resources and skills on higher-value strategic initiatives.</p> <h2>Automating management</h2> <p>Besides the browser-based Data Services Cloud Console, HPE GreenLake for Private Cloud Business Edition offers a set of Application Programming Interfaces (API's) to automate management of HPE GreenLake for Private Cloud Enterprise or even integrate this with the customer's own management tools. These API's are governed by role-based access controls (RBACs) similar to the regular users using the browser-based console. The API's will authenticate using a token, which is created from the client_id and client_secret key pair, and will have the same permissions as the user that created the client_id and client_secret key pair. Each API call will be audited and logged by HPE GreenLake platform and will be listed under the user's id (usually the email address of the user).</p> <p>In this series of blog posts I will demonstrate how we can connect to the APIs and how the APIs can be used in scripting or integrated into management software. For this series Python will be used in the examples, however any other programming language supporting the http protocol can be used.</p> <h3>Configuring API client credentials</h3> <p>To get started with the API you'll need to create a client_id/client_secret pair to authenticate each API request, this key pair is linked to the user creating it and every interaction using this token will be registered in the audit log. The token generated using this key pair has the same permissions as the user who created it. The token is valid for two hours and expires automatically.</p> <ol> <li> <p>Sign in to HPE GreenLake, select your Workspace, then select Manage Workspace.</p> </li> <li> <p>Click the API card.</p> <p><img src="/img/workspace-small.png" alt="" title="Manage workspace"></p> </li> <li> <p>Click Create Credentials.</p> <p><img src="/img/create_credentials.png" alt="" title="Click create credentials"></p> </li> <li> <p>Select the Service Manager you want to access.</p> </li> <li> <p>Enter the Credential Name.</p> </li> <li> <p>Click the Create Credentials button to continue. The Credentials Created screen displays showing your credentials were successfully created.</p> </li> <li> <p>Copy the Client ID and the Client Secret to a safe and secure location. HPE GreenLake platform does not store your Client Secret.</p> </li> <li> <p>Select the Copy icon to save your information to your desktop.</p> </li> <li> <p>Click the Close button to continue. You are returned to the main API page, where you can now generate the access token using a sample code provided.</p> </li> </ol> <p>Example code:</p> <pre><code class="language-python">from oauthlib.oauth2 import BackendApplicationClient from requests.auth import HTTPBasicAuth from requests_oauthlib import OAuth2Session client = BackendApplicationClient('e21f3028-8097-4a4f-b491-a49b1d102d4a') oauth = OAuth2Session(client=client) auth = HTTPBasicAuth('e21f3028-8097-4a4f-b491-a49b1d102d4a', '05656940014f11ef9946a2408e898685') token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth) print(token["access_token"]) </code></pre> <blockquote> <p><em>Note</em>: Make sure the modules oauthlib, requests and requests_oauthlib are installed in your environment, if not you can install these using:</p> </blockquote> <pre><code class="language-powershell">pip install oauthlib pip install requests pip install requests_oauthlib </code></pre> <p>When running the example code JSON formatted data will be returned, like this:</p> <pre><code class="language-json">{ "access_token": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjJGSmhvZ1lRMDZNazNBc2Q4UU8zU09ZVE9wayIsInBpLmF0bSI6ImRlejAifQ.eyJjbGllbnRfaWQiOiJlMjFmMzAyOC04MDk3LTRhNGYtYjQ5MS1hNDliMWQxMDJkNGEiLCJpc3MiOiJodHRwczovL3Nzby5jb21tb24uY2xvdWQuaHBlLmNvbSIsImF1ZCI6ImV4dGVybmFsX2FwaSIsInN1YiI6Im1hcmsudmFuLnNpbGZob3V0QGhwZS5jb20iLCJ1c2VyX2N0eCI6ImVmMDc3MDVjOGVhNjExZWNiNzNiMWEyNTZjMDNiNTQ2IiwiYXV0aF9zb3VyY2UiOiJjY3NfdG9rZW5fbWFuYWdlbWVudCIsInBsYXRmb3JtX2N1c3RvbWVyX2lkIjoiZDI2YmFmMjY4ZWE2MTFlYzljZmUwMmMwMzQ1YzI0M2MiLCJpYXQiOjE3MTM4NjI4OTYsImFwcGxpY2F0aW9uX2luc3RhbmNlX2lkIjoiZWYwMTA4MjMtYjMwMS00MWJmLTk0OWMtYTQ3NzE0OTRiMzg1IiwiZXhwIjoxNzEzODcwMDk2fQ.arTNjVsVZ-wcW5Ic5fuGUvBKAupzWvRokuvdzlW1I2UqXFoqp0-jw1cHVr-QgDUlBPXG6uc-aILnYXWN_h-QWOQQu1c8aTbLaqpfXEL89MndPyErF0x4By21JLoR1mq-8zkMEEJ2CHGOeBYau_hBim-SBr1BtcetX3BcFl4GoGRGS5lzL1nbwkC-Hi_OOs_2UnDH4ajyyPsp_Ka4wmsgHn1aSQhJjDFbxx4WiLmRdp8aNZT5r250v5EWR9qVeqE4TDOfKAf_BBhC2RB00mWAt1F_Rd4rPmMyZm0uPZd5f71aDjd3I5tl0gh-W2Z4HHt0jRKV-L1d4o52jzYkhL1nTA", "expires_at": 1713870095.208547, "expires_in": 7199, "token_type": "Bearer" } </code></pre> <p>The JSON data holds the access_token itself and also the expiration time, which is 2 hours (7200 seconds).</p> <blockquote> <p>Note: For simplicity and demo reasons we store the client_id and client_secret in the code, obviously that should never be done in production! How to implement a more secure method will be explained in the next episode of this series.</p> </blockquote> <h2>Calling an API</h2> <p>A list of available APIs for HPE GreenLake for Private Cloud Business Edition can be found at <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/private-cloud-business/public/">https://developer.greenlake.hpe.com/docs/greenlake/services/private-cloud-business/public/</a> When publishing this blog post the APIs were still in beta, however the GA release is planned shortly after.</p> <p>In this example you will collect an overview of all System Software Catalogs.</p> <pre><code class="language-python">import json from oauthlib.oauth2 import BackendApplicationClient from requests.auth import HTTPBasicAuth from requests_oauthlib import OAuth2Session client = BackendApplicationClient('e21f3028-8097-4a4f-b491-a49b1d102d4a') oauth = OAuth2Session(client=client) auth = HTTPBasicAuth('e21f3028-8097-4a4f-b491-a49b1d102d4a', '05656940014f11ef9946a2408e898685') token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth) access_token= token["access_token"] url = "https://us1.data.cloud.hpe.com/private-cloud-business/v1beta1/system-software-catalogs" headers = {"Authorization": f"Bearer {token}"} response = requests.get(url, headers=headers) print(json.dumps(response.json(), indent=4, sort_keys=True)) </code></pre> <p>This will generate a JSON formatted data stream like:</p> <pre><code class="language-json"> "count": 10, "items": [ { "createdAt": "2024-03-11T01:56:04Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 991063, "hypervisor": null, "hypervisorManager": null, "id": "f3b72767-4f59-4af5-8e34-cc2828baa954", "name": "2.98", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/f3b72767-4f59-4af5-8e34-cc2828baa954", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "2.98" }, { "createdAt": "2024-03-11T01:58:06Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 570404, "hypervisor": null, "hypervisorManager": null, "id": "993fae5e-a928-458e-8547-87e29f956952", "name": "7.61.34.18.37", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/993fae5e-a928-458e-8547-87e29f956952", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:31:52Z", "version": "7.61.34.18.37" }, { "createdAt": "2024-03-11T01:54:04Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 970638, "hypervisor": null, "hypervisorManager": null, "id": "6cff3aef-c12c-41e4-8934-1e02fa19492a", "name": "7.6.35.18.36", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/6cff3aef-c12c-41e4-8934-1e02fa19492a", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "7.6.35.18.36" }, { "createdAt": "2023-11-02T18:39:13Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-juno-1100.html", "generation": 3181010, "hypervisor": { "name": "ESXi 7.0U2 May 2021", "releaseDate": "2021-05-13", "releaseNotesURL": "https://support.hpe.com/hpesc/public/docDisplay?docId=a00113634en_us", "version": "7.0.2-17867351" }, "hypervisorManager": null, "id": "2df73503-6b84-41b7-9c49-eb23b6e850b2", "name": "1.70", "releaseDate": "2023-06-01", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/2df73503-6b84-41b7-9c49-eb23b6e850b2", "serverFirmware": { "name": "Service Pack for ProLiant", "releaseDate": "2022-09-28", "releaseNotesURL": "https://internal.support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=a00127259en_us", "version": "2022.09.1" }, "storageConnectionManager": { "name": "Nimble Connection Manager", "releaseDate": "2020-10-21", "releaseNotesURL": "https://infosight.hpe.com/InfoSight/media/software/active/2/351/NCM701_RelNotes2.pdf", "version": "7.0.1-700003" }, "storageSoftware": { "name": "Nimble OS", "releaseDate": "2023-06-13", "releaseNotesURL": "https://infosight.hpe.com/InfoSight/media/cms/active/NimbleOS_Release_Notes_5.2.1.1100.pdf", "version": "5.2.1.1100-1027043" }, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "1.70" }, { "createdAt": "2024-03-11T01:54:04Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 952334, "hypervisor": null, "hypervisorManager": null, "id": "7332fe0c-9c98-4bf8-8071-d7b2cd8e8b37", "name": "7.6.34.18.36", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/7332fe0c-9c98-4bf8-8071-d7b2cd8e8b37", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "7.6.34.18.36" }, { "createdAt": "2024-03-11T01:56:04Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 988428, "hypervisor": null, "hypervisorManager": null, "id": "50247e6c-13e1-426e-80d5-72aac49cbf3a", "name": "2.97", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/50247e6c-13e1-426e-80d5-72aac49cbf3a", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "2.97" }, { "createdAt": "2023-11-02T18:39:13Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-juno-1100.html", "generation": 3169145, "hypervisor": { "name": "ESXi 7.0U1 May 2021", "releaseDate": "2021-05-13", "releaseNotesURL": "https://support.hpe.com/hpesc/public/docDisplay?docId=a00113633en_us", "version": "7.0.1-17551050" }, "hypervisorManager": null, "id": "2b1b935a-6640-4f65-bbbc-02141582de20", "name": "1.69", "releaseDate": "2023-06-01", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/2b1b935a-6640-4f65-bbbc-02141582de20", "serverFirmware": { "name": "Service Pack for ProLiant", "releaseDate": "2021-05-21", "releaseNotesURL": "https://downloads.hpe.com/pub/softlib2/software1/publishable-catalog/p291731480/v138520/SPP2021.05.0ReleaseNotes.pdf", "version": "2021.05.0" }, "storageConnectionManager": { "name": "Nimble Connection Manager", "releaseDate": "2020-10-21", "releaseNotesURL": "https://infosight.hpe.com/InfoSight/media/software/active/2/351/NCM701_RelNotes2.pdf", "version": "7.0.1-700003" }, "storageSoftware": { "name": "Nimble OS", "releaseDate": "2023-06-13", "releaseNotesURL": "https://infosight.hpe.com/InfoSight/media/cms/active/NimbleOS_Release_Notes_5.2.1.1100.pdf", "version": "5.2.1.1100-1027043" }, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "1.69" }, { "createdAt": "2024-02-20T02:56:40Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-juno-1100.html", "generation": 1447550, "hypervisor": null, "hypervisorManager": null, "id": "6b692b44-ce0d-44a2-8a19-4d9cbfd9330f", "name": "1.77", "releaseDate": "2024-02-11", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/6b692b44-ce0d-44a2-8a19-4d9cbfd9330f", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "1.77" }, { "createdAt": "2024-03-11T01:57:07Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 644305, "hypervisor": null, "hypervisorManager": null, "id": "f66777dd-e380-4319-9a32-a7ee24ff4cd4", "name": "7.61.35.18.37", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/f66777dd-e380-4319-9a32-a7ee24ff4cd4", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:31:52Z", "version": "7.61.35.18.37" }, { "createdAt": "2024-03-11T01:57:05Z", "customerId": "00000000000000000000000000000000", "eula": "https://update.nimblestorage.com/catalog/download/eula-rel-pebble-400.html", "generation": 990969, "hypervisor": null, "hypervisorManager": null, "id": "3f3a18be-00f8-4478-9a40-b0b316d47771", "name": "2.99", "releaseDate": "2024-03-05", "resourceUri": "/private-cloud-business/v1beta1/system-software-catalogs/3f3a18be-00f8-4478-9a40-b0b316d47771", "serverFirmware": null, "storageConnectionManager": null, "storageSoftware": null, "systemsWithUpdatePath": null, "type": "private-cloud-business/system-software-catalog", "updatedAt": "2024-05-03T13:32:55Z", "version": "2.99" } ], "offset": 0, "total": 11 } </code></pre> <p>In the next blog post in this series we will have a look at a more secure implementation of the client_secret, using a key chain manager to read the client_secret.</p>https://developer.hpe.com/2024-may-06/https://developer.hpe.com/2024-may-06/Mon, 06 May 2024 13:00:00 GMThttps://developer.hpe.com/automating-it-operations-with-compute-ops-management-webhooks/https://developer.hpe.com/automating-it-operations-with-compute-ops-management-webhooks/Fri, 03 May 2024 09:54:45 GMT<p>NONE</p>https://developer.hpe.com/doing-science-in-python-wishing-for-more-speed-or-scalability/https://developer.hpe.com/doing-science-in-python-wishing-for-more-speed-or-scalability/Tue, 30 Apr 2024 21:27:01 GMT<p>External blog</p>https://developer.hpe.com/ai-news-21/https://developer.hpe.com/ai-news-21/Mon, 29 Apr 2024 14:58:26 GMT<p>External blog post</p>https://developer.hpe.com/chapels-high-level-support-for-cpu-gpu-data-transfers-and-multi-gpu-programming/https://developer.hpe.com/chapels-high-level-support-for-cpu-gpu-data-transfers-and-multi-gpu-programming/Thu, 25 Apr 2024 22:17:28 GMT<p>External blog</p>https://developer.hpe.com/bulk-onboarding-of-users-in-hpe-greenlake-edge-to-cloud-platform/https://developer.hpe.com/bulk-onboarding-of-users-in-hpe-greenlake-edge-to-cloud-platform/Wed, 24 Apr 2024 13:44:40 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } ol li{ font-size:27px; } </style> <h2>HPE GreenLake API to the rescue</h2> <p>The use case covered in this document is part of what we call the Day 0 activities; tasks that must be done to onboard users to HPE GreenLake platform. When a customer decides to use HPE GreenLake, it is critical that all customer collaborators who require access to HPE GreenLake platform are invited to join. Using the HPE GreenLake console to invite hundreds of collaborators can be tedious and error prone - this is when an API comes to the rescue. The API allows you to write a script that reads a list of users from an Excel spreadsheet and automatically invites these users to access the HPE GreenLake platform.</p> <h2>What are the HPE GreenLake edge-to-cloud platform APIs</h2> <p>The foundational APIs for common HPE GreenLake platform services allow IT administrators and IT operators to programmatically operate and manage users and resources in an HPE GreenLake platform workspace.  </p> <p>This set of APIs for common platform services includes APIs for workspace management, identity and access management, device and subscription, locations, audit logs, and wellness.  </p> <p><em>Note: The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake platform documentation</a> for these APIs leverages OpenAPI specifications and associated reference material. The documentation provides a complete explanation of the operations supported by these APIs for common HPE GreenLake platform services, as well as sample requests and responses.</em>  </p> <p>The following blog posts are an excellent way to learn more about the APIs using Postman:</p> <ul> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-1-introduction-to-the-apis/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 1: Introduction to the APIs</a></li> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-2-configuring-and-managing-a-workspace/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 2: Configuring and managing a workspace</a></li> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 3: Tracking activities and monitoring health</a></li> </ul> <p>In this blog post, I will focus on one specific API call, part of Identity Management. The call is <code>POST /identity/v1/users</code>, which invites users to an HPE GreenLake workspace. Full documentation on this API call can be found in the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/iam/workspaces/public/openapi/workspaces-v1/operation/invite_user_to_account_identity_v1_users_post/">HPE GreenLake developer portal</a>.</p> <h2>Providing the right data to the script</h2> <p>Before writing any code, it’s important to understand what data is required to invite a user. You only need the email address of the invited user- that’s easy! In the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/iam/workspaces/public/openapi/workspaces-v1/operation/invite_user_to_account_identity_v1_users_post/">API reference</a>, you'll also see that there is no way to select a workspace to invite the user to. The reason for this is that the API credentials used to make the call is workspace specific, so it implicitly provides the workspace to which the user will be invited to. This means that you need to collect API access credentials for every workspace that you're adding users to. For the script I am writing here, in the Workspaces tab, I have stored the Client Id corresponding to API Access of a given Workspace. Because I don’t want to save Client Secrets, I will prompt for them and store them in memory.</p> <p>So, my Excel file contains the following 2 sheets:</p> <p><img src="/img/bulkimport-blog-picture-1.png" alt="Users tab in Excel" title="Users tab in Excel"></p> <p><img src="/img/bulkimport-blog-picture-2.png" alt="Workspaces tab in Excel" title="Workspaces tab in Excel"></p> <h2>High-level algorithm</h2> <p>Let’s look at the steps necessary to invite users from my spreadsheets:</p> <ol> <li>Read command parameters to get the Excel filename</li> <li>Open spreadsheet to retrieve data</li> <li>For each workspace in Workspaces sheet <ul> <li>Prompt for Client Secret that matches the Client Id</li> <li>Retrieve a session token using those credentials</li> </ul> </li> <li>For each user in Users sheet <ul> <li>Lookup Client Id using workspace name</li> <li>Call POST /identity/v1/users for user using email</li> <li>Increase counter of invited users</li> </ul> </li> <li>Display list of users invited in each workspace</li> </ol> <h2>Putting things together in PowerShell</h2> <p>I decided to use PowerShell to write this script because it provides easy native access to Excel spreadsheets.</p> <h3>Step 1 – Reading the parameter from the command line.</h3> <pre><code class="language-powershell">Param($XLFile) if ($Null -eq $XLFile) { if ($env:XLFile -eq $Null) { $XLFile = read-host "Enter name of the Excel file" } } </code></pre> <h3>Step 2 – Importing data from the 2 sheets of my spreadsheet.</h3> <pre><code class="language-powershell">$tokens =@{} $invited=@{} if ($XLFile) { $users_excel = import-excel -path $XLFile -dataonly -worksheetname Users $workspaces_excel = Import-Excel -path $XLFile -dataonly -worksheetname Workspaces </code></pre> <p>Note that I initialized 2 hash tables, one called $tokens that will store the token for a given Client Id (i.e Workspace) and another called $invited for storing the number of invited users for a given Client Id.</p> <h3>Step 3 – Iterating over the Workspaces sheet to collect client secrets, and retrieve access tokens.</h3> <pre><code class="language-powershell"># Ask for client_Secret of each workspace in Excel file foreach ($workspace in $workspaces_excel ) { $client_id = $workspace.'Client Id' if ($tokens[$client_id] -eq $null) { # We don't have a token for this client_id yet # We need to ask the Client secret for this workspace $workspace_name = $workspace.'Workspace Name' $client_id = $workspace.'Client Id' $secClientSecret = read-host "Enter HPE GreenLake Client Secret for Workspace $workspace_name" -AsSecureString $bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($secClientSecret) $secret = [System.Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) # use Client Id and Client Secret to retrieve a token $body = "grant_type=client_credentials&#x26;client_id=" + $client_id + "&#x26;client_secret=" + $secret $headers = @{} $headers["Content-Type"] = "application/x-www-form-urlencoded" try { $response = Invoke-webrequest "https://sso.common.cloud.hpe.com/as/token.oauth2" -Method POST -Headers $headers -Body $body # store the token for future use $AccessToken = ($response.Content | Convertfrom-Json).access_token $tokens.Add($client_id,$AccessToken) } catch { Write-Host "Error retrieving access token for workspace $workspace_name!" -ForegroundColor Red exit } } } </code></pre> <p>Note that, at the end of this loop, I have a hash table of tokens indexed by Client Id, which I will use to call the API in the next section.</p> <h3>Step 4 – Iterating over Users sheet to invite each of them.</h3> <pre><code class="language-powershell"># Now walk the list of users to add $invited.Add($client_id,0) foreach ($user in $users_excel ) { $workspace_name = $user.'Workspace Name' # Get client id from workspace name $result = $workspaces_excel | Where-Object { $_.'Workspace Name' -eq $workspace_name } if ($result.Count -eq 0) { Write-Host "Workspace not found for user " $user.email -ForegroundColor Red exit } $client_id = $result[0].'Client Id' Write-Host "Inviting user" $user.email "to workspace./" $workspace_name $AccessToken = $tokens[$client_id] # Create header for next API calls $headers = @{} $headers["Authorization"] = "Bearer $AccessToken" $headers["Accept"] = "application/json" $headers["Content-Type"] = "application/json" # Build body for next API call $_body = @{ "email" = $user.email "sendWelcomeEmail" = $true } $Body = $_body | ConvertTo-Json # Call GLP API to invite user try { $response = Invoke-webrequest -Uri "https://global.api.greenlake.hpe.com/identity/v1/users" -Method POST -Headers $headers -Body $Body $invited[$client_id]++ } catch { Write-Host "Error sending invite for" $user.Email"! Already onboarded?" -ForegroundColor Red Write-Host $Error[0] -ForegroundColor Red continue } sleep 15 } } </code></pre> <p>Note that before the loop, I initialized to zero the count of invited users for a given workspace. Also note the sleep 15 (seconds) at the end of the loop to avoid issues with rate limiting constraints which might raise a status code 429.</p> <p><em>Note: Rate Limiting is a mechanism employed to control and restrict the rate at which requests or interactions are permitted to occur between clients and a service.</em></p> <h3>Step 5: Displaying list of users invited in each workspace.</h3> <pre><code class="language-powershell">else { write-host 'Mailing list file not provided nor found....' exit } Write-host "Done processing Excel file $XLFile!" # ------------------------ Query GL to get list of users for each workspace ------------------------ foreach ($workspace in $workspaces_excel ) { $workspace_name = $workspace.'Workspace Name' $client_id = $workspace.'Client Id' # Create header for next API calls $headers = @{} $AccessToken = $tokens[$client_id] $headers["Authorization"] = "Bearer $AccessToken" $headers["Accept"] = "application/json" $headers["Content-Type"] = "application/json" try { $response = Invoke-webrequest "https://global.api.greenlake.hpe.com/identity/v1/users?filter=&#x26;limit=300&#x26;offset=0" -Method GET -Headers $headers } catch { Write-Host "Cannot get list of users!!" exit } $invited_users=$invited[$client_id] Write-Host $invited_users "user(s) invited to workspace" $workspace_name Write-Host "List of users in workspace:" $workspace_name $_list = $response.Content | ConvertFrom-Json if ($null -ne $_list) { $_users_list = [System.Collections.ArrayList]::new() foreach ($_u in $_list.Items) { $_users_list += @{ 'Username' = $_u.Username 'Status' = $_u.userStatus 'id' = $_u.Id } } } $_users_list | select Username, Status | ft -AutoSize } </code></pre> <h2>Try it!</h2> <p>Let’s run this script, making sure to reference the correct Excel spreadsheet:</p> <pre><code class="language-markdown">PS /Volumes/Dev/GreenLake/GLP-API-Tooling/Scripts> ./bulk_invite.ps1 -XLfile userlist.xlsx Enter HPE GreenLake Client Secret for Workspace HPEDEV -GLCP- Hackshack: ******************************** Enter HPE GreenLake Client Secret for Workspace Super Awesome Company: ******************************** Inviting user xxx@gmail.com to workspace HPEDEV -GLCP- Hackshack Inviting user yyy@lalli.fr to workspace HPEDEV -GLCP- Hackshack Error sending invite for yyy@lalli.fr ! Already onboarded? Inviting user zzz@lalli.fr to workspace Super Awesome Company Inviting user www@gmail.com to workspace Super Awesome Company Error sending invite for www@gmail.com ! Already onboarded? Done processing Excel file userlist.xlsx! 1 user(s) invited to workspace HPEDEV -GLCP- Hackshack List of users in workspace: HPEDEV -GLCP- Hackshack Username Status -------- ------ &#x3C;email> VERIFIED … yyy@lalli.fr VERIFIED … xxx@gmail.com UNVERIFIED … &#x3C;email> VERIFIED 1 user(s) invited to workspace Super Awesome Company List of users in workspace: Super Awesome Company Username Status -------- ------ &#x3C;email> VERIFIED … www@gmail.com VERIFIED … xxx@lalli.fr UNVERIFIED … &#x3C;email> VERIFIED </code></pre> <p>As you can see, the script has invited 1 user in each workspace. The second email address is already a member of the workspace, so no action is necessary.</p> <h2>What’s next?</h2> <p>Through this post, I have shown you how it is possible to integrate with HPE GreenLake platform using the most popular scripting languages, such as PowerShell. You can get the source code for these scripts from <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling">our community tooling repository</a>.</p> <p>If you’re interested in trying out what I just discussed, you might first want to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake APIs mentioned in this blog post. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested in and go! It’s as simple as that.</p> <p>If you still have questions regarding the HPE GreenLake platform APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion on our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We are always here to help.</p>https://developer.hpe.com/from-a-pre-trained-model-to-an-ai-assistant-finetuning-gemma-2b-using-dpo/https://developer.hpe.com/from-a-pre-trained-model-to-an-ai-assistant-finetuning-gemma-2b-using-dpo/Wed, 24 Apr 2024 12:22:14 GMT<p>External blog post</p>https://developer.hpe.com/ai-news-19/https://developer.hpe.com/ai-news-19/Mon, 15 Apr 2024 14:37:30 GMT<p>External blog post</p>https://developer.hpe.com/ai-news-18/https://developer.hpe.com/ai-news-18/Thu, 11 Apr 2024 09:39:30 GMT<p>External blog post</p>https://developer.hpe.com/navier-stokes-in-chapel-—-introduction/https://developer.hpe.com/navier-stokes-in-chapel-—-introduction/Wed, 10 Apr 2024 13:48:07 GMT<p>External blog</p>https://developer.hpe.com/hpe-athonet-llm-platform-divide-et-impera-designing-the-ll-mesh/https://developer.hpe.com/hpe-athonet-llm-platform-divide-et-impera-designing-the-ll-mesh/Tue, 09 Apr 2024 09:11:29 GMT<p>In a recent series of blog posts, I've explored various facets of generative AI and its profound impact on telecommunications and corporate tools. HPE Athonet's revolutionary approaches in the digital landscape were highlighted in these discussions:</p> <ul> <li><a href="https://developer.hpe.com/blog/the-transformative-impact-of-generative-ai-on-telco-products/">"The transformative impact of generative AI on Telco products"</a> discussed the pivotal role of AI in enhancing product offerings.</li> <li><a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-first-pillar-from-personal-assistant-to-collaborative-corporate-tool/">"HPE Athonet LLM Platform: From personal assistant to collaborative corporate tool"</a> highlighted the evolution of the platform into a powerful collaborative suite.</li> <li><a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-driving-users-towards-peak-flow-efficiency/">“HPE Athonet LLM Platform: Driving users towards peak 'Flow' efficiency</a> focused on optimizing user efficiency and engagement.</li> </ul> <p>These insights set the stage for a profound transformation in AI centered around:</p> <ul> <li>Transitioning to a team-focused collaborative framework.</li> <li>Enhancing user experience to boost productivity and focus.</li> <li>Building a resilient infrastructure based on data mesh principles, ensuring robust security and adherence to ethical standards.</li> </ul> <p>Today, the discussion delves deeper into the third pillar inspired by the Data Mesh concepts detailed in the article <a href="https://martinfowler.com/articles/data-mesh-principles.html">“Data Mesh Principles and Logical Architecture”</a> Here I introduce the <strong>"LLM Agentic Tool Mesh"</strong> to create a clean, scalable architecture that can swiftly adapt to the rapid advancements in technology . This tailored approach adapts the Data Mesh framework to the unique needs of LLMs, merging data, context, and reasoning capabilities into a cohesive whole.</p> <p>The LLM Agentic Tool Mesh embodies the data itself and includes reasoning capabilities, orchestrating tools enhanced with LLM technology. Developers using this platform take charge of everything from the underlying data to the APIs and documentation, ensuring a seamless experience for end-users. This initiative is more than a technical upgrade—it’s a significant leap towards digital transformation that promises substantial competitive advantage through enhanced knowledge sharing within the company that adopts it, leading to the customization of products and services tailored specifically for customers.</p> <p>However, there are some challenges, including the need to master new generative AI technologies, the importance of managing data properly by adopting the 'data as code' concept, and the cultural shift required to maintain agility and continuously improve all company processes.</p> <p>Expecting every team tasked with developing a tool to start from scratch is impractical. Instead, it's crucial to offer a supportive platform that delivers the essential services needed for creation, complemented by a federated governance structure that lays out clear rules and guidelines. In the sections that follow, these principles are explored more in detail, alongside additional insights to guide the development process.</p> <h3><strong>Domain-specific LLM tool ownership and decentralization</strong></h3> <p>In the LLM Agentic Tool Mesh framework, ownership of LLM tools is strategically distributed across various teams or departments, with each group taking full responsibility for their respective tool's development, maintenance, and quality assurance. This organizational structure aligns with modern enterprise setups, which are typically divided into specific business domains. By applying LLM Agentic Tool Mesh principles along these natural divisions, tools are finely tuned to meet the unique requirements of each domain, such as customer service or research and development.</p> <p>Central to LLM Agentic Tool Mesh are data and prompts, metadata, and models, all supported by the necessary computational power to process and deliver results efficiently. This architecture not only facilitates streamlined data management but also significantly enhances the model’s capability to learn and adapt. By aligning the decomposition of tools with the business domains, LLM Agentic Tool Mesh ensures that each tool operates within well-defined operational parameters, optimizing both performance and adaptability.</p> <h3><strong>LLM tool as product</strong></h3> <p>Each LLM tool is crafted as a specialized product, meticulously designed with the end-user in mind to ensure user-centricity, efficiency, and effectiveness. This philosophy treats each tool as a comprehensive solution that spans from data generation to consumption, emphasizing a seamless and integrated experience. It is crucial that a dedicated team, including a product owner, is responsible for managing the tool end-to-end, ensuring that all stages—from using documents for information analysis with LLMs to presenting results—are harmoniously interconnected. This holistic approach not only enhances the functionality of each phase but also significantly boosts the overall quality of the tool.</p> <p>The principle of viewing an LLM tool as a product also tackles the persistent issue of data silos. Like data products, there should be a system in place that enables the use of multiple LLM tools in concert. It is essential for these tools to provide clear interfaces and capabilities, including discoverability, security, understandability, and trustworthiness. This ensures that tools not only function independently but also work cohesively within a larger ecosystem, enhancing interoperability and user trust.</p> <h3><strong>LLM platform infrastructure</strong></h3> <p>A self-service infrastructure platform is provided with the tool, enabling teams to autonomously develop, deploy, and manage their LLM tools. This platform supports the necessary frameworks to promote autonomy and innovation. Envisioned as LLM Agentic Tool Mesh, it extends capabilities beyond enhancing user experience for chatbot users to include interfaces for various stakeholders, including document owners, tool developers, and deployment teams. This requires the development of diverse user interfaces tailored to different interactions within the ecosystem.</p> <p><img src="/img/athon_ssp_1.png" alt=""></p> <p>At MWC 2024 (Mobile World Congress), a proof of concept demonstrated the robustness of this architecture by showcasing a chatbot capable of supporting multiple tools through the self-service capabilities of the LLM platform. This facilitates the development and integration of both individual tools and comprehensive chatbot solutions.</p> <p><img src="/img/athon_ssp_2.png" alt=""></p> <p>The architecture leverages a reasoning engine where the LLM serves as a central hub, orchestrating various tools to ensure streamlined communication and functionality. Integrating a tool into this engine is straightforward: simply define the tool and its arguments in a prompt, and the LLM will intelligently determine when to engage the tool based on user requests.</p> <p><img src="/img/athon_ssp_3.png" alt=""></p> <p>Designed to be both extendable and modular, each component of the platform employs design patterns such as the factory method and strategy pattern. These patterns ensure a decoupling of creation from usage, providing a flexible and scalable architecture that is easy to maintain. Each module operates independently yet conforms to a unified global interface, enhancing system integrity and adaptability. The self-serve platform contains the following features:</p> <ul> <li>Utility functions simplify the complexity of underlying libraries, such as LangChain, Hugging Face, and LLaMA Index</li> <li>Domain-specific high-level functions, for example, parsing telecommunications standards using Retrieval-Augmented Generation (RAG).</li> </ul> <p><img src="/img/athon_ssp_4.png" alt=""></p> <h3><strong>Federated governance and standards</strong></h3> <p>While the management of LLM tools is decentralized, a unified framework of governance policies and standards is crucial to ensure consistency, ethical integrity, and overall quality across the platform. Federated governance effectively balances the drive for innovation with rigorous standards for security, privacy, and compliance, which are vital to the operation of the LLM Agentic Tool Mesh. This approach encompasses several key elements:</p> <ul> <li>Decentralized autonomy: Domain-specific teams are empowered to manage their LLM tools, fostering a sense of ownership and responsiveness to specific needs, all while aligning with overarching governance standards of the mesh.</li> <li>Interoperable standards: Common guidelines for security, data privacy, and model behavior are established to ensure all interactions are helpful and harmless, promoting seamless and safe integration across different tools and domains.</li> <li>Ethical compliance: It is critical that all LLMs operate within ethical guidelines and regulatory frameworks to foster trust and reliability. This commitment to ethical compliance supports honest interactions and adherence to broader social and legal standards.</li> <li>Central oversight: A centralized governance body is tasked with enforcing these policies across the mesh, ensuring that all components adhere to responsible AI principles and maintain a high standard of integrity and accountability.</li> <li>Automated governance tools: Advanced technologies are deployed to automate the enforcement of governance standards, ensuring consistent compliance with the principles of being helpful, harmless, and honest.</li> <li>Collaboration and feedback: Continuous collaboration and feedback mechanisms within the LLM Agentic Tool Mesh are crucial for adapting and refining governance practices. This open dialogue encourages ongoing improvement and alignment with emerging technologies and challenges.</li> </ul> <p>By embracing these principles, the LLM Agentic Tool Mesh seeks to cultivate a robust, ethical, and dynamic ecosystem for the development and deployment of LLM tools. This approach ensures that all tools not only add significant value but also meet stringent standards of responsibility and governance. These efforts underscore HPE Athonet's commitment to driving innovation and digital transformation in network management by developing a responsible framework and product.</p>https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-backup-and-recovery/https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-backup-and-recovery/Mon, 08 Apr 2024 00:16:50 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>What's new?</h2> <p>Recently, a new set of REST APIs for HPE GreenLake for Backup and Recovery Service was introduced in the HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/">website</a>. This is the third blog post from the series of blog postings (<a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services">Data-Services</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-virtualization">Virtualization</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-backup-and-recovery">Backup and Recovery</a>) that introduce some useful tips and best practices about using this new set of APIs given a specific use case.</p> <p>This set of APIs provides the capability for manipulating resources made available by HPE GreenLake for Backup and Recovery services. Consequently, any customer can use these APIs to protect their data in a hybrid cloud in the same manner as using the user interface for HPE GreenLake for Backup and Recovery in the HPE GreenLake console. This set of APIs provides user capabilities to perform any of Create, Read, Update and Delete (CRUD) operations against HPE GreenLake Backup and Recovery resources such as: <em>data-orchestrator, protection store gateway, StoreOnce, protection-stores, protection-policies, snapshots, backups, on-premises assets (VM, DataStore, MSSQL, Storage Volumes).</em></p> <p>There will be more resources added into this set of APIs in the future releases that will cover <em>cloud service providers, and cloud assets</em>. For more information on how to use this HPE GreenLake for Backup and Recovery service, please visit the <a href="https://www.hpe.com/us/en/hpe-greenlake-backup-recovery.html">website </a>and the getting started <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=sd00003454en_us">guide</a>.</p> <p>The specification of this API is publicized as an OpenAPI specification in JSON format, and the specification is available for download from this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/guide/">section</a> of the documentation (shown below). The specification follows the OpenAPI Standard v3.1, and it contains all required information so that this JSON file can be consumed by any OpenAPI tools to provide client library, server mock, or documentation as described in this <a href="https://tools.openapis.org/">OpenAPI </a>Initiative.</p> <blockquote> <p><strong>NOTE:</strong> There are two different sets of OpenAPI specs that are downloadable from the documentation page of the Backup and Recovery in the March 2024 release. The two sets represent the two versions of Backup and Recovery APIs that were made available for separate resources, namely <strong>hypervisor-managers</strong> and <strong>the rest of other resources</strong> as shown in this picture below. To get into the page for downloading each of the OpenAPI specification file, scroll through the left windows and select the appropriate API version page.</p> </blockquote> <p><img src="/img/backup-and-recovery-api-front-download-page.jpg" alt="GLBR API documentation website" title="Front page for Backup &#x26; Recovery front page"></p> <p>The Backup and Recovery API specification files contain information that describes the set of REST APIs for HPE GreenLake for Backup and Recovery, such as the endpoints, authentication, syntax of parameters, expected response, and many other objects.</p> <p><img src="/img/backup-and-recovery-json-information.png" alt="GLBR openapi spec" title="JSON open API spec example"></p> <p><em>The above figure shows the example of the downloaded backup-and-recovery v1beta1.json file from the Backup and Recovery API documentation <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/guide/">guide</a>.</em></p> <h2>API versioning</h2> <p>This set of APIs is released with two different specifications which are identified as revision V1 Alpha 1 and V1 Beta 1 at the time of its release in March 2024. The short-term plan is to publish version 1 of the HPE GreenLake APIs for Backup and Recovery Service to replace the V1 Alpha 1 and V1 Beta 1 versions. Once the API is at version 1, the intent is to keep changes to the minimum required and publicly announce changes as they are made. As each individual API is updated, there will also be more capabilities added to any of the resources identified under this set of APIs. For information regarding updates and deprecation, please refer the HPE GreenLake Developer Portal Versioning <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/public/standards/versioning_basics/">guide</a>. You can expect that the API categorized as V1 Alpha 1 will be updated within a short time; hence, I recommend you monitor for any announcement of the next revision of APIs for Backup and Recovery in this documentation <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/guide/">guide</a>.</p> <blockquote> <p><strong>NOTE:</strong> At the time of its release in March 2024, all of resources for HPE GreenLake API for Backup and Recovery are limited to data protection of on-premises assets. The manipulation of the cloud assets will be made available in the next release.</p> </blockquote> <h2>What are Backup and Recovery API resources?</h2> <p>There are a lot of resources that are part of this set of APIs . Therefore, I like to break them down into two parts so that I can present the familiar view of the components that correspond to the resources.</p> <p>The below diagram displays components in the first part of the HPE GreenLake API for Backup and Recovery. These components consist of resources considered to be part of the infrastructure for HPE GreenLake for Backup and Recovery on-premises and in the cloud. These components must be deployed prior to operation of the HPE GreenLake for Backup and Recovery. The on-premises components consist of Data Orchestrator VM, Protection Store Gateway VM or HPE StoreOnce (purpose-built backup appliance). Additionally, the cloud components consist of the data services instance deployed at HPE GreenLake workspace and the cloud protection store to contain the cloud recovery points (backups). The information on how to get started with HPE GreenLake for Backup and Recovery is available in the support <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003102en_us&#x26;page=bar_overview_dscc.html">website</a>.</p> <p><img src="/img/glbr-architecture-overview-1.png" alt="GLBR architecture 1"></p> <p><em>The above figure shows the resources that are part of the infrastructure used to accommodate the data protection for HPE GreenLake for Backup and Recovery.</em></p> <p>The second part of the HPE GreenLake API for Backup and Recovery resources is the resources from which HPE GreenLake for Backup and Recovery provides protection from data loss. In below diagram, you will see those assets which contains the components that need to be protected and the components which the assets are allocated at. Those components, such as virtual machine, datastore, or SQL database application, storage volumes, or physical hosts, are the assets that need to be protected. Nevertheless, other assets in this category include the components for the hypervisor, compute servers, storage array, networking, and the VMware vCenter that manage the hypervisor components. This HPE GreenLake API must maintain the inventory of all the on-premises assets that are protected by querying the VMware vCenter.</p> <p><img src="/img/glbr-architecture-overview-2.png" alt="GLBR architecture 2"></p> <p><em>The above figure displays resources that are parts of the hypervisor and on-premises components that can be protected.</em></p> <blockquote> <p><strong>NOTE:</strong> In the current HPE GreenLake Developer website, there is a single resource categorized as V1 Alpha 1 that will be used to register, unregister, and update the hypervisor-manager to an instance of HPE GreenLake Backup and Recovery. The current supported on-premises hypervisor-manager is VMware vCenter version 7.0 or later. The HPE GreenLake API to discover the already onboarded hypervisor-manager is available from the virtualization API <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/tag/hypervisor-managers/">set</a>.</p> </blockquote> <h3>What about the components in HPE GreenLake Backup and Recovery that are not mentioned above?</h3> <p>There are other resources that exist in the HPE GreenLake for Backup and Recovery that are not mentioned above; however, they are available in the user-interface for HPE GreenLake for Backup and Recovery. A couple of those resources are protection policy and protection group. The protection group is used to consolidate a multiple number of assets with a particular protection policy. On the other hand, the protection-policy is a resource to consolidate the schedule of protection-jobs and the flow of the recovery points at different tiers of protection-store. Together, both protection policy and protection group deliver the management of the Recovery Point Objective for data protection using HPE GreenLake for Backup and Recovery.</p> <p><img src="/img/protection-policy-ui-for-vmware-scheduling.png" alt="Protection policy for vmware schedule"></p> <p><em>The above figure displays resources for schedule and the flow of the recovery-points part of the protection policy.</em></p> <p>Additionally, there is also a resource known as protection job that is obscurely shown in the user-interface; however, it is required to perform the operation of data protection. You will need to manipulate protection jobs to create a protection or to recover an existing recovery point. Additionally, the protection-job is the resource that can be manipulated to suspend or to resume a protection schedule.</p> <p><img src="/img/protection-jobs-ui.png" alt="Protection jobs for scheduling and protection tiers"></p> <p><em>The above figure displays the representation of protection jobs for the protected assets. The protection jobs are available after a protection policy applied to an asset or a protection groups for multiple assets.</em></p> <p>Beyond protection-jobs, there are recovery points (also known as copies). These are resources that are used to represent that recovery points inside the protection store, and they are used to perform the recovery of assets.</p> <blockquote> <p><strong>Note</strong> that even though there are three different categories of recovery-points that are available in the protection policy, there are only two different resources.</p> </blockquote> <p>The first resource is called snapshots to represent copies that exist inside the storage array. The second resource is called backups to represent copies that exist inside the on-premises protection store such as Protection Store Gateway, and copies that exist inside the cloud protection store.</p> <p>The cloud protection store is a repository that sits in the cloud public providers that are managed by HPE, and these can exist either in AWS or Microsoft Azure. All of these recovery points exist in relation to the assets, where the recovery can be initiated using the HPE GreenLake API.</p> <p><img src="/img/protection-copies.png" alt="Protection copies for hierarchy"></p> <h2>Using the Backup and Recovery APIs</h2> <p>This set of Backup and Recovery APIs use the same authorization and permission as the rest of the family of HPE GreenLake API for data services. To ensure that all programmatic interaction with the HPE GreenLake platform services and resources is secure and authenticated, these APIs require an access token. The token is generated using the client ID and client Secret you acquired during the creation of the client API credentials.</p> <p>Documentation concerning getting started with the HPE GreenLake API is provided on the HPE Developer Community <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">website</a> and on the HPE GreenLake Developer portal <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">website</a>. There is also blog <a href="https://developer.hpe.com/blog/learn-what-you-can-do-with-hpe-data-services-cloud-console-api-in-just-3-minutes/">post</a> that describes how to use publicly available tool to manipulate this API without a programming language, such as Postman. An additional blog post that describes using Postman for this API is also available in this <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">link</a>.</p> <p>Lastly, anyone can follow the examples provided by each API reference in the documentation page, such as the <code>GET /backup-recovery/v1beta1/protection-jobs</code>, which is shown in the figure below. The documentation provides detail on the API syntax for a particular method, arguments used for the API, expected successful and failed responses, and several examples of creating the automation script using cURL, JavaScript, Python, and Go. The documentation page also provides the ability to execute the API directly on the documentation page as explained in the previous blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services/">post</a> (Getting started with HP GreenLake Data Services API).</p> <p><img src="/img/reference-document-for-hpe-glbr.png" alt="HPE GLBR API documentation "></p> <p><em>The above figure shows the three-panel of the interactive API reference documentation for one of HPE GreenLake APIs for Backup and Recovery.</em></p> <h2>Some tips and examples</h2> <p>The interactive API reference documentation guide provides information about the parameters and request payload (body) key-pair values required for every available HPE GreenLake API for Backup and Recovery. Additionally, I am presenting some use cases with detailed information on providing the correct parameters or building the correct request payload (body) key-pairs JSON structure required to achieve the use case.</p> <blockquote> <p><strong>NOTE:</strong> The below examples assume that HPE GreenLake Backup and Recovery had been deployed, it was connected to an HPE array onboarded to HPE GreenLake, a VMware vCenter had been discovered, and some virtual machines had been deployed and discovered by HPE GreenLake Backup and Recovery. For more information on getting started with HPE GreenLake Backup and Recovery, please visit the Getting Started guide on HPE support <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=sd00003454en_us&#x26;page=GUID-F25ABD00-C36B-42D8-A443-82584EE8E35A.html">website</a>.</p> </blockquote> <h2>Creating a cloud protection store</h2> <p>A protection store is the critical resource that is required to store the recovery points on-premises and in the cloud. The cloud protection stores are created on top of either the Protection Store Gateway or HPE StoreOnce, because either one is required for connections to cloud protection-stores. To perform this use case in the below example, we will need to discover the StoreOnce and the storage location of the cloud protection store.</p> <blockquote> <p><strong>NOTE:</strong> You will see in this blog post that I used a combination of both HPE GreenLake APIs from <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/guide/">data services</a> and <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/">virtualization</a> to accomplish the below examples.</p> </blockquote> <p>The example below displays the creation of the cloud protection store in the HPE GreenLake protection store at Microsoft Azure cloud storage.</p> <p><img src="/img/ui-to-create-cloud-protection.png" alt="UI to create a cloud protection site on Microsoft Azureat the eastus2 storage location"></p> <p><em>The figure above shows the user interface used to create a cloud store protection at Microsoft Azure in eastus2 storage location using the user interface.</em></p> <p>Here is the list of the steps required to perform this use case using HPE GreenLake API:</p> <ol> <li>I used the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/StoreOncesList/">API</a> to discover the StoreOnce instance that can connect to the cloud protection store and copied the id which will be used as the value for storageSystemId as shown in below JSON request body. The API used for this is: <code>GET /backup-recovery/v1beta1/storonces?limit=20&#x26;offset=0</code>.</li> </ol> <p><img src="/img/api-discover-storeonce.png" alt="Discover deployed StoreOnce to create cloud protection store"></p> <ol start="2"> <li>Next, I discovered the cloud storage at the correct location from the list of the available storage location and copy the <code>storageLocationId</code> as shown in below JSON response body. The discovery was done by using the HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/openapi/data-services-public-v1beta1/operation/ListLocations/">API</a> for Data Services API set. Note from the below figure that I used filter <code>“backup-and-recovery”</code> in capabilities to capture selected storage locations with the correct capability. The below figure shows information about the location (region) where the data will be stored; conversely, it’s located at <code>“Richmond”</code>, cloud service provide was <code>“AZURE”</code>, and cloud services provider identification is <code>“eastus2”</code>. I the copied value <code>"azure:eastus2"</code> from the key <code>id</code> of this API's response body as the <code>storageLocationID</code> value. The API used for this: <code>GET /data-services/v1beta1/storage-locations?filter=”backup-and-recovery” in capabilities</code>.</li> </ol> <p><img src="/img/figure-out-storage-location.png" alt="API to figure out the Azure storage location"></p> <ol start="3"> <li>Moving forward, I composed the cloud protection store at the storage location using values from the previous both API responses. For this API execution, I created a request JSON body structure for <code>POST /backup-recovery/v1beta1/protection-stores</code> that contains some of the key-pair values from the previous API response. The below figure shows that the creation of protection stores using the JSON body structure to compose the cloud protection store that was connected to the HPE StoreOnce.</li> </ol> <blockquote> <p>The below REST API will be executed asynchronously, and I recognized from the Response status that this API was properly executed as shown by the <code>Status 0x202 Accepted</code>. From the Header's location field part of the response body, I copied the task Id and stored that id into variable <code>{{TaskId}}</code> so that I could track the completion of this REST API.</p> </blockquote> <p><img src="/img/api-compose-protection-store.png" alt="API composing the protection store"></p> <blockquote> <p><strong>Note:</strong> The figure below shows all the fields about <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/ProtectionStoreCreate/">this</a> JSON body structures provided on the interactive documentation of <code>POST /backup-recovery/v1beta1/protection-stores</code> under Payload tab.</p> </blockquote> <p><img src="/img/api-to-create-protection-stores-request-json-body.png" alt="API Request body JSON to compose protection Store"></p> <ol start="4"> <li>After roughly five minutes, the cloud protection store was completely created based on response of the following <code>GET /data-services/async-operations/{Id}</code> <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/openapi/data-services-public-v1beta1/operation/GetAsyncOperation/">API</a> where <code>{{TaskId}}</code> came from previously executed API response.</li> </ol> <blockquote> <p><em><strong>Note:</strong></em> I used a set of selection parameters in the figure below to provide a summary of this task information:</p> <pre><code class="language-shellsession">GET /data-services/v1beta1/async-operations/{{taskId}}/select=associatedResources,createdAt,displayName,customerId,logMessages,progressPercent,state </code></pre> <p>I copied the task’s Id from the response header’s location value of the prior API execution into a Postman’s variable called <code>{{taskId}}</code>, and incorporated the variable to the <code>async-operations</code> API execution as shown below.</p> </blockquote> <p><img src="/img/api-async-on-post-protection-stores.png" alt="Task completion on POST protection-stores"></p> <ol start="5"> <li>For reference, I went into the Protection Stores menu as shown in the figure below to show the available HPE GreenLake Backup and Recovery protection-stores, and confirmed that the desired protection store was available.</li> </ol> <p><img src="/img/ui-validated-cloud-protection-store-completed.png" alt="Validating cloud protection store has been created"></p> <h2>Creating a Protection-Policy</h2> <p>A common use-case that every user of HPE GreenLake Backup and Recovery will deploy is the creation of a protection-policy. This resource is important because it sets up the schedule for creation of recovery point. Additionally, it sets up the flow of a recovery point from a primary storage to a storage snapshot, to on-premises protection-store and eventually to the cloud protection-store.</p> <p>A protection policy contains several JSON objects that are displayed in the figure below.</p> <ol> <li>A schedule that contains the frequency for creation of the copy and time to retain the copies.</li> <li>Protection store where the copies are stored: snapshot (in the primary array), on-premises store, and cloud store.</li> <li>Length of time when the copy is being locked to satisfy immutability.</li> <li>The pre-script information that contains the link to the scripts to be executed prior to the creation of the copy.</li> <li>The post-script information that contains the link to the scripts to be executed after the creation of the copy.</li> </ol> <p><img src="/img/api-body-json-request-for-protection-policies.png" alt="API request body JSON for protection policy creation"></p> <p><em>The above figure shows the guide to create a protection-policy request body JSON structure.</em></p> <p>To simplify this example, I created a three-tier protection-policy for VMware as depicted by this snippet from the protection policy’s menu.</p> <p><img src="/img/backup-protection-policy-with-3-tiers.png" alt="3 Tiers protection policy "></p> <p>This is the list of the steps used to create this protection policy:</p> <ol> <li>From the figure below, inside the protection store gateways menu, I discovered the serial number of the protection-store gateway and used it as the filtering parameter to obtain the protection store gateway instance.</li> </ol> <p><img src="/img/psg-ui-to-show-the-serial-no.png" alt="PSG UI to obtain the serial no"></p> <ol start="2"> <li>Afterward, I used <code>GET /backup-recovery/v1beta1/protection-store-gateways</code> <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/ProtectionStoreGatewaysList">API</a> and the <code>filter=serialNumber eq &#x3C;StoreOnce SerialNo></code> to figure out the <code>"&#x3C;protection-store-gateway-id>"</code> that was associated with the protection-stores that would be incorporated into the protection-policy.</li> </ol> <p><img src="/img/api-display-registered-psg.png" alt="API show registered PSG"></p> <ol start="3"> <li> <p>From the figure below, I used <code>GET /backup-recovery/v1beta1/protection-stores</code> <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/ProtectionStoreList/">API</a> to obtain the protection-store ids for both the on-premises protection store and the cloud protection store. To display protection-stores related to the protection store gateway, I used the parameter filter to display the exact the protection-store associated with protection storage gateway of <code>“&#x3C;onprem-protection-store-id>”</code>. The filter parameter that I used are <code>protectionStoreType eq 'ON_PREMISES'</code> and <code>storageSystemInfo/id eq 'protection-store-gateway-id'</code>. Additionally, I used the following <code>select</code> parameter <code>name,displayName,id,status,state,protectionStoreType</code> to provide a shorter response that simplify the discovery of the protection-store on-premises. The API used for this:</p> <pre><code class="language-shellsession">GET /backup-recovery/v1beta1/protection-stores?select=name,displayName,id,status,state,protectionStoreType&#x26;filter=protectionStoreType eq ‘ON_PREMISES’ and storageSystemInfo/id eq “\&#x3C;protection-store-gateway-id\>” </code></pre> </li> </ol> <p><img src="/img/api-to-get-onpremises-protection-store-id.png" alt="API to obtain the onpremises protection store id"></p> <ol start="4"> <li> <p>I repeated the same execution of the <code>GET /backup-recovery/v1beta1/protection-stores</code> to obtain the <code>"&#x3C;cloud-protection-store-id>"</code>. To accomplish that, I used the following <code>filter: protectionStoreType eq 'CLOUD' and storageSystemInfo/id eq “&#x3C;protection-store-gateway-id>”</code>. Additionally, I also used the parameter <code>select: name,displayName,id,status,state,protectionStoreType</code> to provide a shorter response for simpler discovery of the protection-store-id in the cloud. The API used for this:</p> <pre><code class="language-shellsession">GET /backup-recovery/v1beta1/protection-stores?select=name,displayName,id,status,state,protectionStoreType&#x26;filter=protectionStoreType eq ‘CLOUD’ and storageSystemInfo/id eq “&#x3C;protection-store-gateway-id>” </code></pre> </li> </ol> <p><img src="/img/api-discover-cloud-protection-store-id.png" alt="API to obtain cloud protection store id"></p> <ol start="5"> <li>For the next step, I created a request body JSON structure that represents the protection policy schedule and each of the protection stores. Inside this JSON structures for request body, I defined the three objects that represent the <code>SNAPSHOT, BACKUP (on-premises), CLOUD_BACKUP</code>. Note that this structure can be expanded or contracted depending on the required backup strategy. The SNAPSHOT object did not require <code>"&#x3C;protection-store-Id>"</code> as that recovery points will exist inside the primary storage array. This request JSON body structure was required to create the protection policy using HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/DataManagementTemplateCreate/">API</a> <code>POST /backup-recovery/v1beta1/protection-policies</code>.</li> </ol> <blockquote> <p><strong>Note:</strong> I didn’t include objects for immutability, prescript, and postscript into the JSON structure. If it’s not intended, you don’t need to include unused key-pair values into the JSON structure. Additionally, the SNAPSHOT object does no require a <code>protectionStoreId</code>.</p> </blockquote> <pre><code class="language-json">{ "name": "VMware create three tiers", "description": "Snapshot-local-cloud", "protections": [ { "type": "SNAPSHOT", "schedules": [ { "scheduleId": 1, "name": "Array_Snapshot_1", "namePattern": { "format": "Array_Snapshot_{DateFormat}" }, "expireAfter": { "unit": "DAYS", "value": 1 }, "schedule": { "recurrence": "HOURLY", "repeatInterval": { "every": 4 }, "activeTime": { "activeFromTime": "00:00", "activeUntilTime": "23:59" } } } ] }, { "type": "BACKUP", "protectionStoreId": "&#x3C;onprem-protection-store-id>", "schedules": [ { "scheduleId": 2, "name": "On-Premises_Protection_Store_2", "sourceProtectionScheduleId": 1, "namePattern": { "format": "On-Premises_Protection_Store_{DateFormat}" }, "expireAfter": { "unit": "DAYS", "value": 3 }, "schedule": { "recurrence": "DAILY", "repeatInterval": { "every": 1 }, "startTime": "00:00" } } ] }, { "type": "CLOUD_BACKUP", "protectionStoreId": "&#x3C;cloud-protection-stores-id>", "schedules": [ { "scheduleId": 3, "name": "HPE_Cloud_Protection_Store_3", "sourceProtectionScheduleId": 2, "namePattern": { "format": "HPE_Cloud_Protection_Store_{DateFormat}" }, "expireAfter": { "unit": "WEEKS", "value": 1 }, "schedule": { "recurrence": "DAILY", "repeatInterval": { "every": 2 }, "startTime": "00:00" } } ] } ], "applicationType": "VMWARE" } </code></pre> <blockquote> <p><em>The above figure shows JSON structure for request body of <code>POST /backup-recovery/v1beta1/protection-policies</code> for the creation of protection-policy as shown by the example in this section.</em></p> </blockquote> <ol start="6"> <li>Finally, I created the protection policies using the HPE GreenLake API for Backup and Recovery <code>POST /backup-recovery/v1beta1/protection-policies</code>, and I used the above JSON structure in the request body. This is a special POST API execution where the response is returned immediately. The response of this API contained the body of JSON structure that will be useful to identify the protection-jobs such as the <code>“&#x3C;protection-policies-id>”</code>.</li> </ol> <p><img src="/img/api-to-create-protection-policy.png" alt="API to create protection policy"></p> <ol start="7"> <li> <p>The below figure shows the complete response JSON body from the above API that shows the construction of the protection policy with different protection tiers and the schedules associated with the protection tier. The important values were the ids for different protection tiers that will be used for the next example.</p> <pre><code class="language-yaml">ID: “&#x3C;protection-policies-id>" SNAPSHOT: “&#x3C;snapshot-protection-id>” ON-PREMISES: “&#x3C;onprem-protection-id>” CLOUD: “&#x3C;cloud-protection-id>” </code></pre> </li> </ol> <blockquote> <p>The full listing of the response body from <code>POST /backup-recovery/v1beta1/protection-policies</code> is shown in the below code snippet:</p> <pre><code class="language-jsonc">{ "assigned": false, "description": "Snapshot-local-cloud", "id": "&#x3C;protection-policies-id", "name": "VMware create three tiers", "protections": [ { "id": "&#x3C;snapshot-protection-id>", "schedules": [ { "scheduleId": 1, "name": "Array_Snapshot_1", "schedule": { "activeTime": { "activeFromTime": "00:00", "activeUntilTime": "23:59" }, "recurrence": "HOURLY", "repeatInterval": { "every": 4 } }, "expireAfter": { "unit": "DAYS", "value": 1 }, "namePattern": { "format": "Array_Snapshot_{DateFormat}" } } ], "type": "SNAPSHOT" }, { "id": "&#x3C;onprem-protection-id>", "schedules": [ { "scheduleId": 2, "sourceProtectionScheduleId": 1, "name": "On-Premises_Protection_Store_2", "schedule": { "recurrence": "DAILY", "repeatInterval": { "every": 1 }, "startTime": "00:00" }, "expireAfter": { "unit": "DAYS", "value": 3 }, "namePattern": { "format": "On-Premises_Protection_Store_{DateFormat}" } } ], "protectionStoreInfo": { "id": "onprem-protectio-store-id", "name": "Local_CDS-TPM-PSG#1", "type": "backup-recovery/protection-store", "resourceUri": "/backup-recovery/v1beta1/protection-stores/501a99e7-fb79-4fd7-89d5-a5dfb3441859", "protectionStoreType": "ON_PREMISES" }, "type": "BACKUP" }, { "id": “&#x3C;cloud-protection-id>”, "schedules": [ { "scheduleId": 3, "sourceProtectionScheduleId": 2, "name": "HPE_Cloud_Protection_Store_3", "schedule": { "recurrence": "DAILY", "repeatInterval": { "every": 2 }, "startTime": "00:00" }, "expireAfter": { "unit": "WEEKS", "value": 1 }, "namePattern": { "format": "HPE_Cloud_Protection_Store_{DateFormat}" } } ], "protectionStoreInfo": { "id": "cloud-protection-store-id", "name": "Cloud_CDS-TPM-PSGno1_USA, North Virginia", "type": "backup-recovery/protection-store", "region": "USA, North Virginia", "resourceUri": "/backup-recovery/v1beta1/protection-stores/25611df9-xxxx-xxxx-xxxx-aa3a4887869f", "protectionStoreType": "CLOUD" }, "type": "CLOUD_BACKUP" } ], "createdAt": "2024-04-05T03:28:03.000000Z", "createdBy": { "id": "&#x3C;user-id>", "name": "ronald.dharma@hpe.com" }, "generation": 1, "resourceUri": "/backup-recovery/v1beta1/protection-policies/f572ce6e-xxxx-xxxx-xxxx-530284bf2bc4", "consoleUri": "/backup-and-recovery/protection-policies/f572ce6e-xxxx-xxxx-xxxx-530284bf2bc4", "applicationType": "VMWARE", "type": "backup-recovery/protection-policy", "updatedAt": "2024-04-05T03:28:03.000000Z" } </code></pre> </blockquote> <h2>Applying protection policy to a Virtual Machine</h2> <p>Next in the list of use cases for this blog post, I followed the progression for the Day One activities. The next one is to protect a virtual machine that is applying this protection policy to a virtual machine.</p> <p>These are the steps required to apply the protection policy against a virtual machine:</p> <ol> <li> <p>I obtained the values from <code>virtual machine id, name, and type</code> keys that were going to be used as the key-pair values required for the key <code>assetInfo</code> as shown in below figures. To obtain those, I used one of the HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/VirtualMachinesList/">API</a> from Virtualization API set to discover the detail information of a virtual machine <code>“0-Linux-Demo-VM02”</code>. The API used for this:</p> <pre><code class="language-shellsession">GET /virtualization/v1beta1/virtual-machines?sort=name desc&#x26; select=appType,id,name,type,guestInfo,protectionJobInfo&#x26; filter=name eq '0-Linux-Demo-VM02' </code></pre> </li> </ol> <p><img src="/img/api-find-virtual-machines.png" alt="API to find virtual-machines and it&#x27;s properties"></p> <ol start="2"> <li>I created a JSON structure of a request body for applying the protection policy against a virtual machine without a protection group to make a simple example for this blog post. <strong>Note</strong> that this JSON body structure created the association of the asset, which is a virtual machine, against the protection policy. The two other parameters entered here were the consistency <code>(CRASH vs APPLICATION)</code> and technology for protection <code>(VMWARE_CBT vs VOLUME)</code>. The three values obtained from previous response of the previous API <code>POST /backup-recovery/v1beta1/protection-policies</code> associated with SNAPSHOT, BACKUP and CLOUD would be used as part of this request body. Each of these protections will be identified with schedule id marked 1,2 and 3 as shown below. There are other options available as part of this JSON structure of this request body, and they are documented in the Payload tab of this interactive guide for this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/DataManagementJobCreate/">API</a>.</li> </ol> <blockquote> <p><strong>Note:</strong> As of March 2024, value of <code>type</code> from <code>virtualization/virtual-machine</code> was translated to <code>hybrid-cloud/virtual-machine.</code></p> <pre><code class="language-jsonc">{ "assetInfo":{ "id": ”&#x3C;virtual-machine-id>”, "type":"hybrid-cloud/virtual-machine", "name":"0-Linux-Demo-VM02" }, "protectionPolicyId”: “&#x3C;protection-policies-id>”, "overrides":{ "protections":[ { "id": “&#x3C;snapshot-protection-id>”, "schedules":[ { "scheduleId":1, "consistency":"CRASH", "backupGranularity":"VMWARE_CBT" } ] }, { "id": “&#x3C;backup-protection-id>”, "schedules":[ { "scheduleId":2, "consistency":"CRASH", "backupGranularity":"VMWARE_CBT" } ] }, { "id": “&#x3C;cloud-protection-id>”, "schedules":[ { "scheduleId":3, "consistency":"CRASH", "backupGranularity":"VMWARE_CBT" } ] } ] } } </code></pre> </blockquote> <ol start="3"> <li>In the figure below, I used HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/DataManagementJobCreate/">API</a> for Backup and Recovery <code>POST /backup-recovery/v1beta1/protection-jobs</code> with the body JSON structure from the above so that I could associate the protection policy against a virtual machine. This POST API execution completed asynchronously and returned Status <code>0x202</code> with a response. The response header contained the <code>{{taskId}}</code> that I can use to ensure that this API completed properly.</li> </ol> <p><img src="/img/api-applying-the-protection-jobs-against-a-vm.png" alt="API to apply protection policy against the VM"></p> <ol start="4"> <li>The API execution was executed successfully. I validated this completion using the task id that I obtained from the response headers as shown above. The API used for this: <code>GET /data-services/v1beta1/async-operations/{{taskId}}</code></li> </ol> <blockquote> <p><strong>Note</strong>: The execution of this API will trigger a protection execution right after the execution of this API completed.</p> </blockquote> <p><img src="/img/api-task-list-for-protection-policy-application.png" alt="Task list to display completion of the application of protection policy against a VM producing protection jobs"></p> <h2>Triggering a protection</h2> <p>Once the protection policy named "VMware create three tiers" was bound to the virtual machine as shown above, I then issued a trigger to create cloud protection against the virtual machine <code>“0-Linux-Demo-VM02”</code> to create one-off cloud-protection.</p> <p>The steps required to trigger the protection are listed below:</p> <ol> <li> <p>I figured out the <code>protection-job-id</code> that is associated with <code>“0-Linux-Demo-VM02”</code>. and the cloud backup <code>schedule Id</code> of the virtual machine. To achieve that, I used the HPE GreenLake <a href="%60https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/DataManagementJobsList%60">API</a> for Backup and Recovery <code>GET /backup-recovery/v1beta1/protection-jobs</code> and <code>filter assetInfo/id eq "{VM-id}”</code> as shown below. Note that the variable <code>{vmId}</code> contained the value of the virtual machine id as discovered in previous example, namely <code>"&#x3C;virtual-machine-id>"</code>. The response body’s JSON structure contained the id of the protection job associated with <code>“0-Linux-Demo-VM02”</code>. From the same response body, I recognized that cloud protection is the <code>scheduleId no 3</code>. The API used for this:</p> <pre><code class="language-shellsession">GET /backup-recovery/v1beta1/protection-jobs?filter=assetInfo/id eq {{vmId}}&#x26;select=assetInfo,id,operational,protections </code></pre> </li> </ol> <p><img src="/img/api-to-figure-out-protection-jobs.png" alt="API to figure out protection-jobs"></p> <ol start="2"> <li>This use case is equivalent to run now button at the schedule of the <code>“0-Linux-Demo-V02”</code> on the selected cloud protection as shown in below figure. After clicking <code>“Run Now”</code> button, a cloud protection will commence against the virtual machine, a recovery point is going to be created at the cloud protection store. In next step, I will invoke <code>"Run Now"</code> on the <code>HPE Cloud Protection Store</code> using HPE GreenLake API for Backup and Recovery.</li> </ol> <p><img src="/img/gui-run-now-cloud-protection.png" alt="UI for run now protection jobs"></p> <ol start="3"> <li> <p>The HPE GreenLake API used to accomplish the use case shown above was <code>POST /backup-recovery/v1beta1/protection-jobs/:id/run</code>, and the documentation of this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/DataManagementJobRun">API</a> also lists the required JSON structure for the request body. I created a JSON request body structure for this example as follows:</p> <pre><code class="language-jsonc">{ "fullBackup": false, "ScheduleIds": [3] } </code></pre> </li> </ol> <blockquote> <p><strong>Note:</strong> There is a key called <code>“fullBackup”</code> inside the JSON request body to enable the creation of full protection where a backup will be created independently from the existing copies in the protection store. I also entered number 3 into <code>ScheduleIds</code> JSON array, to represent the cloud backup schedule. The below figure shows an example of the execution of <code>"run now"</code> without full backup protection of the third schedule, which is cloud protection of this virtual machine. The value <code>&#x3C;protection-jobs-id></code> will be entered from the parameter of this API in this manner: <code>POST /backup-recovery/v1beta1/protection-jobs/"&#x3C;protection-jobs-id>"/run</code>.</p> </blockquote> <p><img src="/img/api-to-execute-a-protection.png" alt="API to execute a protection run"></p> <ol start="4"> <li>The result from the API execution above can be validated from the API <code>/data-services/v1beta1/async-operations/:id</code> using the task id obtained from the above response header. The API used for this: <code>GET /data-services/v1beta1/async-operations/{{taskId}}</code>.</li> </ol> <p><img src="/img/api-task-list-after-a-run-execution.png" alt="API async-operations of execution of protection-jobs"></p> <p>The activities above were validated from the HPE GreenLake Backup and Recovery list of the recovery points as shown in the figure below.</p> <p><img src="/img/gui-cloud-run-now-completed-succesfully.png" alt="GUI display the completed cloud protection run"></p> <h2>Wow.. those were so cool! Can I recover a new virtual machine from that recovery point that I just created?</h2> <p>Each of the recovery points, regardless of the location of store (array snapshot, On-Premises Protection-Store, or HPE Cloud Protection-Store), can be recovered using the HPE GreenLake APIs for Backup and Recovery.</p> <ol> <li> <p>To restore the backup from a virtual machine, this API requires a request body of JSON structure as documented in the HPE GreenLake developer website. In this blog post, I planned a demo of the steps to recover the virtual machine from a copy that had existed in the HPE Cloud Protection Store into the VMware cluster where the Protection Storage Gateway is hosted (recover as a new virtual machine). Copy the <code>{{vmId}}</code> from the response JSON body to be used on subsequent API execution. The API used for this:</p> <pre><code class="language-shellsession">GET /virtualization/v1beta1/virtual-machines?sort=name desc&#x26;filter=name eq’0-Linux-Demo-VM02’&#x26;select=appType,id,name,type,guestinfo,protectionJobInfo </code></pre> </li> </ol> <p><img src="/img/api-to-discover-vm-for-recovery.png" alt="API to discover backup for recovery of VM"></p> <ol start="2"> <li> <p>Let's obtain the Cloud Recovery Point id for the cloud protection recovery from the virtual machine id from the step #1 using the HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/VirtualMachineBackupList/">API</a> <code>GET /backup-recovery/v1beta1/virtual-machines/:id/backups</code> given the virtual machine id <code>{{vmId}}</code>. Copy the <code>{{backupId}}</code> from the response body from the below figure so that it can be used in the subsequent API execution. The API used for this:</p> <pre><code class="language-shellsession">GET /backup-recovery/v1beta1/virtual-machines/{{vmId}}/backups?select=name,description,backupType,id </code></pre> </li> </ol> <p><img src="/img/api-to-obtain-backup-id-for-recovery.png" alt="API to obtain the backup Id of a cloud recovery point"></p> <ol start="3"> <li> <p>Let's obtain the <code>datastore id</code> and the <code>cluster id</code> from the existing VMFS datastore that can accommodate the datastore type that this VM can be restored. In this hypervisor, I am using the datastore with the name <code>“0-BRS-VMFS-Test3”</code> and enter that as part of the filter into HPE GreenLake <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/DatastoresList">API</a> <code>GET /virtualization/v1beta1/datastores</code>. The API used for this:</p> <pre><code class="language-shellsession">GET /virtualization/v1beta1/datastores?filter=displayName eq '0-BRS-VMFS-Test3'&#x26;select=clusterInfo,datastoreType,name,id </code></pre> </li> </ol> <p><img src="/img/api-obtain-cluster-and-datastore.png" alt="API to obtain the cluster and datastore Ids"></p> <ol start="4"> <li> <p>To obtain the hypervisor Network Id that is required to recover the recovery point into a new virtual machine, I had to discover the hypervisor id. The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/HypervisorManagerList/">API</a> used for this:</p> <pre><code class="language-shellsession">GET /virtualization/v1beta1/hypervisor-manager?select=name,id,state,status,dataOrchestratorInfo,services,hypervisorManagerType,releaseVersion&#x26;filter=state eq "OK" and status eq "OK" and name eq "vCenter Name" </code></pre> </li> </ol> <p><img src="/img/api-obtain-hypervisor-id.png" alt="API to get hypervisorId"></p> <ol start="5"> <li> <p>To set the virtual machine network to the correct network port group, I'll need to look into into the existing virtual machine <code>"0-Linux-Demo-V02"</code>, and discovered the network port group <code>“Mgmt-DPortGroup”</code>. From the list of the network port group, I selected the associated network port group of the virtual machine by filtering the name of the port group. Afterward, we copied the <code>"&#x3C;hypervisor-network-id>"</code> from the response body of this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/HypervisorNetwork/">API</a> to be used for subsequent execution. The API used for this:</p> <pre><code class="language-shellsession">GET /virtualization/v1beta1/hypervisor-managers/{{hyperVisorId}}/networks?select=id,displayName&#x26;filter=displayName eq 'Mgmt-DPortGroup' </code></pre> </li> </ol> <p><img src="/img/api-to-get-the-network-id.png" alt="API to get network ID for VM"></p> <ol start="6"> <li> <p>After I obtained all the values that were required to build the request body to recover a cloud recovery point from <code>“0-Linux-Demo-VM02"</code>, I constructed the JSON request body as shown in the below figure. To restore the recovery point into a new virtual machine, the <code>restoreType</code> key of the request body JSON structure was set to <code>“ALTERNATE”</code> as shown in the below figure. I also provided the new virtual machine name after the recovery as <code>“0-Linux-Demo-VN02-2-05-04-2024_05:48_PM”</code>. The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/openapi/backup-recovery-public-v1beta1/operation/VirtualMachineRestore/">API</a> used for this:</p> <pre><code class="language-shellsession">POST /backup-recovery/v1beta1/virtual-machines/{{vmId}}/restore </code></pre> </li> </ol> <p><img src="/img/api-restoring-a-cloud-protection-recovery-point.png" alt="API to recover a cloud protection copy from a VM"></p> <ol start="7"> <li> <p>To validate that the recovery was completed, I tracked the progress from the response using the <code>async-operations</code> <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/openapi/data-services-public-v1beta1/operation/GetAsyncOperation/">API</a> based on <code>{{taskId}}</code> from previous execution as shown below. The API used for this:</p> <pre><code class="language-shellsession">GET /data-services/v1beta1/async-operations/:id?select=associatedResources,createdAt,endedAt,error,displayName,healthStatus,id,customerId,progressPercent,name,type,state </code></pre> </li> </ol> <p><img src="/img/task-display-recovery-is-completed.png" alt="Task Id confirming the completion of the recovery"></p> <ol start="8"> <li>At the end, I went to the VMware vCenter console used in this example to validate that the virtual machine 0-Linux-Demo-VN02-2-05-04-2024_05:48_PM was indeed part of the virtual machines inventory.</li> </ol> <p><img src="/img/the-vcenter-display-the-recovered-vm.png" alt="vCenter display the VM recovery completed"></p> <h2>Summary</h2> <p>This blog post introduces the new set of REST API from the family of the APIs for data services on HPE Greenlake, namely <strong>HPE GreenLake API for Backup and Recovery</strong>. This set of APIs is documented at the HPE GreenLake for developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/backup-recovery/public/">website</a> using interactive documentation based on OpenAPI Standard version 3.1.</p> <p>Early in this blog post, I laid down the relationship of the resources in this set of HPE GreenLake APIs with the objects in the HPE GreenLake Backup and Recovery user interface. In this blog post, I also introduced the examples from several use cases associated with utilizing HPE GreenLake for Backup and Recovery to provide virtual machine protection from day one.</p> <p>The examples presented in this blog post provided some guides on using combination of the REST APIs that were announced in March 2024 to achieve the goal for protecting a virtual machine.</p> <p>All the execution for the examples were done using Postman API tool without any scripting language to encourage anyone to experiment with the family of REST APIs for data services on HPE GreenLake.</p> <p>Please don’t hesitate to explore this new set of APIs for Cloud Data Services on HPE GreenLake and see how you can improve your agility in managing your data. Any questions on HPE GreenLake Data Services Cloud Console API? Please join <a href="https://developer.hpe.com/slack-signup">the HPE Developer Community Slack Workspace</a>, and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services</a> Slack channel.</p>https://developer.hpe.com/2024-april-05/https://developer.hpe.com/2024-april-05/Fri, 05 Apr 2024 13:00:00 GMThttps://developer.hpe.com/supercharged-chapel-editor-support/https://developer.hpe.com/supercharged-chapel-editor-support/Thu, 04 Apr 2024 13:43:29 GMT<p>External blog</p>https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-virtualization/https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-virtualization/Wed, 03 Apr 2024 21:18:41 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>What’s New?</h2> <p>Recently, a new set of REST APIs for HPE GreenLake edge-to-cloud platform was introduced on the HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/guide/">website</a>. These APIs are grouped under the set which is called HPE GreenLake API for Virtualization. Several articles will be written and posted on HPE Developer Community <a href="https://developer.hpe.com/blog/">blog</a> to help you better understand and work with these APIs in conjunction with the family of APIs for data services on HPE GreenLake.</p> <p>This the second in a series of blog posts (<a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services">data services</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-virtualization">virtualization</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-backup-and-recovery">Backup and Recovery</a>) that will introduce some useful tips and best practices about using this new set of APIs given a specific use case. The purpose of this virtualization API is described from the HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public">Guide</a>: “The HPE GreenLake for Virtualization API provides management of virtual machines and other virtual resources in public clouds and on-premises systems.”</p> <p>At time of release, this set of APIs' supports for on-premises hypervisor includes VMware (7.X and 8.X) and cloud public provider such as AWS Elastic Cloud Compute and Microsoft Azure virtual machine. The resources that are supported on-premises are <em>virtual machines, virtual machine images, datastores, VMware clusters (hosts, folders, networks, tags, networks, resource-pool)</em>. Conversely, the resources that are supported on cloud providers include <em>virtual machine instance, virtual machine images, and virtual machine instance types</em>.</p> <p><img src="/img/virtualization-api-guide-website-and-download.png" alt="" title="The HPE GreenLake API for Virtualization documentation."></p> <p><em>The above figure shows the HPE GreenLake APIs for Virtualization drop down list, and the link to download the OpenAPI specification in JSON format.</em></p> <p>The specification for this set of APIs is published as an OpenAPI specification in JSON format. The specification of this set of APIs is available for download from this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/overview/">section</a> of the documentation (shown below). The specification follows the OpenAPI standard 3.1, which contains all the information required so that this JSON spec-file can be consumed by any OpenAPI tools to generate client library, SDK, server mock, or documentation as described in this OpenAPI <a href="https://tools.openapis.org/">Initiative</a>.</p> <p><img src="/img/virtualization-openapi-3.1-spec-file.png" alt="" title="HPE GreenLake OAS 3.1 specification for Virtualization download."></p> <p><em>The above figure shows a sample of the downloaded OpenAPI specification of the HPE GreenLake API for Virtualization.</em></p> <h2>API versioning</h2> <p>This set of APIs is identified as revision V1 Beta 1 at the time of its introduction in March 2024. Moving forward, the APIs will be updated to their next revision as they evolves toward the long-term release version. As each individual API is updated, there will also be more capabilities added to any of the resources identified under the APIs. Furthermore, there will be more resources that are not currently available for this API added in the future. For information about update stages, and deprecation, please follow the HPE GreenLake Developer Portal Versioning <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/public/standards/versioning_basics/">Guide</a>.</p> <h2>What are these virtualization resources?</h2> <p>The following pictures depict some of the resources that are related to the virtualization APIs that can be discovered inside the two cloud data services that are part of HPE GreenLake. The two services which leverage these virtualization APIs are HPE GreenLake for Backup and Recovery (GLBR), and HPE GreenLake for Private Cloud Business Edition (PCBE). Both services leverage the virtualization APIs to discover assets that would need to be onboarded, protected, orchestrated, nurtured, or retired following the CRUD (create, revise, update, delete) principles of REST API. Each object presented in the pictures below are part of the User Interface that can be manipulated using this virtualization APIs.</p> <blockquote> <p><strong>Note:</strong> Not all virtualization API resources that can be seen at HPE GreenLake for Backup and Recovery user interfaces and HPE GreenLake for Private Cloud Business Edition user interfaces, are going to be available inside this set of APIs upon its first release. Due to sharing of the virtualization services between the two services, any virtualization resources that are added into one HPE GreenLake workspace and used by both services can be manipulated by HPE GreenLake APIs for Virtualization using the same instance Id.</p> </blockquote> <p><img src="/img/onprem-resources-virtualization-api-glbr.png" alt="" title="On-prem resources in GLBR"></p> <p><em>The above figure shows virtualization APIs resources related to VMware in HPE GreenLake for Backup and Recovery.</em></p> <p><img src="/img/onprem-resources-virtualization-api-pcbe.png" alt="" title="on-prem resources in PCBE"></p> <p><em>The above figure shows virtualization APIs resources related to VMware in HPE GreenLake for Private Cloud Business Edition.</em></p> <p><img src="/img/cloud-public-resources-inside-the-glbr.png" alt="" title="Cloud resources in GLBR"></p> <p><em>The above figure shows virtualization APIs resources related to Public Cloud Provider in HPE GreenLake for Backup and Recovery.</em></p> <h2>Using the HPE GreenLake APIs for Virtualization</h2> <p>This set of virtualization APIs uses the same authorization and permission as the rest of the family of HPE GreenLake APIs for data services.</p> <p>To ensure that all programmatic interaction with the HPE GreenLake platform services and resources is secure and authenticated, these APIs require an access token. The token is generated using the client ID and client Secret you obtained during the creation of the client API credentials. Documentation about getting started with the HPE GreenLake API is provided on the HPE Developer Community <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">website</a>, and on the HPE GreenLake Developer portal <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">website</a>.</p> <p>Nevertheless, there is also blog <a href="https://developer.hpe.com/blog/learn-what-you-can-do-with-hpe-data-services-cloud-console-api-in-just-3-minutes/">post</a> that describes how to use publicly available tools to manipulate this API without a programming language, such as Postman. An additional blog post that describes using Postman for this API is also available at this <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console">link</a>. Moreover, there will be blog posts available that provide guidance on how to convert this Open API JSON file based on the OpenAPI 3.1 to any scripting language library.</p> <p>Lastly, anyone can follow the examples provided by each virtualization API referenced in the HPE GreenLake Developer's documentation page, such as shown in the below figure. The documentation provides detail on the API syntax for a particular method, arguments used for the API, successful and failed responses, and several examples using cURL, JavaScript, Python, and Go. The documentation page also provides the ability to execute the API directly on the documentation page as explained in the previous blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services/">post</a> (Getting started with HP GreenLake APIs for Data Services ).</p> <p><img src="/img/get-all-registered-hypervisor-managers-download.png" alt="" title="Get all registered hypervisors managers"></p> <p><em>The above figure shows three-panel interactive API reference documentation for one of HPE GreenLake API for virtualization.</em></p> <h2>Some tips and examples</h2> <p>Even though there is documentation available in the HPE GreenLake Developer portal, here are some recommendations and best practices for using the HPE GreenLake API for virtualization.</p> <blockquote> <p><strong>Note:</strong> All of the below examples assume that you have access to HPE GreenLake workspace. For more information on acquiring an HPE GreenLake workspace, please follow this <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=a00120892en_us">guide</a>.</p> </blockquote> <h3>Discovery of services for a given hypervisor</h3> <p>The discovery of the on-premises assets in a HPE GreenLake workspace is started by obtaining the access token, and applying it to the virtualization API <code>GET {baseURL}/virtualization/v1beta1/hypervisor-managers</code> to discover the hypervisors that are already onboarded into the your workspace.</p> <p>Along with the information about the hypervisor, the response of <code>GET {baseURL}/virtualization/v1beta1/hypervisor-managers</code> provides additional information such as which HPE GreenLake services are associated with the discovered hypervisor. To discover the information, you can use the <code>select</code> parameter with the API to discover properties such as <code>dataOrchestratorInfo</code> and <code>services</code>. The response values that are returned from this API execution provide the information on what services are associated with the hypervisor and the instance of Data Orchestrator VM that is providing protection against that hypervisor.</p> <p>The recommended select parameters to discover the hypervisor and services related to hypervisor is shown below.</p> <pre><code class="language-shellsession">GET https://{baseUrl}/virtualization/v1beta1/hypervisor-managers?select=name,id,state,status,dataOrchestratorInfo,services,hypervisorManagerType,releaseVersion </code></pre> <p>The example of the response using the above recommended parameter is shown below. From this response information, I could derive that the hypervisor with its name <code>cds-tme-vcenter.rtplab.nimblestorage.com</code> was associated with <code>backup-and-recovery</code> service. However, the second hypervisor with its name <code>rtp-arra392-dhci.rtplab.nimblestorage.com</code> was associated with both the <code>hci-manager</code> and <code>backup-and-recovery</code> services. These values gave me an idea that this workspace contained two sets of VMware vCenters, both of which were protected by HPE GreenLake for Backup and Recovery; however, only the second one was part of HPE GreenLake for Private Cloud Business Edition.</p> <pre><code class="language-json">{ "items": [ { "dataOrchestratorInfo": { "id": "d36122b2-xxxx-xxxx-xxxx-15d2521926f0", "resourceUri": "/backup-recovery/v1beta1/data-orchestrators/d36122b2-xxxx-xxxx-xxxx-15d2521926f0", "type": "backup-recovery/data-orchestrator" }, "hypervisorManagerType": "VMWARE_VCENTER", "id": "0c9acaa0-xxxx-xxxx-xxxx-eaa239798f6d", "name": "cds-tme-vcenter.rtplab.nimblestorage.com", "releaseVersion": "7.0.3", "services": [ "backup-and-recovery” ], "state": "OK", "status": "OK" }, { "dataOrchestratorInfo": { "id": "d36122b2-xxxx-xxxx-xxxx-15d2521926f0", "resourceUri": "/backup-recovery/v1beta1/data-orchestrators/d36122b2-xxxx-xxxx-xxxx-15d2521926f0", "type": "backup-recovery/data-orchestrator" }, "hypervisorManagerType": "VMWARE_VCENTER", "id": "213a7f0d-xxxx-xxxx-xxxx-44e597c32121", "name": "rtp-array392-dhci.rtplab.nimblestorage.com", "releaseVersion": "7.0.3", "services": [ "hci-manager", "backup-and-recovery" ], "state": "OK", "status": "OK" } ], "count": 2, "offset": 0, "total": 2 } </code></pre> <p><em>The above code snippet displays the response from <code>GET the hyper-managers</code> API execution</em>.</p> <h3>Creation and management of a virtual machine in a cloud service provider (CSP)</h3> <p>The figure below shows the documentation of HPE GreenLake API for virtualization to deploy a virtual machine in your CSP account: <code>POST /virtualization/v1beta1/csp-machine-instances</code>. Note that this API requires that user to provide a <code>request body</code> as part of the API execution. The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/CreateCSPMachineInstance/">documentation</a> for this API at HPE GreenLake Developer website used another terminology which is called <code>Payload</code>. It is presented as one of the tabs in the <code>Request samples</code> window, as shown in the figure below. This <code>Payload</code> tab provided details of the JSON structure required for the execution of the API <code>POST /virtualization/v1beta1/csp-machine-instance</code>. Inside the JSON body structures, I recognized that I need to provide multiple key-pair values such as</p> <pre><code class="language-shellsession">accountId, imageId, instanceType, region, cspType, keyPairName, name </code></pre> <p><img src="/img/documentation-of-the-post-csp-machine-instance-with-required-payload.png" alt="" title="display csp-machine-instances with the required Payload to POST"></p> <p><em>The above figure shows the documentation on <code>POST /virtualization/v1beta1/csp-machine-instances</code> to deploy a VM inside the cloud service provider.</em></p> <p>At the time of this release (March 2024), the API resource to discover the HPE GreenLake cloud service provider (CSP) <code>account id</code> is not yet available in this released set of HPE GreenLake APIs for Virtualization. To display that id, I used an existing legacy HPE GreenLake API <code>GET https://{baseUrl}/api/v1/csp-accounts</code>. My AWS CSP account was already onboarded into HPE GreenLake Private Cloud Business Edition, hence the prior API returned the <code>accountId</code> that I need to provide as one of the key-pair object in the JSON request body for the virtualization API <code>POST /virtualization/v1beta1/csp-machine-instances</code>.</p> <p><img src="/img/api-of-csp-account-to-obtain-accountid-csptype.png" alt="" title="List available CSP accounts (DSCC API v1.4)"></p> <p><em>The above figure shows the accountId and the cspType required for the deployment of the virtual machine on the CSP account.</em></p> <p>Another important information to deploy a virtual machine into my CSP account is the <code>imageId</code> value. That <code>imageId</code> (shown below) corresponded to the <code>AWS Linux VM (free tier)</code> which was a machine-image (template of the virtual machine) that is going to be deployed at the AWS account. I used the following virtualization API <code>GET /virtualization/v1beta1/csp-machine-images</code> to validate AWS machine image (AMI) from existing virtual machine that had been deployed inside my CSP account.</p> <p><img src="/img/api-of-csp-machine-image-based-on-known-aws-machine-id-in-us-east-1.png" alt="" title="return imageId from csp-machine-images"></p> <p><em>The above figure shows the imageId value obtained from the existing machine instance.</em></p> <p>You can see in the below figure, I performed the deployment of a virtual machine using Postman tool. This tool provided the special field for me to enter the JSON body structure according to the definition of Payload in the developer’s API <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/CreateCSPMachineInstance/">guide</a>. Please note that I used the raw form of the body with JSON structure (shown below). Combining the information that I gathered above, I entered the values required by the JSON structures part of the request body. The key of <code>keyPairName</code> was the name for the certificate key-pair created in my CSP account. I chose <code>RonD-deploy-CSP-1</code> for the name of the deployed virtual machine. Additionally, I used instanceType of <code>t2.micro</code> as the instance that was appropriate for this demo.</p> <p><img src="/img/post-creation-of-a-virtual-machine-inside-the-csp-account.png" alt="" title="Deploy a machine instance based on the payload POST"></p> <p><em>The above figure provides the example of creating a Payload (Body) JSON structure to provision a VM inside the cloud service provider.</em></p> <p>Using the API <code>GET async-operations</code> on the <code>task Id</code> provided from location value in the response header of the above API, I was able to track the completion of the execution of the create virtual-machine in the AWS account. For more information on using the <code>GET async-operations</code> API, please take look at my blog <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services/">post</a> (Getting Started with HPE GreenLake API for Data Services).</p> <pre><code class="language-json">{ “associatedResources”: [], “childTasks”: [ { “name”: “Create Instance”, “resourceUri”: “/data-services/v1beta1/async-operations/286cc55c-8636-4a8a-8428-0a5694f42785”, “type”: “task” } ], “createdAt”: “2024-03-28T01:53:30.395518007Z”, “customerId”: “xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” “displayName”: “Create Instance: RonD-deploy-CSP-1”, “endedAt”: “2024-03-28T01:54:04.245722969Z”, “error”: null, “estimatedRunningDurationMinutes”: 0, “generation”: 3, “groups”: [ { “id”: “eb988b5e2dcb11ec840712b3b5263ef4”, “name”: “Default Group” } ], “healthStatus”: “OK”, “id”: “5f54e591-xxxx-xxx-xxxx-bebadb977f93”, “logMessages”: [ { “message”: “Create Instance: RonD-deploy-CSP-1 task is created”, “timestampAt”: “2024-03-28T01:53:30.395526907Z” }, { “message”: “Create Instance: RonD-deploy-CSP-1 task is running”, “timestampAt”: “2024-03-28T01:53:30.395529707Z” }, { “message”: “Create Instance: RonD-deploy-CSP-1 task is succeeded”, “timestampAt”: “2024-03-28T01:54:04.245712087Z” } ], “name”: “Create Instance: RonD-deploy-CSP-1”, “parentTask”: null, “progressPercent”: 100, “recommendations”: [], “resourceUri”: “/data-services/v1beta1/async-operations/5f54e591-a231-46ce-a2e2-bebadb977f93”, “rootTask”: { “id”: “5f54e591-a231-46ce-a2e2-bebadb977f93”, “name”: “”, “resourceUri”: “/data-services/v1beta1/async-operations/5f54e591-a231-46ce-a2e2-bebadb977f93”, “type”: “task” }, “services”: [ “private-cloud-business-edition” ], “startedAt”: “2024-03-28T01:53:30.395520016Z”, “state”: “SUCCEEDED”, “subtreeTaskCount”: 1, “suggestedPollingIntervalSeconds”: 30, “type”: “task”, “updatedAt”: “2024-03-28T01:54:04.324786754Z”, “userId”: “xxxx.yyyy@abc.com" } </code></pre> <p><em>The above snippet of the response body depicts that execution of <code>POST /virtual/v1beta1/machine-instance</code> that was completed successfully.</em></p> <p>To progress further, I needed to find the id that corresponds to the newly created virtual-machine instance in the AWS CSP account (<code>csp-machine-instance id</code>). To find the virtual machine Id of the deployed VM, I used the legacy HPE GreenLake API <code>GET {baseUrl}/api/v1/csp-machine-instances</code> to get the list of virtual-machine-instance that exist in that AWS account. From the response of that API, I obtained the VM instance Id of that Virtual Machine that was created in the prior example using <code>filter:</code> parameter (shown below).</p> <p><img src="/img/get-csp-machine-instance-id-to-validate-the-body-structure.png" alt="" title="Get information about the VM deployed in the CSP to validate parameters"></p> <p><em>The above figure displays the name and the csp-machine-instance id so that I could manipulate the machine's state.</em></p> <p>To control cost, I used the API <code>POST {baseUrl}/virtualization/v1beta1/csp-machine-instances/:id/power-off</code> to power off the virtual-machine instance in the AWS account. Lastly, I invoked another API <code>DEL {baseUrl}/virtualization/v1beta1/csp-machine-instances/:id</code> to terminate that virtual machine and retire it (delete) from the inventory of my AWS CSP account. Just like any invocation of <code>POST API</code>, the invocation of DEL method was asynchronously executed. Hence, the same strategy of using the <code>GET async-operations</code> API would apply. Finally, the result of the execution will be returned as part of the response API from <code>GET async-operation</code> indicated that VM that I created recently has already been terminated, as shown below.</p> <pre><code class="language-json">{ "associatedResources": [], "childTasks": [ { "name": "Terminate: RonD-deploy-CSP-1", "resourceUri": "/data-services/v1beta1/async-operations/f267be2d-5a04-4f41-a760-c803212840c8", "type": "task" } ], "createdAt": "2024-03-27T22:37:20.509665935Z", "customerId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "displayName": "Terminate: RonD-deploy-CSP-1", "endedAt": "2024-03-27T22:37:37.652544571Z", "error": null, "estimatedRunningDurationMinutes": 0, "generation": 4, "groups": [ { "id": "eb988b5e2dcb11ec840712b3b5263ef4", "name": "Default Group" } ], "healthStatus": "OK", "id": "de6d9725-7803-4ef5-9cce-f40ad05f879f", "logMessages": [ { "message": "Terminate: d4c46c6e-51ff-59c0-a24a-ef52d77f0b16 task is created", "timestampAt": "2024-03-27T22:37:20.509674904Z" }, { "message": "Terminate: d4c46c6e-51ff-59c0-a24a-ef52d77f0b16 task is running", "timestampAt": "2024-03-27T22:37:20.509678268Z" }, { "message": "Terminate: RonD-deploy-CSP-1 task is running", "timestampAt": "2024-03-27T22:37:20.79512845Z" }, { "message": "Terminate: RonD-deploy-CSP-1 task is succeeded", "timestampAt": "2024-03-27T22:37:37.652529902Z" } ], "name": "Terminate: RonD-deploy-CSP-1", "parentTask": null, "progressPercent": 100, "recommendations": [], "resourceUri": "/data-services/v1beta1/async-operations/de6d9725-xxxx-xxxx-xxxx-f40ad05f879f", "rootTask": { "id": "de6d9725-7803-4ef5-9cce-f40ad05f879f", "name": "", "resourceUri": "/data-services/v1beta1/async-operations/de6d9725-xxxx-xxxx-xxxx-f40ad05f879f", "type": "task" }, "services": [ "private-cloud-business-edition" ], "sourceResource": { "name": "RonD-deploy-CSP-1", "resourceUri": "/api/v1/csp-machine-instances/d4c46c6e-xxxx-xxxx-xxxx-ef52d77f0b16", "type": "AWS Instance" }, "startedAt": "2024-03-27T22:37:20.509668598Z", "state": "SUCCEEDED", "subtreeTaskCount": 1, "suggestedPollingIntervalSeconds": 30, "type": "task", "updatedAt": "2024-03-27T22:37:37.688700579Z", "userId": "ronald.dharma@hpe.com" } </code></pre> <p><em>The above figure shows the termination of machine-instance in CSP reached 100% progress and completed successfully.</em></p> <h3>Wow… that was so cool. What about provisioning a VMware datastore on-premises?</h3> <p>Here is another example of using the HPE GreenLake API for virtualization to provision a datastore out of an HPE disaggregated Hyperconverged Infrastructure (dHCI) system for on-premises deployment. The HPE dHCI instance had already been onboarded into the HPE GreenLake for Private Cloud Business Edition service (PCBE). The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/virtualization/public/openapi/virtualization-public-v1beta1/operation/CreateDS/">API</a> that is used for the datastore provisioning is <code>POST {baseUrl} /virtualization/v1beta1/datastores</code>, which also required a definition of this JSON structure with values in the <code>body (Payload)</code>. The information about this request JSON body is presented in the figure below.</p> <p><img src="/img/create-datastore-on-dhci.png" alt=""></p> <p><em>The above figure displays the required body (Payload) used to deploy datastore into a hyper-converged instance with VMware.</em></p> <p>This virtualization API incorporated the virtual machine provisioning policy that is part of the HPE GreenLake for Private Cloud Business Edition. I used the HPE GreenLake for Private Cloud Business Edition API <code>GET /private-cloud-business/v1beta1/vm-provisioning-policies</code> to obtain the <code>provisioningPolicyId</code> and entered it into the Body JSON structure as required.</p> <p><img src="/img/get-virtualization-api-to-obtain-vm-provisioning-policy.png" alt="obtain the provisioning policy for deploying datastores using HPE PCBE" title="get the provisioning policy for deploying datastores in a dHCI using HPE PCBE"></p> <p>I used another API from the virtualization API such as <code>GET {baseUrl}/virtualization/v1beta1/hypervisors-clusters</code> to obtain the <code>targetHypervisorClusterId</code> (shown below).</p> <p><img src="/img/obtain-the-cluster-id-for-deployment-at-dhci-using-pcbe.png" alt="" title="Obtain the cluster id which is required to deploy a datastore in a dHCI using PCBE"></p> <p>Furthermore, I used the legacy API such as <code>GET /api/v1/storage-systems/device-type2</code> to obtain the <code>Storage System Id</code> (shown below).</p> <p><img src="/img/obtain-the-storage-system-id-using-the-legacy-api.png" alt="" title="legacy API to display the storage system Id"></p> <p>Finally, I entered all the above values into the JSON body below as part of the virtualization API to create a datastore in the designated VMware cluster using the attached storage system. The Integer value key <code>sizeInBytes</code> was equivalent to 10 TB. Other String values were obtained from the previous HPE GreenLake API responses as I demonstrated above.</p> <p><img src="/img/post-virtualization-api-with-taskid-succesful.png" alt="" title="Deploy POST to create the datastore in the DHCI using PCBE"></p> <p>From the response of the API <code>GET aync-operations</code> for the <code>{task Id}</code> shown below, I confirmed that creation of the datastore was completed. Note that the creation of the datastore also applied the HPE GreenLake Backup Recovery data protection-policy to the newly created datastore.</p> <pre><code class="language-json">{ "associatedResources": [], "childTasks": [ { "name": "Create datastore: VMFS-ds2", "resourceUri": "/data-services/v1beta1/async-operations/58955ba4-59ba-4f78-a4e8-5116aaba9e7d", "type": "task" } ], "createdAt": "2024-04-21T22:08:25.439053461Z", "customerId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "displayName": "Provisioning datastore VMFS-ds2", "endedAt": "2024-04-21T22:12:00.738144472Z", "error": null, "estimatedRunningDurationMinutes": 0, "generation": 8, "groups": [ { "id": "eb988b5e2dcb11ec840712b3b5263ef4", "name": "Default Group" } ], "healthStatus": "OK", "id": "8de5e0a2-af09-4de3-8334-75638c147735", "logMessages": [ { "message": "Task created", "timestampAt": "2024-04-21T22:08:25.439071294Z" }, { "message": "Task is running", "timestampAt": "2024-04-21T22:08:25.439074148Z" }, { "message": "Preparing parameters", "timestampAt": "2024-04-21T22:08:29.779163168Z" }, { "message": "Creating datastore", "timestampAt": "2024-04-21T22:08:31.200651622Z" }, { "message": "Validating datastore", "timestampAt": "2024-04-21T22:09:44.256999998Z" }, { "message": "Applying protection policy", "timestampAt": "2024-04-21T22:10:14.382076894Z" }, { "message": "Task succeeded", "timestampAt": "2024-04-21T22:12:00.738160833Z" } ], "name": "Provisioning datastore VMFS-ds2", "parentTask": null, "progressPercent": 100, "recommendations": [], "resourceUri": "/data-services/v1beta1/async-operations/8de5e0a2-af09-4de3-8334-75638c147735", "rootTask": { "id": "8de5e0a2-af09-4de3-8334-75638c147735", "name": "", "resourceUri": "/data-services/v1beta1/async-operations/8de5e0a2-af09-4de3-8334-75638c147735", "type": "task" }, "services": [ "private-cloud-business-edition" ], "sourceResource": { "name": "VMFS-ds2", "resourceUri": "/api/v1/datastores/ff5a8bf8-ddc9-5bca-9f9f-c4cf3f099364", "type": "Datastore" }, "startedAt": "2024-04-21T22:08:25.439056085Z", "state": "SUCCEEDED", "subtreeTaskCount": 2, "suggestedPollingIntervalSeconds": 30, "type": "task", "updatedAt": "2024-04-21T22:12:00.793422464Z", "userId": "ronald.dharma@hpe.com" } </code></pre> <p><em>The above figure shows that API <code>POST {baseURL)/virtualization/v1beta1/datastores</code> executed succesfully.</em></p> <h2>Summary</h2> <p>This blog post introduces you to the new set of HPE GreenLake APIs for Virtualization, which is part of data services family APIs for HPE GreenLake. This is the set of APIs to support resources from an on-premises hypervisor, that includes VMware (7.X and 8.X), as well as for cloud-public-providers such as AWS Elastic Cloud Compute and Microsoft Azure Virtual Machines. This set of APIs will evolve through future releases. The March 2024 announcement introduces version V1 Beta 1 of this set of APIs. This set of APIs is documented at HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com">website</a> and it is available for download as Open API 3.1 specification in JSON file.</p> <p>In this post, I started by introducing the HPE GreenLake virtualization API on the API documentation page. Afterward, I provided a couple of tips on using this virtualization REST API such as to discover the on-premises hypervisor resources and to determine the services that were applied against the discovered hypervisor's cluster. Next, I provided a complete guides to deploy a virtual machine inside the cloud service provider (AWS). Lastly, I also introduced an example on how to invoke the DEL method of the virtual machines at the cloud to clean up the deployed virtual machines on the cloud service provider. As a bonus, I also provided an example to deploy datastores in the on-premises VMware infrastructure part of the HPE GreenLake Private Cloud Business Edition. In future blogs, I will provide examples that incorporate HPE GreenLake APIs for virtualization with the family of APIs from different data services on HPE GreenLake.</p> <p>Please don’t hesitate to explore this new set of HPE GreenLake APIs for virtualization and see how you can improve your agility in managing your data. Any questions on HPE GreenLake APIs for virtualization? Or do you have any suggestions or cool ideas that you want share with the community? Please join <a href="https://developer.hpe.com/slack-signup/">the HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services</a> Slack channel.</p>https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-data-services/https://developer.hpe.com/getting-started-with-hpe-greenlake-api-for-data-services/Tue, 02 Apr 2024 00:37:09 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>What’s New?</h2> <p>Recently, a new set of REST APIs for HPE GreenLake edge-to-cloud Platform was introduced on the HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/">website</a>. These APIs are grouped into a set called HPE GreenLake APIs for Data Services. Several articles will be written and posted on <a href="https://developer.hpe.com/blog">HPE Developer's forum blog site</a> to help you better understand and use HPE GreenLake APIs to enhance your DataOps' operations.</p> <p>This is the first in a series of blog postings (<a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-data-services">Data-Services</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-virtualization">Virtualization</a>, <a href="https://developer.hpe.com/blog/getting-started-with-hpe-greenlake-api-for-backup-and-recovery">Backup and Recovery</a> that introduce some useful tips and best practices about using this new set of APIs given a specific use case. The introduction of these APIs arises from the necessity for manipulation of the common resources that are shared by the existing family of data services on HPE GreenLake (DataOps Manager, Block Storage, virtualization, Backup and Recovery, Private Cloud Business Edition). This set of APIs provide users with the ability to perform any Create, Read, Update and Delete (CRUD) operations against these resources: <em>async-operations, dual-auth-operations, issues, secrets, software-releases, storage locations, and tags</em>.</p> <p>The specification for these APIs is publicized as an OpenAPI specification in JSON format, and the specification is available for download from this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/openapi/data-services-public-v1beta1/overview/">section</a> of the documentation (shown below). Anyone can download the JSON file that contain the specification for this set, by clicking on the <strong>Download</strong> button. The specification follows the OpenAPI standard 3.1 that contains all information required so that this JSON OpenAPI spec-file can be consumed by any OpenAPI tools to provide client library, server mock, or documentation as described in this OpenAPI <a href="https://tools.openapis.org/">Initiative</a>.</p> <p><img src="/img/data-services-api-download-page.png" alt="Figure 1. HPE GreenLake API for Data Services List" title="Figure 1. HPE GreenLake API for Data Services List"></p> <p><em>The above figure shows the list of HPE GreenLake APIs for Data Services</em></p> <p><img src="/img/data-services-api-json-file.png" alt="Figure 2. An example of the downloaded data-services.json contents." title="Figure 2. An example of the downloaded data-services.json contents."></p> <p><em>The above figure shows an example of the downloaded data-services.json contents.</em></p> <p>This set of APIs is identified as revision V1 Beta 1 at the time of its introduction in March 2024. Moving forward, the API will be updated to its next revision, moving toward the long-term release version. As each individual API is updated, there will also be more capabilities added to any of the resources identified under this set of APIs. Furthermore, there will be more resources that are not currently available for this API added in the future. For information about management of versioning, please follow the HPE GreenLake Developer Portal Versioning Guide at this <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/public/standards/versioning_basics/">link</a>.</p> <h2>What are these data services' resources?</h2> <p>The following pictures depict some of the resources that are related to Data Services that can be discovered on the main page for the Data Services Cloud Console (UI). Other resources, such as software-releases, storage-locations and tags, are embedded inside data services storage objects. The examples of fore mentioned resources are the software deployed for Backup and Recovery’s Data Orchestrator, the location of the storage repository for Backup and Recovery’s cloud protection store, and tags associated with the storage array.</p> <blockquote> <p><strong>Note:</strong> The resources that are presented as <strong>Tasks</strong> in this main page are identified as <strong>async-operations</strong>, which is the universal API resource used to monitor completion or status of all services under data services on HPE GreenLake. Additionally, the async-operations API is also used to track any API operations that are running in background as covered later in this blog post. Future iterations of the API release will also add more resources, e.g. email-notification, or add more capabilities, e.g. POST/PATCH for tag (currently GET is the only available method for tag).</p> </blockquote> <p><img src="/img/data-services-resources-in-dscc-ui.png" alt="" title="Figure 3 Some of the resources that are managed by Data Services API."></p> <p><em>The above figure shows some of the resources that are managed by HPE GreenLake APIs for data services</em></p> <h2>Using the HPE GreenLake APIs for data services</h2> <p>Documentation on using this set of APIs is provided on the HPE GreenLake Developer Portal at this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/guide/">link</a>. There is also a blog post that describes how to use publicly available tools, such as Postman, to manipulate this set of APIs without using any programming languages found at this <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">link</a>. Additional blog post that describes about using Postman for this set of APIs is also found at this <a href="https://developer.hpe.com/blog/learn-what-you-can-do-with-hpe-data-services-cloud-console-api-in-just-3-minutes/">link</a>. Moreover, there will be blog posts available that provide guidance on how to convert this data services OpenAPI spec that is based on OpenAPI spec 3.1 to a scripting client library in future. Anyone can follow the examples provided by each API reference in the documentation <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/">page</a> such as what’s shown below. The documentation provides details on the API syntax for a particular method, arguments used for the API, successful and failed responses, and several examples using cURL, JavaScript, Python, and Go. The documentation page also provides capability to execute the API directly from the API reference documentation, as explained in the following paragraph.</p> <p><img src="/img/api-documentation-try-it-now.png" alt="" title="Figure 4 Documentation for Data Services REST API."></p> <p><em>The above figure shows the three-panel interactive API reference guide for HPE GreenLake APIs for data services.</em></p> <h2>An example of executing an API Call from the API reference documentation</h2> <blockquote> <p><strong>Note:</strong> All of the below examples assume that you have access to HPE GreenLake workspace. For more information on acquiring an HPE GreenLake workspace, please follow this <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=a00120892en_us">guide</a>.</p> </blockquote> <p>You can start by obtaining an access token for the workspace where you have the permission. For information on getting the access token, please see the following <a href="https://developer.greenlake.hpe.com/docs/greenlake/services">guide</a>, or follow my other blog posts in this <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">link</a>. Once the access token is obtained, please copy it (Ctrl-c short cut in Microsoft Window) so that you can enter that token into the <strong>Security>Bearer Token:</strong> field of the API reference documentation page displaying the list of <code>async-operations</code> (as an example).</p> <p>To start with this example, go to the documentation page for the <code>async-operations</code> on this <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/openapi/data-services-public-v1beta1/operation/ListAsyncOperations/">link</a>. Please click on the <strong>Try It</strong> button located at the top right side of the page, as shown below.</p> <p><img src="/img/tryit-process.png" alt=""></p> <p>Once you are at the next page, select the end-point of the region under <strong>Target Server</strong> menu where your workspace based on your access token is obtained. For my example, it would be: <code>&#x3C;https://us1.data.cloud.hpe.com></code> because my workspace was created at the US region. Afterward, please expand the <strong>Security> Bearer Token:</strong> field to enter the access token (Bearer Token).</p> <p><img src="/img/enter-bearer-token-and-target-end-point.png" alt=""></p> <p>Paste the valid access token into the <strong>Bearer Token</strong> using Ctrl-v (short cut in Microsoft Windows). Keep in mind that the access token will expire after 2 hours past from the time of creation. If the access token has expired, please obtain a new access token using your existing client-credentials.</p> <p><img src="/img/send-the-api-after-token-entered.png" alt=""></p> <p>Click the <strong>Send</strong> button to execute that API and you will see the response page indicating a good status (<code>Status: 0x200</code>). Finish up by clicking on the <strong>Body: Expand All</strong> button to display the completed response body in JSON.</p> <p><img src="/img/response-from-the-tryit-test.png" alt=""></p> <p>Congratulations! You have executed your first HPE GreenLake API for data services using the interactive API reference documentation page in the HPE GreenLake Developer website.</p> <h2>Some tips and examples</h2> <p>Even though there is documentation available in the HPE GreenLake Developer website, here are some of the recommendations and best practices on how to use the API.</p> <h3>async-operations</h3> <p>The responses from this resource are critical for debugging and monitoring the activities that happen from any API executions. Those APIs can be from Virtualization, Backup and Recovery, Block services, or Private Cloud Business Edition. Here is a tip on how to filter out those tasks (<code>async-operations</code>) that belong to a particular service. Use the parameter: <code>filter: &#x3C;service> in services</code> like below.</p> <blockquote> <p><strong>Note</strong>: To simplify the response returned by this API, use the parameter: <code>select: associatedResources,displayName,services,createdAt</code> as shown below.</p> </blockquote> <p><img src="/img/async-operations-invocation-parameters.png" alt="" title="Execution of async-services using filter, sort, and select parameters"></p> <p>The response from the execution of <code>GET /data-services/v1beta1/async-operations</code> is provided below. From this response page, the property <code>associatedResources</code> pointed to the particular asset that encountered the execution. Additionally, the property <code>services</code> indicated which set of service that this <code>associatedResources</code> API execution applicable to.</p> <p><img src="/img/api-response-from-filtered-and-selected-async-response.png" alt=""></p> <p>The corresponding task message from the HPE GreenLake task’s UI is shown below.</p> <p><img src="/img/corresponding-task-for-the-async-operations.png" alt=""></p> <h3>active-issues</h3> <p>This API provides the list of the issues that require attention by your HPE GreenLake users. The HPE GreenLake UI provides a bell icon on the top right of every HPE GreenLake's window (please see the previous UI image under <em>What are these data services resources?</em> paragraph in this blog post) to access issues from every available services under the HPE GreenLake.</p> <p>To limit the display of properties from responses returned by this API, user can use parameter <code>select</code> as part of the parameter of the API execution. However, there are minimal set of properties required by <code>active-issues</code> API to be entered into the <code>select</code> parameter as shown in the below response from the <code>active-issues</code> API.</p> <pre><code class="language-json">https://&#x3C;region-baseUrl>/data-services/v1beta1/issues?select=body { "error": "Missing required field(s) - \"lastOccurredAt,customerId,createdAt,id,resourceUri,generation,type\"", "errorCode": "422", "traceId": "18e7b6adf90315de57f2b177652e3649" } </code></pre> <p>To alleviate this condition, the user can add the required properties to the <code>select</code> parameter in addition to any other property that is desired, such as shown below.</p> <pre><code class="language-json">https://&#x3C;region-baseUrl>/data-services/v1beta1/issues?select=body,lastOccuredAt,customerId,createdAt,id,resourceUri,generation,type </code></pre> <p>Following the above recommendation, the following request body shows where I entered the select parameters as required above.</p> <p><img src="/img/api-request-issue-required-select-parameters.png" alt=""></p> <p>The completed execution of this API is shown below.</p> <p><img src="/img/api-response-with-select-for-active-issues.png" alt="Response from issues with the correct select"></p> <p><em>The above figure shows the output from GET issues given the parameter: <code>select=body,createdAt,customerId,generation,id,lastOccuredAt,resourceURI,type,updatedAt</code>.</em></p> <h3>Completing POST method for REST API using <code>async-operations</code> API with <code>task id</code></h3> <p>Almost any HPE GreenLake REST API for data services with <code>POST, DELETE or PATCH</code> methods (e.g. <code>POST /virtualization/v1beta1/virtual-machines</code>) will be executed asynchronously. The asynchronous execution means that execution of the API will complete and return with <code>response status= 0x202</code>. The POST REST API process will run in the background; nonetheless, this operation must be monitored until it comes to completion.</p> <p>To accomplish that monitoring, the user will receive a <code>task id</code> value as part of <code>location</code> response property as shown in the figure below. The user ould then poll that task id using the <code>GET /data-services/v1beta1/async-operations/{Task Id}</code> to retrieve the progress and status of the completion. Below is an example of this use case, where I executed the creation of virtual machines in the on-premises hypervisor (VMware vCenter)</p> <p>I executed the REST API <code>POST https://{baseURL}/virtualization/v1beta1/virtual-machines</code> and the response was completed with response status code of <code>0x202 (Accepted)</code>. Moreover, you can discover the task Id value: <code>0xcad794d1-27ec-4050-bed4-45d13a8de9d0</code> from the location property.</p> <p><img src="/img/location-output-contains-the-task-id.png" alt="The task Id from response location field"></p> <p><em>The above figure displays the response header from <code>POST https://{baseUrl}/virtualization/v1beta1/virtual-machines</code>.</em></p> <p>From the task Id that was obtained from the response header, I used <code>GET async-operations</code> with the <code>specific task ID</code> (e.g. <code>https://{baseUrl}/data-services/v1beta1/async-operations/cad794d1-27ec-4050-bed4-45d13a8de9d0</code>) to obtain the status and progress of the previously executed REST API.</p> <p><img src="/img/api-request-async-operations-request-of-particular-id.png" alt="Request async-response for a particular id"></p> <p>The following response snippets depict two different responses from the polling using the <code>async-operations</code> API. The first response indicated the progress of the associated API execution at <code>40% (RUNNING)</code>, and the second response indicated the progress at <code>100%(SUCCEEDED)</code>. The progress between the two point of executions was about less than 3 minutes as shown by the difference from the following properties: <code>startedAt</code> and <code>endedAt</code>.</p> <p>Below is the first poll of the VM provisioning REST API task id:</p> <pre><code class="language-json">{ “displayName”: “Provisioning virtual machine 0-RRD-API-Deploy-4”, “endedAt”: “2024-03-24T00:13:53.558031307Z”, “healthStatus”: “OK”, “id”: “cad794d1-27ec-4050-bed4-45d13a8de9d0”, “logMessages”: [ { “message”: “Task created”, “timestampAt”: “2024-03-24T00:13:52.002673131Z” }, { “message”: “Task is running”, “timestampAt”: “2024-03-24T00:13:52.002675372Z” }, { “message”: “Preparing parameters”, “timestampAt”: “2024-03-24T00:13:53.368619324Z” }, { “message”: “Starting virtual machine deployment”, “timestampAt”: “2024-03-24T00:13:53.558043002Z” } ], “name”: “Provisioning virtual machine 0-RRD-API-Deploy-4”, “progressPercent”: 40, “services”: [ “private-cloud-business-edition” ], “startedAt”: “2024-03-24T00:13:52.002663421Z”, “state”: “RUNNING”, “suggestedPollingIntervalSeconds”: 30, “type”: “task”, “updatedAt”: “2024-03-24T00:13:55.846052959Z” } </code></pre> <p><em>The above response displays the result from the first poll of the VM provisioning REST API task Id with <code>progressPercent:40</code> and <code>state: RUNNING</code></em>.</p> <p>Below is the last poll of the VM provisioning REST API task id:</p> <pre><code class="language-json">{ "displayName": "Provisioning virtual machine 0-RRD-API-Deploy-4", "endedAt": "2024-03-24T00:15:49.906710665Z", "error": null, "healthStatus": "OK", "id": "cad794d1-27ec-4050-bed4-45d13a8de9d0", "logMessages": [ { "message": "Task created", "timestampAt": "2024-03-24T00:13:52.002673131Z" }, { "message": "Task is running", "timestampAt": "2024-03-24T00:13:52.002675372Z" }, { "message": "Preparing parameters", "timestampAt": "2024-03-24T00:13:53.368619324Z" }, { "message": "Starting virtual machine deployment", "timestampAt": "2024-03-24T00:13:53.558043002Z" }, { "message": "Applying protection policy", "timestampAt": "2024-03-24T00:15:49.598976645Z" }, { "message": "Virtual machine provisioning completed and initiated a task for applying backup policy.", "timestampAt": "2024-03-24T00:15:49.906721852Z" }, { "message": "Task succeeded", "timestampAt": "2024-03-24T00:15:49.906727488Z" } ], "name": "Provisioning virtual machine 0-RRD-API-Deploy-4", "progressPercent": 100, "services": [ "private-cloud-business-edition" ], "startedAt": "2024-03-24T00:13:52.002663421Z", "state": "SUCCEEDED", "suggestedPollingIntervalSeconds": 30, "type": "task", "updatedAt": "2024-03-24T00:15:49.955699371Z" } </code></pre> <p><em>The above response displays that at the moment of the second poll of the VM provisioning REST API <code>task Id</code>, the creation of the virtual machines on-premises had completed successfully (<code>progressPercent: 100, state: SUCCEEDED</code>).</em></p> <p>At that moment, I also discovered a virtual machine with the name <code>0-RRD-API-Deploy-4</code> available at the VMware cluster where this provisioning was executed.</p> <h1>Summary</h1> <p>This blog post introduces you to the new set of HPE Greenlake APIs for Data Services, to support resources such as: <em>async-operations, dual-auth-operations, issues, secrets, software-releases, storage locations, and tags.</em> This set of APIs will evolve throughout the future toward the long term supported version and the number of APIs will expand to support more resources in the future.</p> <p>This March 2024 announcement introduces revision V1 Beta 1 of the API, which is documented at HPE GreenLake Developer <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/data-services/public/">website</a> using the interactive reference documentation based on OpenAPI 3.1 standard. In this post, I also introduced methods to exercise the API directly from the API reference documentation page using the access token obtained from HPE GreenLake API gateway. Lastly, I provided a list of tips on using this HPE GreenLake API for the specific use cases.</p> <p>Please don’t hesitate to explore this new set of HPE GreenLake APIs for data services and see how you can improve your agility in managing your data. If you have any questions on any of these APIs, or if you are interested sharing your feedback and use cases on this set of APIs, please join the HPE Developer Slack <a href="https://developer.hpe.com/slack-signup">Workspace</a> and start a discussion in our <em><a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services</a></em> Slack channel.</p>https://developer.hpe.com/introducing-chapelcon-24-the-chapel-event-of-the-year/https://developer.hpe.com/introducing-chapelcon-24-the-chapel-event-of-the-year/Tue, 02 Apr 2024 00:05:29 GMT<p>External blog</p>https://developer.hpe.com/deploying-super-mario-game-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/deploying-super-mario-game-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/Fri, 29 Mar 2024 17:53:09 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my recent blog post, I showed you <a href="https://developer.hpe.com/blog/exposing-an-application-using-ingress-and-tls-termination-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/">how to expose applications using Ingress and TLS termination on Kubernetes (K8s) in HPE GreenLake for Private Cloud Enterprise</a>. Let's have a little fun practicing this through a real-world use case where I walk you through the steps of deploying gaming applications, like <em>Super Mario</em> and <em>Tetris</em>, on K8s in the HPE GreenLake for Private Cloud Enterprise. By using K8s Ingress, TLS termination, and a range of suitable tools, <em>Super Mario</em> and <em>Tetris</em> can be made available and securely accessible via HTTPS. The setup I show here strictly adheres to the rigorous security and compliance standards of the K8s production environment in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>Utilizing <em>YAML</em> manifest files or <em>Helm</em> charts along with Docker images, the installation of gaming applications on the K8s cluster is a straightforward process. Tools like <em>kubectl</em>, <em>helm</em>, and <a href="https://kustomize.io/">Kustomize</a> are available for this purpose. The complexity arises when it comes to securely exposing the deployed games for external access over HTTPS, a common requirement for on-premises K8s clusters. This involves the generation and management of SSL/TLS certificates for the games within the cluster. These certificates are vital for secure inter-service communication. Proper installation and management are key to preventing access issues and security threats. As game traffic increases, particularly during peak usage hours, it becomes crucial to set up gaming applications with load balancing access. This presents a significant challenge ensuring the availability of load balancing for gaming applications running on K8s.</p> <p>Here's some of the things you need to deploy <em>Super Mario</em> and <em>Tetris</em> in an HPE GreenLake for Private Cloud Enterprise cluster and expose them using K8s Ingress and TLS termination. Remember that <a href="https://developer.hpe.com/blog/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/">MetalLB</a> is employed to establish the load balancer in the cluster and <a href="https://developer.hpe.com/blog/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/">cert-manager</a> is deployed for the generation and management of SSL/TLS certificates, which are stored as K8s <em>Secret</em> objects and made available to the entire cluster upon creation. The <a href="https://www.nginx.com/products/nginx-ingress-controller/">Nginx Ingress controller</a> is deployed within the cluster. The Ingress TLS configuration is used to decrypt encrypted traffic over HTTPS at the load balancer setup and forward the decrypted traffic to the target gaming applications. This configuration offloads the resource-intensive cryptographic operations to the dedicated load balancer, allowing the backend gaming applications to concentrate on efficiently processing client requests and responses. The gaming applications are deployed with the <em>ClusterIP</em> service type in the backend, providing internal connectivity and being solely accessible from within the cluster. They do not directly handle SSL/TLS encryption and decryption.</p> <p><img src="/img/game-deploy.png" alt=""></p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em> CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The <em>helm</em> CLI tool, version 3.12.0 or later</li> <li>A domain and a list of subdomains to generate the SSL certificate and host the game applications in the cluster</li> <li>The optional <em>openssl</em> CLI tool, for validating the generated certificates</li> </ul> <h3>Deploy Super Mario</h3> <p>The game <em>Super Mario</em>, together with <em>Tetris</em>, can be deployed to the cluster using the <em>YAML</em> manifest files from the GitHub repo <a href="https://github.com/GuopingJia/k8s-games">k8s-games</a>:</p> <pre><code class="language-shell">$ tree k8s-games/ k8s-games/ ├── README.md ├── super-mario │   ├── deployment.yaml │   └── service.yaml └── tetris ├── deployment.yaml └── service.yaml </code></pre> <p>Type the following commands to deploy <em>Super Mario</em> and <em>Tetris</em> to the namespace <em>cfe-games</em> in the cluster:</p> <pre><code class="language-shell">$ kubectl create ns cfe-games namespace/cfe-games created $ kubectl apply -f super-mario/ -n cfe-games deployment.apps/mario-deployment created service/mario-service created $ kubectl apply -f tetris/ -n cfe-games deployment.apps/tetris-deployment created service/tetris-service created </code></pre> <p>Type the command shown below to check the details of the game deployment:</p> <pre><code class="language-shell">$ kubectl get all -n cfe-games NAME READY STATUS RESTARTS AGE pod/mario-deployment-96f79d8f-dw9hh 1/1 Running 0 19s pod/mario-deployment-96f79d8f-wsf7s 1/1 Running 0 13s pod/tetris-deployment-86d744fb47-7kmwl 1/1 Running 0 7s pod/tetris-deployment-86d744fb47-hqmgd 1/1 Running 0 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mario-service ClusterIP 10.104.144.88 &#x3C;none> 80/TCP 22s service/tetris-service ClusterIP 10.111.218.14 &#x3C;none> 80/TCP 10s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mario-deployment 2/2 2 2 24s deployment.apps/tetris-deployment 2/2 2 2 12s NAME DESIRED CURRENT READY AGE replicaset.apps/mario-deployment-96f79d8f 2 2 2 24s replicaset.apps/tetris-deployment-86d744fb47 2 2 2 12s </code></pre> <p>Two games, <em>mario-deployment</em> and <em>tetris-deployment</em>, are deployed in the cluster, each running with 2 Pod replicas by default. They are exposed as the <em>ClusterIP</em> type of services, providing internal connectivity and solely being accessible from within the cluster.</p> <p>You can configure the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/"><em>Horizontal Pod Autoscaling</em> (HPA)</a> in the cluster by using the K8s <em>HorizontalPodAutoscaler</em> resource. It will automatically scale the workload by deploying more Pods in the cluster according to gaming application memory or CPU usage.</p> <p>Type the following command to check that all the game service endpoints have been populated:</p> <pre><code class="language-shell">$ kubectl get endpoints -n cfe-games NAME ENDPOINTS AGE mario-service 10.192.3.118:80,10.192.4.32:80 60s tetris-service 10.192.3.119:3000,10.192.4.33:3000 50s </code></pre> <h3>Set up the load balancer with <em>MetalLB</em></h3> <p>You can install <em>MetalLB</em> and set up the load balancer in the K8s cluster by following the instructions shown in the blog post <a href="https://developer.hpe.com/blog/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/">Setting up the load balancer with MetalLB</a>.</p> <p>Here is the deployed <em>MetalLB</em> to the namespace <em>metallb-system</em> in the cluster:</p> <pre><code class="language-shell">$ kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/controller-57b4fdc957-dr4h4 1/1 Running 0 18d pod/speaker-9kx9h 1/1 Running 0 18d pod/speaker-d6sdh 1/1 Running 0 18d pod/speaker-gxbbx 1/1 Running 0 18d pod/speaker-hflbj 1/1 Running 0 18d pod/speaker-wfw9n 1/1 Running 0 18d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/webhook-service ClusterIP 10.107.242.167 &#x3C;none> 443/TCP 18d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/speaker 5 5 5 5 5 kubernetes.io/os=linux 18d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/controller 1/1 1 1 18d NAME DESIRED CURRENT READY AGE replicaset.apps/controller-57b4fdc957 1 1 1 18d </code></pre> <p>By running the following commands, you can see a range of virtual IP addresses, <em>"10.6.115.251-10.6.115.254"</em>, defined in the CRD resource <em>IPAddressPool</em>, and the layer 2 service IP address announcement in the CRD resource <em>L2Advertisement</em>:</p> <pre><code class="language-shell">$ kubectl get ipaddresspools -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES cfe-pool true false ["10.6.115.251-10.6.115.254"] $ kubectl get l2advertisements -n metallb-system NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES cfe-l2advert ["cfe-pool"] </code></pre> <h3>Generate a self-signed certificate using cert-manager</h3> <p>You can deploy cert-manager to the K8s cluster and generate a self-signed certificate by following the instructions found in the blog post <a href="https://developer.hpe.com/blog/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/">Generating self-signed certificates using cert-manager</a>.</p> <p>Here is the cert-manager deployed to the namespace <em>cert-manager</em> in the cluster:</p> <pre><code class="language-shell">$ kubectl get all -n cert-manager NAME READY STATUS RESTARTS AGE pod/cert-manager-6bcdd5f7c-f7lfw 1/1 Running 0 18d pod/cert-manager-cainjector-5d4577b4d9-jmpsp 1/1 Running 0 18d pod/cert-manager-webhook-bf957dc77-s9r2g 1/1 Running 0 18d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.109.28.203 &#x3C;none> 9402/TCP 18d service/cert-manager-webhook ClusterIP 10.100.82.119 &#x3C;none> 443/TCP 18d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 18d deployment.apps/cert-manager-cainjector 1/1 1 1 18d deployment.apps/cert-manager-webhook 1/1 1 1 18d NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-6bcdd5f7c 1 1 1 18d replicaset.apps/cert-manager-cainjector-5d4577b4d9 1 1 1 18d replicaset.apps/cert-manager-webhook-bf957dc77 1 1 1 18d </code></pre> <p>Below is the deployed self-signed custom resource definition (CRD) <em>Issuer</em> in the namespace <em>cfe-games</em> where the game applications are deployed. You want to generate the certificate to this namespace.</p> <pre><code class="language-shell">$ kubectl get issuer -n cfe-games NAME READY AGE cfe-selfsigned-issuer True 10s </code></pre> <p>Here is the generated self-signed certificate in the namespace <em>cfe-games</em>:</p> <pre><code class="language-shell">$ kubectl get certificate -n cfe-games NAME READY SECRET AGE cfe-selfsigned-tls True cfe-tls-key-pair 8s </code></pre> <p>The K8s <em>Secret</em> <em>'cfe-tls-key-pair'</em> is created automatically in the same namespace as part of certificate deployment:</p> <pre><code class="language-shell">$ kubectl get secrets -n cfe-games cfe-tls-key-pair NAME TYPE DATA AGE cfe-tls-key-pair kubernetes.io/tls 3 35s </code></pre> <p>Type the following <em>openssl</em> command to check the content of the generated certificate:</p> <pre><code class="language-shell">$ openssl x509 -in &#x3C;(kubectl get secret -n cfe-games cfe-tls-key-pair -o jsonpath='{.data.tls\.crt}' | base64 -d) -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 2d:0e:ee:67:d2:e0:e2:e6:bc:f2:9a:da:2b:78:66:86 Signature Algorithm: sha256WithRSAEncryption Issuer: CN = example.com Validity Not Before: Feb 21 17:33:40 2024 GMT Not After : May 21 17:33:40 2024 GMT Subject: CN = example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:d7:88:2a:e6:67:20:62:e4:25:f8:cd:63:b7:75: bf:ac:d4:5a:8a:32:1c:06:29:17:96:cb:6b:36:97: 7f:9b:1d:f2:d6:f2:a4:f1:63:32:9b:7f:42:a1:31: 40:b6:02:ec:0b:37:a6:60:fb:11:72:28:96:91:90: 55:26:c5:58:3c:dd:a0:4b:a2:ab:33:19:29:88:24: da:73:81:af:99:9b:df:7f:26:14:36:1b:56:93:24: e9:91:d0:89:e1:62:d0:45:22:64:0b:c4:1d:96:71: ab:ee:61:94:00:f6:60:71:10:10:fc:3e:d1:6b:b6: 5b:0b:bf:18:0c:86:90:b0:f9:eb:78:8c:dc:90:4e: ef:87:1f:ac:22:56:2b:92:23:ae:fe:bb:48:1e:13: 40:03:b7:54:02:44:8f:ae:c6:61:bf:d4:e9:f7:17: 72:a8:98:72:b7:a6:e0:16:29:8d:ca:4a:1e:08:89: 78:f7:88:b7:ac:d2:b8:8d:89:88:c3:c7:04:f4:ff: 00:64:37:6f:3f:5a:43:2c:ce:e4:69:b2:a8:44:fe: 77:41:ec:97:b8:7b:82:49:b0:65:8e:fc:1f:1c:2b: 37:ea:46:9d:e4:5c:a0:56:9f:d8:3b:78:83:28:b5: ac:a9:61:ce:25:c7:54:c8:a3:96:f6:a8:48:f4:57: 56:3b Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 3F:DD:BB:BB:DB:23:47:E1:EC:39:1E:BE:03:AC:D4:7E:2A:E2:A6:FA X509v3 Subject Alternative Name: DNS:super-mario.example.com, DNS:tetris.example.com, DNS:example.com Signature Algorithm: sha256WithRSAEncryption 78:46:61:2d:b8:27:fe:18:59:b2:57:ef:88:2b:2f:20:9f:a5: 4a:28:33:64:46:78:e3:c4:7f:40:4a:38:ad:ca:0a:2e:7d:31: 7f:70:81:e1:50:b6:4e:a5:02:31:bf:26:44:89:b2:1f:5c:3d: 63:b8:62:bf:9c:b3:f0:96:76:bb:b0:3e:47:0e:bc:5e:fa:9c: 9c:98:36:1d:2f:72:3d:b9:11:30:94:b0:2e:2f:a3:57:18:07: 5d:bf:aa:0d:c6:36:20:2a:8f:a6:11:7c:e4:2f:03:07:2e:c4: cd:33:07:3f:c2:54:30:e0:bf:d1:8e:20:0a:bc:a3:90:39:46: d4:ed:03:c2:71:a1:43:b4:a6:c0:73:13:14:ea:a4:52:39:8f: 72:59:00:1a:5f:1c:6e:1e:b7:4d:b5:9e:43:cd:e7:89:5a:07: ad:ce:41:f4:5a:cd:73:ee:bc:f4:01:73:92:9d:c4:a6:f1:8d: eb:43:af:65:78:8d:f0:e6:c3:df:bc:44:ca:19:c5:da:3f:a2: 4d:89:fa:8e:63:33:3d:4d:8d:b3:98:3b:d9:12:c0:d9:3a:82: 07:bc:81:fb:5d:c9:e5:38:3c:ec:d3:3e:e9:bc:e4:13:84:07: f3:c7:85:8a:46:ba:69:13:c7:a8:14:42:4b:ee:f9:2a:b4:3b: d9:8f:9c:50 </code></pre> <p>The line <em>X509v3 Subject Alternative Name</em> contains the <em>dnsNames</em>, <em>'super-mario.example.com'</em> &#x26; <em>'tetris.example.com'</em>, which host two games, <em>Super Mario</em> and <em>Tetris</em>, respectively in the cluster.</p> <h3>Deploy Nginx Ingress controller</h3> <p>In order for an Ingress to work in the cluster, there must be an Ingress controller being deployed and running. It's the Ingress controller that accesses the certificate and the routing rules defined on the Ingress resource and makes them part of its configuration.</p> <p>A variety of Ingress controllers are available for deployment in the cluster, including <a href="https://doc.traefik.io/traefik/providers/kubernetes-ingress/">Traefik</a>, <a href="https://github.com/haproxytech/kubernetes-ingress#readme">HAProxy</a> and <a href="https://www.nginx.com/products/nginx-ingress-controller/">Nginx Ingress controller</a>. Execute the command below to install the Nginx Ingress controller to the cluster using <em>Helm</em>:</p> <pre><code class="language-shell">$ helm upgrade --install ingress-nginx ingress-nginx \ > --repo https://kubernetes.github.io/ingress-nginx \ > --namespace ingress-nginx --create-namespace Release "ingress-nginx" does not exist. Installing it now. NAME: ingress-nginx LAST DEPLOYED: Wed Mar 6 18:30:55 2024 NAMESPACE: ingress-nginx STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the load balancer IP to be available. You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - pathType: Prefix backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: &#x3C;base64 encoded cert> tls.key: &#x3C;base64 encoded key> type: kubernetes.io/tls </code></pre> <p>The Nginx Ingress controller is deployed to the namespace <em>ingress-nginx</em> in the cluster. Type the following command to check the deployment details:</p> <pre><code class="language-shell">$ kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-controller-5957546d75-zjwjh 1/1 Running 0 15d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.98.254.246 10.6.115.251 80:30209/TCP,443:30833/TCP 15d service/ingress-nginx-controller-admission ClusterIP 10.109.187.223 &#x3C;none> 443/TCP 15d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 15d NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-5957546d75 1 1 1 15d </code></pre> <p>The service <em>ingress-nginx-controller</em> gets deployed as the service type of <em>LoadBalancer</em> with the <em>EXTERNAL-IP</em> assigned as <em>10.6.115.251</em>. This IP address will be used for setting up domain and subdomain name resolution.</p> <h3>Set up Ingress TLS</h3> <p>The Ingress resource with TLS has to be created. Here is a sample Ingress TLS resource <em>ingress-host-based-selfsigned-games.yaml</em>:</p> <pre><code class="language-shell">$ cat ingress-host-based-selfsigned-games.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-host-based-selfsigned annotations: ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/issuer: "cfe-selfsigned-issuer" spec: ingressClassName: nginx tls: - hosts: - example.com secretName: cfe-tls-key-pair rules: - host: super-mario.example.com http: paths: - path: / pathType: Prefix backend: service: name: mario-service port: number: 80 - host: tetris.example.com http: paths: - path: / pathType: Prefix backend: service: name: tetris-service port: number: 80 </code></pre> <p>In the above sample YAML manifest file, there is the <em>'tls'</em> block that contains the hostname <em>'example.com'</em> and the secret <em>cfe-tls-key-pair</em> created in the certification step. There is also the <em>'rules'</em> block in which a list of routing rules is defined per host, e.g., the host <em>'super-mario.example.com'</em> will be routed to the Super Mario game service <em>'mario-service'</em> in the backend.</p> <p>Type the following command to deploy the Ingress resource to the namespace <em>cfe-games</em>:</p> <pre><code class="language-shell">$ kubectl apply -f ingress-host-based-selfsigned-games.yaml -n cfe-games ingress.networking.k8s.io/ingress-host-based-selfsigned created </code></pre> <p>Check the details of the <em>TLS</em> and <em>Rules</em> settings by typing the following command:</p> <pre><code class="language-shell">$ kubectl describe ingress ingress-host-based-selfsigned -n cfe-games Name: ingress-host-based-selfsigned Labels: &#x3C;none> Namespace: cfe-games Address: Ingress Class: nginx Default backend: &#x3C;default> TLS: cfe-tls-key-pair terminates example.com Rules: Host Path Backends ---- ---- -------- super-mario.example.com / mario-service:80 (10.192.4.21:80,10.192.4.22:80) tetris.example.com / tetris-service:80 (10.192.3.231:3000,10.192.4.27:3000) Annotations: cert-manager.io/issuer: cfe-selfsinged-issuer ingress.kubernetes.io/ssl-redirect: true Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 30s nginx-ingress-controller Scheduled for sync Normal CreateCertificate 30s cert-manager-ingress-shim Successfully created Certificate "cfe-tls-key-pair" </code></pre> <h3>Access deployed games</h3> <p>Before accessing the deployed games, you need set up the subdomain name resolution. For the subdomains, <em>super-mario.example.com</em> and <em>tetris.example.com</em>, the workstation host file has been used for DNS resolution.</p> <p>Type the following commands to check that the domain/subdomain name resolution is set up correctly:</p> <pre><code class="language-shell">$ host super-mario.example.com super-mario.example.com has address 10.6.115.251 $ host tetris.example.com tetris.example.com has address 10.6.115.251 </code></pre> <p>You can then access the deployed games using the browser. Start the browser and type the URL <em>super-mario.example.com</em>. It will be redirected over HTTPS with the warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/mario-private.png" alt=""></p> <p>This is due to the fact that the self-signed certifcate is generated in cert-manager and configured in the K8s Ingress resource.</p> <p>Click <em>Not secure</em> and start the Certificate Viewer to check the certificate:</p> <p><img src="/img/mario-certificate.png" alt=""></p> <p>Click <em>Proceed to super-mario.example.com (unsafe)</em>. You will land on the <em>SUPER MARIO</em> game page:</p> <p><img src="/img/super-mario.png" alt=""></p> <p>If you type the URL <em>tetris.example.com</em> to the browser, it will be redirected over HTTPS with the same warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/tetris-private.png" alt=""></p> <p>Click <em>Proceed to tetris.example.com (unsafe)</em>. You will then go to the Tetris <em>Start</em> page:</p> <p><img src="/img/tetris-start.png" alt=""></p> <p>When you click on the <em>Start</em> button, you will land on the <em>Tetris</em> game page:</p> <p><img src="/img/tetris.png" alt=""></p> <p>That's all there is to it! Enjoy playing your games!</p> <h3>Conclusion</h3> <p>This blog post offers you a comprehensive guide on how to deploy <em>Super Mario</em> and <em>Tetris</em> in a K8 cluster and expose those games to be securely accessed via HTTPS in HPE GreenLake for Private Cloud Enterprise. It details the process of configuring TLS termination on an Ingress controller at the load balancer setup, utilizing a K8s Ingress resource and a self-signed TLS certificate generated with cert-manager. This guide fully aligns with the stringent security and compliance requirements of the K8s production environment in HPE GreenLake for Private Cloud Enterprise.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise and get more ideas on how you can use it in your everyday operations.</p>https://developer.hpe.com/okta-sso-integration-for-green-lake-and-aruba-central/https://developer.hpe.com/okta-sso-integration-for-green-lake-and-aruba-central/Tue, 26 Mar 2024 19:04:06 GMT<p>Aruba Central has gone GREEN…GreenLake that is! Aruba Central has recently been integrated into the HPE GreenLake Cloud Platform (GLCP). This provides IT administrators with the ability to view and orchestrate critical network services, such as Wired, Wireless and SD-Branch, through the same dashboard as their compute and storage infrastructure. GLCP also supports Single Sign On (SSO) which helps simplify account management.</p> <p>If you are new to Aruba Central and are looking to enable SSO, this guide is for you. It will walk you through the process of configuring SSO for HPE GreenLake and Aruba Central using Okta.</p> <h3>Before starting</h3> <p>Please review the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">HPE GreenLake</a> User Guide to understand how the SAML framework works in the context of Common Cloud Services for the Aruba Central application.</p> <h3>Configure SSO/SAML applications in Okta</h3> <p>To configure application metadata in Okta, complete the following steps:</p> <ul> <li>Step 1: Create an Okta SAML application</li> <li>Step 2: Configure Sign On settings</li> <li>Step 3: Export the SAML 2.0 IdP metadata</li> <li>Step 4: Configure the SAML connection in HPE GreenLake</li> </ul> <p><strong>Step 1: Create an Okta SAML application</strong></p> <ol> <li>Log in to the Okta administration console.</li> <li>Click <strong>Applications > Create new app integration.</strong> The Create a new app integration window opens.</li> <li>Select SAML 2.0 and click <strong>Next</strong>.</li> </ol> <p><img src="/img/ws-image0.png" alt=""></p> <p>Provide a name for the Aruba GreenLake SSO service (Okta application)</p> <p><img src="/img/ws-image1.png" alt=""></p> <p><strong>Step 2: How to configure Single Sign On settings</strong></p> <ol> <li> <p>Enter the SAML information.</p> <p>Under General:</p> <p><strong>Single Sign on URL:</strong> <a href="https://sso.common.cloud.hpe.com/sp/ACS.saml2">https://sso.common.cloud.hpe.com/sp/ACS.saml2</a></p> <p><strong>Audience URI (SP Entity ID):</strong> <a href="https://sso.common.cloud.hpe.com">https://sso.common.cloud.hpe.com</a></p> <p><strong>Name ID format EmailAddress</strong></p> <p><strong>Application username Email</strong></p> <p><strong>NameID = user.email</strong></p> <p><strong>gl_first_name = user.FirstName</strong></p> <p><strong>gl_last_name = user.LastName</strong></p> <p><strong>hpe_ccs_attribute = (See Below)</strong></p> <p>See here for IdP attribute details: <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us</a></p> <p>As part of the HPE GreenLake cloud platform integration, one of the additional features that was added is the Role Based Access Controls for Aruba Central and all other apps on the platform. A new SAML attribute has been added “hpe_ccs_attribute” which tells HPE GreenLake and Central the exact role/permissions for each user. The following describes how to format the attribute.</p> </li> </ol> <p><img src="/img/ws-image2.png" alt=""></p> <p><img src="/img/ws-image3.png" alt=""></p> <p><img src="/img/ws-image17.png" alt=""></p> <p><img src="/img/ws-image5.png" alt=""></p> <p>The <strong>hpe__ccs__attribute</strong> always starts with <code>version_1#</code>. You must first configure the attributes for HPE GreenLake CSS, and then Central. To do so, enter the PCID for the account, followed by the HPE GreenLake application ID. This will always be <strong>00000000-0000-0000-0000-000000000000</strong>. Following this, enter the role name and <strong>ALL_SCOPES</strong>. Next, enter in the Aruba Central information. Start with the <strong>app cid</strong>, followed by the role name (i.e. Aruba Central Administrator), and then <strong>ALL_SCOPES</strong>.</p> <p>Example:</p> <p><strong>version_1#5b0ec0e8c4f422eca232ba72799953ac:00000000-0000-0000-0000-000000000000:Account Administrator:ALL__SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:</strong></p> <p><strong>Aruba Central Administrator:ALL_SCOPES</strong></p> <p>If you want to add additional HPE GreenLake applications, or if you have multiple Aruba Central accounts, you can add them as well. Just follow the same syntax as before. Once you have the attribute defined, enter it into the SAML attribute statement in Okta as shown below.</p> <p><img src="/img/ws-image6.png" alt=""></p> <p>2. Complete the setup.</p> <p><img src="/img/ws-image7.png" alt=""></p> <p>Click Next and Select “Internal App”, then Finish.</p> <p><strong>Step 3:</strong> <strong>Export the SAML 2.0 IdP metadata</strong></p> <ol> <li> <p>Click Next – Configure the Sign On settings</p> <p>You will find two options are available: <strong>View Setup Instructions</strong> which steps you through the SAML configuration and <strong>Identity Provider metadata</strong>, which will produce an XML file that can be loaded into Aruba Central.</p> <p>Suggestion: Click <strong>Identity Provider metadata</strong> and save the XML data to a file.</p> <p><img src="/img/ws-image9.png" alt=""></p> </li> <li> <p>Click Next.</p> </li> <li> <p>Select Internal app, and Click Finish.</p> </li> </ol> <p><strong>Step 4: Create SAML Authorization Profile in HPE GreenLake Cloud Platform</strong></p> <ol> <li> <p>Log into HPE GreenLake and click Menu > Manage > Authentication and Click Set Up SAML Connection.</p> <p><em>Before you can add a new SAML configuration, you must have at least one user account with that domain already enabled in HPE GreenLake. Also, you must be logged into HPE GreenLake with an account from that domain in order to enable SSO for it.</em></p> <p><img src="/img/ws-image10.png" alt=""></p> </li> <li> <p>Type in the domain you want to enable SSO on:</p> <p><img src="/img/ws-image11.png" alt=""></p> </li> <li> <p>Input the metadata from the step above.</p> <p>While HPE GreenLake does support entering this information manually, it's recommended that you simply upload the XML metadata that was downloaded in the previous step. To do so, Select Metadata File, selecting the XML file. Then, click Next.</p> <p><img src="/img/ws-image12.png" alt=""></p> </li> <li> <p>Enter the SAML attributes to match what was entered in Okta. Set the idle timeout value as well.</p> <p><img src="/img/ws-image13.png" alt=""></p> </li> <li> <p>Then click Next.</p> </li> <li> <p>Create a recover user so that, in the event SSO fails, an admin will still be able to access the HPE GreenLake portal.</p> <p><img src="/img/ws-image14.png" alt=""></p> <p>Congratulations! SSO will now be enabled for HPE GreenLake as well as the Aruba Central application. Log out and on the HPE GreenLake home page, click <strong>Sign in with SSO</strong>.</p> </li> </ol> <p><strong>Testing and troubleshooting:</strong></p> <p>On the HPE GreenLake Cloud Platform home page, click <strong>Sign In with SSO</strong>.</p> <p><img src="/img/ws-image15.png" alt=""></p> <p><img src="/img/ws-image16.png" alt=""></p> <p>Enter the SSO credentials. You will be redirected to Okta to authenticate. Once you successfully authenticate, you will be redirected back to HPE GreenLake. You can then click on the Aruba Central application and be given access based on the configured role/permissions.</p> <p><strong>Additional notes:</strong></p> <ul> <li>There must be at least <strong>one</strong> verified user belonging to the <strong>Domain</strong> prior to configuration.</li> <li>In order to configure SSO, you must be logged into HPE GreenLake with a user from the domain.</li> <li>SSO user access is determined by the “role_name” attribute included in the SAML "hpe_ccs_attribute" provided by the IdP.</li> <li>SSO users can initiate a Single Sign On request by trying to log into Aruba Central (SP-initiated login).</li> <li>For more troubleshooting: <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us</a></li> </ul>https://developer.hpe.com/ai-news-16/https://developer.hpe.com/ai-news-16/Tue, 26 Mar 2024 17:20:55 GMT<p>External post</p>https://developer.hpe.com/chapel-2-0-scalable-and-productive-computing-for-all/https://developer.hpe.com/chapel-2-0-scalable-and-productive-computing-for-all/Thu, 21 Mar 2024 22:42:35 GMT<p>External blog</p>https://developer.hpe.com/hpe-athonet-llm-platform-driving-users-towards-peak-flow-efficiency/https://developer.hpe.com/hpe-athonet-llm-platform-driving-users-towards-peak-flow-efficiency/Wed, 20 Mar 2024 17:36:19 GMT<p>In the preceding blog posts,<a href="https://developer.hpe.com/blog/the-transformative-impact-of-generative-ai-on-telco-products/">"The transformative impact of generative AI on Telco products"</a>and <a href="https://developer.hpe.com/blog/hpe-athonet-llm-platform-first-pillar-from-personal-assistant-to-collaborative-corporate-tool/">"HPE Athonet LLM Platform: From personal assistant to collaborative corporate tool"</a>, I detailed how HPE Athonet is constructing a solution anchored in three fundamental aspects:</p> <ul> <li>Shifting towards a framework that centers on team collaboration.</li> <li>Elevating the user experience to foster productivity and concentration on tasks.</li> <li>Crafting a flexible infrastructure grounded in data mesh methodologies, encompassing the management of security and ethical standards.</li> </ul> <p>These components collectively illustrate HPE Athonet's dedication not solely to the advancement of technology but also to the promotion of true digital evolution within network management through the application of generative AI.</p> <p>This article further explores the second core principle shaping the strategy: <strong>the enhancement of usability and user experience</strong> to assist individuals in maintaining focus on their tasks.</p> <p>While current chatbot solutions offer limited functionalities, HPE Athonet's goal is to introduce a tool that significantly improves task concentration and multitasking capabilities. By creating a system that can be augmented with additional tools or plugins, the intention is to support users in sustaining focus and potentially reaching a state of <strong>"Flow"</strong>, where they are at peak productivity.</p> <p>This strategy reflects design concepts present in some eLearning platforms, focusing on user involvement and efficiency, as outlined in the provided <a href="https://www.ux-republic.com/en/flow-a-key-concept-for-ux-designers/">“Flow, a key concept for UX designers”</a>.</p> <p><img src="/img/athon_ux-copy.png" alt=""></p> <p>Major companies in the tech industry are incorporating generative AI assistants across their office tool suites. HPE Athonet’s approach significantly diverges; rather than amplifying a variety of general-purpose products for individual usage, the aim is to forge a specialized instrument geared towards <strong>augmenting team collaboration</strong> specifically within 5G project realms. This instrument consolidates all vital data and resources, facilitating each team member’s effective engagement and seamless contribution to unified project endeavors.</p> <p>HPE Athonet introduces a novel feature allowing the selection of a particular project, which unlocks a comprehensive archive of all related actions and a set of project-specific tools. For instance, when evolving a new attribute for the HPE Athonet 5G core, the project archive might encompass customer feature inquiries, viability studies, and strategic evaluations by product managers and architects, in addition to relevant 3rd Generation Partnership Project (3GPP) standards details. This amalgamation ensures seamless access to all essential data for the R&#x26;D team via a straightforward chat interaction with the tool as they commence development.</p> <p>The platform accommodates <strong>individual and collective discussions</strong>, maintaining the project archive up-to-date with fresh insights. To further refine the user experience, it is vital not only to address user queries but also to exhibit the source content, potentially showcasing the pertinent document segment with highlighted key areas. Furnishing an answer's relevance score could further enhance user engagement.</p> <p>For tools interfacing with devices or 5G networks to present real-time insights, the integration of sophisticated visualization attributes like Key Performance Indicator (KPI) charts, geographical mappings, and system layouts is vital. These improvements should transcend mere textual exchanges, providing a user interface that skillfully renders data through the most effective multimedia elements. This strategy guarantees <strong>that information is not just reachable but also intuitively understandable</strong>, empowering users to immediately digest complex data and make prompt, well-informed choices.</p> <p>Technical instruments often demand specific input formats; thus, HPE Athonet is also introducing the ability to customize the user interface integration within the chatbot. All these initiatives aim in a unified direction: to assist the user in efficiently accessing or performing required tasks through a straightforward interface that can link to multiple tools.</p> <p><img src="/img/athon_ux_tools.png" alt=""></p> <p>This blog post highlights the second foundational pillar shaping HPE Athonet's forthcoming product: enhancing user experience to foster productivity and task concentration. It demonstrates how HPE Athonet intends to implement its strategy, which also encompasses transitioning to a framework centered around team collaboration and developing a secure, ethical, and adaptable infrastructure. These initiatives highlight HPE Athonet's dedication to innovation and digital transformation within network management.</p> <p>Stay tuned for an upcoming post on the final pillar concerning the development of an LLM Data Platform.</p>https://developer.hpe.com/exposing-an-application-using-ingress-and-tls-termination-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/exposing-an-application-using-ingress-and-tls-termination-on-kubernetes-in-hpe-greenlake-for-private-cloud-enterprise/Wed, 20 Mar 2024 10:06:33 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>Once applications are deployed in a K8s cluster, the next step is to create services that expose these applications. By default, K8s services are created with the <em>ClusterIP</em> type, which supports internal connectivity among different components of the applications. However, these services are not directly accessible from outside the cluster. It can be challenging to securely expose the deployed applications over HTTPS. This involves generating and managing SSL/TLS certificates for multiple applications deployed in the cluster. These certificates are crucial for secure communication between services. Installing them and managing them correctly is essential to avoid access issues and security risks.</p> <p>To address exposing applications over HTTPS, K8s provides the concept of <em>Ingress</em>. An Ingress acts as an entry point for external traffic into the cluster. It can be configured with TLS termination. However, setting up K8s Ingress with TLS termination is intricate. It involves creating a K8s <em>Secret</em> to host the certificate and referencing the secret in the Ingress resource. It also requires an additional load balancer configuration in the cluster.</p> <p>This blog post outlines the comprehensive steps used to expose applications using Ingress and TLS termination on K8s in HPE GreenLake for Private Cloud Enterprise. <a href="https://developer.hpe.com/blog/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/">MetalLB</a> is deployed to the cluster to set up the load balancer. It enables external access to services within the cluster. <a href="https://developer.hpe.com/blog/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/">Cert-manager</a> is used for creating and managing SSL/TLS certificates. The generated certificate is stored as a K8s <em>Secret</em> object. This secret can be mounted by application Pods or used by an Ingress controller. The <a href="https://www.nginx.com/products/nginx-ingress-controller/">Nginx Ingress controller</a> is deployed and configured in the cluster. It handles SSL certificates and facilitates secure access to applications in the backend.</p> <p><img src="/img/tls-termination-s.png" alt=""></p> <p>Despite the complexities, securely exposing applications in a K8s cluster over HTTPS is attainable. It can be achieved by leveraging Ingress and TLS termination, along with a suite of suitable tools and utilities deployed within the K8s cluster in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em> CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The <em>helm</em> CLI tool, version 3.12.0 or later</li> <li>A domain and a list of subdomains to generate the SSL certificate and host the applications in the cluster</li> </ul> <h3>Set up the load balancer with MetalLB</h3> <p>You can install MetalLB and set up the load balancer in the K8s cluster by following the instructions found in the blog post <a href="https://developer.hpe.com/blog/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/">Setting up the load balancer with MetalLB</a>.</p> <p>Here is the deployed MetalLB to the namespace <em>metallb-system</em> in the cluster:</p> <pre><code class="language-shell">$ kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/controller-57b4fdc957-56wv8 1/1 Running 0 22m pod/speaker-c7sgk 1/1 Running 0 22m pod/speaker-dtlpm 1/1 Running 0 22m pod/speaker-gxccz 1/1 Running 0 22m pod/speaker-pwl87 1/1 Running 0 22m pod/speaker-rvvkz 1/1 Running 0 22m pod/speaker-wxd5n 1/1 Running 0 22m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/webhook-service ClusterIP 10.102.54.20 &#x3C;none> 443/TCP 22m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/speaker 6 6 6 6 6 kubernetes.io/os=linux 22m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/controller 1/1 1 1 22m NAME DESIRED CURRENT READY AGE replicaset.apps/controller-57b4fdc957 1 1 1 22m </code></pre> <p>You can see the range of virtual IP addresses, <em>"10.6.115.251-10.6.115.254"</em>, defined in the CRD resource <em>IPAddressPool</em>, and the layer 2 service IP address announcement in the CRD resource <em>L2Advertisement</em>:</p> <pre><code class="language-shell">$ kubectl get ipaddresspools -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES cfe-pool true false ["10.6.115.251-10.6.115.254"] $ kubectl get l2advertisements -n metallb-system NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES cfe-l2advert ["cfe-pool"] </code></pre> <h3>Deploy Nginx Ingress controller</h3> <p>In order for an Ingress to work in the cluster, there must be an Ingress controller being deployed and running. It's the Ingress controller that accesses the certificate and the routing rules defined on the Ingress resource and makes them part of its configuration.</p> <p>A variety of Ingress controllers are available for deployment in the cluster, including <a href="https://doc.traefik.io/traefik/providers/kubernetes-ingress/">Traefik</a>, <a href="https://github.com/haproxytech/kubernetes-ingress#readme">HAProxy</a> and <a href="https://www.nginx.com/products/nginx-ingress-controller/">Nginx Ingress controller</a>. Execute the command below to install the Nginx Ingress controller to the cluster using <em>Helm</em>:</p> <pre><code class="language-shell">$ helm upgrade --install ingress-nginx ingress-nginx \ > --repo https://kubernetes.github.io/ingress-nginx \ > --namespace ingress-nginx --create-namespace Release "ingress-nginx" does not exist. Installing it now. NAME: ingress-nginx LAST DEPLOYED: Wed Mar 6 18:30:55 2024 NAMESPACE: ingress-nginx STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the load balancer IP to be available. You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - pathType: Prefix backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: &#x3C;base64 encoded cert> tls.key: &#x3C;base64 encoded key> type: kubernetes.io/tls </code></pre> <p>The Nginx Ingress controller is deployed to the namespace <em>ingress-nginx</em> in the cluster. Type the following command to check the deployment details:</p> <pre><code class="language-shell">$ kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-controller-548768956f-8bz2q 1/1 Running 0 15m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.108.173.7 10.6.115.251 80:32734/TCP,443:32265/TCP 15m service/ingress-nginx-controller-admission ClusterIP 10.108.100.150 &#x3C;none> 443/TCP 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 15m NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-548768956f 1 1 1 15m </code></pre> <p>The service <em>ingress-nginx-controller</em> gets deployed as the service type of <em>LoadBalancer</em> with the <em>EXTERNAL-IP</em> assigned as <em>10.6.115.251</em>. This IP address will be used for setting up domain and subdomain name resolution.</p> <h3>Generate a self-signed certificate using cert-manager</h3> <p>You can deploy cert-manager to the K8s cluster and generate a self-signed certificate by following the instructions found in the blog post <a href="https://developer.hpe.com/blog/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/">Generating self-signed certificates using cert-manager</a>.</p> <p>Here is the cert-manager deployed to the namespace <em>cert-manager</em> in the cluster:</p> <pre><code class="language-shell">$ kubectl get all -n cert-manager NAME READY STATUS RESTARTS AGE pod/cert-manager-59fbb6655d-h7sqb 1/1 Running 0 18s pod/cert-manager-cainjector-69548575fb-7fqd2 1/1 Running 0 18s pod/cert-manager-webhook-57b78f476d-mp45s 1/1 Running 0 16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.107.221.97 &#x3C;none> 9402/TCP 20s service/cert-manager-webhook ClusterIP 10.104.243.185 &#x3C;none> 443/TCP 19s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 18s deployment.apps/cert-manager-cainjector 1/1 1 1 18s deployment.apps/cert-manager-webhook 1/1 1 1 17s NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-59fbb6655d 1 1 1 19s replicaset.apps/cert-manager-cainjector-69548575fb 1 1 1 19s replicaset.apps/cert-manager-webhook-57b78f476d 1 1 1 18s </code></pre> <p>Below is the deployed self-signed custom resource definition (CRD) <em>Issuer</em> in the namespace <em>nginx-apps</em> in which you want to generate the certificate:</p> <pre><code class="language-shell">$ kubectl get issuer -n nginx-apps NAME READY AGE cfe-selfsigned-issuer True 115s </code></pre> <p>Here is the generated self-signed certificate in the namespace <em>nginx-apps</em>:</p> <pre><code class="language-shell">$ kubectl get certificate -n nginx-apps NAME READY SECRET AGE cfe-selfsigned-tls True cfe-tls-key-pair 2m23s </code></pre> <p>The K8s <em>Secret</em> <em>'cfe-tls-key-pair'</em> is created automatically in the same namespace as part of certificate deployment:</p> <pre><code class="language-shell">$ kubectl get secrets -n nginx-apps cfe-tls-key-pair NAME TYPE DATA AGE cfe-tls-key-pair kubernetes.io/tls 3 2m25s </code></pre> <p>Type the following command to check the <em>commonName</em> and the <em>dnsNames</em> in the generated certificate:</p> <pre><code class="language-shell">$ kubectl describe certificate cfe-selfsigned-tls -n nginx-apps Name: cfe-selfsigned-tls Namespace: nginx-apps Labels: &#x3C;none> Annotations: &#x3C;none> API Version: cert-manager.io/v1 Kind: Certificate Metadata: Creation Timestamp: 2024-03-06T17:58:51Z Generation: 1 Managed Fields: API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:commonName: f:dnsNames: f:isCA: f:issuerRef: .: f:kind: f:name: f:secretName: Manager: kubectl-client-side-apply Operation: Update Time: 2024-03-06T17:58:51Z API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:revision: Manager: cert-manager-certificates-issuing Operation: Update Subresource: status Time: 2024-03-06T17:58:52Z API Version: cert-manager.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:conditions: .: k:{"type":"Ready"}: .: f:lastTransitionTime: f:message: f:observedGeneration: f:reason: f:status: f:type: f:notAfter: f:notBefore: f:renewalTime: Manager: cert-manager-certificates-readiness Operation: Update Subresource: status Time: 2024-03-06T17:58:52Z Resource Version: 2128063 UID: 977eaa8a-1612-489b-a34d-0e78ab113096 Spec: Common Name: example.com Dns Names: green.nginx.example.com blue.nginx.example.com nginx.example.com example.com Is CA: true Issuer Ref: Kind: Issuer Name: cfe-selfsigned-issuer Secret Name: cfe-tls-key-pair Status: Conditions: Last Transition Time: 2024-03-06T17:58:52Z Message: Certificate is up to date and has not expired Observed Generation: 1 Reason: Ready Status: True Type: Ready Not After: 2024-06-04T17:58:52Z Not Before: 2024-03-06T17:58:52Z Renewal Time: 2024-05-05T17:58:52Z Revision: 1 Events: &#x3C;none> </code></pre> <h3>Deploy sample Nginx applications</h3> <p>In order to configure and validate the Ingress TLS termination, three sample Nginx applications will be deployed to the cluster using the YAML manifest files from the GitHub repo <a href="https://github.com/GuopingJia/ingress-demo">ingress-demo</a>:</p> <pre><code class="language-shell">$ tree ingress-demo/ ingress-demo/ ├── apps │   ├── nginx-blue.yaml │   ├── nginx-green.yaml │   └── nginx-main.yaml ├── ingress-host-based-selfsigned.yaml ├── ingress-path-based-selfsigned.yaml └── README.md </code></pre> <p>Each YAML manifest file in the folder <em>'apps'</em> defines the <em>Deployment</em> and the <em>Service</em> resource.</p> <p>Type the following commands to deploy those Nginx applications to the namespace <em>nginx-apps</em>:</p> <pre><code class="language-shell">$ cd ingress-demo/ $ kubectl apply -f apps/nginx-main.yaml -n nginx-apps service/nginx-main created deployment.apps/nginx-main created $ kubectl apply -f apps/nginx-green.yaml -n nginx-apps service/nginx-green created deployment.apps/nginx-green created $ kubectl apply -f apps/nginx-blue.yaml -n nginx-apps service/nginx-blue created deployment.apps/nginx-blue created </code></pre> <p>Type the command shown below to check the details of each application deployment:</p> <pre><code class="language-shell">$ kubectl get all -n nginx-apps NAME READY STATUS RESTARTS AGE pod/nginx-blue-78647f4c4b-z8wq9 1/1 Running 0 10s pod/nginx-green-8956bbd9f-zz7hk 1/1 Running 0 22s pod/nginx-main-64bfd77895-tf7xd 1/1 Running 0 31s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-blue ClusterIP 10.108.51.116 &#x3C;none> 80/TCP 15s service/nginx-green ClusterIP 10.106.115.65 &#x3C;none> 80/TCP 23s service/nginx-main ClusterIP 10.108.33.44 &#x3C;none> 80/TCP 32s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-blue 1/1 1 1 15s deployment.apps/nginx-green 1/1 1 1 24s deployment.apps/nginx-main 1/1 1 1 32s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-blue-78647f4c4b 1 1 1 15s replicaset.apps/nginx-green-8956bbd9f 1 1 1 24s replicaset.apps/nginx-main-64bfd77895 1 1 1 32s </code></pre> <p>Three Nginx services, <em>nginx-main</em>, <em>nginx-blue</em> and <em>nginx-green</em>, are deployed as the <em>ClusterIP</em> type. They provide internal connectivity and can solely be accessed from within the cluster.</p> <p>Type the following command to check that all the application service endpoints have been populated:</p> <pre><code class="language-shell">$ kubectl get endpoints -n nginx-apps NAME ENDPOINTS AGE nginx-blue 10.192.3.78:80 1m nginx-green 10.192.4.45:80 1m nginx-main 10.192.4.44:80 1m </code></pre> <h3>Set up Ingress TLS</h3> <p>The Ingress resource with TLS has to be created. Here is the sample Ingress TLS resource <em>ingress-host-based-selfsigned.yaml</em>, available from the GitHub repo <a href="https://github.com/GuopingJia/ingress-demo">ingress-demo</a>:</p> <pre><code class="language-shell">$ cat ingress-host-based-selfsigned.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-host-based-selfsigned annotations: ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/issuer: "nginx-selfsigned-issuer" spec: ingressClassName: nginx tls: - hosts: - nginx.example.com secretName: cfe-tls-key-pair rules: - host: nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-main port: number: 80 - host: blue.nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-blue port: number: 80 - host: green.nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-green port: number: 80 </code></pre> <p>In the above sample YAML manifest file, there is the <em>'tls'</em> block that contains the hostname <em>'nginx.example.com'</em> and the secret <em>cfe-tls-key-pair</em> created in the certification step. There is also the <em>'rules'</em> block in which a list of routing rules is defined per host, e.g., host <em>nginx.example.com</em> will be routed to the application service <em>nginx-main</em> in the backend.</p> <p>Type the following command to deploy the Ingress resource to the namespace <em>nginx-apps</em>:</p> <pre><code class="language-shell">$ kubectl apply -f ingress-host-based-selfsigned.yaml -n nginx-apps ingress.networking.k8s.io/ingress-host-based-selfsigned created </code></pre> <p>Check the details of the <em>TLS</em> and <em>Rules</em> settings by typing the command shown below:</p> <pre><code class="language-shell">$ kubectl describe ingress ingress-host-based-selfsigned -n nginx-apps Name: ingress-host-based-selfsigned Labels: &#x3C;none> Namespace: nginx-apps Address: Ingress Class: nginx Default backend: &#x3C;default> TLS: cfe-tls-key-pair terminates nginx.example.com Rules: Host Path Backends ---- ---- -------- nginx.example.com / nginx-main:80 (10.192.4.44:80) blue.nginx.example.com / nginx-blue:80 (10.192.3.78:80) green.nginx.example.com / nginx-green:80 (10.192.4.45:80) Annotations: cert-manager.io/issuer: nginx-selfsinged-issuer ingress.kubernetes.io/ssl-redirect: true Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 20s nginx-ingress-controller Scheduled for sync Normal CreateCertificate 20s cert-manager-ingress-shim Successfully created Certificate "cfe-tls-key-pair" </code></pre> <h3>Access deployed Nginx applications</h3> <p>Before accessing the deployed Nginx applications, you need set up the domain and the subdomain name resolution. For the sample domain name <em>nginx.example.com</em>, and its subdomains, <em>blue.nginx.example.com</em> and <em>green.nginx.example.com</em>, the workstation host file has been used for DNS resolution.</p> <p>Type the following commands to check that this is done correctly:</p> <pre><code class="language-shell">$ host nginx.example.com nginx.example.com has address 10.6.115.251 $ host green.nginx.example.com green.nginx.example.com has address 10.6.115.251 $ host blue.nginx.example.com blue.nginx.example.com has address 10.6.115.251 </code></pre> <p>You can then validate the Ingres TLS configuration of the deployed Nginx applications using the browser.</p> <p>Start the browser and type the URL <em>nginx.example.com</em>. It will be redirected over HTTPS with the warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/nginx-main-warning.png" alt=""></p> <p>This is due to the fact that the self-signed certifcate is generated in cert-manager and configured in the K8s Ingress resource.</p> <p>Click <em>Not secure</em> and start the Certificate Viewer to check the certificate:</p> <p><img src="/img/nginx-main-cert.png" alt=""></p> <p>Click <em>Proceed to nginx.example.com (unsafe)</em>. You will then go to the Nginx <em>MAIN</em> page:</p> <p><img src="/img/nginx-main.png" alt=""></p> <p>Type the URL <em>green.nginx.example.com</em> to the browser. It will be redirected over HTTPS with the same warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/nginx-green-warning.png" alt=""></p> <p>Click <em>Proceed to green.nginx.example.com (unsafe)</em>. You will then go to the Nginx <em>GREEN</em> page:</p> <p><img src="/img/nginx-green.png" alt=""></p> <p>The same thing occurs when you type the URL <em>blue.nginx.example.com</em> to the browser. The access will be redirected over HTTPS with the same warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/nginx-blue-warning.png" alt=""></p> <p>Click <em>Proceed to blue.nginx.example.com (unsafe)</em>. You will then go to the Nginx <em>BLUE</em> page:</p> <p><img src="/img/nginx-blue.png" alt=""></p> <p>You have successfully configured the Ingress with the generated TLS certificate and exposed the deployed applications with TLS termination.</p> <h3>Conclusion</h3> <p>This blog post provided a comprehensive guide on how to expose applications and make them accessible securely via HTTPS in a K8 cluster in HPE GreenLake for Private Cloud Enterprise. It detailed the process of configuring TLS termination on an Ingress controller, utilizing a K8s Ingress resource and a self-signed TLS certificate generated with cert-manager. While the emphasis of this post was on self-signed certificates, the outlined procedure is equally applicable to any type of certificates. This flexibility allows customers to follow the steps using their own CA certificates or any commercially issued certificates for Ingress TLS termination, ensuring secure exposure of their applications in the K8s cluster over HTTPS.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/announcing-genai-studio-your-generative-ai-playground-built-on-determined/https://developer.hpe.com/announcing-genai-studio-your-generative-ai-playground-built-on-determined/Wed, 13 Mar 2024 22:07:13 GMT<p>External Blog</p>https://developer.hpe.com/hpe-athonet-llm-platform-first-pillar-from-personal-assistant-to-collaborative-corporate-tool/https://developer.hpe.com/hpe-athonet-llm-platform-first-pillar-from-personal-assistant-to-collaborative-corporate-tool/Wed, 13 Mar 2024 08:17:11 GMT<p>As introduced in the previous blog post "<a href="https://developer.hpe.com/blog/the-transformative-impact-of-generative-ai-on-telco-products/">The transformative impact of generative AI on Telco products</a>, HPE Athonet is building a product based on three pillars:</p> <ol> <li>Transitioning to a team-centric collaborative framework.</li> <li>Enhancing user experience to promote productivity and task focus.</li> <li>Developing a versatile architecture using data mesh principles that also include governance of security and ethical aspects.</li> </ol> <p>Together, these elements represent our commitment not just to technological innovation, but also to fostering a genuine digital transformation in network management using generative AI. This installment delves deeper into the first foundational principles underpinning our strategy: specifically, the shift from utilizing isolated tools to embracing a holistic, team-centric collaborative framework.</p> <p>Sam Altman, CEO of OpenAI, is on a mission to develop an exceptional personal assistant aimed at enhancing individual learning and support. While this initiative is great, HPE Athonet has found that transitioning from a business-to-consumer (B2C) to a business-to-business (B2B) model necessitates a significant shift in focus from serving individuals to empowering teams. So, our team introduced the following <strong>innovation of meaning</strong>:</p> <p><img src="/img/athon_col_tool.png" alt=""></p> <p>The rationale behind this shift is the conviction that teamwork is fundamental to organizational success. The concern is that if this tool is used solely on an individual basis, it might inadvertently amplify personal biases, complicating communication within teams. By <strong>emphasizing teamwork</strong>, one signal's a broader perspective on how this technology should be utilized.</p> <p>To clarify, there are several global initiatives to address and reduce biases in LLMs, particularly those related to data, like the demographic’s ones. These biases can often be minimized or managed using LLMs themselves, employing strategies such as guardrails or reinforcement learning from human feedback methods. Adhering to the <strong>HPE AI Ethics principles</strong> commits to mitigating these biases.</p> <p>It is recognized, however, that there is a distinct category: the cognitive biases, which, unlike data biases, receive less focus in academic research and have fewer mitigation strategies available. Cognitive biases, such as confirmation bias, have profound effects on human interaction, potentially leading to polarization also within small social units like teams or departments. In response to this challenge, HPE Athonet is dedicating efforts towards understanding and addressing cognitive biases. The team is fostering environments that <strong>encourage diverse perspectives by creating 'shared memory' spaces</strong> within projects. In these spaces, employees can contribute with their individual chat memories, facilitating the collection of varied viewpoints and promoting openness and dialogue.</p> <p>In B2B environments, the transfer of employee knowledge into digital tools is crucial, underlining the significance of the design and presentation of these tools to the workforce. Historically, a common approach towards AI in business has been to automate processes and reduce costs, often at the expense of human resources. HPE Athonet's priority diverges from this path; we prioritize enhancing the intrinsic value of our products, focusing not just on reducing expenses but on adding qualitative benefits.</p> <p>Despite the advanced nature and associated costs of these technologies, they are viewed as vital investments in promoting collaboration and creativity within teams. The objective is to <strong>empower individuals</strong> to work more efficiently and to <strong>foster innovation through improved teamwork</strong>. By leveraging AI's unique capabilities, HPE Athonet aims to augment and elevate human efforts, rather than replace them.</p> <p>While it is acknowledged that eliminating biases from LLMs is unfeasible, we believe that evolving towards a more collaborative tool can foster an <strong>'anti-fragile' system</strong> better equipped to manage and mitigate these biases.</p> <p>This blog highlights the initial foundational pillar shaping HPE Athonet's upcoming product: the shift towards a team-centric collaborative framework to leverage generative AI potential. It marks the initial step in a strategy that also aims to enhance user experience and develop a secure, ethical, and versatile architecture. These initiatives underscore HPE Athonet's commitment to innovation and the digital transformation of network management. Stay tuned for upcoming posts on the other pillars.</p>https://developer.hpe.com/ai-news-14/https://developer.hpe.com/ai-news-14/Mon, 11 Mar 2024 17:04:26 GMT<p>External blog post</p>https://developer.hpe.com/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/https://developer.hpe.com/generating-self-signed-certificates-using-cert-manager-for-kubernetes-in-hpe-greenlake-for-private-cloud-entreprise/Mon, 11 Mar 2024 16:19:11 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>This blog post describes the details steps on how to generate a self-signed certificate using cert-manager for Kubernetes (K8s) in HPE GreenLake for Private Cloud Enterprise. The generated self-signed certificates can be used by DevOps teams and developers to configure Transport Layer Security (TLS) termination and expose applications deployed in the K8s cluster securely via HTTPS.</p> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a K8s cluster, view details about existing clusters, and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>Once applications are deployed in a cluster, a common requirement is to expose the applications so that they can be securely accessed over HTTPS. This requires getting a valid SSL/TLS certificate in K8s. Generating and managing SSL/TLS certificates in K8s is not always easy. There is a list of popular tools and utilities, e.g, . <a href="https://www.openssl.org/">OpenSSL</a>, <a href="https://github.com/cloudflare/cfssl">CloudFlare’s CFSSL</a>, <a href="https://github.com/OpenVPN/easy-rsa">OpenVPN’s Easy-RSA</a>, etc, that you can use for generating certificates.</p> <p>However, you still need to follow up that with creating the root certificate authorities, generating certificate signing requests (CSRs), and signing the certificates. The process to generate those items is not very intuitive. Most often than not, it requires the help of a <em>DevOps</em> engineer as well as assistance from different teams who are involved in installing and configuring the certificate chain.</p> <p>This blog post describes the detailed steps involved in the process of generating a <strong>self-signed</strong> certificate using cert-manager for K8s in HPE GreenLake for Private Cloud Enterprise. Cert-manager integrates seamlessly with K8s for automated handling of certificates. It aligns well with the K8s resource model. This makes cert-manager a native and powerful solution for creating and managing certificates within K8s clusters.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em> CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The optional <em>openssl</em> CLI tool, for validating the generated certificate</li> </ul> <h3>Cert-manager</h3> <p><a href="https://cert-manager.io/">Cert-manager</a>, a popular open source certificate management add-on designed to work with K8s, streamlines the process of acquiring, renewing, and utilizing SSL/TLS certificates within a K8s cluster. When deployed in a K8s cluster, cert-manager introduces two custom resource definitions (CRDs): <em>Issuer</em> and <em>Certificate</em>. These CRDs automate the generation and renewal of certificates for various scenarios in K8s. Cert-manager can obtain certificates from a variety of certificate authorities (CAs), including <em>Let’s Encrypt</em>, <em>HashiCorp Vault</em>, and <em>private PKIs</em>. It can also be configured to generate self-signed certificates if needed. When cert-manager creates a certificate, it makes it available to the entire cluster by storing the certificate as a K8s <em>Secret</em> object, which can be mounted by application Pods or used by an Ingress controller. This makes the certificate accessible across all namespaces within the K8s cluster.</p> <h3>Generate a self-signed certificate</h3> <h4>Install cert-manager</h4> <p>As shown on the <a href="https://cert-manager.io/docs/installation/">cert-manager installation page</a>, cert-manager can be installed by typing the following <em>kubectl apply</em> command:</p> <pre><code class="language-shell">$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.yaml namespace/cert-manager created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created configmap/cert-manager created configmap/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created </code></pre> <p>The latest cert-manager <em>v1.14.3</em> will be installed to the namespace <em>cert-manager</em>. Type the following command to check that all the Pods are showing a Running status:</p> <pre><code class="language-shell">$ kubectl get all -n cert-manager NAME READY STATUS RESTARTS AGE pod/cert-manager-6bcdd5f7c-f7lfw 1/1 Running 0 3m36s pod/cert-manager-cainjector-5d4577b4d9-jmpsp 1/1 Running 0 3m36s pod/cert-manager-webhook-bf957dc77-s9r2g 1/1 Running 0 3m36s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.109.28.203 &#x3C;none> 9402/TCP 3m39s service/cert-manager-webhook ClusterIP 10.100.82.119 &#x3C;none> 443/TCP 3m38s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 3m37s deployment.apps/cert-manager-cainjector 1/1 1 1 3m38s deployment.apps/cert-manager-webhook 1/1 1 1 3m37s NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-6bcdd5f7c 1 1 1 3m38s replicaset.apps/cert-manager-cainjector-5d4577b4d9 1 1 1 3m39s replicaset.apps/cert-manager-webhook-bf957dc77 1 1 1 3m38s </code></pre> <p>As part of cert-manager installation, a list of cert-manager related CRDs has been added to the cluster:</p> <pre><code class="language-shell">$ kubectl get crds | grep cert-manager certificaterequests.cert-manager.io 2024-02-02T15:42:53Z certificates.cert-manager.io 2024-02-02T15:42:53Z challenges.acme.cert-manager.io 2024-02-02T15:42:54Z clusterissuers.cert-manager.io 2024-02-02T15:42:55Z issuers.cert-manager.io 2024-02-02T15:42:55Z orders.acme.cert-manager.io 2024-02-02T15:42:56Z </code></pre> <h4>Create an Issuer</h4> <p>An <em>Issuer</em> in cert-manager is a K8s CRD resource that represents a certificate authority (CA) that's able to generate a signed certificate by honoring certificate signing request (CSR). All cert-manager certificates require a referenced issuer that is in a ready condition to attempt to honor the request.</p> <p>Here is a self-signed issuer YAML manifest file <em>issuer-selfsigned.yaml</em>:</p> <pre><code class="language-shell">$ cat issuer-selfsigned.yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: cfe-selfsigned-issuer spec: selfSigned: {} </code></pre> <p>Type the following commands to create a namespace in which you want to generate certificates and deploy the CRD Issuer resource to this namespace. Replace the sample namespace <em>cfe-apps</em> in the commands with your own namespace.</p> <pre><code class="language-shell">$ kubectl create ns cfe-apps namespace/cfe-apps created $ kubectl apply -f issuer-selfsigned.yaml -n cfe-apps issuer.cert-manager.io/cfe-selfsigned-issuer created </code></pre> <p>Type the following command to check the deployed issuer in the namespace. The issuer should show <em>READY</em> as <em><strong>True</strong></em>.</p> <pre><code class="language-shell">$ kubectl get issuer -n cfe-apps NAME READY AGE cfe-selfsigned-issuer True 7s </code></pre> <p>If you want to be able to request certificates from any namespace in a cluster, use the CRD resource called <em>ClusterIssuer</em>.</p> <p>Here is a sample <em>ClusterIssuer</em> YAM manifest file <em>clusterissuer.yaml</em>:</p> <pre><code class="language-shell">$ cat clusterissuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-cluster-issuer spec: selfSigned: {} </code></pre> <h4>Generate a certificate</h4> <p>You can use the CRD resource <em>Certificate</em> to generate a self-signed certificate.</p> <p>Here is a sample <em>Certificate</em> YAML manifest file <em>certificate.yaml</em> :</p> <pre><code class="language-shell">$ cat certificate.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: cfe-selfsigned-tls spec: secretName: cfe-tls-key-pair isCA: true issuerRef: name: cfe-selfsigned-issuer kind: Issuer commonName: "example.com" dnsNames: - nginx.example.com - example.com </code></pre> <p>In this YAML file, the <em>commonName</em> is set to a sample domain <em>'example.com'</em>. The <em>dnsNames</em> includes <em>'example.com'</em> and its subdomain <em>'nginx.example.com'</em>.</p> <p>Cert-manager supports the generation of wildcard certificates, e.g., using '*.example.com', which allows one to secure multiple subdomains under a single certificate. Wildcard certificates cover all subdomains under the specified domain. You need to be cautious when using them, as they grant access to any subdomain matching the pattern.</p> <p>Type the following command to generate the certificate in the namespace <em>cfe-apps</em>:</p> <pre><code class="language-shell">$ kubectl apply -f certificate.yaml -n cfe-apps certificate.cert-manager.io/cfe-selfsigned-tls created </code></pre> <p>Check the generated certificate in the namespace <em>cfe-apps</em> by typing the following command:</p> <pre><code class="language-shell">$ kubectl get certificate -n cfe-apps NAME READY SECRET AGE cfe-selfsigned-tls True cfe-tls-key-pair 23s </code></pre> <p>The K8s secret <em>cfe-tls-key-pair</em> will be created automatically in the same namespace as part of certificate deployment. Type the command shown below to check it:</p> <pre><code class="language-shell">$ kubectl get secrets -n cfe-apps cfe-tls-key-pair NAME TYPE DATA AGE cfe-tls-key-pair kubernetes.io/tls 3 52s </code></pre> <p>The secret <em>cfe-tls-key-pair</em> contains 3 keys, <em>ca.crt</em>, <em>tls.crt</em> and <em>tls.key</em>, which can be checked using the option <strong>-o yaml</strong> in the above <em>get secrets</em> command.</p> <h3>Test the certificate</h3> <p>Type the following <em>openssl</em> command to check the generated certificate:</p> <pre><code class="language-shell">$ openssl x509 -in &#x3C;(kubectl get secret -n cfe-apps cfe-tls-key-pair -o jsonpath='{.data.tls\.crt}' | base64 -d) -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 2a:2a:5d:0f:d1:e2:6f:60:3e:8a:93:4f:f4:e8:52:1e Signature Algorithm: sha256WithRSAEncryption Issuer: CN = example.com Validity Not Before: Feb 21 14:17:18 2024 GMT Not After : May 21 14:17:18 2024 GMT Subject: CN = example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:b7:7d:95:7f:55:a7:32:fd:66:b2:78:c0:2b:1f: 1f:69:c6:de:1f:85:eb:fb:2b:69:f3:60:23:df:9d: 3e:3d:41:df:c9:6b:b0:92:80:fe:6a:6f:19:4d:61: 20:3e:fc:19:af:f1:1d:5e:f6:b6:4f:17:5d:76:99: 3f:f4:d3:4a:70:15:f8:d5:3e:02:5c:c4:29:32:75: cd:e3:5a:07:7d:ea:47:71:37:3b:3d:36:89:36:e5: 8f:0e:03:57:ab:99:b3:6d:47:67:8a:6b:3b:2b:61: b0:08:96:a6:a2:5d:46:ed:ee:f3:5a:e3:6b:1d:05: 08:f1:ab:1b:ea:49:a3:2f:0d:82:37:80:76:00:18: 77:99:39:08:2e:06:54:28:24:e2:c8:9f:48:9c:ec: 75:0e:5e:a6:7b:ce:0b:68:96:d1:1a:4e:56:e1:ca: 42:ab:8e:11:a8:37:e1:70:ae:25:e3:2f:26:f1:7c: 95:fa:da:48:57:1f:a3:d7:47:84:86:9d:76:b3:99: a5:ef:10:98:96:31:ee:32:31:05:bc:5a:c0:94:bd: 25:ba:d6:86:32:d1:a6:3e:8c:21:99:a8:96:d6:5d: 69:35:01:8e:4f:d8:e9:90:78:17:ce:ac:4a:f8:13: 59:9b:e3:a8:9b:59:cc:c6:5f:5b:ca:6c:73:5e:e6: 88:f9 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 53:55:6D:56:AA:75:E2:87:9E:BB:C2:C7:45:32:2F:E3:1C:FF:17:62 X509v3 Subject Alternative Name: DNS:nginx.example.com, DNS:example.com Signature Algorithm: sha256WithRSAEncryption 69:e4:ae:bb:15:c1:d7:1a:54:49:10:6b:04:f9:1b:ed:bf:64: 0f:da:5e:b8:c2:e7:e2:d9:45:9e:66:92:0f:ce:f5:c9:5f:aa: b3:28:36:cd:16:da:6a:60:7f:eb:1d:85:fe:3a:38:65:71:0f: eb:da:e8:9e:1b:dc:f5:b7:14:4f:70:00:fd:bf:44:ed:37:35: bc:67:c7:4f:68:bc:5e:3b:bd:64:aa:5c:cd:1a:4f:11:90:c4: 6f:6a:d2:4b:90:4c:25:e7:ab:83:12:d7:38:b1:bf:70:8c:d5: cc:cb:70:70:b6:de:dc:8f:66:21:42:88:d5:7e:59:5f:6e:83: 73:81:e4:63:57:d1:c6:63:c0:9a:49:09:44:b5:d0:33:6b:3b: fd:3e:e4:c7:b7:d4:e4:72:0d:36:cf:a8:31:26:e3:ce:55:9f: 46:b8:fd:ab:7c:cc:2a:4b:e2:a6:a5:cd:2f:0c:3a:b1:2d:84: 1a:51:8b:e8:73:0f:cb:49:2e:a2:a6:ed:d5:e2:e8:cf:79:44: b9:2b:00:03:86:1a:a6:33:d4:20:33:9c:04:71:43:2d:9c:66: 3b:13:9b:6f:9f:f6:5f:f2:e0:e4:4a:04:64:c3:e6:bd:78:18: 19:22:d9:98:b5:47:85:0d:bd:b6:56:44:e6:89:34:30:90:20: 36:63:4f:1e </code></pre> <p>The line <em>X509v3 Subject Alternative Name</em> contains the <em>dnsNames</em> specified in the YAML file <em>certificate.yaml</em> during the certificate generation.</p> <h3>Integrate certificate with applications</h3> <p>There are several ways to integrate the generated certificates into applications deployed in the K8s cluster and configure applications to be accessed securely over HTTPS.</p> <p>The simplest way is to create the K8s <em>Deployment</em> resource with TLS block and <em>containerPort</em> configuration.</p> <p>Here is one sample Nginx Deployment YAML manifest file <em>nginx-deployment.yaml</em> that integrates the generated certificate:</p> <pre><code class="language-shell">$ cat nginx-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-app spec: replicas: 1 template: spec: containers: - image: nginx name: nginx ports: - containerPort: 443 tls: - secretName: cfe-tls-key-pair </code></pre> <p>By specifying the <em>containerPort</em> as <em>443</em> and referring the <em>secretName</em> to the generated K8s secret <em>cfe-tls-key-pair</em> under <em>tls</em> section, it enables TLS termination for the Nginx application.</p> <p>There is another way to integrate the certificate and configure it using the K8s <em>Ingress</em> resource with TLS parameters. This configuration requires a working Ingress controller setup in the cluster. There is a list of Ingress controllers, like: <a href="https://doc.traefik.io/traefik/providers/kubernetes-ingress/">Traefik</a>, <a href="https://github.com/haproxytech/kubernetes-ingress#readme">HAProxy</a>, <a href="https://www.nginx.com/products/nginx-ingress-controller/">Nginx Ingress controller</a>, you can deploy in the cluster.</p> <p>Here is one such sample Ingress YAML manifest file <em>ingress-nginx-selfsigned.yaml</em>:</p> <pre><code class="language-shell">$ cat ingress-nginx-selfsigned.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-ingress-selfsigned annotations: ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/issuer: "cfe-selfsinged-issuer" spec: ingressClassName: nginx tls: - hosts: - nginx.example.com secretName: cfe-tls-key-pair rules: - host: nginx.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-app port: number: 80 </code></pre> <p>It assumes the Nginx Ingress controller is deployed in the cluster. It configures the TLS block with the hostname <em>'nginx.example.com'</em> and the generated K8s secret.</p> <p>One benefit of this approach is that the sample Nginx application can be deployed in the cluster with the default service type <em>ClusterIP</em>, which provides internal connectivity and can solely be accessed from within the cluster. The Ingress controller will provide external access and handle SSL by accessing the certificate in the cluster and route the traffic to the deployed Nginx application in the backend.</p> <p>Type the following command to deploy the Ingress resource to the namespace <em>cfe-apps</em>:</p> <pre><code class="language-shell">$ kubectl apply -f ingress-nginx-selfsigned.yaml -n cfe-apps ingress.networking.k8s.io/nginx-ingress-selfsigned created </code></pre> <p>After deploying the Ingress using the above command, together with Nginx application deployment, to the namespace <em>cfe-apps</em>, you can validate the Ingress TLS using the browser.</p> <p>Start the browser and type the URL <em>nginx.example.com</em>, it will be rediected over HTTPS with the warning message <em>'Your connection is not private'</em>:</p> <p><img src="/img/nginx-private.png" alt=""></p> <p>You can click <em>Not secure</em> and start the Certificate Viewer to check the TLS certificate before clicking <em>Proceed to nginx.example.com (unsafe)</em> to go to the Nginx page:</p> <p><img src="/img/nginx-cert.png" alt=""></p> <h3>Conclusion</h3> <p>This blog post described the steps to generate a self-signed certificate using cert-manager for K8s in HPE GreenLake for Private Cloud Enterprise. Self-signed certificates provide an easy way to prove your own identity for the applications deployed in K8s cluster. This is a good option for development and testing environments. However, because self-signed certificates are not trusted certificates, they should not be used for production applications. For production use cases, you can try out cert-manager with <a href="https://letsencrypt.org/">Lets Encrypt</a>. You can refer to <a href="https://cert-manager.io/docs/">cert-manager documentation</a> on how to use it with the type of <em>Let’s Encrypt</em> challenges, as well as other sources than <em>Let’s Encrypt</em>.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/getting-started-with-the-hpe-greenlake-developer-portal/https://developer.hpe.com/getting-started-with-the-hpe-greenlake-developer-portal/Fri, 08 Mar 2024 17:37:23 GMT<p>You know about the <a href="https://developer.hpe.com/">HPE Developer Community portal</a> that provides an entry point to everything you need to know about HPE from a software developer and IT Ops standpoint, including how to access APIs (application programming interfaces), SDKs (software development kits), and training. The HPE Developer Community portal covers a wide range of HPE products, from HPE OneView, iLO, and Cray to the HPE GreenLake platform.</p> <p>From the HPE Developer Community portal, you can access the HPE GreenLake-specific developer portal where you can find the API documentation along with some trial capabilities. </p> <p>When reaching the <a href="https://developer.greenlake.hpe.com/">HPE GreenLake Developer Portal</a>, you are presented with three tiles that allow you to make the best use of the portal.</p> <p><img src="/img/blog-greenlake-dev-portal1.png" alt=""></p> <h2>Guides: Learn how to get started using the HPE GreenLake Developer Portal</h2> <p>The minimal requirement of being able to start using the portal is to have an HPE user account or an HPE GreenLake Account. To leverage the HPE GreenLake platform APIs, the user will need to obtain API client credentials for the HPE GreenLake platform. With the API client credentials, the user will be able to create an access token.  </p> <p>One cannot generate an access token without having an HPE account associated to a workspace. You will be guided through all the different steps to create the account either through the HPE GreenLake developer portal or through the HPE GreenLake portal. </p> <p>It is important to note that some content on the portal is private or only visible to a restricted group (for example, partners or developers). Within each group, several roles are identified and have specific privileges: </p> <p> <em>Public</em> </p> <p>Guest: Every visitor will have this role with a single permission to read all public content on the portal. </p> <p>Authenticated-User: Every logged in user will have this role. </p> <p><em>Partner</em> </p> <p>Users with the Partner role can access restricted content that is specific to a Partner.  </p> <h2>Services:  Discover the HPE GreenLake platform APIs  </h2> <p>Clicking on the second tile (Services), provides you with access to the HPE GreenLake APIs. But, as mentioned previously, an API needs a token to be tested. Therefore, the very first step described in the documentation API <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/#generate-or-reset-application-credentials">here</a> is to create API client credentials for the HPE GreenLake platform and generate an access token. </p> <p><img src="/img/blog-greenlake-dev-portal2.png" alt=""></p> <p>Once you have your token, you can try the different APIs that are related to the different services listed on the left-hand side of the page.  </p> <p><a href="https://developer.greenlake.hpe.com/docs/greenlake/services/audit-logs/public/">The HPE GreenLake audit log service</a>, for instance, will offer a collection of RESTful APIs for publishing audit logs, managing configurations, and retrieving application-specific and overall platform logs.</p> <p>Each service section offers, at a minimum, an overview, a guide, the Open API specification, and the associated API reference documentation. Some services provide additional parts to dive into more specific areas of the service. For instance, HPE GreenLake for Compute Ops Management service provides additional details about event webhooks and jobs. A webhook is an HTTP-based callback function that allows lightweight, event-driven communication between two APIs.  A job is a multi-step task that is managed by Compute Ops Management to perform an action on a resource. For example, performing power operations or firmware updates on a server. </p> <p>The Overview gives a definition of the service. The Guide explains the core details of the service from an API standpoint. Finally, the API reference documentation will provide you with a download link to the Open API specification file in JSON format of the API. This JSON file can be used in Postman, allowing you to build collection of REST API calls. In fact, the HPE Developer Community team has already put together a nice HPE GreenLake platform API collection. You can get the Postman collection from the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">HPE Developer Community tooling GitHub repository</a></p> <p>Finally, “la cerise sur le gateau” as we say in French, the try part. There you can make real API calls related to the selected service, bearing in mind that you have an HPE account, joined a workspace, and that you have generated the necessary API client credentials and access token. </p> <p>By simply clicking the Try it button, you can get access to the API. </p> <p><img src="/img/blog-greenlake-dev-portal3.png" alt=""></p> <p>Now, you can pass your first API call. Simply paste the token in the security window and hit the Send button. </p> <p><img src="/img/blog-greenlake-dev-portal4.png" alt=""></p> <p>As a result, a 200 status will inform you that the call was passed correctly, and the Headers window will display the result of the call. </p> <p>On the right-hand side of the windows, you will see that samples are being presented. You will find Curl, JavaScript, Node.js, Java, Python versions that you can copy / paste to test, allowing you to integrate them easily. </p> <p><img src="/img/blog-greenlake-dev-portal5.png" alt=""></p> <p>In addition, a Change Log section will inform you about the versioning of the API and changes related to it.</p> <h2>HPE Developer Community</h2> <p>The third tile allows you to reach out to us, [The HPE Developer Community](<a href="https://developer.hpe.com/">https://developer.hpe.com/</a>). Our web portal as mentionned in the introductionnary words of this blog, will help you find all things software at HPE <a href="https://developer.hpe.com/community">Join us</a> to collaborate and build applications and integrations with HPE products using the latest software and open source technologies.</p> <p>The HPE GreenLake Developer Portal offers one last important feature:</p> <h2><strong>The search window:</strong></h2> <p>Rather than browsing through the page to look for a necessary piece of information, you can use the search box and get the result you need straight away. Let's say you would like to know about HPE GreenLake for Compute Ops Management' APIs. A quick search will present you with many relevant entries.</p> <p><img src="/img/blog-greenlake-dev-portal6.png" alt=""></p> <h2>Summary </h2> <p>This blog post is intended to help you get started with the HPE GreenLake Developer Portal. Additional posts on the HPE GreenLake platform APIs have been produced by the HPE Developer Community team. There is <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-1-introduction-to-the-apis/">a three-part series</a> that takes you through the preparation steps you need  to use the APIs for common platform services and walks you through the steps required to obtain an OAuth access token to make secure REST API calls to the HPE GreenLake platform APIs. Then, you can dive into the Postman collection to learn how you, as an IT (Information Technology) administrator of the HPE GreenLake platform, can configure and manage workspace resources (users’ identity, devices, and subscriptions), and how you can track activities within your workspace to monitor the overall health of services and devices in your workspace. There is also a blog post about <a href="https://developer.hpe.com/blog/hpe-greenlake-edge-to-cloud-platform-scripting-fundamentals/">HPE GreenLake edge-to-cloud platform scripting fundamentals</a> where you can learn about the HPE GreenLake platform APIs through simple coding examples leveraging bash, PowerShell, and Python.</p> <p>Another way to experience these APIs is to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake APIs mentioned in this blog post. The workshops are free, available 24/7, and quite easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you are interested and go! It is as simple as that.  </p> <p>If you still have any questions regarding the HPE GreenLake platform APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We are always here to help.</p>https://developer.hpe.com/the-transformative-impact-of-generative-ai-on-telco-products/https://developer.hpe.com/the-transformative-impact-of-generative-ai-on-telco-products/Thu, 07 Mar 2024 16:28:15 GMT<p>In February 2024, Harvard Business Review published an insightful article titled “<a href="https://hbr.org/2024/01/turn-generative-ai-from-an-existential-threat-into-a-competitive-advantage">Turn Generative AI from an Existential Threat into a Competitive Advantage</a>”. The article outlines three strategic approaches for integrating generative AI into business operations to secure a competitive edge:</p> <ol> <li><strong>Adopting public AI tools:</strong> Utilizing readily available AI technologies, like ChatGPT, offers early movers a competitive edge by enhancing efficiency and innovation. Early adoption offers a competitive edge, though the advantage may be short-lived as such tools become universally essential.</li> <li><strong>Customizing AI tools:</strong> Tailoring AI tools with company-specific data improves customer experiences through personalized services and interfaces. This strategy, fueled by user feedback, advances towards more sophisticated AI applications, enhancing both customization and usability.</li> <li><strong>Automating data feedback loops</strong>: Implementing self-improving AI through automatic feedback loops from customer interactions boosts personalization and efficiency. Despite the challenge of integrating these mechanisms smoothly, it offers a sustainable competitive advantage.</li> </ol> <p>At HPE Athonet, our goal is to achieve this third level of implementation to enhance internal efficiency and maximize the value we deliver to our customers. This enables them to leverage our private networks for groundbreaking innovations. To achieve this, we are developing a generative AI platform founded on three core principles:</p> <ol> <li><strong>Transition from individual tools to a team-centric collaborative framework:</strong> Shifting the focus towards collaborative efforts that enhance team synergy and productivity.</li> <li><strong>Designing a user experience to enhance focus and facilitate flow:</strong> Crafting an interface that minimizes distractions and supports users in achieving a state of deep concentration, or 'Flow'.</li> <li><strong>Building a versatile architecture incorporating data mesh and microservices principles:</strong> Developing a flexible system architecture that integrates advanced data management and service-oriented designs to support scalable and efficient operations.</li> </ol> <p>We showcased a proof of concept (PoC) of our innovative solution at the Mobile World Congress in Barcelona, February 2024. Our chatbot demonstrated its capability to fetch information and execute commands over Athonet's 5G networks effectively.</p> <p><img src="/img/athon_mwc1.png" alt=""></p> <p>The PoC highlighted the ability of our reasoning engine to orchestrate multiple tools, enabling seamless connection to Athonet 5G networks via APIs and efficient retrieval of document information, both features accessible through chat interactions.</p> <p><img src="/img/athon_mwc2.png" alt=""></p> <p>This pioneering approach not only showcases the potential and flexibility of this architecture in network systems management but also emphasizes the profound impact such technologies can have on the digital transformation journey. We designed this strategy with the aim to elevate HPE Athonet into a future where AI not only enhances but significantly boosts our operational capabilities and the value we deliver to our customers.</p> <p>Stay tuned for our upcoming posts where we will delve deeper into the core components and principles of this idea.</p>https://developer.hpe.com/2024-march-05/https://developer.hpe.com/2024-march-05/Tue, 05 Mar 2024 13:00:00 GMThttps://developer.hpe.com/finetuning-mistral-7b-with-lora-and-deepspeed/https://developer.hpe.com/finetuning-mistral-7b-with-lora-and-deepspeed/Wed, 28 Feb 2024 16:21:51 GMT<p>External blog post</p>https://developer.hpe.com/changes-to-chapel-2-0-since-its-first-release-candidate/https://developer.hpe.com/changes-to-chapel-2-0-since-its-first-release-candidate/Wed, 28 Feb 2024 01:53:46 GMT<p>External blog</p>https://developer.hpe.com/the-latest-ai-models-datasets-and-research/https://developer.hpe.com/the-latest-ai-models-datasets-and-research/Tue, 27 Feb 2024 15:40:41 GMT<p>External blog post</p>https://developer.hpe.com/why-is-redfish®-different-from-other-rest-apis-part-2/https://developer.hpe.com/why-is-redfish®-different-from-other-rest-apis-part-2/Thu, 22 Feb 2024 17:22:57 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/why_is_redfish_different/why_is_redfish_different_part2">Server Management Portal</a>.</big></p> <br>https://developer.hpe.com/streamline-your-server-deployments-bare-metal-provisioning-with-hpe-compute-ops-management-and-ansible/https://developer.hpe.com/streamline-your-server-deployments-bare-metal-provisioning-with-hpe-compute-ops-management-and-ansible/Wed, 21 Feb 2024 13:16:34 GMT<style>ul li{ font-size:27px;padding-bottom: 0.5em;}</style> <style>ol li{ font-size:27px;padding-bottom: 0.5em;}</style> <style>ul ul li {padding-bottom: 8px;}</style> <style>ol ol li {padding-bottom: 8px;}</style> <style>li > ul {margin-top: 10px;}</style> <style>li > ul > li:last-child {margin-bottom: 0px;}</style> <style> i{ color:grey;font-family:'Courier New';font-size:22px; } </style> <style>   img {     max-width: 100%;     height: auto;     border: 1px solid #ccc;     margin: 20px;     box-shadow: 2px 2px 5px #ccc;   } </style> <p>In the rapidly-evolving world of IT infrastructure management, achieving speed, efficiency, and reliability in server provisioning can make a significant difference. This is where cutting-edge tools like <a href="https://www.hpe.com/us/en/hpe-greenlake-compute-ops-management.html">HPE GreenLake for Compute Ops Management</a> and <a href="https://www.ansible.com/">Ansible</a> come into play. Together, they create a robust platform for managing your infrastructure seamlessly. In this blog post, I will introduce an exciting new GitHub project that exemplifies how to harness these tools for optimal bare metal provisioning.</p> <h2>Introducing a new GitHub project</h2> <p>I am excited to share a new project. It is an open-source initiative hosted on GitHub that aims to enhance the integration between HPE GreenLake for Compute Ops Management and Ansible. This endeavor is focused on making it easier to configure, manage and provision bare metal servers at scale.</p> <p>The initial aim of this project was to focus on server provisioning for the ESXi, RHEL and Windows Server platforms. However, it also aims to provide an overview of the various capabilities of the Compute Ops Management API. The project effectively demonstrates a wide range of API interactions, covering everything from initial installation (Day0 operations) through the early stages of active use (Day1) to ongoing maintenance (Day2) with automated firmware updates.</p> <p>Main operations include:</p> <ul> <li>Collecting server information</li> <li>Identifying storage destinations for the operating system install</li> <li>Configuring server settings: <ul> <li>BIOS settings</li> <li>Storage configurations</li> <li>OS provisioning</li> </ul> </li> <li>Creating tailor-made kickstart scripts and assembling ISOs</li> <li>Starting and monitoring OS image installation</li> <li>Installing and monitoring HPE Agentless Management Service (AMS) and Smart Update Tool (SUT)</li> <li>Creating server groups with specific settings</li> <li>Adding servers to temporary and permanent server groups</li> <li>Executing firmware updates</li> <li>Monitoring task execution</li> <li>Managing errors related to tasks</li> </ul> <p>In this project, automating the provisioning of operating systems on bare metal servers is made simple and accessible to anyone with basic knowledge of Ansible, HPE Compute Ops Management, and kickstart techniques. While it is generally a complex process that requires a wide range of skills, this project simplifies it with the use of auto-customized kickstarts, auto-generated ISO files and by exploiting the very compelling features of HPE Compute Ops Management server groups.</p> <h2>Key highlights of this project</h2> <ul> <li><strong>Automated provisioning</strong>: Kickstart your server setups without tedious manual configuration.</li> <li><strong>Centralized control</strong>: Manage your entire fleet of servers from a single pane of glass.</li> <li><strong>Scalable architecture</strong>: Effortlessly scale your infrastructure to meet growing business demands.</li> <li><strong>Pre-built playbooks</strong>: Jumpstart your automation with curated collection, crafted for various deployment scenarios (Microsoft Windows Server, VMware ESXi and Red Hat Enterprise Linux).</li> <li><strong>Custom Ansible variables</strong>: Enable you to define environment-specific parameters, ensuring that each server gets a configuration that fits its role in the infrastructure, ensuring that you have granular control over server provisioning.</li> <li><strong>Comprehensive documentation</strong>: Detailed guides, videos and examples help you customize the workflow to your specific requirements.</li> </ul> <h2>Mastering server management with HPE GreenLake for Compute Ops Management</h2> <p>HPE GreenLake for Compute Ops Management is a comprehensive solution for hardware resource management, providing a seamless way to handle server deployments. With its ability to manage health monitoring, orchestrate server configuration and firmware update workflows, and automate bare metal provisioning, administrators can ensure their data centers operate optimally with less effort and greater oversight. To learn more, see <a href="https://www.hpe.com/emea_europe/en/hpe-greenlake-compute-ops-management.html">HPE GreenLake for Compute Ops Management</a></p> <h2>Bridging HPE GreenLake for Compute Ops Management and Ansible</h2> <p>HPE GreenLake for Compute Ops Management provides the foundational management capabilities essential for maintaining data center health and efficiency. When combined with the automation capabilities of Ansible, IT administrators can achieve unprecedented levels of automation.</p> <h2>Bringing it all together</h2> <ul> <li><strong>Automated Workflows</strong>: Convert time-consuming manual processes into automated workflows that can be tracked and managed easily.</li> <li><strong>Scalable Infrastructure</strong>: Embrace growth without compromising on performance or manageability.</li> <li><strong>Reduced Human Error</strong>: Minimize mistakes by standardizing server configurations across the board.</li> </ul> <h2>Mastering parallel execution with Ansible</h2> <p>A key attribute of Ansible that I looked for in this project is its impressive capability to execute tasks concurrently across multiple systems, thereby accelerating deployment processes. This feature is called "forks" in Ansible. Set to 5 by default, the forks value is adjustable based on available system resources (CPU and memory), signifying that Ansible can carry out playbook tasks in parallel across 5 hosts from the inventory list. This parallel execution is among Ansible's outstanding functionalities, enhancing the effectiveness of bare-metal provisioning substantially. Moreover, this approach ensures consistent configurations across all provisioned hosts.</p> <h2>Where to start?</h2> <p>To gain an understanding of the project, please refer to the <a href="https://github.com/jullienl/HPE-COM-baremetal/blob/main/readme.md">readme.md</a> file within the project's repository. It will provide you with detailed instructions on:</p> <ul> <li><a href="https://github.com/jullienl/HPE-COM-baremetal#prerequisites">The necessary prerequisites for utilizing this project</a></li> <li><a href="https://github.com/jullienl/HPE-COM-baremetal#ansible-control-node-information">The process for setting up the Ansible control node</a></li> <li><a href="https://github.com/jullienl/HPE-COM-baremetal#preparation-to-run-the-playbooks">The initial steps required prior to executing a playbook</a></li> </ul> <h2>How to run a playbook?</h2> <p>A single command is required to provision all hosts listed in an inventory file:</p> <pre><code class="language-shell">ansible-playbook &#x3C;provisioning_file>.yml -i &#x3C;inventory_file> --ask-vault-pass --ask-become-pass </code></pre> <p>Where <code>&#x3C;provisioning_file></code> should be replaced with <code>ESXi_provisioning</code>, <code>RHEL_provisioning</code>, or <code>WIN_provisioning</code> depending on the target operating system. Similarly, replace <code>&#x3C;inventory_file></code> with the appropriate inventory filename such as <code>hosts_ESXi</code>, <code>hosts_RHEL</code>, or <code>hosts_WIN</code>.</p> <p>Upon running this command, Ansible will prompt you to enter the vault password and the sudo password to proceed with the provisioning process.</p> <h2>Explore my video series</h2> <p>Dive into this series of videos showcasing the seamless bare metal operation across three major operating systems. Each video provides a walk through over the different variables involved and the files that are required to update HPE drivers and software, along with the explanation of the different steps of each playbook.</p> <h3>Windows Server Bare Metal Provisioning on 2 x HPE ProLiant DL360 Gen10 Plus</h3> <p><a href="https://www.youtube.com/watch?v=A6RD6nIAFmw"><img src="https://img.youtube.com/vi/A6RD6nIAFmw/hqdefault.jpg" alt="Automatic Windows server"></a></p> <h3>RHEL 9.3 Bare Metal Provisioning on 2 x HPE ProLiant DL360 Gen10 Plus</h3> <p><a href="https://www.youtube.com/watch?v=6_o8yB4cvag"><img src="https://img.youtube.com/vi/6_o8yB4cvag/hqdefault.jpg" alt="Automatic RHEL Bare Metal"></a></p> <h3>VMware ESXi Bare Metal Provisioning on 2 x HPE ProLiant DL360 Gen10 Plus</h3> <p><a href="https://www.youtube.com/watch?v=_ySgROdd_Bw"><img src="https://img.youtube.com/vi/_ySgROdd_Bw/hqdefault.jpg" alt="Automatic ESX Bare Metal"></a></p> <br /> <p>Join me on my <a href="https://github.com/jullienl/HPE-COM-baremetal">GitHub repository</a>, where a wealth of information awaits you in the README file. Learn how to effectively utilize this project, from cloning it into your environment to commencing with its use, and witness the ways it can streamline your bare metal provisioning workflow.</p> <p>Stay tuned as I continue to update and maintain this project, incorporating user <a href="mailto:lio@hpe.com">feedback</a> and the latest advancements that HPE GreenLake will offer.</p> <p>Get started now and begin transforming your server deployment strategy today!</p> <p> </p>https://developer.hpe.com/why-is-redfish®-different-from-other-rest-apis-part-1/https://developer.hpe.com/why-is-redfish®-different-from-other-rest-apis-part-1/Mon, 19 Feb 2024 16:20:51 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/why_is_redfish_different/why_is_redfish_different_part1">Server Management Portal</a></big></p> <br>https://developer.hpe.com/mesa-and-co-designing-model-architectures-with-hardware/https://developer.hpe.com/mesa-and-co-designing-model-architectures-with-hardware/Mon, 05 Feb 2024 16:47:48 GMT<p>External blog post</p>https://developer.hpe.com/2024-February-05/https://developer.hpe.com/2024-February-05/Mon, 05 Feb 2024 13:00:00 GMThttps://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-3-tracking-activities-and-monitoring-health/https://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-3-tracking-activities-and-monitoring-health/Fri, 02 Feb 2024 10:38:42 GMT<p><strong>Editor's note:</strong> This blog post series may refer to older release of the HPE GreenLake platform APIs. For information about the current release of the HPE GreenLake service APIs, please visit the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake API catalog</a>.</p> <p>This is part three of a blog series that showcases the capabilities of the APIs for common HPE GreenLake platform services using a real customer scenario, displaying it from the perspective of a user of the platform, such as an IT administrator.</p> <p>Continuing on from the <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-2-configuring-and-managing-a-workspace/">second part of this series</a>, where I had put on my IT administrator’s hat for a <em><strong>Standard Enterprise</strong></em> workspace, I will now explore the set of REST API calls used for tracking activities in the workspace and monitoring the overall health of HPE services and HPE products in the workspace.</p> <h2>Tracking activities in the workspace</h2> <p><strong>Audit log</strong> service records the occurrence of events emitted by users, any device or service in the workspace. These logs can be used for tracking user activities, doing root cause analysis of an incident, investigating breaches, and for auditing purposes.</p> <p>Let’s assume that I have been notified of unusual activities in the workspace and I would like to act to begin an investigation and identify the root cause of the incident in the workspace. This involves an analysis of logs for the services and the platform in the workspace, and tracking activities for users. To conduct this analysis, I will use the set of audit log API calls from the Postman collection folder: <em><strong>Tracking GLP Workspace/Step5-audit-log/audit-log/v1beta1/logs</strong></em>.</p> <h3>Collecting service-specific logs and platform logs</h3> <p>The <strong>GET</strong> REST API request <em><strong>Get all audit logs of an application</strong></em> derived from the API call <em><strong>GET all audit logs of an application or user</strong></em> is used to get logs from a specific service in the workspace and platform logs in the workspace. I just need to specify the service identifier or the HPE GreenLake platform identifier to obtain the list of logs for a particular service or for the platform. In the example below, I specify the identifier of the HPE GreenLake platform and limit the output for activities that occurred after a certain date and time in UTC following the <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601 standard</a>:</p> <p><code>GET {{baseUrl}}/audit-log/v1beta1/logs?filter=application/id eq '{{GLP_Application_Id}}'&#x26;filter=createdAt ge '2023-12-10T11:00:00.00000Z'&#x26;limit=50&#x26;offset=0</code></p> <p><img src="/img/blog-part3-auditlog-application-specific-logs-image2.png" alt="Figure 1: Audit logs for the HPE GreenLake platform in the workspace" title="Figure 1: Audit logs for the HPE GreenLake platform in the workspace"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 1: Audit logs for the HPE GreenLake platform in the workspace</span></p> </blockquote> </blockquote> <h3>Tracking user-specific activities</h3> <p>The <strong>GET</strong> REST API call <em><strong>Get all audit logs of a user</strong></em> derived from the REST API call <em><strong>Get all audit logs of an application or user</strong></em> is used to track activities for a specific user in the workspace based on the criteria specified in the filter query parameters. In this example I limit the output for a particular user’s activities that occurred after a certain date and time in UTC following the <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601 standard</a>:</p> <p><code>GET {{baseUrl}}/audit-log/v1beta1/logs?filter=user/username eq '&#x3C;UserEmail@example.com>'&#x26;filter=createdAt ge '2023-12-14T11:00:00.00000Z'&#x26;limit=300</code></p> <blockquote> <p><strong>Note:</strong> You can specify additional query parameters to limit the scope of the output to a specific category of activities. For example, <em>User Management</em>, <em>Device Management</em>, or <em>Customer Management</em>.</p> </blockquote> <p><img src="/img/blog-part3-auditlog-tracking-user-activities-image1.png" alt="Figure 2: Tracking activities for a specific user" title="Figure 2: Tracking activities for a specific user"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 2: Tracking activities for a specific user</span></p> </blockquote> </blockquote> <h3>Monitoring health events for the workspace</h3> <p>The HPE GreenLake platform provides a <strong>wellness</strong> service to enable you to monitor the overall health of the managed services and devices in the workspace. The wellness service API provides programmatic access to view health events and insights about HPE services and HPE products in the workspace.</p> <p>I will use the set of Wellness API calls from the Postman collection folder: <em><strong>Tracking GLP Workspace/Step6-Wellness</strong></em>.</p> <p>The <strong>GET</strong> REST API call <em><strong>Get a list of wellness events</strong></em> is used to retrieve a list of health events for services (for example, Networking, Storage, Compute or HPE GreenLake platform services) and HPE device models in the workspace. In the example below, I use a set of query parameters to view health information for a specific device model, service and severity:</p> <p><code>GET {{baseUrl}}/wellness/v2beta2/events?filter=serviceName in ('Storage')&#x26;filter=productName eq 'HPE Alletra 6000'&#x26;filter=condition/severity eq 'warning'&#x26;limit=100</code></p> <p><img src="/img/blog-part3-list-wellness-v2beta2-events-for-service-product-severity-image3.png" alt="Figure 3: Retrieve list of health events for a specific service, device model and severity" title="Figure 3: Retrieve list of health events for a specific service, device model and severity"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 3:Retrieve list of health events for a specific service, device model and severity</span></p> </blockquote> </blockquote> <p>To view information about a specific event ID, I can use the REST API request <em><strong>Get wellness event with specific ID</strong></em> by specifying the event ID as a Path variable in the parameters of the API call:</p> <p><code>GET {{baseUrl}}/wellness/v2beta2/events/:id</code></p> <p><img src="/img/blog-part3-view-wellness-v2beta2-event-for-specific-event-id-image4.png" alt="Figure 4: View information for a specific event ID" title="Figure 4: View information for a specific event ID"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 4: View information for a specific event ID</span></p> </blockquote> </blockquote> <h2>Summary</h2> <p>This blog series walks you through the APIs for common HPE GreenLake platform services <strong>for a single-tenant workspace</strong> environment from the perspective of an IT administrator. I took advantage of the Postman collection available on the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">HPE Developer Community tooling GitHub repository</a> to help you get started with these APIs, learn the REST API call syntax for the API requests through the use of examples, how to <strong>programmatically</strong> configure and manage workspace resources such as users and infrastructure devices, and how to track activities and monitor health events for the workspace.</p> <p>To learn more about all the REST API calls for the platform, I invite you to refer to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake platform documentation</a> for these APIs. The documentation leverages OpenAPI specifications and associated reference documentation for these API services. It provides a complete explanation of the operations supported by these APIs for common HPE GreenLake platform services, as well as sample requests and responses.</p> <p>If you’re interested in trying out what I just discussed, you might want to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake APIs mentioned in this blog series. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested in and go! It’s as simple as that.</p> <p>If you still have any questions regarding the HPE GreenLake platform APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-2-configuring-and-managing-a-workspace/https://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-2-configuring-and-managing-a-workspace/Wed, 31 Jan 2024 17:29:10 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><strong>Editor's note:</strong> This blog post series may refer to older release of the HPE GreenLake platform APIs. For information about the current release of the HPE GreenLake service APIs, please visit the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake API catalog</a>.</p> <p>This is Part 2 of a blog series that showcases the capabilities of APIs for common HPE GreenLake platform services through a real customer use case, presenting it from the perspective of a user of the platform, such as an IT administrator.</p> <p><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-1-introduction-to-the-apis/">In the previous blog post</a>, I described the current set of APIs for the HPE GreenLake platform, and I covered the Postman collection aspect I built to get started with these APIs.</p> <p>In this second part of the series, I will put on my IT administrator’s hat and assume the role of the HPE GreenLake platform Workspace Administrator for a <em><strong>Standard Enterprise</strong></em> workspace. This type of workspace is a <em>single-tenant</em> environment for a single customer and organization. As workspace administrator, I have full privileges to provision, manage and monitor the users and IT resources in the workspace.</p> <p>As I do so, I will show you how to use these foundational, common APIs to <strong>programmatically</strong> configure and manage workspace resources (users and infrastructure devices) like an administrator would do using the HPE GreenLake platform User Interface (UI). I will walk you through some of the most common REST API calls to the HPE GreenLake platform API services based on a typical customer scenario that will allow you to learn what you can do on the platform using this set of APIs:</p> <ul> <li> <p>Configuring and managing an HPE GreenLake platform workspace:</p> <ul> <li>Invite users to join a workspace.</li> <li>Add a device (for example an HPE Aruba Access Point) and subscription to the workspace.</li> <li>Attach the device to a regional instance of a service deployed in the workspace (for example, HPE Aruba Networking Central service in Central Europe) and a subscription key (a license) to operate the device.</li> <li>Remove service and subscription assignments for a device.</li> </ul> </li> </ul> <h2>Obtaining the OAuth access token</h2> <p>As I prepare to access and use the HPE GreenLake platform workspace resources through REST API calls to the platform API services, I first need to generate an OAuth access token. Refer to <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-1-introduction-to-the-apis/">my blog post Part 1</a> to learn how to generate the access token using the Postman collection available <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">here</a>.</p> <p>Once the access token is generated, it will be used as the <strong>authorization bearer token</strong> for all subsequent REST API calls.</p> <h2>Inviting a tenant user to collaborate in the workspace</h2> <p>As an administrator in my HPE GreenLake platform workspace, I can easily invite other members of my organization to join the workspace by sending them a sign-up link in email. Here I am using the <strong>POST</strong> REST API call - <em><strong>Invite a user</strong></em>, taken from the Postman collection folder: <em><strong>Configuring and Managing GLP Workspace/Step2-IAM/Identity/v1/users</strong></em>.</p> <p><code>POST {{baseUrl}}/identity/v1/users</code></p> <p><img src="/img/blog-part2-invite-user-image1.png" alt="Figure 1: Invite a user REST API call" title="Figure 1: Invite a user REST API call"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 1: Invite a user REST API call</span></p> </blockquote> <p><strong>Note:</strong> You must be assigned the HPE GreenLake platform <em>Workspace Administrator</em> role to invite a user to join the workspace.</p> </blockquote> <p>Let’s look at this REST API request <strong>syntax</strong>. All REST API calls to HPE GreenLake platform services are made by providing:</p> <ul> <li>The single unified domain endpoint (<em><a href="https://global.api.greenlake.hpe.com">https://global.api.greenlake.hpe.com</a></em>) defined in the baseURL variable,</li> <li>The HTTP request method such as GET, POST, PUT/PATCH, or DELETE. In this example, the POST method is specified to create a new user instance in the workspace.</li> <li>The path (<em>API-group-name/API-Version/Workspace-Resources</em>). The path specifies the API group name (here identity), the version of the API (here v1), and the resource path in the workspace (here users).</li> <li>A data payload when using a method that involves changing (PUT/PATCH) or creating (POST) an object instance. In this example, as the method involves creating a user instance in the workspace, the data payload (the Body) specifies the e-mail user address, and a welcome email is sent to the user to invite the user to join the workspace.</li> </ul> <p>To get started, I hit the <strong>Send</strong> button and I get a <em><strong>201 Created</strong></em> response indicating that the user has been successfully invited. The user then receives an email to confirm and accept the invitation. The user will then be invited to create and activate an HPE account to join the workspace. Users who are invited are not <strong>verified</strong> users until they accept the email confirmation.</p> <p>I can then verify the user has joined the workspace by using the GET REST API call <em><strong>Get invited users by Username</strong></em> below:</p> <p><code>GET {{baseUrl}}/identity/v1/users?filter=username eq '&#x3C;user’s email address>'</code></p> <p><img src="/img/blog-part2-get-invited-user-image2.png" alt="Figure 2: Checking the status of the invited user in the workspace" title="Figure 2: Checking the status of the invited user in the workspace"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 2: Checking the status of the invited user in the workspace</span></p> </blockquote> </blockquote> <p>The REST API call syntax is the same as the previous API request, but here a <strong>GET</strong> method is used to list the users in the workspace.</p> <p>One or more <strong>query parameters</strong> indicated after the question mark (“<strong>?</strong>”) in the URL can be used to filter the data that an API request returns. Typical query parameters are:</p> <ul> <li><strong>filter:</strong> filter the set of resources returned based on the criteria specified in the filter query parameter</li> <li><strong>limit:</strong> maximum number of records to return</li> <li><strong>offset:</strong> resource offset to start the response from</li> </ul> <p>In this example, I use <em><strong>filter</strong></em> as the query parameter to limit the scope of the output of the call to just the invited user’s email specified in the filter query parameter.</p> <p>I now hit the <strong>Send</strong> button. The request indicates success (<em><strong>Status: 200 OK</strong></em>). In the response, a <em>userStatus</em> of <em>VERIFIED</em> means that the user has activated the HPE account and joined the workspace. A user who has already activated their HPE account will automatically be added to the workspace upon invitation. A <em>userStatus</em> of <em>UNVERIFIED</em> would mean that the user has not created and activated the HPE account yet.</p> <blockquote> <p><strong>Note:</strong> Thanks to the Postman <em>Post-response Script</em> associated with this request, the unique identifier of the invited user is automatically saved as collection variable. The identifier of the user is needed should an administrator want to disassociate (delete) a user from the workspace using the REST API call <em><strong>DELETE Disassociate a user</strong></em>.</p> </blockquote> <h2>Managing IT resources (devices and subscriptions) into the workspace</h2> <p>A typical scenario to manage infrastructure resources from the HPE GreenLake platform would be where one would:</p> <ol> <li>Add an infrastructure device and a subscription key for this device to the workspace. In this scenario, I will add an HPE Aruba Access Point and associated subscription key to the inventory of the workspace.</li> <li>Attach the device to a service to manage and operate the device. In this scenario, I will assign the HPE Aruba Access Point to the HPE Aruba Networking Central service already deployed in the workspace. The HPE Aruba Networking Central service is a SaaS-based User Interface that lets customers manage their fleet of networking equipment from edge-to-cloud from a single web interface.</li> <li>Assign a subscription key to the device. A subscription key is a license key needed to activate the device and allows the IT administrator to use and operate it using the appropriate service management console such as HPE Aruba Networking Central for networking devices, HPE GreenLake for Compute Ops Management for compute servers, and Data Services Cloud Console for storage arrays.</li> <li>Remove assignment of a service or a subscription for a device.</li> </ol> <h3>Adding a device and subscription</h3> <p>Here I am going to use the REST API calls from the Postman collection folder: <em><strong>Configuring and Managing GLP Workspace/Step4-Service Catalog, Devices and Subscriptions</strong></em>.</p> <p>The <strong>POST</strong> REST API call <em><strong>Add devices - Aruba Access Point with Tag</strong></em> from <em><strong>/devices/v1beta1/devices</strong></em> subfolder allows me to add an HPE Aruba Networking device to the inventory in the workspace by providing device details in the data payload (Body) of the request. The device information for HPE Aruba Networking equipment includes the <em>Serial Number</em> and the <em>MAC address</em>.</p> <p>Optionally I can assign a “<strong>tag</strong>” to the device while adding it to the inventory. Tags are <em>name-value</em> pairs that can be very useful for identifying and categorizing groups of resources. In the example below, the tag’s name is “<em>Location</em>”, and the value is “<em>Lab Building 2</em>”.</p> <p><code>POST {{baseUrl}}/devices/v1beta1/devices</code></p> <pre><code class="language-json">{ "compute": [], "storage": [], "network": [ { "serialNumber": "&#x3C;SerialNumber of the device>", "macAddress": "&#x3C;MAC-address of the device>", "tags": { "Location": "Lab Building 2" } } ] } </code></pre> <p>The “<strong>Add devices</strong>” API call is an asynchronous operation. Asynchronous operations are API operations that cannot be completed immediately. The response of the request indicates <em><strong>Status: 202 Accepted</strong></em> and contains the <em>transaction Id</em> of the asynchronous operation that I can use as a Path variable in the subsequent <strong>GET</strong> API call <em><strong>Get progress or status of async operations in devices</strong></em> to verify whether the asynchronous operation is successful or not:</p> <p><code>GET {{baseUrl}}/devices/v1beta1/async-operations/:id</code></p> <p><img src="/img/blog-part2-adding-device-async-operation-image5.png" alt="Figure 3: Checking status of asynchronous operation for adding devices" title="Figure 3: Checking status of asynchronous operation for adding devices"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 3: Checking the status of the asynchronous operation for adding devices</span></p> </blockquote> </blockquote> <p>A similar sequence of REST API calls can be used to add a subscription key in the workspace inventory and verify the status of the asynchronous operation. The <strong>POST</strong> REST API call <em><strong>Add subscriptions key for AP device</strong></em> derived from the API call <em><strong>POST Add subscriptions</strong></em> from <em><strong>/subscriptions/v1beta1</strong></em> subfolder allows me to add a subscription for HPE Aruba Networking devices to the inventory in the workspace by providing the <em>subscription key</em> in the data payload (Body) of the request. The API call is an asynchronous operation.</p> <p><code>POST {{baseUrl}}/subscriptions/v1beta1/subscriptions</code></p> <pre><code class="language-json">{ "subscriptions": [ { "key": "&#x3C;Subscription key>" } ] } </code></pre> <p>The GET API call <em><strong>Get progress or status of async operations in subscriptions</strong></em> is used to verify status of the asynchronous operation:</p> <p><code>GET {{baseUrl}}/subscriptions/v1beta1/async-operations/:id</code></p> <p>I can now use the two subsequent REST API calls below to fetch detailed information about the device and the subscription key I have just added to the inventory:</p> <ul> <li>The <em><strong>Get devices managed in a workspace</strong></em> API call allows me to obtain detailed information about the device by specifying the <em>SerialNumber</em> of the device in the filter query parameter. This request will allow me to save the unique identifier of the device as a collection variable.</li> <li>Similarly, the <em><strong>Get subscriptions of a workspace</strong></em> API call allows me to get detailed information about the <em>subscription key</em> and fetch the unique identifier of the subscription key as collection variable.</li> </ul> <p><code>GET {{baseUrl}}/devices/v1beta1/devices?filter=serialNumber eq '&#x3C;SerialNumber>'</code></p> <p><code>GET {{baseUrl}}/subscriptions/v1alpha1/subscriptions?filter=key eq '&#x3C;SubcriptionKey>'</code></p> <blockquote> <p><strong>Note:</strong> I will need the <em>identifier</em> of the device to attach the device to a regional instance of a service management console. I will also need the <em>identifier</em> of the subscription key to assign the subscription key to the device as explained in the next step.</p> </blockquote> <h3>Assigning the device to a regional instance of a service</h3> <p>Next, using the <strong>PATCH</strong> <em><strong>Update devices - Assign Application to a device</strong></em> REST API request (derived from the <em><strong>PATCH Update devices API call</strong></em>), I can attach the device to a regional instance of the HPE Aruba Networking Central management console service already deployed in the workspace. The identifier of the device is specified as a query parameter, the HPE Aruba Networking Central service <em>identifier</em> and <em>region</em> are specified in the data payload (Body) as shown below:</p> <p><code>PATCH {{baseUrl}}/devices/v1beta1/devices?id={{DeviceId}}</code></p> <pre><code class="language-json">{ "application": { "id": "{{Aruba_Application_Id}}" }, "region": "&#x3C;region>" } </code></pre> <p><img src="/img/blog-part2-assign-app-to-device-image6.png" alt="Figure 4: Assign device to a regional service instance" title="Figure 4: Assign device to a regional service instance"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 4: Assign device to a regional instance of the HPE Aruba Networking Central service</span></p> </blockquote> </blockquote> <p>This API call is an asynchronous operation, and I can use the <strong>GET</strong> API call <em><strong>Get progress or status of async operations in devices</strong></em> to verify the status of the operation.</p> <h3>Applying a subscription key to the device</h3> <p>Similarly, I can use the same REST API call to assign a subscription key to the device specifying the identifier of the device as the query parameter and the identifier of the subscription key in the data payload (Body):</p> <p><code>PATCH {{baseUrl}}/devices/v1beta1/devices?id={{DeviceId}}</code></p> <pre><code class="language-json">{ "subscription": [ { "id": "{{SubscriptionKeyId}}" } ] } </code></pre> <p><img src="/img/blog-part2-assign-subscriptionkey-to-device-image7.png" alt="Figure 5: Assign a subscription key to a device" title="Figure 5: Assign a subscription key to a device"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 5: Assign a subscription key to a device</span></p> </blockquote> </blockquote> <p>This API call is an asynchronous operation, and I can use the <strong>GET</strong> API call <em><strong>Get progress or status of async operations in devices</strong></em> to verify the status of the operation.</p> <h3>Removing assignment of service and subscription</h3> <p>During ongoing operations in the workspace, I may need to remove assignment of a service and a subscription for a particular device. I can use the <strong>PATCH</strong> REST API calls <em><strong>Update devices – Unassign Application for a Device</strong></em> and <em><strong>Update devices - Unassign Subscription Key for a device</strong></em> respectively.</p> <p>To remove an assignment of a service for a device, leave the <em>application</em> field as empty value in the data payload as shown here:</p> <p><code>PATCH {{baseUrl}}/devices/v1beta1/devices?id={{DeviceId}}</code></p> <pre><code class="language-json">{ "application": { } } </code></pre> <p>To remove an assignment of a subscription key for a device, leave the <em>subscription</em> field as empty value in the data payload as shown here:</p> <p><code>PATCH {{baseUrl}}/devices/v1beta1/devices?id={{DeviceId}}</code></p> <pre><code class="language-json">{ "subscription": [ ] } </code></pre> <h2>Summary</h2> <p>This blog post walks you through the APIs for common HPE GreenLake platform services <strong>for a single-tenant workspace</strong> environment from the perspective of an IT administrator. I took advantage of the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">Postman collection</a> to help you get started with these APIs, learn through examples the REST API call syntax for the API requests and how to <strong>programmatically</strong> configure and manage workspace resources such as users, users’ roles and infrastructure devices.</p> <p>To learn more about all the REST API calls for the platform, I invite you to refer to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake platform documentation</a> for these APIs. You can get the Postman collection from the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">HPE Developer Community tooling GitHub repository</a>.</p> <p>In <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">the next part</a> of this blog series, I will explore the set of APIs used for tracking activities and monitoring overall health of services and devices in the workspace.</p> <p>If you’re interested in trying out what I just discussed, you might want to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake APIs mentioned in this blog post. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested in and go! It’s as simple as that.</p> <p>If you still have any questions regarding the HPE GreenLake platform APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/finetuning-an-llm-using-huggingface-determined/https://developer.hpe.com/finetuning-an-llm-using-huggingface-determined/Wed, 31 Jan 2024 15:29:49 GMT<p>External blog post</p>https://developer.hpe.com/comparing-standard-library-sorts-the-impact-of-parallelism/https://developer.hpe.com/comparing-standard-library-sorts-the-impact-of-parallelism/Tue, 30 Jan 2024 19:16:00 GMT<p>External blog</p>https://developer.hpe.com/mambabyte-multimodal-pathway-and-crossmae/https://developer.hpe.com/mambabyte-multimodal-pathway-and-crossmae/Mon, 29 Jan 2024 15:59:48 GMT<p>External blog post</p>https://developer.hpe.com/how-to-backup-and-restore-stateful-applications-on-kubernetes-using-kasten-k10-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/how-to-backup-and-restore-stateful-applications-on-kubernetes-using-kasten-k10-in-hpe-greenlake-for-private-cloud-enterprise/Fri, 26 Jan 2024 10:07:05 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a Kubernetes (K8s) cluster, view details about existing clusters, and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>In the blog post <a href="https://developer.hpe.com/blog/getting-started-with-volume-snapshots-on-a-kubernetes-cluster-in-hpe-greenlake-for-private-cloud-enterprise/">Getting started with volume snapshots on K8s cluster</a>, I explained how to create a volume snapshot of a persistent volume in a MySQL database instance running on a K8s cluster deployed on HPE GreenLake for Private Cloud Enterprise. In this blog post, I will show you how to backup and restore the stateful applications deployed in a K8s cluster in HPE GreenLake for Private Cloud Enterprise using Kasten K10. Kasten K10 uses the volume snapshot capability in the HPE Container Storage Interface (CSI) driver for K8s to connect to different HPE storage systems and take volume snapshots of persistent volumes in K8s. It provides a user-friendly and intuitive interface and platform for easy and reliable backup and restore of the stateful applications running in the cluster.</p> <h3>Kasten K10</h3> <p>Kasten K10 is a data management platform purpose-built for K8s that was developed by Kasten. Following Veeam's acquisition of Kasten early in 2020, Kasten K10 is often referred to as Kasten by Veeam.</p> <p>Kasten K10 has been named <a href="https://www.veeam.com/news/kasten.html">a Leader and Outperformer in GigaOm’s K8s Data Protection report for the third consecutive year</a>. It offers an easy-to-use, scalable, and secure system for K8s backup/restore, disaster recovery and mobility of K8s applications.</p> <p>Apart from direct integration with a number of storage providers, Kasten K10 supports invoking volume snapshots operations via the CSI driver for K8s. By using the volume snapshot capability in the CSI driver for K8s, Kasten K10 can access different types of storage systems that enable you to backup and restore persistent volumes of the stateful applications running on K8s.</p> <h3>HPE CSI driver for K8s</h3> <p>The CSI defines a standard interface that allows container orchestration systems, such as K8s, to access storage systems. The CSI driver for K8s is a software component that implements the CSI specification and enables K8s to communicate with external storage systems. K8s supports many CSI drivers. HPE CSI Driver for K8s is one of the CSI drivers developed by HPE that uses the CSI to perform data management operations on different HPE storage systems, such as Nimble Storage, 3PAR and Primera.</p> <p>The K8s cluster provisioned in HPE GreenLake for Private Cloud Enterprise comes with the HPE CSI driver for K8s pre-installed and configured. The details of the HPE CSI driver for K8s deployed in the cluster under the namespace <em>'hpe-storage'</em> are shown below:</p> <pre><code class="language-shell">$ kubectl get all -n hpe-storage NAME READY STATUS RESTARTS AGE pod/hpe-csi-controller-54cf448d85-g4w4c 9/9 Running 0 56d pod/hpe-csi-node-5xtdb 2/2 Running 0 56d pod/nimble-csp-74d57f9487-qxwln 1/1 Running 0 56d pod/primera3par-csp-59f5dfc499-hfghx 1/1 Running 0 56d pod/snapshot-controller-5fd799f6b5-f6k7n 1/1 Running 6 (22d ago) 56d pod/snapshot-controller-5fd799f6b5-z62dc 1/1 Running 2 (27d ago) 56d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alletra6000-csp-svc ClusterIP 10.101.79.85 &#x3C;none> 8080/TCP 56d service/alletra9000-csp-svc ClusterIP 10.97.147.230 &#x3C;none> 8080/TCP 56d service/nimble-csp-svc ClusterIP 10.110.238.43 &#x3C;none> 8080/TCP 56d service/primera3par-csp-svc ClusterIP 10.101.42.76 &#x3C;none> 8080/TCP 56d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/hpe-csi-node 1 1 1 1 1 &#x3C;none> 56d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hpe-csi-controller 1/1 1 1 56d deployment.apps/nimble-csp 1/1 1 1 56d deployment.apps/primera3par-csp 1/1 1 1 56d deployment.apps/snapshot-controller 2/2 2 2 56d NAME DESIRED CURRENT READY AGE replicaset.apps/hpe-csi-controller-54cf448d85 1 1 1 56d replicaset.apps/nimble-csp-74d57f9487 1 1 1 56d replicaset.apps/primera3par-csp-59f5dfc499 1 1 1 56d replicaset.apps/snapshot-controller-5fd799f6b5 2 2 2 56d </code></pre> <p>HPE CSI driver for K8s supports both dynamic persistent volumes and volume snapshots. The following are the <em>StorageClasses</em> and the <em>VolumeSnapshotClass</em> that are configured in the cluster:</p> <pre><code class="language-shell">$ kubectl get storageclasses NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gl-sbc-hpe csi.hpe.com Delete Immediate true 56d gl-sbp-frank-gl1-sstor01 (default) csi.hpe.com Delete Immediate true 56d hpe-hdd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d hpe-nvme-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d hpe-ssd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d $ kubectl get volumesnapshotclasses NAME DRIVER DELETIONPOLICY AGE gl-sbp-frank-gl1-sstor01 csi.hpe.com Delete 56d </code></pre> <p><a href="https://www.kasten.io/kubernetes/resources/blog/kubernetes-backup-with-hpe-csi-and-kasten-k10">The joint partnership between HPE and Veeam</a> supports HPE CSI driver for K8s and Kasten K10 as a data management solution for K8s backup and recovery. The following sections will show you how to install Kasten K10 on the cluster and how to use it with the HPE CSI driver for K8s to backup and restore the persistent volumes of the stateful applications running in the cluster using volume snapshots.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The kubectl CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The <a href="https://helm.sh/docs/intro/install/">Helm</a> CLI tool, version 3.12.1 or later</li> <li>The optional mysql CLI tool, for accessing the deployed sample MySQL database service</li> </ul> <h3>Install Kasten K10</h3> <p>Kasten K10 can be deployed on K8s like any other application and it runs in its own namespace.</p> <p>Following the <a href="https://docs.kasten.io/latest/index.html">Kasten K10 installation page</a>, Kasten K10 can be installed to the cluster with the following commands using helm:</p> <pre><code class="language-shell">$ helm repo add kasten https://charts.kasten.io/ $ helm repo update $ helm install k10 kasten/k10 --namespace=kasten-io --create-namespace NAME: k10 LAST DEPLOYED: Thu Jan 18 22:34:17 2024 NAMESPACE: kasten-io STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing Kasten\u2019s K10 Data Management Platform 6.5.2! Documentation can be found at https://docs.kasten.io/. How to access the K10 Dashboard: To establish a connection to it use the following `kubectl` command: `kubectl --namespace kasten-io port-forward service/gateway 8080:8000` The Kasten dashboard will be available at: `http://127.0.0.1:8080/k10/#/` </code></pre> <p>With the above commands, Kasten K10 is installed to the namespace <em>'kasten-io'</em> in the cluster. To validate the installation, type the following command to watch the status of all Pods. Helm installs a list of Pods to the namespace. It takes a while before all those Pods start running.</p> <pre><code class="language-shell">$ kubectl get pods -n kasten-io -w NAME READY STATUS RESTARTS AGE aggregatedapis-svc-6fc8fcf7bd-cdw8p 1/1 Running 0 15m auth-svc-6fcb76d7df-pt748 1/1 Running 0 15m catalog-svc-7c6f8b76fb-bsdqn 2/2 Running 0 15m controllermanager-svc-5fffc97d7-b5whv 1/1 Running 0 15m crypto-svc-8568584f9f-br8kn 4/4 Running 0 15m dashboardbff-svc-b58b6d8cd-gnt5n 2/2 Running 0 15m executor-svc-cb5fd4698-7zqjg 1/1 Running 0 15m executor-svc-cb5fd4698-n27d5 1/1 Running 0 15m executor-svc-cb5fd4698-rvj4v 1/1 Running 0 15m frontend-svc-6c5677595b-9tsmj 1/1 Running 0 15m gateway-54d778c955-n9wt5 1/1 Running 0 15m jobs-svc-668b76cb86-q27nk 1/1 Running 0 15m k10-grafana-889ff545b-g7px7 1/1 Running 0 15m kanister-svc-76cdb967bd-hkhql 1/1 Running 0 15m logging-svc-79599589f6-hdsp5 1/1 Running 0 15m metering-svc-55f84f7766-rsm5f 1/1 Running 0 15m prometheus-server-689ccf5f57-j9hpz 2/2 Running 0 15m state-svc-b4b996d9b-jnbrl 3/3 Running 0 15m </code></pre> <p>After all the Pods are in running states, edit the service <em>gateway</em> to change its service type from <em>ClusterIP</em> to <em>NodePort</em>. This will generate a service port and expose the service via the configured gatway host name plus the generated port.</p> <pre><code class="language-shell">$ kubectl edit svc gateway -n kasten-io … spec: selector: service: gateway sessionAffinity: None type: NodePort … service/gateway edited </code></pre> <p>Type the following command to get the <em>gateway</em> service endpoint:</p> <pre><code class="language-shell">$ kubectl get svc gateway -n kasten-io -o jsonpath={.metadata.annotations.hpecp-internal-gateway/8000} gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10021 </code></pre> <p>The Kasten K10 service dashboard can now be accessed by pointing your browser to the URL '<em><a href="http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10021/k10/#/">http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10021/k10/#/</a>'</em> :</p> <p><img src="/img/k10-login.png" alt=""></p> <p>Click <em>Accept Terms</em> after specifying your email and company name. This will land you on the Kasten K10 dashboard:</p> <p><img src="/img/k10-dashboard.png" alt=""></p> <p>Kasten K10 automatically discovers all the applications and their data across namespaces in the cluster. The K10 dashboard displays a list of applications that are mapped to namespaces. It also displays a summary of the cluster’s backup data footprint, showing <em>0.0 B</em> when accessing the dashboard for the first time.</p> <p>To use Kasten K10 with HPE CSI driver for K8s, you need to ensure the configured <em>VolumeSnapshotClass</em> in the cluster contains the K10 annotation <em><strong>k10.kasten.io/is-snapshot-class: "true"</strong></em>. Typing the following command to add this required K10 annotation to the <em>VolumeSnapshotClass</em>:</p> <pre><code class="language-shell">$ kubectl get volumesnapshotclasses NAME DRIVER DELETIONPOLICY AGE gl-sbp-frank-gl1-sstor01 csi.hpe.com Delete 69d $ kubectl annotate volumesnapshotclasses gl-sbp-frank-gl1-sstor01 k10.kasten.io/is-snapshot-class=true volumesnapshotclasses.snapshot.storage.k8s.io/gl-sbp-frank-gl1-sstor01 annotated </code></pre> <p>Whenever Kasten K10 detects volumes that were provisioned via the CSI driver deployed in the cluster, it will look for a <em>VolumeSnapshotClass</em> with this K10 annotation for the identified CSI driver and use it to create snapshots.</p> <p>Type the following command to verify the <em>VolumeSnapshotClass</em> has the required K10 annotation added:</p> <pre><code class="language-shell">$ kubectl get volumesnapshotclass gl-sbp-frank-gl1-sstor01 -o yaml -o jsonpath='{.metadata.annotations}' | jq . | grep kasten "k10.kasten.io/is-snapshot-class": "true", </code></pre> <h3>Deploy MySQL database</h3> <p>In order to show backup and restore process, a MySQL database instance from <a href="https://github.com/GuopingJia/mysql-app">my GitHub repo</a> will be deployed as a sample stateful application to the cluster.</p> <p><strong>1. Install MySQL database</strong></p> <p>MySQL database requires a persistent volume to store data. Here is the <em>PersistentVolumeClaim</em> (PVC) YAML manifest file <em>mysql-pvc.yaml</em> in the repo's <em>'base'</em> folder:</p> <pre><code class="language-shell">$ cat mysql-app/base/mysql-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc namespace: mysql labels: app: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi </code></pre> <p>This PVC file, together with other YAML manifest files in the folder <em>'base'</em>, will be used to install the MySQL database instance using <a href="https://kustomize.io/">Kustomize</a>.</p> <pre><code class="language-shell">$ tree mysql-app/base mysql-app /base ├── kustomization.yaml ├── mysql-deployment.yaml └── mysql-pvc.yaml </code></pre> <p>The file <em>kustomization.yaml</em> lists all YAML files in its resources section, together with the secret generator for MySQL password:</p> <pre><code class="language-shell">$ cat mysql-app/base/kustomization.yaml secretGenerator: - name: mysql-pass namespace: wordpress literals: - password=CfeDemo@123 resources: - mysql-deployment.yaml - mysql-pvc.yaml </code></pre> <p>Type below command to install the MySQL database to the namespace <em>'mysql'</em>:</p> <pre><code class="language-shell">$ kubectl apply -k mysql-app/base namespace/mysql created secret/mysql-pass-m62cbhd9kf created service/mysql created persistentvolumeclaim/mysql-pvc created deployment.apps/mysql created </code></pre> <p>Type the following command to check the MySQL database deployment state. The MySQL Pod should be in <em>Running</em> status:</p> <pre><code class="language-shell">$ kubectl get all -n mysql NAME READY STATUS RESTARTS AGE pod/mysql-6974b58d48-wb8g5 1/1 Running 0 14s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP None &#x3C;none> 3306/TCP 24s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 1/1 1 1 23s NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-6974b58d48 1 1 1 24s </code></pre> <p>You can check that the <em>PersistentVolume</em> (PV) and the PVC get provisioned as part of the MySQL database deployment:</p> <pre><code class="language-shell">$ kubectl get persistentvolumes NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO Delete Bound mysql/mysql-pvc $ kubectl get persistentvolumeclaims -n mysql NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pvc Bound pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO gl-sbp-frank-gl1-sstor01 9m50s </code></pre> <p><strong>2. Access MySQL database</strong></p> <p>In order to access MySQL database service using the mysql CLI, first set the port-forward of <em>service/mysql</em>:</p> <pre><code class="language-shell">$ kubectl port-forward service/mysql -n mysql 42281:3306 Forwarding from 127.0.0.1:42281 -> 3306 Forwarding from [::1]:42281 -> 3306 Handling connection for 42281 </code></pre> <p>The deployed MySQL database service can be accessed by typing the following mysql command:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0,282 sec) </code></pre> <p><strong>3. Populate MySQL database</strong></p> <p>The MySQL application repo has a <em>'test'</em> folder that contains a list of scripts for populating data records and testing the contents:</p> <pre><code class="language-shell">$ tree mysql-app/test mysql-app/test ├── employees.sql ├── load_departments.dump ├── load_dept_emp.dump ├── load_dept_manager.dump ├── load_employees.dump ├── load_salaries1.dump ├── load_salaries2.dump ├── load_salaries3.dump ├── load_titles.dump ├── show_elapsed.sql ├── test_employees_md5.sql └── test_employees_sha.sql </code></pre> <p>Type the following command to populate a sample <em>employees</em> data to the MySQL database:</p> <pre><code class="language-shell">$ cd mysql-app/test $ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 &#x3C; employees.sql INFO CREATING DATABASE STRUCTURE INFO storage engine: InnoDB INFO LOADING departments INFO LOADING employees INFO LOADING dept_emp INFO LOADING dept_manager INFO LOADING titles INFO LOADING salaries data_load_time_diff NULL </code></pre> <p>The added sample data records called <em>employees</em> can be checked and verified by running the commands shown below:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | employees | | mysql | | performance_schema | +--------------------+ 4 rows in set (0,237 sec) $ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 -t &#x3C; test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | OK | ok | | dept_emp | OK | ok | | dept_manager | OK | ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:27 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ </code></pre> <h3>Back up MySQL database</h3> <p>In order to back up the MySQL database, go to the Kasten K10 dashboard and click the <em>Applications</em>. Find the deployed MySQL database <em>'mysql'</em> from the application list and expand its menu. Then click the <em>Snapshot</em> button.</p> <p><img src="/img/k10-backup-button.png" alt=""></p> <p>Using all the default options from <strong>Snapshot <em>mysql</em></strong>, click <em>Snapshot Application</em> button:</p> <p><img src="/img/k10-backup.png" alt=""></p> <p>The snapshot of the MySQL database will be started. This takes a few seconds. When you go back to the K10 dashboard, you should see the completed <em>Backup</em> entry under <strong>Actions</strong> with protected object as <em>mysql</em>:</p> <p><img src="/img/k10-dashboard-backup.png" alt=""></p> <p>You can also check the <strong>Data Usage</strong> page to see the data used by database backups:</p> <p><img src="/img/k10-data-backup.png" alt=""></p> <p>In the cluster, after snapshot of the MySQL database, you can check that there is a <em>VolumeSnapshot</em> <em>'k10-csi-snap-ltxzrwxgp6r5pwkp'</em> created from the source PVC <em>'mysql-pvc'</em> in the namespace <em>mysql</em>, together with a <em>VolumeSnapshotContent</em> object created at cluster level. The <em>READYTOUSE</em> of the <em>VolumeSnapshot</em> should show as <em>true</em>:</p> <pre><code class="language-shell">$ kubectl get volumesnapshot -n mysql NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE k10-csi-snap-ltxzrwxgp6r5pwkp true mysql-pvc 1Gi gl-sbp-frank-gl1-sstor01 snapcontent-f3890356-d47f-4b36-a7e4-eb4c5792ec59 6d12h 6d12h $ kubectl get volumesnapshotcontents NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE snapcontent-f3890356-d47f-4b36-a7e4-eb4c5792ec59 true 1073741824 Delete csi.hpe.com gl-sbp-frank-gl1-sstor01 k10-csi-snap-ltxzrwxgp6r5pwkp mysql 6d12h </code></pre> <p>This volume snapshot can be used for MySQL database restore.</p> <h3>Restore MySQL database</h3> <p>Before showing the database restore, I will first delete a table from MySQL database to simulate a loss of data. Then, I will perform the database recovery using the Kasten K10.</p> <p><strong>1. Delete table</strong></p> <p>Delete data from the table '<em>departments'</em> by typing the following commands:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 -Demployees Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 15 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [employees]> show tables; +----------------------+ | Tables_in_employees | +----------------------+ | current_dept_emp | | departments | | dept_emp | | dept_emp_latest_date | | dept_manager | | employees | | salaries | | titles | +----------------------+ 8 rows in set (0,237 sec) MySQL [employees]> delete from departments; Query OK, 9 rows affected (1,523 sec) </code></pre> <p>If you rerun the testing script <em>test_employees_sha.sql</em>, it will show the failures of <em>CRC</em> and <em>count</em>, which indicate the loss of data in the MySQL database:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 -t &#x3C;test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 0 | | | dept_emp | 0 | | | dept_manager | 0 | | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | not ok | not ok | | dept_emp | not ok | not ok | | dept_manager | not ok | not ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:24 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | FAIL | | count | FAIL | +---------+--------+ </code></pre> <p><strong>2. Perform MySQL database restore</strong></p> <p>In order to restore the MySQL database, go to the Kasten K10 dashboard, locate the MySQL database <em>'mysql'</em> from the application list, expand the menu of <em>mysql</em>, and then click the <em>Restore</em> button:</p> <p><img src="/img/k10-restore-button.png" alt=""></p> <p>Select a restore point from the list and click it. The <strong>Restore Point</strong> page will show up:</p> <p><img src="/img/k10-restore-point.png" alt=""></p> <p>Use all the default options from <strong>Restore Point</strong> and click the <em>Restore</em> button:</p> <p><img src="/img/k10-restore.png" alt=""></p> <p>The restore of the MySQL database will be started from the selected restore point. It will take a few seconds. Go back to the Kasten K10 dashboard. You should see the completed <em>Restore</em> entry under <strong>Actions</strong> with target namespace as <em>mysql</em>:</p> <p><img src="/img/k10-dashboard-restore.png" alt=""></p> <p><strong>3. Verify MySQL database</strong></p> <p>Connect to the MySQL database service and re-run the testing script <em>test_employees_sha.sql</em>. You should see the testing script now reports everything is <em>OK</em>:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 42281 -t &#x3C; test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | OK | ok | | dept_emp | OK | ok | | dept_manager | OK | ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ +--------------+---------------+-----------+ | departments | OK | ok | | dept_emp | OK | ok | | dept_manager | OK | ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:31 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ </code></pre> <p>This indicates the MySQL database has been recovered from its backup and the MySQL database data is back!</p> <h3>Summary</h3> <p>In this blog post, I explored the functionalities of Kasten K10 and HPE CSI driver for K8s. Using the volume snapshot capability in HPE CSI driver for K8s, I demonstrated how to use Kasten K10 to backup the persistent volume of a sample MySQL database deployed in the cluster in HPE GreenLake for Private Cloud Enterprise. I then illustrated how to restore the database from the backup. Kasten K10, with its user-friendly and intuitive interface, simplifies the backup and recovery of stateful applications running in the cluster. It enhances the efficiency and reliability of data management in a K8s cluster.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/kubernetes-monitoring-using-prometheus-and-grafana-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/kubernetes-monitoring-using-prometheus-and-grafana-in-hpe-greenlake-for-private-cloud-enterprise/Thu, 25 Jan 2024 14:13:04 GMT<h3>Introduction</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a> ("containers service"), one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a Kubernetes (K8s) cluster, view details about existing clusters, and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>In this blog post, I will discuss K8s monitoring and show you how to add a monitoring stack using Prometheus and Grafana to a K8s cluster in HPE GreenLake for Private Cloud Enterprise. By setting up Prometheus as the data source and importing different dashboard templates into Grafana, various aspects of K8s, including metrics, performance, and health, can be monitored in the K8s cluster.</p> <h3>Why monitor K8s</h3> <p>Though K8s dramatically simplifies application deployment in containers and across clouds, it adds a new set of complexities for managing, securing and troubleshooting applications. Container-based applications are dynamic and designed using microservices, where the number of components increases by an order of magnitude. K8s security is self-configured, typically being specified in code through K8s yaml manifests, Helm charts, or templating tools. Properly configuring for workloads, clusters, networks, and infrastructure is crucial for averting issues and limiting the impact if a breach occurs. Dynamic provisioning via Infrastructure as Code (IaC), automated configuration management and orchestration also add to monitoring and troubleshooting complexity. As such, K8s monitoring is critical to managing application performance, service uptime, and enabling troubleshooting. Having a good monitoring tool is essential.</p> <h3>Prerequisites</h3> <p>Before starting, make sure your setup has the following:</p> <style> li { font-size: 100%; line-height: 23px; max-width: none; } </style> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>Terraform, installed using <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">these steps</a></li> <li>The kubectl CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The Helm CLI tool, version 3.12.0 or later</li> </ul> <h3>Prometheus and Grafana</h3> <p><a href="https://prometheus.io/docs/introduction/overview/">Prometheus</a> is a robust open-source monitoring and alerting tool used to collect, store, query, and provide alerts on time-series data. It employs a pull-based model to gather metrics from instrumented targets and features a powerful query language (PromQL) for data analysis. It enables developers to monitor various aspects of their systems, including metrics, performance, and health.</p> <p><a href="https://grafana.com/">Grafana</a> is a powerful data visualization and monitoring tool. It serves as the interface for developers to visualize and analyze the data collected by Prometheus. With its rich set of visualization options and customizable dashboards, Grafana empowers developers to gain real-time insights into their systems’ performance, identify trends, and detect anomalies. By leveraging Grafana’s capabilities, developers can create comprehensive visual representations of their systems’ metrics, facilitating informed decision-making and proactive system management.</p> <p>In the following sections, I will show you how to add a monitoring stack using Prometheus and Grafana to a K8s cluster deployed in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Deploy Prometheus and Grafana using Terraform</h3> <p>Prometheus and Gafana will be deployed to the K8s cluster using the <a href="https://registry.terraform.io/providers/HPE/hpegl/latest">HPE GreenLake Terraform provider <em>hpegl</em></a>, together with the <a href="https://registry.terraform.io/providers/hashicorp/helm/latest">Helm provider from Hashicorp</a>.</p> <h4>Create a Terraform config</h4> <p>Here is the terraform config file. You can refer to <a href="https://developer.hpe.com/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/">Infrastructure-as-code on HPE GreenLake using Terraform</a> for the details about HPE GreenLake Terraform provider and its usage.</p> <pre><code class="language-markdown">$ cat main.tf terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider "hpegl" { caas { api_url = var.api_url } } data "hpegl_caas_cluster" "iaccluster" { name = var.cluster_name space_id = var.space_id } provider "helm" { kubernetes { host = yamldecode(base64decode(data.hpegl_caas_cluster.iaccluster.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.iaccluster.kubeconfig)).users[0].user.token insecure = true } } resource "helm_release" "prometheus-stack" { name = "prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" chart = "prometheus" version = "23.0.0" namespace = "monitoring" create_namespace = true set { name = "server.service.type" value = "NodePort" } set { name = "prometheus-node-exporter.hostRootFsMount.enabled" value = "false" } set { name = "prometheus-node-exporter.hostNetwork" value = "false" } set { name = "prometheus-node-exporter.hostPID" value = "false" } } resource "helm_release" "grafana-dashboard" { name = "grafana-dashboard" repository = "https://grafana.github.io/helm-charts" chart = "grafana" version = "6.57.4" namespace = "monitoring" create_namespace = true set { name = "service.type" value = "NodePort" } set { name = "persistence.enabled" value = "true" } } </code></pre> <p>There are a few things worth noting in the above Terraform configuration file:</p> <style> li { font-size: 100%; line-height: 23px; max-width: none; } </style> <ul> <li>In Grafana, the persistence by default is disabled. In case Grafana Pod gets terminated for some reason, you will lose all your data. In production deployments, such as HPE GreenLake for Private Cloud Enterprise: Containers, this needs to be enabled, by setting <em>persistence.enabled</em> as <em>true</em>, to prevent any data loss.</li> <li>In Prometheus, the <em>DaemonSet</em> deployment of the node exporter tries to mount the <em>hostPath</em> volume to the container root “/”, which violates the deployed OPA (Open Policy Agent) policy to the K8s cluster for filesystem (FS) mount protections. As such, the DaemonSet deployment will never be ready and keeps showing the warning events as <em>Warning FailedCreate daemonset-controller Error creating: admission webhook "soft-validate.hpecp.hpe.com" denied the request: Hostpath ("/") referenced in volume is not valid for this namespace because of FS Mount protections.</em>. You need to disable the <em>hostRootFsMount</em>, together with <em>hostNetwork</em> and <em>hostPID</em>, to comply with the security policy in the cluster.</li> <li>Both Prometheus and Grafana services are deployed as <em>NodePort</em> service types. Those services will be mapped to the gateway host with automatically generated ports for easy access and service configuration.</li> </ul> <h4>Initialize working directory</h4> <p>With above <em>main.tf</em> config file, the working directory can be initialized by running the following command:</p> <pre><code class="language-shell"> $ terraform init Initializing the backend... Initializing provider plugins... - Finding hpe/hpegl versions matching ">= 0.2.2"... - Finding latest version of hashicorp/helm... - Installing hpe/hpegl v0.3.17... - Installed hpe/hpegl v0.3.17 (signed by a HashiCorp partner, key ID D1F277A1AC66CE3D) - Installing hashicorp/helm v2.10.1... - Installed hashicorp/helm v2.10.1 (signed by HashiCorp) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. </code></pre> <p>From the output of the command run, the provider <em>helm</em> is also installed to the Terraform working directory in addition to the HPE GreenLake Terraform provider <em>hpegl</em>.</p> <h4>Deploy Prometheus and Grafana</h4> <p>Type the following command to apply the Terraform configuration and deploy Prometheus and Grafana to the K8s cluster while responding <em>yes</em> at the prompt to confirm the operation. You may want to first try a dry run to preview the changes to your infrastructure based on the data you provide in your Terraform file by running <em>terraform plan</em>.</p> <pre><code class="language-shell"> $ terraform apply --var-file=variables.tfvars Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # helm_release.grafana-dashboard will be created + resource "helm_release" "grafana-dashboard" { + atomic = false + chart = "grafana" + cleanup_on_fail = false + create_namespace = true + dependency_update = false + disable_crd_hooks = false + disable_openapi_validation = false + disable_webhooks = false + force_update = false + id = (known after apply) + lint = false + manifest = (known after apply) + max_history = 0 + metadata = (known after apply) + name = "grafana-dashboard" + namespace = "monitoring" + pass_credentials = false + recreate_pods = false + render_subchart_notes = true + replace = false + repository = "https://grafana.github.io/helm-charts" + reset_values = false + reuse_values = false + skip_crds = false + status = "deployed" + timeout = 300 + verify = false + version = "6.57.4" + wait = true + wait_for_jobs = false + set { + name = "persistence.enabled" + value = "true" } + set { + name = "service.type" + value = "NodePort" } } # helm_release.prometheus-stack will be created + resource "helm_release" "prometheus-stack" { + atomic = false + chart = "prometheus" + cleanup_on_fail = false + create_namespace = true + dependency_update = false + disable_crd_hooks = false + disable_openapi_validation = false + disable_webhooks = false + force_update = false + id = (known after apply) + lint = false + manifest = (known after apply) + max_history = 0 + metadata = (known after apply) + name = "prometheus-stack" + namespace = "monitoring" + pass_credentials = false + recreate_pods = false + render_subchart_notes = true + replace = false + repository = "https://prometheus-community.github.io/helm-charts" + reset_values = false + reuse_values = false + skip_crds = false + status = "deployed" + timeout = 300 + verify = false + version = "23.0.0" + wait = true + wait_for_jobs = false + set { + name = "prometheus-node-exporter.hostNetwork" + value = "false" } + set { + name = "prometheus-node-exporter.hostPID" + value = "false" } + set { + name = "prometheus-node-exporter.hostRootFsMount.enabled" + value = "false" } + set { + name = "server.service.type" + value = "NodePort" } } Plan: 2 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes helm_release.grafana-dashboard: Creating... helm_release.prometheus-stack: Creating... helm_release.grafana-dashboard: Still creating... [10s elapsed] helm_release.prometheus-stack: Still creating... [10s elapsed] helm_release.grafana-dashboard: Still creating... [20s elapsed] helm_release.prometheus-stack: Still creating... [20s elapsed] helm_release.grafana-dashboard: Still creating... [30s elapsed] helm_release.prometheus-stack: Still creating... [30s elapsed] helm_release.grafana-dashboard: Still creating... [40s elapsed] helm_release.grafana-dashboard: Creation complete after 43s [id=grafana-dashboard] helm_release.prometheus-stack: Still creating... [40s elapsed] helm_release.prometheus-stack: Still creating... [50s elapsed] helm_release.prometheus-stack: Still creating... [1m0s elapsed] helm_release.prometheus-stack: Still creating... [1m10s elapsed] helm_release.prometheus-stack: Creation complete after 1m18s [id=prometheus-stack] Apply complete! Resources: 2 added, 0 changed, 0 destroyed. </code></pre> <h4>Check Prometheus and Grafana</h4> <p>After a few minutes of Terraform running, both Prometheus and Grafana will be deployed in the K8s cluster to the namespace <em>monitoring</em>.</p> <p>Type the following command to check the deployed monitoring resources. All the Pods should show that they are in <em>Running</em> states.</p> <pre><code class="language-markdown">$ kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/grafana-dashboard-5674bcd6d4-zh8zk 1/1 Running 0 4d17h pod/prometheus-stack-alertmanager-0 1/1 Running 0 4d17h pod/prometheus-stack-kube-state-metrics-6fb8684695-r7zp6 1/1 Running 0 4d17h pod/prometheus-stack-prometheus-node-exporter-bcmt2 1/1 Running 0 4d17h pod/prometheus-stack-prometheus-node-exporter-hgr2x 1/1 Running 0 4d17h pod/prometheus-stack-prometheus-pushgateway-559cc996d5-jrjdn 1/1 Running 0 4d17h pod/prometheus-stack-server-7646574d75-zswws 2/2 Running 0 4d17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/grafana-dashboard NodePort 10.97.7.23 &#x3C;none> 80:30488/TCP 4d17h service/prometheus-stack-alertmanager ClusterIP 10.101.171.170 &#x3C;none> 9093/TCP 4d17h service/prometheus-stack-alertmanager-headless ClusterIP None &#x3C;none> 9093/TCP 4d17h service/prometheus-stack-kube-state-metrics ClusterIP 10.98.134.240 &#x3C;none> 8080/TCP 4d17h service/prometheus-stack-prometheus-node-exporter ClusterIP 10.104.44.251 &#x3C;none> 9100/TCP 4d17h service/prometheus-stack-prometheus-pushgateway ClusterIP 10.104.146.18 &#x3C;none> 9091/TCP 4d17h service/prometheus-stack-server NodePort 10.106.152.143 &#x3C;none> 80:30245/TCP 4d17h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-stack-prometheus-node-exporter 2 2 2 2 2 kubernetes.io/os=linux 4d17h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/grafana-dashboard 1/1 1 1 4d17h deployment.apps/prometheus-stack-kube-state-metrics 1/1 1 1 4d17h deployment.apps/prometheus-stack-prometheus-pushgateway 1/1 1 1 4d17h deployment.apps/prometheus-stack-server 1/1 1 1 4d17h NAME DESIRED CURRENT READY AGE replicaset.apps/grafana-dashboard-5674bcd6d4 1 1 1 4d17h replicaset.apps/prometheus-stack-kube-state-metrics-6fb8684695 1 1 1 4d17h replicaset.apps/prometheus-stack-prometheus-pushgateway-559cc996d5 1 1 1 4d17h replicaset.apps/prometheus-stack-server-7646574d75 1 1 1 4d17h NAME READY AGE statefulset.apps/prometheus-stack-alertmanager 1/1 4d17h </code></pre> <p>Type the <em>helm list</em> command to show both Prometheus and Grafana helm charts and versions that are deployed to the <em>monitoring</em> namespace in the cluster:</p> <pre><code class="language-markdown">$ helm list -n monitoring NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION grafana-dashboard monitoring 1 2023-11-22 15:28:07.986364628 +0100 CET deployed grafana-6.57.4 9.5.5 prometheus-stack monitoring 1 2023-11-22 15:28:13.290386574 +0100 CET deployed prometheus-23.0.0 v2.45.0 </code></pre> <h3>Set up Prometheus and Grafana for K8s monitoring</h3> <h4>Access Prometheus</h4> <p>Prometheus can be accessed by pointing your browser to the URL <em><a href="http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10015">http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10015</a></em>, extracted by using the following command:</p> <pre><code class="language-markdown">$ kubectl get service/prometheus-stack-server -n monitoring -o jsonpath='{.metadata.annotations.hpecp-internal-gateway/80}' gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10015 </code></pre> <p>You can execute the query in Prometheus by using some metrics, e.g., <em>node_procs_running</em>:</p> <p><img src="/img/prometheus.png" alt=""></p> <h4>Access Grafana</h4> <p>Grafana can be accessed by pointing your browser to the URL <em><a href="http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10016">http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10016</a></em>. The URL and the admin password can be extracted by using the following commands:</p> <pre><code class="language-markdown">$ kubectl get service/grafana-dashboard -n monitoring -o jsonpath='{.metadata.annotations.hpecp-internal-gateway/80}' gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10016 $ kubectl get secrets -n monitoring grafana-dashboard -o jsonpath='{.data.admin-password}' | base64 -d cs3O6LF2H9m0jLrgdR8UXplmZG22d9Co9WbnJNzx </code></pre> <p><img src="/img/grafana.png" alt=""></p> <h4>Configure Grafana data sources</h4> <p>From the Grafana Administration page, Prometheus can be configured as the data source by specifying the HTTP URL as <em><a href="http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10015/">http://gl-tor-upc-cp-gw-node1.customer.yyz.gl-hpe.local:10015/</a></em>:</p> <p><img src="/img/grafana-datasources.png" alt=""></p> <h4>Import Grafana dashboards</h4> <p>From <a href="https://grafana.com/grafana/dashboards/">Grafana Labs</a>, there is a list of Grafana dashboard templates you can download and then import as monitoring dashboards to the Grafana.</p> <p><img src="/img/grafana-dashboard-import.png" alt=""></p> <p>Here is the imported dashboard for <em>K8s cluster monitoring (via Prometheus)</em>. It shows overall cluster CPU / Memory / Filesystem usage.</p> <p><img src="/img/grafana-cluster-monitoring.png" alt=""></p> <p>Here is another imported dashboard for <em>K8s Pod metrics</em>. It shows individual Pod CPU / memory usage.</p> <p><img src="/img/grafana-pod-metrics.png" alt=""></p> <h3>Summary</h3> <p>This blog post described the detailed process to deploy and set up Prometheus and Grafana as a monitoring stack in a K8s cluster in HPE GreenLake for Private Cloud Enterprise. Prometheus excels at collecting and storing time-series data, enabling developers to monitor various aspects of K8s, including metrics, performance, and health. Grafana complements Prometheus by providing developers with intuitive dashboards and visualizations, enabling them to gain meaningful insights into K8s performance and behavior. Integration of Prometheus and Grafana by deploying them in the K8s cluster adds a monitoring stack. It empowers users to gain a deep understanding of the cluster’s internal states and behaviors, enabling them to identify potential issues, optimize performance and enhance overall reliability.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/hpe-greenlake-edge-to-cloud-platform-scripting-fundamentals/https://developer.hpe.com/hpe-greenlake-edge-to-cloud-platform-scripting-fundamentals/Wed, 24 Jan 2024 09:51:57 GMT<style> ul li{ font-size:27px; } </style> <style> ol li{ font-size:27px; } </style> <h2>What are the HPE GreenLake cloud APIs  </h2> <p>The foundational APIs for common HPE GreenLake cloud services allow IT administrators and IT operators to programmatically operate and manage users and resources in an HPE GreenLake cloud workspace.   </p> <p>This set of APIs for common platform services includes APIs for workspace management, identity and access management, device and subscription, locations, audit logs, and wellness.   </p> <blockquote> <blockquote> <p><em>Note: The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake cloud documentation</a> for these APIs leverages OpenAPI specifications and associated reference material. The documentation provides a complete explanation of the operations supported by these APIs for common HPE GreenLake cloud services, as well as sample requests and responses.</em>   </p> </blockquote> </blockquote> <p> The following blog posts are an excellent way to learn more about the APIs using Postman. </p> <ul> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-1-introduction-to-the-apis/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 1: Introduction to the APIs</a></li> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-2-configuring-and-managing-a-workspace/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 2: Configuring and managing a workspace</a></li> <li><a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">Get started with the foundational APIs for the HPE GreenLake edge-to-cloud platform – Part 3: Tracking activities and monitoring health</a></li> </ul> <p>In this blog post here, I will explore the usage of the HPE GreenLake cloud APIs one step further by using the APIs to build automation scripts or custom integrations. To develop my script, I will use bash, python and PowerShell. </p> <h2>Let’s pick a use case</h2> <p>Let’s say I’d like to check what is in my audit log at regular intervals in order to keep an eye on my HPE GreenLake workspace. The following graphics explain what I will be doing: </p> <p><img src="/img/don-picture.png" alt="Figure 1: Illustrating the interactions made between workspace users and the HPE GreenLake cloud" title="Figure 1: Illustrating the interactions made between workspace users and the HPE GreenLake cloud"></p> <blockquote> <p>Figure 1: Illustrating the interactions made between workspace users and the HPE GreenLake cloud</p> </blockquote> <p>For reference, I can also check the content of this audit log in the HPE GreenLake cloud console, under the Manage Workspace tab. </p> <p><img src="/img/auditlogui.jpg" alt="Figure 2: Audit log in HPE GreenLake cloud console" title="Figure 2: Audit log in HPE GreenLake cloud console"></p> <blockquote> <p>Figure 2: Audit log in HPE GreenLake platform console</p> </blockquote> <h2>High-level algorithm </h2> <p>Let’s look at the steps necessary to accomplish this. </p> <ol> <li>Gather details about the API access</li> <li>Get an API access/session token </li> <li>Compute date for filtering events </li> <li>Call audit log API </li> <li>Extract data and print results </li> <li>Wait a bit and go to Step 3 </li> </ol> <h2>Give me a token, my friend!</h2> <p>The HPE GreenLake cloud console provides a way to create API client credentials (up to 5 per workspace) in the form of a Client ID and a Client Secret pair, which, in turn, I am going to use to generate a session token.</p> <blockquote> <blockquote> <p><em>Note: To make REST API calls to HPE GreenLake cloud APIs, you will need to select “HPE GreenLake platform” as an option when configuring API client credentials. To learn how to create API client credentials for HPE GreenLake cloud APIs, check out the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-23E6EE78-AAB7-472C-8D16-7169938BE628.html">Configuring API client credentials</a> and <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-771F9B3A-B029-43E5-A38F-6D8D04178FAB.html">Requesting access to HPE GreenLake cloud APIs</a> in the HPE GreenLake cloud user guide.</em></p> </blockquote> </blockquote> <p>My script will prompt for these two values (<strong>client_id</strong> and <strong>client_secret</strong>) and will make sure that <strong>client_secret</strong> is never printed anywhere. Because these values are quite long, I will also test the presence of the operating system’s environment variables CLIENTID and CLIENTSECRET. If present, I will use their values and not prompt the user. </p> <p>From the HPE GreenLake cloud console, I’ve learned that the cURL command to get a session token is: </p> <pre><code class="language-markdown">curl -s --location 'https://sso.common.cloud.hpe.com/as/token.oauth2' \  --header 'Content-Type: application/x-www-form-urlencoded' \  --data-urlencode 'grant_type=client_credentials' \  --data-urlencode 'client_id='&#x3C;CLIENTID>' \  --data-urlencode 'client_secret='&#x3C;CLIENTSECRET>' </code></pre> <p>You can see this in the API section of the Manage Workspace screen of the HPE GreenLake cloud console shown below: </p> <p><img src="/img/apiaccess.jpg" alt="Figure 3: API access management page" title="Figure 3: API access management page"></p> <blockquote> <p>Figure 3: API access management page</p> </blockquote> <p><img src="/img/codesample-for-token.jpg" alt="Figure 4: cURL sample code for generating access token" title="Figure 4: cURL sample code for generating access token"></p> <blockquote> <p>Figure 4: cURL sample code for generating access token</p> </blockquote> <p>The JSON response is received from this call in the following format: </p> <pre><code class="language-json">{ "access_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjJGSmhvZ1lRMDZNazNBc2Q4UU8zU09ZVE9wayIsInBpLmF0bSI6ImRlejAifQ.eyJjbGllbnRfaWQiOiI1ZDVjMDVjMi00OGM5LTRmNzAtOWY4ZS1iMzIwZmQxNTA0NmYiLCJpc3MiOiJodHRwczovL3Nzby5jb21tb24uY2xvdWQuaHBZGUyODI1ZjIxMWVjOGE4NGZlZGViY2I0YTc1NCIsImF1dGhfc291cmNlIjoiY2NzX3Rva2VuX21hbmFnZW1lbnQiLCJwbGF0Zm9ybV9jdXN0b21lcl9pZCI6IjMwMQlkZTI4MjVmMjExZWM4YTg0ZmVkZWJjYjRhNzU0IiwiaWF0IjoxNzAyMDUxMDg1LCJhcHBsaWNhdGlvbl9pbnN0YW5jZV9pZCI6IjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsImV4cCI6MTcwMjA1ODI4NX0.ocpBLPKG5XdL1s_ndMmuySGt5S2-ngcaZDTrb3P0L_M4px-6_7JOavSOW-x_lCns1i1mrYKk6vfswgsRtVVq7HQA-NT8PCbxNGBVzeBjhf0SLYtPUsDLr8tfZgIH3-tE4KoW9frAWVOM1plJ5DL8i7xIpj33yyrQiLEb84IAq5TuLQ6KesSvgatQyKgB4dGRZ6lISqh9jeXU7ZuoO2rnFRC8wDcPlx-XNX3oGM0-ZO5U-NXckdmxhaWMETKmDxnvvqmLbr_jvDxUwZWCcbPfIYyqP_OYpCljhtAPkGbj8U4V0xF7HMBms1qazSy9ZVgfJEPwvbdRwo5iRKAxi7oFnQ",  "token_type":"Bearer",  "expires_in":7199 } </code></pre> <p>The response provides an access token of type “Bearer” with a time to live of 7200 seconds (2 hours). You should renew a token before expiration, but for the purposes of this blog, I will just check and terminate cleanly if it happens. </p> <blockquote> <blockquote> <p><em>Note: The token is returned as a standard JWT (JSON Web Token) described by <a href="https://datatracker.ietf.org/doc/html/rfc7519">RFC 7519</a>. You can dissect the content of your token using <a href="https://jwt.io/">https://jwt.io/</a>. Part of the data provided in the content is the date of expiration.</em></p> </blockquote> </blockquote> <h2>Querying the audit log </h2> <p>According to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/audit-logs/public/">API Reference documentation</a> for the Audit Log service, I can query the log using:</p> <pre><code class="language-markdown">GET /audit-log/v1/logs </code></pre> <p>I can also see from the documentation, that I can use a filter to keep only logs after a certain date using the following parameter: </p> <pre><code class="language-markdown">GET /audit-log/v1/logs?filter=createdAt ge '2023-07-24T04:21:22.00Z' </code></pre> <blockquote> <blockquote> <p><em>Note: the format of the date used by the API, which is <a href="https://www.iso.org/standard/70908.html">ISO 8601</a> of the form: YYYY-MM-DDTHH:MM:SS.ss-/+FF:ff. For example: '2023-07-24T04:21:22.00Z' for 4:21AM on the 24th of July, 2023 in UTC (Z=Zero Meridian)</em> </p> </blockquote> </blockquote> <p>This call needs an <strong>Authorization</strong> header which contains the access_token preceded with the string “Bearer ”. It is also a best practice to provide an <strong>Accept</strong> header to specify that a response in JSON (application/json) is expected, although this has become the default nowadays. </p> <p>The JSON response received from this API call should be in the form of: </p> <pre><code class="language-json">{  "items":[],  "count": 0,  "offset": 0,  "total": 0,  "remainingRecords": true  } </code></pre> <p>In this response, <strong>count</strong> provides the size of <strong>items</strong>, the returned array of items and <strong>total</strong>, the total number of existing items. If <strong>total</strong> is greater than <strong>count</strong>, I would need to call the same API multiple times, specifying an <strong>offset</strong> parameter to get the next batch, until <strong>total</strong> is reached or <strong>remainingRecords</strong> is false. </p> <p><strong>Items</strong> is an array of audit items, with a single audit item being defined as shown below: </p> <pre><code class="language-json">{  "id": "string",  "type": "/audit-log/logs",  "application":     {  "id": "string"  },  "region": "string",  "user":     {  "username": "string"  },  "category": "string",  "description": "string",  "workspace":   {  "id": "string",  "workspaceName": "string"  },  "createdAt": "2019-08-24T14:15:22Z",  "updatedAt": "2019-08-24T14:15:22Z",  "generation": 0,  "additionalInfo": { },  "hasDetails": true  } </code></pre> <p>To keep it simple and human readable, in my scripts, I will only display: </p> <ul> <li>user.username </li> <li>description </li> <li>createdAt </li> <li>additionalInfo.ipAddress</li> </ul> <h2>Putting it all together in bash </h2> <p>Let’s first try to assemble a bash script. In the next sections, I will show you how to achieve the same using PowerShell and Python. </p> <h3>Step 1: Gather details about the API Access </h3> <p>The first section of the script needs to take care of collecting the CLIENTID and CLIENTSECRET, checking first for environment variables, and prompting the user if no environment variables are set. Reading the CLIENTSECRET shall be done in a secure way to avoid displaying it. </p> <pre><code class="language-shell">if [[ -z "${CLIENTID}" ]]; then    read -p "Enter your HPE GreenLake Client ID: " client_id  else    client_id="${CLIENTID}"  fi  if [[ -z "${CLIENTSECRET}" ]]; then  client_secret=""  pass_var="Enter your HPE GreenLake Client Secret: "        while IFS= read -p "$pass_var" -r -s -n 1 letter  do      if [[ $letter == $'\0' ]]             then          break      fi      client_secret="${client_secret}$letter"           pass_var="*"              done  else    client_secret="${CLIENTSECRET}"  fi </code></pre> <h3>Step 2: Get a session token </h3> <pre><code class="language-shell">access_token="Bearer "`curl -s --location 'https://sso.common.cloud.hpe.com/as/token.oauth2' \  --header 'Content-Type: application/x-www-form-urlencoded' \  --data-urlencode 'grant_type=client_credentials' \  --data-urlencode 'client_id='$client_id'' \  --data-urlencode 'client_secret='$client_secret''  | jq .access_token | xargs` </code></pre> <h3>Step 3: Compute date for filtering events </h3> <p>I can start this infinite loop and each time, I will collect the audit logs that were generated during the last minute. To do so, I will need to compute the time now and subtract 1 minute. </p> <pre><code class="language-shell">for (( ; ; ))  do    d=`date -v "-1M" -u +"%Y-%m-%dT%H:%M:%S.00Z"`    echo Last check at \(UTC\): $d    echo '---------------------' </code></pre> <h3>Step 4: Call audit log API </h3> <p>I can now call the API with the right authorization header and set the filter parameter, <strong>startTime</strong> greater than the computed date: </p> <pre><code class="language-shell">http_response=$(curl -s -o out.json -w "%{http_code}" --location "https://global.api.greenlake.hpe.com/audit-log/v1/logs?filter=startTime%20ge%20'$d'" \  --header 'Accept: application/json' \  --header "Authorization: $access_token") </code></pre> <h3>Step 5: Extract data and print results </h3> <p>I need to check that the call returned an HTTP status code of 200 (Success) and then use <a href="https://jqlang.github.io/jq/download/">jq</a> to display the selected fields: </p> <pre><code class="language-shell">  if [ "$http_response" != "200" ]; then        echo "Error calling the API or token has expired!"        exit $http_response    else        cat out.json | jq '.items[] | { createdAt: .createdAt, username: .user.username, description: .description, ipAddress: .additionalInfo.ipAddress}'    fi </code></pre> <h3>Step 6: Wait a bit and go to Step 3 </h3> <p>I decided that a 1mn was a good interval, so I used the sleep command to wait 60 seconds and go back to the beginning of the infinite loop (Step 3). </p> <pre><code class="language-shell"> sleep 60 done </code></pre> <h3>Running the bash code </h3> <p>When the code is invoked in bash (tested on MacOS and Ubuntu), I can see that, every minute, the script displays time and audit logs (if any have occurred): </p> <pre><code class="language-shell">$ ./spy_workspace.sh  Last check at (UTC): 2023-12-18T09:05:53.00Z  ---------------------  {    "createdAt": "2023-12-18T09:06:11.000000Z",    "username": "&#x3C;usename>"    "description": "Loading workspace 3009de2825f211ec8a84fedebcb4a754 for user &#x3C;username>"   "ipAddress": "&#x3C;ip address>"  }  {    "createdAt": "2023-12-18T09:05:55.000000Z",    "username": "&#x3C;usename>",    "description": "User &#x3C;usename> logged in via ping mode.",    "ipAddress": "&#x3C;ip address>"  }  Last check at (UTC): 2023-12-18T09:06:54.00Z  ---------------------  Last check at (UTC): 2023-12-18T09:07:55.00Z  --------------------- </code></pre> <blockquote> <blockquote> <p><em>Note: The audit log API returns logs in LIFO (Last In First Out) mode. This is great for a GUI interface; however, it makes things a little more complicated for CLI and scripts. Sorting the logs is outside the scope of the blog post.</em></p> </blockquote> </blockquote> <p>When the token expires after 2 hours, I can catch the error, display a message, and exit. </p> <pre><code class="language-shell">Last check at (UTC): 2023-12-18T11:05:55.00Z  ---------------------  Error calling the API or token has expired! </code></pre> <h2>Putting it all together in PowerShell </h2> <p>Let’s now see how I could do the same (or better) using PowerShell: </p> <h3>Step 1: Gather details about the API Access </h3> <pre><code class="language-powershell">if ($Env:CLIENTID -eq $null) {      $ClientID = read-host "Enter your HPE GreenLake Client ID"   }  else {      $ClientID = $Env:CLIENTID   }  if ($Env:CLIENTSECRET -eq $null) {      $secClientSecret = read-host  "Enter your HPE GreenLake Client Secret" -AsSecureString      $bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($secClientSecret)      $ClientSecret = [System.Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr)   }  else {      $ClientSecret = $Env:CLIENTSECRET  } </code></pre> <h3>Step 2: Get a session token </h3> <pre><code class="language-powershell">$headers = @{}   $body = "grant_type=client_credentials&#x26;client_id=" + $ClientID + "&#x26;client_secret=" + $ClientSecret  # Get a Token  $headers = @{}   $headers["Content-Type"] = "application/x-www-form-urlencoded"  try {      $response = Invoke-webrequest "https://sso.common.cloud.hpe.com/as/token.oauth2" -Method POST -Headers $headers -Body $body  }  catch {      Write-Host "Error retrieving access token!"       exit  }  # Capturing API Access Token  $AccessToken = ($response.Content  | Convertfrom-Json).access_token </code></pre> <p>I can now prepare the headers for Step 4 </p> <pre><code class="language-powershell"># Headers creation  $headers = @{}   $headers["Authorization"] = "Bearer $AccessToken"  $headers["Accept"] = "application/json" </code></pre> <h3>Step 3: Compute date for filtering events </h3> <pre><code class="language-powershell">While ($true) {      $d=((Get-Date).AddMinutes(-1)).ToUniversalTime()      $sd=$d.tostring('yyyy-MM-ddTHH:mm:ss.00Z')      write-host "Last check at (UTC): " $sd      write-host "--------------------" </code></pre> <h3>Step 4: Call audit log API </h3> <p>Here, you’ll see that I can leverage exceptions that PowerShell supports: </p> <pre><code class="language-powershell">    try {          $response = Invoke-webrequest "https://global.api.greenlake.hpe.com/audit-log/v1/logs?filter=startTime%20ge%20'$sd'" -Method GET -Headers $headers       }      catch {          write-host "Error calling the API or token has expired!"          exit      } </code></pre> <h3>Step 5: Extract data and print results </h3> <pre><code class="language-powershell">$my_json=$response | ConvertFrom-Json      foreach ($i in $my_json.items){          write-host "createdAt:" $i.createdAt          write-host "username: " $i.user.username           write-host "description: " $i.description           write-host "ipAddress: " $i.additionalInfo.ipAddress          write-host "--------------"      } </code></pre> <h3>Step 6: Wait a bit and go to Step 3 </h3> <pre><code class="language-powershell">  start-sleep -Seconds 60  } </code></pre> <h3>Running the PowerShell code </h3> <p>Running the code in PowerShell (tested on Windows and MacOS) provides similar results: </p> <pre><code class="language-markdown">PS> ./spy-workspace.ps1  Last check at (UTC):  2023-12-18T15:16:59.00Z                                                            --------------------  createdAt: 12/18/2023 3:17:40 PM                                                                                 username:  &#x3C;usename>  description:  User &#x3C;usename> logged out  ipAddress:  &#x3C;ip address>  --------------  createdAt: 12/18/2023 3:17:36 PM  username:  &#x3C;usename>  description:  Platform customer 3009de2825f211ec8a84fedebcb4a754 profile Address updated  ipAddress:  &#x3C;ip address>  --------------  createdAt: 12/18/2023 3:17:17 PM  username:  &#x3C;usename>  description:  Loading workspace 3009de2825f211ec8a84fedebcb4a754 for user &#x3C;usename>  ipAddress:  &#x3C;ip address>  --------------  createdAt: 12/18/2023 3:17:11 PM  username:  &#x3C;usename>  description:  User &#x3C;usename> logged in via ping mode.  ipAddress:  &#x3C;ip address>  -------------- </code></pre> <blockquote> <blockquote> <p><em>Note: The audit log API returns logs in LIFO (Last In First Out) mode. This is great for a GUI interface; however, it makes things a little more complicated for CLI and scripts. Sorting the logs is outside the scope of the blog post.</em>   </p> </blockquote> </blockquote> <p>Here I can catch the exception when the token has expired, display a message, and stop: </p> <pre><code class="language-markdown">Last check at (UTC):  2023-12-18T17:17:02.00Z                                                                             --------------------  Error calling the API or token has expired! </code></pre> <h2>Putting it all together in Python </h2> <h3>Step 1: Gathering details about the API Access </h3> <pre><code class="language-python">from time import sleep  from oauthlib.oauth2 import BackendApplicationClient         from requests.auth import HTTPBasicAuth         from requests_oauthlib import OAuth2Session         from datetime import datetime, timedelta  import requests  import json  import os  import pwinput    client_id = os.environ.get("CLIENTID", "")  client_secret = os.environ.get("CLIENTSECRET", "")  if client_id == "":      client_id = input("Enter your HPE GreenLake Client ID: ")    if client_secret == "":      client_secret = pwinput.pwinput("Enter your HPE GreenLake Client Secret: ")  client = BackendApplicationClient(client_id)         oauth = OAuth2Session(client=client)         auth = HTTPBasicAuth(client_id,client_secret) </code></pre> <h3>Step 2: Get a session token </h3> <pre><code class="language-python">try:      token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth)         except:      print("Error retrieving access token.")      exit()  my_token = "Bearer " + token["access_token"] </code></pre> <h3>Step 3: Compute date for filtering events </h3> <pre><code class="language-python">while True:  # Get date in right format      now = datetime.utcnow() + timedelta(minutes = -1)      date = now.strftime("%Y-%m-%dT%H:%M:%S.00Z")      print("Last check at (UTC): ", date)      print('-------------------') </code></pre> <h3>Step 4: Call audit log API </h3> <pre><code class="language-python">my_headers = {          'accept': 'application/json',          'Authorization': my_token,      }      my_url = "https://global.api.greenlake.hpe.com/audit-log/v1/logs?filter=startTime%20ge%20'" + date + "'"  # Fetch audit logs since last minute      response = requests.get(url=my_url, headers=my_headers)      try:          response.raise_for_status()      except:          print("Error calling the API or token has expired!")          exit() </code></pre> <h3>Step 5: Extract data and print results </h3> <pre><code class="language-python">json = response.json()       e = 0      while (e &#x3C; json['count']):          print('createdAt: '+ json['items'][e]['createdAt'])          print('username: ' + json['items'][e]['user']['username'])          print('description: ' + json['items'][e]['description'])          try:              print('ipAddress: ' + json['items'][e]['additionalInfo']['ipAddress'])          except: {              print('ipAddress:')          }          print('-----------')          e = e + 1 </code></pre> <h3>Step 6: Wait a bit and go to Step 3 </h3> <pre><code class="language-python">    sleep(60) </code></pre> <h3>Running the Python code  </h3> <p>I can now run this script (tested on MacOS) and get the same behavior: </p> <pre><code class="language-markdown">$ python3 ./spy_workspace.py  Last check at (UTC):  2023-12-18T15:17:10.00Z  -------------------  createdAt: 2023-12-18T15:17:36.000000Z  username: &#x3C;username>  description: Platform customer 3009de2825f211ec8a84fedebcb4a754 profile Address updated  ipAddress: &#x3C;ip address>  ----------- </code></pre> <p>Similar to how it was done in the other versions, I can catch an exception calling the audit log API, display a message and terminate: </p> <pre><code class="language-markdown">Last check at (UTC):  2023-12-18T17:17:23.00Z  -------------------  Error calling the API or token has expired! </code></pre> <h2>What’s next? </h2> <p>The next step for the HPE GreenLake cloud APIs is to provide language specific SDK, which would provide better handling in PowerShell and Python, with stronger type checking and exception handling. In the meantime, I have shown you through this blog post that it is already possible to integrate with HPE GreenLake cloud using the most popular scripting languages. You can get the source code for these scripts from <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling">our community tooling repository</a>.</p> <p>If you’re interested in trying out what I just discussed, you might first want to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake cloud APIs mentioned in this blog post. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested and go! It’s as simple as that.</p> <p>If you still have any questions regarding the HPE GreenLake cloud APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/getting-started-with-volume-snapshots-on-a-kubernetes-cluster-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/getting-started-with-volume-snapshots-on-a-kubernetes-cluster-in-hpe-greenlake-for-private-cloud-enterprise/Tue, 23 Jan 2024 14:24:28 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h3>Overview</h3> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a> ("containers service"), one of the HPE GreenLake cloud services available on the HPE GreenLake for Private Cloud Enterprise, allows customers to create a Kubernetes (K8s) cluster, view details about existing clusters, and deploy containerized applications to the cluster. It provides an enterprise-grade container management service using open source K8s.</p> <p>In this blog post, I discuss first the persistent volumes and volume snapshots in K8s. Then, I describe the Container Storage Interface (CSI) and HPE CSI driver for K8s in HPE GreenLake for Private Cloud Enterprise. With a MySQL database instance deployed as a sample stateful application using persistent volume in the cluster, I show the detailed steps u sed to create a volume snapshot of the database as a backup using HPE CSI driver for K8s. Finally, I demonstrate how to restore the MySQL database using the created volume snapshot.</p> <h3>Persistent volumes and volume snapshots</h3> <p>In K8s, a persistent volume (PV) is a piece of storage in the cluster that has been provisioned, either statically by an administrator or dynamically using <em>StorageClasses</em>. It provides a way for data to persist beyond the lifecycle of individual Pods. PV provides the necessary data persistence for stateful applications, ensuring that they function correctly even in the event of Pod or node failures. It's a key component in managing storage in K8s. As such, backing up PVs has become a critical aspect of managing stateful applications in K8s.</p> <p>A volume snapshot is a copy of the data stored in a PV in K8s at a specific point in time. It provides the ability to create a snapshot of a PV from stateful applications. A volume snapshot can be used to back up data from a PV, restore a PV from a previous state, or create a new PV from a snapshot. A volume snapshot provides K8s users with a standardized way to copy the contents of a PV at a particular point in time without creating an entirely new volume. As an example of how this is used, this functionality can enable database administrators to backup databases before performing edit or delete modifications.</p> <p>Support of volume snapshots in K8s is only available for CSI driver deployed in the cluster.</p> <h3>HPE CSI driver for K8s</h3> <p>The CSI defines a standard interface for container orchestration systems, like K8s, to expose arbitrary block and file storage systems to their containerized workloads. Support for CSI in K8s was introduced as <em>alpha</em> in its v1.9 release, and promoted to <em>beta</em> in its v1.10 release. Implementation of the CSI has been in <em>GA</em> in K8s since v1.13 release. With the adoption of CSI, the K8s volume layer becomes truly extensible. Using CSI, 3rd party storage providers, such as HPE, can write and deploy plugins exposing new storage systems in K8s without ever having to touch the core K8s code. This gives K8s users more options for storage and makes the system more secure and reliable.</p> <p>A CSI driver for K8s is a plugin that allows K8s to access different types of storage systems, such as <em>Azure Disks</em>, <em>AWS EBS</em>, and <em>HPE Storage</em>, etc. HPE CSI driver for K8s is one of those CSI driver plugins that follows the K8s CSI specification and enables K8s to use various HPE storage systems, such as <em>Nimble Storage</em>, <em>3PAR</em> and <em>Primera</em>.</p> <p>As part of K8s cluster provisioning in HPE GreenLake for Private Cloud Enterprise, HPE CSI driver for K8s has been installed in the cluster. The installation consists of two components, a <em>controller</em> component and a <em>per-node</em> component.</p> <ol> <li> <p>The controller component is deployed as a <em>Deployment</em> on any node in the K8s cluster. It implements the CSI Controller service and a list of sidecar containers, such as <em>external-provisioner</em>, <em>external-attacher</em>, <em>external-snapshotter</em>, and <em>external-resizer</em>, etc. These controller sidecar containers typically interact with K8s objects, make calls to the driver’s CSI Controller service, manage K8s events and make the appropriate calls to the CSI driver.</p> </li> <li> <p>The per-node component is deployed as a <em>DaemonSet</em> on every node in the cluster. It implements the CSI Node service, together with the <em>node-driver-registrar</em> sidecar container, which registers the CSI driver to kubelet that runs on every cluster node and is responsible for making the CSI Node service calls. These calls mount and unmount the storage volume from the HPE storage system, making it available to the Pod to consume.</p> </li> </ol> <p>Details about the deployed HPE CSI driver for K8s in the cluster to its namespace <em>hpe-storage</em> are shown below:</p> <pre><code class="language-markdown">$ kubectl get all -n hpe-storage NAME READY STATUS RESTARTS AGE pod/hpe-csi-controller-54cf448d85-g4w4c 9/9 Running 0 56d pod/hpe-csi-node-5xtdb 2/2 Running 0 56d pod/nimble-csp-74d57f9487-qxwln 1/1 Running 0 56d pod/primera3par-csp-59f5dfc499-hfghx 1/1 Running 0 56d pod/snapshot-controller-5fd799f6b5-f6k7n 1/1 Running 6 (22d ago) 56d pod/snapshot-controller-5fd799f6b5-z62dc 1/1 Running 2 (27d ago) 56d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alletra6000-csp-svc ClusterIP 10.101.79.85 &#x3C;none> 8080/TCP 56d service/alletra9000-csp-svc ClusterIP 10.97.147.230 &#x3C;none> 8080/TCP 56d service/nimble-csp-svc ClusterIP 10.110.238.43 &#x3C;none> 8080/TCP 56d service/primera3par-csp-svc ClusterIP 10.101.42.76 &#x3C;none> 8080/TCP 56d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/hpe-csi-node 1 1 1 1 1 &#x3C;none> 56d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hpe-csi-controller 1/1 1 1 56d deployment.apps/nimble-csp 1/1 1 1 56d deployment.apps/primera3par-csp 1/1 1 1 56d deployment.apps/snapshot-controller 2/2 2 2 56d NAME DESIRED CURRENT READY AGE replicaset.apps/hpe-csi-controller-54cf448d85 1 1 1 56d replicaset.apps/nimble-csp-74d57f9487 1 1 1 56d replicaset.apps/primera3par-csp-59f5dfc499 1 1 1 56d replicaset.apps/snapshot-controller-5fd799f6b5 2 2 2 56d </code></pre> <p>As part of HPE CSI driver configuration, a list of <em>StorageClasses</em> is created that refers to the CSI driver name. The <em>PersistentVolumeClaim</em> (PVC) can then be created, which uses the <em>StorageClass</em> to dynamically provision a PV backed by the HPE storage systems. Apart from features such as dynamic provisioning, raw block volumes, inline ephemeral volumes, and volume encryption, HPE CSI driver implements and supports volume snapshot on a K8s cluster. As you can see in above deployment, the common snapshot controller <em>snapshot-controller</em> and a <em>VolumeSnapshotClass</em>, together with a list of snapshot <em>CustomResourceDefinitions</em> (CRDs), all get deployed and added to the cluster.</p> <p>Here is the list of <em>StorageClasses</em> and the <em>VolumeSnapshotClass</em> created in this cluster:</p> <pre><code class="language-markdown">$ kubectl get storageclasses NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gl-sbc-hpe csi.hpe.com Delete Immediate true 56d gl-sbp-frank-gl1-sstor01 (default) csi.hpe.com Delete Immediate true 56d hpe-hdd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d hpe-nvme-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d hpe-ssd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56d $ kubectl get volumesnapshotclasses NAME DRIVER DELETIONPOLICY AGE gl-sbp-frank-gl1-sstor01 csi.hpe.com Delete 56d </code></pre> <p>Now that you understand the basics, in the following sections, I will describe how to create volume snapshots of persistent volumes in K8s using the HPE CSI driver for K8s.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following:</p> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The kubectl CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>The optional mysql CLI tool, for accessing the deployed sample MySQL database service</li> </ul> <h3>Deploy MySQL database</h3> <p>Before showing the volume snapshots, a MySQL database instance from <a href="https://github.com/GuopingJia/mysql-app">my GitHub repo</a> will be deployed as a sample stateful application to the cluster.</p> <p><strong>1. Install MySQL database</strong></p> <p>MySQL database requires a persistent volume to store data. Here you can see the PVC YAML manifest file <em>mysql-pvc.yaml</em> in the repo's <em>base</em> folder:</p> <pre><code class="language-markdown"> $ cat mysql-app/base/mysql-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc namespace: mysql labels: app: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi </code></pre> <p>This PVC file, together with other YAML manifest files in the folder <em>base</em>, will be used to install a MySQL database instance using <a href="https://kustomize.io/">Kustomize</a>.</p> <pre><code class="language-markdown"> $ tree mysql-app/base mysql-app/base ├── kustomization.yaml ├── mysql-deployment.yaml └── mysql-pvc.yaml </code></pre> <p>The file <em>kustomization.yaml</em> lists all YAML files in its resources section, together with the secret generator for the MySQL password:</p> <pre><code class="language-markdown">$ cat mysql-app/base/kustomization.yaml secretGenerator: - name: mysql-pass namespace: mysql literals: - password=CfeDemo@123 resources: - mysql-deployment.yaml - mysql-pvc.yaml </code></pre> <p>Type command shown below to install the MySQL database to the namespace <em>mysql</em>:</p> <pre><code class="language-shell">$ kubectl apply -k mysql-app/base namespace/mysql created secret/mysql-pass-m62cbhd9kf created service/mysql created persistentvolumeclaim/mysql-pvc created deployment.apps/mysql created </code></pre> <p>Type the following command to check the MySQL database deployment state. The MySQL Pod should be in <em>Running</em> status:</p> <pre><code class="language-shell">$ kubectl get all -n mysql NAME READY STATUS RESTARTS AGE pod/mysql-6974b58d48-wb8g5 1/1 Running 0 14s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP None &#x3C;none> 3306/TCP 24s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 1/1 1 1 23s NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-6974b58d48 1 1 1 24s </code></pre> <p>You can check that the PVC and the PV were created as part of the MySQL database deployment:</p> <pre><code class="language-markdown">$ kubectl get persistentvolumes NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO Delete Bound mysql/mysql-pvc $ kubectl get persistentvolumeclaims -n mysql NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pvc Bound pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO gl-sbp-frank-gl1-sstor01 9m50s </code></pre> <p><strong>2. Access MySQL database</strong></p> <p>In order to access the MySQL database service using the mysql CLI, you must first set the port-forward of <em>service/mysql</em>:</p> <pre><code class="language-markdown">$ kubectl port-forward service/mysql -n mysql :3306 Forwarding from 127.0.0.1:41797 -> 3306 Forwarding from [::1]:41797 -> 3306 </code></pre> <p>The deployed MySQL database service can be accessed by typing the following mysql command:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0,237 sec) </code></pre> <p><strong>3. Populate MySQL database</strong></p> <p>The MySQL application repo has a <em>test</em> folder that contains a list of scripts for populating data records and testing the contents:</p> <pre><code class="language-shell">$ tree mysql-app/test mysql-app/test ├── employees.sql ├── load_departments.dump ├── load_dept_emp.dump ├── load_dept_manager.dump ├── load_employees.dump ├── load_salaries1.dump ├── load_salaries2.dump ├── load_salaries3.dump ├── load_titles.dump ├── show_elapsed.sql ├── test_employees_md5.sql └── test_employees_sha.sql </code></pre> <p>Type the following command to populate a sample <em>employees</em> data to the MySQL database:</p> <pre><code class="language-shell">$ cd mysql-app/test $ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 &#x3C; employees.sql INFO CREATING DATABASE STRUCTURE INFO storage engine: InnoDB INFO LOADING departments INFO LOADING employees INFO LOADING dept_emp INFO LOADING dept_manager INFO LOADING titles INFO LOADING salaries data_load_time_diff NULL </code></pre> <p>The added sample data records called <em>employees</em> can be checked and verified by running the commands shown below:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | employees | | mysql | | performance_schema | +--------------------+ 4 rows in set (0,237 sec) $ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 -t &#x3C; test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | OK | ok | | dept_emp | OK | ok | | dept_manager | OK | ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:27 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ </code></pre> <h3>Create volume snapshot</h3> <p>Here is the <em>VolumeSnapshot</em> YAML manifest file that creates a volume snapshot from the source PVC <em>'mysql-pvc'</em>:</p> <pre><code class="language-markdown">$ cat volumesnapshot.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysql-snapshot namespace: mysql spec: volumeSnapshotClassName: gl-sbp-frank-gl1-sstor01 source: persistentVolumeClaimName: mysql-pvc </code></pre> <p>Type the following command to create the volume snapshot:</p> <pre><code class="language-markdown">$ kubectl apply -f volumesnapshot.yaml volumesnapshot.snapshot.storage.k8s.io/mysql-snapshot created </code></pre> <p>You can check that a <em>VolumeSnapshot</em> <em>'mysql-snapshot'</em> is created in the namespace <em>mysql</em> together with a <em>VolumeSnapshotContent</em> object created at cluster level. The <em>READYTOUSE</em> of the <em>VolumeSnapshot</em> should show as <em>true</em>:</p> <pre><code class="language-markdown">$ kubectl get volumesnapshot -n mysql NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE mysql-snapshot true mysql-pvc 1Gi gl-sbp-frank-gl1-sstor01 snapcontent-41de6346-1ba3-4ce7-9483-2ca074e476a2 2m21s 2m22s $ kubectl get volumesnapshotcontents NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE snapcontent-41de6346-1ba3-4ce7-9483-2ca074e476a2 true 1073741824 Delete csi.hpe.com gl-sbp-frank-gl1-sstor01 mysql-snapshot mysql 2m50s </code></pre> <h3>Restore MySQL database using volume snapshot</h3> <p>Before showing the database restore, I will first delete a table from MySQL database to simulate a loss of data. Then, I will perform the database recovery from the created volume snapshot.</p> <h4>Delete table</h4> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 5 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> use employees; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MySQL [employees]> show tables; +----------------------+ | Tables_in_employees | +----------------------+ | current_dept_emp | | departments | | dept_emp | | dept_emp_latest_date | | dept_manager | | employees | | salaries | | titles | +----------------------+ 8 rows in set (0,237 sec) MySQL [employees]> delete from departments; Query OK, 9 rows affected (1,523 sec) </code></pre> <p>If you rerun the testing script <em>test_employees_sha.sql</em>, it will show the failures of <em>CRC</em> and <em>count</em>, which indicate the loss of data in the MySQL database:</p> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 41797 -t &#x3C;test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 0 | | | dept_emp | 0 | | | dept_manager | 0 | | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | not ok | not ok | | dept_emp | not ok | not ok | | dept_manager | not ok | not ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:24 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | FAIL | | count | FAIL | +---------+--------+ </code></pre> <h4>Perform MySQL database restore</h4> <p><strong>1. Scale MySQL database deployment config to 0</strong></p> <p>Before starting the MySQL database restore, you first need to stop the mysql Pod by scaling the replicas in the MySQL deployment to 0:</p> <pre><code class="language-markdown">$ kubectl scale deployment.apps/mysql -n mysql --replicas=0 deployment.apps/mysql scaled </code></pre> <p>Type the following command to check that the MySQL database deployment has 0 replica:</p> <pre><code class="language-markdown">$ kubectl get all -n mysql NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP None &#x3C;none> 3306/TCP 28m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 0/0 0 0 28m NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-6974b58d48 0 0 0 28m </code></pre> <p><strong>2. Create a new PVC using volume snapshot</strong></p> <p>Here is the PVC YAML manifest file that creates a new PVC <em>mysql-pvc-restore</em> from the volume snapshot <em>mysql-snapshot</em>:</p> <pre><code class="language-shell">$ cat mysql-pvc-restore.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc-restore namespace: mysql labels: app: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi dataSource: name: mysql-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io $ kubectl apply -f mysql-pvc-restore.yaml persistentvolumeclaim/mysql-pvc-restore created </code></pre> <p>You will see the new PVC <em>mysql-pvc-restore</em>, together with its PV, is created:</p> <pre><code class="language-shell">$ kubectl get persistentvolumeclaims -n mysql NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pvc Bound pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO gl-sbp-frank-gl1-sstor01 33m mysql-pvc-restore Bound pvc-92940c36-eb1d-4de5-9c1e-57261ccbecad 1Gi RWO gl-sbp-frank-gl1-sstor01 8s $ kubectl get persistentvolumes NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-3e55e9b3-097f-4ddf-bdcb-60825a7905ec 1Gi RWO Delete Bound mysql/mysql-pvc gl-sbp-frank-gl1-sstor01 42m pvc-92940c36-eb1d-4de5-9c1e-57261ccbecad 1Gi RWO Delete Bound mysql/mysql-pvc-restore gl-sbp-frank-gl1-sstor01 8m48s </code></pre> <p><strong>3. Edit MySQL deployment config</strong></p> <p>Type the following command to edit the MySQL deployment config and change the PVC name from <em>mysql-pvc</em> to <em>mysql-pvc-restore</em>:</p> <pre><code class="language-shell">$ kubectl edit deployment.apps/mysql -n mysql … volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc-restore … deployment.apps/mysql edited </code></pre> <p><strong>4. Scale MySQL database deployment config back to 1</strong></p> <p>Start the mysql Pod by scaling the replicas in the MySQL deployment back to 1:</p> <pre><code class="language-shell">$ kubectl scale deployment.apps/mysql -n mysql --replicas=1 deployment.apps/mysql scaled $ kubectl get all -n mysql NAME READY STATUS RESTARTS AGE pod/mysql-697499cd4c-k4phg 1/1 Running 0 13s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP None &#x3C;none> 3306/TCP 36m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 1/1 1 1 36m NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-697499cd4c 1 1 1 107s replicaset.apps/mysql-6974b58d48 0 0 0 36m </code></pre> <p><strong>5. Verify MySQL database data records</strong></p> <p>You can connect to the MySQL database service and rerun the testing script. You should see the testing script now reports everything is <em>OK</em>:</p> <pre><code class="language-markdown">$ k port-forward service/mysql -n mysql :3306 Forwarding from 127.0.0.1:43959 -> 3306 Forwarding from [::1]:43959 -> 3306 </code></pre> <pre><code class="language-shell">$ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 43959 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | employees | | mysql | | performance_schema | +--------------------+ 4 rows in set (0,238 sec) $ mysql -h 127.0.0.1 -uroot -pCfeDemo@123 -P 43959 -t &#x3C;test_employees_sha.sql +----------------------+ | INFO | +----------------------+ | TESTING INSTALLATION | +----------------------+ +--------------+------------------+------------------------------------------+ | table_name | expected_records | expected_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+------------------+------------------------------------------+ | table_name | found_records | found_crc | +--------------+------------------+------------------------------------------+ | departments | 9 | 4b315afa0e35ca6649df897b958345bcb3d2b764 | | dept_emp | 331603 | d95ab9fe07df0865f592574b3b33b9c741d9fd1b | | dept_manager | 24 | 9687a7d6f93ca8847388a42a6d8d93982a841c6c | | employees | 300024 | 4d4aa689914d8fd41db7e45c2168e7dcb9697359 | | salaries | 2844047 | b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f | | titles | 443308 | d12d5f746b88f07e69b9e36675b6067abb01b60e | +--------------+------------------+------------------------------------------+ +--------------+---------------+-----------+ | table_name | records_match | crc_match | +--------------+---------------+-----------+ | departments | OK | ok | | dept_emp | OK | ok | | dept_manager | OK | ok | | employees | OK | ok | | salaries | OK | ok | | titles | OK | ok | +--------------+---------------+-----------+ +------------------+ | computation_time | +------------------+ | 00:00:29 | +------------------+ +---------+--------+ | summary | result | +---------+--------+ | CRC | OK | | count | OK | +---------+--------+ </code></pre> <p>This indicates the database restore using the volume snapshot succeeded and the MySQL database data is back!</p> <h3>Summary</h3> <p>In this blog post, I described persistent volumes, volume snapshots, and the CSI driver for K8s. Using HPE CSI driver for K8s, I demonstrated how to create a volume snapshot of a MySQL database and how to restore a database using the created volume snapshot in the cluster. The volume snapshot capability can be easily integrated with third-party tools like <a href="https://www.veeam.com/products/cloud/kubernetes-data-protection.html">Kasten K10 by Veeam</a> as an automatic backup and recovery solution. It can significantly simplify the process and enhance the robustness of data management in a K8s cluster. Feel free to take a look at my blog post <a href="https://developer.hpe.com/blog/how-to-backup-and-restore-stateful-applications-on-kubernetes-using-kasten-k10-in-hpe-greenlake-for-private-cloud-enterprise/">How to backup and restore stateful app using Kasten 10</a>.</p> <p>Please keep coming back to the <a href="https://developer.hpe.com/blog/">HPE Developer Community blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/deploying-cribl-stream-containers-on-hpe-greenlake/https://developer.hpe.com/deploying-cribl-stream-containers-on-hpe-greenlake/Wed, 17 Jan 2024 21:25:38 GMT<p>Hewlett Packard Enterprise and <a href="https://cribl.io/">Cribl</a> bring together breakthrough technology to optimize and modernize observability data management, offering new levels of performance and platform independence.</p> <p>The challenges of security and log management are only partly solved by existing software solutions. HPE and Cribl address the remaining problems of optimizing, routing, and replaying logs to provide independence from the industry’s software products in this space. HPE provides a robust way to run multiple log management software solutions and the Cribl Stream in a modern, easy-to-use, and robust platform. Together, HPE and Cribl reduce the total cost of ownership of log management systems by optimizing the software, accelerating the infrastructure, and reducing management costs.</p> <p>Cribl Stream is an observability and data streaming platform for real-time processing of logs, metrics, traces, and observability data that enables the ITops, SRE, SecOps and observability teams to collect the data they want, shape the data in the formats they need, route the data wherever they want it to go, and replay data on-demand; thereby enabling customers to observe more and spend less, to have choice and flexibility, and to provide control over their data. HPE GreenLake is a private and hybrid cloud service that delivers the benefits of public cloud to your on-premises environment.</p> <p>Cribl software can be deployed as stand alone software or run on a fully managed HPE GreenLake platform to offer further ease-of-use for organizations that want the benefits of cloud in an on-premise private cloud offering.</p> <p>Deploying Cribl Stream containers on HPE GreenLake is a simple and effective way to implement a vendor-agnostic observability pipeline. Cribl Stream containers offer a number of advantages, including agility, cost savings, security, and management simplicity. <a href="https://www.hpe.com/us/en/software/marketplace/cribl-stream.html">Cribl software</a> is available in the<a href="https://www.hpe.com/us/en/software/marketplace.html"> HPE GreenLake Marketplace</a>.</p> <p>Deploying Cribl Stream containers on HPE GreenLake offers a number of advantages, including:</p> <ul> <li><strong>Agility:</strong> Cribl Stream containers can be deployed quickly and easily on HPE GreenLake, giving you the agility to scale your observability pipeline up or down as needed.</li> <li><strong>Cost savings:</strong> Cribl Stream containers can help you reduce the cost of your observability pipeline by optimizing your data storage and processing through data reduction, data normalization and log routing.</li> <li><strong>Security:</strong> Cribl Stream containers can help you secure your data by encrypting it at rest and in transit.</li> <li><strong>Management simplicity:</strong> HPE GreenLake provides a single management console for managing your Cribl Stream containers, making it easy to keep your observability pipeline running smoothly.</li> </ul> <p><img src="/img/cribl-on-hpe-architecture.png" alt="Cribl architecture diagram" title="Cribl architecture"></p> <h4>Prerequisites</h4> <p>Before you deploy Cribl Stream containers on HPE GreenLake, you will need to:</p> <ul> <li>Have an active HPE GreenLake agreement and deployed HPE GreenLake for Private Cloud Enterprise and an account on <a href="https://common.cloud.hpe.com/">https://common.cloud.hpe.com/</a>.</li> <li>Install the HPE Ezmeral Runtime Enterprise <a href="https://docs.ezmeral.hpe.com/runtime-enterprise/56/reference/kubernetes/tenant-project-administration/Dashboard__Kubernetes_TenantProject_Administrator.html">Kubectl executable</a>.</li> <li>Create a HPE Ezmeral Runtime Enterprise <a href="https://youtu.be/HSYWa2MalF4">Kubernetes cluster</a>.</li> <li>Install the Cribl Stream <a href="https://docs.cribl.io/stream/getting-started-guide/">Kubernetes operator</a>.</li> </ul> <p>Steps to deploy Cribl Stream containers on HPE GreenLake:</p> <ol> <li>Create a Cribl Stream deployment file. This file will specify the Cribl Stream containers that you want to deploy, as well as the resources that they need.</li> <li>Deploy the Cribl Stream containers to your HPE GreenLake cluster using the Cribl Stream Kubernetes operator.</li> <li>Verify that the Cribl Stream containers are running and healthy.</li> <li>Configure Cribl Stream to collect and process your data.</li> <li>Send your data to your analysis platform of choice.</li> </ol> <h4>Example deployment file</h4> <p>The following example deployment file deploys a Cribl Stream container that collects and processes logs from a Kubernetes cluster:</p> <pre><code class="language-yaml">apiVersion: apps/v1 kind: Deployment metadata: name: cribl-stream spec: replicas: 1 selector: matchLabels: app: cribl-stream template: metadata: labels: app: cribl-stream spec: containers: - name: cribl-stream image: cribl/cribl-stream:latest ports: - containerPort: 9000 volumeMounts: - name: cribl-stream-config mountPath: /etc/cribl-stream volumes: - name: cribl-stream-config configMap: name: cribl-stream-config </code></pre> <h4>Deploying Cribl Stream using Helm Charts</h4> <p>The Cribl Stream helm charts can be found on github (<a href="https://github.com/criblio/helm-charts">https://github.com/criblio/helm-charts</a>). This assumes that the namespace is set to <code>cribl</code>.</p> <p>Log into cloud CLI or jump box and issue the following commands:</p> <pre><code class="language-shell">export KUBECONFIG=&#x3C;path_to_kube_settings> kubectl get nodes -n cribl kubectl get svc -n cribl </code></pre> <p>Label the leader node and the worker nodes:</p> <pre><code class="language-shell">kubectl label nodes &#x3C;leader_node> stream=leader kubectl label nodes &#x3C;worker_node> stream=worker </code></pre> <p>Validate by running:</p> <pre><code class="language-shell">kubectl get nodes --show-labels </code></pre> <p>Create and modify the <code>values.yaml</code> file for workers and leader nodes. For the leader nodes, create a file named <code>Leader_values.yaml</code> and modify line 97:</p> <pre><code class="language-yaml">nodeSelector: stream: leader </code></pre> <p>For the worker nodes, create a file named <code>Worker_values.yaml</code> and modify line 97:</p> <pre><code class="language-yaml">nodeSelector: stream: worker </code></pre> <p>Next, set the labels for your workers and leader node.</p> <p>To do this, you'll first need to get a list of all the nodes and the labels associated with them.</p> <pre><code class="language-shell">kubectl get nodes --show-labels </code></pre> <p>Now, identify the nodes and make sure to label the nodes according to their role for this deployment.</p> <p>Here is an example of setting the host <code>k8s-cribl-master-t497j-92m66.gl-hpe.net</code> as a leader:</p> <pre><code class="language-shell">kubectl label nodes k8s-cribl-master-t497j-92m66.gl-hpe.net stream=leader </code></pre> <p>Here is an example of setting the host <code>k8s-cribl-wor8v32g-cdjdc-8tkhn.gl-hpe.net</code> as a worker node:</p> <pre><code class="language-shell">kubectl label nodes k8s-cribl-wor8v32g-cdjdc-8tkhn.gl-hpe.net stream=worker </code></pre> <p>If you accidentally label a node and want to remove or overwrite the label, you can use this command:</p> <pre><code class="language-shell">kubectl label nodes k8s-cribl-wor8v32g-cdjdc-876nq.gl-hpe.net stream=worker --overwrite=true </code></pre> <p>Once the labels have been set, you are ready to run the helm command and deploy Cribl Stream on your environment. The first command will deploy the Cribl Leader node:</p> <pre><code class="language-shell">helm install --generate-name cribl/logstream-leader -f leader_values.yaml -n cribl </code></pre> <p>When successful, you will see output similar to what's shown below:</p> <pre><code class="language-shell">NAME: logstream-leader-1696441333 LAST DEPLOYED: Wed Oct 4 17:42:16 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None </code></pre> <p>Note that this will deploy the leader node with the parameters found in the <code>leader_values.yaml</code> file and into the namespace <code>cribl</code>.</p> <p>Next, deploy the worker nodes using the <code>worker_values.yaml</code> file into the namespace <code>cribl</code>.</p> <pre><code class="language-shell">helm install --generate-name cribl/logstream-workergroup -f workers_values.yaml </code></pre> <p>When successful, you will see a similar output like the one below:</p> <pre><code class="language-shell">NAME: logstream-workergroup-1696441592 LAST DEPLOYED: Wed Oct 4 17:46:36 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None </code></pre> <p>Now you can validate the deployment by running the following command:</p> <pre><code class="language-shell">kubectl get svc </code></pre> <p>You should see the following results:</p> <pre><code class="language-shell">NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 &#x3C;none> 443/TCP 22d logstream-leader-1696441333 LoadBalancer 10.111.152.178 &#x3C;pending> 9000:31200/TCP 9m56s logstream-leader-1696441333-internal ClusterIP 10.105.14.164 &#x3C;none> 9000/TCP,4200/TCP 9m56s logstream-workergroup-1696441592 LoadBalancer 10.102.239.137 &#x3C;pending> 10001:30942/TCP,9997:32609/TCP,10080:32174/TCP,10081:31898/TCP,5140:30771/TCP,8125:31937/TCP,9200:32134/TCP,8088:32016/TCP,10200:32528/TCP,10300:30836/TCP 5m35s </code></pre> <p>Note: the names and IP addresses will differ from the above example. To test that the deployment was successful, you can run the following command and log into your deployment using the localhost and port 9000:</p> <pre><code class="language-shell">kubectl port-forward service/logstream-leader-1696441333 9000:9000 &#x26; </code></pre> <h4>Uninstalling Cribl using Helm</h4> <p>You can uninstall the Cribl deployment for both the leader and worker nodes by running the following commands respectively:</p> <pre><code class="language-shell">helm uninstall logstream-leader-1696441333 -n default helm uninstall logstream-workergroup-1696441592 -n default </code></pre> <p>Make sure to use your leader and worker group name when uninstalling Cribl from your deployment.</p> <h4>Configuring Cribl Stream</h4> <p>Once you have <a href="https://docs.cribl.io/stream/deploy-kubernetes-leader/">deployed the Cribl Stream</a> containers, you need to configure them to collect and process your data. You can do this by editing the Cribl Stream configuration file. The Cribl Stream documentation provides detailed instructions on how to configure Cribl Stream.</p> <h4>Sending your data to your analysis platform of choice</h4> <p>Once you have configured Cribl Stream to collect and process your data, you need to send it to your analysis platform of choice. Cribl Stream supports a wide range of analysis platforms, including Elasticsearch, Splunk, and Kafka.</p> <h4>Conclusion</h4> <p>For more information on Cribl Stream, check out <a href="https://www.hpe.com/psnow/doc/a50006507enw">Optimized Enterprise Logging Solution With HPE Ezmeral And Cribl Business white paper</a>.</p> <p>For more blog posts related to HPE Ezmeral Software, keep coming back to the HPE Developer Community blog and search on HPE Ezmeral.</p>https://developer.hpe.com/how-multimodal-llms-work/https://developer.hpe.com/how-multimodal-llms-work/Wed, 17 Jan 2024 18:30:00 GMT<p>External blog</p>https://developer.hpe.com/working-with-benk-a-storage-provisioning-and-io-performance-benchmark-suite-for-kubernetes/https://developer.hpe.com/working-with-benk-a-storage-provisioning-and-io-performance-benchmark-suite-for-kubernetes/Fri, 12 Jan 2024 20:03:16 GMT<p>Recently Hewlett Packard Enterprise (HPE) published an open source benchmark suite for storage drivers capable of dynamically provisioning persistent volumes to Kubernetes. The suite is called <a href="https://github.com/hpe-storage/benk">Benk</a> and it is an acronym that plays with the word bench as in benchmark and Kubernetes. Benk is used internally at HPE for mapping performance metrics around the provisioning process itself as well as for IO performance. It’s still a bit rough around the edges and not feature complete, but it’s still a very useful tool to capture performance across large swaths of configurations with a high degree of automation and repeatability without too much user attendance.</p> <p>A few of the features include:</p> <ul> <li>Highly customizable rendering of Kubernetes resources through kustomize templates</li> <li>A simple Kubernetes batch job that manage all the provisioning, decommissioning and benchmarking</li> <li>A single configuration file per job that abstracts Kubernetes constructs and IO parameters</li> <li>Use of industry standard Flexible I/O tester (FIO) for filesystem benchmarking</li> <li>Easy-to-build-your-own output templates using Jinja2 with the rich metrics in JSON format</li> </ul> <p>Let’s walk through a practical example of configuring, running and reporting a benchmark with Benk.</p> <h1>Prerequisites</h1> <p>At the time of writing, parts of Benk can only be run on Mac and Linux due to a dependency on Bash. It will of course run in WSL on Windows. Besides cloning the <a href="https://github.com/hpe-storage/benk">GitHub repository</a> with <code>git</code>, a recent version of <code>kubectl</code> and Python 3.x needs to be installed.</p> <p>Some pro-efficiency working with scripts, JSON and Kubernetes is helpful to better understand the workflows.</p> <h1>Hello Benk!</h1> <p>The synopsis section of Benk on GitHub highlights the main steps to run the default job. It's a good measuring stick one can use to understand if the lights come on and determine if the system under test is ready for a more complex job.</p> <p>I'll walk you through each step and explain more as I go along. Descriptions are below the commands unless noted.</p> <pre><code class="language-text">git clone https://github.com/hpe-storage/benk &#x26;&#x26; cd benk </code></pre> <p>Cloning and entering the <code>benk</code> directory is now considered your home. Your logs will go into <code>logs</code>, your workload configurations are in <code>kustomize/overlays</code> and the Jinja2 templates are in <code>jinja2</code>.</p> <pre><code class="language-text">pip3 install -r requirements.txt </code></pre> <p>This will install the required Python packages for the Benk <code>outputter.py</code> script. Python is not required if you don’t intend to render any output and you just want to use Benk for load generation.</p> <pre><code class="language-text">cp kustomize/base/config-dist.env kustomize/base/config.env cp kustomize/base/storageclass-dist.yaml kustomize/base/storageclass.yaml </code></pre> <p>The two “dist” files are the base of your configuration. A <code>config.env</code> file that contains information about your storage backend that will be inserted into the <code>StorageClass</code> <code>Secret</code> reference. While the base configuration is heavily biased towards the HPE CSI Driver for Kubernetes, any storage driver that uses a <code>StorageClass</code> for dynamic provisioning for filesystems <code>Persistent Volume Claims</code> (PVC) can be used. Using the <code>storageclass.yaml</code> to configure your driver and pertinent details is at your discretion.</p> <pre><code class="language-text">kubectl create ns benk </code></pre> <p>All <code>Namespace</code> resources will be provisioned into the “benk” <code>Namespace</code>.</p> <pre><code class="language-text">kubectl apply -k kustomize/overlays/default kubectl wait -n benk --for=condition=complete job/benk </code></pre> <p>This will create the default job. Wait for it to complete. It runs a 80/20 read/write random 8K workload for 30 seconds on a single replica <code>Deployment</code> using a single <code>PVC</code>.</p> <pre><code class="language-text">kubectl logs -n benk job/benk | jq </code></pre> <p>This will render the log in colorized and indented JSON. The log is pretty substantial and intentionally left out from the blog. An example log entry from the above job is available <a href="https://github.com/hpe-storage/benk/tree/main/jinja2">on GitHub</a>.</p> <p>There’s also a tiny output template provided in the repository <code>jinja2/example-default.yaml.j2</code> that pulls some data out of the log file.</p> <pre><code class="language-text">kubectl logs -n benk job/benk | ./src/benk/outputter.py -t jinja2/example-default.yaml.j2 -l- --- report: name: Example YAML template logfile: &#x3C;stdin> jobs: - threads: 1 runtime: 49s iops: 1258 bandwidth: 10MB/s bs: 8k </code></pre> <p>Congratulations. You've just completed the three corners of running Benk. You've configured the environment and a job, ran the job successfully and distilled the results into something more easily readable by humans for our demonstration.</p> <h1>The sequencer</h1> <p>The core of Benk, for now, is to run preconfigured jobs with kustomize. It may seem like a lot of work for very little output. That’s why the repository contain two important scripts, <code>sequencer.sh</code> and <code>sequencer-cluster.sh</code>. These scripts will help you run multiple jobs and structure the output to create more meaty reports.</p> <p>First, copy the “default” kustomize directory into eight separate directories.</p> <pre><code class="language-text">for i in {1..8}; do cp -a kustomize/overlays/default kustomize/overlays/mytest-${i}; done </code></pre> <p>Now, edit each <code>config.env</code> in each directory. I use <code>vi</code> to “:wn” myself through the templates.</p> <pre><code class="language-text">vi kustomize/overlays/mytest-*/config.env </code></pre> <p>In each iteration, double <code>workloadThreads</code> so you end up with a sequence like 1, 2, 4, 8, 16, 32, 64 and 128. Would it not be useful to run these in sequence and report the results in the same template used previously to understand if the system scales to when adding more threads to the workload? (The entire demo environment runs in virtual machines on decade-old hardware, so please don’t judge.)</p> <p>Run the sequence:</p> <pre><code class="language-text">./sequencer.sh mytest- </code></pre> <p>What happens here is that the sequencer script will use globbing to find all overlays that have the prefix “mytest-”. Be mindful of how you name things to avoid unexpected jobs to be executed.</p> <p>Note that the same report can be used for all the jobs.</p> <pre><code class="language-text">./src/benk/outputter.py -l logs/run-mytest-&#x3C;unique timestamp>.log -t jinja2/example-default.yaml.j2 --- report: name: Example YAML template logfile: logs/run-mytest-20240111161937.log jobs: - threads: 1 runtime: 55s iops: 1664 bandwidth: 13MB/s bs: 8k - threads: 2 runtime: 49s iops: 3291 bandwidth: 26MB/s bs: 8k - threads: 4 runtime: 48s iops: 8044 bandwidth: 63MB/s bs: 8k - threads: 8 runtime: 53s iops: 12075 bandwidth: 94MB/s bs: 8k - threads: 16 runtime: 46s iops: 16357 bandwidth: 128MB/s bs: 8k - threads: 32 runtime: 51s iops: 17284 bandwidth: 135MB/s bs: 8k - threads: 64 runtime: 48s iops: 17489 bandwidth: 137MB/s bs: 8k - threads: 128 runtime: 54s iops: 18761 bandwidth: 147MB/s bs: 8k </code></pre> <p>Observe that it shows diminishing returns by adding more than 16 threads to this workload.</p> <h1>A/B comparison</h1> <p>A/B testing is something that has gained popularity in human interaction testing for websites. This is also a crucial testing methodology used to analyze systems performance by simply changing a single parameter and rerunning the exact same test to compare the outcomes. We’re in luck as Benk allows reporting on two log files at once with just a slightly different data structures being fed to the Jinja2 templates.</p> <p>Now, change the block size for the previous workload example. We’re going to test out if increasing the block size will increase the bandwidth for the workload.</p> <p>Change the “workloadBlockSize” to “64k” with the <code>vi</code> ceremony.</p> <pre><code class="language-text">vi kustomize/overlays/mytest-*/config.env </code></pre> <p>Re-run the sequencer.</p> <pre><code class="language-text">./sequencer.sh mytest- </code></pre> <p>You'll have to use a different template, as Jinja2 now has to deal with managing two log files at a time. The syntax for the <code>outputter.py</code> script is also slightly different.</p> <pre><code class="language-text">./src/benk/outputter.py -a logs/run-mytest-&#x3C;unique timestamp>.log -b logs/run-mytest-&#x3C;unique timestamp>.log -t jinja2/example-default-ab.md.j2 </code></pre> <p>This report template will summarize the findings in a markdown table suitable to include in a GitHub pull request or similar to illustrate a certain discovery.</p> <pre><code class="language-text">| Threads | A (MB/s) | B (MB/s) | Diff | | ------- | -------- | -------- | ---- | | 1 | 13 | 55 | 4.2x | | 2 | 26 | 106 | 4.1x | | 4 | 63 | 181 | 2.9x | | 8 | 94 | 450 | 4.8x | | 16 | 128 | 661 | 5.2x | | 32 | 135 | 748 | 5.5x | | 64 | 137 | 840 | 6.1x | | 128 | 147 | 833 | 5.7x | </code></pre> <p>Clearly, the performance increases nearly 5x across the board. We also have a discrepancy in the dataset. Which one?</p> <p>A good exercise would be to factor in latency into the report in order to understand the impact, which there will be, as performance plateaus when you add more workloads – usually as a result of higher latency. But, how high?</p> <h1>More sequencers and examples!</h1> <p>There’s also a <code>sequencer-cluster.sh</code> script that allows users to orchestrate load generation across multiple clusters attached to one or many storage systems to isolate problems with high concurrency. The possibilities are quite endless.</p> <p>You can learn more about multi-cluster scaling in <a href="https://github.com/hpe-storage/benk#multi-cluster-testing">the GitHub repository</a>.</p> <p>You'll also find more practical examples in <a href="https://github.com/hpe-storage/benk/tree/main/examples">the GitHub repository</a> stemming from real world performance testing conducted by the HPE Hybrid Cloud solutions team.</p> <h1>Summary</h1> <p>Hopefully, this blog post gets you started on ideas you’d like to build and use cases you want to explore. The possibilities are endless. Bear in mind that Benk is, by all means, provided as-is. In addition, to be fully transparent, this is a very early implementation that HPE chose to open source in order to better collaborate with customers and partners to isolate performance bottlenecks. For example, not all advertised features in <code>config.env</code> have been implemented yet and the CLI is not yet completed.</p> <p>HPE invites collaboration and accepts pull requests to Benk on GitHub. The <a href="https://github.com/hpe-storage/benk/issues/2">first issue</a> discusses a solution on how to collapse initial configuration and reporting into a single intuitive Python CLI.</p> <p>Let us know what you’re building or have questions. The team behind the tool is available on HPE Developer Community Slack in the <a href="https://hpedev.slack.com/archives/C81QZ4X62">#Kubernetes</a> channel. Sign up <a href="https://developer.hpe.com/slack-signup">here</a> and sign in at <a href="https://hpedev.slack.com">hpedev.slack.com</a>.</p>https://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-1-introduction-to-the-apis/https://developer.hpe.com/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-–-part-1-introduction-to-the-apis/Fri, 12 Jan 2024 15:33:48 GMT <style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p><strong>Editor's note:</strong> This blog post series may refer to older release of the HPE GreenLake platform APIs. For information about the current release of the HPE GreenLake service APIs, please visit the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake API catalog</a>.</p> <p>HPE’s unified management plane for hybrid cloud, the HPE GreenLake platform, provides a set of <em>common services</em> that are used by cloud services that run on top of the platform. Cloud services rely on these common services for user's authentication, authorization, devices and subscriptions management, monitoring, audit trail and more.</p> <p>The HPE GreenLake platform now provides a collection of RESTful application programming interfaces (APIs) for these foundational, common services.</p> <p>If you are looking for a quick way to discover what you can do with the HPE GreenLake platform APIs using popular tools that don’t require programming (such as <a href="https://www.postman.com/product/what-is-postman/">Postman</a>), this blog post series is for you. This series will offer you the opportunity to automate IT operations via these APIs to achieve velocity, be more agile, get consistent results, reduce costs, and scale.</p> <p>In Part 1 of this series, I will help you get started with the HPE GreenLake platform APIs by taking advantage of a Postman collection I built for you. I will describe the current set of APIs for HPE GreenLake platform. I will also show you how to obtain an OAuth access token to make subsequent secure REST API calls to the HPE GreenLake platform APIs.</p> <p>In <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-2-configuring-and-managing-a-workspace/">Part 2</a> and <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">Part 3</a> of the blog series, I will take you on a deep dive into the foundational HPE GreenLake platform APIs through a typical customer scenario in which one automates IT operations via APIs such as managing users and resources, tracking activities and monitoring the overall health of services and devices in a <em><strong>Standard Enterprise</strong></em> workspace. This type of workspace is a single-tenant environment for a single customer and organization.</p> <p>Let’s embark on this exciting journey into the HPE GreenLake platform APIs.</p> <h2>Introducing the foundational APIs for the HPE GreenLake platform</h2> <p>The foundational APIs for the HPE GreenLake platform services are designed to enable IT administrators and IT operators to automate IT operations by <strong>programmatically</strong> managing users and resources in an HPE GreenLake platform <strong>workspace</strong>.</p> <blockquote> <p><strong>Note:</strong> A workspace is an identity and access management boundary. HPE GreenLake customers can organize their users, hardware and services into one or more workspaces. Users and resources must be in a workspace to be operated according to specific user permissions.</p> </blockquote> <p>For example, the current set of APIs for common platform services allows HPE GreenLake customers and partners to <strong>programmatically</strong> add users, add devices and associated subscriptions (licenses), track users’ activities and monitor the overall health of the managed services and devices in the workspace.</p> <blockquote> <p><strong>Important note:</strong> This set of APIs for common platform services differentiates from the specific APIs for the cloud services that HPE GreenLake administrators can deploy in their workspace to operate and manage workloads and their underlying infrastructure for networking, compute, storage and data services. You can find more information about these services’ specific APIs in the HPE Developer Community portal: <a href="https://developer.hpe.com/greenlake/aruba-central/home/">HPE Aruba Networking Central</a>, <a href="https://developer.hpe.com/greenlake/hpe-greenlake-for-compute-ops-management/home/">HPE Compute Ops Management</a>, and <a href="https://developer.hpe.com/greenlake/data-services-on-the-hpe-greenlake-platform/home/">HPE GreenLake for Data Services</a>.</p> </blockquote> <p>The set of APIs for common platform services includes:</p> <ul> <li> <p><strong>Identity and Access management (IAM) services:</strong> Identity and Access management services control access to HPE GreenLake workspace. The services ensure that users are granted appropriate access rights based on their roles. IAM includes the following services:</p> <ul> <li><strong>Workspace management service:</strong> Workspace management service allows you to manage workspace information and operate tenants for a Managed Service Provider (MSP) workspace.</li> <li><strong>Identity management service:</strong> Identity management service allows you to manage the workspace users. The service allows you to invite users to join the workspace, retrieve a list of existing users in the workspace and delete users from the workspace.</li> <li><strong>API client credentials service:</strong> HPE GreenLake API Client Credentials service allows programmatic access to manage workspace API Client credentials.</li> </ul> </li> <li> <p><strong>Location management:</strong> Location management service manages service delivery information (SDI), including device location and support contact information.</p> </li> <li> <p><strong>Device inventory management:</strong> Device service maintains the inventory of all devices (networking, compute and storage devices) connected to the workspace.</p> </li> <li> <p><strong>Subscription management:</strong> Subscription management service maintains the subscriptions and licenses for cloud management of devices for networking, compute and storage, and cloud software as-a-service.</p> </li> <li> <p><strong>Service catalog management:</strong> Service Catalog service allows you to manage the workspace services and service managers that are used to operate and manage workloads and their underlying infrastructure. These services run on top of the HPE GreenLake platform.</p> </li> <li> <p><strong>Audit log management:</strong> Audit log service records the occurrence of events emitted by any device or service. These logs can also be used for auditing purposes, track user activity, investigate breaches and ensure compliance with regulatory requirements.</p> </li> <li> <p><strong>Wellness event service:</strong> Wellness service presents wellness events for several HPE services and products in the workspace. In a near future, it will also enable you to open a support ticket corresponding to a wellness event when appropriate.</p> </li> </ul> <p>These APIs conform to <a href="https://spec.openapis.org/oas/latest.html">OpenAPI specifications</a> and are <a href="https://restfulapi.net/">RESTful</a>. This makes them easy to learn, discoverable by code, and accessible with any programming language. By using OAuth protocol to authenticate and authorize API client applications, secure and time-limited access to the collection of HPE GreenLake platform service APIs are provided via an access token. The token ensures that client API requests access HPE GreenLake platform services and resources securely and according to the authorization granted to the user who created the access token.</p> <p>The REST APIs support standard HTTP request methods (GET, POST, PATCH, PUT and DELETE). An HTTP request is made by providing a single unified domain endpoint (<em><a href="https://global.api.greenlake.hpe.com">https://global.api.greenlake.hpe.com</a></em>) to the HPE GreenLake platform APIs, HTTP request method, access token and data payload. The HTTP response for these requests is returned in a JSON format.</p> <blockquote> <p><strong>Note:</strong> The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake platform documentation</a> for these APIs leverages OpenAPI specifications and associated reference documentations. The documentation provides a complete explanation of the operations supported by these APIs for common HPE GreenLake platform services, as well as sample requests and responses.</p> </blockquote> <h2>Postman</h2> <p>Postman is an API platform for building and using APIs. You can sign into your Postman account either from the <a href="https://identity.getpostman.com/login">web application</a> or from the desktop application. If you don’t have a Postman account already, you can sign up for a Postman account <a href="https://identity.getpostman.com/signup">here</a> or download the desktop application <a href="https://www.postman.com/downloads/">here</a>.</p> <h2>Preparing to use the APIs for common platform services</h2> <p>As an IT administrator, before you can work with the APIs for common HPE GreenLake platform services, you will need to:</p> <ol> <li>Create an HPE account and a company workspace for your organization. Ensure you get assigned the <em><strong>Workspace Administrator</strong></em> role in HPE GreenLake platform for your organization workspace.</li> </ol> <blockquote> <p><strong>Note:</strong> You can refer to the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=index.html">HPE GreenLake platform user guide</a> to learn how to create an HPE account, a workspace and assign roles.</p> </blockquote> <ol start="2"> <li>Generate personal API client credentials for the <em>HPE GreenLake platform</em>. The credentials consist of a <em>ClientID</em> and <em>ClientSecret</em> pair that represents the permissions granted to the user who creates the personal API client credentials. <strong>Save</strong> the <em>ClientID</em> and <em>ClientSecret</em> to a safe location. You will need the credentials to generate and refresh an expired OAuth based access token when making REST API calls. Once the token is generated or refreshed, it can be used as an <strong>authorization bearer token</strong> to make further secure REST API calls to the APIs for HPE GreenLake platform common services.</li> </ol> <blockquote> <p><strong>Note:</strong> To make REST API calls to HPE GreenLake platform APIs, you will need to select “<strong>HPE GreenLake platform</strong>” as an option when configuring personal API client credentials. To learn how to create personal API client credentials for HPE GreenLake platform APIs, check out the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-23E6EE78-AAB7-472C-8D16-7169938BE628.html">Creating a personal API client</a> and <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-771F9B3A-B029-43E5-A38F-6D8D04178FAB.html">Requesting access to HPE GreenLake platform APIs</a> in the HPE GreenLake platform user guide.</p> </blockquote> <ol start="3"> <li> <p>Gather the unique identifier of your organization workspace: Go to <strong>Manage Workspace</strong> in the <a href="https://common.cloud.hpe.com/">HPE GreenLake platform Graphical User Interface</a> to get the identifier of your workspace. <strong>Save</strong> the <em>workspace identifier</em>.</p> </li> <li> <p>Obtain the unique <em><strong>identifier</strong></em> of your services deployed in your workspace. These services are typically HPE Aruba Networking Central, Data Services, and HPE Compute Ops Management used to manage and operate your networking, storge and compute infrastructure. One method is to use your Internet browser, log in to the HPE GreenLake platform UI and launch the <strong>inspect element</strong> feature of your browser to inspect the <strong>Network</strong> activity. In your workspace, select <strong>Services</strong> and check the network activity in the inspect element. In the left-end panel, select <strong>provisions</strong>, and select <strong>Response</strong> in the Network activity panel to display the list of services provisioned in your workspace. <strong>Save</strong> the <em><strong>identifier</strong></em> (displayed as <em>application_id</em> in the <em>Response</em> tab) for each of your <em>PROVISIONED</em> services. Another method is to use the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/service-catalog/public/">Service catalog API</a> to list the services that are <em><strong>provisioned</strong></em> in your workspace. You will need the information about the provisioned services when making REST API calls to the foundational, common services for the HPE GreenLake platform.</p> </li> <li> <p>Get information (email address) for a user to invite to your workspace.</p> </li> <li> <p>Get information for a Networking device (Serial Number and MAC address), or for a Storage device (Serial Number), or for a Compute device (Serial Number and Product ID) to allow you to manage these devices from the HPE GreenLake platform workspace using the APIs. The product’s serial number and other identifying details are information you received in the product order confirmation email. You will also need to get the associated subscription keys for these devices.</p> </li> </ol> <h2>Setting the Postman collection for the HPE GreenLake platform APIs</h2> <p>As you know, one of the benefits of working within a community is the ability to take advantage of open collaboration, sharing hints, tools, and resources. Although you can build your own Postman collection by downloading the OpenAPI specification files from the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/">HPE GreenLake documentation</a> and importing them to Postman, you can take advantage of the Postman collection I built for you. The Postman collection for the <em>APIs for common HPE GreenLake platform services</em> is available in the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">HPE Developer Community tooling repository</a>. Simply download the JSON file and import it to Postman. Then set the collection variables as explained in the next section.</p> <blockquote> <p><strong>Note:</strong> As HPE will enrich the APIs for common platform services over time, I will update the Postman collection as appropriate. So, check out the link above regularly to download the latest release of the Postman collection.</p> </blockquote> <h3>Defining the HPE GreenLake platform APIs collection variables</h3> <p>The collection I built makes use of collection variables that are available throughout all the REST API requests in the collection. Select the collection and then select the <strong>Variables</strong> tab as shown below:</p> <p><img src="/img/blog-part1-collection-variables-image1.png" alt="Figure 1: HPE GreenLake platform API collections variables" title="Figure 1: HPE GreenLake platform API collections variables"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 1: HPE GreenLake platform API collections variables</span></p> </blockquote> </blockquote> <p>Define the <strong>current value</strong> of the collection variables to match your HPE GreenLake platform workspace context:</p> <ul> <li><strong>BaseUrl:</strong> This variable defines the base URL of the REST API requests. It matches the single unified domain endpoint (<em><a href="https://global.api.greenlake.hpe.com">https://global.api.greenlake.hpe.com</a></em>) to APIs for common HPE GreenLake platform services.</li> <li><strong>ClientId</strong> and <strong>ClientSecret:</strong> These variables should be set with the value of your personal Client Application API credentials you previously created using the HPE GreenLake platform GUI. These variables are used to request an OAuth access token by authenticating with the authorization server referenced in the <strong>sso_URI</strong> variable or the <strong>TokenIssuerURL</strong> variable.</li> <li><strong>sso_URI:</strong> This variable is the URI of the OAuth authorization server. If your organization has set up their own HPE GreenLake SAML Single Sign-On (SSO) authorization server to create an access token, replace the current default value with your SSO URI. Otherwise you can keep the value for this variable as currently set to <em>sso.common.cloud.hpe.com/as/token.oauth2</em>.</li> <li><strong>TokenIssuerURL:</strong> This variable is the URL of the Token Issuer you obtain when you create your personal API client credentials from the HPE GreenLake GUI for your workspace.</li> <li><strong>BearerToken:</strong> Do not edit this variable. Keep the value field empty. The collection variable <em>BearerToken</em> will be set automatically upon successful execution of the <em><strong>Generate AccessToken</strong></em> API call as explained in the next step.</li> <li><strong>Workspace ID:</strong> This variable should be set with the value of the <em>identifier</em> of your Workspace you previously saved.</li> <li><strong>Aruba_Application_Id</strong>, <strong>COM_Application_Id</strong>, and <strong>DSCC_Application_Id:</strong> These variables should be set with the value of the <em>identifier</em> of the services you deployed in your workspace to manage your infrastructure services for networking, compute and storage.</li> <li><strong>GLP_Application_Id:</strong> This variable is the <em>identifier</em> of the HPE GreenLake platform. This is always set to value “00000000-0000-0000-0000-000000000000”.</li> <li><strong>ServiceManagerName:</strong> This variable should be set with the name of a cloud service manager used to manage your infrastructure equipment for compute, storage and networking. For example "<em>Aruba Central</em>", "<em>Compute Ops Management</em>", or "<em>Data Services</em>".</li> <li><strong>RegionId:</strong> This variable should be set with the name of the region where your service managers are installed. For example "<em>eu-central</em>".</li> </ul> <p><strong>Note:</strong> Do not edit the other variables. Keep the value field empty. The collection variables will be set automatically upon successful execution of REST API calls using Postman <em><strong>Scripts</strong></em>.</p> <h2>Acquire an OAuth access token as your session bearer token</h2> <p>The APIs for common HPE GreenLake platform services use a bearer token as an authorization type to ensure that all REST API requests access authorized platform services securely. So, you first need to obtain a token from the OAuth authorization server before you can make any REST API calls to the HPE GreenLake platform services. To do so, proceed as follows:</p> <ul> <li>From your collection, generate the token using the <em><strong>Generate AccessToken</strong></em> API call or the <em><strong>Generate AccessToken with TokenIssuerURL</strong></em> from the <em><strong>Step1-Generate Token</strong></em> folder. Click the <strong>Send</strong> button.</li> <li>Verify you get a status code of 200 for a successful response with token value in the response body.</li> </ul> <p>The <em>Generate AccessToken</em> API call has defined a Post-response script in the <strong>Scripts</strong> tab (formerly known as Postman Tests script) to programmatically set the collection variable <em>BearerToken</em> as shown in the picture below. The programmatically defined token is then used to authenticate any subsequent REST API calls.</p> <p><img src="/img/blog-part1-bearertoken-testscript-image2.png" alt="Figure 2: Defining collection variables programmatically in script" title="Figure 2: Defining collection variables programmatically in script"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 2: Defining collection variables programmatically in script</span></p> </blockquote> <p><strong>Note:</strong> Access bearer tokens expire after 15 minutes for HPE GreenLake platform tokens. Run the <em>Generate AccessToken</em> API request or the <em>Generate AccessToken with TokenIssuerURL</em> API request again to refresh the token before or after it expires.</p> </blockquote> <h2>Make subsequent secure REST API calls to HPE GreenLake platform services</h2> <p>All subsequent REST API requests are authenticated by presenting the access token as the authorization bearer token to the APIs for common HPE GreenLake platform services. The services validate the access token, and if valid, serve the requests.</p> <p>As shown in the two pictures below, all REST API requests in the collection will inherit the authorization bearer token that is specified at the collection level:</p> <p><img src="/img/blog-part1-collection-level-authorization-image3.png" alt="Figure 3: Authorization type (bearer token) specified at the collection level" title="Figure 3: Authorization type (bearer token) specified at the collection level"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 3: Authorization type (bearer token) specified at the collection level</span></p> </blockquote> </blockquote> <p><img src="/img/blog-part1-collection-api-call-authorization-image4.png" alt="Figure 4: REST API request with authorization type inherited from parent collection" title="Figure 4: REST API request with authorization type inherited from parent collection"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> Figure 4: REST API request with authorization type inherited from parent collection</span></p> </blockquote> </blockquote> <p>To validate the access token, pick the next REST API call <em><strong>Get workspace information</strong></em> from the <em><strong>Step1-Generate Token</strong></em> folder. Click the <strong>Send</strong> button and verify you get a <em>status code of 200</em> for a successful response. You will get a JSON representation of your HPE GreenLake platform workspace. An example is shown below:</p> <pre><code class="language-json">{ "id": "&#x3C;Your WorkspaceId>", "type": "workspace", "generation": 26, "createdAt": "2021-10-05T15:37:45.991228", "updatedAt": "2023-12-08T08:42:31.708206", "workspaceName": "&#x3C;Your Workspace Name>", "createdBy": "&#x3C;name of person who created the workspace>" } </code></pre> <p><strong>Congratulations!</strong> You have placed your first API calls to the common HPE GreenLake platform services using Postman.</p> <h2>Summary</h2> <p>This blog post helps you get started with the HPE GreenLake platform APIs by taking advantage of Postman collection. It explains you the preparation steps you need to take to use the APIs for common platform services and walks you through the steps required to obtain an OAuth access token to make secure REST API calls to the HPE GreenLake platform APIs.</p> <p>You can get the Postman collection from the <a href="https://github.com/hpe-dev-incubator/GLP-API-Tooling/tree/main/Postman-Collections">HPE Developer Community tooling GitHub repository</a>.</p> <p>Don’t miss <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-2-configuring-and-managing-a-workspace/">Part 2</a> and <a href="https://developer.hpe.com/blog/get-started-with-the-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform-%E2%80%93-part-3-tracking-activities-and-monitoring-health/">Part 3</a> of this blog series, where you will further explore the rest of the collection to learn how you, as an IT administrator of the HPE GreenLake platform, can configure and manage workspace resources (users’ identity, devices and subscriptions) and how you can track activities within your workspace and monitor overall health of services and devices in your workspace.</p> <p>If you’re interested in trying out what I just discussed, you might want to check out one of our hands-on Workshops-on-Demand that lets you play with the HPE GreenLake APIs mentioned in this blog post. The workshops are free, available 24/7, and very easy to use. They give you a real-world experience without any risk. Check out our <a href="https://developer.hpe.com/hackshack/workshops">catalog of workshops</a>, register for the one you’re interested in and go! It’s as simple as that.</p> <p>If you still have any questions regarding the HPE GreenLake platform APIs, join the <a href="https://developer.hpe.com/slack-signup/">HPE Developer Community Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake-api</a> channel. We’re always here to help.</p>https://developer.hpe.com/2024-January-11/https://developer.hpe.com/2024-January-11/Thu, 11 Jan 2024 13:00:00 GMThttps://developer.hpe.com/introduction-to-gpu-programming-in-chapel/https://developer.hpe.com/introduction-to-gpu-programming-in-chapel/Thu, 11 Jan 2024 06:25:58 GMT<p>External blog</p>https://developer.hpe.com/determined-ai-weekly-update-6/https://developer.hpe.com/determined-ai-weekly-update-6/Tue, 09 Jan 2024 09:55:36 GMT<p>External blog</p>https://developer.hpe.com/be-an-hpe-dev-blogger/https://developer.hpe.com/be-an-hpe-dev-blogger/Thu, 04 Jan 2024 13:03:49 GMT<p>Want to participate more in the HPE Developer Community but don’t know how? Have you ever dreamt about being published? Consider writing an article for the <a href="https://developer.hpe.com/blog">HPE Developer blog!</a> Share your knowledge with like-minded developers, designers, data scientists, and IT admins all over the world. Maybe you’ve just discovered something new you’d like to share. Maybe you found an easier way to do something. Others in the community have probably encountered similar situations and would be eager to hear your tips.</p> <p>As daunting as it may feel, writing a blog post is not as hard as you think. The HPE Developer Community team has templates you can use to organize your thoughts and draft an article. You can submit your post in the way that best suits your needs. And we’re right here to help you along the way. Not confident about your writing or language skills? No worries… we even have an editor on board who can help make your article shine. Become a star blog contributor, as this post gives you all the tools you need.</p> <h2>What’s the blog process?</h2> <p>If you’d like to be published on the HPE Developer blog, it’s really very easy. Here are the basic steps:</p> <ol> <li>Draft your article either directly in the Netlify CMS or in one of the templates provided</li> <li>Submit your draft for a quick technical and editorial review</li> <li>Make the recommended changes</li> <li>Once approved, the article will be posted and promoted</li> </ol> <h3>General guidelines</h3> <p>Whenever you’re writing, it’s important to remember who your audience is. Articles placed on the <a href="https://developer.hpe.com/blog">HPE Developer blog</a> are seen by developers, data scientists, data architects, IT admins and managers worldwide. It is a highly technical audience who will discount anything that reeks of marketing. So, consider your topic and audience carefully. Simple advertisements for a product won’t be published. If you have any question as to the appropriateness of your topic, feel free to <a href="mailto:hpedev@hpe.com">reach out to us</a>.</p> <p>Is there a limit on word count? Not really. You can write a post as short as 250-500 words. You can also write a much longer post. Best industry practices indicate that an ideal blog post length is between 2,100-2,400 words. If your post is very long, the editor may suggest breaking it up into a multi-part series. This often works quite well, providing you with greater visibility in the community.</p> <p>If you need to have a post published by a certain date, please let us know in advance so we can prioritize our review appropriately.</p> <h2>There are two paths you can take</h2> <p>The HPE Developer portal is set up so you can submit your articles directly into the Netlify Content Management System (CMS) and have them reviewed by the HPE Developer Community team via GitHub. This is the preferred method for submitting your drafts. We also accept direct submissions of Microsoft Word or Google Docs files, in case you do not have a GitHub account or require this ability for other reasons. Templates and instructions on how to provide submissions via these documents are provided below.</p> <p><img src="/img/blogger-two-paths-2.jpg" alt=""></p> <p>The major difference between the two methods is that there is a greater amount of responsibility placed on the writer with the Netlify CMS method. With an external document submission, the HPE Developer Community team takes care of placing the document into the Netlify CMS. The preferred method is to use the Netlify CMS, but for those who feel they might need a bit more assistance or who don’t have a GitHub account, we offer Microsoft Word and Google Docs templates you can use to submit via email.</p> <h3>Netlify CMS method:</h3> <p>You’ll find complete instructions for entering your content into the Netlify CMS and submitting your post for review in the <a href="https://github.com/hpe-dev-incubator/hpe-dev-portal/blob/master/docs/ContributorGuide-v2.md">HPE DEV External Contributor Guide</a>. Once you have created a new blog entry, entered your content, added your tags and saved your work, your post becomes visible in the Editorial Workflow in the Drafts section. Make sure you move your post entry from the <strong>Drafts</strong> column to the <strong>In Review</strong> column in the <strong>Editorial Workflow</strong>. This will automatically open a Pull Request (PR) in the GitHub repository associated with the Netlify CMS. This indicates to the HPE DEV team that the post is ready, initiating the review process.</p> <blockquote> <p>Note: If you need to make edits after you’ve placed it in the <strong>In Review</strong> column, there is no need to move it back into the <strong>Drafts</strong> column. In fact, if you do so, it will delete the history, so please just keep it in the <strong>In Review</strong> column while you make your edits.</p> </blockquote> <p><img src="/img/blogger-workflow.jpg" alt=""></p> <h3>Microsoft Word Templates:</h3> <p>To use the Microsoft Word templates, click on the link. It will automatically download the file to your local workstation. Save it to your local drive and create your article from there. When you are ready submit your draft <a href="mailto:hpedev@hpe.com">to the editor via email</a> for review.</p> <ul> <li><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/8/HPE-DEV-BASIC-BLOG-TEMPLATE-FINAL.docx">Microsoft Word Basic Blog template</a></li> <li><a href="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/8/HPE-DEV-TECHNICAL-TUTORIAL-TEMPLATE-FINAL.docx">Microsoft Word Technical Tutorial template</a></li> </ul> <h3>Google Docs Templates:</h3> <p>To use the Google Docs templates, click on the link. It will bring you to the Google account where the document is stored. Go to <strong>File</strong> and pull down to <strong>Make a Copy</strong>. Save it to your drive and rename the file. Once you have filled in the sections, <em>share the document</em> with the <a href="mailto:denis.choukroun@hpe.com">technical review team</a> and the <a href="mailto:dale.rensing@hpe.com">content editor</a>.</p> <ul> <li><a href="https://docs.google.com/document/d/1uAHcsJxavfmC0oRoccjBFI_WmuALDWhOINATiCEoDIw/edit?usp=sharing">Google Docs Basic Blog template</a></li> <li><a href="https://docs.google.com/document/d/1bY0QL0TYgQtzjCF4JpsLbDMvPUMarIwQoVZFjQYej1Y/edit?usp=sharing">Google Docs Technical Tutorial template</a></li> </ul> <p>Remember to include hi res images as separate .jpg attachments to the email.</p> <h2>Ready to get writing?</h2> <p>Here are some tips to help you get your blog read. As you draft your article, keep in mind Aristotle’s advice on how to communicate for impact by breaking your post into three parts; the <strong>Introduction</strong>, the <strong>Body</strong>, and the <strong>Summary</strong>. You want to first introduce what you are going to cover in the post. Then, go into details on your topic, telling the reader everything you want to convey in the body. At the end, reiterate and summarize your key points.</p> <p><img src="/img/blogger-aristotle.jpg" alt=""></p> <p>But there are three other things you’ll also want to include; a <strong>Problem Statement</strong>, a <strong>Benefit Statement</strong>, and a <strong>Call to Action</strong>. The first two should be in your introduction, and the last, in your summary.</p> <p>The <strong>Problem Statement</strong> piques the interest of the reader. When the reader sees this, they often recognize an issue that’s familiar to them, which encourages further reading. The <strong>Benefit Statement</strong> almost serves as a summary of what the remainder of the article covers. It explains how the <strong>Problem Statement</strong> is addressed; the <em>why you should read this</em>, if you will.</p> <p>The <strong>Call to Action</strong> is placed at the end in your summary. This gives you the opportunity to reach out to the reader and ask them to do something. It could be to go to another web page, to connect with you on Slack or Twitter, or to remember salient points from your article.</p> <p>A few other things to keep in mind as you draft your article:</p> <ul> <li>Read through a few of the existing <a href="https://developer.hpe.com/blog">blog posts</a> first to get ideas.</li> <li>Avoid the “royal we”. Imagine you are talking to the reader directly and use “I” and “you”, instead.</li> <li>Don’t use quotation marks (“ “) for things that are not actual quotes or code requiring this sort of notation. For emphasis, use <em>italics</em> or <strong>bold.</strong></li> <li>Include images. Images are often pulled from your blog post and used as part of the social promotion for your post.</li> </ul> <h2>The review process</h2> <p>Please note that the length of the review cycle depends on the technical depth of the piece, the writer’s fluency in written English, the immediacy of the need, and current backlog.</p> <h3>Netlify CMS:</h3> <p>The review process is covered in detail in the <a href="https://github.com/hpe-dev-incubator/hpe-dev-portal/blob/master/docs/ContributorGuide-v2.md">HPE DEV External Contributor Guide</a>. But to quickly summarize:</p> <ul> <li>The HPE Developer Community team will review your markdown copy on the Netlify CMS.</li> <li>They’ll offer technical and editorial suggestions, commenting on each line directly on GitHub Pull Request.</li> <li>The writer is notified of the edits by email.</li> <li>It is up to the writer to make the suggested edits. The writer may determine that an edit is not appropriate and can comment back.</li> <li>Once agreement is established, the post will be published.</li> </ul> <h3>Microsoft Word and Google Docs Templates:</h3> <ul> <li>The submitted draft is reviewed by the team and returned to the writer with Track Changes turned on.</li> <li>The writer may accept all changes or determine if they should be accepted or rejected one by one.</li> <li>Once agreement is established, the post will be reformatted into markdown by the HPE Developer Community team and published.</li> </ul> <h2>Let’s get writing!</h2> <p>Writing articles for the HPE Developer blog can be a fun and rewarding experience. It gives you a chance to work more closely with the HPE Developer Community team and often keys us into more opportunities of working together to create other assets, like a Workshop-on-Demand or a Munch &#x26; Learn session. We have a number of bloggers who consistently come back to add more to their portfolio.</p> <p>We’ve tried to make it as easy for you to contribute as possible, offering multiple ways to create and submit your posts. If you have any questions, please feel free to reach out to us either through <a href="mailto:hpedev@hpe.com">email</a> or <a href="https://slack.hpedev.io/">Slack</a>. We’re really looking forward to seeing what you have to offer, and we bet our readers are, too!</p>https://developer.hpe.com/open-source-monitoring-at-the-hpe-customer-technology-center-böblingen/https://developer.hpe.com/open-source-monitoring-at-the-hpe-customer-technology-center-böblingen/Tue, 02 Jan 2024 15:34:10 GMT<h1>The challenge</h1> <p>The HPE Customer Technology Center (CTC) Böblingen is a customer visit center in Germany where a wide range of HPE solutions are demonstrated. The core infrastructure of the CTC is based on HPE SimpliVity. Its performance, capacity and uptime has been monitored for some time using Prometheus and Grafana. You can find the implementation of the HPE SimpliVity-Prometheus exporter described in the <a href="https://www.hpe.com/psnow/doc/a50000514enw" target="_blank">HPE SimpliVity Prometheus exporter whitepaper</a>. Over time, this monitoring environment was extended with the <a href="https://github.com/hpe-storage/array-exporter" target="_blank">HPE array exporter</a> for HPE storage arrays (HPE Alletra 6000, HPE Alletra 9000 and HPE Alletra MP).</p> <p>Over the last few summers, the CTC lab team was challenged with how to cool the growing infrastructure. The cooling capacity initially planned was no longer sufficient given the new generation servers and storage that were deployed in the CTC. The team found that the lab temperature was rising to a level that caused several incidents where HPE ProLiant servers were automatically shutting down to avoid damages. To remedy this situation, the team looked for a CTC lab temperature warning system that would be easy to implement and would provide a "heads up" of any rise in temperature in the CTC and allow us to shutdown equipment that was not urgently needed for customer demos - ensuring that the booked customer demos would still be able to run.</p> <h1>Redfish® iLO RESTful API as a solution?</h1> <p>Each HPE ProLiant server has a lot of temperature sensors spread across the system board, CPU, memory and power supplies. This detailed temperature data is available and visualized on each ILO interface in the Power &#x26; Thermal section:</p> <p><img src="/img/ilotemperature.png" alt="" title="ILO Temperature Information"></p> <p>The first sensor, 01-Inlet Ambient temperature, is measuring the inlet temperature at the front of the server and can be used as an indicator of the temperature of the overall CTC environment. Now that I have identified a usable data point an automated reading of the temperature values is needed. Since I already used the the HPE Simplivity REST API to build the HPE SimpliVity-Prometheus exporter used to monitor the core infrastructure, it was only natural that I would look into the <a href="https://developer.hpe.com/platform/ilo-restful-api/home/" target="_blank">Redfish® iLO RESTful API</a>. The Redfish® API ecosystem is an open industry-standard specification (published by the Distributed Management Task Force <a href="http://www.dmtf.org/standards/redfish" target="_blank">DMTF</a> and provides remote server provisioning, inventory and monitoring.</p> <p>A directory of the available Redfish® resources at an ILO interface can be retrieved with the following RESTful API command: </p> <pre><code class="language-http">GET {{baseUrl}}/redfish/v1/resourcedirectory </code></pre> <p>I used <a href="https://www.postman.com/" target="_blank">Postman</a> as a developer tool to retrieve and search the resource directory. The above RESTful API comand was used in Postman with a basic username/password authorization at one of the ILOs in the CTC, where the baseUrl is defined as <a href="https://iloIPaddress">https://iloIPaddress</a>. I searched in the obtained resource directory for "thermal" in order to find the correct command to retrieve thermal data:</p> <p><img src="/img/redfishresourcedirectory.png" alt="" title="Redfish resource directory (Postman output)"></p> <p>I found that I could retrieve thermal server data using the following RESTful command:</p> <pre><code class="language-http">GET /redfish/v1/chassis/1/thermal </code></pre> <p>And using this Get command with Postman confirmed that I could get the values of all temperature sensors of the server:</p> <p><img src="/img/redfishtemparture.png" alt="" title="ILO Redfish thermal information (Postman output)"></p> <h1>Building the HPE server exporter</h1> <p>Once I had the Prometheus-Grafana based monitoring established in the Customer Technology Center and knew the RESTful command required to retrieve the data I was looking for, the only missing piece was a Prometheus exporter for the HPE ILO interfaces and the according Grafana dashboards. This required me to write another Prometheus exporter - one designed for the HPE ILO interface. I decided to use Python as the programming language for this connector, because I could build on the knowledge gained when I developed the HPE SimpliVity-Prometheus exporter. </p> <p>First I had to decide which redfish library I wanted to use - the <a href="https://github.com/DMTF/python-redfish-library" target="_blank">DMTF Redfish Python Library</a> or the <a href="https://github.com/HewlettPackard/python-ilorest-library" target="_blank">HPE Python Redfish library</a> - as they cannot coexist both in a Python environment. I decided to use the DMTF Redfish Python library, since I didn't need the additional features of the HPE Python Redfish library for my use case. The retrieved data can easily made digestible for Prometheus with the prometheus_client Python library. The additional libraries, that I used are mainly for providing the input data in XML-format (etree library) and for the permanent monitoring loop (time, sys and os library) and logging (datetime library). Hence, I needed the following libraries included in my Python script:</p> <pre><code class="language-python">from lxml import etree import time from datetime import datetime from prometheus_client import Counter, Gauge, start_http_server, Info import sys import redfish import os </code></pre> <p>I decided to use one username for all ILOs that I wanted to add into the monitoring realm. Login with read only access privileges were sufficient for the Prometheus exporter user defined local on the ILOs.</p> <p><img src="/img/ilouser.png" alt="" title="ILO Prometheus user"></p> <p>Testing the initial Prometheus exporter showed that I could consolidate multiple server ILOs into a single exporter while keeping the data collection time below 30 to 60 seconds.  The ILO Prometheus exporter itself has a simple structure - I didn't want to complicate things if not absolutely necessary:</p> <p><img src="/img/prometheusconnectorstructure.png" alt="" title="ILO Prometheus Connector - Architecture"></p> <p>First, the exporter reads the input data; i.e. the ILO IP addresses and the ILO username and password. Next, the exporter retrieves the correct URLs for each monitored server and stores it in a Python directory. Afterwards, the script starts the http server and the corresponding Prometheus counters and gauges. Having completed these preparation steps, the script enters an endless loop that first captures the start time of the current iteration, second retrieves the data from the monitored server, third displays the captured data and waits until the current monitoring interval is completed. Additionally, I have added that at least once per day the input data and monitor URLs are refreshed. This allows me to change the list of monitored systems without starting and stopping the script. I just need to edit the input data file to adjust the list of monitored systems.</p> <h2>Main routine</h2> <p>As a result, the Python main routine of the ILO Prometheus exporter is as simple as shown below (I removed some of the log commands to improve the readability):</p> <pre><code class="language-python">if __name__ == "__main__": input = getServerList() # Get the monitoring URLs of the server monitor_urls = get_server_urls(input['user'], input['password'], input['server'], input['lfile']) # Start the http_server and the counters, gauges start_http_server(input['port']) c = Counter('ilorest_sample','ILO REST sample number') node = Gauge('ilorest_node','ILO Node Data',['nodename','rack','nodemetric','metricdetail']) delta = Gauge('ConnectorRuntime','Time required for last data collection in seconds') inode = Info('node','Additional Node Info',['node']) # Start the endless loop while True: t0 = time.time() start0 = t0 c.inc() for server in monitor_urls: try: server_metrics = get_server_data(input['user'], input['password'], monitor_urls[server], input['lfile']) display_results(node, inode, server_metrics, monitor_urls[server]) except Exception as ex: log=logopen(input['lfile']) logwriter(log,'Exception') logwriter(log,str(ex.__context__)) logclose(log) pass t1 = time.time() delta.set((t1-t0)) while ((t1-t0) &#x3C; input['mintervall']): time.sleep(1.0) t1 = time.time() # update once per day the input if ( t1 - start0 > 86400): start0 = t1 input = getServerList() monitor_urls = get_server_urls(input['user'], input['password'], input['server'], input['lfile']) </code></pre> <p>As you can see, I used four Python functions within the main routine:</p> <ul> <li><em>getServerList</em></li> <li><em>get_server_urls</em></li> <li><em>get_server_data</em></li> <li><em>display_results</em></li> </ul> <p>The first one - <em>getServerList</em> - is just reading the input data out of an XML file. Since this is a pretty standard task in Python, I do not want to discuss this function here further. It should be sufficient to know, that it is reading the list of server IPs and additional runtime parameters (username, password, monitoring intervall, …) into a Python directory that is used in the subsequent parts of the script. I want to focus on the other three functions that are more specific to the ILO Prometheus exporter.</p> <h2>get_server_urls</h2> <p>The get_server_urls routine uses the list of ILO IP addresses to gather the correct URLs out of the Redfish resource directory. If the ILO is alive  a Redfish client object for this ILO is created and the Redfish resource directory is read. The client object is stored together with the URLs for the thermal, power and computer system information in the python directory server_urls, which is the return value of the routine.</p> <pre><code class="language-python">def get_server_urls( login_account, login_password, server, lfile): server_urls={} if LINUX: cmd='ping -c 2 ' log=logopen(lfile) logwriter(log,'get_server_urls') for s in server: ilo_url="https://"+s['ilo'] s["url"]=ilo_url if LINUX: response=os.system(cmd+s['ilo']) # works on Linux without root priviliges else: response = ping(address=s['ilo'],count=1) # requires root priviliges on Linux if(response.is_alive): response = 0 else: response = 256 if (response == 0): try: # Create a Redfish client object REDFISHOBJ = redfish.redfish_client(base_url=ilo_url, username=login_account, password=login_password) # Login with the Redfish client REDFISHOBJ.login() s["redfish"]=REDFISHOBJ resource_instances = get_resource_directory(REDFISHOBJ, lfile) for instance in resource_instances: if '#ComputerSystem.' in instance ['@odata.type']: s["ComputerSystem"]=instance['@odata.id'] if '#Power.' in instance ['@odata.type']: s["Power"]=instance['@odata.id'] if '#Thermal.' in instance ['@odata.type']: s["Thermal"]=instance['@odata.id'] if len(s) > 4: server_urls[s['ilo']]=s logwriter(log,s['ilo']+' completed') except Exception as ex: logwriter(log,'Exception - get_server_urls: '+s['ilo']) logwriter(log,str(ex.__context__)) if len(s) > 4: server_urls[s['ilo']]=s pass else: logwriter(log,'Exception - ILO is not reachable: '+s['ilo']) logclose(log) return server_urls </code></pre> <h2>get_server_data</h2> <p>The get_server_data routine takes the obtained server_urls of one of the monitored servers together with the Prometheus user information to retrieve the power and thermal data as well as some general system information data. The captured data is packed into a server_data directory and returned to the main program.</p> <pre><code class="language-python">def get_server_data( login_account, login_password, server, lfile): server_data={} try: REDFISHOBJ = server["redfish"] # System Data server_data['System'] = REDFISHOBJ.get(server["ComputerSystem"]).obj # Power Data uri = REDFISHOBJ.get(server["Power"]).obj.Oem.Hpe['Links']['PowerMeter']['@odata.id'] server_data["PowerMeter"] = REDFISHOBJ.get(uri).obj # Temperatures server_data['Temperatures'] = REDFISHOBJ.get(server["Thermal"]).obj["Temperatures"] return server_data; except Exception as ex: sys.stderr.write("ERROR during get_server_date: "+server["url"]) if ex.status == 401: REDFISHOBJ = redfish.redfish_client(base_url=server["url"], username=login_account, password=login_password) REDFISHOBJ.login() server["redfish"] = REDFISHOBJ log=logopen(lfile) logwriter(log,'Exception') logwriter(log,str(ex.__context__)) logclose(log) pass </code></pre> <h2>display_results</h2> <p>The display_results routine takes the server_data retrieved by the get_server_data routine and puts it into the corresponding Prometheus node gauges.</p> <pre><code class="language-python">def display_results( node, inode, server_metrics, server): hostname = (server_metrics['System']['HostName']).split('.')[0].replace('-','_') cn = server['ilo'] inode.labels(cn).info({"Model":server_metrics['System']["Model"],"Manufacturer":server_metrics['System']["Manufacturer"],"SerialNumber":server_metrics['System']["SerialNumber"],"Hostname":hostname}) node.labels(cn,server['Rack'],'Power','State').set(power_state[server_metrics['System']["PowerState"]]) node.labels(cn,server['Rack'],'Power','Average').set(server_metrics["PowerMeter"]['Average']) node.labels(cn,server['Rack'],'Power','Maximum').set(server_metrics["PowerMeter"]['Maximum']) node.labels(cn,server['Rack'],'Power','Minimum').set(server_metrics["PowerMeter"]['Minimum']) for temperature in server_metrics['Temperatures']: if temperature['Status']['State'] == 'Enabled': node.labels(cn,server['Rack'],'Temperature',temperature["Name"]).set(temperature['ReadingCelsius']) return 0 </code></pre> <p>The most current and complete HPE ILO Prometheus exporter (iloPromConnector.v4.3.py at the time of writing this blog), together with a routine (createILOcreadentials.v1.0.py) to generate XML-input file with encrypted user credentials, is available on <a href="https://github.com/tbeha/iloPrometheus" target="_blank">Github</a>.</p> <h1>Visualize the data</h1> <p>The CTC monitoring environment uses Prometheus as a time series database to store and Grafana to visualize the gathered data. Prometheus, Grafana, the <a href="https://developer.hpe.com/blog/get-started-with-prometheus-and-grafana-on-docker-with-hpe-storage-array-exporter/" target="_blank">HPE Storage Array Exporter for Prometheus</a> and the <a href="https://www.hpe.com/psnow/doc/a50000514enw" target="_blank">HPE SimpliVity Exporter</a> are deployed as Kubernetes pods in order to easily scale and adjust the CTC monitoring environment.</p> <p>In order to insert the CTC power and temperature monitoring into this environment, I first had to containerize the HPE Server Exporter.  I based the container on Ubuntu Linux, added Python and the necessary Python libraries to it and stored the server exporter script in the directory /opt/prometheus:</p> <pre><code class="language-dockerfile"># User Ubuntu as the base Image FROM ubuntu # LABEL maintainer="Thomas Beha" LABEL version="4.3" LABEL copyright="Thomas Beha, 2023" LABEL license="GNU General Public License v3" LABEL DESCRIPTION="CTC ILO Redfish Prometheus Connector Python container based on Ubuntu" # RUN apt-get update RUN apt-get -y install python3.6 &#x26;&#x26; \ apt-get -y install python3-pip &#x26;&#x26; \ apt-get -y install iputils-ping RUN /usr/bin/pip3 install requests &#x26;&#x26; \ /usr/bin/pip3 install fernet &#x26;&#x26; \ /usr/bin/pip3 install cryptography &#x26;&#x26; \ /usr/bin/pip3 install lxml &#x26;&#x26; \ /usr/bin/pip3 install redfish &#x26;&#x26; \ /usr/bin/pip3 install prometheus_client # copy the necessary python files to the container RUN mkdir /opt/prometheus COPY ./iloPromConnector.v4.3.py /opt/prometheus </code></pre> <p>I used this Dockerfile to generate a container image and uploaded it into my registry. Before I could define the Kubernetes pod based on this container image, I had to decide how to provide the input data to the pod. The current HPE server exporter expects to have the input data available at /opt/prometheus/data. The easiest and most flexible way to provide the input data is to use a Kubernetes config map. This allowed me to use the same container image multiple times, each time with a different config map. The large number of servers hosted in the CTC required multiple server exporter, each one with a different list of server to monitor. An example of a config map that I used is shown below:</p> <pre><code class="language-yaml">apiVersion: v1 kind: ConfigMap metadata: name: iloprom1-xml namespace: svtprometheus data: iloprometheus.key: |- LVkjkV5j40Fl6U0O7ratU0RDf5W1NmAE0oqoQ2DaE74= iloprometheus.xml: |- &#x3C;data> &#x3C;username>Prometheus&#x3C;/username> &#x3C;user>gAAAAABig6ZJgST83R7v5AFjsOKiYt0bdledJpF3-TnUBR5871eNttG8P-O_eYA55-5NyyZgmmeWRnWy3hPAbUFk18_sWNw==&#x3C;/user> &#x3C;password>gAAAAABig6ZJAZL8wS41b0snZ76VWo17JysUq0-k5NUCU1y5AAPw5p1MphkFcB_u6z0TgcPbmR9yJcX39Cx0Egwv5Lou8A==&#x3C;/password> &#x3C;monitoringintervall>30&#x3C;/monitoringintervall> &#x3C;logfile>iloprometheus.log&#x3C;/logfile> &#x3C;port>9091&#x3C;/port> &#x3C;server> &#x3C;ILO_ip>10.1.40.5&#x3C;/ILO_ip>,&#x3C;Rack>i5&#x3C;/Rack>,&#x3C;Loc>16&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.1.40.6&#x3C;/ILO_ip>,&#x3C;Rack>i5&#x3C;/Rack>,&#x3C;Loc>14&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.1.40.7&#x3C;/ILO_ip>,&#x3C;Rack>i5&#x3C;/Rack>,&#x3C;Loc>19&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.1.40.8&#x3C;/ILO_ip>,&#x3C;Rack>i5&#x3C;/Rack>,&#x3C;Loc>18&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.0.44.11&#x3C;/ILO_ip>,&#x3C;Rack>F12&#x3C;/Rack>,&#x3C;Loc>24&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.0.44.12&#x3C;/ILO_ip>,&#x3C;Rack>F12&#x3C;/Rack>,&#x3C;Loc>22&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.0.44.13&#x3C;/ILO_ip>,&#x3C;Rack>F12&#x3C;/Rack>,&#x3C;Loc>20&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.1.24.66&#x3C;/ILO_ip>,&#x3C;Rack>C5&#x3C;/Rack>,&#x3C;Loc>30&#x3C;/Loc> &#x3C;/server> &#x3C;server> &#x3C;ILO_ip>10.0.43.14&#x3C;/ILO_ip>,&#x3C;Rack>i12&#x3C;/Rack>,&#x3C;Loc>9&#x3C;/Loc> &#x3C;/server> &#x3C;/data> </code></pre> <p>As you can see, the config map contains two data parts: the iloprometheus.key and the iloprometheus.xml section. The first one stores the encryption key, while the second one stores username and password encrypted with the encryption key iloprometheus.key, the monitoring intervall, the logfile name and the port used by the exporter together with the server data. I used the rack location, in addition to the ILO IP address, in order to be able to visualize a rack temperature diagram if needed.</p> <p>The config map was stored in a YAML-file (iloprom1.yml) and then applied in my Kubernetes environment:</p> <pre><code class="language-shell">kubectl apply -f iloprom1.yml </code></pre> <p>With the config map in place, I was able to define the Kubernetes pod for the HPE Server exporter. The pod uses the iloprometheus:v4.3.1 container image, mounts the config map data to the /opt/prometheus/data directory in the container and starts the Python script /opt/prometheus/iloPromConnector.v4.3.py.</p> <pre><code class="language-shell">cat &#x3C;&#x3C; 'EOF' | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: ilo1 namespace: svtprometheus labels: app: ilo1 spec: selector: matchLabels: app: ilo1 template: metadata: labels: app: ilo1 spec: containers: - name: ilo1 image: tb1378/iloprometheus:v4.3.1 command: ["/usr/bin/python3"] args: ["/opt/prometheus/iloPromConnector.v4.3.py"] volumeMounts: - name: iloprometheusxml mountPath: /opt/prometheus/data volumes: - name: iloprometheusxml configMap: name: iloprom1-xml # the correct name of the configmap needs to be added here. EOF </code></pre> <p>After I checked that the pod was up and running, I found that I still needed to define the Kubernetes service using this pod/app in order to have the exporter data available for Prometheus.</p> <pre><code class="language-shell">cat &#x3C;&#x3C; 'EOF' | kubectl apply -f - apiVersion: v1 kind: Service metadata: name: ilo1 namespace: svtprometheus # labels: # hpecp.hpe.com/hpecp-internal-gateway: "true" spec: selector: app: ilo1 ports: - port: 9091 # The Port of that the Prometheus connector uses targetPort: 9091 protocol: TCP type: NodePort # expose the Prometheus connector if you want/need to debug it. EOF </code></pre> <p>Next, I updated the Prometheus config map to make sure that the server exporter data was collected and stored in the Prometheus time series database. Hence, I added a job for the server exporter to the existing job list of the Prometheus config map:</p> <pre><code class="language-shell"> - job_name: 'iloredfish' static_configs: - targets: ['ilo1:9091','ilo2:9091','ilo3:9091','ilo4:9091','ilo5:9091','ilo6:9091'] honor_timestamps: true scrape_interval: 60s scrape_timeout: 20s metrics_path: /metrics scheme: http </code></pre> <p>As you can see from the above, I actually used six server exporter (ilo1 up to ilo6) in the CTC monitoring environment to cover the complete list of server hosted in the CTC. After having the data collected into the Prometheus timeseries database, I defined some Grafana dashboards to visualize the collected data. I don't want to get into the details of defining Grafana dashboards here - this is well documented in the Grafana documentation. I just want to give you an example of a dashboard I have used:</p> <p><img src="/img/ctcracktemppower.2.png" alt="" title="CTC Rack Temperature &#x26; Power Overview"></p> <h1>Conclusion</h1> <p>In this blog post, I documented my approach on solving a challenge we had in the Customer Technology Center Böblingen by using available API interfaces and building on an existing Kubernetes, Prometheus and Grafana environment. I decided to write this rather long blog in order to give you all necessary steps I did in detail with the goal to show you what can be done with public APIs, a little bit of Python scripting and the use of open source tools like Prometheus and Grafana. I do hope that this is helpful when you need to develop your own data collectors.</p> <p>The complete Python script together with sample Grafana dashboards and a Jupyter notebook for the deployment as a Kubernetes service is available at <a href="https://github.com/tbeha/iloPrometheus" target="_blank"><a href="https://github.com/tbeha/iloPrometheus">https://github.com/tbeha/iloPrometheus</a></a>.</p> <p>Check out what the Customer Technology Center in Boeblingen looks like <a href="https://www.hpe.com/us/en/about/customer-centers/boeblingen.html" target="_blank">here</a>.</p>https://developer.hpe.com/mixtral-8x7b-that-may-pave-the-trend-to-adopt-the-mixture-of-experts-model/https://developer.hpe.com/mixtral-8x7b-that-may-pave-the-trend-to-adopt-the-mixture-of-experts-model/Wed, 20 Dec 2023 01:25:25 GMT<p>On December 8th, 2023, Mistral AI released Mixtral 8x7B, a high-quality sparse mixture of experts model (SMoE) with open weights (ref: <a href="https://mistral.ai/news/mixtral-of-experts/">Mistral AI</a>). What is this Mixtral 8x7B? Very simply, it takes 8 x Mixtral 7B models and combines them to make one model. For those who are unfamiliar with the term Mixture of Experts (MOE), it is a machine learning technique that uses multiple expert networks to divide a problem space into homogeneous regions. As noted in this <a href="https://arxiv.org/pdf/2305.14705.pdf">paper</a>, MOE models build upon the observation that language models can be decomposed into smaller, specialized sub-models, or "experts". These models or experts focus on distinct aspects of the input data, thereby enabling more efficient computation and resource allocation. It has become quite popular because AI models with sparsely activated MOE significantly reduce the computational involved with large language models (LLMs).</p> <p>So, how does it work? If you are into LLM modeling, you probably know that the most simple representation of an LLM would be an input going into a single model and producing an output. In an MOE model, your input goes through a gateway that decides, based on the input context, which of the eight models are to be used to process the input. It could be just one model or possibly two or more that will process the input and coordinate to product the output. Below is a simplified pictorial representation of the process:</p> <p><a href="https://soonhengblog.files.wordpress.com/2023/12/image-6.png"><img src="https://soonhengblog.files.wordpress.com/2023/12/image-6.png?w=900" alt=""></a></p> <p>To best understand this, you could compare it to your supervisor looking at the nature of a job and finding the best expert within his team (i.e. MOE model) to complete the task rather than giving all the tasks to a single person (i.e. the traditional LLM model). In this way, it is much more efficient, except that you now need to maintain eight models instead of one.</p> <p>Generally, models often reuse the same parameters for all inputs. But MOE models uses different parameters based on the input given. Routing multiple models is not as easy as you think. There is an overhead in communication between the models as well as the routing between them. This <a href="https://arxiv.org/abs/2101.03961">paper</a> explains it well. The gating network (typically a neural network), produces a sparse distribution over the available experts. It might only choose the top-k highest-scoring experts, or softmax gating, which encourages the network to select only a few experts. Getting the right balance is not easy, meaning that you may end up choosing the same expert every single time versus evenly distributing it across the experts.</p> <p>How good is Mixtral 8x7B? According to <a href="https://mistral.ai/news/mixtral-of-experts/">mistral AI</a>, Mixtral 8x7B outperforms Llama 2 70B on most benchmarks with 6x faster inference. It is the strongest open-weight model with a permissive license and the best model overall regarding cost/performance trade-offs. In particular, it matches or outperforms GPT3.5 on most standard benchmarks (see diagram below). As you may have guessed, there isn't a performance comparison against GPT 4. Hopefully, we will see one put together in the near future.</p> <p><a href="https://soonhengblog.files.wordpress.com/2023/12/image-7.png"><img src="https://soonhengblog.files.wordpress.com/2023/12/image-7.png?w=890" alt=""></a></p> <p>From the benchmarks, you can see that even the scrore for the LLaMA 2 70B parameters outperform Mixtral 8x7B parameters by only a few percentage points. Mixtral also shows improvements in reducing hallucinations and biases in the <a href="https://mistral.ai/news/mixtral-of-experts/">TruthfulQA/BBQ/BOLD</a> benchmark.</p> <p>If you look at the post on "X" (below), the filename says it all.</p> <p><a href="https://soonhengblog.files.wordpress.com/2023/12/image-8.png"><img src="https://soonhengblog.files.wordpress.com/2023/12/image-8.png?w=521" alt=""></a></p> <p>Mixtral 8x7B stands out for its high-quality performance. The model boasts a range of capabilities, including the ability to handle a context of 32k tokens and support for multiple languages, including English, French, Italian, German, and Spanish. Mixtral is a decoder-only sparse mixture-of-experts network. Its architecture enables an increase in parameters while managing cost and latency. Take note that all the expert models are loaded into VRAM, which means it requires lots of super-fast VRAM. Also, the model only uses a fraction of the total set of parameters per token. According to <a href="https://mistral.ai/news/mixtral-of-experts/">Mistral AI</a>, Mixtral has 46.7B total parameters, but it only uses 12.9B parameters per token. What this mean is that it processes both input and generate output at same cost as a 12.9B parameters. Hence, it lowers LLM computational cost.</p> <p>You might be wondering if you can deploy one and try it out? This would be unlikely for most general users, unless they happen to have some spare 2 x 80GB 100A or 4 x 40 GB 100A GPs lying around or have access to them through a cloud provider. However, there are some demo platforms you can go to and try it out. Here are a few you can look into: Perplexity, Poe, Vercel, and Replicate. Below is a screenshot of what I found on Perplexity:</p> <p><a href="https://soonhengblog.files.wordpress.com/2023/12/image-5.png"><img src="https://soonhengblog.files.wordpress.com/2023/12/image-5.png?w=1024" alt=""></a></p> <p>Is this something new? Nope. This <a href="https://ieeexplore.ieee.org/document/6797059">paper</a> called "Adaptive Mixtures of Local Experts" written by the GodFather of AI, Geoffrey Hinton back in 1991, already talk about this concept. Like most AI, the advancement in computing technology, the affordability of computational cost and the availability of mass amount of data make those technology conceptualized in the early days, possible now.</p> <p>So, why the sudden interest in Mixture of Experts? Although OpenAI hasn't publicly commented on the details of the GPT-4 technical specifications, it has been shared within the OpenAI developer community that GPT-4 uses a Mixture of Experts model. With the release of a Mixtral 8x7B model to the open source world, it could very well set a trend for future model developers to embark on a similar path. Who really knows? Only time will tell.</p>https://developer.hpe.com/open-sourcing-packetrusher-a-5g-core-performance-tester/https://developer.hpe.com/open-sourcing-packetrusher-a-5g-core-performance-tester/Mon, 18 Dec 2023 10:04:11 GMT<p>In the fast-evolving landscape of 5G technology, the demand for robust and efficient testing tools has never been higher. Enter PacketRusher, a cutting-edge 5G core Network performance testing tool. As the lead developer behind this groundbreaking project, I am thrilled to share the power and potential PacketRusher holds in revolutionizing the way we test and optimize 5G networks.</p> <h2>PacketRusher</h2> <p>PacketRusher is a tool dedicated to the performance testing and automatic validation of 5G Core Networks. It tests a 5G Core Network via its external interfaces as defined by the 3rd Generation Partnership Project (3GPP): N1, N2 and N3 interfaces.</p> <p>The tool can emulate various types of user equipment (eg. 5G phones or 5G-enabled IoT devices) on the N1 interface using the Non-access Stratum (NAS-5GS) 5G protocol. It can do stress testing of the 5G Core with up to 100k simulated user equipment doing simultaneously multiple 5G Control Plane’s procedures: Registration with authentication and Deregistration, Creation and Deletions of Protocol Data-Unit (PDU) Sessions.</p> <p>These user equipements can be used together with PacketRusher’s simulated gNodeBs (eg. antennas in Radio Access Network) implementing the N2 interface. These simulated gNodeB implements a set of pre-defined procedures on top of both New Generation Application Protocol (NGAP) which allow the usage of advanced gNodeB features as user equipment handover between gNodeB with PDU Session continuity.</p> <p>The other most important feature of PacketRusher is its highly performant N3 generic tunnel. Alongside its control plane testing features, PacketRusher is also able to stress test the UPF of a 5G Core with user equipment being able to do each up to 5 GB/s of userplane traffic.</p> <p>All of these features of PacketRusher can be offered from a simple Linux virtual machine, that can compiled in minutes, without the needs for expensive commercial tools, or 100k phones on a table ;-)</p> <h2>Architecture</h2> <p>Before I show you how PacketRusher works, it's important to have an understanding of the components of a minimal 5G network:</p> <p><img src="/img/high-level-diagram-of-a-5g-deployment.png" alt="High-level diagram of a 5G deployment" title="High-level diagram of a 5G deployment"></p> <p>To summarize, there are essentially three main components:</p> <ul> <li>User equipment, which is any 5G device used directly by a user to communicate. For example, a smartphone or an IoT device.</li> <li>The gNodeB, which is the radio used for wireless communication between the User Equipment and the 5G network core.</li> <li>And finally, the 5G network core itself, which manages all mobile network functionalities. This is the component that will manage phone authentication in the network, quality of service, routing to the data network, and so on...</li> </ul> <p><img src="/img/packetrusher-architecture.png" alt="High-level diagram of the PacketRusher&#x27;s architecture and its interaction with a 5G Core&#x27;s AMF and UPF" title="High-level diagram of the PacketRusher&#x27;s architecture and its interaction with a 5G Core&#x27;s AMF and UPF"></p> <p>PacketRusher's behavior is quite simple; it essentially simulates two of the three major components of a 5G network:</p> <ul> <li>the User Equipments, i.e. the phones</li> <li>and the gNodeB, i.e. the antennas.</li> <li>At the same time, it connects to the 5G Core via its external interfaces, as if it were a black box.</li> </ul> <p>PacketRusher simulates both components en masse, which means it can simulate several user devices and gNodeBs at the same time.</p> <h2>Community contributions</h2> <p>Excitingly, PacketRusher has caught the attention of a leading industry player. Orange, a key player in the telecommunications sector, has recognized the potential of PacketRusher and is actively integrating it into their open-source GitHub project, <a href="https://github.com/Orange-OpenSource/towards5gs-helm">toward-5gs</a>.</p> <p>I would also like to mention the outstanding work of Github User <a href="https://github.com/s5uishida">s5uishida</a> who made a <a href="https://github.com/s5uishida/simple_measurement_of_upf_performance">high-quality performance comparison</a> of open-source UPF using PacketRusher.</p> <h2>Join the 5G revolution</h2> <p>How 5G evolves will be based on community collaboration and you're invited to be a part of this exciting journey. Whether you are a seasoned developer, a researcher, or simply passionate about the possibilities of 5G, your contributions can make a significant impact.</p> <p>To get started, visit the <a href="https://github.com/HewlettPackard/PacketRusher">GitHub repository</a> to access the source code, documentation, and engage with the community. Your feedback, suggestions, and contributions are invaluable in shaping the future of 5G technology.</p> <p>At HPE, we are committed to pushing the boundaries of innovation, and with the release of PacketRusher as open-source, we are laying the foundation for a new era of connectivity.</p> <p>PacketRusher is not just a testing tool; it's a community-driven effort to push the boundaries of 5G technology. Let's build the future of 5G together!</p> <p><a href="https://www.linkedin.com/in/valentin-d-emmanuele/">Valentin D'Emmanuele</a>,<br> Lead Developer and Maintainer, PacketRusher</p>https://developer.hpe.com/dark-mode-theming-in-grommet-theme-color-customization/https://developer.hpe.com/dark-mode-theming-in-grommet-theme-color-customization/Fri, 15 Dec 2023 16:54:03 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/g1lightdark3_1-1603892557138.png" alt="g1lightdark3_1"></p> <p>This is the final post in a three-part series. Parts <a href="/blog/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme">1</a> and <a href="/blog/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes">2</a> introduced how to apply a theme, change the theme’s mode between light and dark, and, finally, provide an app’s user the ability to change the theme’s mode to their preference.</p> <ul> <li><a href="/blog/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme">Part 1 - How to set up and apply a theme</a></li> <li><a href="/blog/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes">Part 2 - Adding dark and light theme modes</a></li> <li>Part 3 - Theme color customization</li> </ul> <p>In this final post, I’ll demonstrate how to add custom light and dark mode colors to the theme. I will cover the following content:</p> <ul> <li>Creating a custom theme file</li> <li>Defining the color palette</li> <li>Mapping the color palette to component definitions in the theme</li> <li>Merging the customizations with an existing theme</li> </ul> <h2>Customizing the Theme</h2> <p>Up to this point, we have been using Grommet’s theme. Let’s say, however, we’d like to tweak the Grommet theme by adding some custom colors for my fictional company, Acme, Inc.</p> <p>To do this, continue modifying your app from where <a href="/blog/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes">Part 2</a> concluded, or reference this <a href="https://codesandbox.io/s/grommet-theme-toggle-2addtogglebutton-txbux?file=/src/App.js">Codesandbox</a> if you are catching up and joining midstream.</p> <p>Acme’s brand colors are Ruby, Gold, and Amethyst, with some warm greys for backgrounds and text. The hex values for Acme’s color palette, plus values for the light and dark variants, are provided below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/g3-table-1603909712399.JPG" alt="g3 table"></p> <p>To begin the custom Acme, Inc. theme, <strong>create a theme file</strong>, <strong>define the color palette</strong>, and then <strong>map the color palette</strong> to the Grommet components for which you want the colors to be applied.</p> <h2>Create Theme File</h2> <p>In the <code>src</code> directory, create a theme file called <code>acme-theme.js</code> with the theme object <code>acme</code> as an export:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/g2part-3-image-1603892563485.png" alt="g2part 3 image"></p> <h2><a id="head2"></a>Define the Color Palette</h2> <p>Define the color palette by naming each color, plus their dark and light variants. Colors are defined in the theme object’s <code>global.colors</code> property. For each color, the following structure is used:</p> <pre><code class="language-markdown">uniqueColorName: { dark: ‘hexidecimal value or reference a color name’, light: ‘hexidecimal value or reference a color name’, }, uniqueColorName! : ‘hexidecimal value or reference a color name’, </code></pre> <p>Notice that any Grommet component with color as an attribute can accept either a string or an object containing <code>dark</code> and <code>light</code> properties. <code>Dark</code> specifies the color Grommet uses when the element is on a dark background; <code>light</code> specifies the color for use on a light background.</p> <p>Additionally, by convention, a color name followed by a ‘!’ bang represents the color’s “true” value rather than a dark/light variant.</p> <p>Add Acme, Inc. colors to <code>acme-theme.js</code> like so:</p> <pre><code class="language-javascript">export const acme = { global: { colors: { ruby: { dark: "#d4111e", light: "#f58990" }, "ruby!": "#EF3F4C", gold: { dark: "#df9007", light: "#e7b86b" }, "gold!": "#F9B644", amethyst: { dark: "#9B59B6", light: "#C39BD3" }, "amethyst!": "#AF7AC5", "grey-1": "#ECE9E3", "grey-2": "#CECCC6", "grey-3": "#737069", "grey-4": "#52504C", } } }; </code></pre> <h2><a id="head3"></a>Map Colors to Grommet Namespaces and Components</h2> <p>Now that the colors are defined, it’s time to apply them. For the purposes of this tutorial, I will only map the colors to a handful of theme properties. This will demonstrate the process of implementation, but it is certainly not exhaustive. For the curious, inspecting <a href="https://github.com/grommet/grommet/blob/master/src/js/themes/base.js">Grommet’s base theme</a> is a great place to take inventory of the many possibilities to fully customize your own theme.</p> <p>Modify <code>acme-theme.js</code> by mapping the colors to background, brand, control, and anchor properties:</p> <pre><code class="language-javascript">export const acme = { global: { colors: { /* BEGIN: Color Palette Definition */ ruby: { dark: "#d4111e", light: "#f58990" }, "ruby!": "#EF3F4C", gold: { dark: "#df9007", light: "#e7b86b" }, "gold!": "#F9B644", amethyst: { dark: "#9B59B6", light: "#C39BD3" }, "amethyst!": "#AF7AC5", "grey-1": "#ECE9E3", "grey-2": "#CECCC6", "grey-3": "#737069", "grey-4": "#52504C", /* END: Color Palette Definition */ /* BEGIN: Mapping Colors to Grommet Namespaces */ background: { dark: "grey-4", light: "grey-1" }, "background-back": { dark: "grey-4", light: "grey-1" }, "background-front": { dark: "grey-3", light: "grey-2" }, brand: "ruby!", control: { dark: "brand", light: "brand" }, input: { background: "blue" }, text: { dark: "grey-1", light: "grey-3" } }, focus: { border: { color: "gold" } } /* END: Mapping Colors to Grommet Namespaces */ }, /* BEGIN: Mapping Colors to Components */ anchor: { color: { dark: "gold", light: "amethyst!" } } /* END: Mapping Colors to Components */ }; </code></pre> <h2>Merging a Custom Theme with an Existing Theme</h2> <p>In <code>App.js</code>, add the following imports:</p> <ul> <li> <p><code>import { deepMerge } from “grommet/utils”;</code> - A function allowing an existing theme to be customized or extended</p> </li> <li> <p><code>import { acme } from “./acme-theme”;</code> - Our custom theme file</p> </li> <li> <p><code>import { DemoSection } from “./DemoSection”;</code> - A section of sample components to see how theme customizations are applied</p> </li> </ul> <p>Then, add the line <code>const theme = deepMerge(grommet, acme)</code>. The imported <code>deepMerge</code> function incorporates the <code>acme</code> specifications into the <code>grommet</code> theme, resulting in a new custom <code>theme</code>.</p> <pre><code class="language-javascript">import React from "react"; import { Grommet, grommet, Anchor, Box, Button, Heading, Paragraph } from "grommet"; import { deepMerge } from "grommet/utils"; import { acme } from "./acme-theme"; import { DemoSection } from "./DemoSection"; const theme = deepMerge(grommet, acme); </code></pre> <p>Finally, swap out the <code>grommet</code> theme with the newly created <code>theme</code>.</p> <pre><code class="language-javascript"> &#x3C;Grommet full theme={theme} themeMode={darkMode ? "dark" : "light"}> </code></pre> <p>That concludes this tutorial. Your final code and resulting app should resemble this <a href="https://codesandbox.io/s/grommet-theme-toggle-3customizeourtheme-9wqfb?file=/src/App.js">Codesandbox</a>. I hope you have enjoyed this three-part tutorial.</p> <p>As review, here’s how the app was modified:</p> <ul> <li>Created a custom theme file.</li> <li>Defined the color palette and its namespaces.</li> <li>Mapped the color namespaces to Grommet namespaces and component definitions.</li> <li>Finally, merged the theme customizations with Grommet’s theme.</li> </ul> <h2>Next Steps for Exploration</h2> <p>Now that you have seen how easy it is to apply a theme to Grommet, set and toggle the theme’s light/dark modes, and even start applying custom colors to your own theme, here are some great next steps you can take:</p> <ul> <li>Check out <a href="https://theme-designer.grommet.io/">Grommet’s Theme Designer</a> (Beta) and other <a href="https://developer.hpe.com/platform/grommet/home/">Grommet Resources</a>.</li> <li>Explore Grommet’s other theme properties which can be customized.</li> <li>Create and apply your own theme to your own project, then share it on Grommet’s <a href="https://grommet.slack.com/archives/CG25TE0KZ">#i-made-this</a> Slack channel for community members to enjoy.</li> </ul>https://developer.hpe.com/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes/https://developer.hpe.com/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes/Fri, 15 Dec 2023 15:46:57 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/f1lightdark2-1603286799167.png" alt="f1lightdark2"></p> <p>In <a href="/blog/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme">Part 1</a> of this 3-part series on Grommet’s support for light and dark modes, I covered setting up a simple Grommet app and applying a theme to that app. Here in Part 2, I’ll guide you through the steps required to implement dark/light theme modes. At the conclusion of this post, the app will have some basic UI components and a control to toggle the interface between light and dark modes.</p> <ul> <li><a href="/blog/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme">Part 1 - How to set up and apply a Theme</a></li> <li>Part 2 - Adding dark and light theme modes</li> <li><a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">Part 3 - Theme color customizations</a></li> </ul> <p>In this post, I’ll cover content regarding adding a theme toggle button, including:</p> <ul> <li> <p>Introducing the <code>themeMode</code> prop, which allows specifying which version of the theme the app renders.</p> </li> <li> <p>Adding a button to the interface to serve as a control to toggle the value that gets passed to <code>themeMode</code>.</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/f10thememodetoggle-1603286872853.gif" alt="f10thememodetoggle"></p> <h2>Adding a Theme Toggle Button</h2> <p>For this exercise, you’ll continue modifying your app from the example you created in Part 1 of this series. If you are catching up and joining midstream, you can reference this <a href="https://codesandbox.io/s/grommet-theme-toggle-1adding-theme-rg91i?file=/src/App.js">Codesandbox</a>.</p> <p>First, I’m going to show you how to add a button that, when clicked, will toggle the theme between light and dark modes.</p> <p>In App.js:</p> <p>Add the <em><strong><a href="https://v2.grommet.io/grommet#themeMode">themeMode</a></strong></em> prop to the <code>&#x3C;Grommet></code> component and set its value to <code>"dark"</code>. The value referenced by <code>themeMode</code> specifies whether Grommet should use the dark or light versions of the theme.</p> <pre><code class="language-javascript"> &#x3C;Grommet full theme={grommet} themeMode="dark"> </code></pre> <p>This should result in:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/f4part-2-toggle-1603286827841.png" alt="f4part 2 toggle"></p> <p>Next, add a button to serve as the toggle control.</p> <p>Import <code>Button</code> as a component from Grommet</p> <pre><code class="language-javascript">import { Grommet, Anchor, Box, Button, List, Heading, Paragraph, Text } from "grommet"; </code></pre> <p>Below the <code>&#x3C;List></code>, add a theme toggle button with some formatting props and an <code>onClick</code> handler.</p> <pre><code class="language-javascript"> &#x3C;Button label="Toggle Theme" primary alignSelf="center" margin="large" onClick={() => {}} /> </code></pre> <h2>Enable Toggling of ThemeMode’s State</h2> <p>Next, make the theme mode dynamic by adding a variable <code>darkMode</code> to hold the current theme mode, storing it in the component’s state, and adjusting the state each time the theme toggle button is clicked.</p> <p>Create variable <code>darkMode</code> and its state using React’s <a href="https://reactjs.org/docs/hooks-state.html"><code>useState</code> Hook</a>.</p> <pre><code class="language-javascript">const App = () => { const [darkMode, setDarkMode] = React.useState(false); return ( &#x3C;Grommet full theme={grommet} themeMode="dark"> </code></pre> <p>Modify the button’s <code>onClick</code> handler to toggle <code>darkMode</code> between <code>true</code> and <code>false</code>.</p> <pre><code class="language-html">&#x3C;Button label="Toggle Theme" primary alignSelf="center" margin="large" onClick={() => setDarkMode(!darkMode)} /> </code></pre> <p>Next, replace <code>themeMode</code>’s value to be “dark” when <code>darkMode</code> is true, and “light” when <code>darkMode</code> is false.</p> <pre><code class="language-javascript">&#x3C;Grommet full theme={grommet} themeMode={darkMode ? "dark" : "light"} > </code></pre> <p>The theme mode toggling should be good to go. Give the toggle button a few clicks!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/f2thememodetogglemid-1603286807584.gif" alt="f2thememodetogglemid"></p> <h2>Incorporating More Visual Interest</h2> <p>Finally, to better demonstrate a changing theme, let’s add some more interesting visuals to the application.</p> <p>Remove the following from App.js</p> <pre><code class="language-javascript"> &#x3C;Paragraph>We will be modifying this project to:&#x3C;/Paragraph> &#x3C;List data={projectTasks}> {(task, index) => ( &#x3C;Text key={index}> {index + 1}) {task} &#x3C;/Text> )} &#x3C;/List> </code></pre> <p>Then, import the DemoSection from <code>DemoSection.js</code> and add it below the toggle button. DemoSection contains a sampling of Grommet components to better demonstrate the effect themeMode has across components.</p> <pre><code class="language-javascript"> import { DemoSection } from "./DemoSection"; </code></pre> <p>Then add DemoSection directly beneath this button.</p> <pre><code class="language-javascript"> &#x3C;Button label="Toggle Theme" primary alignSelf="center" margin="large" onClick={() => setDarkMode(!darkMode)} /> &#x3C;DemoSection /> </code></pre> <p>At this point, your code and resulting app should resemble what is shown in this <a href="https://codesandbox.io/s/grommet-theme-toggle-2addtogglebutton-txbux?file=/src/App.js">Codesandbox</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/f14part-2-non-animated-white-theme-toggle-1603286900031.png" alt="f14part 2 non animated white theme toggle"></p> <p>As quick review, here’s what we’ve done to modify the app:</p> <ul> <li> <p>Added <code>themeMode</code> as a prop on the <code>&#x3C;Grommet></code> component. The value provided to <code>themeMode</code> specifies which mode of the theme to use.</p> </li> <li> <p>Created a state variable called <code>darkMode</code> to store whether the theme should currently be in dark mode.</p> </li> <li> <p>Enabled the value of the <code>themeMode</code> prop to update when the value of <code>darkMode</code> changes.</p> </li> <li> <p>Added a <code>&#x3C;Button></code> to serve as control to toggle the theme mode by toggling the state of <code>darkMode</code>.</p> </li> <li> <p>Lastly, made the app interface a bit more interesting to demonstrate how various components are affected by toggling the <code>themeMode</code>. That’s it for Part 2! In <a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">Part 3</a>, I’ll demonstrate how to customize a theme with custom dark and light mode colors. Don’t forget to check back at the HPE DEV blog to catch <a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">Part 3</a> of this series. Again, if you have any questions, please feel free to reach out to me and others in the Grommet group on our Slack channel.</p> </li> </ul>https://developer.hpe.com/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme/https://developer.hpe.com/dark-mode-theming-in-grommet-how-to-set-up-and-apply-a-theme/Fri, 15 Dec 2023 15:45:24 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/darkmode-intro-part1-1603293464825.png" alt="darkmode intro part1"></p> <p>This post is Part 1 of a three-part series.</p> <ul> <li>Part 1 - How to set up and apply a theme</li> <li><a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes/">Part 2 - Adding dark and light theme modes</a></li> <li><a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">Part 3 - Theme color customization</a></li> </ul> <p>As a UI/UX developer at HPE, I use <a href="https://grommet.io">Grommet</a> extensively. I am also a regular contributor on the <a href="https://grommet.slack.com">Grommet Slack</a> channels where the Theming capabilities, especially Grommet’s support for light and dark modes, are a consistent topic of interest.</p> <p><a href="https://grommet.io">Grommet</a> has robust support for light and dark themes. Due to the apparent interest in this topic, I thought I’d share my approach on how to get started with theme mode styling in <a href="https://grommet.io">Grommet</a>.</p> <p>To illustrate the method I use, I’m going to create a simple application demonstrating the ability to toggle between light and dark modes. By the end of this 3-part blog series, I will have demonstrated how to:</p> <ul> <li>Apply a theme in a Grommet application</li> <li>Incorporate dark and light theme modes</li> <li>Modify a theme to apply custom colors in dark and light modes</li> </ul> <p>The final product will be a simple application with a custom theme applied and the ability for a user to toggle between the app’s light and dark modes.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/themetutorialapp-1602698870239.gif" style="height:300px, width:300px"> <h1>Setup for the tutorial</h1> <h2>Prerequisites</h2> <p>This tutorial assumes familiarity with HTML, JavaScript, and React. However, even if any of these are new to you, you should be able to follow along and complete the exercise.</p> <h2>Get the Starter Code</h2> <p>Open this <a href="https://codesandbox.io/s/grommet-theme-toggle-0starter-1k1cv?file=/src/App.js">starting code</a> in your browser. Create your own copy by clicking 'Fork' from the menu in the upper right corner. The starter app will look like this:</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture-2-1602661773922.png"> <h2>Starter Code Orientation</h2> <h3>Development Environment</h3> <p>For this tutorial, I’m using <a href="https://codesandbox.io/">Codesandbox</a> as the development environment. Codesandbox presents the user with:</p> <ul> <li>An interface for the project’s file and directory structure</li> <li>A text editor for editing files</li> <li>A browser window to view and interact with the application</li> </ul> <h3>Project Structure and Dependencies</h3> <p>The project structure mimics a minimal Create React App (CRA) structure and is organized like so:</p> <ul> <li> <p>/public</p> <ul> <li>index.html (this gets loaded by the browser and is what renders src/index.js)</li> </ul> </li> <li> <p>/src</p> <ul> <li>index.js (This is the project’s root file. I won’t be modifying it.)</li> <li>App.js</li> <li>DemoSection.js (This is a component I will use later in the tutorial)</li> </ul> </li> <li> <p>Package.json (Create React App ships with <code>react</code>, <code>react-dom</code>, and <code>react-scripts</code> as dependencies. Additionally, I’ve added <code>grommet</code> and its peer dependency, <code>styled-components</code>, to the starter project.)</p> </li> </ul> <h3>App.js</h3> <p>For this tutorial, most of the development will happen in App.js. The App.js file consists of three parts; imports, projectTasks array, and the App component.</p> <p>Import of React and Grommet components:</p> <pre><code class="language-javascript">import React from "react"; import { Grommet, Anchor, Box, List, Heading, Paragraph, Text } from "grommet"; </code></pre> <p><code>projectTasks</code> array with the tasks to be implemented</p> <pre><code class="language-javascript">const projectTasks = [ "Apply a Theme - Add an existing theme to provide some styling", `Add Theme Toggle Button - Add a button, which when clicked, will toggle the theme between light and dark modes`, "Customize Theme - Add custom colors to theme" ]; </code></pre> <p>The <code>App</code> is composed of the following Grommet components:</p> <ul> <li><code>&#x3C;Grommet></code> is the top level Grommet container</li> <li><code>&#x3C;Box></code> for some basic page layout</li> <li>For the page’s content, I have <code>&#x3C;Heading></code>,<code>&#x3C; Paragraph></code>, and <code>&#x3C;List></code> which iterates over the <code>projectTasks</code> array, returning a <code>&#x3C;Text></code> component for each task.</li> </ul> <pre><code class="language-javascript">const App = () => { return ( &#x3C;Grommet full> &#x3C;Box gap="small" pad="large" align="start"> &#x3C;Heading level="1">Hello Grommet Theme Toggle&#x3C;/Heading> &#x3C;Paragraph> This is the first step in a &#x3C;Anchor href="#">theming tutorial&#x3C;/Anchor>{" "} using Grommet. &#x3C;/Paragraph> &#x3C;Paragraph>We will be modifying this project to:&#x3C;/Paragraph> &#x3C;List data={projectTasks}> {(task, index) => ( &#x3C;Text key={index}> {index + 1}) {task} &#x3C;/Text> )} &#x3C;/List> &#x3C;/Box> &#x3C;/Grommet> ); }; export default App; </code></pre> <h1>Applying a Theme</h1> <p>To provide some initial styling, I’ll apply the <code>grommet</code> theme. In <a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">Part 3</a> of this series, I will show you how to customize and incorporate a custom theme.</p> <p>In App.js, import and apply the Grommet theme:</p> <p>Import grommet theme</p> <pre><code class="language-javascript"> import { grommet } from "grommet";  </code></pre> <p>Apply it by adding <code>theme={grommet}</code> to the Grommet component.</p> <pre><code class="language-javascript"> &#x3C;Grommet full theme={grommet}> </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture-3-1602661789429.png" alt="matt1-picture 3"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture-4-1602661802053.png" alt="matt1-picture 4"></p> <p>Your code and resulting app should resemble this <a href="https://codesandbox.io/s/grommet-theme-toggle-1adding-theme-rg91i?file=/src/App.js">Codesandbox</a>, and the app’s UI should have updated with the applied theme.</p> <p>At this point, you now have an understanding for how to apply a theme to a Grommet app. This is the foundation from which we will build custom theming. In my next post, I will show you how to add dark/light theming and give users control over toggling between theme modes. In my final post, I will wrap up with a how to customize a theme with custom dark and light mode colors.</p> <p>Stay tuned to the <a href="/blog">HPE DEV blog</a> to catch Parts <a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-adding-dark-and-light-theme-modes/">2</a> and <a href="https://developer.hpe.com/blog/dark-mode-theming-in-grommet-theme-color-customization/">3</a> of this series. If you have any questions, please feel free to reach out to me and others in the Grommet group on our <a href="https://app.slack.com/client/T04LMHMUT/C04LMHN59">Slack channel</a>.</p>https://developer.hpe.com/announcing-chapel-1-33/https://developer.hpe.com/announcing-chapel-1-33/Thu, 14 Dec 2023 23:14:56 GMT<p>External blog</p>https://developer.hpe.com/sc23-from-the-chapel-language-perspective/https://developer.hpe.com/sc23-from-the-chapel-language-perspective/Thu, 07 Dec 2023 18:24:50 GMT<p>External blog</p>https://developer.hpe.com/2023-December-05/https://developer.hpe.com/2023-December-05/Mon, 04 Dec 2023 13:00:00 GMThttps://developer.hpe.com/discover-the-power-of-data-center-monitoring-using-redfish-telemetry-and-cloud-native-tooling-part-2-streaming/https://developer.hpe.com/discover-the-power-of-data-center-monitoring-using-redfish-telemetry-and-cloud-native-tooling-part-2-streaming/Fri, 01 Dec 2023 21:50:51 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Ensuring the seamless operation of your data center is critical to business continuity. In my <a href="https://developer.hpe.com/blog/discover-the-power-of-data-center-monitoring-using-redfish-telemetry-and-cloud-native-tooling/"><strong>first article in this series</strong>,</a> I showed you how coupling Redfish telemetry, a standardized way to access telemetry data from servers and other devices, with cloud-native monitoring tools can improve the efficiency and insightfulness of your data center monitoring. In this second article, I'll show you how you can unlock the full potential of data center monitoring with Redfish telemetry streaming and cutting-edge cloud-native tools. Together, we'll delve deep into the world of streaming, as I showcase the efficiency and scalability of HPE iLO's Redfish telemetry streaming using event subscription.</p> <h1>A holistic approach</h1> <p>To fully tap the potential of data center monitoring, embark on a comprehensive journey by exploring my latest technical whitepaper, <strong>"<a href="https://www.hpe.com/psnow/doc/a50009739enw">Data center monitoring using Redfish telemetry and cloud-native tooling: Part 2 — streaming</a>"</strong>. This guide not only introduces you to the intricacies of Redfish telemetry streaming but also demonstrates the seamless integration with open-source cloud-native tools such as Telegraf, InfluxDB, and Grafana. The telemetry data source, HPE iLO, provides a wealth of performance metrics through the Redfish interface, and event subscription ensures a continuous flow of metric reports from the telemetry service.</p> <h1>Key highlights from the whitepaper</h1> <p>In my whitepaper, I provide you with a deep understanding of:</p> <ul> <li><strong>Redfish telemetry streaming:</strong> Gain insights into the efficiency and scalability of Redfish telemetry streaming using event subscription for metric collection.</li> <li><strong>Cloud-native tools:</strong> Explore the capabilities of Telegraf, InfluxDB, and Grafana, and witness how they elevate data center monitoring to new heights.</li> <li><strong>Architecture deep dive:</strong> Delve into the intricacies of the integrated solution's architecture, unraveling how data seamlessly flows from the source to the visualization layer.</li> <li><strong>Setup process:</strong> Follow a step-by-step guide on implementing this monitoring solution in your data center, ensuring a smooth and efficient setup.</li> <li><strong>Benefits:</strong> Uncover the array of benefits this approach brings, including real-time monitoring, effortless scalability, and the power to craft customizable visualizations.</li> </ul> <h1>Take the leap</h1> <p>Discover how the fusion of Redfish telemetry streaming with cloud-native tools revolutionizes data center monitoring. Dive into the depths of insights and efficiency by reading my whitepaper. Transform your data center monitoring with HPE iLO today! For more enlightening articles on HPE iLO, stay tuned to the <a href="https://developer.hpe.com/blog/">HPE Developer blog</a>.</p>https://developer.hpe.com/automate-itops-announcing-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform/https://developer.hpe.com/automate-itops-announcing-foundational-apis-for-the-hpe-greenlake-edge-to-cloud-platform/Fri, 01 Dec 2023 16:43:11 GMT<p>External blog</p>https://developer.hpe.com/configuring-sso-for-hpe-greenlake-central-private-cloud-enterprise-and-hpe-greenlake-glcp-using-okta/https://developer.hpe.com/configuring-sso-for-hpe-greenlake-central-private-cloud-enterprise-and-hpe-greenlake-glcp-using-okta/Wed, 29 Nov 2023 12:41:00 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Enterprises looking to use HPE GreenLake for Private Cloud Enterprise can benefit from the use of SSO, as it has been integrated onto the HPE GreenLake edge-to-cloud platform (also known as HPE GreenLake platform), which supports single sign-on.</p> <p>In this blog post, I will walk you through the process of configuring Okta Active Directory (AD) to authenticate users into the HPE GreenLake for Private Cloud Enterprise application on the HPE GreenLake platform using SAML Identity Provider (IdP) for single sign-on.</p> <h3>Before starting</h3> <p>Please review the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-D7192971-EF71-4304-B51E-548E7954E644.html">HPE GreenLake</a> User Guide to understand how the SAML framework works in the context of HPE GreenLake for Private Cloud Enterprise Services for the HPE GreenLake edge-to-cloud platform.</p> <h3>Configure SSO/SAML applications in Okta</h3> <p>To configure application metadata in Okta, complete the following steps:</p> <ul> <li>Step 1: Create an Okta SAML application</li> <li>Step 2: Configure Sign On settings</li> <li>Step 3: Export the SAML 2.0 IdP metadata</li> <li>Step 4: Configure the SAML connection in the HPE GreenLake platform</li> </ul> <p><strong>Step 1: Create an Okta SAML application</strong></p> <ol> <li>Log into the Okta administration console.</li> <li>Click <strong>Applications > Create new app integration.</strong> The Create a new app integration window opens.</li> <li>Select SAML 2.0 and click <strong>Next</strong>.</li> </ol> <p><img src="/img/ws-image0.png" alt=""></p> <p>Provide a name for the SAML application which gets connected to the HPE GreenLake platform:</p> <p><img src="/img/saml_app-okta.jpg" alt=""></p> <p><strong>Step 2: How to configure single sign-on settings</strong></p> <ol> <li> <p>Enter the SAML information.</p> <p>Under General:</p> <p><strong>Single Sign on URL:</strong> <a href="https://sso.common.cloud.hpe.com/sp/ACS.saml2">https://sso.common.cloud.hpe.com/sp/ACS.saml2</a></p> <p><strong>Audience URI (SP Entity ID):</strong> <a href="https://sso.common.cloud.hpe.com">https://sso.common.cloud.hpe.com</a></p> <p><strong>Name ID format EmailAddress</strong></p> <p><strong>Application username Email</strong></p> <p><strong>NameID = user.email</strong></p> <p><strong>gl_first_name = user.FirstName</strong></p> <p><strong>gl_last_name = user.LastName</strong></p> <p><strong>hpe_ccs_attribute = (See Below)</strong></p> <p>See here for IdP attribute details: <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us"></a><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-D7192971-EF71-4304-B51E-548E7954E644.html">https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us&#x26;page=GUID-D7192971-EF71-4304-B51E-548E7954E644.html</a></p> <p>A new SAML attribute has been added “hpe_ccs_attribute” which tells HPE GreenLake platform and HPE GreenLake for Private Cloud Enterprise application the exact role/permissions for each user. The following describes how to format the attribute.</p> <p><code>Format: {version}#{pcid}:{app id}:{role_name}:{ALL_SCOPES}</code></p> <p>Note : At present HPE GreenLake for Private Cloud Enterprise application role should be excluded.</p> </li> </ol> <p><img src="/img/hpe-greenlake-saml-attributes.jpg" alt=""></p> <p><img src="/img/workspace-pcid.jpg" alt=""></p> <p><img src="/img/glp_role_name.jpg" alt=""></p> <p>The <strong>hpe_ccs_attribute</strong> always starts with version1#. You must first configure the attributes for HPE GreenLake platform and to do so, enter the Platform Customer ID (PCID) for the account (this is the identifier assigned to your HPE GreenLake platform Workspace), followed by the HPE GreenLake platform application ID. This will always be <strong>00000000-0000-0000-0000-000000000000</strong>. Following this, enter the role name and ALL_SCOPES**.** </p> <p>Example:</p> <p><code>version_1#7ede5c36b7b911edacf45a78eb8b07d1:00000000-0000-0000-0000-000000000000:Observer:ALL_SCOPES</code></p> <p><img src="/img/saml_settings.jpg" alt=""></p> <p>2. Complete the setup.</p> <p><img src="/img/ws-image7.png" alt=""></p> <p>Click <strong>Next</strong> and select <strong>Internal App</strong>, then <strong>Finish</strong>.</p> <p><strong>Step 3:</strong> <strong>Export the SAML 2.0 IdP metadata</strong></p> <ol> <li> <p>Click Next – Configure the single sign-on settings</p> <p>You will find two options are available: <strong>View Setup Instructions</strong> which steps you through the SAML configuration and <strong>Identity Provider metadata</strong>, which will produce an XML file that can be loaded into HPE GreenLake platform application.</p> <p>Suggestion: click <strong>Identity Provider metadata</strong> and save the XML data to a file.</p> <p><img src="/img/ws-image9.png" alt=""></p> </li> <li> <p>Click <strong>Next</strong>.</p> </li> <li> <p>Select <strong>Internal app</strong>, and click <strong>Finish</strong>.</p> <h5><strong>Step 3.1 : Access to the SAML application and HPE GreenLake platform is determined by assigning only those members or group to the SAML application.</strong></h5> </li> </ol> <p><img src="/img/customer-user-assignment-to-saml.jpg" alt=""></p> <p><strong>Step 4:</strong> <strong>Configure the SAML connection in the HPE GreenLake platform</strong></p> <ol> <li> <p>Log into HPE GreenLake platform and click <strong>Menu</strong> > <strong>Manage</strong> > <strong>Authentication</strong> and click <strong>Set Up SAML Connection</strong>.</p> <p><em>Before you can add a new SAML configuration, you must have at least <strong>one</strong> user account with that <strong>domain</strong> already enabled in HPE GreenLake platform. Also, you must be logged into HPE GreenLake platform with an account from that domain in order to enable SSO for it.</em></p> <p><img src="/img/ws-image10.png" alt=""></p> </li> <li> <p>Type in the domain you want to enable SSO on:</p> <p><img src="/img/glp_domain.jpg" alt=""></p> </li> <li> <p>Input the metadata from the step above.</p> <p>While HPE GreenLake platform does support entering this information manually, it's recommended that you simply upload the XML metadata that was downloaded in the previous step. To do so, select <strong>Metadata File</strong>, selecting the XML file. Then, click <strong>Next</strong>.</p> <p><img src="/img/ws-image12.png" alt=""></p> </li> <li> <p>Enter the SAML attributes to match what was entered in Okta. Set the idle timeout value as well.</p> <p><img src="/img/config_setting_sso_appjpg.jpg" alt=""></p> </li> <li> <p>Then click <strong>Next</strong>.</p> </li> <li> <p>Create a recovery user so that, in the event SSO fails, an admin will still be able to access the HPE GreenLake platform.</p> <p><img src="/img/recovery_user.jpg" alt=""></p> <p>Congratulations! SSO will now be enabled for HPE GreenLake platform as well as the HPE GreenLake for Private Cloud Enterprise application. Log out and on the HPE GreenLake platform home page, click <strong>Sign in with SSO</strong>.</p> </li> </ol> <p><strong>Testing and troubleshooting:</strong></p> <p>On the HPE GreenLake edge-to-cloud platform home page, click <strong>Sign In with SSO</strong>.</p> <p><img src="/img/ws-image15.png" alt=""></p> <p><img src="/img/hpe-greenlake-sso-page.jpg" alt=""></p> <p>Enter the SSO credentials. You will be redirected to Okta to authenticate. Once you successfully authenticate, you will be redirected back to HPE GreenLake platform. You can then click on the HPE GreenLake for Private Cloud Enterprise application and be given access based on the configured role/permissions.</p> <p><strong>Additional notes:</strong></p> <ul> <li>There must be at least <strong>one</strong> verified user belonging to the <strong>Domain</strong> prior to configuration.</li> <li>In order to configure SSO, you must be logged into the HPE GreenLake edge-to-cloud platform with a user from the domain.</li> <li>SSO user access is determined by the “role_name” attribute included in the SAML hpe_ccs_attribute provided by the IdP.</li> <li>For more troubleshooting: <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us</a></li> <li>Customer users should be given access to SAML application.</li> <li>After authentication when clicking the HPE GreenLake for Private Cloud Enterprise application**,** if it leads to the below error, it will take 1 hr to sync. If it does not do so within that time period, the customer should contact their HPE administrator. </li> </ul> <p>I hope this blog post answers any questions you may have had in regards to how to configure single sign-on for HPE GreenLake for Private Cloud Enterprise on the HPE GreenLake platform using Okta Active Directory. Please return back to the <a href="https://developer.hpe.com/blog">HPE Developer Community blog</a> for more tips and tricks on working with the HPE GreenLake platform.</p>https://developer.hpe.com/help-us-prioritize-features-for-future-releases-of-chapel/https://developer.hpe.com/help-us-prioritize-features-for-future-releases-of-chapel/Wed, 15 Nov 2023 18:36:57 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In a <a href="https://chapel-lang.org/blog/posts/announcing-chapel-1.32/">recent blog post</a>, the Chapel team gave a preview of Chapel 1.32, the release candidate for the official 2.0 version which will declare a core subset of the language and library features as "stable". These features are ones that we intend to support in their current form going forward, such that code relying on them will not break across releases.</p> <p>With this candidate release available, we thought now would be a good time to get a sense for what remaining currently unstable features are important to our user base, so that we can better prioritize what to stabilize next. To do so, we've created a Chapel program (<a href="https://github.com/chapel-lang/chapel/blob/main/tools/unstableWarningAnonymizer/unstableAnonScript.chpl">available here</a>) that will summarize the unstable warnings in your key programs. This summary will not include any identifying details like module or variable names, so even if your source code is not intended for public eyes, you should be able to send us the result of running this script without worry.</p> <p>This script is intended to be used with the most recent Chapel release (1.32), as that release has the most complete coverage of unstable features. If you use it with earlier releases, please let us know when providing your result file.</p> <p>Here's how to use this script:</p> <ol> <li>Compile the program linked above like you would any normal Chapel program. This should create an executable named 'unstableAnonScript'. For example:</li> </ol> <pre><code class="language-markdown">chpl unstableAnonScript.chpl </code></pre> <ol start="2"> <li>Compile and run your program as you would normally, with the addition of the compilation flag <code>--warn-unstable</code> (which will cause any use of unstable features to trigger a warning), and save the full output into a file.</li> </ol> <pre><code class="language-markdown">chpl --warn-unstable myProgram.chpl >myUnstableOutput.txt 2>&#x26;1 ./myProgram >>myUnstableOutput.txt 2>&#x26;1 </code></pre> <p>After these commands, <code>myUnstableOutput.txt</code> (or whatever you've named it) should contain any unstable warnings you may trigger in your code, as well as any other potential output that occurs when compiling and running your program.</p> <ol start="3"> <li>Run the built script over your output file. For our convenience, it would be helpful to run with the <code>--csv</code> and <code>--sorted</code> flags, or <code>-c</code> and <code>-d</code> if you want to use the shorter version of those flags.</li> </ol> <pre><code class="language-markdown">./unstableAnonScript --csv --sorted --inputFiles myUnstableOutput.txt --outputFile mySummary.csv or ./unstableAnonScript -c -d -i myUnstableOutput.txt -o mySummary.csv </code></pre> <p>Note that <code>--inputFiles</code>/<code>-i</code> can take multiple files, so if you have multiple chapel programs you'd like to share the results for, you can combine the results together by specifying the unstable warnings from all of those programs at the same time:</p> <pre><code class="language-markdown">./unstableAnonScript -c -d -n -i myUnstableOutput1.txt myUnstableOutput2.txt myUnstableOutput3.txt -o mySummary.csv </code></pre> <p>After any of these commands, <code>mySummary.csv</code> (or whatever you've named it) should contain a comma-separated list of the unstable warnings generated by your program(s) and their counts, sorted from most common to least. There should be no identifying information in this file, so at this point, it should be safe to send the file to us.</p> <p>You could also additionally run with the <code>--numFiles</code> flag (<code>-n</code> for short), which will include the number of different files where each unstable warning was generated:</p> <pre><code class="language-markdown">./unstableAnonScript --csv --sorted --numFiles --inputFiles myUnstableOutput.txt --outputFile mySummary.csv or ./unstableAnonScript -c -d -n -i myUnstableOutput.txt -o mySummary.csv </code></pre> <p>This information would be helpful for our metrics, but is not essential.</p> <p>Please send your resulting file to <a href="https://chapel.discourse.group/c/users/6">chapel users mailing list on Discourse</a>. If you would like to send it to just the Chapel team at HPE, you can either send it to the <a href="mailto:chapel+info@discoursemail.com">chapel+info@discoursemail.com</a> or to <a href="https://chapel.discourse.group/u/lydia">Lydia Duncan at Discourse</a> directly. Note that if you do not already have a Discourse account, you may be asked to create one to send it.</p> <p>If you have any questions or concerns about this, please don't hesitate to voice them.</p> <p>Thank you for your continued interest in Chapel, and for helping us prioritize on the features that matter most to you.</p>https://developer.hpe.com/programing-with-chapel-making-the-power-of-parallelism-and-supercomputers-more-accessible/https://developer.hpe.com/programing-with-chapel-making-the-power-of-parallelism-and-supercomputers-more-accessible/Tue, 14 Nov 2023 16:38:00 GMT<p>External blog</p>https://developer.hpe.com/2023-November-02/https://developer.hpe.com/2023-November-02/Thu, 02 Nov 2023 13:00:00 GMThttps://developer.hpe.com/highly-available-nfs-workload-on-hpe-greenlake-for-private-cloud-enterprise-using-serviceguard-for-linux/https://developer.hpe.com/highly-available-nfs-workload-on-hpe-greenlake-for-private-cloud-enterprise-using-serviceguard-for-linux/Mon, 30 Oct 2023 17:51:45 GMT<h1>Introduction</h1> <p>HPE GreenLake for Private Cloud Enterprise offers the ability to manage and provision compute through machine-readable definition files, otherwise known as Infrastructure-as-Code (IaC). This offers many significant benefits; for example, it helps increase operational agility, simplifies management, reduces errors, and saves cost.</p> <p>In this blog post, you will discover how to provision a highly available and scalable NFS server solution on HPE GreenLake for Private Cloud Enterprise platform in conjunction with HPE Serviceguard for Linux. This blog post covers provisioning of VMs and other required components using Terraform, and the second part talks about installing and configuring an NFS server and Serviceguard for Linux (SGLX) to provide a highly available NFS service.</p> <h2>HPE GreenLake for Private Cloud Enterprise</h2> <p>One of the options provided through HPE GreenLake is to make it easy for customers to order and operate a private cloud with a mix of virtual machines, containers, and physical servers. This service allows customers to create resources such as virtual machines using just a few button clicks. It provides access via a public API, allowing developers to use an Infrastructure-as-Code type of tool to automate the provisioning, i.e., Terraform.</p> <h2>Terraform</h2> <p>Terraform is an open-source Infrastructure-as-Code framework originally created by HashiCorp that is written in Go. It uses a declarative language (HashiCorp Configuration Language HCL or JSON more recently) to describe the desired state of the infrastructure in terms of cloud, virtual machines, networks, storage, and many other components. Terraform uses the concept of “providers” to integrate with all major public clouds. Terraform is a so-called idempotent system in the sense that it does not generate any side effects if applied multiple times on infrastructure already in the desired state. Terraform has gained quite the momentum in the last few years.</p> <h2>HPE Serviceguard for Linux</h2> <p>HPE Serviceguard for Linux (SGLX) is a high availability (HA) and disaster recovery (DR) clustering solution that increases uptime for your critical applications by protecting them from a multitude of infrastructure and application faults across physical or virtual environments over any distance. The solution also reduces the impact of unplanned downtime with no compromise on data integrity or performance, and it helps achieve near zero planned downtime for maintenance.</p> <h2>Ansible</h2> <p>Ansible Automation Platform is an end-to-end automation platform to configure systems, deploy software, and orchestrate advanced workflows. It includes resources to create, manage, and scale across the entire enterprise.</p> <h1>Preparing for an Infrastructure-as-Code implementation</h1> <h2>Terraform installation</h2> <p>The first step is to create a virtual machine that can act as a Jump Host and where you will execute your code. Virtual machines created in later steps will be reachable from your Jump Host if you configure your networking infrastructure appropriately. You can create a virtual machine in the Virtual Machines service of HPE GreenLake for Private Cloud Enterprise as follows:</p> <ol> <li>Log into <strong>HPE GreenLake Central</strong> by navigating to <a href="https://client.greenlake.hpe.com">https://client.greenlake.hpe.com</a></li> <li>Launch the <strong>Virtual Machines Service Console</strong></li> <li>Select <strong>Provisioning --> Instances</strong> and create a Linux-based virtual machine using whichever VM (Virtual Machine) image, Plan, and the network you find most suitable.</li> <li>(optional) Configure your virtual machine so you can access your GitHub or other code repository.</li> <li>Log into your virtual machine and install Terraform: Hashi Corp has provided a useful <a href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli">tutorial</a> on how to do this for various distributions of Linux</li> <li>Verify the installation by executing this command: <code>terraform --help</code></li> </ol> <p>At this point, you are ready to start building your infrastructure description file.</p> <h2>Building a Terraform configuration file from scratch</h2> <p>Let’s start building this Terraform (TF) file using your favourite editor.</p> <h3>Selecting a Terraform provider</h3> <p>The first section of the file will enumerate the “providers” you rely upon for building your infrastructure, and they could be multiple providers in a single TF file. In the case here, you only have the HPE GreenLake provider referenced as hpe/hpegl in the official Terraform registry. The first lines of your Terraform configuration file should look like this:</p> <pre><code class="language-json"># Load HPE GreenLake terraform provider terraform { required_providers { hpegl = { source = "hpe/hpegl" version = "0.3.17" } } } </code></pre> <p>You can find out more about the HPE GreenLake Terraform provider from its <a href="https://registry.terraform.io/providers/HPE/hpegl/latest">Terraform Registry page</a>. This page also provides a link to the GitHub repository corresponding to this provider. The docs folder is your best source of information for using the different data sources and resources provided by the provider. If you navigate to the resources section, you will see that one resource you can configure with this provider is a VM instance. This article will focus on this resource.</p> <p><em><strong>Note:</strong></em> Because this is open source, do not hesitate to open issues, or even a pull request, if you identify an issue.</p> <h3>Setting up the Terraform provider</h3> <p>Set up the required parameters for hpegl provider that was specified earlier. As previously explained, you can either explicitly set those parameters in your TF file or have them set in a series of environment variables or have a mix of both. It is recommended to add the following two parameters in your TF file:</p> <pre><code class="language-json"># Setup provider environment (location and space) provider "hpegl" { vmaas { location = "FTC06" space_name = "Default" } } </code></pre> <p>The rest (such as tenant id, user id and user secret key) can be placed in a RC file, which you can source before running your Terraform command. You can find your location and your space name from the HPE GreenLake for Private Cloud Enterprise Overview.</p> <p>In the example shown below, FTC06 is our location:</p> <p><img src="/img/picture1.png" alt=""></p> <p>In the capture below, Default is the space you will use for your work with Terraform. You can check your available Spaces from the HPE GreenLake console under your profile icon, Change Space.</p> <p><img src="/img/picture2.png" alt=""></p> <h2>Setting up API Client access</h2> <p>Next, you need to create a new API Client access dedicated to Terraform. You can do this from the HPE GreenLake console under your settings icon, select <strong>User Management</strong> and then the <strong>API Clients</strong> tab.</p> <p><img src="/img/picture3.png" alt=""></p> <p>Create a new API Client and be sure to note down the Issuer, Client ID and Client Secret values which are shown. The value for the tenant ID may be seen in the Tenant ID field under the API Access menu and in the URL of your browser.</p> <p><img src="/img/picture4.png" alt=""></p> <p>With this you can now build a resource file that defines the following environment variables:</p> <pre><code class="language-shell">export HPEGL_TENANT_ID=&#x3C;Your Tenant ID> export HPEGL_USER_ID=&#x3C;Client ID of the API Client> export HPEGL_USER_SECRET=&#x3C;Secret Key displayed when you created the API Client> export HPEGL_IAM_SERVICE_URL=&#x3C;Issuer URL> </code></pre> <p>And execute it on your machine to set these environment variables.</p> <h2>Assign Roles to API Client</h2> <p>Once your API Client has been created, you need to assign a Role and a Space. You can assign a Role and a Space by clicking on your new API Client and then clicking the Create Assignment button. Since intent is to use this API Client to create resources in the Virtual Machines Service, we need to assign an appropriate Virtual Machines Role. Choose a Role like ‘Private Cloud Tenant Contributor’ and choose the same Space as used earlier, I.e., ‘Default.’</p> <p><em><strong>Note:</strong></em> More details on HPE GreenLake user roles can be found in the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">HPE GreenLake documentation</a>.</p> <h3>Set API Client Usernames and Passwords</h3> <p>When a user creates virtual machines using the HPE GreenLake for Private Cloud Enterprise: Virtual Machines user interface, they first set the Linux and Windows username and password. Once this is done, any virtual machines subsequently created by that user will inherit these credentials. The user can later use these credentials to log into these virtual machines.</p> <p>API Clients which are used to create virtual machines can also set Linux and Windows username and password values. Since the API Client does not use the HPE GreenLake for Private Cloud Enterprise: Virtual Machines user interface, this must be done via an API call.</p> <p>Here is a sample script which reads the VM_USERNAME and VM_PASSWORD environment variables and uses the values for Linux and Windows username and password for the API Client. The script assumes a Location value of ‘FTC06’ and Space value of ‘Default’.</p> <p>To execute this script, first set appropriate values for the VM_USERNAME and VM_PASSWORD environment variables. Next, execute the resource file, which was created earlier, which sets the HPEGL** environment variables for your API Client.</p> <p>Finally, execute the script below:</p> <pre><code class="language-shell">#!/bin/bash export LOCATION='FTC06' export SPACE='Default' export SPACE_ENCODED=$(echo -n -e "$SPACE" | od -An -tx1 | tr ' ' % | xargs printf "%s") ACCESS_TOKEN=$(curl -s -k -X POST \ "${HPEGL_IAM_SERVICE_URL}/v1/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "client_id=${HPEGL_USER_ID}" \ -d "client_secret=${HPEGL_USER_SECRET}" \ -d grant_type=client_credentials \ -d scope=hpe-tenant | jq -r '.access_token') echo "Token: ${ACCESS_TOKEN}" curl -s -k -X GET \ "https://client.greenlake.hpe.com/api/iac-vmaas/v1/whoami?space=${SPACE_ENCODED}&#x26;location=${LOCATION}" \ -H "Authorization: ${ACCESS_TOKEN}" | jq '.' # Sets user settings curl -s -k -X POST \ "https://client.greenlake.hpe.com/api/iac-vmaas/v1beta1/user-settings?space=${SPACE_ENCODED}&#x26;location=${LOCATION}" \ -H "Authorization: ${ACCESS_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "user": { "linuxUsername": '${VM_USERNAME}', "linuxPassword": '${VM_PASSWORD}', "windowsUsername": '${VM_USERNAME}', "windowsPassword": '${VM_PASSWORD}' } }' </code></pre> <p>Querying for infrastructure components</p> <p>Your next step with the TF file is to query the HPE GreenLake provider to collect information needed to create your first VM instance. From the documentation, you can see that you need to gather the following information:</p> <p>• Cloud ID</p> <p>• Group ID</p> <p>• Layout ID</p> <p>• Plan ID</p> <p>• Instance type code</p> <p>• Network ID</p> <p>• Resource Pool ID</p> <p>• Template ID</p> <p>• Folder Code</p> <p>For this, you will use the Terraform data statements. For example, the following statement retrieves the Cloud ID and stores it (in variable called cloud), which we can later retrieve using: data.hpegl_vmass_cloud.cloud.id</p> <pre><code class="language-json"># Retrieve cloud id data "hpegl_vmaas_cloud" "cloud" { name = "HPE GreenLake VMaaS Cloud" } </code></pre> <p>Using a similar technique, you can retrieve the rest of the data you need:</p> <pre><code class="language-json"># And a network data "hpegl_vmaas_network" "blue_segment" { name = "Blue-Segment" } data "hpegl_vmaas_cloud_folder" "compute_folder" { cloud_id = data.hpegl_vmaas_cloud.cloud.id name = "ComputeFolder" } # Locate a resource pool data "hpegl_vmaas_resource_pool" "cl_resource_pool" { cloud_id = data.hpegl_vmaas_cloud.cloud.id name = "gl-ftc06-G2i-vm-02" } # And a group data "hpegl_vmaas_group" "default_group" { name = "Default" } # Locate a plan data "hpegl_vmaas_plan" "g2i_medium" { name = "G2i-medium" } # A layout data "hpegl_vmaas_layout" "vmware" { name = "Vmware VM" instance_type_code = "vmware" } # And a template data "hpegl_vmaas_template" "vanilla" { name = "redhat8-20220331T1850" } </code></pre> <p>You can get information about each of the data statements supported by the hpegl provider from <a href="https://github.com/hpe/terraform-provider-hpegl/tree/main/docs/data-sources">GitHub.</a></p> <h2>Creating VM resources</h2> <p>The next step is to use a Terraform resource statement to create a random integer (used in VM names) and a second resource to request the creation of several VM instances:</p> <pre><code class="language-json">resource "random_integer" "random" { min = 1 max = 50000 } resource "hpegl_vmaas_instance" " my_HA_NFS" { count = 2 name = "drbd-${count.index}-${random_integer.random.result}" hostname = "drbd-${count.index}-${random_integer.random.result}" cloud_id = data.hpegl_vmaas_cloud.cloud.id group_id = data.hpegl_vmaas_group.default_group.id layout_id = data.hpegl_vmaas_layout.vmware.id plan_id = data.hpegl_vmaas_plan.g2i_medium.id instance_type_code = data.hpegl_vmaas_layout.vmware.instance_type_code network { id = data.hpegl_vmaas_network.blue_segment.id } volume { name = "root_vol" size = 50 datastore_id = "auto" } volume { name = "drbd_vol" size = 50 datastore_id = "auto" } config { resource_pool_id = data.hpegl_vmaas_resource_pool.cl_resource_pool.id template_id = data.hpegl_vmaas_template.vanilla.id no_agent = false asset_tag = "vm_terraform_sglx" folder_code = data.hpegl_vmaas_cloud_folder.compute_folder.code create_user = true } } </code></pre> <p>Finally, we will create a VM to act as Serviceguard quorum node:</p> <pre><code class="language-jsoniq">resource "hpegl_vmaas_instance" "my_quorum" { count = 1 name = "drbd-${count.index}-qs-${random_integer.random.result}" hostname = "drbd-${count.index}-qs-${random_integer.random.result}" cloud_id = data.hpegl_vmaas_cloud.cloud.id group_id = data.hpegl_vmaas_group.default_group.id layout_id = data.hpegl_vmaas_layout.vmware.id plan_id = data.hpegl_vmaas_plan.g2i_medium.id instance_type_code = data.hpegl_vmaas_layout.vmware.instance_type_code network { id = data.hpegl_vmaas_network.blue_segment.id } volume { name = "root_vol" size = 50 datastore_id = "auto" } config { resource_pool_id = data.hpegl_vmaas_resource_pool.cl_resource_pool.id template_id = data.hpegl_vmaas_template.vanilla.id no_agent = false asset_tag = "vm_terraform_sglx_quorum" folder_code = data.hpegl_vmaas_cloud_folder.compute_folder.code create_user = true } } </code></pre> <p>3 VMs need to be created to setup SGLX. 2 VMs will be used to create Serviceguard for Linux nodes where the NFS service will be up and running. The third VM will act as a quorum server for the Serviceguard cluster to ensure that split brain of the cluster does not impact the availability of the monitored workload.</p> <p><em><strong>Note:</strong></em> You can get information about each of the resource statements supported by the hpegl provider from <a href="https://github.com/hpe/terraform-provider-hpegl/tree/main/docs/resources">GitHub</a>.</p> <p><em><strong>Note:</strong></em> An existing Serviceguard Quorum Server in your environment can be used instead of provisioning a third VM, provided the Quorum Server is reachable to the 2 VM’s that were created.</p> <h3>Terraform init</h3> <p>Before you can use Terraform, you need to initialize it from the configuration file we have created. This is done with the following step:</p> <p><code>terraform init</code></p> <h3>Terraform ready to plan</h3> <p>To validate your configuration file, it is recommended to run the terraform validate command. Once ready, the terraform plan command will provide the a summary of the deployment that would be built when terraform apply method would be used. Once you agree with the plan and confirm, you can apply the configuration.</p> <h3>Terraform ready to apply</h3> <p>The command you need to use is now:</p> <pre><code class="language-shell">terraform apply </code></pre> <p>This will rerun the plan command, then prompt you to confirm before it starts building what is in the plan. Here is some sample output from the <code>terraform apply</code> command:</p> <pre><code class="language-markdown">Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes hpegl_vmaas_instance.my_drbd[1]: Creating... hpegl_vmaas_instance.my_quorum[0]: Creating... hpegl_vmaas_instance.my_drbd[0]: Creating... hpegl_vmaas_instance.my_quorum[0]: Still creating... [10s elapsed] hpegl_vmaas_instance.my_drbd[1]: Still creating... [10s elapsed] hpegl_vmaas_instance.my_drbd[1]: Still creating... [10s elapsed] hpegl_vmaas_instance.my_drbd[1]: Creation complete after 2m8s [id=3105] hpegl_vmaas_instance.my_drbd[0]: Creation complete after 2m8s [id=3111] hpegl_vmaas_instance.my_quorum[0]: Creation complete after 2m8s [id=3108] Apply complete! Resources: 4 added, 0 changed, 0 destroyed. </code></pre> <p>Once the command completes, your virtual machines are ready.</p> <h1>Configuring a Highly Available NFS solution</h1> <p>Now that the VMs are provisioned, we can now deploy HPE Serviceguard for Linux on these VMs to create a cluster to provide high availability for the applications running on the VMs, NFS server in this case.</p> <h2>Installing Serviceguard for Linux</h2> <p>Serviceguard and all its components can be installed using Ansible playbooks. Clone the repository on ansible control node:</p> <pre><code class="language-shell">git clone https://github.com/HewlettPackard/serviceguard.git cd serviceguard/ansible-sglx </code></pre> <p>Checkout the stable branch. For ex: to checkout branch 1.0,</p> <p><code>git checkout Stable-v1.0</code></p> <p>To upgrade to the latest version of the playbooks:</p> <p><code>git pull https://github.com/HewlettPackard/serviceguard.git</code></p> <p>Master playbook <code>site.yml</code> contains the roles which will be executed for the inventory defined in hosts. When the master playbook is run, the version specified in the parameters file will be installed. The parameters for the master playbook, roles are configured in <code>group_vars/all.yml</code>.</p> <p>Now let's look into some of the fields in this file which needs to be configured. It’s time to configure the version of Serviceguard to be installed, in this case SGLX 15.10.00 will be installed.</p> <p><code>sglx_version : 15.10.00</code></p> <p>Provide the Serviceguard for Linux ISO location on the controller node.</p> <pre><code class="language-yaml">sglx_inst_upg_mode: iso sglx_inst_upg_additional_params: .. iso_params: iso_location: &#x3C;absolute path of the iso on ansible controller node> </code></pre> <p>Provide the Serviceguard Flex storage add-on ISO location on the control node.</p> <pre><code class="language-yaml"> storage_flex_add_on: install_upgrade: yes install_upg_mode: iso iso_params: iso_location: &#x3C;absolute path of the iso on ansible controller node> </code></pre> <p>Next, install Serviceguard NFS add-on.</p> <pre><code class="language-yaml">sglx_add_on_inst_upg_params: sglx_addon: nfs </code></pre> <p>Serviceguard installation mandates a replicated user configuration. As part of the installation, a replicated user for Serviceguard Manager (sgmgr) is created on the hosts and the password for the same can be configured under the below parameter.</p> <p><code>sglx_sgmgr_password: "{{ vault_sglx_sgmgr_password }}"</code></p> <p>Ansible vault will be used to encrypt this password, when you run the command as below:</p> <p><code>ansible-vault encrypt_string 'your_password' --name 'vault_sglx_sgmgr_password'</code></p> <p>The generated output must be substituted in</p> <pre><code class="language-yaml">vault_sglx_sgmgr_password: !vault | $ANSIBLE_VAULT;1.1;AES256 34363834323266326237363636613833396665333061653138623431626261343064373363656165 6639383863383633643035656336336639373161323663380a303331306337396435366535313663 31336636333862303462346234336138393135393363323739633661653534306162323565646561 6662396366333534350a663033303862646331613765306433353632316435306630343761623237 3863 </code></pre> <p>Once these parameters are populated, you can modify the hosts file to add the two VMs that were provisioned earlier where the cluster will be formed, and the quorum server that was provisioned earlier. In this case, it’s as shown below:</p> <pre><code class="language-yaml">[sglx-storage-flex-add-on-hosts] drbd-0-808 drbd-1-808 [sglx-cluster-hosts:children] sglx-storage-flex-add-on-hosts [quorum-server-hosts] drbd-0-qs-808 [primary] drbd-0-808 [secondary] drbd-1-808 </code></pre> <p>When the parameters specified above are configured, playbook <code>site.yml</code> can be run from the directory where the repository is cloned on the ansible control node.</p> <pre><code class="language-shell">cd serviceguard/ansible-sglx ansible-playbook -i hosts -v --vault-password-file &#x3C;path_to_vault_password_file> site.yml </code></pre> <p>This completes the Serviceguard software installation.</p> <h2>Configuring data replication using Serviceguard flex Storage Add-on</h2> <p>Serviceguard for Linux Flex Storage Add-on is a software-based, shared-nothing, replicated storage solution that mirrors the content of block devices. NFS server export data will be replicated to all Serviceguard cluster nodes using this add-on. The Ansible snippet below can be used to configure the replication.</p> <pre><code class="language-yaml">- hosts: sglx-storage-flex-add-on-hosts tasks: - name: Populate /etc/drbd.d/global_common.conf file become: True shell: | echo "global { usage-count yes; } common { handlers { disconnected /usr/local/cmcluster/conf/scripts/sgenss/replication_software/drbd/drbd_disconnect_handler.sh; } options { auto-promote no; } }" > /etc/drbd.d/global_common.conf - name: Create first part of drbd config file become: True shell: | echo "resource drbd0 {" >/etc/drbd.d/drbd0.res echo " disk /dev/sdb;" >> /etc/drbd.d/drbd0.res echo " device /dev/drbd0;" >> /etc/drbd.d/drbd0.res echo " meta-disk internal;" >> /etc/drbd.d/drbd0.res - name: Create second part of drbd config file become: True vars: this_nodename: "{{ hostvars[item]['ansible_hostname'] }}" shell: | echo " on {{ this_nodename }} {" >> /etc/drbd.d/drbd0.res echo " address {{ item }}:7789;" >> /etc/drbd.d/drbd0.res echo " node-id {{ my_index }};" >> /etc/drbd.d/drbd0.res echo " }" >> /etc/drbd.d/drbd0.res loop: "{{ ansible_play_batch }}" loop_control: index_var: my_index - name: Set initial empty mesh list set_fact: mesh: "" - name: Build list of nodes for connection-mesh entry loop: "{{ ansible_play_batch }}" set_fact: mesh: "{{ mesh + hostvars[item]['ansible_hostname'] + ' ' }}" - name: Check mesh nodes debug: var=mesh - name: Create connection-mesh portion of config file become: True shell: | echo " connection-mesh {" >> /etc/drbd.d/drbd0.res echo " hosts {{ mesh|trim }};" >> /etc/drbd.d/drbd0.res echo " }" >> /etc/drbd.d/drbd0.res - name: Create last part of drbd config file become: True shell: | echo "}" >> /etc/drbd.d/drbd0.res - name: Create drbd0 device become: True shell: | drbdadm create-md drbd0 when: res.rc != 0 - name: Start DRBD service become: True systemd: name: drbd enabled: True state: started - hosts: primary tasks: - name: Enable this node as Primary become: True shell: | drbdadm primary drbd0 --force when: res.rc != 0 </code></pre> <h2>Configuring LVM</h2> <p>Once data replication is configured on the nodes, you can configure LVM on top of the DRBD disk <em>/dev/drbd0</em>. The following Ansible snippet can be used to configure the LVM volume group named nfsvg and logical volume names nfsvol of size 45GB.</p> <pre><code class="language-yaml">--- - hosts: sglx-storage-flex-add-on-hosts tasks: - name: Modify lvm configuration become: True lineinfile: path: /etc/lvm/lvm.conf regexp: "# volume_list" line: volume_list=["nfsvg","nfsvg/nfsvol","@tag1","@*"] state: present backup: True - name: reject disk in lvm configuration become: True lineinfile: path: /etc/lvm/lvm.conf regexp: ".*/dev/cdrom.*" line: ' filter = [ "r|/dev/sdb|", "a|/dev/drbd0|" ] ' state: present backup: True - hosts: primary tasks: - name: Create a volume group on /dev/drbd0 become: True lvg: vg: nfsvg pvs: /dev/drbd0 - name: create logical volume for nfs become: True lvol: vg: nfsvg lv: nfsvol size: 45g force: True - name: Format filesystem become: True filesystem: dev: /dev/nfsvg/nfsvol fstype: xfs </code></pre> <h2>Setting up the NFS server</h2> <p>Now start the NFS service and export the NFS share from the primary node using the Ansible snippet below.</p> <pre><code class="language-yaml">--- - hosts: sglx-storage-flex-add-on-hosts tasks: - name: Install NFS Server and related components become: True ansible.builtin.yum: name: - nfs-utils state: present ignore_errors: True - name: Enable NFS related services become: True systemd: name: "{{ item }}" enabled: True with_items: - rpcbind - nfs-server - name: Start NFS related services become: True systemd: name: "{{ item }}" state: started with_items: - rpcbind - nfs-server - name: Add /etc/exports entry and create NFS mount point become: True shell: | mkdir -p /nfs chmod go+rwx /nfs echo '/nfs *(rw,sync,no_root_squash)' > /etc/exports - hosts: primary tasks: - name: mount nfs on primary become: True shell: | mount /dev/nfsvg/nfsvol /nfs exportfs -a </code></pre> <h2>Creating an SGLX cluster and providing HA to the NFS workload</h2> <p>Once NFS share is configured, look into creating an SGLX cluster and deploy the NFS workload in the SGLX environment to make it highly available. The below snippet will help you achieve this.</p> <pre><code class="language-yaml">--- - hosts: primary - name: Build string of primary nodes set_fact: primary_nodes: "{{ primary_nodes | default ('') + ' -n ' + hostvars[item].ansible_hostname }}" with_items: - "{{ groups['primary'] }}" - name: Build string of secondary nodes set_fact: secondary_nodes: "{{ secondary_nodes | default ('') + ' -n ' + hostvars[item].ansible_hostname }}" with_items: - "{{ groups['secondary'] }}" - name: Build string of quorum nodes set_fact: quorum_nodes: "{{ quorum_nodes | default ('') + ' -q ' + hostvars[item].ansible_hostname }}" with_items: - "{{ groups['quorum-server-hosts'] }}" - name: Run cmdeploycl command become: True ansible.builtin.expect: command: "$SGSBIN/cmdeploycl {{ primary_nodes }} {{secondary_nodes }} {{ quorum_nodes }}" responses: password: "{{ root_pass }}" timeout: 300 - name: Update cluster config become: True shell: | rm -rf /tmp/cluster.txt $SGSBIN/cmgetconf > /tmp/cluster.txt echo "GENERIC_RESOURCE_NAME CGR_SGeNSS_drbd" >> /tmp/cluster.txt echo "GENERIC_RESOURCE_TYPE simple" >> /tmp/cluster.txt echo "GENERIC_RESOURCE_CMD $SGSBIN/scripts/sgenss/replication_software/drbd/cluster_generic_resource.sh" >> /tmp/cluster.txt echo "GENERIC_RESOURCE_SCOPE node" >> /tmp/cluster.txt echo "GENERIC_RESOURCE_RESTART none" >> /tmp/cluster.txt echo "GENERIC_RESOURCE_HALT_TIMEOUT 10000000" >> /tmp/cluster.txt - name: Run cmapplyconf command become: True shell: | $SGSBIN/cmapplyconf -v -C /tmp/cluster.txt -f - name: Create a DRBD and NFS package become: True shell: | rm -rf /tmp/nfs_drbd.conf $SGSBIN/cmmakepkg -m sgenss/rf_drbd -m tkit/nfs/nfs /tmp/nfs_drbd.conf - name: update the drbd resource name become: True replace: path: /tmp/nfs_drbd.conf regexp: "{{ item.regexp }}" replace: "{{ item.rep }}" with_items: - { regexp: 'res0', rep: 'drbd0'} - name: Make change to package configuration become: True lineinfile: path: /tmp/nfs_drbd.conf regexp: "{{ item.regexp }}" line: "{{ item.line }}" state: present with_items: - { regexp: '^package_name', line: 'package_name nfs_drbd'} - { regexp: '^#vg', line: 'vg nfsvg'} - { regexp: '^tkit/nfs/nfs/XFS', line: 'tkit/nfs/nfs/XFS "-o rw,sync,no_root_squash *:/nfs"'} - { regexp: '^tkit/nfs/nfs/QUOTA_MON', line: 'tkit/nfs/nfs/QUOTA_MON no'} - name: Add additional NFS configuration become: True lineinfile: path: /tmp/nfs_drbd.conf insertafter: EOF line: | fs_name /dev/nfsvg/nfsvol fs_directory /nfs fs_type "xfs" fs_mount_opt "-o rw" ip_subnet 10.10.180.0 ip_address 10.10.180.99 - name: check the package and apply it become: True shell: | $SGSBIN/cmcheckconf -P /tmp/nfs_drbd.conf $SGSBIN /cmapplyconf -P /tmp/nfs_drbd.conf -f - name: enable the package become: True shell: | $SGSBIN /cmmodpkg -e nfs_drbd </code></pre> <p>The NFS server is now deployed in Serviceguard cluster with high availability.</p> <h1>Conclusion</h1> <p>In this blog, you learned how to use platforms like Terraform and Ansible to easily provision and deploy a highly available NFS server solution with Serviceguard for Linux on an HPE GreenLake Private Cloud Enterprise environment.</p> <p>Serviceguard for Linux has high availability and disaster recovery solutions available for various applications such as SAP, Oracle, SQL Server On Linux, EDB Postgres, etc. For more information regarding these solutions you can refer to the HPE Serviceguard for Linux operational guide for workloads and solutions available at <a href="https://www.hpe.com/info/linux-serviceguard-docs">https://www.hpe.com/info/linux-serviceguard-docs</a></p> <p>Check back on the <a href="https://developer.hpe.com/blog">HPE Developer Community blog</a> for more articles on HPE GreenLake for Private Cloud.</p>https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-scaling-and-orchestrating-modern-applications-for-the-enterprise/https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-scaling-and-orchestrating-modern-applications-for-the-enterprise/Thu, 19 Oct 2023 14:34:37 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my blog post on <a href="https://developer.hpe.com/blog/hpe-greenlake-for-private-cloud-enterprise-glpce-deploying-and-scaling-traditional-applications/">HPE GreenLake for Private Cloud Enterprise: Deploying and scaling traditional applications</a>, I highlighted how HPE GreenLake for Private Cloud Enterprise seamlessly integrates traditional applications with modern demands, transforming infrastructure into programmable code for optimal flexibility and security. Its strategic approach to scalability ensures businesses consistently operate at their best, making applications resilient to ever-changing requirements. In this post, I'll delve into the features and capabilities of HPE GreenLake for Private Cloud Enterprise, with a specific focus on its support for scaling containers using Kubernetes (K8s). Let's explore the advancements and offerings of the HPE GreenLake for Private Cloud Enterprise platform.</p> <h1><strong>Introduction</strong></h1> <p>In today’s rapidly evolving digital landscape, it's imperative for enterprises to have agile and secure solutions that can effortlessly adapt to emerging trends. HPE GreenLake for Private Cloud Enterprise: Containers ("containers service") offers a robust solution tailored to meet these specific demands. Containers service efficiently adjusts resources based on changing workloads, ensuring optimal use. This flexibility eliminates unexpected costs, allowing businesses to pay only for the resources they use. A unified dashboard makes this process even more transparent and manageable.</p> <p>HPE GreenLake for Private Cloud Enterprise enables seamless integration between on-premises infrastructure and cloud platforms. This capability ensures that, as workloads move or scale between these environments, the performance remains consistent, providing a dependable experience for users. It's not just about meeting current needs; it's about anticipating the future. By aligning with the latest standards and supporting innovative architectural trends, HPE GreenLake for Private Cloud Enterprise positions businesses at the cutting edge of technology. In essence, containers service is a harmonious blend of traditional and modern needs, paving a clear path for businesses to move confidently into the future with unmatched agility and security.</p> <h2><strong>Kubernetes (K8s) on HPE GreenLake Private Cloud Enterprise</strong></h2> <p>HPE GreenLake for Private Cloud Enterprise provides Kubernetes deployment via its container services. Notably, it's a CNCF-compliant K8s distribution, ensuring adherence to industry standards.</p> <p><strong>Key Points:</strong></p> <ol> <li>On-premises Kubernetes: It is designed for on-premises deployments with the ability to scale based on business needs.</li> <li>Ready to use: The platform is pre-configured, reducing setup time.</li> <li>Performance and security: It provides consistent performance and security for operations.</li> <li>Quick Kubernetes start: The platform features one-click provisioning that gets Kubernetes operations running fast.</li> <li>Central Kubernetes management: Its management console allows for easy control over Kubernetes clusters.</li> <li>Pay-as-you-go pricing: It uses a flexible pricing model where businesses pay for what they use.</li> </ol> <p>Containers service stands out by delivering the security, efficiency, and cost-effectiveness modern businesses seek, maximizing resource use. In tandem with Kubernetes, containers service provides a smooth and managed infrastructure. Enhanced by HPE Ezmeral Runtime Enterprise, it gives businesses a refined platform for deploying applications, ensuring scalability, and streamlined operations.</p> <p><strong>Under the hood – A fungible infrastructure resource pool</strong><br> As discussed in <a href="https://developer.hpe.com/blog/hpe-greenlake-for-private-cloud-enterprise--Exploring-a-flexible-infrastructure-resource-pool/">Part 1</a> of this blog series, HPE GreenLake for Private Cloud Enterprise integrates an adaptable infrastructure that leverages the strengths of its infrastructure resource pool and detailed capacity planning. This approach not only addresses present business requirements but also anticipates future needs. Within HPE GreenLake for Private Cloud Enterprise, the resource pool offers flexibility, letting businesses choose container deployments on bare metal, virtual machines, or a combination of both, based on their specific demands. Diving into the heart of HPE GreenLake for Private Cloud Enterprise: Containers, we encounter two primary deployment models for Kubernetes: on virtual machines and on bare metal.</p> <p>Kubernetes on bare metal:</p> <ul> <li>Tailored for expansive, consistent deployments where performance is paramount. * Efficiency takes center stage, devoid of VM-associated costs.</li> <li>Simplified management structure cuts down on overhead</li> <li>Perfect for business-critical applications or those where speed and responsiveness are crucial, like financial or imaging apps.</li> </ul> <p>Kubernetes on virtual machines:</p> <ul> <li>Delivers optimal performance in diverse environments, making it an ideal choice for development</li> <li>Flexibility in deployments with the convenience of having mixed workloads run simultaneously</li> <li>Typical applications include general-purpose web apps and event-driven microservices.</li> </ul> <p>In essence, whether your workload requires the adaptable environment of VMs or the robust power of bare metal, containers service ensures your Kubernetes deployments are optimized to your unique needs.</p> <h2><strong>K8s in Action on HPE GreenLake for Private Cloud Enterprise</strong></h2> <p>In the evolving landscape of enterprise IT, scalability isn't just a luxury; it's a necessity. With HPE GreenLake for Private Cloud Enterprise and Kubernetes at its helm, businesses are equipped to meet dynamic demands head-on. When administrators first work with HPE GreenLake, they can manually allocate resources based on anticipated needs. But as traffic unpredictably rises, Kubernetes springs into action, enabling it to scale. Kubernetes, with its sophisticated orchestration capabilities, monitors workloads in real-time on HPE GreenLake for Private Cloud Enterprise. Should traffic surge unexpectedly, Kubernetes autonomously scales containers, ensuring optimal performance without overburdening resources. The combined prowess of manual fine-tuning with Kubernetes' automated scalability represents the future-forward approach of HPE GreenLake for Private Cloud Enterprise, promising enterprises reliability, efficiency, and adaptability all in one package.</p> <p><strong>Kubernetes Cluster blueprints:</strong></p> <p><img src="/img/screenshot-4-creation-of-kubernetes-cluster-blueprint.png" alt="Screenshot 1: Creation of Kubernetes cluster blueprint" title="Screenshot 1: Creation of Kubernetes cluster blueprint"></p> <p>In screenshot 1, Kubernetes cluster blueprints in containers service serve as templates to simplify cluster deployments, including K8s setups. They set configurations like the Kubernetes version, storage class, and node details. Blueprints ensure consistent deployments and allow users to use standard templates or create their own for specific needs. Let me show you a concise demo on how to effortlessly configure clusters and optimize auto-scaling parameters!</p> <p><strong>How to create K8s clusters and scale worker nodes (up or down) in a running cluster</strong></p> <p><img src="/img/screenshot-2-k8s-cluster-creation-and-configuring-scaled-worker-nodes.png" alt="Screenshot 2: K8s Cluster creation and configuring scaled worker nodes" title="Screenshot 2: K8s Cluster creation and configuring scaled worker nodes"></p> <p>In screenshot 2 , I created a cluster. I selected configurations and resources, set parameters, and containers service provisions and initialized the cluster.</p> <p><strong>Scaling worker nodes in containers service</strong>: With containers service, you can scale worker nodes based on workload. You just increase or decrease the number of worker nodes in a running cluster to align with resource requirements.</p> <p><strong>Autoscaler in Kubernetes clusters</strong>: The Autoscaler adjusts the Kubernetes cluster's size based on specific conditions. It scales up when pods can't be scheduled due to resource limitations and scales down if nodes are underutilized for over 10 minutes.</p> <p><strong>Key points:</strong></p> <ul> <li>Scaling range: Defined by a minimum and maximum node count. The range is between 1 and 200 nodes. By default, autoscaling is off with equal min-max values.</li> </ul> <p><strong>Scaling criteria</strong>:</p> <ol> <li> <p>Pending pods due to limited resources trigger a scale-up.</p> </li> <li> <p>Underutilized nodes for 10 minutes prompt a scale-down, but certain conditions can prevent this:</p> <ul> <li>Recent scale-up within 10 minutes</li> <li>Failed scale-down in the last 3 minutes</li> <li>Nodes with critical pods or those facing specific constraints</li> </ul> </li> <li> <p>Autoscaler operates from the control plane, checking conditions every minute.</p> <p><img src="/img/screenshot-3-scale-cluster.png" alt="Screenshot 3: Scale cluster" title="Screenshot 3: Scale cluster"></p> </li> </ol> <p><strong>How to use configure:</strong></p> <ol> <li>Ensure the cluster is ready and familiarize yourself with autoscaler guidelines.</li> <li>Navigate to HPE GreenLake for Private Cloud Enterprise > Containers > Selected Cluster.</li> <li>Choose Scale from Actions.</li> <li>Set the min-max node count values to either enable or disable autoscaling as shown in Screenshot 3.</li> <li>Optionally, add or remove node pools.</li> <li>To see the autoscaler logs, utilize the given kubectl command.</li> </ol> <p><strong>Conclusion – The perfect synergy</strong><br> HPE GreenLake for Private Cloud Enterprise, in conjunction with Kubernetes, addresses a broad spectrum of enterprise applications. Whether dealing with brownfield applications that have evolved over time or greenfield applications that are freshly developed, this combination ensures seamless integration and deployment.</p> <p>Containers service’s ability to scale resources up or down based on workload demands ensures that businesses can respond effectively to varying operational requirements. Additionally, the integrated framework provides a secure stack, reinforcing infrastructure integrity, governance, compliance, and application security. In essence, the union of HPE GreenLake for Private Cloud Enterprise and Kubernetes provides a comprehensive solution that caters to both existing and new enterprise applications, fostering a flexible, responsive, and secure environment.</p>https://developer.hpe.com/announcing-chapel-1-32/https://developer.hpe.com/announcing-chapel-1-32/Thu, 05 Oct 2023 17:16:19 GMT<p>External blog</p>https://developer.hpe.com/discover-the-power-of-data-center-monitoring-using-redfish-telemetry-and-cloud-native-tooling/https://developer.hpe.com/discover-the-power-of-data-center-monitoring-using-redfish-telemetry-and-cloud-native-tooling/Thu, 05 Oct 2023 17:03:36 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Monitoring data center infrastructure is critical to ensuring optimal performance, resource utilization, and timely issue detection. Redfish, an open industry standard for hardware management, provides a standardized way to access telemetry data from servers and other devices. Coupling Redfish telemetry with cloud-native monitoring tools offers a robust solution for real-time monitoring, data analysis, visualization, and alerting.</p> <h2><strong>Why does data center monitoring matter?</strong></h2> <p>Data centers are the backbone of modern businesses, housing critical applications, databases, and services. Ensuring the seamless operation of these data centers is essential for business continuity and meeting the demands of today's digital world. Data center monitoring helps with following:</p> <ul> <li><strong>Performance optimization:</strong> Monitoring helps identify bottlenecks, inefficiencies, and potential failures, allowing for proactive optimization.</li> <li><strong>Resource utilization:</strong> Tracking resource usage ensures that capacity is allocated effectively, saving costs and energy.</li> <li><strong>Predictive maintenance:</strong> Real-time insights enable predictive maintenance, reducing downtime and associated costs.</li> <li><strong>Compliance:</strong> Monitoring helps meet regulatory requirements by maintaining accurate records and ensuring security.</li> </ul> <h2><strong>A comprehensive approach</strong></h2> <p>My latest technical whitepaper, <em>“<a href="https://www.hpe.com/psnow/doc/a00134351enw">Data center monitoring using Redfish telemetry and cloud-native tooling</a>”</em>, presents a comprehensive approach to data center monitoring by integrating the Redfish telemetry with cloud-native open-source tools including Telegraf, Prometheus, Alertmanager, and Grafana. These tools work seamlessly together to provide a holistic view of your data center infrastructure. The telemetry data source for this stack is HPE Integrated Lights-Out (iLO) which exposes metrics via the Redfish interface.</p> <p>Here's a glimpse of what you'll discover in the whitepaper:</p> <ul> <li><strong>Understanding Redfish telemetry:</strong> Learn about the Redfish standard and how it simplifies hardware telemetry data collection.</li> <li><strong>Cloud-native tools:</strong> Explore the capabilities of Telegraf, Prometheus, Alertmanager, and Grafana and how they enhance data center monitoring.</li> <li><strong>Architecture:</strong> Dive into the architecture of this integrated solution, detailing how data flows from the source to the visualization layer.</li> <li><strong>Setup process:</strong> Follow a step-by-step guide on how to set up this monitoring solution in your data center.</li> <li><strong>Benefits:</strong> Understand the benefits of this approach, including real-time monitoring, scaling, and customizable visualization.</li> </ul> <h2><strong>Take the next step</strong></h2> <p>Discover how the integration of Redfish telemetry service with cloud-native tools can transform your data center monitoring, making it not only efficient but also highly insightful by reading my whitepaper and dive into the world of data center monitoring with HPE iLO today! Keep coming back to the <a href="https://developer.hpe.com/blog">HPE Developer blog</a> for more articles on HPE iLO.</p>https://developer.hpe.com/2023-October-05/https://developer.hpe.com/2023-October-05/Thu, 05 Oct 2023 13:00:00 GMThttps://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-glpce-deploying-and-scaling-traditional-applications/https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise-glpce-deploying-and-scaling-traditional-applications/Tue, 03 Oct 2023 03:45:24 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my first blog post on <a href="https://developer.hpe.com/blog/hpe-greenlake-for-private-cloud-enterprise--Exploring-a-flexible-infrastructure-resource-pool/">HPE GreenLake for Private Cloud Enterprise: Exploring a flexible infrastructure resource pool</a> I underscored the flexible nature of HPE GreenLake for Private Cloud Enterprise infrastructure. Its transparent cost analytics stand out as a significant benefit, aiding organizations in making well-informed financial and infrastructural choices. In our present digital era, having the right infrastructure is imperative, a need that HPE GreenLake for Private Cloud Enterprise aptly fulfills through its robust support for automated tools like Terraform and Ansible, ensuring seamless and efficient infrastructure management and scaling.</p> <p>In this second segment, I will delve further into the enhanced features of HPE GreenLake for Private Cloud Enterprise: Virtual Machines. This exploration will highlight the substantial support these services extend to modern businesses. By demystifying the intricacies of deploying and efficiently scaling traditional applications, HPE GreenLake for Private Cloud Enterprise stands as a pivotal ally for contemporary enterprise operations.</p> <p><strong>Introduction</strong></p> <p>In the current business landscape, organizations face the intricate task of integrating their applications, built on traditional time-tested and trusted technology and processes, like three-tier architectures, with the rapid pace and adaptability of modern applications, like containers and serverless computing. These older technologies have proven their reliability and effectiveness over time, forming the backbone of many enterprise systems. Meanwhile, applications have become modernized, emphasizing scalability, agility, and efficiency, and often relying on cloud-native architectures and services to deliver seamless, robust functionality.</p> <p>Navigating the integration of these disparate systems requires a thoughtful approach, balancing the stability and reliability of traditional architectures with the innovative features of modern application development and deployment strategies. This blending ensures organizations can harness the full potential of both technological paradigms to drive operational excellence and competitive advantage in the marketplace. Addressing this challenge, the HPE GreenLake for Private Cloud Enterprise: Virtual Machines serves as a comprehensive platform. It is designed to support the simultaneous operation and integration of both traditional and modern application types.</p> <p><strong>Traditional architecture</strong></p> <p>Figure 1 illustrates a three-tier architecture with the pivotal roles of various components. Their collective functionality in ensuring consistent application responsiveness, user engagement, and efficient processing of requests is highlighted. The architecture is delineated into the Load Balancer, the web-mobile tier, the application/business Logic tier, and the database tier, each playing a significant role in ensuring the smooth and efficient functioning of the application. Amidst this backdrop, HPE GreenLake for Private Cloud Enterprise emerges as a significant facilitator for deploying traditional applications. It streamlines the complexities intertwined with deployment, enhancing the synchronization between different architectural tiers.</p> <p>The real advantage that HPE GreenLake for Private Cloud Enterprise delivers lies in its ability to deploy essential components like Tier 0 (top or external tier) and Tier 1 (middle or distribution tier) routers, load balancers, and firewalls. This ability enables customers to seamlessly establish a three-tier application in the private cloud environment, ensuring robustness and efficiency in their operations. Beyond this, HPE GreenLake for Private Cloud Enterprise empowers customers to construct a comprehensive blueprint of a three-tier application, leveraging it for consistent and repeatable deployments. This approach not only simplifies deployment challenges but also enhances the efficiency and reliability of deploying traditional three-tier applications in a cloud environment.</p> <p><img src="/img/a-traditional-n-tier-application-to-be-hosted-on-virtual-machine-service.png" alt="Figure 1: A Traditional n-tier application to be hosted on Virtual Machine Service" title="Figure 1: A Traditional n-tier application to be hosted on Virtual Machine Service"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 1: A Traditional n-tier application to be hosted on Virtual Machine Service </span></p> </blockquote> </blockquote> <p>By integrating these capabilities, HPE GreenLake for Private Cloud Enterprise stands out as a solution that not only supports the deployment of traditional applications but also underscores the advancements in modern applications, ensuring seamless cloud transitions and robust integration. It adeptly positions enterprises to navigate the intricate waters of technological evolution, ensuring they remain at the forefront of innovation and operational efficiency.</p> <p><strong>Deploying and scaling traditional applications</strong></p> <p>The Virtual Machine service enables the utilization of virtual machine infrastructure through code, offering support for tools like Terraform and Ansible. This promotes rapid deployments, uniformity, and enhanced scalability. It also strengthens security and consistency by reducing manual steps, priming businesses for efficient digital asset management. In this blog post, I’ll help you navigate the HPE GreenLake for Private Cloud Enterprise console to initialize assets for hosting and deploying code. I begin by setting up web layer instances with the compute-optimized (c2i) type and scale them twice based on memory thresholds. Using an Ansible playbook, I will be able to automate tasks and incorporate a load balancer to prevent any single point of failure. Although the focus is on port 80 for HTTP, options for port 443, SSL certificates, and other TCP ports are available for diverse cluster scaling. Screenshots will illustrate instance launches upon reaching scaling thresholds.</p> <p>Next, I will explore the two key phases of application deployment and scaling:</p> <h3>1. Design and Development:</h3> <p>In this section, I guide you through managing groups and establishing scaling parameters for CPU and memory. During the Design and Development phase, initiate hosting via the HPE GreenLake for Private Cloud Enterprise console and configure web layer instances using the c2i type. With Ansible's help, introduce threshold-based scaling and incorporate a load balancer for uninterrupted service. While we'll emphasize port 80, several other ports are accessible as well.</p> <h3>2. Runtime</h3> <p>In the Runtime phase, as instances reach scaling thresholds, they are activated and tracked. For detailed insights, I provide a series of screenshots in this post.</p> <p>Now lets begin by login into the HPE GreenLake for Private Cloud Enterprise console, then head to the Virtual Machines section displayed on the dashboard.</p> <p><strong>Design and development phase</strong>:</p> <ol> <li> <p>Creating a Group: Start by accessing the Virtual Machines link on the dashboard. Click on <strong>Launch Service console</strong> and then select <strong>Groups</strong> from the <strong>Infrastructures</strong> dropdown. Here, initiates the creation of a designated group, as illustrated in screenshot 1. This approach allows you to bundle associated resources, ensuring streamlined management and clear oversight.</p> <p><img src="/img/managing-groups-and-setting-scaled-thresholds-for-cpu-and-memory.png" alt="Screenshot 1: Managing groups and setting scaled thresholds for CPU and memory." title="Screenshot 1: Managing groups and setting scaled thresholds for CPU and memory."></p> </li> </ol> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 1: Managing groups and setting scaled thresholds for CPU and memory. </span></p> </blockquote> </blockquote> <ol start="2"> <li> <p>Setting scale threshold: Navigate to <strong>Library</strong> and choose <strong>Scale</strong> <strong>threshold</strong>. Here is where you will define specific criteria for automatic scaling as shown in screenshot 1. With this in place, the system self-adjusts, activating anywhere from 1 to 4 instances based on memory consumption. When memory usage is minimal, the system conserves resources by operating fewer instances. Conversely, as memory usage approaches its limit, the system scales upwards, guaranteeing consistent performance without the need for human input.</p> </li> <li> <p>Create NSX-T Load Balancer: Screenshot 2 shows creating the <strong>Scale-Post-LB</strong> load balancer, set to <strong>Small</strong> size and <strong>Enabled</strong>, is in an Up administrative state. Connected to the main network router, <strong>Tier-1 Gateway</strong>, it logs activity at a Warning level. <img src="/img/nsx-t-load-balancer.png" alt="Screenshot 2: NSX-T Load Balancer" title="Screenshot 2: NSX-T Load Balancer"></p> </li> </ol> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 2: NSX-T Load Balancer </span></p> </blockquote> </blockquote> <p>This tool distributes traffic across servers, enhancing performance and reliability, while granting access to all groups.</p> <ol start="4"> <li>Instance provisioning: Screenshot 3 takes you through the detailed steps of setting up an instance:</li> </ol> <p>a. Initiating instance creation: Begin by navigating to <strong>Provisioning</strong>, then select <strong>Instances</strong> and click on Create Instance. At this juncture, identify and assign the right group for your resources. It is now also the time to detail key attributes, such as the instance name, its operating environment, and any relevant labels. This meticulous labeling aids in precise resource tracking and management.</p> <p><img src="/img/screenshot-3-creating-and-configuring-instance.png" alt="Screenshot 3: Creating and configuring instance" title="Screenshot 3: Creating and configuring instance"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 3: Creating and configuring instance </span></p> </blockquote> </blockquote> <p>b. Layout configuration: Subsequently, shift your focus to integrating the instance with a load balancer. While many prominent load-balancing options are available, including the likes of F5, the NSX-T Load Balancer (screenshot 2) native to HPE GreenLake for Private Cloud Enterprise is a good fit. It's equipped with a comprehensive set of features tailored to meet our deployment requirements, ensuring even distribution of incoming traffic across multiple endpoints.</p> <p>c. Final settings: Concluding the process, determine the root volume size to match your data storage necessities. Concurrently, select and designate the network on which these instances will function. This guarantees flawless interaction and operation within the stipulated digital ecosystem.</p> <p><img src="/img/configuring-automation-scale-factor-and-load-balancer.png" alt="Screenshot 4: Configuring automation, scale factor and load balancer" title="Screenshot 4: Configuring automation, scale factor and load balancer"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 4: Configuring automation, scale factor and load balancer </span></p> </blockquote> </blockquote> <p>In Screenshot 4, as you navigate the automation phase of the instance wizard:</p> <ol> <li> <p>Initiate the workflow configuration, opting for Ansible based on your needs. It's worth noting the availability of a Node.js workflow alternative.</p> </li> <li> <p>The Scale type selection follows, referencing the threshold previously established in Screenshot 1. For clarity, you can revisit Screenshot 1.</p> </li> <li> <p>In Screenshot 2, you’ll see how to set up an NSX-T Load Balancer. Port 80 is primarily designated for HTTP traffic. While port 80 is the focus, other ports like 443 for HTTPS or custom ports for database tasks (e.g., 1433, 1521) can be configured. It's crucial to note that, initially, configurations apply only to the servers behind the load balancer. When a new instance arises due to scaling thresholds, it inherits these settings but isn't automatically added to the load balancer. A separate script is required to include the new instance to the load balancer.</p> </li> <li> <p>It's essential to highlight that activating port 443 for HTTPS provides the capability to configure and implement SSL certificates, bolstering security.</p> </li> <li> <p>Additionally, there's flexibility to choose the installation of custom agents on the instances, such as third-party monitoring tools or security vulnerability scanners.</p> <p><img src="/img/tagging-the-resources..png" alt="Screenshot 5: Tagging the resources." title="Screenshot 5: Tagging the resources."></p> </li> </ol> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 5: Tagging the resources. </span></p> </blockquote> </blockquote> <p>In Screenshot 5, you’ll notice an invaluable feature: the ability to create tags during the instance setup. Tagging resources allows organizations to effortlessly track and evaluate their resource consumption, providing insights into patterns, trends, and potential areas of optimization. Furthermore, these tags become indispensable when delving into spend analytics, helping to allocate and manage costs effectively. This functionality is especially crucial within the HPE GreenLake suite, as it promotes both transparency and strategic decision-making for businesses.</p> <p><strong>Runtime</strong>: Observe screenshot 6, where you’ll notice instances are activated and are attached to load balancer.</p> <p><img src="/img/provisioning-%E2%80%93-2-instances-up-and-attached-to-load-balancer.png" alt="Screenshot 6: Provisioning – 2 instances up and attached to load balancer." title="Screenshot 6: Provisioning – 2 instances up and attached to load balancer."></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 6: Provisioning – 2 instances up and attached to load balancer. </span></p> </blockquote> </blockquote> <p>Provisioning: Screenshot 6 shows the final phase of the instance wizard: the Review screen. This overview permits users to review and affirm their established configurations. By clicking the complete button, the provisioning process is activated. Although the instances themselves do not undergo optimization, the earlier described workflow and automation contribute to enhancing application performance. This enhancement is achieved by efficiently scaling out or scaling in the number of virtual machines based on the predefined scale factor, ensuring optimal application operation without manually adjusting the VM count.</p> <p><img src="/img/scaled-up-vms.png" alt="Screenshot 7: Scaled up VMs" title="Screenshot 7: Scaled up VMs"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 7: Scaled up VMs. </span></p> </blockquote> </blockquote> <p>Scaling observation: Now, draw your attention to the tripartite progression of instance scaling that is shown in Screenshot 7.</p> <ol> <li><strong>Initial state</strong>: This stage represents the baseline, illustrating the system's configuration before any significant memory demand. It's the foundation, where the system awaits triggers or conditions to initiate any scaling.</li> <li><strong>Intermediate phase</strong>: During the transition, memory demand escalates and meets the scaling threshold. This juncture is pivotal — the system responds by initializing new instances. It's an adaptive phase, wherein the system's agility becomes evident, reacting in real-time to the increasing demands.</li> <li><strong>Final state</strong>: The culmination is a full display of responsiveness. With memory utilization hitting its designated peak, the system maximizes its resources. Four active instances, mirroring the set scaling threshold, now operate cohesively. This not only reflects an adaptive system but also showcases a setup geared for efficiency and robust performance, ensuring user demands are consistently met.</li> </ol> <p><strong>Conclusion</strong></p> <p>In this post, I showed you how HPE GreenLake for Private Cloud Enterprise: Virtual Machines simplifies the deployment and scaling of traditional applications, making it an efficient choice for modern digital business needs. The Design and Development phase highlights the ease of managing groups and setting scaling thresholds for CPU and memory, emphasized by the detailed visual walkthrough.</p> <p>Through the HPE GreenLake for Private Cloud Enterprise console and Ansible automation, the establishment of a robust and efficient scaling environment is showcased, ensuring operational consistency and responsiveness.</p> <p>In the Runtime phase, the system’s agility comes to the forefront. The seamless integration with NSX-T Load Balancer and clear, step-by-step provisioning and setup underscore the system’s focus on optimal performance and reliability. The nuanced instance setup process, from initiation to layout configuration and tagging, exemplifies the comprehensive control and oversight organizations hold over their resources and deployment processes.</p> <p>In conclusion, the HPE GreenLake for Private Cloud Enterprise stands out as a robust, intuitive, and adaptive platform for efficiently deploying and scaling traditional applications. This detailed overview underscores the significant value it brings to enterprises, ensuring consistent meeting of user demands while promoting operational efficiency and innovation. Stay tuned for the final post in this series.</p>https://developer.hpe.com/the-ilorest-tool-in-conjunction-with-directly-attached-and-controller-connected-drives/https://developer.hpe.com/the-ilorest-tool-in-conjunction-with-directly-attached-and-controller-connected-drives/Sat, 30 Sep 2023 06:13:14 GMT<p><strong>Introduction</strong></p> <p>In the domain of data storage and server setups, the method of connecting hard drives holds considerable sway over performance, scalability, and the general operational capabilities. Two prevalent techniques for linking hard drives to a server or storage system include direct-attached and controller-connected setups. This article will delve into a comprehensive examination of these two methodologies, investigating their respective merits, practical applications, and their impact on contemporary computing environments. Additionally, it will focus on examining how the iLORest tool facilitates Drive Firmware (FW) updates in both of these configurations.</p> <p><strong>Direct-attached drives</strong></p> <p>Direct-attached drives, as the name suggests, are storage devices that are directly connected to a server or host system. These drives can be connected using various interfaces, such as SATA (Serial Advanced Technology Attachment), NVMe (Non-Volatile Memory Express), or SAS (Serial Attached SCSI). Let's examine the key aspects of direct-attached drives:</p> <ol> <li><strong>Simplicity</strong>: Direct-attached drives are straightforward to set up and manage. They are typically installed inside the server's chassis or externally connected via cables.</li> <li><strong>Performance</strong>: These drives often offer excellent performance, especially when used in high-speed interfaces like NVMe. They are well-suited for applications that demand low latency and high throughput.</li> <li><strong>Scalability</strong>: While direct-attached drives are easy to install, their scalability is somewhat limited. Expanding storage capacity often requires physically adding more drives to the server, which may not be suitable for large-scale data storage needs.</li> <li><strong>Use cases</strong>: Direct-attached drives are commonly used in small to medium-sized businesses, as well as for specific applications where high-speed local storage is essential, such as gaming servers or databases.</li> </ol> <p>When updating configurations where direct-attached drives are being used, iLORest automatically identifies drives that are directly attached, initiates the upload of the firmware component, and generates a UEFI task to perform the update. The UEFI task is scheduled to execute during the next server reboot, and the firmware update is carried out at that time as well.</p> <p><strong>Controller-connected drives</strong></p> <p>Controller-connected drives, on the other hand, involve a more complex setup. These drives are connected to a storage controller or RAID (Redundant Array of Independent Disks) controller, which is then connected to the server. This controller manages the storage devices and can offer several advantages:</p> <ol> <li><strong>Scalability</strong>: Controller-connected drives are highly scalable. Storage controllers can handle a large number of drives, allowing for significant storage expansion without the need to clutter the server with additional physical drives.</li> <li><strong>Data protection</strong>: Many controller systems offer RAID configurations, which provide data redundancy and protection against drive failures. This is crucial for mission-critical applications and data centers.</li> <li><strong>Centralized management</strong>: Storage controllers often come with management interfaces that allow for centralized monitoring, configuration, and maintenance of all connected drives. This simplifies storage administration in large-scale environments.</li> <li><strong>Use cases</strong>: Controller-connected drives are commonly used in enterprise environments, data centers, and any scenario where data reliability, scalability, and centralized management are paramount.</li> </ol> <p>In this particular setup, iLORest identifies drives connected to the controller, initiates the upload of the firmware component, and then proceeds to flash it directly using iLO (BMC). In this configuration, there is no necessity to restart the server.</p> <p>However, if the server incorporates a mix of direct-attached and controller-connected drives, both the direct flashing and UEFI task become essential. Consequently, a reboot is needed to finalize the firmware update process.</p> <p><strong>Choosing the right attachment method</strong></p> <p>The choice between direct-attached drives and controller-connected drives depends on your specific requirements and use cases. Here are some considerations to guide your decision:</p> <ul> <li><strong>Performance vs. Scalability</strong>: If you need high-performance local storage and have a limited number of drives, direct-attached drives may be sufficient. However, if scalability is crucial and data protection is a concern, controller-connected drives are a better choice.</li> <li><strong>Data redundancy</strong>: For applications where data loss is unacceptable, such as financial systems or healthcare databases, controller-connected drives with RAID configurations offer a higher level of data protection.</li> <li><strong>Cost</strong>: Direct-attached drives are often more cost-effective for small to medium-sized deployments. Controller-connected drives can be more expensive due to the additional hardware required.</li> <li><strong>Management complexity</strong>: Consider the level of management and administration your storage solution requires. Controller-connected drives offer centralized management but may require more initial setup.</li> </ul> <p>In summary, gaining a comprehensive understanding of how the iLORest tool updates firmware for both direct-attached and controller-connected drives is crucial. This knowledge forms the basis for making well-informed decisions when designing and overseeing your storage infrastructure. Each approach carries its own set of advantages and limitations, and your choice should align with the specific requirements and objectives of your organization. Whether you prioritize aspects such as performance, scalability, data protection, or cost-effectiveness, the selection of the appropriate hard drive attachment method can greatly influence the effectiveness of your IT infrastructure.</p>https://developer.hpe.com/streamline-and-optimize-ml-workflows-with-hpe-ezmeral-unified-analytics/https://developer.hpe.com/streamline-and-optimize-ml-workflows-with-hpe-ezmeral-unified-analytics/Wed, 27 Sep 2023 13:00:00 GMT<p>This is part two of a blog series that showcases the capabilities of <a href="https://www.hpe.com/us/en/hpe-ezmeral-unified-analytics">HPE Ezmeral Unified Analytics</a> through the real-world example of stock market predicting. In that <a href="https://developer.hpe.com/blog/seamless-data-engineering-for-financial-services/">previous blog post</a>, we covered the data engineering aspects of Apache Spark and Superset to streamline pipelines and visualize data. In this post, the same use case is retained, stock market forecasting, to showcase how HPE Ezmeral simplifies building, deploying, and productizing machine learning models and pipelines. </p> <p><strong>Model building and training</strong></p> <p>Business problem: Forecast stock prices of different companies listed in the National Stock Exchange (NSE) of India.</p> <p>Step 1 Data Gathering:<br> The data is streamed from external servers hosted publicly and then saved to a HPE Ezmeral Data Fabric volume as explained in the previous blog.</p> <p>Step 2 Data Preprocessing:<br> The stock market does not function on holidays and weekends but the model expects continuous data. For this reason, the data feature, along with the open and closing price for each stock, is being used to impute the model with the previous working day’s data.</p> <p><img src="/img/step2-data-preprocessing.png" alt="" title="Figure 1. Selecting data for preprocessing. "></p> <p>Step 3 Modeling:<br> First, the data is divided between training and validation and then used to train a long short term memory (LSTM) model. After training, the model is evaluated on the error metric.</p> <p>LSTM is a variety of <a href="https://www.techtarget.com/searchenterpriseai/definition/recurrent-neural-networks#:~:text=A%20recurrent%20neural%20network%20is,predict%20the%20next%20likely%20scenario.">recurrent neural networks (RNNs)</a> capable of learning long-term dependencies, especially in sequence prediction problems. </p> <p><strong>Model tracking and monitoring using MLflow</strong></p> <p>MLflow is an open-source platform designed to manage the end-to-end machine learning lifecycle. It provides tools for tracking experiments, packaging code, sharing models, and managing deployment. MLflow was developed by Databricks, a company that specializes in big data and AI. They have gained popularity within the machine learning community for their ability to streamline and organize various stages of machine learning workflows.</p> <p>MLflow consists of several key components:</p> <p>Tracking: This component allows you to record and compare different experiments, including the parameters, metrics, and artifacts (such as models, visualizations, and data) associated with each experiment. It helps keep track of the different iterations and configurations tried during the model development process.</p> <p>Projects: MLflow projects provide a way to package code into a reproducible format, allowing you to define and share your machine learning projects with others. This ensures that the code and dependencies used for a particular experiment can be easily reproduced by others.</p> <p>Models: MLflow model management capabilities enable you to easily log, version, and deploy machine learning models. It supports various machine learning frameworks, for example TensorFlow, PyTorch, and scikit-learn. MLflow also makes it possible to deploy models directly onto different platforms. </p> <p>Registry: The MLflow model registry provides a centralized repository for storing, versioning, and managing models. It allows teams to collaborate on model development and helps maintain a history of different model versions.</p> <p>UI and APIs: MLflow makes it easy to use and integrate programmatically by offering a web-based user interface as well as APIs. </p> <p><strong>Experiments</strong></p> <p>An experiment is registered in the MLflow tracking service, and different runs are carried out with different parameters and features. The parameters can then be logged into MLflow and viewed on the UI. Different runs are then compared so that the best metric can be chosen and moved into production. </p> <p>To begin, allow the experiment to access MLflow from the Jupyter notebook. Then, as we have used TensorFlow LSTM as our ML model, we use mlflow.tensorflow.autolog() to log automatically the training parameters, hyperparameters and model metrics.</p> <p><img src="/img/figure-1a-experiment.png" alt="" title="Figure 2. Creating an experiment with auto logging. "></p> <p>Once the model is ready, the run details are available via MLflow APIs like search_run(). The best run can be auto-selected as per our requirements.</p> <p><img src="/img/fig-1b-best-run-experiment.png" alt="" title="Figure 3. Selecting the best run of the experiment and loading the model."></p> <p>On the Web UI for MLflow, you can access multiple runs of the ML experiment for different parameter combinations. Details of the run can be accessed by clicking on the corresponding run name.</p> <p><img src="/img/fig-2-multiple-runs.png" alt="" title="Figure 4. Results from multiple runs made on the experiment with different parameters in each run. "></p> <p>Model performance can be obtained by comparing various runs. In the image below, all six runs have been selected and you can see that the model with epochs=20, performs best with the least loss and mean absolute error (MAE)</p> <p><img src="/img/fig-3-multiple-runs-compared.png" alt="" title="Figure 5. Comparison of all runs based on a relevant metric. "></p> <p>Now that the best model has been identified, it can be registered programmatically or through the MLflow Web UI. Once registered, the model can be deployed to the appropriate staging-production region. </p> <p><img src="/img/fig-4-model-registry-different-models.png" alt="" title="Figure 6. Model registry with different models. "></p> <p><strong>Code repository</strong></p> <p>A code repository is a storage location for code and other software development assets, such as documentation, tests, and scripts. They are often used to manage and organize a software project's codebase and collaborate with other project developers.</p> <p>The source code for the model is stored in a private repository on GitHub then seamlessly pulled into the IDE using git commands. The updated model can be pushed to different branches of the source repo. </p> <p><img src="/img/code-repository.png" alt="" title="Figure 7. Source code for models is stored on Github in a private repository. "></p> <p><strong>Model deployment</strong></p> <p>The next step is to deploy the best model chosen from the MLflow experiment. Here, Kserve, which is a standard model inference platform on Kubernetes that is build for highly scalable use cases, is being used.</p> <p>Kubeflow Serving, also known as KServe, is a component of the Kubeflow ecosystem and is designed to serve machine learning models in Kubernetes environments. Kubeflow itself is an open-source platform for deploying, monitoring, and managing machine learning models on Kubernetes. KServe provides several features and benefits:</p> <ol> <li>Scalability: It is designed to handle serving machine learning models at scale. You can easily scale up or down based on the traffic and load requirements.</li> <li>Multi-framework support: KServe supports serving models trained in various machine learning frameworks, such as Tensor Flow, PyTorch, scikit-learn, and others.</li> <li>Advanced deployment strategies: KServe supports various deployment strategies, for example  Canary deployments or Blue-Green deployments,  allowing you to roll out new model versions gradually and monitor their performance.</li> <li>Monitoring and metrics: It provides metrics and monitoring capabilities, allowing you to keep track of model performance, errors, and other relevant statistics.</li> <li>Customization: KServe is highly customizable, allowing you to define how you want to serve your models, handle preprocessing and post-processing tasks, and even add custom logic to your serving containers.</li> <li>Integration with Kubeflow pipelines: If you're using Kubeflow pipelines for end-to-end machine learning workflows, KServe can be integrated seamlessly to deploy models as part of your pipeline.</li> </ol> <p>Visit the <a href="https://kserve.github.io/website/0.10/modelserving/control_plane/">Kserve documentation</a> for more information.</p> <p>The first step of model deployment is to define the service account with MinIO object store secrets.</p> <p><img src="/img/fig-1a-define-service-acct.png" alt="" title="Figure 8. Defining service account with MinIO secret. "></p> <p>Next, script the KServe yaml file to begin the inference service. </p> <p><img src="/img/fig-1b-inference-service.png" alt="" title="Figure 9. The inference service is defined. "></p> <p>Next, a new model server must be created in Kubeflow and the yaml file uploaded. Once it is running, you can send your test data through REST APIs to get the response.</p> <p><strong>Model retraining</strong></p> <p>The models deployed into production are constantly monitored to identify inconsistencies and degradation in performance. Because models can skew to the old data used for training resulting in under performance, models should be retrained whenever performance drops below a defined threshold. To retrain a model, a pipeline is developed that could trigger all the steps associated with model retraining to occur in the appropriate sequence using Kubeflow pipelines.  </p> <p>Python programs need to be modified to run as a Kubeflow pipeline because each step runs as a containerized module. The whole cycle of data collection, preprocessing, and modeling are divided into separate components with each process isolated and running as independent containers inside individual Kubernetes pods. All the dependencies of each task must be mentioned inside the task due to the isolated nature of the environment. The artifacts that are used in the process, such as input data files and model objects, are saved in specific locations and referenced by the tasks. </p> <p>To prepare the Kubeflow pipelines, define each task as a function. First “read_data()” is defined to read the input file from the data volume on the Kubeflow pipeline.</p> <p><img src="/img/fig-1-define-each-task.png" alt="" title="Figure 10. Each task is defined as a function that includes all necessary packages. "></p> <p>Once defined, functions are converted into a Kubeflow component using “create_component_from_func()”.</p> <p><img src="/img/fig-2-converting-each-task.png" alt="" title="Figure 11. Each task is converted into a component that includes base line images and packages. "></p> <p>Once each of the tasks are defined, the Kubeflow pipelines are defined to call each of the steps using DSL (Domain Specific Language).</p> <p><img src="/img/fig3-define-the-pipeline.png" alt="" title="Figure 12. Pipelines are defined along with tasks in the appropriate sequence. "></p> <p>When the Kubeflow pipeline is set up, the pipeline can run using the Kubeflow client where the required arguments and parameters are passed for the specific run.</p> <p><img src="/img/kubeflow-setup-run.png" alt="" title="Figure 13. Kubeflow client is created and executed with appropriate arguments. "></p> <p>In the Kubeflow UI, “stock_pipeline_run” would be triggered to provide a green-red status of each step as it executes. </p> <p><img src="/img/fig-5-simplified-pipeline.png" alt="" title="Figure 14. Status of each pipeline step and it&#x27;s status. "></p> <p><strong>Conclusion</strong></p> <p>As you can see, HPE Ezmeral Unified Analytics has significantly enhanced the end-to-end pipeline of financial services use cases by providing a comprehensive platform tailored to the specific needs of ML practitioners and data scientists.</p> <p>In summary:</p> <p>Seamless integration: HPE Ezmeral Unified Analytics offers seamless integration of various data engineering and ML components like Spark, EzPresto, Superset, MLflow, Kubeflow. This integration streamlines the entire ML workflow, reducing friction between dissimilar stages and enables more efficient development.</p> <p>Data exploration and visualization: By seamlessly integrating Apache Superset into the Ezmeral ecosystem, organizations can harness the power of data-driven insights. Teams can easily access and visualize data, track experiment results, and make data-informed decisions to improve model quality and performance.</p> <p>Reproducibility: Kubeflow ensures reproducibility by allowing users to define and version their entire ML pipelines using HPE Ezmeral Unified Analytics. This means that every step, from data preparation to model deployment, can be tracked, versioned, and replicated easily, leading to better model accountability and auditability.</p> <p>Scalability: Kubeflow on HPE Ezmeral Unified Analytics excels at scaling applications. Kube Flow leverages this capability to scale ML workloads as needed, handling large datasets and complex models with ease. This scalability is crucial for training and serving models in real-world, production environments.</p> <p>Model serving: HPE Unified Analytics through Kubeflow Serving (KServe) simplifies model deployment and serving. It provides features like advanced deployment strategies, model versioning, and monitoring, making it easier to roll out new models, manage production deployments, and ensure model performance and reliability.</p> <p>Automated workflow orchestration: Kubeflow pipelines simplify the orchestration of ML workflows as code. With HPE Unified Analytics, this automation reduces manual effort, enhances reproducibility, and accelerates experimentation, resulting in more efficient model operations.</p> <p>Contributors to this blog post include Suvralipi Mohanta (<a href="mailto:suvralipi.mohanta@hpe.com">suvralipi.mohanta@hpe.com</a>), Harikrishnan Nair (<a href="mailto:harikrishnan.nair@hpe.com">harikrishnan.nair@hpe.com</a>), Ashok Manda (<a href="mailto:ashok.manda@hpe.com">ashok.manda@hpe.com</a>) and Joann Starke (<a href="mailto:joann.starke@hpe.com">joann.starke@hpe.com</a>).</p>https://developer.hpe.com/gitops-the-next-step-in-cloud-native-development/https://developer.hpe.com/gitops-the-next-step-in-cloud-native-development/Mon, 25 Sep 2023 15:37:56 GMT<p><strong>Editor’s note: This article was originally posted on HPE Enterprise.nxt on June 2, 2021</strong></p> <hr> <p>What do you get when you combine DevOps management and the Git distributed version control system? Say hello to GitOps. </p> <p>Automation makes software better and more reliable. GitOps takes automation a step further and merges it with deployment. </p> <p><a href="https://www.hpe.com/us/en/insights/articles/devops-lessons-in-software-infrastructure-and-business-success-1702.html">DevOps</a>, in which system administrators work hand-in-hand with developers, can speed up software development and operational deployments from months to days. At the same time, we use <a href="https://www.hpe.com/us/en/insights/articles/devops-lessons-in-software-infrastructure-and-business-success-1702.html">Kubernetes</a> to orchestrate containers over clouds to speed up software development and operational deployments. </p> <p>Ultimately, both approaches lend themselves to <a href="https://www.hpe.com/us/en/insights/articles/continuous-integration-and-delivery-tool-basics-1807.html">continuous Integration/continuous delivery (CI/CD</a>). Wouldn't it be great, thought <a href="https://www.weave.works/%22%20/t%20%22_blank">Weaveworks </a>CEO Alexis Richardson, if we could combine these approaches and use the <a href="https://www.weave.works/blog/gitops-git-push-all-the-things%22%20/t%20%22_blank">Git distributed version control system as the ultimate source of truth</a>? So, when there's a dispute over the correct state of the site, people know where to go for the correct version. </p> <p>It turns out Richardson was on to something. Cornelia Davis, Weaveworks' CTO, recently said, "<a href="https://www.itprotoday.com/development-techniques-and-management/why-gitops-model-future-devops">I believe that GitOps is the model that will dominate operations</a>. … I think, five years from now, everybody will be doing some level of GitOps." </p> <p>Why? Because GitOps is a set of practices that enables you to manage and deploy highly distributed software that's constantly changing without breaking a sweat. </p> <p>Now if just a single vendor was promoting its approach, you might be wise to be skeptical. But it's not just Weaveworks. Priyanka Sharma, general manager at the <a href="https://www.cncf.io/">Cloud Native Computing Foundation (CNCF)</a>, believes GitOps is becoming to Kubernetes what Git already is to Linux: the fundamental building tool for the next generation of <a href="https://www.hpe.com/us/en/insights/articles/how-to-implement-cloud-native-computing-with-kubernetes-1710.html">cloud-native computing</a>.</p> <p>GitOps is designed for and, as it now stands, really applicable to just orchestrated cloud-native applications. </p> <p><img src="/img/gitops_-the-next-step-in-cloud-native-development.png" alt="Block Text" title="Block text"></p> <h2>So what is GitOps today? </h2> <p>Officially, Weaveworks defines GitOps as "a way to do Kubernetes cluster management and application delivery. <a href="https://www.weave.works/technologies/gitops/%22%20/t%20%22_blank">GitOps works by using Git as a single source of truth for declarative infrastructure and applications</a>. With GitOps, the use of software agents can alert on any divergence between Git with what's running in a cluster, and if there's a difference, Kubernetes reconcilers automatically update or roll back the cluster depending on the case. With Git at the center of your delivery pipelines, developers use familiar tools to make pull requests, to accelerate and simplify both application deployments and operations tasks to Kubernetes." </p> <p>Superficially, GitOps is quite simple. GitOps uses a version control system, Git, to house all information, documentation, and code for a Kubernetes deployment. Kubernetes then automatically deploys changes to the cluster. </p> <p>Of course, simple concepts are always more complex in reality. Let's look at the fundamentals. </p> <h3>1) Everything that can be described must be stored in Git. </h3> <p>By using Git as the source of truth, it is possible to observe your cluster and compare it with the desired state. The goal is to describe everything: policies, code, configuration, and even monitored events and version control. Keeping everything under version control enforces convergence where changes can be reapplied if at first they didn't succeed. </p> <p>These descriptions of the entire system are described declaratively in <a href="https://yaml.org/">YAML</a>. This is a human-readable data serialization language. YAML is commonly used for configuration files and data storage and transmission. </p> <p>If this sounds a lot like DevOps, you're right, it does. For example, <a href="https://www.ansible.com/%22%20/t%20%22_blank">Ansible</a>, <a href="https://azure.microsoft.com/en-us/services/devops/pipelines/">Azure Pipelines</a>, <a href="https://saltproject.io/">Salt</a>, and <a href="https://puppet.com/">Puppet</a> all use YAML. No matter the program, the idea is the same: Use declarations written in YAML to control operations. This approach is also known as <a href="https://stackify.com/what-is-infrastructure-as-code-how-it-works-best-practices-tutorials/">infrastructure as code</a> (IAC). </p> <p>Within GitHub YAML files, you find not instructions like "Start 10 MySQL servers" but declarations—for instance, "There are 10 MySQL servers. These are their names." </p> <p>The bottom line is to take fundamental DevOps and IAC concepts and move them to the cloud-native world. </p> <h3>2) Use a Kubernetes controller that follows an operator pattern </h3> <p>With a Kubernetes controller that follows the operator pattern, your cluster is always in sync with your Git repository, the source of truth. Since the desired state of your cluster is kept in Git YAML files, you can easily spot differences between your Git files and the running cluster. </p> <p>For GitOps, the most popular controller is the open source program <a href="https://fluxcd.io/">Flux</a>. This is a collection of tools for keeping Kubernetes clusters in sync with YAML files in Git repositories and automating configuration updates when there's new code to deploy. Indeed, although Flux's code is changing rapidly, <a href="https://radar.cncf.io/2020-06-continuous-delivery">Flux has already been recommended by the CNCF for adoption</a> by CD users. </p> <p>With Flux, you can describe everything about the entire desired state of your system in Git. This includes apps, configuration, dashboards, monitoring, and everything else. </p> <p>This means everything—and I mean everything—is controlled through pull requests. There's no learning curve for new programmers; they just use the Git commands they already know. If there's a production issue, it's fixed via a pull request instead of manually changing the running system. As a big side benefit, your Git history automatically provides a log of transactions, enabling you to recover your system state from any snapshot. </p> <p>Flux also uses <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Kubernetes Custom Resources</a>, object state reports, and via Kubernetes Events, integration with <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">Kubernetes role-based access control (RBAC)</a>, making it declaratively configurable. </p> <p>The next version, Flux v2, is being built from the ground up to use Kubernetes' API extension system and to integrate with <a href="https://prometheus.io/">Prometheus</a> and other core Kubernetes components. In version 2, Flux supports multi-tenancy and can sync an arbitrary number of Git repositories, among other long-requested features. The Flux people are building this with the <a href="https://toolkit.fluxcd.io/components/%22%20/t%20%22_blank">GitOps Toolkit</a>, a set of composable APIs and specialized tools for building CD on top of Kubernetes. </p> <p>There are non-Flux Kubernetes platforms that support GitOps as well, including <a href="https://www.hpe.com/content/hpe/country/us/en/software/ezmeral-runtime.html">HPE Ezmeral Container Platform</a>. Ezmeral <a href="https://docs.containerplatform.hpe.com/53/reference/kubernetes/Policy_Management_Overview.html">delivers GitOps through its Centralized Policy Management capabilities</a> using <a href="https://argoproj.github.io/argo-cd/">Argo CD</a>, a declarative, GitOps continuous delivery tool for Kubernetes. </p> <h3>3) Software agents are used to ensure correctness and act on divergence, a.k.a. <a href="https://cluster-api.sigs.k8s.io/developer/providers/implementers-guide/controllers_and_reconciliation.html">Kubernetes reconcilers</a> </h3> <p>As Richardson puts it, "One of the most important functions of GitOps is to enable a group of system changes to be applied correctly and then verified. After that, GitOps should enable the users and orchestrators to be notified [alerted] if any of the systems has drifted from the correct state so that it may then be converged back to the correct desired state, which is in Git." </p> <p>So, how would this work in practice? Something like this. </p> <p>You make your changes to your Git files. Then, <a href="https://www.jenkins.io/">Jenkins</a>, the open source automation server, pushes these changes to the <a href="https://www.projectquay.io/">Quay</a> container image registry. Jenkins then pushes the new config, and <a href="https://helm.sh/">Helm</a>, the Kubernetes package manager, charts to the master Git storage bucket. Once the merge request is complete, the automated GitOps operator detects the change and calls Flux to make the changes operational by deploying the updated YAML files to the master Git repository and hence to the operational Kubernetes cluster. </p> <p>That may sound complicated, but once it's set up and debugged, it will greatly speed up your deployment of applications. What do you end up with? GitOps is a CI/CD system that's greater than the sum of its parts.</p> <p>As a corollary to this, you don't use <a href="https://kubectl.docs.kubernetes.io/%22%20/t%20%22_blank">kubectl</a>, the Kubernetes command-line interface, because all changes are handled automatically by the GitOps pipeline. Indeed, Richardson says it's not a good idea to deploy directly to the cluster using kubectl. That's because by relying on kubectl, you're making production potentially open to shell-based hacking attacks. </p> <h2>Why GitOps? </h2> <p>Sharma says, just as "Kubernetes unleashes the power of cloud computing for building software fast and resiliently, GitOps is basically utilizing the Git workflows that every developer is used to." So, how important is this approach? "Not everyone who is touching Kubernetes is using GitOps, but I know everyone wants to because it would make their life easier." </p> <p>"Everyone" includes the <a href="https://github.com/gitops-working-group/gitops-working-group">CNCF GitOps Working Group</a>, which is working on best practices for the still quite new GitOps. Its members include Hewlett Packard Enterprise, Amazon Web Services, GitHub, Codefresh, Microsoft, and, of course, Weaveworks. </p> <p>Besides best practices, the working group is also <a href="https://github.com/gitops-working-group/gitops-working-group">hammering out the GitOps Manifest</a>. This is very much a work in progress. When done, it will define GitOps' principles and technical aspects in a vendor- and implementation-neutral manner. It will also lay out a common understanding of GitOps systems based on shared principles rather than on individual opinion. Another aim is to encourage innovation by clarifying the technical outcomes rather than the code, tests, or organizational elements needed to achieve them. </p> <p>Sharma, <a href="https://gitlab.com/users/sign_in%22%20/t%20%22_blank">GitLab</a>'s director of technical evangelism, says it best: "For those shops doing DevOps, this approach can be appealing because <a href="https://thenewstack.io/what-is-gitops-and-why-it-might-be-the-next-big-thing-for-devops/%22%20/t%20%22_blank">GitOps brings the workflow closer to the developer</a>." </p> <p>Programmers just keep using the Git tools they already know to push code into production. And since the workflow goes directly through Git, it's recorded and logged. Sharma says, "There is an audit trail, the ability to revert problematic changes, and ultimately a single source of truth of what is happening in the system from both the software development and infrastructure perspective." </p> <p>Richardson says, "Imagine a world where every time you do a deployment it's correct. And if it's not correct, then the deployment fails completely, so you can try again or make other intelligent decisions. … That is just an incredible cost-saver in operational overhead—moving from an unsafe, semireliable system to one that is basically more robust." </p> <p>But it's not magic. Simply storing your old operational patterns into Git won't get you anywhere. As Cornelia Davis, Weaveworks CTO, comments, "<a href="https://sdtimes.com/softwaredev/gitops-its-the-cloud-native-way/">Just because you put something in Git doesn't make it GitOps.</a> It isn't actually the central part of GitOps. Ops is the central part of GitOps." </p> <p>The biggest mistake people make about GitOps, Davis says, is people don't get that things are always correcting themselves and you always have to respond to change with reconciliation loops. You must think about this and use it in your approach or you'll just be recycling old mistakes in a new concept. </p> <p>Do it right, however, and you can expect to reap the following: </p> <ol> <li>Increased CI/CD productivity. </li> <li>A better developer experience, by letting developers push code instead of managing containers. </li> <li>Improved stability, thanks to Git's audit log of Kubernetes cluster changes. </li> <li>Better reliability as a result of Git's built-in revert/rollback and fork from a single source of truth. </li> <li>And last but never least, improved cost efficiency from less downtime and improved productivity. </li> </ol> <p>Sound good to you? It does to me. I predicted long before most people did that Kubernetes would become the container orchestration program. I'm now going to go on a limb and predict that GitOps, in turn, is going to become the way most of us will end up deploying programs to the cloud in the next few years. It just makes too much sense for it not to work. </p> <p>Tom Phelan, Fellow, big data and storage organization at Hewlett Packard Enterprise, contributed to this article. </p> <h2>Lessons for leaders </h2> <ul> <li>Cloud-native development does an excellent job of enabling new capabilities, including GitOps. </li> <li>GitOps enables more efficient processes and, ultimately, better service for customers than traditional approaches. </li> <li>One of the best reasons to give GitOps a chance is that it uses tools that developers already know. </li> </ul> <br /> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.</span></p> </blockquote> </blockquote> <br /> <h2>About the author: </h2> <p><a href="https://www.hpe.com/us/en/insights/contributors/steven-j-vaughan-nichols.html">Steven Vaughan-Nichols</a> </p> <p> CEO, Vaughan-Nichols &#x26; Associates 53 publications </p> <p>Steven J. Vaughan-Nichols, a.k.a. sjvn, has been writing about technology and the business of technology since CP/M-80 was the cutting-edge PC operating system, 300bps was a fast Internet connection, WordStar was the state-of-the-art word processor, and we liked it. His work has been published in everything from highly technical publications (IEEE Computer, ACM NetWorker, Byte) and business publications (eWeek, InformationWeek, ZDNet) to popular technology magazines (Computer Shopper, PC Magazine, PC World) and the mainstream press (Washington Post, San Francisco Chronicle, Businessweek). </p>https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise--Exploring-a-flexible-infrastructure-resource-pool/https://developer.hpe.com/hpe-greenlake-for-private-cloud-enterprise--Exploring-a-flexible-infrastructure-resource-pool/Mon, 18 Sep 2023 10:08:35 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h2>Introduction </h2> <p>Welcome to the HPE GreenLake for Private Cloud Enterprise (GLPCE) blog series: Showcasing how GLPCE features address business challenges. In the series I delve deep into the advanced capabilities of modern cloud infrastructure, illuminating the path to heightened innovation and sustainable growth for organizations. By consistently offering up-to-date insights and expert perspectives, I aim to be your trusted guide in this rapidly shifting digital landscape. Join me in this exploration, and together, we'll chart the course for the next era of enterprise cloud technology. </p> <p>Enterprises today face a complex data landscape due to the rise of edge technologies. They deal with data that originates from and is processed in either in data centers, at the edge, or a combination of both. As more data is produced at the edge, the importance of local processing grows. This, combined with the challenges of private cloud scalability and costs, means organizations need to adopt more streamlined and efficient approaches. HPE GreenLake for Private Cloud Enterprise (GLPCE) offers a solution. GLPCE blends the strengths of public and private clouds together. While it offers the flexibility and scalability demonstrated by public clouds, it also retains the advantages of private cloud implementations, such as data locality, dedicated infrastructure, and the capacity to integrate with and capitalize on pre-existing on-premises IT infrastructure investments. By using standardized instance types, GLPCE ensures that organizations maintain agility without compromising on cost-effectiveness.  </p> <p>HPE GreenLake for Private Cloud Enterprise (GLPCE) offers a robust solution with its ability to flexibly allocate resources, prevent bottlenecks, and ensure cost-aware scalability. A standard GLPCE setup may include a single or multi rack configuration with various nodes, such as 4 from the bare metal pool with a mix of general purpose and memory optimized instance types, 8 dedicated to a Virtual Machines pool, and 4 for container tasks, emphasizing its versatile deployment capabilities.</p> <p>HPE GreenLake for Private Cloud Enterprise (GLPCE) provides a variety of instance types, such as:  </p> <ul> <li>General purpose: a good balance of compute, memory, and networking resources, and suitable for a wide variety of workloads. </li> <li>Compute optimized: designed for workloads that require high CPU performance, such as web servers, CI/CD pipelines, and container and VM orchestration. </li> <li>Memory optimized: designed for workloads that require high memory performance, such as in-memory databases and analytics. </li> <li>Storage optimized: designed for workloads that require high storage performance, such as data lakes and Splunk. </li> </ul> <p>In this first blog post, I will delve into the challenges organizations face in the current digital landscape. While private clouds offer notable advantages like geo-locality and enhanced performance, they also come with the added complexity of managing scalability, costs, and agility. Addressing this, HPE GreenLake for Private Cloud Enterprise (GLPCE) emerges as a solution that blends the flexibility of public clouds with the specific controls of private ones, ensuring an infrastructure that's both agile and cost-efficient. A standout aspect of GLPCE is its flexible infrastructure resource pool, designed for immediate resource allocation, enabling organizations to swiftly adjust to varying digital traffic demands. Furthermore, as a managed solution, GLPCE alleviates the administrative burden, allowing businesses to focus on their core operations while enjoying the benefits of an efficient cloud infrastructure.</p> <p>Join me in this first post of a three-part series on HPE GreenLake for Private Cloud Enterprise (GLPCE). I will dive deep into GLPCE, spotlighting its unique strengths. With each post, you can gain valuable insights into the future of enterprise cloud technology. Step in, and let's explore together. </p> <h2>Features </h2> <p>Navigating cloud computing complexities calls for a solution that's efficient and user-friendly. HPE GreenLake for Private Cloud Enterprise (GLPCE) meets these needs, providing features designed to simplify and enhance cloud functionalities.  </p> <ul> <li>Flexible resource allocation: GLPCE enables targeted allocation and release of computing instance types from a dynamic infrastructure resource pool based on demand. </li> <li>Prevention of bottlenecks: GLPCE allows for adjusting resources for each service based on current needs, ensuring smooth operations, and preventing potential service interruptions or slowdowns. </li> <li>Cost-awareness: Through GLPCE's Consumption Analytics Service, administrators gain a clear insight into both private and public cloud costs, enabling a more streamlined approach to budgeting and cost management. </li> </ul> <h2>Allocation / Deallocation </h2> <p>HPE GreenLake for Private Cloud Enterprise (GLPCE) supports resource allocation in line with expected workloads. Should a cluster harbor idle resources, they can be redirected to areas of higher demand. It offers consumption analytics and ongoing monitoring tools to ensure efficient resource utilization. I'll delve deeper into this concept with a forthcoming visual representation.</p> <p><img src="/img/blog1.png" alt="Screenshot 1: Initial allocation " title="Screenshot 1: Initial allocation "></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 1: Initial allocation </span></p> </blockquote> </blockquote> <h2>Resource allocation strategy</h2> <p><strong>Initial allocation</strong>: As seen in screenshot 1, the initial resource allocation process is primarily based on anticipated workload demands. When preparing for extensive data-processing tasks, customers can allocate a higher number of memory-optimized (M2i) instance types to Cluster 2 within the virtualization resource pool. On the other hand, for every day, routine operations, Cluster 1 is typically outfitted with general purpose (G2i) instance types to ensure smooth and efficient operations. This allocation strategy aims to match resource types with the specific requirements of each cluster. </p> <p><img src="/img/blog2.png" alt="Screenshot 2: Reallocation " title="Screenshot 2: Reallocation "></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em">Screenshot 2: Reallocation </span></p> </blockquote> </blockquote> <p><strong>Reallocation</strong>: As illustrated in screenshot 2, there are times when instances within the bare metal resource pool remain unused. These idle resources are not merely dormant; they hold value. As shown in screenshot 2, these instance types can be quickly reallocated to any virtualization clusters based on their current needs. Such dynamic reallocation ensures optimal resource utilization, adapting promptly to varying workload demands.</p> <h2>Tune resource allocations </h2> <p><strong>Monitoring for efficiency/capacity</strong>: Using HPE GreenLake for Private Cloud Enterprise, you can continuously monitor the utilization rates across all three services: bare metal, virtual machines, and containers. To understand how such resource utilization is tracked and assessed, check out the next section on Capacity planning and monitoring.</p> <p><img src="/img/picture3-20-sep-2023.png" alt="Screenshot 3: Responsive adjustments " title="Screenshot 3: Responsive adjustments "></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"><em>Screenshot 3: Responsive adjustments</em> </span></p> </blockquote> </blockquote> <p><strong>Responsive adjustments</strong>: Referring to screenshot 3, observe that if Cluster 2 possesses idle memory optimized (M2i) instances, they can be shifted from Cluster 2 either to the bare metal pool or to Cluster 1, depending on demand. Conversely, if Cluster 1 has surplus general purpose (G2i) units, they can be relocated in a similar manner. Such adaptability ensures that resource utilization is consistently maximized, preventing them from sitting unused when they could be beneficial elsewhere.</p> <h2>Benefits </h2> <p>HPE GreenLake for Private Cloud Enterprise (GLPCE) stands out with its versatile offerings: </p> <ol> <li><strong>Flex capacity for varied workloads</strong>: Adapts to any challenge, be it data-heavy tasks needing memory-optimized solutions or general-purpose assignments. The infrastructure's flexibility allows you to dynamically allocate resources from different instance types, ensuring you're always prepared for varying workloads. </li> <li><strong>Cost efficiency</strong>: The intelligent utilization of varied instance types avoids redundant costs by preventing over-provisioning, leading to tangible savings. </li> <li><strong>Performance enhancement</strong>: No matter the environment - bare metal, VM, or container - resources are meticulously tuned to guarantee top-tier operational efficiency. </li> </ol> <p>For a Cloud or an IT admin, this translates to: </p> <ul> <li><strong>Resource agility</strong>: Resources are allocated quickly and as per demand, ensuring seamless coding and testing phases. </li> <li><strong>Optimized app development</strong>: Tailored resources mean your applications run at their prime, ensuring your work shines. </li> <li><strong>Hassle-free environment transition</strong>: While GLPCE doesn't make transitioning between BM, VM, and Containers effortless, it does simplify the infrastructure aspect, ensuring that whatever environment you're developing for or operating within is backed by a robust, flexible, and optimal resource. </li> <li><strong>Informed application optimization</strong>: With GLPCE’s optimized infrastructure, developers gain insights into the utilization and cost implications of their applications. This transparency empowers them to refine and optimize their apps, ensuring efficient resource use and cost-effectiveness during the application's lifecycle. </li> <li><strong>Performance assurance</strong>: You develop with the certainty that applications will consistently deliver peak performance, enhancing the end-user experience. </li> </ul> <p>In wrapping up, HPE GreenLake for Private Cloud Enterprise is more than just a sophisticated enterprise infrastructure; it caters to both the developer and the admin. For developers, it simplifies workflows and magnifies their contributions. For admins, it offers tools and insights that align with organizational goals. The value that GLPCE delivers makes developers and admins advocates for the adoption of GLPCE within their organizations. This dual appeal makes GLPCE a prime selection for entities seeking contemporary, streamlined, and role-friendly infrastructure solutions.</p> <h2>Capacity planning &#x26; monitoring </h2> <p>Here’s how you can plan and monitor capacity usage in HPE GreenLake for Private Cloud Enterprise (GLPCE) across the three services: </p> <p><img src="/img/picture5-19-sep-23.png" alt="Screenshot 4: Bare metal service monitoring" title="Screenshot 4: Bare metal service monitoring"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"><em>Screenshot 4: Bare metal service monitoring</em> </span></p> </blockquote> </blockquote> <p>In Screenshot 4, the Capacity tab displays bare metal resources categorized by "Compute Group, Site, and Instance Type." Users can set thresholds for CPU and memory usage. Yellow bars in the CPU and memory columns signify that resource usage is nearing the configured maximum threshold.</p> <p><img src="/img/picture6-19-sep-23.png" alt="Screenshot 5: Virtual Machine service monitoring" title="Screenshot 5: Virtual Machine service monitoring"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"><em>Screenshot 5: Virtual Machine service monitoring</em></p> </blockquote> </blockquote> <p>In Screenshot 5, the Capacity tab categorizes Virtual Machine service resources by "Cluster and Instance Type." The screen displays four key metrics: CPU usage, CPU allocated, memory usage, and memory allocated.  </p> <ul> <li>CPU and memory usage thresholds reflect active resource consumption by hosted apps/processes. By defining a range, we can optimize resources, preventing underuse or potential performance issues. </li> <li>CPU and memory allocated thresholds show reserved resources per cluster/instance type, regardless of activity. This ensures we identify over-allocated or under-allocated resources.</li> </ul> <p><img src="/img/picture7-19-sep-23.png" alt="Screenshot 6: Container service monitoring" title="Screenshot 6: Container service monitoring"></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"><em>Screenshot 6: Container service monitoring</em></p> </blockquote> </blockquote> <p>In Screenshot 6, the Capacity tab displays grouping of Container resources by "Cluster, Instance Type, and Site." GLPCE supports two container deployment models: on Virtual Machines and on bare metal. While allocated thresholds are predetermined, this interface allows for the customization of usage thresholds. Stay tuned for our upcoming blog post on container service scaling.</p> <h2>Conclusion </h2> <p>In conclusion, HPE GreenLake for Private Cloud Enterprise offers organizations greater flexibility through its flexible infrastructure resource pool and capacity planning. This not only addresses current needs but also prepares for future demands. Moreover, GLPCE's consumption analytics provide clear cost insights, facilitating informed financial decisions. In the digital age, choosing the right infrastructure is essential. With GLPCE, organizations gain both scalability and transparency in costs. Consider HPE GreenLake for Private Cloud Enterprise for a comprehensive infrastructure solution.</p> <p>Stay tuned for upcoming blog posts as I delve deeper into GLPCE's dynamic features, underscoring how they come together to create an agile and resilient foundation for the modern digital enterprise.</p>https://developer.hpe.com/closing-the-gap-between-hpc-and-ai/https://developer.hpe.com/closing-the-gap-between-hpc-and-ai/Fri, 15 Sep 2023 06:34:38 GMT<p>Scientists and engineers face a number of technical hurdles to utilizing artificial intelligence (AI) techniques alongside high-performance scientific computing applications. Recently, a multidisciplinary team of researchers at Hewlett Packard Enterprise (HPE), the Institute of Aerodynamics and Gas Dynamics of the University of Stuttgart, and the High-Performance Computing Center Stuttgart (HLRS), created Relexi to overcome these technical challenges and provide the community with a large-scale simulation framework that can be tailored to their use cases. This solution was built on top of another existing open-source library developed at HPE called <a href="https://developer.hpe.com/platform/smartsim/home/">SmartSim</a>.</p> <p>Researchers were looking for machine learning (ML)-based techniques that could help with common engineering problems, such as using a simulation of fluid flow to improve the performance of an airplane wing. Relexi was able to realize this by combining FLEXI, a computational fluid dynamics solver, and Tensorflow, one of the most popular machine learning packages, to create a reinforcement learning framework by which a computer can ‘learn’ how to optimize these types of problems.</p> <p>FLEXI was developed at the Institute for Aerodynamics and Gas Dynamics (IAG) in the lab for Numerical Methods in Fluid Mechanics of Prof. Dr. Ing. Andrea Beck over the course of many years, and is used for computationally demanding, high-resolution computational fluid dynamics (CFD) simulations. Engineers use the code, for example, to simulate the flow around an airfoil to optimize the performance of new airplane wing designs. Modeling such phenomena based on physical principles is currently only practical using powerful, highly parallelized supercomputers. Recently, however, researchers at the IAG have been exploring how machine learning could enhance FLEXI’s capabilities and make such simulations more accessible. Tested on HLRS’s Hawk supercomputer, this approach holds the potential to make high-performance simulation more accessible for state-of-the-art engineering applications.</p> <h2><strong>Reinforcement learning makes simulations easier</strong></h2> <p>Turbulence is one of the most challenging problems for engineers to deal with because it exists at a variety of scales, all the way down to the molecular level. Normally, the simulations that represent the smallest structures of a turbulent flow are very computationally intensive and expensive. Additionally, there is no single formula that describes the effects of turbulence well in the types of coarser simulations that most engineers are practically restricted to. Traditionally, fluid dynamicists have needed to derive approximations using complex mathematical techniques, most of which form the foundation of countless PhD theses, with no clear ‘winner’ emerging among the field. More recently, researchers have tried using a machine learning approach called supervised learning that analyzes a training dataset to develop a turbulence model. This approach has also proved to be insufficient, because of discrepancies between the training set and the actual simulation environment, which are not easy to resolve this way.</p> <p>Scientists in the Beck Lab have been exploring whether another ML approach, called reinforcement learning, could help to overcome these limitations. In reinforcement learning, a machine learning model is not trained on a static dataset, but rather on data generated by the actual system it will later model. The team calls the new framework <a href="https://github.com/flexi-framework/relexi">Relexi</a>, and in recent publications they have demonstrated its effectiveness in finding suitable turbulence models and making the simulations more accurate. Figure 1 below shows the general architecture of Relexi.</p> <p><img src="/img/blog-hpc-ai.png" alt="Figure 1: General architecture of Relexi." title="Figure 1: General architecture of Relexi. To train the agent, Relexi launches simulations on distributed worker nodes during each iteration of the training process. The entire training process is configured by Relexi’s YAML configuration file. The communication between TensorFlow and FLEXI is implemented via the Orchestrator database and the SmartRedis (SR) Clients provided by SmartSim."></p> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"><em>Fig. 1: General architecture of Relexi. To train the agent, Relexi launches simulations on distributed worker nodes during each iteration of the training process. The entire training process is configured by Relexi’s YAML configuration file. The communication between TensorFlow and FLEXI is implemented via the Orchestrator database and the SmartRedis (SR) Clients provided by SmartSim.</em></span></p> </blockquote> </blockquote> <p>To combine HPC and AI, Relexi uses an iterative workflow. Simulation data from FLEXI are fed into a reinforcement learning algorithm, becoming training data that the program uses to optimize the parameters of the turbulence model. The turbulence model then predicts the eddy viscosity as input data for the larger FLEXI simulation, which once again generates data for another round of training. By producing many FLEXI instances over multiple iterations, the optimization of the turbulence model eventually converges to a point where the simulation remains stable and accurate. The scientists can then be confident that their model is suitable for other types of applications.</p> <h2><strong>SmartSim enables the implementation of hybrid AI/HPC workflows</strong></h2> <p>Beck and her colleagues initially developed a proof of this concept on a desktop computer, but it required 36 hours to perform just 1,000 iterations, not enough to achieve a stable and reliable turbulence model. To scale it up to simulate more realistic systems, they turned to HLRS, whose Hawk supercomputer, a 26-petaflop system installed by HPE in 2020, offered an ideal platform. Considering the state of the art in implementing hybrid workflows, however, implementing this approach was far from straightforward.</p> <p>This is because traditional simulation and newer artificial intelligence approaches typically operate on completely different computing architectures. Except in rare situations, CFD codes like FLEXI are optimized for CPUs, are written in languages like Fortran, C, or C++, and are managed using parallel programming models like MPI and OpenMP. In contrast, AI methods run most effectively on GPUs, are based on other programming languages like Python, rely on libraries like TensorFlow or PyTorch, and use containers. Coupling legacy CFD codes with AI libraries is therefore one of the great challenges that high-performance computing is currently facing, particularly in a case like Relexi, where communication between HPC and AI needs to happen continuously in a single workflow.</p> <p>Hawk was initially built exclusively using CPUs, but in 2021 HLRS installed an expansion that includes 24 HPE Apollo 6500 Gen10 Plus systems with 192 NVIDIA A100 GPUs, offering 120 petaflops of AI performance. This expanded architecture now offers the capability to run hybrid workflows that combine traditional simulation with data science methods in a more efficient manner.</p> <p>A critical piece that was needed to make Relexi work, however, was an open-source library developed at HPE called SmartSim. This library provides a framework for efficiently moving data between legacy scientific applications developed for CPU architectures and newer data analytics methods requiring GPUs. In this case, the team used SmartSim to mediate efficient communication between FLEXI instances and the reinforcement learning program during runtime. Although SmartSim has previously been used for performing inference inside of applications at-scale in weather and climate simulations, this collaboration with the Beck Lab, HLRS and HPE was an early experiment to see how SmartSim could be used to construct a reinforcement learning solution within the context of a large-scale simulation.</p> <p>The implementation of SmartSim within FLEXI was relatively straightforward, as the programmers needed to change only 16 lines of code in FLEXI to make it work. The approach also scaled extremely well to 16 compute nodes (2,048 cores) on Hawk, with minimal drop-offs in performance. The team’s study demonstrated that by running higher numbers of simulations per training iteration, the turbulence model converged more quickly. They also showed that the more data that is produced, the better the resulting turbulence model will be.</p> <h2><strong>Relexi could support other CFD software</strong></h2> <p>Relexi was developed to enhance FLEXI, but is designed to be modular so that other CFD software can also be used as well. In this way, it holds great potential to support other hybrid applications that combine traditional high-performance computing with AI methods. The Beck Lab, HLRS, and HPE will continue developing Relexi, and plan to test it on a real-world simulation to determine the flow around a plane wing in the near future.</p> <h2><strong>Additional resources</strong></h2> <p>SmartSim on GitHub: <a href="https://github.com/CrayLabs/SmartSim">https://github.com/CrayLabs/SmartSim</a></p> <p>FLEXI and Relexi on GitHub:</p> <p><a href="https://github.com/flexi-framework/flexi">https://github.com/flexi-framework/flexi</a></p> <p><a href="https://github.com/flexi-framework/relexi">https://github.com/flexi-framework/relexi</a></p> <h2><strong>Related publications:</strong></h2> <p>Kurz M, Offenhäuser P, Beck A. 2023. <a href="https://www.sciencedirect.com/science/article/abs/pii/S0142727X2200162X?via%3Dihub">Deep reinforcement learning for turbulence modeling in largy eddy simulations</a>. Int J Heat Fluid Flow. 99: 109094.</p> <p>Kurz M, Offenhäuser P, Viola D, Shcherbakov O, Resch M, Beck A. 2022. <a href="https://www.sciencedirect.com/science/article/pii/S1877750322002435">Deep reinforcement learning for computational fluid dynamics on HPC systems</a>. J Comp Sci. 65: 101884.</p> <p>Kurz M, Offenhäuser P, Viola D, Shcherbakov O, Resch M, Beck A. 2022. <a href="https://www.softwareimpacts.com/article/S2665-9638(22)00106-3/fulltext">Relexi — A scalable open source reinforcement learning framework for high-performance computing</a>. Software Impacts 14: 100422.</p>https://developer.hpe.com/seamless-data-engineering-for-financial-services/https://developer.hpe.com/seamless-data-engineering-for-financial-services/Mon, 11 Sep 2023 13:56:53 GMT<p>Welcome to the three-part blog series showcasing the remarkable capabilities of HPE Ezmeral Unified Analytics through a real-world use case: Stock Market Prediction. In Part 1 of this series, I will delve into the data engineering aspect of the platform, exploring how it facilitates seamless data management and analysis.</p> <p>In Part 2 of the blog series, we will take you on a deep dive into the platform's ML/AI capabilities. Together, we will explore how the transformed data can be utilized for model building, leveraging Jupyter notebooks to perform interactive data exploration, pre-processing, and model training. Additionally, you will see how HPE Ezmeral Unified Analytics integrates seamlessly with MLflow for efficient model management and KServe for inference, allowing you to track and reproduce experiments easily.</p> <p>Finally, in Part 3 of the series, I will focus on automation using MLOps.  Now, let's embark on this exciting journey into the design and implementation of this cutting-edge solution.</p> <h2>What is HPE Ezmeral Unified Analytics?</h2> <p>HPE Ezmeral Unified Analytics software is a usage-based Software-as-a-Service (SaaS) platform that fully manages, supports, and maintains hybrid and multi-cloud modern analytics workloads through open-source tools. It goes beyond traditional analytics by seamlessly integrating machine learning and artificial intelligence capabilities, empowering users to develop and deploy data, analytics, and AI applications. By providing access to secure, enterprise-grade versions of popular open-source frameworks, the platform enables efficient and flexible scalability while securely accessing data stored in distributed data platforms. With its consistent SaaS experience, organizations can unlock data and insights faster, make data-driven predictions, and gain valuable business insights for faster decision-making, regardless of whether they operate on private, public, or on-premises infrastructure.</p> <p><img src="https://lh6.googleusercontent.com/I8m-F21ZhxExbOUCZsUjrAlQ6i1oapby2gnfJcelgsVDgSXEtQt_hvOeQSLWBwXAPVydqEEwMjiC_C2_mG3d7EKFL6uN6igCaetumxr-PFPvHtBNa6DiMKgp-qSvan4WzHf3UfRRwhxPOvgsD9sN7gA" alt=""></p> <p>This use case involves leveraging external pricing server/rest API calls, which are streamed into the data lake/data warehouse of a cloud provider (Microsoft Azure) using Spark from HPE Ezmeral Unified Analytics. Let me demonstrate how this platform enables data analysis using EzPresto (an enterprise-supported version of Presto) and empowers the creation of live dashboards using Superset.</p> <h2>Step 1: <strong>Data Gathering</strong></h2> <p>The data consists of stock prices of different companies listed in National Stock Exchange (NSE) of India. The files consist of historical data from the year 2000 to 2021, which was transformed to a streaming data source. The data was pulled from external servers hosted publicly and then saved to HPE Ezmeral Data Fabric Volume.</p> <p> <img src="https://lh4.googleusercontent.com/EqEcRejQD_tZemLF6h9J5I6KIZrzTJdqeAkvGzDUI0KN-x4XD5FrNTRzz2N8VvmRVwhk5YG_RtpClmbctddZyhz7t47ooK8ZipEEsHWyuEk3sTKavVtUtskbBGi1tKrBSBCP2aGWkqMxTdtQIKjQdlo" alt=""></p> <h2>Step 2: <strong>Data Ingestion</strong></h2> <h2>Apache Livy</h2> <p>HPE Ezmeral Unified Analytics gives access to Apache Livy, which enables easy interaction with the Spark cluster via REST interface. It simplifies the access between Spark cluster and application servers. It enables long running Spark contexts that can be used for multiple Spark jobs and multiple clients. Multiple Spark context can be managed that runs on the Spark Clusters. Spark applications can be either batch jobs or real-time streaming applications as per the business needs. Financial services have both long running batch applications as well as streaming applications, Apache Livy provides seamless management of Spark for the data engineers and application support team.</p> <p>Apache Livy on the HPE Ezmeral platform enables programmatic, fault-tolerant, multi-tenant submission of Spark jobs from web/mobile apps (no Spark client needed). So, multiple users can interact with the Spark cluster concurrently and reliably. Livy speaks either Scala or Python, so clients can communicate with the Spark cluster via either language remotely. Also, batch job submissions can be done in Scala, Java, or Python.</p> <p>It enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark context management, all via a simple REST interface or an RPC client library.</p> <p>HPE Ezmeral Unified Analytics provides functions like %reload_ext sparkmagics and %manage_spark for seamless connection to the Spark cluster. %reload_ext sparkmagics loads the Spark session and authenticates the user for secured access to the Spark session. %manage_spark will create the Spark session with predefined Spark cluster configuration in the background.</p> <p><img src="https://lh3.googleusercontent.com/uLQ2f7fgl0NeTwPaf9L9WvumSN4prHhtxZl7jdM1fdCKfOjGJmGqphn5DPqiUkDhP4-sNUEbpLlW_EuyEzx8zGx9lzuDPjrM6j6SGo6Rbm8LO1zWenZBpKQGfuNNk4IjG1gfZX5O__F1HBA4jv_pH-w" alt=""></p> <p>Once the Livy session is enabled, the code can be run on the notebook servers.</p> <p><img src="https://lh3.googleusercontent.com/NLB3_xj7PZ0W5w9JDXVvMGL3gHOZpV1pdWlMf2_toidQjxWAv7kRuY9n8AQRatl55Ht1ULl-v-BeiWB0TPJi-Pv4f-nwyFMEsjluZPUIzY-gXThHbzLGLs_3Nv7bjHD3EX-pV-1l4BZv3Geanr0VN9k" alt=""></p> <h2>Spark Streaming</h2> <p>Financial applications like real-time transaction processing, fraud detection, trade matching and settlement systems are widely distributed and deal with large volume and variety of data. These systems require parallel processing of transactions in a distributed computing architecture. Hence, Spark streaming best suits the needs of such financial applications like the stock market prediction analysis.</p> <p>Spark Streaming is a real-time data processing module in Apache Spark, a popular distributed computing framework for big data processing. It enables processing and analysis of live data streams in a scalable and fault-tolerant manner. Spark Streaming brings the power and flexibility of Spark's batch processing capabilities to real-time data streams, making it a versatile choice for various real-time data processing use cases.</p> <p>Micro-Batch Processing: Spark Streaming follows the micro-batch processing model, where it divides the continuous stream of data into small, discrete batches. Each batch of data is processed as a RDD (Resilient Distributed Dataset), which is Spark's fundamental data abstraction for distributed computing. This approach allows Spark Streaming to process data in mini-batches, providing low-latency processing and better resource utilization.</p> <p>Data Sources and Sinks: Spark Streaming can ingest data from various sources, including Kafka, Flume, Kinesis, HDFS, TCP sockets, and more. It supports a wide range of input formats, making it compatible with different streaming data pipelines. Similarly, Spark Streaming can write the processed data to various sinks, such as HDFS, databases (e.g., MySQL, Cassandra), and external systems.</p> <h2>Notebook servers</h2> <p>HPE Ezmeral Unified Analytics is equipped with notebook servers that can execute Python commands seamlessly along with scalable resources like CPUs, GPUs, and memory. Notebook servers can be spun up on Kubeflow using pre-defined Jupyter notebook images or custom-built notebook images based on your requirement. It will take a few minutes to bring the notebook server up and running. </p> <p><img src="/img/05-kubeflow-new-notebook-screen-shot.png" alt=""></p> <p>Once it is available, you can connect to the notebook server either on HPE Ezmeral Unified Analytics Notebooks Tab or directly from the Kubeflow Notebooks.</p> <p> <img src="https://lh6.googleusercontent.com/bY2CLk9ru69g0lTgPTpHECnv1oS_LZrRpeP7h3zN3-LlwQQMIAVE5LPeuD7aaOu0xPmrQN7tNwv_gqSYWfWwrTIrXUNe4U179lxbHUwOdlio0SeDohJB4nfan2IGYhg00xL8LylNoDVlV6rAUSoABFI" alt=""></p> <h2>MySQL Database</h2> <p>A MySQL database was created and hosted in Microsoft Azure to capture the structured streaming data to a single table. The database server is configured to permit access to the select IP addresses.</p> <h2>Step 3: <strong>Streaming data to database</strong></h2> <p>The data is read from HPE Ezmeral Data Fabric volume by the Spark Streaming engine in constant time intervals. The Spark engine converts the files into batches and does some data engineering like transformations and aggregations on the data. Finally, it is saved to MySQL database using jdbc connections. It is mandatory for all the incoming files to share the same schema.</p> <p>3.1 Load the required Spark libraries.</p> <p>Once connected to the Livy server, the Spark connection is configured and managed internally by HPE Ezmeral Unified Analytics platform. Now you can directly import the required libraries and you’ll be ready to use Spark. </p> <p><img src="https://lh5.googleusercontent.com/3-oBLpDhQ6DY4QJSn4K9-nR3fXB2IMtpQuphVxIcR9rd6SpSVaolvNg_xTp9VwUX726oJZJE8Cb9ii4xhrrJc-QtTee5xn2jes1qLDPnRVj3GKQUoWoCvl43IeACRSzCMJmVtDIOTK-p6CZ5eZFmj6s" alt=""></p> <p>3.2 Define the Data Schema</p> <p>Define the data schema for the data to stream in the application.</p> <p><img src="https://lh6.googleusercontent.com/vwbcqYFB0FdwVTsb_lFDHyBZKBWRrv18B0P3BWRaHWG2q7tXS-nNTrUEjA_GnJVP5e5r2RnZZ3ds3GLCVTY0XZOj6L9WJtL-f3ll_COXW1YIW8PjfqcVNQGPE31ODOhhEFeAKLvGDvBi-_0v9r2z3eE" alt=""></p> <p>3.3 Read the input stream of files from the external server</p> <p><img src="https://lh5.googleusercontent.com/LRu4GZMlkTQZ8crHQBXBXd5aFtFDzATuVJXSXOpAcQDb3DSQ8NUH4PQuXAsATtrTcUON1j9quD28gFuzc_Bgj3er771mdrR62JtSKezBrcTlxPCwI8tVXseRg1pxeSxjh-ktolizmQV08H8u8_QaJ20" alt=""></p> <p>3.4 Write the output stream to the destination path.</p> <p><img src="https://lh5.googleusercontent.com/aUcWv_pT4sW9AUowSlfnBHKBRy_BCebTyWO99epi3koEaw8srIYP5uLFQc6WKFUt9ZaFKCfrSS1ewWeWScU2-Tv65fHmcM2CPEGGEN8l7YZZv8zis_LEBq4qcr3qS2_8g3eBPFrqXdvLuXwhShdOL3Y" alt=""></p> <p>3.5 Read the data using Spark SQL and perform Exploratory Data Analysis.</p> <p><img src="https://lh4.googleusercontent.com/gMBQ7IbeVKFi9YDjjVn485VXx8Kt-jcJvNBFV7HsWYjGqnSZzpdEKrgfvypRiTCUIYMVLlOag3FDjwhNFPdVJyZrXqqf9cOiLJkJ6T89MajBJccQizawt0lqo7gPrVM4DPTEbx9dJj5zR7BHVwbnFUs" alt=""></p> <h2>Step 4: <strong>Connecting the database to HPE Ezmeral Unified Analytics</strong></h2> <p>HPE Ezmeral Unified Analytics provides users with a quick and simple process to connect to external data sources like different databases, Hive, Snowflake, Teradata, etc. Here a new data source connection is added, and the source is selected as MySQL. The connection is established once the jdbc connection url, username and password are validated.</p> <p><img src="https://lh6.googleusercontent.com/BmmyC843T9xG9NukA_XqQ7-qxTD8S9SP3y1IH6UGnMflFmEYDMAN-J6-YgURJAuBFPC5fNkrXZBGKdHDthHo-nRqOQO_5R3W04FtYGq67yIiJRmCxQrBq0qZC-4lhzeD7wXe5-H8Bem1_X9EXDTCK2M" alt=""></p> <p>This will connect the database to EzPresto, which is a distributed analytic query engine for big data, integrated into HPE Ezmeral Unified Analytics. This enables users to query the tables in the database using SQL commands. This service helps users to use the database seamlessly by enabling them to insert, delete, update and query records from the tables. The data can be accessed from a remote server or on HPE Ezmeral Data Fabric Software. </p> <p><img src="https://lh3.googleusercontent.com/EJZsyxnH6tUhlduTbkddolYL3QHc3QUmAihD_QmE8L0_p1j_y1DfgA8HE0ug1dF3RecvRuomELlrp7LtSYIqWuk6U2vs15MR6SvmnLz-1zTeZGf_v9iXzkCu43kyrEpucyHnBVRTvAXb7nff5uWXEiM" alt=""></p> <h2>Step 5: <strong>Visualization using Superset</strong></h2> <p>Apache Superset, a data visualization tool has been integrated into HPE Ezmeral Unified Analytics, which helps with the graphical representation of data, from simple line charts to highly detailed geospatial charts. The dashboards help users to get a clear picture of the KPIs and other relevant metrices of the business.</p> <p>Here, a new dashboard is created in HPE Ezmeral Unified Analytics, and the connection to the database is established. Different visuals on the stock data are integrated into the dashboard and it is customized to auto refresh to a customer-defined time interval. Once the data starts streaming, the dashboard updates the visuals periodically and the latest data is available on the dashboard for analysis.</p> <p><img src="https://lh6.googleusercontent.com/MvJQkbWinxA42eJ6J0-mOK-9sBNdh6JOeU0gMAplO308IyLPIFB1H7oweUZ9cTMngpkr4qyi9saWsQ9PaNFGSBxF8O9DE8HctUaqHmvDPGFmEL586UAySdnAW6b9KFxrC2vbUyiCP-WUvkeDjd5BJxI" alt=""></p> <p><img src="https://lh3.googleusercontent.com/QmnYcBblPyGIxZ-6lElbDCn0lCcG7PcPqSxHCwFslo3IVG9ExQxvSWdvmB1s5W6uOeiCJMCmSEuIZSUBtF8gJxrHP1nIw8qPyUTZH3QEn4kvPbW-GyXfNLvP4bAAbdIgbtOg6MsccLM-USWpFuDeKJ8" alt=""></p> <p><img src="https://lh3.googleusercontent.com/nQul3Zy2RuOYA7RXOqokiTIhCLQYDktS_VzCU5YBVLSjItmICHnRCQlLp_pzyLR1tnHKbvCcXLBnG7E7JeYfKh7pBC_OGGpxS5ewCKfi2IWvzXA7LVIq05Lk9DNFVdlc9EuqsxDN_xiPGrMsAWW8B38" alt=""></p> <h2>Summary</h2> <p>In concluding Part 1 of this blog series, you’ve journeyed through the data engineering and analytics aspects of using Spark, EzPresto, and Superset, powered by HPE Ezmeral Unified Analytics. With a spotlight on assimilating external pricing data to craft a dynamic dashboard, I hope I have illuminated how this platform brings together best of breed open-source tools to transform complex data into valuable insights.</p> <p>Don't miss Part 2, where you’ll get to explore the machine learning capabilities of our platform. To get familiar with HPE Unified Analytics Software, <a href="https://www.hpe.com/us/en/hpe-ezmeral-unified-analytics.html">try it</a> for free or visit our <a href="https://docs.ezmeral.hpe.com/unified-analytics/11/index.html">website</a> for details. Let's unlock the future of analytics together!</p> <p>Contributors to this blog post include Suvralipi Mohanta (<a href="suvralipi.mohanta@hpe.com">suvralipi.mohanta@hpe.com</a>), Harikrishnan Nair (<a href="mailto:harikrishnan.nair@hpe.com">harikrishnan.nair@hpe.com</a>), and Joann Starke (<a href="joann.starke@hpe.com">joann.starke@hpe.com</a>).</p>https://developer.hpe.com/2023-September-05/https://developer.hpe.com/2023-September-05/Tue, 05 Sep 2023 13:00:00 GMThttps://developer.hpe.com/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/set-up-load-balancer-with-metallb-in-hpe-greenlake-for-private-cloud-enterprise/Fri, 01 Sep 2023 08:59:11 GMT<p>This blog post describes how to set up the load balancer using <a href="https://metallb.universe.tf/">MetalLB</a> for a Kubernetes (K8s) cluster in <a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a>. It allows customers to configure load balancing services for their workloads deployed in K8s clusters.</p> <h3>Overview</h3> <p>Using <a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, customers can create services with the type of <em>NodePort</em> for their workloads deployed in K8s clusters using the label <em>hpecp.hpe.com/hpecp-internal-gateway=true</em>. The services will be automatically exposed to a container platform gateway host with assigned ports. The deployed workloads will become accessible externally using the gateway host name and the assigned ports as access URLs.</p> <p>Different from various public cloud providers, such as <em>GCP</em>, <em>AWS</em> and <em>Microsoft Azure</em>, HPE GreenLake for Private Cloud Enterprise doesn’t support network load balancers by default. As cluster administrators, after you create K8s cluster, you can integrate cluster with any load balancers in place to support K8s services of type <em>LoadBalancer</em>. This blog post will show you how to use MetalLB to provide load balancing services for K8s clusters in HPE GreenLake for Private Cloud Enterprise. It provides customers with the flexibility to configure custom load balancers for their deployed workloads in HPE GreenLake for Private Cloud Enterprise.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you meet the following requirements:</p> <style> li { font-size: 100%; line-height: 23px; max-width: none; } </style> <ul> <li>A K8s cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The kubectl CLI tool, together with the kubeconfig file for accessing the K8s cluster</li> <li>A range of virtual IP addresses. Those IP addresses should not be used in any existing K8s clusters. They will be assigned to the load balancer services.</li> </ul> <h3>Deploy MetalLB for load balancing</h3> <p><a href="https://metallb.universe.tf/">MetalLB</a> is a software solution that provides a network load balancer implementation for Kubernetes clusters using standard routing protocols. By installing MetalLB, it will support the LoadBalancer services within the Kubernetes clusters.</p> <p>This section describes the detailed steps to deploy MetalLB and configure it to support the <em>LoadBalancer</em> services in the Kubernetes clusters.</p> <h4>1. Deploy MetalLB</h4> <p>MetalLB can be deployed by applying the following YAML manifest file:</p> <pre><code class="language-markdown">$ MetalLB_RTAG=$(curl -s https://api.github.com/repos/metallb/metallb/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//') $ echo $MetalLB_RTAG 0.13.10 $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v$MetalLB_RTAG/config/manifests/metallb-native.yaml namespace/metallb-system created customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created serviceaccount/controller created serviceaccount/speaker created role.rbac.authorization.k8s.io/controller created role.rbac.authorization.k8s.io/pod-lister created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/controller created rolebinding.rbac.authorization.k8s.io/pod-lister created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created secret/webhook-server-cert created service/webhook-service created deployment.apps/controller created daemonset.apps/speaker created validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created </code></pre> <p>The above command installs the latest MetalLB <em>v0.13.10</em> to the K8s cluster. It first creates the namespace <em>metallb-system</em>, sets up the role-based access control (<em>RBAC</em>), creates a list of customer resource definitions (CRDs) and finally deploys a list of pods and services.</p> <p>You can check and confirm all pods and services are deployed successfully:</p> <pre><code class="language-markdown">$ kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/controller-7967ffcf8-8lgwc 0/1 Running 0 37s pod/speaker-24l42 1/1 Running 0 36s pod/speaker-g2q9h 1/1 Running 0 36s pod/speaker-kkmsj 1/1 Running 0 36s pod/speaker-ss4w7 1/1 Running 0 36s pod/speaker-xl7bv 1/1 Running 0 36s pod/speaker-zfl7s 1/1 Running 0 36s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/webhook-service ClusterIP 10.105.154.106 &#x3C;none> 443/TCP 38s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/speaker 6 6 4 6 4 kubernetes.io/os=linux 37s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/controller 0/1 1 0 38s NAME DESIRED CURRENT READY AGE replicaset.apps/controller-7967ffcf8 1 1 0 38s </code></pre> <h4>2. Define a range of IP addresses</h4> <p>After all MetalLB components are deployed, you can start creating and allocating a range of IP addresses, which can be used by MetalLB to assign IP addresses to services.</p> <p>The customer resource definition (CRD) <em>IPAddressPool</em> will be used for defining the range of IP addresses. After it’s deployed to the cluster, all the IP addresses will be allocated for MetalLB to use.</p> <pre><code class="language-markdown">$ cat IPAddressPool.yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: cfe-pool namespace: metallb-system spec: addresses: - 172.16.17.250-172.16.17.254 $ kubectl apply -f IPAddressPool.yaml ipaddresspool.metallb.io/cfe-pool created $ kubectl get IPAddressPool -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES cfe-pool true false ["172.16.17.250-172.16.17.254"] </code></pre> <p>The above command allocates the IP pool that has the IP range 172.16.17.250-172.16.17.254. The IP addresses in the <em>IPAddressPool</em> can be defined by <em>CIDR</em> and <em>IPV6</em> addresses as well.</p> <h4>3. Announce the service IP addresses</h4> <p>Once the IP addresses are allocated, you must announce service IPs. The <a href="https://metallb.universe.tf/configuration/#announce-the-service-ips">MetalLB Configuration site</a> shows a list of configuration approaches you can use to announce service IPs. The below example shows the details of using the <em>Layer 2</em> mode to configure service IP addresses. This approach does not need any protocol specific configuration, only IP addresses from the <em>IPAddressPool</em>.</p> <pre><code class="language-markdown">$ cat L2Advertisement.yaml apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: example namespace: metallb-system spec: ipAddressPools: - cfe-pool $ kubectl apply -f L2Advertisement.yaml l2advertisement.metallb.io/example created $ kubectl get L2Advertisement -n metallb-system NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES example ["cfe-pool"] </code></pre> <h3>Deploy Nginx app as the service type <em>LoadBalancer</em></h3> <p>As a sample web application, the <em>Nginx</em> with the service type of <em>LoadBalancer</em> will be deployed to the K8s cluster using the following YAML manifest file:</p> <pre><code class="language-markdown">$ cat nginx-deployment.yaml apiVersion: v1 kind: Service metadata: name: cfe-nginx-app labels: app: nginx-app spec: type: LoadBalancer ports: - port: 80 name: http selector: app: nginx-app --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx-app name: cfe-nginx-app spec: replicas: 1 selector: matchLabels: app: nginx-app template: metadata: labels: app: nginx-app spec: volumes: - name: webdata emptyDir: {} initContainers: - name: web-content image: busybox volumeMounts: - name: webdata mountPath: "/webdata" command: ["/bin/sh", "-c", 'echo "&#x3C;h1> Hi, this is the sample &#x3C;font color=blue>Nginx App&#x3C;/font> deployed as the Load Balancer service type !&#x3C;/h1>" > /webdata/index.html'] containers: - image: nginx name: nginx volumeMounts: - name: webdata mountPath: "/usr/share/nginx/html" $ kubectl apply -f nginx-deployment.yaml service/cfe-nginx-app created deployment.apps/cfe-nginx-app created </code></pre> <p>You can check the Nginx application deployment by typing the following command, using the label <em>app=nginx-app</em>, and confirm all pods and services are in running states. For the sample service <em>cfe-nginx-app</em>, you should see it’s been deployed as the <em>LoadBalancer</em> type and an IP address, <em>172.16.17.250</em>, gets assigned as its <em>EXTERNAL-IP</em> :</p> <pre><code class="language-markdown">$ kubectl get all -l app=nginx-app NAME READY STATUS RESTARTS AGE pod/cfe-nginx-app-66cb4f5bbf-4nfw5 1/1 Running 0 3m20s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cfe-nginx-app LoadBalancer 10.98.244.64 172.16.17.250 80:31631/TCP 3m22s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cfe-nginx-app 1/1 1 1 3m21s NAME DESIRED CURRENT READY AGE replicaset.apps/cfe-nginx-app-66cb4f5bbf 1 1 1 3m22s </code></pre> <p>To verify the deployed Nginx application is working, launch your web browser and open <em><a href="http://172.16.17.250">http://172.16.17.250</a></em>. The following should now show in your browser:</p> <p><img src="/img/web-nginx-app.png" alt=""></p> <h3>Summary</h3> <p>This blog post describes the detailed process used to deploy and set up MetalLB to support customers to configure load balancers for K8s clusters in HPE GreenLake for Private Cloud Enterprise. By deploying load balancing configuration, it provides an externally accessible IP address that sends traffic to the deployed workload. It also allows customers to use <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Kubernetes Ingress</a>, in place of <em>Service</em>, for different traffic routing support of their deployed workloads in K8s clusters. This supports deploying applications by passing the customer certificates through their own authority. It unblocks a list of potential use cases and enhances HPE GreenLake by providing additional flexibilities.</p> <p>You can keep coming back to the <a href="https://developer.hpe.com/blog">HPE Developer blog</a> to learn more about HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/hpe-nonstop-devops/https://developer.hpe.com/hpe-nonstop-devops/Wed, 30 Aug 2023 13:50:59 GMT<p>The DevOps approach has gained significant momentum over the past decade as today’s enterprises require shorter development and deployment cycles to gain faster time to market and obtain fast feedback for improvement. DevOps helps in continual product rollouts by bridging silos and streamlining change management practices, improving not only  development efficiency but also ensuring consistency and reliability of deployments.</p> <p>Gartner forecasts a 5.1% increase in Worldwide IT Spending in 2023, with organizations focused on realizing operational efficiency, cost reductions and/or cost avoidance during the current economic uncertainty. One can infer from this that investments in DevOps and improving DevOps practices is likely to increase. As HPE NonStop customers increasingly embrace DevOps, this is the right time to look at DevOps offerings for the HPE NonStop platform. And now is even more important than ever, as HPE NonStop is takings it steps towards becoming cloud ready!</p> <p>DevOps is a culture and a continuous process of improving efficiency. It can be applied to the various phases of software development, deployment, and post deployment with the aid of tools and automation. While the culture aspect is organization specific, in this article, I will discuss the tools recommendation, automation and artifacts that aid the DevOps adoption for HPE NonStop.</p> <h1>HPE NonStop is DevOps ready!</h1> <p>In DevOps, you build and operate Continuous Integration/Continuous Delivery (CI/CD) pipelines. Continuous Integration includes phases such as plan, code, build, and test, while Continuous Delivery (CD) covers the continuous delivery and deployment aspects, including release, deployment, operation, and monitoring of the software. </p> <p>HPE NonStop is DevOps ready. This means that, for each of the phases, there are tool sets specific to the platform identified and workflows that can be automated to achieve efficient, repeatable, and reliable production ready software.</p> <h1>HPE NonStop DevOps starter kits</h1> <p>HPE NonStop supports multiple languages and usage of DevOps for HPE NonStop is classified based on languages. (<a href="https://github.com/HewlettPackard/NonStop/blob/main/nsdevops/images/CustomerUsageProfileClassification.jpg">CustomerUsageProfileClassification</a>). This classification is not only required for toolset recommendations, but also for demonstrating the tool usages through starter kits. The starter kits are customer usage profile specific, ready to use, developer-friendly and production-ready. Each starter kit consists of a sample application (typically client/server), a set of pipeline scripts and a README file with usage instructions. These are hosted on GitHub <a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops">https://github.com/HewlettPackard/NonStop/tree/main/nsdevops</a></p> <p>The intent of the starter kit is to demonstrate pipelines and tools for a language usage profile. These reusable scripts and pipelines in the starter kit are an effortless way to get started with DevOps and try it out with customer applications.</p> <p>The starter kits currently available are:</p> <style> table { display: block; width: 100%; width: max-content; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; padding: 10px !important; } </style> <table> <thead> <tr> <th><strong>Name</strong></th> <th><strong>Description</strong></th> <th><strong>Location</strong></th> </tr> </thead> <tbody> <tr> <td>C Starter Kit</td> <td>CI sample for C/C++ based applications built using cross-compilers</td> <td><a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops/c">C</a></td> </tr> <tr> <td>Java Starter Kit</td> <td>CI sample for Java based applications built off-platform compilers</td> <td><a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops/java">java</a></td> </tr> <tr> <td>Java JNI Starter Kit</td> <td>CI sample for polyglot Java and C applications built on-platform on HPE NonStop</td> <td><a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops/javajni">javajni</a></td> </tr> <tr> <td>Python Starter Kit</td> <td>CI sample for Pythons based applications</td> <td><a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops/python">python</a></td> </tr> <tr> <td>CD Starter Kit</td> <td>Continuous Deployment using HPE NonStop Manageability Framework (NSMF) and Ansible</td> <td><a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops/cd">cd</a></td> </tr> </tbody> </table> <p>Most of the starter kits demonstrate continuous integration (CI). For the continuous deployment, NSMF and Ansible-based deployments are used. While the CD starter kit is demonstrated with a Java application, the dev, test and production environment setup and concepts remain the same for application of any language profile. The CD starter kit can also be extended to HPE NonStop system configuration management and HPE NonStop system administration activities.</p> <h1>Using the Starter Kits  </h1> <p>Developers new to DevOps should use the <a href="https://github.com/HewlettPackard/NonStop/blob/main/nsdevops/HPE%20Nonstop%20Server-Modern%20DevOps-Instructions-for-CI-CD-Setup%20Documnet_v1.2.pdf">HPE NonStop ModernDevOps - Instructions for CI-CD setup</a> for preparing the environment to get started. If the enterprise already has a DevOps setup, go through the recommended setup (<a href="https://github.com/HewlettPackard/NonStop/blob/main/nsdevops/images/RecommendedSetup.jpg">Recommended Setup</a>) and the instructions in the above guide to ensure HPE NonStop systems are configured to be accessible in the CI Tool.</p> <p>Getting started with the starter kits is easy.</p> <ul> <li>First download the starter kit from the HPE NonStop <a href="https://github.com/HewlettPackard/NonStop/tree/main/nsdevops">git repository</a>.</li> <li>Select the appropriate folder applicable to language profile.</li> <li>Upload the code from that folder to the organization GIT. Go through the instructions in the README.md file of the starter kit and update the pipeline script to point to the right repository.</li> <li>Make appropriate changes based on the environment. Commit the changes to the organization git. The pipelines will start executing.</li> </ul> <p>The scripts can be reused for applications in the same language profile with minor changes, such as modifying GIT repository and build steps, if any.</p> <h1>Starter Kits are cloud ready!</h1> <p>As more development environments are moving to public cloud, cloud vendors are offering DevOps services. HPE NonStop development environment (NSDEE) and cross-compilers and tools are also available on the public cloud now.</p> <p>The starter kits have ready-to-use pipelines for popular cloud vendors and their DevOps services, such as AzureDevOps and AWS CodeBuild, can be used to demonstrate how the HPE NonStop development and deployment can be integrated easily using those cloud services.</p> <p>For the platform agnostic language profiles on HPE NonStop, such as Python and Java, follow the instructions given in the vendors’ DevOps documentation. Alternately, specific scripts are provided in the starter kit. The buildspec.yml file, if present in the root folder of the repository – i.e. the AWS CodeBuild, will automatically build the project. Similarly, while using Azure, create a build script or choose to use the one provided with the starter kit.</p> <p>The C, Python, Java starter kits are cloud ready. Currently, support is provided for AWS CloudBuild or AzureDevOps. In future, the HPE NonStop development team will include other cloud vendor specific scripts and integrations in the starter kits.</p> <p>DevOps is supported on HPE NonStop, and starter kits are here to help you get started. Go, explore the HPE NonStop DevOps starter kits now!</p>https://developer.hpe.com/a-guide-to-connecting-tableau-with-the-hpe-nonstop-database-sql-mx-v2/https://developer.hpe.com/a-guide-to-connecting-tableau-with-the-hpe-nonstop-database-sql-mx-v2/Mon, 28 Aug 2023 11:15:01 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>HPE NonStop is a platform and an operating environment that turns your applications into fault-tolerant apps. Trusting in over 40 years of continued evolution and developments on the platform, financial institutions payment processors, manufacturers and retailers continue to put their most mission-critical workloads on HPE NonStop.</p> <p>HPE NonStop offers a modern RDBMS, with interfaces, tools, and experts available to lead the integration into customers’ IT environments.</p> <p>HPE NonStop SQL/MX is an HPE-patented database that inherits the fault-tolerant design and coherent integration between OS and the database, and adopts industry-standard features and tools.</p> <h2>What does it mean to adopt industry standard features and tools?</h2> <p>This means that software and tools programmed to use industry-standard <a href="https://insightsoftware.com/blog/what-is-odbc/">Open Database Connectivity drivers (ODBC)</a> and <a href="https://www.ibm.com/docs/en/informix-servers/12.10?topic=started-what-is-jdbc">Java Database Connectivity (JDBC) drivers</a> will be able to connect to the HPE NonStop database.</p> <p>There are many data visualization and analytics tools that provide such connectors, and, in this tutorial, we will be connecting the SQL/MX database to Tableau, a leading data visualization tool used for data analysis and business intelligence.</p> <p><img src="/img/connectingtableau-tohpenonstopaql.png" alt="Connecting Tableau to HPE NonStop" title="Connecting Tableau to HPE NonStop"></p> <h2>What is Tableau?</h2> <p>Tableau is an excellent data visualization and business intelligence tool used for reporting and analysing vast volumes of data, helping users create charts, graphs, and dashboards that assist in business decision making. It helps users see and understand their data through its intuitive interface and feature-packed application. Tableau has spent a decade as a leader in the Gartner Magic Quadrant for Analytics and Business Intelligence space, and remains one of the top business intelligence platforms for graduates just out of college and <a href="https://www.tableau.com/solutions/customers">top companies, such as LinkedIn and RedHat</a>.</p> <p><img src="/img/sampletableaudashboard-withsql.png" alt="Sample Tableau Dashboard with SQL/MX Database" title="Sample Tableau Dashboard with SQL/MX Database"></p> <h2>Connecting Tableau to the SQL/MX database</h2> <p>This article serves as a companion to the <a href="https://help.tableau.com/current/guides/get-started-tutorial/en-us/get-started-tutorial-home.htm">existing Tableau Tutorial to their desktop client</a>. The tutorial is quite good, so make sure you check it out and continue exploring Tableau’s features while being connected to your SQL/MX database.</p> <p>This document will provide specific guidance for connecting the Tableau software to the SQL/MX database.</p> <h3>Connection and ODBC prerequisites:</h3> <p>This tutorial assumes that HPE NonStop ODBC 3.x Unicode driver has already been installed. Check out the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00045523en_us&#x26;docLocale=en_US">HPE NonStop ODBC/MX Client Drivers User Guide</a> for more information on the driver.</p> <p>You can refer to this tutorial to get help in configuring your ODBC <a href="https://shanice-abigail.medium.com/python-how-to-use-odbc-to-connect-hpe-nonstop-sql-mx-44ca90047eb3">— Getting Prerequisites > Configuring your ODBC</a>.</p> <p>This tutorial also assumes that on your host, HPE NonStop SQL/MX has been installed, MXCS is running, and a MXCS data source has been added and started. Check with your administrator for the IP address, port number etc. (If you’re the administrator you can find out more in this <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=emr_na-a00090054en_us">manual</a>).</p> <h3>Data model and database structure:</h3> <p>The data for this tutorial was taken from Tableau’s tutorial and desktop client for “Superstore”. The raw data can be downloaded from Tableau’s desktop client, or through this link - <a href="https://github.com/shaniceabigail/medium-materials/raw/main/2022%20tableau%20demo/Superstore.xls">Superstore.xls from Tableau’s Tutorial (Click to Download)</a>.</p> <p>Each excel sheet is visualized as one table in the database. To import the data into SQL/MX, you need to:</p> <ol> <li>Save each excel sheet as a .csv file.</li> <li>Create tables in the database through MXCI.</li> <li>Import from OSS using the command: <strong>/usr/tandem/sqlmx/bin/import</strong> [catalog].[schema].[table name] <strong>-I</strong> [table name]<strong>.csv</strong>. For example: <em>/usr/tandem/sqlmx/bin/import catalog.schema.orders -I orders.csv</em></li> </ol> <h2>Connecting to the HPE NonStop server</h2> <p>Create a new workbook in Tableau and select the “Other Databases (ODBC)” option when choosing the type of connection to a server.</p> <p><img src="/img/newtableauconnection.png" alt="New Tableau Connection" title="New Tableau Connection"></p> <h3>Connection requirements</h3> <p>Here are the details you will need:</p> <ul> <li>Database username</li> <li>Database password</li> <li>Catalog name</li> <li>Schema name</li> </ul> <p><img src="/img/hpenonatopodbcdatasource.png" alt="HPE NonStop ODBC DataSource" title="HPE NonStop ODBC DataSource"></p> <p>You will see another window prompt — make sure that you select the DSN (data source name) that you have registered in your ODBC configuration.</p> <h3>Server details and format of server connection string</h3> <p>Server attributes needed:</p> <ul> <li>IP address</li> <li>Port number</li> <li>Catalog name</li> </ul> <p><strong>Format:</strong></p> <p><em>TCP:[IP address]/[Port number]</em></p> <p><em>Database name: [Catalog name]</em></p> <p><img src="/img/databaseserverdetails.png" alt="Database server details" title="Database server details"></p> <h2>Setting up tables for Tableau’s Superstore tutorial</h2> <p>If you’ve made it to this step, it means that your SQL/MX database has successfully connected to the Tableau software. Congratulations!</p> <p>Now you can configure the relationship between the tables in this database.</p> <p><img src="/img/succesfulconnectiontodb.png" alt="Shows a successful connection to database" title="Shows a successful connection to database"></p> <h3>Selecting tables for Superstore</h3> <p>Select the database and schema where you have created and populated the database. Click and drag the tables into the orange space indicated.</p> <p><img src="/img/dragtablesintothearea.png" alt="Click and drag tables into the area" title="Click and drag tables into the area"></p> <h3>Creating relationships between tables</h3> <ul> <li>Start with the “Orders” table, and then the “People” table</li> <li>A line between the tables, and a box showing the relation between the tables will appear</li> <li>Check the fields used to link the 2 tables together</li> <li>Repeat with the “Returned” table and “Orders” table</li> </ul> <p><img src="/img/relationshiporderstable-peopletable.png" alt="Relationship between orders table and People table" title="Relationship between orders table and People table"></p> <p>Once completed, you should be all set. Now you can continue onto the Tableau tutorial <a href="https://help.tableau.com/current/guides/get-started-tutorial/en-us/get-started-tutorial-drag.htm">Step 2: Drag and Drop to take a first look</a>.</p> <h2>Other data visualization tools that can work with SQL/MX</h2> <p>Tableau is not the only data visualization tool that works with HPE NonStop SQL/MX’s ODBC driver. Other applications such as Power BI, and even Excel, can also connect in a similar way to the HPE NonStop database.</p> <p><img src="/img/populardatavisualizationtools.png" alt="Popular data visualization tools" title="Popular data visualization tools"></p> <p>It’s important to note that not only is HPE NonStop SQL/MX ANSI compliant, but it also adopts ODBC and JDBC standards, allowing effortless database connection with a state-of-the-art fault tolerance.</p> <p>Don’t forget to check back frequently on the <a href="https://developer.hpe.com/">HPE Developer Community portal</a> to find more information on HPE NonStop.</p>https://developer.hpe.com/sivayanama-simplifying-open-api-specification-with-reusable-common-data-models/https://developer.hpe.com/sivayanama-simplifying-open-api-specification-with-reusable-common-data-models/Sat, 26 Aug 2023 08:06:19 GMT<p>The <a href="https://www.openapis.org">OpenAPI specification</a> (OAS) is one of the most widely followed API contracts. It is language-agonistic. With the help of the information found in the OpenAPI specification, clients can better understand APIs and how to invoke them without having access to the code or worrying about the implementation details.</p> <p>At times, this open specification file can be too complex to manage and understand. In this article, I will discuss the techniques to simplify the open specification with loosely coupled reusable data models.</p> <h2>Items found in the OpenAPI Specification</h2> <p>OpenAPI specifications files have many definitions in them. However, the below list of entities are typically bigger in terms of the number of lines and tend to be reused in the specification file.</p> <ul> <li>schemas</li> <li>pathitems</li> <li>parameters</li> <li>requestBodies</li> <li>responses</li> <li>headers</li> <li>examples</li> <li>linkscallbacks</li> </ul> <p>I will discuss four approaches to deal with these definitions.</p> <h2>Inline definition</h2> <p>With inline definition, the definition is inline right at the reference point, as shown below:</p> <pre><code class="language-yaml">schema: type: object properties: id: type: string name: type: string </code></pre> <h2>Inline inside components object</h2> <p>The components object in the OpenAPI specification is the home of reusable object definitions. However, these defined objects must be explicitly referred to outside of the components section wherever required.</p> <pre><code class="language-yaml">components: parameters: tenantId: in: path name: tenantId schema: type: string required: true description: Describes the clientId or mspId of tenant </code></pre> <p><em>$ref</em> is one of the fixed fields in the schema. It is a string value that refers to other components in the OpenAPI document, internally and externally. The above defined <em>tenantId</em> parameter can be referred to below:</p> <pre><code class="language-yaml">$ref: '#/components/parameters/tenantId’ </code></pre> <h2>Externalized definition</h2> <p>Data models can be defined outside of the OpenAPI specification file using a $ref reference to an external definition. For example:</p> <pre><code class="language-yaml">example-multiple-threshold-type-example-request: $ref: ./models/opsramp-monitoring-management/multiple-threshold-type-example-request-v1.yaml </code></pre> <h3>Content of multiple-threshold-type-example-request-v1.yaml</h3> <p>Please note that the request object definition should be defined within the <em>value:</em> as shown below:</p> <pre><code class="language-yaml">value: id: 1ecf993f-9b54-4ce3-9581-c365188f7e58 name: OpsRamp Gateway Performance Template description: Monitors basic UNIX parameters like UNIXCPU, UNIXSTORAGE, UNIXUPTIME, UNIXMEMORY, UNIXLOAD and UNIXStats resourceType: DEVICE collectorType: OpsRamp Gateway status: state: ACTIVE generation: 2 tags: Performance Monitors createdDate: '2022-10-09T15:03:44+0000' updatedDate: '2022-10-09T15:23:40+0000' scope: Client templateFamily: Performance Monitors Family notes: Sample notes related to performance monitors </code></pre> <h2>Externalized definition with local aliases</h2> <p>The above externalized definition can be further improved by defining local aliases. The local aliases can be used instead of repeating the relative path of the definition in all references. In the example shown below, the external definition referenced with the $ref can be referred by <em>#components/parameters/tenanId</em>:</p> <pre><code class="language-yaml">components: parameters: tenantId: $ref: ./models/opsramp-monitoring-management/tenantId-v1.yaml # this can be referred by #components/parameters/tenantId </code></pre> <h2>Advantages of externalized definitions</h2> <p>Externalized definitions have many advantages over traditional inline definitions, such as:</p> <ul> <li>Loosely coupled data models, and definitions from the OpenAPI Specification</li> <li>Reusable data models with common definitions</li> <li>This will reduce the OpenAPI specification files significantly</li> <li>Easy to manage and govern OpenAPI specification files</li> </ul> <h2>Conclusion</h2> <p>In this article, I explained four techniques used to simplify the OpenAPI Specification using loosely coupled, reusable data model definitions and pointed out the advantages of these approaches. Check back for more articles on this and other subjects on the HPE Developer <a href="https://developer.hpe.com/blog">blog</a>.</p>https://developer.hpe.com/getting-started-with-the-hpe-data-services-cloud-console-powershell-sdk/https://developer.hpe.com/getting-started-with-the-hpe-data-services-cloud-console-powershell-sdk/Mon, 07 Aug 2023 09:01:09 GMT<p>HPE provides the Data Services Cloud Console (DSCC), a SaaS-based cloud console, to eliminate silos and to reduce complexity by delivering a unified management of HPE storage arrays. It's UI may be sufficient for many customers to manage their HPE storage assets, but lately I was approached from multiple customers that were looking for an efficient way to manage their HPE storage assets with PowerShell scripts. When I started looking into the available options to access the DSCC API with PowerShell, I first found the <a href="https://developer.hpe.com/blog/new-powershell-toolkit-available-for-managing-hpe-data-services-cloud-console/">PowerShell toolkit for managing the HPE Data Services Cloud Console</a> on the <a href="https://developer.hpe.com/">HPE Developer Community Portal</a>. Working with this PowerShell toolkit, I realized that it is, on one hand, an honorable first approach by my colleagues to write such a toolkit, but it is not complete and not yet fully tested. Hence, a different solution needed to be found.</p> <p>The DSCC provides an public OpenAPI 3.x spec that can be used to generate your preferred language SDK as it is described for Python in this <a href="https://developer.hpe.com/blog/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/">blog</a>. Using the OpenAPI 3.x spec of the DSCC API has the benefit of providing complete list of possible API operations. And once the procedure to generate the SDK is in place, a new SDK can easily be created if new features are exposed via the DSCC API.</p> <p>In this blog post, I will describe the steps I used to generate a DSCC PowerShell SDK and show how this SDK is used for daily tasks.</p> <h1>Creating the DSCC Powershell SDK</h1> <p>My first step to generate the DSCC Powershell SDK was to download the current <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.yaml">YAML</a>- or <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.json">JSON</a> OpenAPI 3.x spec of the DSCC API that can be found on the <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">Data Services Cloud API documentation</a> page.</p> <p><img src="/img/dscc-api-spec.png" alt="DSCC API Spec" title="DSCC API Documentation"></p> <p>You can also download the DSCC API spec YAML file with the following PowerShell command:</p> <pre><code class="language-powershell">invoke-webrequest -uri https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.yaml -outfile storage-api.yaml </code></pre> <p>I ensured that I had the latest Microsoft PowerShell 7 installed on the Microsoft Windows system that I used to generate the DSCC PowerShell SDK. The easiest way to get the latest PowerShell 7 installed is to use the winget utility.</p> <pre><code class="language-powershell">winget search Microsoft.PowerShell winget install --id Microsoft.PowerShell --source winget winget install --id Microsoft.PowerShell.Preview --source winget </code></pre> <p>There are multiple generators available to create a SDK out of the OpenAPI 3.x spec. In the end, I had the best result with <a href="https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/6.6.0/">openapi-generator-cli-6.6.0.jar</a></p> <p>The minimum requirement to execute this JAR File is to have as a minimum JAVA Runtime Environment (JRE) version 8 installed. I used the <a href="https://learn.microsoft.com/en-us/java/openjdk/download">Microsoft OpenJDK 17.0.8 LTS</a> build as JRE on my system. Having all requirements in place, I simply created the DSCC PowerShell SDK with the following command:</p> <pre><code class="language-powershell">java -jar openapi-generator-cli-6.6.0.jar generate -i storage-api.yaml -g powershell -o dscc-powershell-sdk </code></pre> <p>The output directory that I specified with the -o flag of the generate command contains the following files:</p> <p><img src="/img/openapigeneratoroutput.png" alt="" title="Openapi Generator Output"></p> <p>After generating the DSCC PowerShell SDK, I needed to install the PowerShell modules (stored in the src directory) locally before I could use it in PowerShell scripts. The local installation was done simply by running the Build.ps1 script located in the openapi-generator-cli output directory that is specified with the -o flag.</p> <pre><code>powershell Build.ps1 </code></pre> <p>Having completed all these steps, I was ready to use the DSCC PowerShell SDK to automate daily DSCC tasks.</p> <h1>How to use the DSCC PowerShell SDK</h1> <p>The DSCC PowerShell SDK output directory contains a README.md file that gives a complete overview of all API Endpoints.</p> <p><img src="/img/dscc_ps_api_endpoints.png" alt="" title="DSCC PowerShell SDK API Endpoints"></p> <p>To do what I did, you can scroll through the README.md file and use the links in the file to get to the detailed information for every available API endpoint. The detailed information includes usage examples for the commands, like the following one that is used for getting all storage systems, <em>Invoke-SystemsList</em>:</p> <pre><code class="language-powershell"># general setting of the PowerShell module, e.g. base URL, authentication, etc $Configuration = Get-Configuration # Configure HTTP basic authorization: JWTAuth $Configuration.Username = "YOUR_USERNAME" $Configuration.Password = "YOUR_PASSWORD" $Limit = 10 # Int32 | Number of items to return at a time (optional) $Offset = 5 # Int32 | The offset of the first item in the collection to return (optional) $Filter = "name eq VEGA_CB1507_8400_2N_150" # String | oData query to filter systems by Key. (optional) $Sort = "id asc,name desc" # String | Query to sort the response with specified key and order (optional) $Select = "id" # String | Query to select only the required parameters, separated by . if nested (optional) # Get all storage systems try { $Result = Invoke-SystemsList -Limit $Limit -Offset $Offset -Filter $Filter -Sort $Sort -Select $Select } catch { Write-Host ("Exception occurred when calling Invoke-SystemsList: {0}" -f ($_.ErrorDetails | ConvertFrom-Json)) Write-Host ("Response headers: {0}" -f ($_.Exception.Response.Headers | ConvertTo-Json)) } </code></pre> <p>You may be wondering ... how did I get from the example given in the README.md file to usable code. First of all, I needed a Client Id and Client Secret in order to open a connection to the DSCC. If you do not have yet a Client Id / Client Secret pair, then you need to create one as it is described in the <!--StartFragment--><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Using HPE GreenLake Console's API Gateway for Data Services Cloud Console</a><!--EndFragment--> blog post.</p> <p>Using the Client Id / Client Secret pair, I was able to retrieve an Access Token that was valid for two hours.</p> <pre><code class="language-powershell">Import-Module -Name '.\dscc-powershell-sdk\src\PSOpenAPITools' $Configuration = Get-Configuration $Configuration.BaseUrl = 'https://eu1.data.cloud.hpe.com' $Configuration.Username = 'Your Client Id' $Configuration.Password = 'Your Client Secret' $AuthUri = "https://sso.common.cloud.hpe.com/as/token.oauth2" [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 $AuthHeaders = @{ 'Content-Type' = 'application/x-www-form-urlencoded' } $AuthBody = [ordered]@{ 'grant_type' = 'client_credentials' 'client_id' = $Configuration.Username 'client_secret' = $Configuration.Password } try { $Configuration.AccessToken = ( invoke-restmethod -uri $AuthURI -Method Post -headers $AuthHeaders -body $AuthBody ).access_token } catch { Write-Host ("Exception occurred when retrieving access toke: {0}" -f ($_.ErrorDetails | ConvertFrom-Json)) Write-Host ("Response headers: {0}" -f ($_.Exception.Response.Headers | ConvertTo-Json)) exit(-1) } </code></pre> <p>As you can see the Access Token, Client Id, Client Secret and the Base URL are stored in a global variable: $Configuration. The retrieved Access Token is valid for two hours and needs to be refreshed afterwards.</p> <p>Please note: the $Configuration.BaseUri depends on the DSCC location. You will need to use BaseUri given for your DSCC location. At the time of writing this blog the three possible DSCC BaseUri are:</p> <ul> <li>Europe: <a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></li> <li>USA: <a href="https://us1.data.cloud.hpe.com">https://us1.data.cloud.hpe.com</a></li> <li>APJ: <a href="https://jp1.data.cloud.hpe.com">https://jp1.data.cloud.hpe.com</a></li> </ul> <p>If you want to have a list of storage systems that you have access to on the DSCC, you can use the <em>Invoke-SystemList</em> API endpoint as shown in the following example:</p> <pre><code class="language-powershell">try { $Systems = (Invoke-SystemsList).items $SystemIds = @{} foreach($s in $Systems){ $SystemIds += @{$s.name = $s.Id} } } catch { Write-Host ("Exception occurred when calling Invoke-DeviceType1SystemsList: {0}" -f ($_.ErrorDetails | ConvertFrom-Json)) Write-Host ("Response headers: {0}" -f ($_.Exception.Response.Headers | ConvertTo-Json)) } </code></pre> <p>After getting the systems list with the <em>Invoke-SystemsList</em> command, I stored the system name and system ID in an array because I needed the system ID for many of the possible API commands. Generally it is the case that the systems, volume, hosts, initiators etc. you want to manipulate in your scripts are defined by their IDs. For example, if you work with the DSCC API, you will work with the id of the object you want to change or delete.</p> <p>For instance, if you want to create a new volume, then you need to define the volume parameters, like name, size, etc., and the system where you want to create the volume. A new volume can be created with the <em>Invoke-VolumeCreate</em> command that has two input parameters: the system ID and the volume parameters that are stored in a <em>CreateVolumeInput</em> object using the <em>Initialize-CreateVolumeInput</em> method.</p> <pre><code class="language-powershell">$SystemId = $System.Primera650 $CreateVolumeInput = Initialize-CreateVolumeInput -Comments "DSCC API -Thomas Beha" ` -Count 1 ` -DataReduction $true ` -Name "DSCC-API-Vol" ` -SizeMib 16384 ` -SnapCpg "SSD_r6" ` -UserCpg "SSD_r6" try { $Result = Invoke-VolumeCreate -SystemId $SystemId -CreateVolumeInput $CreateVolumeInput } catch { Write-Host ("Exception occurred when calling Invoke-CreateVolume: {0}" -f ($_.ErrorDetails | ConvertFrom-Json)) Write-Host ("Response headers: {0}" -f ($_.Exception.Response.Headers | ConvertTo-Json)) } </code></pre> <p>The <em>Invoke-VolumeCreate</em> command will return the information of the generated task. You can use the <em>Get-Task</em> command to monitor the progress of this task.</p> <pre><code class="language-powershell">$Status = Get-Task $Result.taskUri while( ($Status.progressPercent -lt 100) -and ($Status.state -ne "FAILED")){ $Status.progressPercent Start-Sleep -Seconds 5 $Status = Get-Task $TaskId } </code></pre> <p>These are the steps I used to create a new Volume on my HPE Primera 650 array with a name of DSCC-API-Vol and a size of 16GB.</p> <p><img src="/img/dscc-api-vol.png" alt="" title="DSCC-API-Vol UI view"></p> <h1>Summary</h1> <p>In conclusion, the DSCC PowerShell SDK, derived from the DSCC Open API 3.x spec, includes at the time of writing this blog already 349 commands that can be used to manage your storage with the DSCC API. At the moment, it t is the most complete PowerShell tool to manage DSCC managed HPE storage arrays. Since it is also easily updated with new features once they are available - simply recreate it with the most current DSCC OpenAPI 3.x spec - it is the recommended way for customers to build PowerShell scripts accessing the DSCC API.</p> <p>I hope the above example helps get you started with the DSCC PowerShell SDK. While I have not tested every possible command, I have not found any issue working with the SDK on the extensive daily task list received from my customers.</p>https://developer.hpe.com/2023-August-01/https://developer.hpe.com/2023-August-01/Tue, 01 Aug 2023 13:00:00 GMThttps://developer.hpe.com/sivayanama-testing-opsramp-apis-with-pytest-fixtures/https://developer.hpe.com/sivayanama-testing-opsramp-apis-with-pytest-fixtures/Wed, 26 Jul 2023 06:58:31 GMT<p>The OpsRamp platform provides a rich set of APIs. By using these APIs, customers can build solutions to automate various IT Operations Management (ITOM) workflows. These could include discovery and monitoring, event and incident management, or remediation and automation. Testing these APIs and the resulting automated solution is critical to ensure the reliability and stability of the workflow.</p> <p>PyTest is a powerful Python testing framework and it is widely used to test APIs. Fixtures are one of the most important capabilities of the PyTest framework. In this article, I will discuss some advanced techniques for testing OpsRamp APIs using PyTest fixtures.</p> <h2>What fixtures are</h2> <p>PyTest fixtures are a special type of Python function that provisions a fixed baseline for testing. With the help of this baseline, you can ensure tests are run reliably and produce consistent results. In addition, you can ensure that the same tests are repeatable.</p> <h2>Install PyTest</h2> <p>Run the below command.</p> <pre><code class="language-shell">pip install -U pytest </code></pre> <h2>Verify PyTest installation</h2> <p>Run the below command.</p> <pre><code class="language-shell">pytest --version pytest 7.4.0 </code></pre> <h2>Define fixture</h2> <p>You can define fixtures by decorating a simple Python function with <a href="https://docs.pytest.org/en/6.2.x/reference.html#pytest.fixture">@pytest.fixture</a>.</p> <p>Python Decorators are a very powerful and convenient way to alter the behaviour of functions. For more details, refer to <a href="https://www.geeksforgeeks.org/decorators-in-python">Python Decorators</a>.</p> <pre><code class="language-python">import pytest @pytest.fixture def hello_world(): return 'Hello, Happy testing' </code></pre> <h2>Invoke fixtures</h2> <p>Test functions can invoke fixtures just by declaring the required fixtures as arguments.</p> <pre><code class="language-python">def test_hello_world(hello_world): assert hello_world == 'Hello, Happy testing' </code></pre> <h2>Invoke OpsRamp API</h2> <p>As shown below, execute OpsRamp <a href="https://develop.opsramp.com/v2/api/auth/tenancy-auth-oauth-token"><em>Get Access Token</em> API</a>.</p> <pre><code class="language-python">import http.client import json import traceback import urllib from asyncio.log import logger import pytest def get_auth_header(api_endpoint, client_id, client_secret, grant_type="client_credentials"): """ Returns bearer token string in the blow format bearer adfa-afdaf-1599-402c-a3ee-1ed24f597cc8 """ api_connection = http.client.HTTPSConnection(api_endpoint) params = {"client_id": client_id, "client_secret": client_secret, "grant_type": "client_credentials"} payload = urllib.parse.urlencode(params) headers = { 'accept': "application/json", 'content-type': "application/x-www-form-urlencoded", } </code></pre> <h2>Define fixture to get access token</h2> <p>You can transform the above function as a PyTest fixture by simply decorating the code as shown below.</p> <pre><code class="language-python">@pytest.fixture def get_auth_header(api_endpoint, client_id, client_secret, grant_type="client_credentials"): </code></pre> <h2>Autouse fixtures</h2> <p>Sometimes some of your fixtures are required for all other tests. The above Get access token is the perfect use case for this. Every API call depends on this API. In this case, you can designate this fixture as an “autouse fixture” by passing in <em>autouse=True</em> to the fixture’s decorator. The autouse fixture capability will reduce a lot of redundant requests.</p> <pre><code class="language-python">@pytest.fixture(autouse=True) def get_auth_header(api_endpoint, client_id, client_secret, grant_type="client_credentials"): </code></pre> <h2>Provisioning of third-party plugin fixtures</h2> <p>You do not necessarily have to define all fixtures on your code. Fixtures can be provided by third-party plugins as well and you are free to use them. You just need to install the required plugins. You can see an example with the pytest-datadir plugin.</p> <p>Install this plugin as shown below.</p> <pre><code class="language-shell">pip install pytest-datadir </code></pre> <pre><code class="language-python">@pytest.fixture(autouse=True) def get_bearer_token(shared_datadir,request): json_path = (shared_datadir / "auth_header_input.json") </code></pre> <p><img src="https://lh4.googleusercontent.com/dacTgDdw17BzeyCitShA73WSip9LVtenQoNN-uraaN5tKEU5cA_xP3cEmNPWmTzU3A1HegdoOVvwPbyYqQuoLeEk4W766nIvpBdoTzUdIiT2dXiOQG0_h7atQWS7-T9qvRrieuhlEV84VS15ir11Ocw" alt=""></p> <p>The get_bearer_token fixture is using <a href="https://pypi.org/project/pytest-datadir/">shared_datadir</a> fixture from a third-party plugin. The <em>Shared_datadir</em> fixture provisions the data folder as <em>pathlib.Path</em> object.</p> <h2>Fixture finalization</h2> <p>Usually, you do teardown or cleanup activities for each test. There are many good reasons for this, such as:</p> <ul> <li>The executed tests will not impact other tests</li> <li>Do not want to have the testing environment piled up with tons of test data</li> <li>Every test should execute in a clean state</li> </ul> <p>Fixtures offer a very powerful teardown/cleanup capability known as finalization.</p> <pre><code class="language-python">@pytest.fixture() def create_client(shared_datadir, get_bearer_token, request): client = Client() client.logger.debug("IN***") payload_file = (shared_datadir / "create_client1.json") input_payload = client.http_helper.to_json(payload_file) # client_response=client.http_helper.do_post(url, input_payload, get_bearer_token) client_response = client.create_client(input_payload, get_bearer_token) def terminate_client(): client.logger.debug("IN***") util = Utils() clientId = Client.clientId if len(str(clientId)) > 0: oauth_token = get_bearer_token client.terminate_client(clientId, oauth_token) client.logger.debug("OUT***") request.addfinalizer(terminate_client) client.print_log(client_response) client.logger.debug("OUT***") return client_response </code></pre> <p>You should add the finalizer function to the request’s context object of the test as shown below:</p> <p><em>request.addfinalizer(terminate_client)</em></p> <p>In the above example, you do all the test set-up on the client and execute the test. Once the test is complete, the finalizer function runs and cleans up the whole thing.</p> <h2>Conclusion</h2> <p>In this article, I have explained what PyTest fixtures are and how to define, invoke and implement fixtures. I also showed you how to leverage the testing of OpsRamp APIs using these fixtures and how to do teardown activities using the finalizer functions. Check back often on the <a href="https://developer.hpe.com/blog">HPE Developer Community blog</a> to find more blog posts on OpsRamp.</p>https://developer.hpe.com/office-of-the-cto-announces-hiring-at-new-center-of-excellence-in-galway/https://developer.hpe.com/office-of-the-cto-announces-hiring-at-new-center-of-excellence-in-galway/Tue, 25 Jul 2023 10:59:41 GMT<p>At Hewlett Packard Enterprise (HPE), our purpose is to advance the way people live and work. And we invite you to join us in our mission! Hewlett Packard Enterprise and the Office of the Chief Technology Officer recently announced hiring for several roles at a new Center of Excellence in Galway, Ireland. The new cloud R&#x26;D center will help expand the capabilities of the HPE GreenLake edge-to-cloud platform.</p> <h2>HPE expands presence in Galway</h2> <p>The Office of the CTO is focused on leading HPE engineering teams to build differentiated inventions and co-inventions that drive secure, sustainable and transformative outcomes, guiding the industry into a more sustainable, equitable and efficient future through engineering design and innovation.</p> <p>This new cloud R&#x26;D center is integral to the development of the HPE GreenLake platform, which is the foundation of HPE’s as-a-service offerings. The HPE GreenLake platform is used by our customers to accelerate data-first modernization with cloud services that can run on premise, at the edge, in a co-location facility, and in the public cloud.</p> <p>HPE already maintains a large presence in Galway and serves as the hub for Digital Services R&#x26;D, HPE GreenLake Cloud Services &#x26; Solutions R&#x26;D, and Cyber Security in Europe.</p> <h2>The Office of the CTO wants you!</h2> <p>The new roles announced by the Office of the CTO will round out a team of software R&#x26;D professionals who will support customers and partners with a unified hybrid cloud experience and easy access to cloud services.</p> <p>Now is your opportunity to join the team as we advance the platform forward. The new Center of Excellence will recruit top talent, from graduates to experienced technology professionals, across a range of roles including: architects, software engineers, product, engineering and project managers, scrum maters, researchers, user experience and test engineers, security specialists, data analysts and AI professionals.</p> <h2>Join us today!</h2> <p>At Hewlett Packard Enterprise, we seek candidates who are relentlessly curious and embrace courage over comfort. We prioritize candidates who have a strong desire to be a force for good and are dedicated to use technology to make the world better. We value diverse backgrounds and those with a variety of experiences.</p> <p>Our team members help create history, pushing the industry forward by helping to advance our customers’ and partners’ businesses in ways they could not previously have imagined. If you’re looking for an opportunity that’s as challenging as it is rewarding, come join us.</p> <p>Check out our <a href="https://careers.hpe.com/us/en/hpe-chief-technology-office">hottest Office of the CTO employment opportunities</a> here. You can also view more information using the following links:</p> <p>Global: <a href="https://careers.hpe.com/hpe-octo">https://careers.hpe.com/hpe-octo</a></p> <p>Ireland: <a href="https://careers.hpe.com/us/en/hpe-octo-ireland-jobs">https://careers.hpe.com/us/en/hpe-octo-ireland-jobs</a></p> <p>India: <a href="https://careers.hpe.com/us/en/hpe-octo-india">https://careers.hpe.com/us/en/hpe-octo-india</a></p> <p>US: <a href="https://careers.hpe.com/us/en/hpe-octo-us">https://careers.hpe.com/us/en/hpe-octo-us</a></p> <p>Ft. Collins (campaigns only): <a href="https://careers.hpe.com/hpe-octo-ft-collins">https://careers.hpe.com/hpe-octo-ft-collins</a></p>https://developer.hpe.com/open-sourcing-workshops-on-demand-part4-managing-the-backend/https://developer.hpe.com/open-sourcing-workshops-on-demand-part4-managing-the-backend/Tue, 18 Jul 2023 09:28:35 GMT<p>In previous articles of this series dedicated to the <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">open sourcing of our Workshops-on-Demand project</a>, I covered the reasons why we open sourced the project and how we did it. I also explained in details how you could install your own Workshops-on-Demand backend server. I also took the time to detail the automation that was hosted on this backend server. Today, I plan to describe to you the management of this backend server. This is what is often referred to as Day2 operations.</p> <p>Once up and running, the main purpose of the backend server is to deliver workshops-on-Demand. But to do so, it may require updates, upgrades, and/or new kernels for the JupyterHub server. If new workshops are created, this means you'll need new jinja templates for related workshops' scripts (i.e <code>create&#x3C;WKSHP>.sh</code>, <code>cleanup&#x3C;WKSHP>.sh</code>, <code>reset&#x3C;WKSHP>.sh</code>, among others). This also means new variable files. And obviously, these templates and variables will need to be taken into account by scripts and notebooks. Some tasks handle all of this. And that's what I'll show now.</p> <h4>Backend server management:</h4> <p>If you take a look at the file structure of the <code>wod-backend</code> directory, you will discover that the team did its best to sort things properly depending on their relationship to system or workshops.</p> <h5>Content of the backend server:</h5> <p>Simple tree view of the wod-backend directory:</p> <p><img src="/img/wod-blogserie2-tree1.png" alt="" title="Tree view of wod-backend directory"></p> <p>The <code>ansible</code> folder contains all the necessary playbooks and variables files to support the main functions of the backend server. It provides playbooks for a minimal installation of the servers or appliances. It also allows the setup of the different types of servers (i.e backend, frontend, and/or api-db), appliances (virtual machines or containers), or workshops as well as maintenance tasks.</p> <p>At the root of this directory can be found:</p> <p><code>Check*.yml playbooks</code>: These playbooks are used to perform checks on the different systems. These checks ensure that this a compliant WoD system by checking firewall rules and many other things. You will see this a bit later in more details.</p> <p><code>Copy_folder.yml</code>: Historically, this is one of very first playbook we used and therefore, it is very important to me. It performs the necessary actions to deploy and personnalize (by substituting Ansible variables) the selected notebook to the appropriate student home folder.</p> <p><code>compile_scripts.yml</code>: Should you need to hide from the student a simple api call that is made on some private endpoint with non-shareable data (credentials for instance), this playbook will make sure to compile it and create a executable file allowing it to happen.</p> <p><code>distrib.yml</code>: This playbook retrieves the distribution name and version from the machine it is run on.</p> <p><code>install_*.yml</code>: These playbooks take care of installing the necessary packages needed by the defined type (frontend, backend, api-db, base-system or even appliance).</p> <p><code>setup_*.ym</code>: There are several types of setup playbooks in this directory.</p> <ul> <li><code>setup_WKSHP-*.yml</code>: These playbooks are responsible for preparing a base appliance for a given workshop by adding and configuring the necessary packages or services related to the workshop.</li> <li><code>setup_appliance.yml</code>: This playbook is used to perform the base setup for a JupyterHub environment server or appliance. It includes setup_base_appliance.yml playbook.</li> <li><code>setup_base_appliance</code>: This takes care of setting the minimal requierements for an appliance. It includes <code>install_base_system.yml</code> playbook. On top of it, it creates and configures the necessary users.</li> <li><code>setup_docker_based_appliance.yml</code>: Quite self explanatory ? it performs setup tasks to enable docker on a given appliance.</li> </ul> <p>It also hosts the <code>inventory</code> file describing the role of JupyterHub servers. Place your JupyterHub machine (FQDN) in a group used as PBKDIR namerole.</p> <pre><code class="language-shellsession"># # Place to your JupyterHub machine (FQDN) in a group used as PBKDIR name. # [production] 127.0.0.1 ansible_connection=localhost </code></pre> <p>The <code>conf</code> folder hosts configuration files in a Jinja format. Once expanded, the resulting files will be used by relevant workshops. I will explain in a future article all the steps and requirements to create a workshop.</p> <p>As part of the refactoring work to open source the project, we reaaranged the different scripts' locations. We have created an install folder to handle the different installation scripts either from a JupyterHub's perpective or from an appliance's standpoint, too.</p> <p>We separated the workshops' related scripts from the system ones. When one creates a workshop, one needs to provide a series of notebooks and in some cases some scripts to manage the creation and setup of a related appliance along with additional scripts to manage its lifecycle in the overall Workshops-on-Demand architecture (Create, Cleanup, Reset scripts at deployment or Cleanup times). These scripts need to be located in the <code>scripts</code> folder. On the other hand, the system scripts are located in the <code>sys</code> folder.</p> <p><img src="/img/tree-wkshop2.png" alt="" title="Tree view of the sys directory"></p> <p>This directory hosts important configuration files for both the system and JupyterHub. You can see for instance <code>fail2ban</code> configuration files. Some Jinja templates are present here, too. These templates will be expanded through the <code>deliver</code> mechanism allowing the creation of files customized with Ansible variables. All the wod related tasks are prefixed with wod for better understanding and ease of use.</p> <p>These Jinja templates can refer to some JupyterHub kernel needs like <code>wod-build-evcxr.sh.j2</code> that aims at creating a script allowing the rust kernel installation. Some other templates are related to the system and JupyterHub. <code>wod-kill-processes.pl.j2</code> has been created after discovering the harsh reality of online mining. In a ideal world, I would not have to explain further as the script would not be needed. Unfortunately, this is not the case. When one offers access to some hardware freely online, sooner or later, he can expect to see his original idea to be hyjacked.</p> <p>Let's say that you want to provide some AI/ML 101 type of workshops. As part of it, you may consider providing servers with some GPUs. Any twisted minded cryptominer discovering your resources will definitely think he's hits the jackpot! This little anecdot actually happened to us and not only on GPU based servers, some regular servers got hit as well. We found out that performance on some servers became very poor and when looking into it, we found some scripts that were not supposed to run there. As a result, we implemented monitors to check the load on our servers and made sure that to kill any suspicious processes before kicking out the misbehaving student.</p> <p><code>wod-test-action.sh.j2</code> is another interesting template that will create a script that can be used for testing workshops. This script mimics the procmail API and actually enables you to test the complete lifecycle of a workshop from deployment to cleanup or reset.</p> <pre><code class="language-shellsession">wodadmin@server:/usr/local/bin$ ./wod-test-action.sh Syntax: wod-test-action.sh &#x3C;CREATE|CLEANUP|RESET|PURGE|PDF|WORD> WKSHOP [MIN[,MAX] ACTION is mandatory wodadmin@server:/usr/local/bin$ </code></pre> <p>It requires the verb, the workshop's name and the student id. Using the script, one does not need to provide participant id. The script is run locally on the JupyterHub server.</p> <pre><code class="language-shellsession">wodadmin@server:/usr/local/bin$ ./wod-test-action.sh Syntax: wod-test-action.sh &#x3C;CREATE|CLEANUP|RESET|PURGE|PDF|WORD> WKSHOP [MIN[,MAX] ACTION is mandatory wodadmin@server:/usr/local/bin$ ./wod-test-action.sh CREATE WKSHP-API101 121 Action: CREATE We are working on WKSHP-API101 Student range: 121 Sending a mail to CREATE student 121 for workshop WKSHP-API101 220 server.xyz.com ESMTP Postfix (Ubuntu) 250 2.1.0 Ok 250 2.1.5 Ok 354 End data with &#x3C;CR>&#x3C;LF>.&#x3C;CR>&#x3C;LF> 250 2.0.0 Ok: queued as 9749E15403AB 221 2.0.0 Bye </code></pre> <p>In order to retrieve the result of the script, you simply need to run a <code>tail</code> command.</p> <pre><code class="language-shellsession">wodadmin@server:~$ tail -100f .mail/from ++ date .... From xyz@hpe.com Fri Mar 3 09:08:35 2023 Subject: CREATE 121 0 Folder: /home/wodadmin/wod-backend/scripts/procmail-action.sh CREATE 11 + source /home/wodadmin/wod-backend/scripts/wod.sh .... + echo 'end of procmail-action for student 121 (passwd werty123) with workshop WKSHP-API101 with action CREATE at Fri Mar 3 09:11:39 UTC 2023' </code></pre> <p>The very last line of the trace will provide you with the credentials necessary to test your workshop.</p> <p>There are two types of activities that can occur on the backend server: punctual or regular. The punctual activity is one that is performed once every now and then. The regular one is usually set up on the backend server as a cron job. Sometimes however, one of these cron tasks can be forced manually if necessary. One of the most important scheduled task is the <code>deliver</code> task. I will explain it later on in this chapter. I will start now by explaining an important possible punctual task, the update of the backend server.</p> <h4>Update of the backend server:</h4> <p>The backend server hosts all the necessary content for delivering workshops: it supplies notebooks,scripts and playbooks to deploy and personalize them. It also hosts some services that are needed by the overall architecture solution (JupyterHub, Procmail, Fail2ban among others).</p> <p>Services are installed once and for all at the installation time. These services may evolve over time. One may need to update the JupyterHub application to fix a bug or get new features. In the same fashion, you may consider bumping from one Python version to a new major one. If you are willing to update these services or add new ones, you will need to update the relevant installation playbooks in <code>wod-backend/ansible</code> directory.</p> <p>Here is a small extract of the <code>install_backend.yml</code> playbook: Full version <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/install_backend.yml">here</a></p> <pre><code class="language-shellsession">vi install_backend - hosts: all gather_facts: true vars: IJAVAVER: "1.3.0" KUBECTLVER: "1.21.6" tasks: - name: Include variables for the underlying distribution include_vars: "{{ ANSIBLEDIR }}/group_vars/{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml" - name: Base setup for a JupyterHub environment server or appliance include_tasks: "{{ ANSIBLEDIR }}/setup_base_appliance.yml" - name: Add CentOS SC repository into repo list become: yes become_user: root yum: name: centos-release-scl-rh state: present when: - ansible_distribution == "CentOS" - ansible_distribution_major_version >= "7" - name: Add conda GPG Key to APT become: yes become_user: root apt_key: url: https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc state: present when: - ansible_distribution == "Ubuntu" - ansible_distribution_major_version >= "20" # TODO: Do it for EPEL if really needed - name: Add conda APT repository become: yes become_user: root apt_repository: repo: deb [arch=amd64] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main state: present when: - ansible_distribution == "Ubuntu" </code></pre> <p>Possible Use Cases:</p> <ul> <li>Upgrade to a newer version of JupyterHub</li> <li>Add a new kernel to JupyterHub</li> <li>Add a new Ansible Galaxy collection</li> <li>Add a new PowerShell library</li> <li>Add a new package needed by a workshop.</li> </ul> <p>For e.g:</p> <ul> <li>Kubectl client</li> <li>Terraform client</li> <li>PowerShell module</li> <li>Python Library</li> </ul> <p>You will start by moving to your public backend forked repository and apply the necessary changes before committing and push locally.</p> <p>Then you will perform a merge request with the main repository. We plan to integrate here in a proper CICD (continuous integration continous development) pipeline to allow a vagrant based test deployment. Whenever someone performs a merge request on the main repo, the test deployment task kicks in and deploys a virtual backend server on which the new version of the installation process is automatically tested. When successful, the merge request is accepted. Once merged, you will need to move to your backend server and perform git remote update and git rebase on the wod-backend directory. Once done, you will then be able to perform the installation process.</p> <h4>Regular maintenance of the backend server:</h4> <p>On a daily basis, some tasks are launched to check the integrity of the backend server. Some tasks are related to the security integrity of the system. The following playbook is at the heart of this verification: <strong>wod-backend/ansible/check_backend.yml</strong>. Full version of the file is available <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/check_backend.yml">here</a> for review.</p> <p>It checks a quite long list of items like:</p> <ul> <li>Wod System compliancy: is this really a wod system? by calling out <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/check_system.yml">check_system.yml</a> playbook.</li> </ul> <p>This first check includes:</p> <ul> <li> <p>nproc hard and soft limits</p> </li> <li> <p>nofile hard and soft limits</p> </li> <li> <p>Setup sysctl params</p> <ul> <li>net.ipv4.tcp_keepalive_time, value: "1800"</li> <li>kernel.threads-max, value: "4096000"</li> <li>kernel.pid_max, value: "200000"</li> <li>vm.max_map_count, value: "600000"</li> <li>Setup UDP and TCP firewall rules</li> </ul> </li> <li> <p>Enable services:</p> <ul> <li>Firewalld</li> <li>Ntp</li> </ul> </li> <li> <p>Student Management:</p> <ul> <li>Ensure limits are correct for students accounts</li> <li>Copy the skeleton content under /etc/skel</li> <li>Test <code>.profile</code> file</li> <li>Ensure vim is the default EDITOR</li> <li>Setup <code>logind.conf</code></li> <li>Manage <code>/etc/hosts</code> file</li> <li>Install the pkg update script</li> <li>Setup <code>crontab</code> for daily pkg security update</li> <li>Deliver create/reset/setup scripts as ansible template for variable expansion</li> <li>Install utility scripts</li> <li>Deliver the system scripts (<code>cleanup-processes.sh.j2</code>)</li> <li>Installation of the cleanup-processes script</li> <li>Setup weekly cleanup processes task</li> <li>Enable WoD service</li> <li>Test private tasks YAML file</li> <li>Call private tasks if available. It performs the private part before users management to allow interruption of the deliver script during normal operations - waiting till end of users management can take hours for 2000 users. Potential impact: private scripts are run before users creation, so may miss some part of setup.</li> </ul> </li> <li> <p>User Management:</p> <ul> <li>Remove existing JupyterHub users</li> <li>Remove Linux users and their home directory</li> <li>Ensure dedicated students groups exist</li> <li>Ensure Linux students users exists with their home directory</li> <li>Ensure JupyterHub students users exist</li> <li>Setup ACL for students with JupyterHub account</li> <li>Setup default ACL for students with JupyterHub account</li> </ul> </li> </ul> <p>A similar set of scripts exist for the different parts of the solution (<a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/check_api-db.yml">check_api-db.yml</a> for api-db server, <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/check_frontend.yml">check_frontend.yml </a>for frontend server for instance).</p> <p>You should now have a better understanding of the maintenance tasks associated to the backend server. Similar actions are available for the other components of the project. Checking tasks have been created for the frontend and api-db server. Having now mostly covered all the subjects related to the backend server from an infrastructure standpoint, it is high time to discuss the content part. In my next blog, I plan to describe the workshop creation process. Time to understand how to build up some content for the JupyterHub server!</p> <p>If we can be of any help in clarifying any of this, please reach out to us on <a href="https://slack.hpedev.io/">Slack</a>. Please be sure to check back at <a href="https://developer.hpe.com/blog">HPE DEV</a> for a follow up on this. Also, don't forget to check out also the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a>! Willing to collaborate with us? Contact us so we can build more workshops!</p>https://developer.hpe.com/2023-July-10/https://developer.hpe.com/2023-July-10/Thu, 06 Jul 2023 13:00:00 GMThttps://developer.hpe.com/announcing-chapel-1-31/https://developer.hpe.com/announcing-chapel-1-31/Thu, 22 Jun 2023 23:45:57 GMT<p>External blog</p>https://developer.hpe.com/end-to-end-easy-to-use-pipeline-for-training-a-model-on-medmnist-v2-using-hpe-machine-learning-development-environment-flask/https://developer.hpe.com/end-to-end-easy-to-use-pipeline-for-training-a-model-on-medmnist-v2-using-hpe-machine-learning-development-environment-flask/Fri, 16 Jun 2023 16:00:00 GMT<p>In this blog post, we’ll be covering how <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> can add value to your machine learning workflow, as well as how to utilize HPE Machine Learning Development Environment and Flask together to train and serve a model on a medical domain-specific use case. An end-to-end workflow and step-by-step instructions are provided in the "Practice" section. If you want to jump right in, the <a href="https://github.com/ighodgao/determined_medmnist_e2e">repository</a> contains all code referenced in this post as well as instructions to run.</p> <h1>Introduction </h1> <h2>MedMNIST </h2> <p>Cancer is a horrible disease, and hospitals and research labs are increasingly using AI technology, such as convolutional neural networks (CNNs), to assist in image-based diagnoses. <a href="https://medmnist.com/">MedMNIST</a> is a meta-dataset which contains 12 2-D datasets and 6 3-D computer vision datasets for various biomedical image classification problems. One of these datasets is PathMNIST, a colon cancer classification dataset. PathMNIST is derived from a prior study, and contains images of colorectal cancer histology slides, some of which contain cancer-associated stroma. Researchers can train a model on this dataset (for example, a CNN model) to help them identify cancer-associated-stroma in future patients.  </p> <p><img src="/img/pathmedmnist.jpg" alt=""></p> <p>The goal of training a model on this dataset is to accurately classify images into their respective categories, for example, this image of adipose tissue: </p> <p><img src="/img/adipose.jpeg" alt=""></p> <p><code>"prediction": "adipose"</code></p> <h2>HPE Machine Learning Development Environment </h2> <p>On its own, <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> helps developers and scientists focus on innovation by removing the complexity and cost associated with at-scale machine learning model training. In this post we’re going to explore the advantages of building an end-to-end ML workflow using HPE Machine Learning Development Environment – occasionally referring to HPE’s open source platform, <a href="http://determined.ai/">Determined</a> - for model training. We also provide an example end-to-end solution using Flask for model deployment, as well as the steps taken to develop this example in the "Practice" section. </p> <h3>Why use HPE Machine Learning Development Environment? </h3> <p>At its core, Determined and HPE Machine Learning Development Environment are training platforms that reduce complexity for ML researchers and help research teams collaborate. </p> <p>Researchers currently write training scripts that include not only the core ML functionalities to train a model on a dataset, but also code to manage the underlying infrastructure such as training on multiple GPUs, running a hyperparameter search, visualizing the training progress, and saving model checkpoints. Researchers should not have to focus on infrastructure problems – taking away these software engineering and systems administration-related tasks can allow researchers to focus on what’s important: building great models.  </p> <p>Additionally, collaboration is an important part of ML development. Many research teams don’t have the appropriate resources to share experiment results or share GPU infrastructure, resulting in a lack of reproducibility and ad-hoc resource management. This frustration due to a lack of high-quality resources causes slow progress and is a common reason why ML projects fail. Even at the smallest scale – say, one data scientist working alone – using a tool like HPE Machine Learning Development Environment can drastically speed up iteration time by removing the need to write boilerplate code.</p> <p>In this blog post, you'll get to see firsthand how HPE Machine Learning Development Environment can remove infrastructure code in a real-world research script and, at the same time, provide out-of-the-box distributed training, checkpointing, hyperparameter search, and visualization functionailty, drastically accelerating research teams’ capabilities. You'll also learn about features that allow teams to collaborate effectively. </p> <p>If you are interested in more details about how this example was developed, take a look at the "Practice" section. For a full, in-depth, model porting guide, check out this <a href="https://docs.determined.ai/latest/tutorials/pytorch-porting-tutorial.html">model porting guide.</a> The code for this example and the instructions used to run it can be found in the <a href="https://github.com/ighodgao/determined_medmnist_e2e">repository</a>.</p> <table> <thead> <tr> <th><strong>Feature</strong></th> <th><strong>Without HPE Machine Learning Development Environment</strong></th> <th><strong>With HPE Machine Learning Development Environment</strong></th> </tr> </thead> <tbody> <tr> <td>Distributed Training</td> <td>Configure using open-source tools of your choice (e.g. Ray, Horovod)</td> <td>Fault tolerant distributed training automatically enabled</td> </tr> <tr> <td>Experiment Visualization</td> <td>Write custom code or configure using open-source tools of your choice, (e.g. Weights &#x26; Biases, Tensorboard)</td> <td>Training metrics (model accuracy, model loss) available natively in WebUI, including Tensorboard extension</td> </tr> <tr> <td>Checkpointing</td> <td>Write custom logic to save checkpoints during training, which may not be robust to code failures, or configure using open-source tools of your choice</td> <td>Automatic, robust checkpoint management (e.g. best checkpoint saved at end of training, automatic checkpoint deletion, save checkpoint on experiment pause)</td> </tr> <tr> <td>Hyperparameter Search</td> <td>Write custom code or configure using tools of your choice (e.g. Ray Tune, Optuna)</td> <td>State-of-the-art hyperparameter search algorithm (Adaptive ASHA) automatically available out of the box</td> </tr> </tbody> </table> <p>As you can see, without a centralized training platform to handle all necessary features in one place, users are left to write custom code or use a variety of open-source tools. This can get complicated very quickly, as it’s difficult to manage multiple dependencies, and compatibility issues start to arise between tools.</p> <p>In many cases, HPE Machine Learning Development Environment can reduce the length of a training script to nearly half its original size, due to the sheer amount of boilerplate code normally required to enable these features. Let’s take a closer look at each of these to see HPE Machine Learning Development Environment in action.</p> <p>Let's take a closer look at the core features of HPE Machine Learning Development Environment!</p> <h3>Distributed training </h3> <p>Distributed training refers to the process of distributing a model training workload across multiple devices, such as GPUs. It’s very common for machine learning workloads to run for weeks on end due to large model and dataset sizes, so distributing mode training across GPUs can drastically speed up the time it takes to develop a machine learning model, from weeks to hours.  </p> <p>However, this is difficult to set up and difficult to manage: manual interaction with GPUs through code is often necessary when setting up distributed training, and, once set up, managing distributed training is cumbersome due to issues like fault tolerance. Fault tolerance refers to the ability of a system to gracefully handle and continue a training job even if something on the infrastructure level goes wrong, such as a device failing. Setting up a fault tolerant solution manually is an enormous lift on an ML team, and not normally within the scope of a researcher’s abilities.  </p> <p>Determined not only takes away the need to automatically interface with individual GPUs, but is also fault-tolerant. Let’s take a look at how the <a href="https://github.com/MedMNIST/experiments/blob/main/MedMNIST2D/train_and_eval_pytorch.py">original training script</a> handles running the model on GPUs.</p> <p>In line 291, the <code>gpu_ids</code> are received from input arguments from the user: </p> <pre><code class="language-python"> gpu_ids = args.gpu_ids </code></pre> <p> The device is configured using only the first <code>gpu_id</code> (distributed training is not yet enabled): </p> <pre><code class="language-python"> device = torch.device('cuda:{}'.format(gpu_ids[0])) if gpu_ids else torch.device('cpu') </code></pre> <p>And the model is ported to the GPU in line 90: </p> <pre><code class="language-python"> model = model.to(device) </code></pre> <p>As well as the inputs to the model, for example, in line 190: </p> <pre><code> outputs = model(inputs.to(device)) </code></pre> <p>The variable “device” is referenced a total of 27 times in the training script, for purposes like porting other inputs to the GPU, and passing the device variable around to different functions so they are aware of the GPU. This is a perfect example of how a researcher would normally need to manage training on a GPU – manually. And we haven't even started distributed training yet!</p> <p>With Determined or HPE Machine Learning Development Environment, none of this manual device management is necessary. Simply using one of our high-level APIs gives you access to not only running on GPUs but running distributed training on multiple GPUs out-of-the-box. The only configuration needed (after porting your model to one of our APIs) would be to set the number of desired resources (GPUs) to use in your experiment settings, e.g.:</p> <pre><code class="language-python">resources: slots_per_trial: 4 </code></pre> <p>After taking these steps, you’d be able to watch your experiment progress in the WebUI and be assured that you’re utilizing multiple GPUs. By viewing the “Cluster” tab, you will notice that 4/5 CUDA Slots are allocated and in use: </p> <p><img src="/img/screenshot2.png" alt=""></p> <h3>Experiment visualization and metric logging </h3> <p>Visualization tools are important when developing models due to the probabilistic nature of machine learning. Debugging a model often involves analyzing a model’s training journey by visualizing metrics at different timestamps during an experiment. Commonly used tools for visualization often require manual configuration. Let’s take a look at how the <a href="https://github.com/MedMNIST/experiments/blob/main/MedMNIST2D/train_and_eval_pytorch.py">original training script</a> handles visualization:  </p> <p>The original script uses a library called <a href="https://tensorboardx.readthedocs.io/en/latest/tensorboard.html#module-tensorboardX">tensorboardX</a>: </p> <pre><code class="language-python">from tensorboardX import SummaryWriter </code></pre> <p>Using this library, a writer object is created for handling visualization data: </p> <pre><code class="language-python">writer = SummaryWriter(log_dir=os.path.join(output_root, 'Tensorboard_Results')) </code></pre> <p>The writer object is referenced a total of 9 times throughout the script.  </p> <p>In addition, training and testing metrics are manually calculated and logged in various places throughout the script, e.g.:  </p> <pre><code class="language-python">logs = ['loss', 'auc', 'acc'] train_logs = ['train_'+log for log in logs] val_logs = ['val_'+log for log in logs] test_logs = ['test_'+log for log in logs] log_dict = OrderedDict.fromkeys(train_logs+val_logs+test_logs, 0) </code></pre> <pre><code class="language-python"> train_log = 'train auc: %.5f acc: %.5f\n' % (train_metrics[1], train_metrics[2]) val_log = 'val auc: %.5f acc: %.5f\n' % (val_metrics[1], val_metrics[2]) test_log = 'test auc: %.5f acc: %.5f\n' % (test_metrics[1], test_metrics[2]) log = '%s\n' % (data_flag) + train_log + val_log + test_log print(log) </code></pre> <p>With Determined, no manual metric tracking or logging is necessary. When porting your model to one of our high-level APIs, the default training and testing metrics, such as model losses, are automatically configured and rendered natively in the WebUI: </p> <p><img src="/img/screenshot1.png" alt=""></p> <h3>Automatic checkpointing </h3> <p>Checkpointing a model throughout an experiment is important to maintain training progress and for preserving the best model at the end of an experiment. Let’s take a look at how the original training script handles model checkpointing.</p> <p>The original training script saves the last model at the very end of the training process, in a location specified by the user, but does not checkpoint throughout the training job: </p> <pre><code class="language-python"> path = os.path.join(output_root, 'best_model.pth') torch.save(state, path) </code></pre> <p>This approach is problematic for the following reasons: </p> <ul> <li>The last model, at the end of a training job, is not always the best model. </li> <li>The script could fail before the end of a training job, resulting in no model checkpoint, which wastes time and resources. </li> </ul> <p>Determined automatically checkpoints in the following situations: </p> <ul> <li>Periodically throughout the course of model training, to keep a record of the training progress. </li> <li>During training – If a trial fails on epoch 9 and the last checkpoint was saved during epoch 1, Determined will save yet another checkpoint at epoch 9, making this a very efficient system. </li> <li>Upon completion of the trial. </li> </ul> <p>Additional checkpoint configuration settings can be modified to make this even more customizable. Checkpoints can be easily examined through the WebUI even after an experiment completes, and downloaded easily through the Determined APIs: </p> <p><img src="/img/screenshot3.png" alt=""></p> <h3>Hyperparameter search </h3> <p>Hyperparameter search refers to the process of searching for the optimal configuration settings for your machine learning model – for example, searching for the optimal learning rate or convolution sizes in a convolutional neural network. The original training script does not implement hyperparameter search. This is not surprising, as hyperparameter search is yet another heavy lift for a researcher who wants to focus on experimentation, not the grunt work of setting up a hyperparameter search.  </p> <p>With Determined or HPE Machine Learning Development Environment, configuring hyperparameter search is easy. Defining hyperparameters, either static or in ranges, is easy through experiment configuration: </p> <pre><code class="language-yaml">hyperparameters: global_batch_size: 128 data_flag: pathmnist dataset_name: "pathmnist.npz" model_flag: resnet18 lr: 0.001 gamma: 0.1 resize: True task: "multi-class" num_epochs: 15 </code></pre> <p>In this case, the “hyperparameters” feature can also be used to switch between datasets and types of training tasks if configured appropriately in the training script – which is neat! </p> <h2>Collaboration </h2> <p>Data scientists and researchers rarely work alone, especially in today’s data-driven technological boom. Sharing experiment results across a large team and sharing a single GPU cluster is not traditionally straightforward, but HPE Machine Learning Development Environment makes for a much better experience: </p> <h3>Resource management </h3> <p>At the enterprise level, HPE Machine Learning Development Environment automatically scales up and down workloads depending on priority and resource availability. For example, if a researcher is running a hefty training job and is utilizing all 10 GPUs on a shared cluster, but their colleague needs two of them for a higher priority smaller job, HPE Machine Learning Development Environment can temporarily scale back the larger job utilizing all 10 GPUs down to using only 8, leaving room for the smaller training job.  </p> <h3>Model sharing and reproducibility </h3> <p>The HPE Machine Learning Development Environment WebUI makes it easy to track experiments and see which model configurations resulted in particular results, across a team. This makes reproducibility easy, something that is of utmost importance when developing models for a use case like predicting cancer in biomedical images.  </p> <p>Now that you have a good background on the HPE Machine Learning Development Environment, l'll show you how you can build an E2E solution for your model training.</p> <h1>Practice: Building an E2E solution </h1> <h2>Step 1: Model training </h2> <p>To get started, we’ll take a look at the original training script provided <a href="https://github.com/MedMNIST/experiments/blob/main/MedMNIST2D/train_and_eval_pytorch.py">here</a>. </p> <p>This script contains all the functionality needed to download the dataset as well as train and test the model (a commonly used convolutional neural network architecture – ResNet), on the PathMNIST data. </p> <p>The original script has some boilerplate code that can be removed since HPE Machine Learning Development Environment handles functionality such as distributed training and experiment visualization. For a full guide to model porting, refer to the <a href="https://hpe-mlde.determined.ai/latest/tutorials/pytorch-porting-tutorial.html#pytorch-porting-tutorial">model porting guide</a>. Walk through the steps taken to port the code to HPE Machine Learning Development Environment using the PyTorch API. The PyTorch API is a high-level API which allows you to utilize the full functionality of HPE Machine Learning Development Environment out-of-the-box. </p> <h2>Step 1.1: Connect to HPE Machine Learning Development Environment  </h2> <p>Once your admin has provisioned a cluster with HPE Machine Learning Development Environment and configured you as a user, connect to HPE Machine Learning Development Environment by exporting the DET_MASTER variable: </p> <pre><code class="language-yaml">export DET_MASTER=&#x3C;your cluster address> </code></pre> <p>Then log in as your user: </p> <pre><code class="language-yaml">det login user &#x3C;your username> </code></pre> <p>You can check to make sure you are logged in via </p> <pre><code class="language-yaml">det user whoami </code></pre> <p>Refer to the Determined User Guide and the Reference Page for more information on how to interact with the cluster.  </p> <p><em>In steps 1.2 - 1.4, I'm going to describe how to port this model to HPE Machine Learning Development Environment step-by-step, (model_def.py). If you are interested in running the final training job, skip to step 1.5.</em> </p> <h2>Step 1.2: Port model definition  </h2> <p>To train your own custom model using one of Determined’s high level APIs, such as the PyTorch API, you need to port your code to the API first. A template of all the functions Determined needs to run your training loops is provided <a href="https://docs.determined.ai/latest/training/apis-howto/api-pytorch-ug.html#pytorch-trial">here</a>. Fill them out, one-by-one, to port your code. Once these functions are populated, Determined can use these, along with the provided configuration file, to run your experiment. </p> <p>To start, create a class definition that inherits from PyTorchTrial (this is the template referred to above). Create a context (line 29). This is like an interface to the Determined master, allowing you to communicate back and forth with it. Also include the model, optimizer, and criterion in the initialization function so that Determined is aware of them. Make each object an attribute of the class as shown below and transfer relevant hyperparameters to <code>config.yaml</code>. As shown here, we can obtain hyperparameters defined in this configuration file by calling <code>self.context.get_hparam()</code>, making it easier to change these hyperparameters without modifying training code.  </p> <pre><code class="language-python">class MyMEDMnistTrial(PyTorchTrial): def __init__(self, context: PyTorchTrialContext) -> None: self.context = context self.info = INFO[self.context.get_hparam("data_flag")] task = self.info["task"] n_classes = len(self.info["label"]) self.context = context if self.context.get_hparam("model_flag") == "resnet18": model = resnet18(pretrained=False, num_classes=n_classes) elif self.context.get_hparam("model_flag") == "resnet50": model = resnet50(pretrained=False, num_classes=n_classes) else: raise NotImplementedError self.model = self.context.wrap_model(model) optimizer = torch.optim.Adam( self.model.parameters(), lr=self.context.get_hparam("lr") ) self.optimizer = self.context.wrap_optimizer(optimizer) if self.context.get_hparam("task") == "multi-label, binary-class": self.criterion = nn.BCEWithLogitsLoss() else: self.criterion = nn.CrossEntropyLoss() </code></pre> <h2>Step 1.3: Port data loaders  </h2> <p>Next, port the training and evaluation data loaders to the following class functions as follows. This is the code that trains one batch of data. You no longer need the standard for loop to iterate over batches or epochs inside these functions, since Determined will handle the training loop for you. </p> <pre><code class="language-python"> def build_training_data_loader(self) -> DataLoader: DataClass = getattr(medmnist, self.info["python_class"]) if self.context.get_hparam("resize"): data_transform = transforms.Compose( [ transforms.Resize((224, 224), interpolation=PIL.Image.NEAREST), transforms.ToTensor(), transforms.Normalize(mean=[0.5], std=[0.5]), ] ) else: data_transform = transforms.Compose( [transforms.ToTensor(), transforms.Normalize(mean=[0.5], std=[0.5])] ) train_dataset = DataClass( split="train", transform=data_transform, download=False, as_rgb=True, root=DATASET_ROOT, ) train_loader = determined.pytorch.DataLoader( dataset=train_dataset, batch_size=self.context.get_per_slot_batch_size(), shuffle=True, ) return train_loader </code></pre> <p>In each function, initialize and return the relevant PyTorch DataLoader object (e.g. <code>val_loader from build_validation_data_loader</code>, and <code>train_loader in build_training_data_loader</code>).  </p> <h2>Step 1.4: Port training and evaluation functions </h2> <p>Finally, port your training and evaluation functions to the following class functions by including the relevant steps to train and evaluate the model on one batch of data. Here, make sure to remove the for-loop iterating over epochs and include only the relevant code corresponding to one data batch. HPE Machine Learning Development Environment handles the training loop behind the scenes.  </p> <pre><code class="language-python"> def train_batch( self, batch: TorchData, epoch_idx: int, batch_idx: int ) -> Dict[str, Any]: inputs, targets = batch outputs = self.model(inputs) if self.context.get_hparam("task") == "multi-label, binary-class": targets = targets.to(torch.float32) loss = self.criterion(outputs, targets) else: targets = torch.squeeze(targets, 1).long() loss = self.criterion(outputs, targets) self.context.backward(loss) self.context.step_optimizer(self.optimizer) return {"loss": loss} def evaluate_batch(self, batch: TorchData) -> Dict[str, Any]: inputs, targets = batch outputs = self.model(inputs) if self.context.get_hparam("task") == "multi-label, binary-class": targets = targets.to(torch.float32) loss = self.criterion(outputs, targets) m = nn.Sigmoid() outputs = m(outputs) else: targets = torch.squeeze(targets, 1).long() loss = self.criterion(outputs, targets) m = nn.Softmax(dim=1) outputs = m(outputs) targets = targets.float().resize_(len(targets), 1) return {"test_loss": loss} </code></pre> <p>Notice that in all functions defined as part of MyMEDMnistTrial, all instances of manual device management, metric logging, and accuracy calculations have been removed. HPE Machine Learning Development Environment’s Trial APIs handle these automatically, given only the Trial definition and configuration file.  </p> <p>After porting the model, compare the original training script and our newly defined model_def.py training file. At 302 vs. 177 lines of code, we have cut the training script nearly in half! </p> <h3><em>Optional</em>: Upload dataset to S3 bucket  </h3> <p>The PathMNIST dataset includes images that are preprocessed to 28x28 images with corresponding classification labels, complete with data augmentation. The original training script downloads this dataset and performs data normalization prior to model training.  </p> <p>When submitting experiments to HPE Machine Learning Development Environment, the dataset will be downloaded to the master in the same way, which may cause a small delay. To avoid this, upload your data to an S3 bucket and access it via the following instead: </p> <p>Modify <code>config.yaml</code> to include the data url as follows: </p> <pre><code class="language-yaml">data: url: https://medmnist-pathmnist.s3.us-east-2.amazonaws.com/pathmnist.npz </code></pre> <p>Access the dataset via the following in the class initialization function: </p> <pre><code class="language-yaml">os.makedirs(DATASET_ROOT, exist_ok=TRUE) wget.download( context.get_data_config()["url"], out=os.path.join(DATASET_ROOT, "pathmnist.npz"), ) </code></pre> <h2>Step 1.5: Train the model </h2> <p>In HPE Machine Learning Development Environment, an experiment is a training job that consists of one or more variations, or trials, on the same model </p> <p>To begin training our PathMNIST model, you need to submit an experiment to HPE Machine Learning Development Environment. After setting up Determined, use the CLI to submit an experiment (refer to full setup instructions in the <a href="https://github.com/ighodgao/determined_medmnist_e2e">repository</a>): </p> <p><code>det e create config.yaml .</code> </p> <p>View progress on the WebUI located at &#x3C;DET_MASTER>:8080, which you can reach by pasting this address into a browser window. </p> <p>You can configure additional training parameters by modifying config.yaml. Refer to the <a href="https://docs.determined.ai/latest/reference/reference-training/experiment-config-reference.html#experiment-configuration-reference">Experiment Configuration Reference</a> for more information. </p> <p>And that’s it! Now that you have a trained PathMNIST model, progress to deployment, found in Step 2.  </p> <h2>Step 2: Model deployment </h2> <p>Once you have finished training with HPE Machine Learning Development Environment, you can deploy your model using any solution.  </p> <p>In Step 2, you'll see an example of using Flask to deploy your model as a RESTful API. </p> <p><em>Steps 2.1-2.4 describe the steps needed to create a Flask server (deploy.py). If you are interested in deploying the model directly, skip to steps 2.5-2.6.</em> </p> <h2>Step 2.1: Load the model from your saved checkpoints in your experiment </h2> <pre><code class="language-python"># Load the Determined model checkpoint = client.get_experiment(os.getenv("EXPERIMENT_ID")).top_checkpoint() path = checkpoint.download() trial = pytorch.load_trial_from_checkpoint_path(path) model = trial.model model.eval() </code></pre> <p>Now that you have a trained model, to properly load the model, reference checkpoints from the experiment.</p> <h2>Step 2.2: Define a function to preprocess your data </h2> <pre><code class="language-python">def preprocess_data(image): # Resize the image image = image.resize((28, 28)) # Convert the image to a NumPy array image = np.array(image) # Add a channel dimension if it's a grayscale image if len(image.shape) == 2: image = np.expand_dims(image, axis=-1) # Normalize the image by dividing by 255 and subtracting 0.5 image = (image / 255.0) - 0.5 # Transpose the image dimensions image = np.transpose(image, (2, 0, 1)) # Convert the image to a torch tensor processed_data = torch.tensor(image, dtype=torch.float32) # Add a batch dimension processed_data = processed_data.unsqueeze(0) return processed_data </code></pre> <p>Here, perform the same normalizations done when training the model on the training dataset.Step 2.3: Define a Flask server.</p> <pre><code class="language-python"># Define a Flask route for serving predictions @app.route("/predict", methods=["POST"]) def predict(): # Get the input image file from the request file = request.files["file"] # Read the image file into a PIL Image object image = Image.open(io.BytesIO(file.read())) # Preprocess the input data processed_data = preprocess_data(image) # Use the Determined model to make a prediction output_tensor = model(processed_data) # Convert the output tensor to a numpy array output_array = output_tensor.detach().numpy() # Apply softmax to get probabilities for each class probabilities = F.softmax(output_tensor, dim=1) # Convert the output tensor to a numpy array output_array = probabilities.detach().numpy() # Get the predicted class label class_label = np.argmax(output_array) class_labels = [ "adipose", "background", "debris", "lymphocytes", "mucus", "smooth muscle", "normal colon mucosa", "cancer-associated stroma", "colorectal adenocarcinoma epithelium", ] # Return the predicted class label as a JSON response return jsonify({"prediction": class_labels[class_label]}) </code></pre> <p>Here, you will define a server that accepts an image file, performs the preprocessing steps defined in the <code>preprocess_data</code> function, and outputs the prediction as a json object.  </p> <h2>Step 2.4: Define a function to start the Flask server: </h2> <pre><code class="language-python"># Start the Flask server if __name__ == "__main__": app.run(debug=True, host="0.0.0.0", port=5000) </code></pre> <p>In this example the server is deployed on localhost on port 5000. </p> <h2>Step 2.5: Obtain test data</h2> <p>Your test data should be in the form of .jpeg, .png, or .jpg images. Deploy the Flask server and submit requests for inference.</p> <p>Full instructions for running the example can be found in the <a href="https://github.com/ighodgao/determined_medmnist_e2e">repository</a>. </p> <p>For example, running prediction on the following image of adipose tissue results in: </p> <p><img src="/img/adipose.jpeg" alt=""></p> <p><code>"prediction": "adipose"</code></p> <p>Deploying your model can make it a lot easier to use in production, and while this example uses Flask as a simple proof-of-concept, tools like KServe can help deploy your models in production. </p> <h1>Conclusion  </h1> <p>That’s it! I've have shown you how to implement an end-to-end machine learning workflow using HPE Machine Learning Development Environment and Flask. For more information about HPE’s AI toolkit please visit the <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence.html">HPE AI Solutions homepage here.</a> </p> <p>If you enjoyed reviewing this model training and deployment example, <a href="https://join.slack.com/t/determined-community/shared_invite/zt-1txc10qgy-yJ2puE6DxgrhdH9TIK93tw">we invite you to get in touch with the Determined Community</a> and try this out for yourself by following the Determined <a href="https://docs.determined.ai/latest/">Documentation</a>.  </p> <p>If you and your team are ready for premium machine learning support from HPE, please please contact the team via the <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment homepage</a>.</p>https://developer.hpe.com/production-ready-object-detection-model-training-workflow-with-hpe-machine-learning-development-environment/https://developer.hpe.com/production-ready-object-detection-model-training-workflow-with-hpe-machine-learning-development-environment/Fri, 16 Jun 2023 16:00:00 GMT<p>This in-depth blog tutorial is divided into five separate sections, where I will recount the seamless user experience one has when working with <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a>, pointing out how easy it is to achieve machine learning at scale with HPE.  </p> <p>Throughout the different parts of this tutorial, I will review the end-to-end training of an object detection model using NVIDIA’s PyTorch Container from <a href="https://www.nvidia.com/en-us/gpu-cloud/">NVIDIA's NGC Catalog</a>, a Jupyter Notebook, the open-source training platform from <a href="http://www.determined.ai/">Determined AI</a>, and <a href="https://www.kubeflow.org/docs/external-add-ons/kserve/kserve/">Kserve</a> to deploy the model into production.  </p> <h1>Part 1: End-to-end example training object detection model using NVIDIA PyTorch Container from NGC</h1> <h2><a href="https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#installation"></a>Installation</h2> <hr> <p><em><strong>Note: this object detection demo is based on the <a href="https://github.com/pytorch/vision/tree/v0.11.3">TorchVision package GitHub repository</a>.</strong></em></p> <p>This notebook walks you each step to train a model using containers from the NGC Catalog. We chose the GPU optimized PyTorch container as an example. The basics of working with docker containers apply to all NGC containers.</p> <h2>NGC</h2> <p>The <a href="https://www.nvidia.com/en-us/gpu-cloud/">NGC catalog from NVIDIA</a> offers ready-to-use containers, pre-trained models, SDKs, and Helm charts for diverse use cases and industries to speed up model training, development, and deployment. For this example, I'm pulling the popular PyTorch container from NGC and show you how to:</p> <ul> <li>Install the Docker engine on your system</li> <li>Pull a PyTorch container from the NGC catalog using Docker</li> <li>Run the PyTorch container using Docker</li> </ul> <p>Let's get started!</p> <hr> <h3><a href="https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#1-install-the-docker-engine"></a>1. Install the Docker Engine</h3> <p>Go to the <a href="https://docs.docker.com/engine/install/">Docker Installation Engine documentation</a> to install the Docker Engine on your system.</p> <h3><a href="https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#2-download-the-tensorflow-container-from-the-ngc-catalog"></a>2. Download the TensorFlow container from the NGC Catalog</h3> <p>Once the Docker Engine is installed on your machine, visit the <a href="https://catalog.ngc.nvidia.com/containers">NVIDIA NGC Container Catalog</a> and search for<a href="https://ngc.nvidia.com/catalog/containers"></a>and search for the TensorFlow container. Click on the TensorFlow card and copy the pull command. <a href="https://raw.githubusercontent.com/kbojo/images/master/NGC.png"><img src="https://raw.githubusercontent.com/kbojo/images/master/NGC.png" alt=""></a></p> <p>Open the command line of your machine and paste the pull command into your command line. Execute the command to download the container.</p> <p><code>$ docker pull nvcr.io/nvidia/pytorch:21.11-py3</code></p> <p>The container starts downloading to your computer. A container image consists of many layers; all of them need to be pulled.</p> <h3><a href="https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#3-run-the-tensorflow-container-image"></a>3. Run the TensorFlow container image</h3> <p>Once the container download is completed, run the following code in your command line to run and start the container:</p> <p><code>$ docker run -it --gpus all -p 8888:8888 -v $PWD:/projects --network=host nvcr.io/nvidia/pytorch:21.11-py3</code> </p> <p><a href="https://raw.githubusercontent.com/kbojo/images/master/commandline1.png"><img src="https://raw.githubusercontent.com/kbojo/images/master/commandline1.png" alt=""></a></p> <h3><a href="https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#4-install-jupyter-lab-and-open-a-notebook"></a>4. Install Jupyter lab and open a notebook</h3> <p>Within the container, run the following commands:</p> <p><code>pip install torchvision==0.11.3 jupyterlab</code></p> <p><code>jupyter lab --ip=0.0.0.0 --port=8888 --allow-root</code></p> <p>Open up your favorite browser and enter: <a href="http://localhost:8888/?token=*yourtoken">http://localhost:8888/?token=*yourtoken</a>*. <a href="https://raw.githubusercontent.com/kbojo/images/master/commandline2.png"><img src="https://raw.githubusercontent.com/kbojo/images/master/commandline2.png" alt=""></a></p> <p>You should see the Jupyter Lab application. Click on the plus icon to launch a new Python 3 notebook. Follow the instructions regarding the image classification with the TensorFlow example provided in Part 2.</p> <p>Now that you have your Docker engine installed and the PyTorch Container running, we need to fetch and prepare our training dataset. You'll see that coming up in Part 2 below.</p> <hr> <h1>Part 2: Data preparation</h1> <p><em>Note this Demo is based on NGC Docker image</em> <code>nvcr.io/nvidia/pytorch:21.11-py3</code></p> <p>This notebook walks you through each step required to train a model using containers from the NGC catalog. We chose the GPU optimized PyTorch container as an example. The basics of working with docker containers apply to all NGC containers.</p> <p>Here, I will show you how to:</p> <ul> <li>Download the xView Dataset</li> <li>How to convert labels to coco format</li> <li>How to conduct the preprocessing step, <strong>Tiling</strong>: slicing large satellite imagery into chunks</li> <li>How to upload to S3 bucket to support distributed training</li> </ul> <p>Let's get started!</p> <hr> <h2>Pre-reqs, set up Jupyter Notebook environment using NGC container</h2> <h3>Execute Docker run to create NGC environment for data preparation</h3> <p>Make sure to map host directory to Docker directory. You will use the host directory again to do the following:</p> <ul> <li><code>docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/ubuntu:/home/ubuntu -p 8008:8888 -it nvcr.io/nvidia/pytorch:21.11-py3 /bin/bash</code></li> </ul> <h3>Run Jupyter Notebook command within Docker container to access it on your local browser</h3> <ul> <li><code>cd /home/ubuntu</code></li> <li><code>jupyter lab --ip=0.0.0.0 --port=8888 --NotebookApp.token='' --NotebookApp.password=''</code></li> <li><code>git clone https://github.com/interactivetech/e2e_blogposts.git</code></li> </ul> <h3>Download the xView dataset</h3> <p>The dataset you will be using is from the <a href="https://challenge.xviewdataset.org">DIUx xView 2018 Challenge</a> by U.S. National Geospatial-Intelligence Agency (NGA). You will need to <a href="https://challenge.xviewdataset.org/welcome">create an account</a>, agree to the terms and conditions, and download the dataset manually.</p> <p>You can also <a href="https://challenge.xviewdataset.org/data-download">download the dataset</a>.</p> <pre><code class="language-python"># run pip install to get the SAHI library !pip install sahi scikit-image opencv-python-headless==4.5.5.64 </code></pre> <pre><code class="language-python"># Example command to download train images with wget command, you will need to update the url as the token is expired" !wget -O train_images.tgz "https://d307kc0mrhucc3.cloudfront.net/train_images.tgz?Expires=1680923794&#x26;Signature=pn0R9k3BpSukGEdjcNx7Kvs363HWkngK8sQLHxkDOqqkDAHSOCDBmAMAsBhYZ820uMpyu4Ynp1UAV60OmUURyvGorfIRaVF~jJO8-oqRVLeO1f24OGCQg7HratHNUsaf6owCb8XXy~3zaW15FcuORuPV-2Hr6Jxekwcdw9D~g4M2dLufA~qBfTLh3uNjWK5UCAMvyPz2SRLtvc3JLzGYq1eXiKh1dI9W0DyWXov3mVDpBdwS84Q21S2lVi24KJsiZOSJqozuvahydW2AuR~tbXTRbYtmAyPF9ZqT8ZCd9MLeKw2qQJjb7tvzaSZ0F9zPjm2RS8961bo6QoBVeo6kzA__&#x26;Key-Pair-Id=APKAIKGDJB5C3XUL2DXQ" </code></pre> <pre><code class="language-python"># Example command to download train images with wget command, you will need to update the url as the token is expired" !wget -O train_labels.tgz "https://d307kc0mrhucc3.cloudfront.net/train_labels.tgz?Expires=1680923794&#x26;Signature=YEX~4gioZ7J0pAjEPx7BjJfnOa2j412mx2HlStlqa0cHj-T0T21vo17S8Fs71DXgPlZ5qnIre2-icc7wQ~EuQV-HL1ViS8qH1Aubgj9i0pnHZL07ktiyulX7QStOLywxJ7bOOmQ37iFF~-OcJW3MZfQCTWrP~LdlZMmXz0yGs5WEIYeMyvfUfIhGvrpHcJ14Z3czasSMeOKfwdQsUJoRcFTbmlbZk98IVeEWjmnGTfxGbPBdMmQ96XdT4NohggtzGdqeZhGNfwm7dKGSUbXvGCoFe~fIjBz0~5BvB6rNIaMaFuBA6aGTbCLeG8FlvijcECouhZdMTHmQUlgtSlZjGw__&#x26;Key-Pair-Id=APKAIKGDJB5C3XUL2DXQ" </code></pre> <pre><code class="language-python"># unzip images and labels from /home/ubuntu/e2e_blogposts/ngc_blog !tar -xf train_images.tgz -C xview_dataset/ </code></pre> <pre><code class="language-python"># unzip labels from /home/ubuntu/e2e_blogposts/ngc_blog directory !tar -xf train_labels.tgz -C xview_dataset/ </code></pre> <h2>1. Convert TIF to RGB</h2> <pre><code class="language-python"># Here loop through all the images and convert them to RGB, this is important for tiling the images and training with pytorch # will take about an hour to complete !python data_utils/tif_2_rgb.py --input_dir xview_dataset/train_images \ --out_dir xview_dataset/train_images_rgb/ </code></pre> <h2>2. How to convert labels to COCO format</h2> <p>Run a script to convert the dataset labels from .geojson format to COCO format. <a href="https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch">Read more details about the COCO format at this link.</a></p> <p>The result will be two files (in COCO formal) generated <code>train.json</code> and <code>val.json</code></p> <pre><code class="language-python"># make sure train_images_dir is pointing to the .tif images !python data_utils/convert_geojson_to_coco.py --train_images_dir xview_dataset/train_images/ \ --train_images_dir_rgb xview_dataset/train_images_rgb/ \ --train_geojson_path xview_dataset/xView_train.geojson \ --output_dir xview_dataset/ \ --train_split_rate 0.75 \ --category_id_remapping data_utils/category_id_mapping.json \ --xview_class_labels data_utils/xview_class_labels.txt </code></pre> <h2>3. Slicing/Tiling the dataset</h2> <p>Here, you will be using the SAHI library to slice our large satellite images. Satellite images can be up to 50k^2 pixels in size, which wouldn't fit in GPU memory. You can alleviate this problem by slicing the image.</p> <pre><code class="language-python">!python data_utils/slice_coco.py --image_dir xview_dataset/train_images_rgb/ \ --train_dataset_json_path xview_dataset/train.json \ --val_dataset_json_path xview_dataset/val.json \ --slice_size 300 \ --overlap_ratio 0.2 \ --ignore_negative_samples True \ --min_area_ratio 0.1 \ --output_train_dir xview_dataset/train_images_rgb_no_neg/ \ --output_val_dir xview_dataset/val_images_rgb_no_neg/ </code></pre> <h2>4. Upload to S3 bucket to support distributed training</h2> <p>Now, you can upload your exported data to a publicly accessible AWS S3 bucket. For a large-scale distributed experiment, this will enable you to access the dataset without installing the dataset on the device. View <a href="https://docs.determined.ai/latest/model-dev-guide/load-model-data.html">Determined Documentation</a> and <a href="https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/">AWS instructions</a> to learn how to upload your dataset to an S3 bucket. Review the <code>S3Backend</code> class in <code>data.py</code></p> <p>Once you create an S3 bucket that is publicly accessible, here are example commands to upload the preprocessed dataset to S3:</p> <ul> <li><code>aws s3 cp --recursive xview_dataset/train_sliced_no_neg/ s3://determined-ai-xview-coco-dataset/train_sliced_no_neg</code></li> <li><code>aws s3 cp --recursive xview_dataset/val_sliced_no_neg/ s3://determined-ai-xview-coco-dataset/val_sliced_no_neg</code></li> </ul> <p>Now that the satellite imagery data is in an S3 bucket and is prepped for distributed training, you can progress to model training and inferencing via the NGC container.</p> <h1>Part 3: End-to-End example training object detection model using NVIDIA PyTorch container from NGC</h1> <h2>Training and inference via NGC Container</h2> <p>This notebook walks you through each step to train a model using containers from the NGC Catalog. I chose the GPU-optimized PyTorch container for this example. The basics of working with Docker containers apply to all NGC containers.</p> <p>We will show you how to:</p> <ul> <li>Execute training an object detection model on satellite imagery using TensorFlow and Jupyter Notebook</li> <li>Run inference on a trained object detection model using the SAHI library</li> </ul> <p>Note this object detection demo is based on <a href="https://github.com/pytorch/vision/tree/v0.11.3">this PyTorch repo</a> and ngc docker image <code>nvcr.io/nvidia/pytorch:21.11-py3</code></p> <p>It is assumed that, by now, you have completed step 2 of dataset preprocessing and have your tiled satellite imagery dataset completed and in the local directory <code>train_images_rgb_no_neg/train_images_300_02</code></p> <p>Let's get started!</p> <h2>Execute Docker run to create NGC environment for data prep</h2> <p>Make sure to map host directory to Docker directory. You will use the host directory again to:</p> <ul> <li><code>docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/ubuntu:/home/ubuntu -p 8008:8888 -it nvcr.io/nvidia/pytorch:21.11-py3 /bin/bash</code></li> </ul> <h2>Run Jupyter Notebook command within Docker container to access it on your local browser</h2> <ul> <li><code>cd /home/ubuntu</code></li> <li><code>jupyter lab --ip=0.0.0.0 --port=8888 --NotebookApp.token='' --NotebookApp.password=''</code></li> </ul> <pre><code class="language-python">%%capture !pip install cython pycocotools matplotlib terminaltables </code></pre> <h2>TLDR; Run training job on 4 GPUs</h2> <p>The below cell will run a multi-gpu training job. This job will train an object detection model (faster-rcnn) on a dataset of satellite imagery images that contain 61 classes of objects.</p> <ul> <li>Change <code>nproc_per_node</code> argument to specify the number of GPUs available on your server</li> </ul> <pre><code class="language-python">!torchrun --nproc_per_node=4 detection/train.py\ --dataset coco --data-path=xview_dataset/ --model fasterrcnn_resnet50_fpn --epochs 26\ --lr-steps 16 22 --aspect-ratio-group-factor 3 </code></pre> <h3>1. Object detection on satellite imagery with PyTorch (single GPU)</h3> <p>Follow and run the code to train a Faster RCNN FPN (Resnet50 backbone) that classifies images of clothing.</p> <pre><code class="language-python">import sys sys.path.insert(0,'detection') </code></pre> <pre><code class="language-python"># Import python dependencies import datetime import os import time import torch import torch.utils.data import torchvision import torchvision.models.detection import torchvision.models.detection.mask_rcnn from coco_utils import get_coco, get_coco_kp from group_by_aspect_ratio import GroupedBatchSampler, create_aspect_ratio_groups from engine import train_one_epoch, evaluate import presets import utils from coco_utils import get_coco, get_coco_kp from train import get_dataset, get_transform from group_by_aspect_ratio import GroupedBatchSampler, create_aspect_ratio_groups from engine import train_one_epoch, evaluate from models import build_frcnn_model from PIL import Image from torchvision.models.detection.faster_rcnn import FastRCNNPredictor from collections import OrderedDict from tqdm import tqdm from vis_utils import load_determined_state_dict, visualize_pred, visualize_gt, predict import numpy as np import matplotlib.pyplot as plt </code></pre> <pre><code class="language-python">output_dir='output' data_path='xview_dataset/' dataset_name='coco' model_name='fasterrcnn_resnet50_fpn' device='cpu' batch_size=8 epochs=26 workers=4 lr=0.02 momentum=0.9 weight_decay=1e-4 lr_scheduler='multisteplr' lr_step_size=8 lr_steps=[16, 22] lr_gamma=0.1 print_freq=20 resume=False start_epoch=0 aspect_ratio_group_factor=3 rpn_score_thresh=None trainable_backbone_layers=None data_augmentation='hflip' pretrained=True test_only=False sync_bn=False </code></pre> <pre><code class="language-python"># Import the dataset. # Data loading code print("Loading data") dataset, num_classes = get_dataset(dataset_name, "train", get_transform(True, data_augmentation), data_path) dataset_test, _ = get_dataset(dataset_name, "val", get_transform(False, data_augmentation), data_path) print(dataset.num_classes) print("Creating data loaders") train_sampler = torch.utils.data.RandomSampler(dataset) test_sampler = torch.utils.data.SequentialSampler(dataset_test) group_ids = create_aspect_ratio_groups(dataset, k=aspect_ratio_group_factor) train_batch_sampler = GroupedBatchSampler(train_sampler, group_ids, batch_size) train_batch_sampler = torch.utils.data.BatchSampler( train_sampler, batch_size, drop_last=True) data_loader = torch.utils.data.DataLoader( dataset, batch_sampler=train_batch_sampler, num_workers=workers, collate_fn=utils.collate_fn) data_loader_test = torch.utils.data.DataLoader( dataset_test, batch_size=1, sampler=test_sampler, num_workers=0, collate_fn=utils.collate_fn) </code></pre> <pre><code class="language-python"># Getting three examples from the test dataset inds_that_have_boxes = [] test_images = list(data_loader_test) for ind,(im,targets) in tqdm(enumerate(test_images),total=len(list(data_loader_test))): # print(ind,targets) if targets[0]['boxes'].shape[0]>0: # print(targets[0]['boxes'].shape[0]) # print(ind,targets) inds_that_have_boxes.append(ind) images_t_list=[] targets_t_list=[] for ind in tqdm(range(3)): im,targets = test_images[ind] images_t_list.append(im[0]) targets_t_list.append(targets[0]) </code></pre> <pre><code class="language-python"># Let's have a look at one of the images. The following code visualizes the images using the matplotlib library. im = Image.fromarray((255.*images_t_list[0].cpu().permute((1,2,0)).numpy()).astype(np.uint8)) plt.imshow(im) plt.show() </code></pre> <pre><code class="language-python"># Let's look again at the first three images, but this time with the class names. for i,t in zip(images_t_list,targets_t_list): im = Image.fromarray((255.*i.cpu().permute((1,2,0)).numpy()).astype(np.uint8)) plt.imshow(im) plt.show() im = visualize_gt(i,t) plt.imshow(im) plt.show() </code></pre> <pre><code class="language-python"># Let's build the model: print("Creating model") print("Number of classes: ",dataset.num_classes) model = build_frcnn_model(num_classes=dataset.num_classes) _=model.to('cpu') </code></pre> <pre><code class="language-python"># Compile the model: # Define loss function, optimizer, and metrics. params = [p for p in model.parameters() if p.requires_grad] optimizer = torch.optim.SGD( params, lr=lr, momentum=momentum, weight_decay=weight_decay) lr_scheduler = lr_scheduler.lower() if lr_scheduler == 'multisteplr': lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=lr_steps, gamma=lr_gamma) elif lr_scheduler == 'cosineannealinglr': lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=epochs) else: raise RuntimeError("Invalid lr scheduler '{}'. Only MultiStepLR and CosineAnnealingLR " "are supported.".format(args.lr_scheduler)) </code></pre> <pre><code class="language-python"># Train the model: # Let's train 1 epoch. After every epoch, training time, loss, and accuracy will be displayed. print("Start training") start_time = time.time() for epoch in range(start_epoch, epochs): train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq) lr_scheduler.step() if output_dir: checkpoint = { 'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'lr_scheduler': lr_scheduler.state_dict(), 'args': args, 'epoch': epoch } utils.save_on_master( checkpoint, os.path.join(output_dir, 'model_{}.pth'.format(epoch))) utils.save_on_master( checkpoint, os.path.join(output_dir, 'checkpoint.pth')) # evaluate after every epoch evaluate(model, data_loader_test, device=device) total_time = time.time() - start_time total_time_str = str(datetime.timedelta(seconds=int(total_time))) print('Training time {}'.format(total_time_str)) </code></pre> <pre><code class="language-python">def build_frcnn_model2(num_classes): print("Loading pretrained model...") # load an detection model pre-trained on COCO model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) # get the number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) model.min_size=800 model.max_size=1333 # RPN parameters model.rpn_pre_nms_top_n_train=2000 model.rpn_pre_nms_top_n_test=1000 model.rpn_post_nms_top_n_train=2000 model.rpn_post_nms_top_n_test=1000 model.rpn_nms_thresh=1.0 model.rpn_fg_iou_thresh=0.7 model.rpn_bg_iou_thresh=0.3 model.rpn_batch_size_per_image=256 model.rpn_positive_fraction=0.5 model.rpn_score_thresh=0.05 # Box parameters model.box_score_thresh=0.0 model.box_nms_thresh=1.0 model.box_detections_per_img=500 model.box_fg_iou_thresh=1.0 model.box_bg_iou_thresh=1.0 model.box_batch_size_per_image=512 model.box_positive_fraction=0.25 return model </code></pre> <pre><code class="language-python"># Let's see how the model performs on the test data: model = build_frcnn_model2(num_classes=61) ckpt = torch.load('model_8.pth',map_location='cpu') model.load_state_dict(ckpt['model']) _=model.eval() </code></pre> <pre><code class="language-python">_=predict(model,images_t_list,targets_t_list) </code></pre> <p>In the next part of this blog post, I will show you how to scale your model training using using distributed training within HPE Machine Learning Development Environment &#x26; System.</p> <h1>Part 4: Training on HPE Machine Learning Development &#x26; System</h1> <p><a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> is a training platform software that reduces complexity for ML researchers and helps research teams collaborate. HPE combines this incredibly powerful training platform with best-of-breed hardware and interconnect in <a href="https://www.hpe.com/us/en/hpe-machine-learning-development-system.html">HPE Machine Learning Development System</a>, an AI turnkey solution that will be used for the duration of the tutorial.</p> <p>This notebook walks you through the commands to run the same training you did in stepin Step 3, but using the HPE Machine Learning Development Environment together with the PyTorchTrial API. All the code is configured to run out of the box. The main change is defining a <code>class ObjectDetectionTrial(PyTorchTrial)</code> to incorporate the model, optimizer, dataset, and other training loop essentials. You can view implementation details by looking at <code>determined_files/model_def.py</code></p> <p>Here, I will show you how to:</p> <ul> <li>Run a distributed training experiment</li> <li>Run a distributed hyperparameter search</li> </ul> <p><em>Note: This notebook was tested on a deployed HPE Machine Learning Development System cluster, running HPE Machine Learning Development Environment (0.21.2-dev0).</em></p> <p>Let's get started!</p> <hr> <h2>Pre-req: Run startup-hook.sh</h2> <p>This script will install some python dependencies, and install dataset labels needed when loading the xView dataset:</p> <pre><code class="language-bash">## Temporary disable for Grenoble Demo wget "https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/train_sliced_no_neg/train_300_02.json" mkdir /tmp/train_sliced_no_neg/ mv train_300_02.json /tmp/train_sliced_no_neg/train_300_02.json wget "https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/val_sliced_no_neg/val_300_02.json" mkdir /tmp/val_sliced_no_neg mv val_300_02.json /tmp/val_sliced_no_neg/val_300_02.json </code></pre> <p><em>Note that completing this tutorial requires you to upload your dataset from Step 2 into a publicly accessible S3 bucket. This will enable for a large scale distributed experiment to have access to the dataset without installing the dataset on device. View <a href="https://docs.determined.ai/latest/model-dev-guide/load-model-data.html">Determined Documentation</a> and <a href="https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/">AWS instructions</a> to learn how to upload your dataset to an S3 bucket. Review the</em> <code>S3Backend</code> class in <code>data.py</code></p> <p>When you define your S3 bucket and uploaded your dataset, make sure to change the <code>TARIN_DATA_DIR</code> in <code>build_training_data_loader</code> with the defined path in the S3 bucket.</p> <pre><code class="language-python">def build_training_data_loader(self) -> DataLoader: # CHANGE TRAIN_DATA_DIR with different path on S3 bucket TRAIN_DATA_DIR='determined-ai-xview-coco-dataset/train_sliced_no_neg/train_images_300_02/' dataset, num_classes = build_xview_dataset(image_set='train',args=AttrDict({ 'data_dir':TRAIN_DATA_DIR, 'backend':'aws', 'masks': None, })) print("--num_classes: ",num_classes) train_sampler = torch.utils.data.RandomSampler(dataset) data_loader = DataLoader( dataset, batch_sampler=None, shuffle=True, num_workers=self.hparams.num_workers, collate_fn=unwrap_collate_fn) print("NUMBER OF BATCHES IN COCO: ",len(data_loader))# 59143, 7392 for mini coco </code></pre> <pre><code class="language-python">!bash demos/xview-torchvision-coco/startup-hook.sh </code></pre> <h2>Define environment variable DET_MASTER and login in terminal</h2> <p>Run the below commands in a terminal, and complete logging into the Determined cluster by changing <username> to your username.</p> <ul> <li><code>export DET_MASTER=10.182.1.43</code></li> <li><code>det user login &#x3C;username></code></li> </ul> <h2>Define Determined experiment</h2> <p>In <a href="https://www.determined.ai/">Determined</a>, a <em>trial</em> is a training task that consists of a dataset, a deep learning model, and values for all of the model’s hyperparameters. An <em>experiment</em> is a collection of one or more trials: an experiment can either train a single model (with a single trial), or can train multiple models via a hyperparameter sweep a user-defined hyperparameter space.</p> <p>Here is what a configuration file looks like for a distributed training experiment.</p> <p>Below is what the <code>determined_files/const-distributed.yaml</code> contents look like. <code>slots_per_trial: 8</code> defines that we will use 8 GPUs for this experiment.</p> <p>Edit the change the workspace and project settings in the <code>determined_files/const-distributed.yaml</code> file</p> <pre><code class="language-yaml">name: resnet_fpn_frcnn_xview_dist_warmup workspace: &#x3C;WORKSPACE_NAME> project: &#x3C;PROJECT_NAME> profiling: enabled: true begin_on_batch: 0 end_after_batch: null hyperparameters: lr: 0.01 momentum: 0.9 global_batch_size: 128 # global_batch_size: 16 weight_decay: 1.0e-4 gamma: 0.1 warmup: linear warmup_iters: 200 warmup_ratio: 0.001 step1: 18032 # 14 epochs: 14*1288 == 18,032 step2: 19320 # 15 epochs: 15*1288 == 19,320 model: fasterrcnn_resnet50_fpn # Dataset dataset_file: coco backend: aws # specifiy the backend you want to use. one of: gcs, aws, fake, local data_dir: determined-ai-coco-dataset # bucket name if using gcs or aws, otherwise directory to dataset masks: false num_workers: 4 device: cuda environment: image: determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-mpi-0.19.10 environment_variables: - NCCL_DEBUG=INFO # You may need to modify this to match your network configuration. - NCCL_SOCKET_IFNAME=ens,eth,ib bind_mounts: - host_path: /tmp container_path: /data read_only: false scheduling_unit: 400 min_validation_period: batches: 1288 # For training searcher: name: single metric: mAP smaller_is_better: true max_length: batches: 38640 # 30*1288 == 6440# Real Training records_per_epoch: 1288 resources: slots_per_trial: 8 shm_size: 2000000000 max_restarts: 0 entrypoint: python3 -m determined.launch.torch_distributed --trial model_def:ObjectDetectionTrial </code></pre> <pre><code class="language-python"># Run the below cell to kick off an experiment !det e create determined_files/const-distributed.yaml determined_files/ </code></pre> <pre><code>Preparing files to send to master... 237.5KB and 36 files Created experiment 77 </code></pre> <h2>Launching a distributed hyperparameter search experiment</h2> <p>To implement an automatic hyperparameter tuning experiment, define the hyperparameter space, e.g. by listing the decisions that may impact model performance. You can specify a range of possible values in the experiment configuration for each hyperparameter in the search space.</p> <p>View the <code>x.yaml</code> file that defines a hyperparameter search where the model architecture that achieves the best performance on the dataset is found..</p> <pre><code class="language-yaml">name: xview_frxnn_search workspace: Andrew project: Xview FasterRCNN profiling: enabled: true begin_on_batch: 0 end_after_batch: null hyperparameters: lr: 0.01 momentum: 0.9 global_batch_size: 128 weight_decay: 1.0e-4 gamma: 0.1 warmup: linear warmup_iters: 200 warmup_ratio: 0.001 step1: 18032 # 14 epochs: 14*1288 == 18,032 step2: 19320 # 15 epochs: 15*1288 == 19,320 model: type: categorical vals: ['fasterrcnn_resnet50_fpn','fcos_resnet50_fpn', 'ssd300_vgg16','ssdlite320_mobilenet_v3_large','resnet152_fasterrcnn_model','efficientnet_b4_fasterrcnn_model','convnext_large_fasterrcnn_model','convnext_small_fasterrcnn_model'] # Dataset dataset_file: coco backend: aws # specifiy the backend you want to use. one of: gcs, aws, fake, local data_dir: determined-ai-coco-dataset # bucket name if using gcs or aws, otherwise directory to dataset masks: false num_workers: 4 device: cuda environment: environment_variables: - NCCL_DEBUG=INFO # You may need to modify this to match your network configuration. - NCCL_SOCKET_IFNAME=ens,eth,ib bind_mounts: - host_path: /tmp container_path: /data read_only: false scheduling_unit: 400 # scheduling_unit: 40 min_validation_period: batches: 1288 # For Real training searcher: name: grid metric: mAP smaller_is_better: false max_length: batches: 51520 # 50*1288 == 51520# Real Training records_per_epoch: 1288 resources: # slots_per_trial: 16 slots_per_trial: 8 shm_size: 2000000000 max_restarts: 0 entrypoint: python3 -m determined.launch.torch_distributed --trial model_def:ObjectDetectionTrial </code></pre> <pre><code class="language-python"># Run the below cell to run a hyperparameter search experiment !det e create determined_files/const-distributed-search.yaml determined_files/ </code></pre> <pre><code>Preparing files to send to master... 312.2KB and 40 files Created experiment 79 </code></pre> <h2>Load checkpoint of trained experiment</h2> <p>Replace the <code>&#x3C;EXP_ID></code> and run the below cells with the experiment ID once the experiment is completed.</p> <pre><code class="language-python">from determined_files.utils.model import build_frcnn_model from utils import load_model_from_checkpoint from determined.experimental import Determined,client </code></pre> <pre><code class="language-python">experiment_id = 76 MODEL_NAME = "xview-fasterrcnn" checkpoint = client.get_experiment(experiment_id).top_checkpoint(sort_by="mAP", smaller_is_better=False) print(checkpoint.uuid) loaded_model = load_model_from_checkpoint(checkpoint) </code></pre> <p>Now that you have a checkpoint from the trained object detection model, you can deploy it to Kserve to run inference and predictions.</p> <h1>Part 5: Deploying trained model on Kserve</h1> <p>This notebook walks you each step to deploy a custom object detection model on KServe.</p> <p>Here, I will show you how to:</p> <ul> <li>Install Kserve natively using Kind and Knative</li> <li>Create a Persistent Volume Claim for local model deployment</li> <li>Preparing custom model for Kserve inference</li> <li>Deploying model using a KServe InferenceService</li> <li>Complete a sample request and plot predictions</li> </ul> <p><em>Note: This notebook was tested on a Linux-based machine with Nvidia T4 GPUs. We also assume Docker is installed in your Linux system/environment</em></p> <p>Let's get started!</p> <h2>Pre-reqs: Setting up Python and Jupyter Lab environment</h2> <p>Run the below commands to set up a Python virtual environment, and install all the Python packages needed for this tutorial</p> <pre><code class="language-cwl">sudo apt-get update &#x26;&#x26; sudo apt-get install python3.8-venv python3 -m venv kserve_env source kserve_env/bin/activate pip install kserve jupyterlab torch-model-archiver pip install torch==1.11.0 torchvision==0.12.0 matplotlib jupyter lab --ip=0.0.0.0 \ --port=8008 \ --NotebookApp.token='' \ --NotebookApp.password='' </code></pre> <h2>Install Kserve natively using Kind and Knative</h2> <h3>Install Kind</h3> <p>Open a terminal and run the following bash commands to install a Kubernetes cluster using Kind:</p> <ul> <li><code>curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.18.0/kind-linux-amd64</code></li> <li><code>chmod +x ./kind</code></li> <li><code>sudo mv ./kind /usr/local/bin/kind</code></li> </ul> <p>After running these commands, create a cluster by running the command: <code>kind create cluster</code></p> <h3>Install Kubectl</h3> <p>Run the following bash commmands in a terminal to to install the kubectl runtime:</p> <ul> <li><code>curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"</code></li> <li><code>curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"</code></li> <li><code>sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl</code></li> </ul> <h3>Install Kserve</h3> <p>Run this bash script to install KServe onto our default Kubernetes cluster, note this will install the following artifacts:</p> <ul> <li>ISTIO_VERSION=1.15.2, KNATIVE_VERSION=knative-v1.9.0, KSERVE_VERSION=v0.9.0-rc0, CERT_MANAGER_VERSION=v1.3.0</li> <li><code>bash e2e_blogposts/ngc_blog/kserve_utils/bash_scripts/kserve_install.sh</code></li> </ul> <h3>Patch domain for local connection to KServe cluster/environment</h3> <p>Run this command to patch your cluster when you want to connect to your cluster on the same machine:</p> <p><code>kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving</code></p> <h3>Run port forwarding to access KServe cluster</h3> <ul> <li><code>INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')</code></li> <li><code>kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80</code></li> </ul> <p>Make sure to open a new terminal to continue the configuration.</p> <h3>Create a persistent volume claim for local model deployment</h3> <p>You will be creating a persistent volume claim to host and access the PyTorch-based object detection model locally. A persistent volume claim requires three k8s artifacts:</p> <ul> <li>A persistent volume</li> <li>A persistent volume claim</li> <li>A k8s pod that connects the PVC to be accessed by other k8s resources</li> </ul> <h3>Creating a persistent volume and persistent volume claim</h3> <p>Below is the yaml definition that defines the Persistent Volume (PV) and a PersistentVolumeClaim (PVC). We already created a file that defines this PV in <code>k8s_files/pv-and-pvc.yaml</code></p> <pre><code class="language-yaml">apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/home/ubuntu/mnt/data" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 1Gi </code></pre> <p>To create the PV and PVC, run the command: <code>kubectl apply -f k8s_files/pv-and-pvc.yaml</code></p> <h3>Create k8s pod to access PVC</h3> <p>Below is the yaml definition that defines the k8s Pod that mounts the PersistentVolumeClaim (PVC). We already created a file that defines this PV in <code>k8s_files/model-store-pod.yaml</code></p> <pre><code class="language-yaml">apiVersion: v1 kind: Pod metadata: name: model-store-pod spec: volumes: - name: model-store persistentVolumeClaim: claimName: task-pv-claim containers: - name: model-store image: ubuntu command: [ "sleep" ] args: [ "infinity" ] volumeMounts: - mountPath: "/pv" name: model-store resources: limits: memory: "1Gi" cpu: "1" </code></pre> <p>To create the Pod, run the command: <code>kubectl apply -f k8s_files/model-store-pod.yaml</code></p> <h2>Preparing custom model for Kserve inference</h2> <p>Here we will complete some preparation steps to deploy a trained custom FasterRCNN Object Detection model using KServe. A pre-requisite is to download a checkpoint from a determined experiement. You can read this <a href="https://docs.determined.ai/latest/training/model-management/checkpoints.html#downloading-checkpoints-using-the-cli">tutorial</a> on how to download a checkpoint using the Determined CLI. For this tutorial, you can download an already prepared checkpoint using the following bash command:</p> <ul> <li><code>wget -O kserve_utils/torchserve_utils/trained_model.pth https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/trained_model.pth</code></li> </ul> <h3>Stripping the checkpoint of the optimizer state dictionary</h3> <p>Checkpoints created from a Determined experiment will save both the model parameters and the optimizer parameters. You will need to strip the checkpoint of all parameters except the model parameters for inference. Run the bash command to generate <code>train_model_stripped.pth</code>:</p> <p>Run the below command in a terminal:</p> <pre><code class="language-cwl">python kserve_utils/torchserve_utils/strip_checkpoint.py --ckpt-path kserve_utils/torchserve_utils/trained_model.pth \ --new-ckpt-name kserve_utils/torchserve_utils/trained_model_stripped.pth </code></pre> <h3>Run TorchServe Export to create .mar file</h3> <p>Run the below command to export the PyTorch checkpoint into a .mar file that is required for torchserve inference. The Kserve InferenceService will automatically deploy a Pod with a docker image that support TorchServe inferencing.</p> <pre><code class="language-cwl">torch-model-archiver --model-name xview-fasterrcnn \ --version 1.0 \ --model-file kserve_utils/torchserve_utils/model-xview.py \ --serialized-file kserve_utils/torchserve_utils/trained_model_stripped.pth \ --handler kserve_utils/torchserve_utils/fasterrcnn_handler.py \ --extra-files kserve_utils/torchserve_utils/index_to_name.json </code></pre> <p>After command finishes, run the command to move the file to our prepared <code>model-store/</code> directory:</p> <ul> <li><code>cp xview-fasterrcnn.mar kserve_utils/torchserve_utils/model-store -v</code></li> </ul> <h3>Copy <code>config/</code> and <code>model-store/</code> folders to the K8S PVC Pod</h3> <p>This is the directory structure needed to prepare your custom PyTorch model for KServe inferencing:</p> <pre><code class="language-markdown">├── config │ └── config.properties ├── model-store │ ├── properties.json │ └── xview-fasterrcnn.mar </code></pre> <h4>What the config.properties file looks like</h4> <pre><code class="language-markdown">inference_address=http://0.0.0.0:8085 management_address=http://0.0.0.0:8085 metrics_address=http://0.0.0.0:8082 grpc_inference_port=7070 grpc_management_port=7071 enable_metrics_api=true metrics_format=prometheus number_of_netty_threads=4 job_queue_size=10 enable_envvars_config=true install_py_dep_per_model=true model_store=/mnt/models/model-store model_snapshot={"name": "startup.cfg","modelCount": 1,"models": {"xview-fasterrcnn": {"1.0": {"defaultVersion": true,"marName": "xview-fasterrcnn.mar","serialized-file":"trained_model_stripped.pth","extra-files":"index_to_name.json","handler":"fasterrcnn_handler.py","minWorkers": 1,"maxWorkers": 5,"batchSize": 1,"maxBatchDelay": 100,"responseTimeout": 120}}}} </code></pre> <h4>What the properties.json looks like</h4> <pre><code class="language-json">[ { "model-name": "xview-fasterrcnn", "version": "1.0", "model-file": "", "serialized-file": "trained_model_stripped.pth", "extra-files": "index_to_name.json", "handler": "fasterrcnn_handler.py", "min-workers" : 1, "max-workers": 3, "batch-size": 1, "max-batch-delay": 100, "response-timeout": 120, "requirements": "" } ] </code></pre> <p>Note that there is a <code>config/</code> folder that includes a config.properties. This defines A. There is also a <code>model-store/</code> directory that contains are exported models and a <code>properties.json</code> file. You will need this file for B.</p> <p>Now, run several kubectl commands to copy over these folders into your Pod and into the PVC defined directory.</p> <ul> <li><code>kubectl cp kserve_utils/torchserve_utils/config/ model-store-pod:/pv/config/</code></li> <li><code>kubectl cp kserve_utils/torchserve_utils/model-store/ model-store-pod:/pv/model-store/</code></li> </ul> <p>Run these commands to verify the contents have been copied over to the pod.</p> <ul> <li><code>kubectl exec --tty model-store-pod -- ls /pv/config</code></li> <li><code>kubectl exec --tty model-store-pod -- ls /pv/model-store</code></li> </ul> <h2>Deploying a model using a KServe InferenceService</h2> <h3>Create Inference Service</h3> <p>Below is the yaml definition that defines the KServe InferenceService that deploys models stored in the PVC. A file that defines this PV has already been created in <code>k8s_files/torch-kserve-pvc.yaml</code></p> <pre><code class="language-yaml">apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "torchserve" spec: predictor: pytorch: storageUri: pvc://task-pv-claim/ </code></pre> <p>To create the Pod, run the command: <code>kubectl apply -f kserve_utils/k8s_files/torch-kserve-pvc.yaml</code></p> <ul> <li>After running the previous command <code>kubectl get inferenceservice</code>, you should see that the inferenceservice is not loaded yet</li> <li>Keep running the command every minute until you see the InferenceService loaded (view screenshot below of example)</li> <li>Next, run command <code>kubectl get pods</code> to get underlying pod that is running inference service. Copy the pod name (example seen in screenshot)</li> <li>Finally run command: <code>kubectl logs -f &#x3C;POD_NAME></code> to see the logs and if the model was successfully loaded.</li> </ul> <h3>Complete a sample request and plot predictions</h3> <pre><code class="language-python">import json import requests import base64 from PIL import Image from PIL import ImageDraw import matplotlib.pyplot as plt </code></pre> <pre><code class="language-python">filename='kserve_utils/torchserve_utils/example_img.jpg' im = Image.open(filename) </code></pre> <p>Here is the test image that will be sent to the deployed model.</p> <pre><code class="language-python">im </code></pre> <p><img src="/img/output_26_0.png" alt="png"></p> <p>Now, encode the image into the base64 binary format.</p> <pre><code class="language-python">image = open(filename, 'rb') # open binary file in read mode image_read = image.read() image_64_encode = base64.b64encode(image_read) bytes_array = image_64_encode.decode('utf-8') request = { "instances": [ { "data": bytes_array } ] } result_file = "{filename}.{ext}".format(filename=str(filename).split(".")[0], ext="json") print("Result File: ",result_file) with open(result_file, 'w') as outfile: json.dump(request, outfile, indent=4, sort_keys=True) </code></pre> <pre><code>Result File: kserve_utils/torchserve_utils/example_img.json </code></pre> <p>Now you can submit the image to the deployed endpoint and visualize the predictions!</p> <pre><code class="language-python">headers = { "Host": "torchserve.default.example.com" } data = open(result_file) response = requests.post('http://localhost:8080/v1/models/xview-fasterrcnn:predict', headers=headers, data=data) resp = json.loads(response.content) print("Number of Predictions: ", len(resp['predictions'][0])) </code></pre> <pre><code>Number of Predictions: 95 </code></pre> <pre><code class="language-python">draw = ImageDraw.Draw(im) </code></pre> <pre><code class="language-python">for pred in resp['predictions'][0]: assert len(list(pred.keys())) == 2 cl_name = list(pred.keys())[0] bboxes = pred[cl_name] # print(cl_names) # print(bboxes) # print("score: ",pred['score']) if pred['score'] > 0.4: # bboxes = [int(i) for i in bboxes] # print(bboxes[0],type(bboxes[0])) draw.rectangle([bboxes[0],bboxes[1],bboxes[2],bboxes[3]],outline=(255,0,0),fill=None,width=1) draw.text([bboxes[0],bboxes[1]-10],"{} :{:.2f}".format(cl_name,pred['score']),fill=(250,0,0)) </code></pre> <p>Here, you can see the model predictions overlaid onto the input image.</p> <pre><code class="language-python">plt.figure(figsize=(12,12)) plt.imshow(im) </code></pre> <pre><code>&#x3C;matplotlib.image.AxesImage at 0x7ff6341ccf10> </code></pre> <p><img src="/img/output_34_1.png" alt="png"></p> <h2>Conclusion</h2> <p>There are numerous ways to get started with end-to-end ML model training with HPE. You can find the <a href="https://github.com/interactivetech/e2e_blogposts/tree/main/ngc_blog">full example on GitHub here</a> To get started with Determined AI’s open source model training platform, visit the <a href="https://docs.determined.ai/latest/">Documentation</a> page.  </p> <p>If you’re ready to begin your ML journey with HPE, we’re excited to help you get started! <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> comes with premium HPE support, which you can read more about <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">here</a>.  </p> <p>HPE also offers a purpose-built, turnkey AI infrastructure for model development and training at scale. <a href="https://www.hpe.com/us/en/hpe-machine-learning-development-system.html">Get in touch with us about HPE Machine Learning Development System here</a>.</p>https://developer.hpe.com/meet-open-source-contributor-and-fpart-fpsync-project-developer-ganael-laplanche/https://developer.hpe.com/meet-open-source-contributor-and-fpart-fpsync-project-developer-ganael-laplanche/Mon, 12 Jun 2023 19:54:14 GMT<p>As part of our blog series on open source experts, the HPE Developer team recently met up with Ganael Laplanche, the project developer for <a href="https://www.fpart.org/">Fpart</a>, a sysadmin-oriented tool that helps users sort files and pack them into bags or 'partitions'. Here, we'll introduce you to his work, how it came about, and learn more about what got Ganael involved with working with open source software.</p> <h1>Ganael, can you tell us a little about the tools Fpart and Fpsync?</h1> <p>The project started when I was working for a renowned center for biomedical research after a discussion with a friend of mine. We wanted to implement a fast bin-packing tool to produce filesystem tree partitions with the same size and number of files. The tool quickly evolved and got support for hooks that can be triggered when a partition is generated.</p> <p>At that time, we needed to move petabyte-scale filesystems to freshly-acquired storage arrays. With its new hooking system, Fpart seemed to be a good basement to launch small migration jobs in parallel through our SSH cluster. Initial tests (<a href="https://connect.ed-diamond.com/GNU-Linux-Magazine/glmf-164/parallelisez-vos-transferts-de-fichiers">see our article in French</a>) were successful but we were still depending on our on-site scheduler to orchestrate submitted jobs and it was to be retired sooner or later. We needed a new scheduler.</p> <p>That's where <a href="https://www.fpart.org/fpsync/">Fpsync</a> comes into play : the tool wraps Fpart and embeds its own scheduler to trigger small <a href="https://rsync.samba.org/">Rsync</a> jobs to parallelize data migration <em>by itself</em>. It can leverage your SSH cluster to get the best from your data servers, acting as a powerful, standalone, data migration tool.</p> <p>Of course, as an ardent open source supporter, those tools were released with an open source license (BSD 2-Clause "Simplified" License). They were quickly adopted by large companies (Intel, AWS, Microsoft, Alibaba, Oracle, ...) as well as research centers to migrate petabyte-scale filesystems.</p> <h1>What attracted you to free software?</h1> <p>I first discovered free software by reading magazines that were surfing on Linux hype during mid-90's (and trying their GNU/Linux distros offered on CDROM). But I really began to understand what free software meant later during my studies. I was immediately seduced by the thought that it exemplified humanity's best attribute: the willingness to share knowledge in order to move forward together.</p> <p>As a student, this was very important to me: it enabled me to learn more, as the code is freely available and the open source community very responsive. I quickly felt that I owed the community something in return; I didn't want to use all that free software (as in free beer) without giving something back. So I started looking at how I could make my own contribution. This is where FreeBSD played a important role, acting as a catalyst...</p> <h1>Why did you come to FreeBSD as a development platform?</h1> <p>There are several reasons for that choice. As a curious student, I tried <a href="https://www.freebsd.org/">FreeBSD</a> in the early 2000's, testing version 4.5. What impressed me at that time was its documentation (<a href="https://docs.freebsd.org/en/books/handbook/">"handbook"</a>) and man ("manual") pages. While GNU/Linux appeared complex to me, FreeBSD suddenly became more clear. With a very nice and welcoming community, it was the perfect platform for a newcomer into the UNIX world. I became hooked on FreeBSD and haven't returned to any other system since.</p> <p>I later came to understand another reason why FreeBSD appeared more clear. It is a homogeneous system, not a patchwork of very different projects. This makes a world of difference, as a <em>specific</em> version of FreeBSD represents a <em>specific</em> version of base components (called "world") and kernel, offering up a complete system. World and kernel are all maintained by the same entity (<a href="https://docs.freebsd.org/en/articles/contributors/">FreeBSD developers</a>) and, because of this, everything is consistent - from any options to the documentation and man pages. This delivers great value for users and guarantees a level of robustness and stability for the system.</p> <p>FreeBSD is a good choice for developers because it is POSIX compliant. This is important if you want to produce portable code. Also, it is very easy to access source code for world, kernel and ports (third-party applications ported to FreeBSD). One can easily patch things and test the modifications, which is a bit harder on other systems where you would often have to install a dedicated source package to be able to patch it.</p> <p>Finally, the system is a pleasure to administrate and update. I think I have not needed to reinstall my machine since the late 2000's; I've only performed updates since. Third-party applications can now be easily installed and upgraded using binary packages, which was not the case when I first started using FreeBSD.</p> <p>These are all the reasons why I use FreeBSD on my systems - not just for servers and development, but also as a daily desktop OS. Lots of people still think FreeBSD is not ready for everyday use on the desktop, but I am living proof that this is not true!</p> <h1>What other open source projects are you involved with?</h1> <p>I became a FreeBSD developer in 2011 and I now maintain more than 40 ports (A port is a set of patches and build options that makes a software work on FreeBSD. It also acts as the basis for binary packages). Maintaining ports is a fantastic hobby because on the one hand, you have the chance to work on your favorite OS, and on the other hand, you can contribute patches back upstream. This way, you are always connected with the different communities.</p> <p>Aside from my FreeBSD activities, I have several <a href="https://contribs.martymac.org/">personal projects</a>. I mentioned Fpart and Fpsync, but I am also the author of ldapscripts, a set of tools used to simplify user and group management within an LDAP directory. They are quite old now, but they still do the job. I also worked on various smaller projects, such as sms1xxx kernel module (a port of Linux' Siano DVB-T driver to FreeBSD, now deprecated in favor of webcamd), evtViewer (a viewer for Ms event log files) or Grpar (a Build engine group archive extract tool). I also wrote several courses (in French).</p> <p>I also try to contribute to software I use when I find a bug (either by fixing it or at least by reporting it).</p> <h1>Is there anything else you’d like to share with our readers?</h1> <p>I owe a lot to free software. That's mostly what allowed me to learn computing, making my career possible. That's why I contribute back the most I can.</p> <p>But that takes time (that is, personal time) and money (we need machines to test on, as well as power to run them). I am glad to see more and more companies supporting open source. Recently, HPE provided me with a replacement for my old server, I'll never thank them enough for that kindness! This HPE ProLiant ML350 allows me to perform faster Fpart and Fpsync tests as well as compile code far more quicker than with my old machine. This is a sign that things are changing. I think everybody now understands why it is so important to support open source development. Providing hardware is a simple yet very efficient way of supporting open source developers, sharing code is another one. Let's encourage companies to continue that way!</p> <p>As for individuals, do not hesitate to report bugs or share code. You will participate in making great things and get fantastic feedback from the community!</p>https://developer.hpe.com/announcing-hpe-swarm-learning-2-0-0/https://developer.hpe.com/announcing-hpe-swarm-learning-2-0-0/Mon, 12 Jun 2023 16:48:53 GMT<p>We’re excited to announce the HPE Swarm Learning 2.0.0 community release!!</p> <p>In the previous version of HPE Swarm Learning, if the sentinel Swarm Network (SN) node goes down during Swarm training, the training process would stop, and there was no way to resume it. However, with this release, we have addressed the issue by implementing a mesh topology (connectivity) between SNs, replacing the previous star topology where only the sentinel SN was connected to other SNs.</p> <p>Also, we now support multiple blockchain miners instead of just one miner in the sentinel SN. Now, even if the initial sentinel SN goes down, since other SNs also function as miners, it allows the training to continue uninterrupted. Additionally, when the initial sentinel SN is down and if a new SN wants to join the network, it can seamlessly integrate and join the Swarm network with the help of any other SN node. This <strong>high availability configuration</strong> ensures improved resilience and robustness of HPE Swarm Learning.</p> <p>In the HPE Swarm Learning sync stage (defined by sync frequency), when it is time to share the learning from the individual model, one of the Swarm Learning (SL) nodes is designated as the “leader” node. This leader node collects the individual models from each peer node and merges them into a single model by combining parameters of all the individuals. The <strong>Leader Failure Detection and Recovery (LFDR)</strong> feature enables SL nodes to continue Swarm training during the merging process when an SL leader node fails. A new SL leader node is selected to continue the merging process. If the failed SL leader node comes back after the new SL leader node is in action, the failed SL leader node is treated as a normal SL node and contributes its learning to the swarm global model.</p> <p>With the HPE Swarm Learning v2.0.0 release, a user can now extend a Swarm client to support other machine learning platforms as well. Currently Swarm client supports machine learning platforms like PyTorch and Keras (based on Tensorflow 2 in backend). Please find the instructions to extend Swarm client <a href="https://github.com/HewlettPackard/swarm-learning/blob/master/lib/src/README.md">here</a>.</p> <h4><strong>2.0.0 release contains following updates:</strong></h4> <ul> <li> <p>High availability for SN</p> <ul> <li>Handles Sentinel node failure</li> <li>Ensures any SN node can act as sentinel while adding new node</li> <li>Supports mesh topology of SN network</li> </ul> </li> <li> <p>High availability for SL leader</p> <ul> <li>Elects a new merge leader when a leader failure is detected</li> <li>Handles stale leader recovery</li> </ul> </li> <li> <p>Swarm Learning Management UI (SLM-UI)</p> <ul> <li>Supports Swarm product installation through SLM-UI</li> <li>Deploys and manages Swarm Learning through SLM-UI</li> </ul> </li> <li> <p>Swarm client library</p> <ul> <li>Extends Swarm Learning for new ML platforms</li> </ul> </li> <li> <p>Improved diagnostics and utility script for logs collection.</p> </li> </ul> <h4>For complete details on this new release, please refer to the following resources:</h4> <ul> <li><a href="https://github.com/HewlettPackard/swarm-learning">HPE Swarm Learning home page</a></li> <li><a href="https://github.com/HewlettPackard/swarm-learning/blob/master/lib/src/README.md">HPE Swarm Learning client readme</a></li> </ul> <p>For any questions, start a discussion in our <a href="https://hpedev.slack.com/archives/C04A5DK9TUK">#hpe-swarm-learning</a> slack channel on <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a></p>https://developer.hpe.com/2023-June-06/https://developer.hpe.com/2023-June-06/Mon, 05 Jun 2023 13:00:00 GMThttps://developer.hpe.com/automate-boot-from-san-target-configurations-using-redfish/https://developer.hpe.com/automate-boot-from-san-target-configurations-using-redfish/Thu, 01 Jun 2023 20:13:38 GMT<p>As the scale and complexity of infrastructure deployed in a customer environment increases, the need for consistency, ease of management and agility in responding to the changing needs of the infrastructure consumers becomes a key priority for IT staff. Operational efficiency has become a key business imperative for customers hosting any large IT footprint, and the operational efficiency gains are often achieved through automation of repetitive tasks performed by IT staff to deploy, manage, and operate IT infrastructure.</p> <p>There are several use-cases of infrastructure automation that exist, and in this blog post we will address the aspect of configuration of OS boot for a server from a Storage Area Network (SAN) based storage array. Specifically, we will discuss the procedure for configuring an HPE ProLiant DL Gen10, Gen10 plus and Gen11 servers to boot from a SAN target Logical Unit Number (LUN) using the Redfish® standard DMTF API.</p> <p><strong>Note</strong>:</p> <ul> <li>The steps outlined in this document were tested on HPE DL 380 Gen10 and Gen10 plus servers with Emulex based SN1610E FibreChannel (FC) cards, HPE iLO firmware 2.81 and Service Pack for ProLiant (SPP) available as of March 2023.</li> <li>QLogic SN1610Q cards do not support the steps outlined in this document (as of April 2023). However, future firmware releases might enable this Redfish API-based configuration method.</li> </ul> <h1>Pre-requisite information</h1> <p>Before we dive into the steps for performing the automation, you will need to gather a few pieces of information, such as</p> <ul> <li>HPE iLO Management port IP address and credentials.</li> <li>Storage target World Wide Number (WWN) and LUN id. A storage administrator can typically provide you with this information.</li> </ul> <p>Download and install the <a href="https://www.hpe.com/info/resttool">HPE RESTful Interface Tool</a> (version 4.1.0.0 or later) on the system you will initiate the automation actions from.</p> <p>Once the RESTful Interface tool (referred to as iLOREST in the remainder of the document) has been installed, you can review the current boot target and their order of execution by following the steps below</p> <p><code>ilorest login &#x3C;ilo-ip> -u &#x3C;ilo-user> -p password</code></p> <p><code>ilorest bootorder</code></p> <p>Refer to the iLOREST<a href="https://hewlettpackard.github.io/python-redfish-utility/#bootorder-command">documentation</a> for more information on this command.</p> <h1>Configuration Steps</h1> <p>Now that you have gathered the pre-requisite information and installed the iLOREST tool, let’s discuss the detailed steps for implementing the automation.</p> <h2>Step 1: Ensure that the <code>FCScanPolicy</code> value is set to ‘AllTargets’</h2> <pre><code class="language-markdown">ilorest get FCScanPolicy --selector Bios </code></pre> <p>Expected output:</p> <pre><code class="language-markdown">FCScanPolicy=AllTargets </code></pre> <p>If the value is not set to AllTargets, you can change it using the following:</p> <pre><code class="language-markdown">ilorest set "FCScanPolicy"="AllTargets" --selector Bios ilorest commit ilorest reboot </code></pre> <p><strong>Note</strong>:</p> <p>After restarting the server, remember to retrieve the updated Bios settings from the iLO of the server with the following command:</p> <pre><code class="language-markdown">ilorest select Bios. --refresh </code></pre> <p>You can monitor the server restart state and continue the configuration when it reaches the <code>FinishedPost</code> state, and the <code>vMainDeviceDiscoveryComplete</code> device discovery state. More details about monitoring server state is described in this <a href="https://developer.hpe.com/blog/master-the-redfish-server-states-to-improve-your-monitoring-and-manageme/">article</a>.</p> <p>The server state can be retrieved with the following command:</p> <pre><code class="language-markdown">ilorest serverstate </code></pre> <p>The device discovery state can be obtained with the following commands:</p> <pre><code class="language-markdown">ilorest select ComputerSystem. --refresh ilorest get Oem/Hpe/DeviceDiscoveryComplete/DeviceDiscovery --json </code></pre> <p>After restarting the server, you can verify that the settings have taken effect using the BIOS menus as shown in Figure 1 or using the commands:</p> <pre><code>ilorest select Bios. --refresh ilorest get FCScanPolicy </code></pre> <p><img src="/img/picture1.png" alt="Figure 1: FCScanPolicy Bios setting" title="Figure 1: FCScanPolicy Bios setting"></p> <h2>Step 2: Identify the right FC Card</h2> <p>The server you are trying to configure might have one or more FC cards in it, and you need to identify the right card using the steps below.</p> <p>List out available network adapters (includes FC cards)</p> <pre><code class="language-markdown">ilorest rawget /redfish/v1/Chassis/1/NetworkAdapters/ </code></pre> <p>The previous command returns the following output:</p> <pre><code class="language-json">{ "@odata.context": "/redfish/v1/$metadata#NetworkAdapterCollection.NetworkAdapterCollection", "@odata.etag": "W/"F09DE05C"", "@odata.id": "/redfish/v1/Chassis/1/NetworkAdapters/", "@odata.type": "#NetworkAdapterCollection.NetworkAdapterCollection", "Description": "The collection of network adapter resource instances available in this chassis.", "Members": [ { "@odata.id": "/redfish/v1/Chassis/1/NetworkAdapters/DC080000/" }, { "@odata.id": "/redfish/v1/Chassis/1/NetworkAdapters/DE083000/" }, { "@odata.id": "/redfish/v1/Chassis/1/NetworkAdapters/DE081000/" }, { "@odata.id": "/redfish/v1/Chassis/1/NetworkAdapters/DE07B000/" } ], "Members@odata.count": 4, … </code></pre> <p>Make a note of the device IDs in the output above - DC080000, DE083000, DE081000, DE07B000. The values for these items might be different on the server you are using.</p> <p>Query these items using the device ID to verify it is the right FC card that you would like to configure, for example using DE083000 device (your output and device IDs might be different).</p> <pre><code class="language-markdown">ilorest rawget /redfish/v1/Chassis/1/NetworkAdapters/DE083000 </code></pre> <p>Look for the following in the output.</p> <pre><code class="language-markdown">"Location": { "PartLocation": { "LocationOrdinalValue": 4, "LocationType": "Slot", "ServiceLabel": "PCI-E Slot 4" } … "Manufacturer": "Hewlett Packard Enterprise", "Model": "HPE StoreFabric SN1610E 32Gb 2-port Fibre Channel Host Bus Adapt", "Name": "HPE SN1610E 32Gb 2p FC HBA" </code></pre> <p>The above values can help you ascertain that you have the right card, and its device ID.</p> <h2>Step 3: Verify that Redfish configuration is Enabled</h2> <p>Before you can change the boot from SAN configurations on an FC card, you will have to set <code>RedfishConfiguration</code> key to "Enabled" for the specific adapter, if it has not already been done.</p> <p>The current value of this key can be viewed by querying the adapter.</p> <pre><code class="language-markdown">ilorest rawget /redfish/v1/Chassis/1/NetworkAdapters/DE083000 </code></pre> <p>If the value is not set, please perform the following steps:</p> <ol> <li> <p>Create a file named enable-redfish.txt with the following contents</p> <pre><code class="language-json">{ "/redfish/v1/Chassis/1/NetworkAdapters/DE083000/": { "Oem": { "Hpe": { "RedfishConfiguration": "Enabled" } } } } </code></pre> </li> <li> <p>Apply this patch via iLOREST</p> <pre><code class="language-markdown">ilorest rawpatch &#x3C;file path>\enable-redfish.txt </code></pre> </li> <li> <p>To make this patch permanent, flush the configuration to the iLO NVRAM by creating a file name flushtonvm.txt, with the following contents:</p> <pre><code class="language-json">{ "/redfish/v1/Chassis/1/NetworkAdapters/DE083000/Actions/Oem/Hpe/HpeNetworkAdapter.FlushConfigurationToNVM/": { } } </code></pre> </li> <li> <p>Post the flush action via iLOREST</p> <pre><code class="language-markdown">ilorest rawpost &#x3C;file path>\flushtonvm.txt </code></pre> </li> <li> <p>Reboot the server</p> <pre><code class="language-markdown">ilorest reboot </code></pre> </li> </ol> <h2>Step 4: Set boot from SAN value to "Enable" for the adapter</h2> <p>For each SN1610E adapter on your server that you would like to enable for SAN boot, there is a variable "BootMode" that needs to have a value of "FibreChannel".</p> <p>This can be viewed from BIOS configuration UI at the location shown in Figure 2.</p> <p><img src="/img/picture2.png" alt="" title="Figure 2: Set boot from SAN"></p> <p>This value is part of the FC adapter configuration, and is visible in the output of the following command:</p> <pre><code class="language-markdown">ilorest rawget /redfish/v1/Chassis/1/NetworkAdapters/DE083000/NetworkDeviceFunctions/1/ </code></pre> <p>Look for "BootMode" in the output generated.</p> <p>To modify the value of this property,</p> <ol> <li> <p>Create a file named enable-sanboot.txt with the following:</p> <pre><code class="language-json">{ "/redfish/v1/Chassis/1/NetworkAdapters/DE083000/NetworkDeviceFunctions/1/": { "BootMode":"FibreChannel" } } </code></pre> </li> <li> <p>Apply this patch via iLOREST</p> <pre><code class="language-markdown">ilorest rawpatch &#x3C;file path>\enable-sanboot.txt </code></pre> </li> <li> <p>Flush or commit the changes to ilo NVM using the flushtonvm.txt file created before</p> <pre><code class="language-markdown">ilorest rawpost &#x3C;file path>\flushtonvm.txt </code></pre> <h2>Step 5: Create the SAN boot target entry</h2> <p>Once the boot from SAN is enabled on the FC card, you will need to create a mapping for the Storage LUN that would be used as a boot entry for the server.</p> <p>This entry is created within the BootTargets collection for the Port on the Adapter.</p> <pre><code class="language-markdown">ilorest rawget /redfish/v1/Chassis/1/NetworkAdapters/DE083000/NetworkDeviceFunctions/1/ </code></pre> <p>In the output, look for entries such as:</p> <pre><code class="language-json">"FibreChannel": { "BootTargets": [ { "BootPriority": 0, "LUNID": "00:00:00:00:00:00:00:00", "WWPN": "00:00:00:00:00:00:00:00" } </code></pre> <p>To create this entry, you will need to find out the Storage WWN and the LUN number exported to this server.</p> <ol> <li> <p>Create a patch file (example bootentry.txt file contents shown below).</p> <pre><code class="language-json">{ "/redfish/v1/Chassis/1/NetworkAdapters/DE083000/NetworkDeviceFunctions/1": { "FibreChannel": { "BootTargets":[ { "BootPriority": 0, "LUNID": "00:00:00:00:00:00:00:00", "WWPN": "00:00:00:00:00:00:00:C3" } ] } } } </code></pre> </li> <li> <p>Apply this patch file via iLOREST</p> <pre><code class="language-markdown">ilorest rawpatch &#x3C;file path>\bootentry.txt </code></pre> </li> <li> <p>Commit the changes by flashing the ilo NVRAM using the flushtonvm.txt file created before.</p> <pre><code class="language-markdown">ilorest rawpost &#x3C;file path>\flushtonvm.txt </code></pre> </li> <li> <p>Restart the server.</p> <pre><code class="language-markdown">ilorest reboot </code></pre> </li> <li> <p>Once the server has restarted, you can view the boot order to verify that the new entry for the SAN boot target has been created.</p> <pre><code class="language-markdown">ilorest bootorder </code></pre> <h1>Summary</h1> <p>In this blog post, we have learnt how easy it is to automate the configuration of boot from SAN targets for HPE servers using the iLOREST tool. There are other aspects of server provisioning, monitoring, and management that you can automate with this tool and using the Redfish API interface supported by HPE iLO.</p> <p>To learn more about these topics, check out <a href="https://developer.hpe.com/search/?term=redfish">HPE Developer portal</a> for additional blogs and technical articles.</p> </li> </ol> </li> </ol>https://developer.hpe.com/new-for-hpe-discover-2023-hack-shack-serves-as-home-base-for-hpe’s-office-of-the-cto/https://developer.hpe.com/new-for-hpe-discover-2023-hack-shack-serves-as-home-base-for-hpe’s-office-of-the-cto/Mon, 22 May 2023 11:06:25 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Hewlett Packard Enterprise (HPE) will be hosting Discover 2023 from June 20-22 at the Venetian Convention and Expo Center in Las Vegas, Nevada. The HPE Developer Community team is excited to be there once again to greet customers and partners in the Hack Shack. There will be some exciting changes this year. Read on to learn more about what will be happening in the Hack Shack!</p> <h2>What’s new for the Hack Shack?</h2> <p>Since 2017, the HPE Hack Shack has been enjoyed by developers as a place to meet with HPE software engineers and other software subject matter experts, to brainstorm about solutions to issues they may be experiencing back home, and to learn a bit more about what HPE is doing in the software space. It’s also provided an opportunity for attendees to enjoy a little down time, relaxing over a few games and software-focused Hack Shack challenges.</p> <p>This year, the Hack Shack will serve as the Technology Hub for technologists of all types; be they developers, data scientists, ITOps or DevOps personnel. It will be the home base for the HPE Office of the Chief Technology Officer (CTO), where you’ll be able to meet and chat with engineering representatives and subject matter experts (SMEs) who work on the HPE GreenLake edge-to-cloud platform. Here, you’ll get to have in-depth conversations about the platform that was created to eliminate complexity, delivering a cloud experience for customer workloads wherever their data may be.</p> <p>The Hack Shack will complement the CTO Theater, where over a dozen different HPE GreenLake platform sessions will take place covering everything from platform APIs to future innovations, by providing a place where you can continue the conversations and hone in on any specific questions you might have.</p> <h2>What is the HPE Office of the CTO?</h2> <p>Today’s highly interconnected world requires easier, more secure ways to access and use the data that has become our life force. The HPE Office of the Chief Technology Officer (CTO) leads HPE’s hotbed of innovation. Office of the CTO team members are heavily involved in R&#x26;D to find and develop new ways to do this. They help create history, pushing the industry forward by redefining what cloud computing means and advancing customers’ and partners’ businesses in ways they could not previously have imagined.</p> <p>The organization is responsible for the HPE GreenLake edge-to-cloud platform, bringing the economy, ease of use, and scalability of a cloud experience to where a customer’s data lives. In addition, the group incubates projects around a variety of emerging technologies to bring the company’s most innovative products and services to market.</p> <p>As companies strive for a future where they are fully autonomous, HPE GreenLake will help them get there, delivering AI-driven smart services that are:</p> <ul> <li>Fully automated</li> <li>Continuous</li> <li>Self-healing</li> <li>Self-aware</li> <li>Fully distributed</li> <li>Auto-compliant</li> <li>Self-learning</li> <li>Self-securing</li> </ul> <p>The Office of the CTO is driving this transformation through its work on the HPE GreenLake platform and its innovation. The group is home to some of the brightest and best talent that HPE has to offer, and is working to attract even more.</p> <h2>What’s happening specifically in the Hack Shack?</h2> <p><img src="/img/hack-shack-image.png" alt="Hack Shack at HPE Discover" title="Hack Shack at HPE Discover"></p> <h3>Prize winning activities and celebration party</h3> <p>As in previous years, the Hack Shack will feature a lot of fun activities. The HPE Developer Community Virtual Treasure Hunt is a fan favorite. This year, 12 winners will be awarded a developer-branded hoodie/sweatshirt! There will be games set up in the yard, like Jenga and corn hole, as well.</p> <p>The traditional Hack Shack Celebration party will be held on Wednesday, June 21st. Make sure you stop on by to join in on the fun, food, and games!</p> <h3>Hack Shack challenges</h3> <p>For those looking for more mind-stimulating activities, there will be 4 Hack Shack challenges this year focused on the HPE GreenLake edge-to-cloud platform. They will center on the HPE GreenLake Aruba Central API, the HPE Data Services Cloud Console API, the HPE Compute Ops Management API and the HPE GreenLake Terraform provider. These are perfect for those who’d like some hands-on experience in working with these APIs.</p> <h3>Post-presentation “Meet the experts” sessions</h3> <p>Event attendees are invited to Meet-the-Expert sessions immediately following presentations being held in the CTO Theater. After attending the session in the theater, you’ll have the opportunity to follow the speaker back to the Hack Shack and continue the conversation in a fun and friendly environment.</p> <h3>Customer-engaging meetups</h3> <p>New for this year, the Hack Shack will host 4 different meetup sessions for customers to meet with the CTO, other HPE executives, and HPE Fellows. These meetups are interactive sessions designed to collect customer feedback. In many of them, you’ll be able to share your experiences, challenges, and expectations related to a particular topic. Take advantage of these opportunities to connect with HPE executives on:</p> <ul> <li> <p><strong>HPE GreenLake Private Cloud Enterprise</strong></p> <p>In this meetup, share your thoughts on what services, in addition to the resource capacity, could be offered from a managed private cloud services offering. We’d love to hear what key features, functionalities, and capabilities you’d like to see offered as part of, or in addition to the existing bare-metal, VMs and container offerings.</p> </li> <li> <p><strong>Unlocking Data Insights: Maximizing Your Cloud's Potential</strong></p> <p>In this meetup, we’d love to hear your thoughts on what the minimum data services a cloud platform should offer. Focus on key features, functionalities, and capabilities that you consider essential in any cloud data service.</p> </li> <li> <p><strong>Site-as-a-Service</strong></p> <p>Starting with a brief presentation from the Office of the CTO on HPE current and near-term projects, this meetup will focus on Site-as-a-Service, where HPE performs edge management on behalf of an enterprise, as well as work in distributed data and workload management at the edge. After the presentation, we will lead an open discussion on the challenges and requirements of the use and management of distributed edge sites in the enterprise and how HPE can help address them.</p> </li> <li> <p><strong>Simplifying Hybrid Cloud Operations</strong></p> <p>In this meetup, we invite you to share your perspectives and experiences in managing IT operations across a hybrid cloud environment. We’ll delve into various challenges and use cases in monitoring, managing events, and automating tasks using the OpsRamp solution. Don’t miss out on this opportunity to engage with industry peers and have a say in product directions by joining this session.</p> </li> </ul> <h2>What sessions are being held in the CTO Theater?</h2> <p>To get details on these 15 sessions, go to the <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1050&#x26;locale=en_US">Content Catalog</a> and search on CTO has the keyword.</p> <ul> <li>API for HPE GreenLake block storage and next-gen platforms [<strong>CTO5731</strong>]</li> <li>Future of ITOps on the HPE GreenLake platform [<strong>CTO5734</strong>]</li> <li>Gain visibility and insights for optimizing the sustainability of your IT estate [<strong>CTO5689</strong>]</li> <li>Getting started with the HPE GreenLake platform [<strong>CTO5732</strong>]</li> <li>HPE GreenLake for Private Cloud Enterprise: APIs and developer experience [<strong>CTO5736</strong>]</li> <li>Intro to the HPE GreenLake for Compute Ops Management API [<strong>CTO5723</strong>]</li> <li>Introducing HPE GreenLake for Disaster Recovery [<strong>CTO5726</strong>]</li> <li>Introduction to Aruba Central automation [<strong>CTO5725</strong>]</li> <li>Join the HPE Developer Community to collaborate and build [<strong>CTO5727</strong>]</li> <li>Seamless onboarding to the HPE GreenLake platform: user management and SSO configuration [<strong>CTO5733</strong>]</li> <li>Simplify management of multi-cloud hybrid IT with HPE [<strong>CTO5735</strong>]</li> <li>Under the hood of the HPE GreenLake platform [<strong>CTO5729</strong>]</li> <li>Unified API strategy enhances HPE GreenLake experience [<strong>CTO5728</strong>]</li> <li>What’s next for HPE? OCTO platform innovations [<strong>CTO5737</strong>]</li> <li>Zero trust through SPIFFE at scale [<strong>CTO5730</strong>]</li> </ul> <h2>Is there any swag or party this year?</h2> <p>Yes! The celebration party will be held at 5.30 pm on Wednesday evening. Attendees will be treated to food and refreshments and hear from some of the renowned members of our community. Those who attend the celebration party will receive a developer-branded cap.</p> <p>We’ll also be giving out t-shirts to those who participate in any of our Hack Shack activities, such as the virtual treasure hunt or a Workshop-on-Demand.</p> <p><img src="/img/swag-2.png" alt="Giveaways at HPE Discover" title="Giveaways at HPE Discover"></p> <h2>Join us in the Hack Shack!</h2> <p>If you haven’t stopped by the Hack Shack before – maybe because you thought it was just for developers – make sure you join us this year. Learn about the HPE GreenLake edge-to-cloud platform. Learn what resources HPE can provide to help you build on and use to work with the platform. It’s a great opportunity to get to meet with the experts and answer any questions you might have.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <a href="https://attend.hpe.com/discover2023/index.cfm?iLangID=1"><button type="button" class="button">Register now!</button></a> </div>https://developer.hpe.com/upgrade-kubernetes-clusters-using-hpe-greenlake-terraform-provider/https://developer.hpe.com/upgrade-kubernetes-clusters-using-hpe-greenlake-terraform-provider/Tue, 16 May 2023 09:00:43 GMT<p>IaC, or Infrastructure as code, is a practice of automating the process of managing and provisioning infrastructure through the use of code instead of using manual processes. It gives organizations the tools required to create, manage, and destroy compute resources by statically defining and declaring these resources in codeblocks. It helps increase operational agility, simplify management, reduce errors, and save cost.</p> <p>In this post, I will explore options used to declare and upgrade Kubernetes clusters on HPE GreenLake using the HPE GreenLake Terraform Provider.</p> <p>In terms of upgrades, the following 2 scenarios are supported:</p> <ol> <li>Scaling of a cluster's worker nodes. Please refer to the blog post <a href="https://developer.hpe.com/blog/scale-kubernetes-cluster-using-hpe-greenlake-terraform-provider/">Scale Kubernetes Clusters using HPE GreenLake Terraform Provider</a> to check out scaling options available for worker nodes.</li> <li>Upgrade the Kubernetes version of the cluster. This step is covered in this blog.</li> </ol> <h1>Prerequisite</h1> <p>Before implementing the steps shown in this tutorial, please read the blog post <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code - Part 1</a>, which includes the steps required to create a Kubernetes cluster. This post expands upon that scenario by examining how to upgrade a cluster's Kubernetes version.</p> <h1>Verify existing Kubernetes cluster</h1> <p>After the cluster is created following the instructions found in the <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code - Part 1 blog post</a>, launch the HPE GreenLake Central console and verify that the cluster is present under the appropriate tenant.</p> <p>You should see the <strong>tf-test</strong> cluster present under <strong>Dashboard -> Manage your Private Cloud -> Containers</strong>.</p> <p><img src="https://developer.hpe.com/img/cluster_list.jpg" alt=""></p> <p>Shown below is the reference Terraform configuration file for the existing cluster.</p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_site" "blr" { name = "BLR" space_id = var.HPEGL_SPACE } data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } resource hpegl_caas_cluster test { name = "tf-test" blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id site_id = data.hpegl_caas_site.blr.id space_id = var.HPEGL_SPACE } </code></pre> <h1>Upgrade the Kubernetes version</h1> <p>For the Kubernetes version upgrade, you need to specify the new version of Kubernetes that is available for upgrade in the resources block.</p> <ol> <li><strong>kubernetes_version</strong>: Use the Kubernetes version that pops up on the cluster details page in the UI.</li> </ol> <p><img src="/img/blog1.jpg" alt=""></p> <p>Below you can see the reference Terraform configuration for updating the cluster's Kubernetes version.</p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_site" "blr" { name = "BLR" space_id = var.HPEGL_SPACE } data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } resource hpegl_caas_cluster test { name = "tf-test" blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id site_id = data.hpegl_caas_site.blr.id space_id = var.HPEGL_SPACE kubernetes_version = "1.23.13-hpe1" } </code></pre> <h2>Run Terraform plan</h2> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run <strong>terraform plan.</strong></p> <pre><code class="language-shellsession">$ terraform plan hpegl_caas_cluster.test: Refreshing state... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # hpegl_caas_cluster.test will be updated in-place ~ resource "hpegl_caas_cluster" "test" { id = "a32fabb9-7c19-42d1-9a38-ebf122810c0a" ~ kubernetes_version = "1.22.9-hpe1" -> "1.23.13-hpe1" name = "tf-test" # (17 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. </code></pre> <h2>Run Terraform apply</h2> <p>Terraform apply executes the actions proposed in the Terraform plan and updates the resources. Run the command <strong>terraform apply</strong> and type <strong>yes</strong> when asked to <strong>Enter a value</strong>.</p> <pre><code class="language-shellsession">$ terraform apply hpegl_caas_cluster.test: Refreshing state... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # hpegl_caas_cluster.test will be updated in-place ~ resource "hpegl_caas_cluster" "test" { id = "a32fabb9-7c19-42d1-9a38-ebf122810c0a" ~ kubernetes_version = "1.22.9-hpe1" -> "1.23.13-hpe1" name = "tf-test" # (17 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes hpegl_caas_cluster.test: Modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 1m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 3m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 5m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 7m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 9m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 11m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 13m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 15m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 17m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 19m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 21m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 23m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 25m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 27m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 29m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 31m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 33m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 35m10s elapsed] hpegl_caas_cluster.test: Modifications complete after 35m18s [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Apply complete! Resources: 0 added, 1 changed, 0 destroyed. </code></pre> <p><strong>Note:</strong> Kubernetes version update can be performed simultainously with scaling of worker nodes by introducing the worker_nodes block in the terraform configuration file.</p> <h1>Summary</h1> <p>In this blog, I covered how to update Kubernetes clusters with Terraform provider for HPE GreenLake. I showed you how to update the Kubernetes version of the cluster. Also I discussed how update can be performed while scaling a cluster.</p> <p>I hope you found this information interesting and useful while considering the upgrade of Kubernetes cluster with HPE GreenLake Terraform provider. Use the following links to understand more about Terraform and HPE GreenLake Terraform Provider.</p> <ul> <li><a href="https://www.terraform.io/">Learn more about Terraform</a></li> <li><a href="https://www.hpe.com/us/en/greenlake.html">Learn more about HPE GreenLake</a></li> <li><a href="https://registry.terraform.io/providers/HPE/hpegl/latest/docs">Learn more about the HPE GreenLake Terraform provider</a></li> </ul> <p>Don’t forget, you can always find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog/tag/hpe-greenlake">HPE Developer blog</a>.</p>https://developer.hpe.com/federating-spire-on-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/federating-spire-on-hpe-greenlake-for-private-cloud-enterprise/Mon, 15 May 2023 03:37:14 GMT<p>SPIRE is designed to enable widespread deployment of Mutual TLS (mTLS), a method for mutual authentication, between workloads in distributed systems. In our previous <a href="https://developer.hpe.com/blog/integrating-istio-and-spire/">blog </a>post, we explained how you can deploy a Kubernetes cluster on HPE GreenLake for Private Cloud Enterprise and integrate Istio and SPIRE to enable advanced analysis and visualization of the service mesh.</p> <p>In this blog post, we will install and federate SPIRE across two Kubernetes clusters deployed on HPE GreenLake for Private Cloud Enterprise: cluster 1 and cluster 2. We will then show you how to deploy a sample application to verify the federation and visualize the communication across services through a graph.</p> <p><img src="/img/spire-federation.png" alt="SPIRE Federation"></p> <h1>Step 1. Installing SPIRE</h1> <p>Using the QuickStart files provided in this <a href="https://github.com/cxteamtrials/caas-trials-content/tree/main/services/spire/federation">link</a>, get started installing SPIRE on both Clusters. Since there are two clusters in our example, the trust domain configured for the first cluster is <em><strong>cluster 1.demo</strong></em> and the other is <em><strong>cluster2.demo</strong></em>.</p> <p><em>Note: You may configure your own custom trust domains for the clusters by replacing these values across the configuration files.</em></p> <h2>1.1 Clone the repo using the command</h2> <pre><code class="language-shellsession">git clone https://github.com/cxteamtrials/caas-trials-content.git </code></pre> <h2>1.2 Apply the QuickStart file on each cluster using the following commands</h2> <p>As the Kubectl command is required for installation and configuration, please refer to our first <a href="https://developer.hpe.com/blog/integrating-istio-and-spire/">blog </a>post, which explains how to obtain the Kubeconfig file to manage the K8s clusters using Kubectl.</p> <pre><code class="language-shellsession">#for K8s cluster 1: kubectl apply -f services/spire/federation/spire-quickstart-cluster-1.yaml #for K8s cluster 2: kubectl apply -f services/spire/federation/spire-quickstart-cluster-2.yaml </code></pre> <p>This step will install SPIRE into your Kubernetes clusters, along with two additional components: the SPIFFE CSI Driver and the SPIRE Kubernetes Controller manager, which facilitates the registration of workloads and establishment of federated relationships. </p> <p>Verify the installation by checking to see if all the pods are running and that the containers within them are up.</p> <pre><code class="language-shellsession">Cluster1:~ # kubectl get po -n spire NAME READY STATUS RESTARTS AGE spire-agent-92q5m 3/3 Running 0 37d spire-agent-jhgwf 3/3 Running 0 37d spire-agent-sm8gt 3/3 Running 0 37d spire-server-574474c7dc-gbzl6 2/2 Running 1 (11d ago) 37d </code></pre> <pre><code class="language-shellsession">Cluster2:~ # kubectl get po -n spire NAME                             READY   STATUS     RESTARTS       AGE spire-agent-wttmd               3/3     Running   1 (24h ago)   24h spire-server-574474c7dc-2bfcx   2/2     Running   0             24h </code></pre> <h1>Step 2. Installing Istio</h1> <p>On each of your Kubernetes clusters, install Istio and patch Istio ingress gateway. Istio can detect the existence of a UNIX Domain Socket that implements the Envoy SDS API on a defined socket path, allowing Envoy to communicate and fetch identities directly from it. SPIRE can be configured for Istio workloads through an integration with Envoy’s SDS API. </p> <h2>2.1 Download the latest release</h2> <p>You can download the latest release using the official Istio repository or just copy the following command, which would do the same thing for you.</p> <pre><code class="language-shellsession">curl -L https://istio.io/downloadIstio | sh - </code></pre> <p>Change to the Istio directory (cd command), and set the path by using this command:</p> <pre><code class="language-shellsession">cd istio-1.17.1 export PATH=$PWD/bin:$PATH </code></pre> <h2>2.2 Install Istio with custom patch</h2> <p>Install Istio with custom patches for the Ingress-gateway as well as for Istio-proxy.  </p> <p>Get the Istio-spire-config patch using this <a href="https://github.com/cxteamtrials/caas-trials-content/blob/main/services/istio/release-1.17/spire">link</a>, and install that patch using the following commands:</p> <pre><code class="language-shellsession">#For K8s cluster 1: istioctl apply -f services/istio/release-1.17/spire/spire-patch-cluster1.yaml #For K8s cluster 2: istioctl apply -f services/istio/release-1.17/spire/spire-patch-cluster2.yaml </code></pre> <p>Installing Istio with the custom patch will share the spiffe-csi-driver with the Ingress Gateway and the sidecars that are going to be injected on workload pods, granting them access to the SPIRE Agent’s UNIX Domain Socket. </p> <h2>2.3 Patch Istio Ingress Gateway</h2> <h3>2.3.1 Apply SPIFFE ID</h3> <p>First, you must get and apply one of SPIRE controller manager’s <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRD (Custom Resource Definition)</a> ClusterSPIFFEID. The CRD - ClusterSPIFFEID is a cluster-wide resource used to register workloads with SPIRE. The ClusterSPIFFEID can target all workloads in the cluster or can be optionally scoped to specific pods or namespaces via label selectors.  </p> <p>Create a ClusterSPIFFEID CRD to generate registration entries in SPIRE server for all workloads labeled <em><strong>spiffe.io/spire-managed-identity: true.</strong></em></p> <p>Apply the ClusterSPIFFEID used for this demo to both clusters. </p> <pre><code class="language-shellsession">kubectl apply -f services/spire/clusterspiffeid-example.yaml </code></pre> <h3>2.3.2 Patch Ingress Gateway</h3> <p>Now, simply patch the ingress-gateway with <em><strong>spiffe.io/spire managed-identity: true label.</strong></em></p> <p>This patch will register your ingress-gateway pod into the server.</p> <pre><code class="language-shellsession">kubectl patch deployment istio-ingressgateway -n istio-system -p '{"spec":{"template":{"metadata":{"labels":{"spiffe.io/spire-managed-identity": "true"}}}}}' </code></pre> <h3>2.3.3 Check your work</h3> <p>After patching, confirm that your ingress-gateway pod, istiod and all their containers work.</p> <pre><code class="language-shellsession">Cluster1:~ # kubectl get po -n istio-system NAME READY STATUS RESTARTS AGE istio-ingressgateway-5d77cdd9d-gh9w4 1/1 Running 0 37d istiod-d5bc8669c-4bdvh 1/1 Running 0 37d </code></pre> <pre><code class="language-shellsession">Cluster2:~ #  kubectl get po -n istio-system NAME READY STATUS RESTARTS AGE istio-ingressgateway-64bd5ccbbb-kqs2h 1/1 Running 0 37d istiod-d5bc8669c-thbpj 1/1 Running 0 37d </code></pre> <h1>Step 3. Federating SPIRE</h1> <h2>3.1 Expose SPIRE server bundle endpoint</h2> <p>Assign an external IP to your spire-server-bundle-endpoint service on each cluster.  </p> <p>SPIFFE (<em>Secure Production Identity Framework For Everyone</em>) is a specification for implementing identity for workloads, and SPIRE is the code that implements this specification in practice. A SPIFFE bundle is a resource that contains the public key material needed to authenticate credentials from a particular trust domain. A SPIFFE bundle endpoint is a resource (represented by a URL) that serves a copy of a SPIFFE bundle for a trust domain. SPIFFE control planes may both expose and consume these endpoints to transfer bundles between themselves, thereby achieving federation. The SPIRE server is used to host the “spire-server-bundle-endpoint” service that serves the SPIFFE bundle to an external SPIRE agent of a different trust domain.  </p> <p>MetalLB is used to assign the IP for this service. MetalLB hooks into your Kubernetes cluster and provides a network load-balancer implementation. In short, it allows you to create Kubernetes services of type LoadBalancer in clusters that don’t run on a cloud provider, and thus cannot simply hook into paid products to provide load balancers. </p> <p>Follow this <a href="https://metallb.universe.tf/configuration/_advanced_ipaddresspool_configuration/#controlling-automatic-address-allocation">link</a> to set up an automatic address allocation for all services that have type external IP. </p> <p>The configuration should now look something like this:</p> <pre><code class="language-shellsession">Cluster1:~ # kubectl get svc -n spire NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE spire-controller-manager-webhook-service ClusterIP 10.111.48.177 &#x3C;none> 443/TCP 37d spire-server NodePort 10.106.72.102 &#x3C;none> 8081:30256/TCP 37d spire-server-bundle-endpoint LoadBalancer 10.99.0.208 172.16.17.9 8443:30889/TCP 37d </code></pre> <pre><code class="language-shellsession">Cluster2:~ # kubectl get svc -n spire NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE spire-controller-manager-webhook-service ClusterIP 10.97.108.123 &#x3C;none> 443/TCP 37d spire-server NodePort 10.104.109.247 &#x3C;none> 8081:30256/TCP 37d spire-server-bundle-endpoint LoadBalancer 10.104.151.184 172.16.17.3 8443:30889/TCP 37d </code></pre> <h2>3.2 Create cluster federated trust domain </h2> <p>The Cluster Federated Trust Domain CRD is used to federate the clusters with each other.  </p> <p>It requires the following federation configurations: </p> <h3>Cluster Federated Trust Domain:</h3> <p><img src="/img/table1.png" alt="Cluster Federated Trust Domain"></p> <h3>Bundle Endpoint:</h3> <p><img src="/img/table2.png" alt="Bundle Endpoint"></p> <p>The sample CRD’s can be applied to each cluster.</p> <pre><code class="language-shellsession">#For K8s cluster 1: kubectl apply -f services/spire/federation/clusterfederatedtrustdomain-cluster1.yaml #For K8s cluster 2: kubectl apply -f services/spire/federation/clusterfederatedtrustdomain-cluster2.yaml </code></pre> <p>The Bundle endpoint URL and the TrustDomainBundle fields must be edited to be configured to the specifications of your cluster. </p> <p>Using the previous step, the Bundle endpoint URL can be obtained. To obtain the trust domain bundle, edit the clusters as follows: </p> <pre><code class="language-shellsession">Cluster1:~ # kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server bundle show -format spiffe > cluster1.bundle Cluster2:~ # kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server bundle show -format spiffe > cluster2.bundle </code></pre> <p>Doing a cat command of the bundles reveals the keys to be copied into the trust domain bundle field.  </p> <p>After the Bundle endpoint URL and the TrustDomainBundle fields are configured and applied, check the status of the SPIRE pods to make sure they are running and check the logs of the spire-server to verify successful federation.</p> <pre><code class="language-shellsession">Cluster1:~ # kubectl logs -n spire -c spire-server spire-server-574474c7dc-gbzl6 </code></pre> <p><img src="/img/table3.png" alt="Cluster-1 Logs"></p> <pre><code class="language-shellsession">Cluster2:~ # kubectl logs -n spire -c spire-server spire-server-574474c7dc-2bfcx </code></pre> <p><img src="/img/table4.png" alt="Cluster-2 Logs"></p> <h1>Step 4. Deploying a sample application</h1> <p>Now that SPIRE is federated and communication across clusters can be facilitated, here's how you can deploy a sample application that verifies this functionality.   </p> <h2>4.1 Deploy a resource in Cluster-1</h2> <p>In Cluster 1, apply a new ClusterSpiffeID called <em><strong>curl-greeter</strong></em> that registers resources with the label <strong>spiffe.io/spire-managed-identity=curl-greeter</strong> that can be federated with cluster2. Create a resource called <em><strong>curl-greeter</strong></em> that has the label: <strong>spiffe.io/spire-managed-identity=curl-greeter</strong> and annotation: <strong>inject.istio.io/templates=sidecar, spire</strong> </p> <pre><code class="language-shellsession">#Apply SPIFFEID kubectl apply -f /services/spire/federation/clusterspiffeid-curl-greeter-cluster1.yaml #Create Curl-Greeter Resource kubectl run curl-greeter --image=radial/busyboxplus:curl --labels="spiffe.io/spire-managed-identity=curl-greeter" --overrides='{ "apiVersion": "v1", "spec": { "template": {"metadata": {"annotations": { "inject.istio.io/templates":"sidecar,spire" } } }}}' -i --tty </code></pre> <h2>4.2 Deploy Bookinfo Sample Application in Cluster-2</h2> <p>In Cluster 2, apply a new ClusterSpiffeID called <em><strong>federated</strong></em> that registers resources with the label <strong>spiffe.io/spire-managed-identity=spire</strong> that can be federated with cluster1. Then apply the bookinfo sample application manifest.</p> <pre><code class="language-shellsession">#Apply SPIFFEID kubectl apply -f /services/spire/federation/clusterspiffeid-federated-cluster2.yaml #Apply Bookinfo Manifest kubectl apply -f services/istio/release-1.17/bookinfo.yaml </code></pre> <h2>4.3 Check if all the resources created are up and running</h2> <pre><code class="language-shellsession">Cluster1:~ # kubectl get po NAME                              READY   STATUS    RESTARTS   AGE curl-greeter                      2/2     Running   0          15h </code></pre> <pre><code class="language-shellsession">Cluster2:~ # kubectl get po NAME                               READY   STATUS    RESTARTS   AGE details-v1-bff8759df-vkvb4       2/2     Running   0          16h greeter-client-76686757cd-6j2ft   2/2     Running   0          21h productpage-v1-98887b9b-x5k24     2/2     Running   0          16h ratings-v1-7ddbb859fc-htmfq       2/2     Running   0          16h reviews-v1-67b576c8bf-jr6tj       2/2     Running   0          16h reviews-v2-7ffbdcc5f7-m2c29       2/2     Running   0          16h reviews-v3-6dbfcc6d89-zn9tw       2/2     Running   0          16h </code></pre> <h2>4.4 Check that the workload identity was issued by SPIRE</h2> <p>SPIRE certificate at Curl-Greeter </p> <pre><code class="language-shellsession">Cluster1:~ # istioctl proxy-config secret curl-greeter -o json | jq -r \ > '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem Cluster1:~ # openssl x509 -in chain.pem -text | grep SPIRE         Subject: C = US, O = SPIRE, x500UniqueIdentifier = c3dd29b5f4a326f8a567854407456ea9 </code></pre> <p><em>Note: Similarly, certificates for other pods can be checked.</em> </p> <h2>4.5 Create a secret</h2> <p>Create a secret to hold bookinfo product page credentials like this: </p> <pre><code class="language-shellsession">kubectl create -n istio-system secret generic bookinfo-credential \  \--from-file=tls.key=bookinfo.com.key \  \--from-file=tls.crt=bookinfo.com.crt \  \--from-file=ca.crt=bookinfo.ca.crt </code></pre> <p>Obtain the tls cert and key using this command: </p> <pre><code class="language-shellsession">kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server x509 mint -spiffeID spiffe://cluster2.demo/ns/default/sa/bookinfo-productpage </code></pre> <p><em>Note: Copy the SVID section into a new file bookinfo.com.crt and Private key section into bookinfo.com.key.</em> </p> <p>Obtain the ca cert using this command: </p> <pre><code class="language-shellsession">kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server bundle list </code></pre> <p><em>Note: Copy the ca cert under the cluster1.demo section into a new file bookinfo.ca.cert.</em> </p> <p>Apply the gateway configuration for the bookinfo application found <a href="https://github.com/cxteamtrials/caas-trials-content/blob/main/services/istio/release-1.17/samples/bookinfo-gateway.yaml">here</a>: (It uses the istio-ingress gateway) </p> <pre><code class="language-shellsession">kubectl apply -f services/istio/release-1.17/bookinto-gateway.yaml </code></pre> <h2>4.6 Generate traffic to the sample application on cluster2 from the curl greeter on cluster 1</h2> <p>Curl Command at Cluster 1 (IP addr: istio-ingress-gateway external ip) </p> <pre><code class="language-shellsession">Cluster1:~ # curl -v -HHost:bookinfo.com --resolve "bookinfo.com:443:172.16.17.2"   --cacert bookinfo.ca.crt --cert svid.pem --key key.pem   "https://bookinfo.com:443/productpage" -k * Added bookinfo.com:443:172.16.17.2 to DNS cache * Hostname bookinfo.com was found in DNS cache *   Trying 172.16.17.2:443... * TCP_NODELAY set * Connected to bookinfo.com (172.16.17.2) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: *   CAfile: bookinfo.ca.crt   CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Request CERT (13): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Certificate (11): * TLSv1.3 (OUT), TLS handshake, CERT verify (15): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: *  subject: C=US; O=SPIRE; x500UniqueIdentifier=a09f8093609833482827d697f2719205 *  start date: May  9 08:45:18 2023 GMT *  expire date: May  9 09:45:28 2023 GMT *  issuer: C=US; O=SPIFFE *  SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5569570e8050) > GET /productpage HTTP/2 > Host:bookinfo.com > User-Agent: curl/7.66.0 > Accept: */* > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * old SSL session ID is stale, removing * Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)! &#x3C; HTTP/2 200 &#x3C; content-type: text/html; charset=utf-8 &#x3C; content-length: 4294 &#x3C; server: istio-envoy &#x3C; date: Tue, 09 May 2023 08:49:40 GMT &#x3C; x-envoy-upstream-service-time: 24 </code></pre> <p>Obtain cert and key file using: </p> <pre><code class="language-shellsession">kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server x509 mint -spiffeID spiffe://cluster.1demo/curl-greeter </code></pre> <p><em>Note: Copy the SVID section into a new file svid.pem and Private key section into key.pem.</em> </p> <p>Obtain ca cert using: </p> <pre><code class="language-shellsession">kubectl exec -n spire -c spire-server deployment/spire-server -- /opt/spire/bin/spire-server bundle list </code></pre> <p><em>Note: Copy the ca cert under the cluster2.demo section into a new file bookinfo.ca.cert.</em> </p> <h2>4.7 Visualize using Service Mesh</h2> <p>Using the Kiali dashboard, observe the graphs of generated traffic. </p> <p>The graph below shows services communication, and the locks symbolize mTls protocol. </p> <p><img src="/img/mtls.png" alt="mTLS Graph"></p> <h1>Summary</h1> <p>The goal of this blog post was to guide you through federating SPIRE across two Kubernetes clusters deployed on HPE GreenLake for Private Cloud Enterprise. It shows you how to do this by creating a cluster federated trust domain and federated ClusterSpiffeIDs for your sample application workloads and then helps you visualize your service mesh through Kiali Dashboard.</p>https://developer.hpe.com/2023-May-03/https://developer.hpe.com/2023-May-03/Fri, 05 May 2023 13:00:00 GMThttps://developer.hpe.com/netcdf-in-chapel-part-2-reading-a-dataset-in-parallel/https://developer.hpe.com/netcdf-in-chapel-part-2-reading-a-dataset-in-parallel/Wed, 03 May 2023 16:50:00 GMT<p>Placeholder</p>https://developer.hpe.com/netcdf-in-chapel-part-1-interfacing-with-the-c-library/https://developer.hpe.com/netcdf-in-chapel-part-1-interfacing-with-the-c-library/Thu, 27 Apr 2023 05:31:00 GMT<p>External blog</p>https://developer.hpe.com/new-powershell-library-for-the-hpe-greenlake-cloud-platform/https://developer.hpe.com/new-powershell-library-for-the-hpe-greenlake-cloud-platform/Wed, 26 Apr 2023 11:33:07 GMT<style>ul li{ font-size:28px;padding-bottom: 10px; line-height: 1.2}</style> <style>ol li{ font-size:28px;padding-bottom: 0.5em; line-height: 1.2}</style> <style> img { max-width: 100%; height: auto; border: 1px solid #ccc; margin: 20px; box-shadow: 2px 2px 5px #ccc; } </style> <p>The purpose of this blog is to familiarize readers with the recently released PowerShell library for the HPE GreenLake edge-to-cloud platform. This library allows PowerShell developers, IT automation experts, and DevOps professionals to use the HPE GreenLake platform's API without having to rely on the Graphical User Interface (GUI).</p> <p>With the introduction of this new library, anyone with basic PowerShell skills can now automate their interaction with the HPE GreenLake API, leverage the many resources offered by the HPE GreenLake platform, and enjoy increased productivity and efficiency.</p> <p>Are you seeking agility and the ability to automate routine tasks through HPE GreenLake? If so, this library is the ideal solution for you!</p> <p>Among many other operations, this library allows you to create users, assign roles, send invitations to users, add and archive devices, assign applications and attach subscriptions, add tags to any device, and get all kinds of information about any HPE GreenLake resource.</p> <p>In addition, you can implement resource restriction policies, fully automate device onboarding, generate API credentials for specific application instances (such as HPE Compute Ops Management or HPE Data Services Cloud Console), and extend your API calls to any HPE GreenLake application instance on the fly.</p> <p>The HPE GreenLake platform provides a shared set of common cloud services for different application instances and services. This common service experience brings all HPE service offerings together into a single, comprehensive customer experience that this library leverages.</p> <p><img src="/img/ccs.png" alt=""></p> <p>HPE GreenLake CCS, short for HPE GreenLake Common Cloud Services, offers a collection of API-enabled services that serve various functions, with the primary ones being:</p> <ul> <li><strong>Registration</strong>: in charge of the user registration. Includes email verification, account creation with HPE GreenLake platform's Identity Provider (IdP) including integration with Ping Identity as the OpenID Connect (OIDC) Relying Party (RP) provider for user authenticity verification and token issuance</li> <li><strong>Authentication (AuthN)</strong>: takes care of the user authentication and HPE GreenLake CCS to applications authentication. Includes unified login, Single or Multi-factor authentication (MFA) or Single sign-on (SSO) with third party, federated authentication with customer’s identity DP, single logout, user management, and supports increased security with PKCE (Proof Key for Code Exchange, pronounced ‘pixy”) for OAuth 2.0</li> <li><strong>Authorization (AuthZ)</strong>: provides authorization service for HPE GreenLake CCS. Includes unified Role-Based Access Control (RBAC) for users, custom roles and Resource Restriction Policy (RRP) including role creation, resource assignment to a role, role assignment to user, etc.</li> <li><strong>Device activation and inventory</strong>: provides Zero Touch Provisioning (ZTP) and Asset inventory (contract and customer order processing). Includes device firmware management (firmware repository for resources, latest FW check, FW upgrade, baseline upgrade)</li> <li><strong>Subscription management</strong>: offers subscription inventory, support for different consumption models</li> </ul> <p><img src="/img/glcp2.png" alt=""></p> <p>The majority of these essential functions can be performed effortlessly and with great efficiency using the HPE GreenLake PowerShell library.</p> <p>PowerShell offers numerous benefits, such as flexibility and ease of learning, and with this new library, you can significantly boost your productivity and be agile while reducing the risk of manual errors. Most, if not all, of the HPE GreenLake tasks can be automated in a smarter way through its use, making it a must-have tool for any IT automation engineer or DevOps personnel.</p> <h1>Introduction to the HPE GreenLake PowerShell library</h1> <p>The new HPE GreenLake PowerShell library can be found on the Hewlett Packard Enterprise GitHub repository at the following <a href="https://github.com/HewlettPackard/POSH-HPEGreenLake">location</a>.</p> <p><img src="/img/github_glcp.png" alt=""></p> <p>The supported PowerShell editions for this new library are Desktop (with 5.1 and above) and Core (with 7.x supported on Windows, Linux and Mac).</p> <p>This module is also published in the <a href="https://www.powershellgallery.com/packages/HPEGreenLake">PowerShell Gallery</a>. The PowerShell Gallery is a repository for sharing and distributing PowerShell modules and scripts. It's a community-driven platform that provides access to various PowerShell resources, enabling you to easily discover, install, and publish your own PowerShell content. The PowerShell Gallery can be accessed through the PowerShellGet module, which comes pre-installed with Windows PowerShell 5.0 and above.</p> <p>In most GitHub repositories, the <strong>README.md</strong> file is a crucial file that provides essential information about the project. It typically contains instructions on how to install and use the module, as well as any other important details that potential users or contributors may need to know. Other files and folders in the repository include source code, documentation, configuration files, samples, and more. These files are organized into different directories to help keep things organized and easy to find. As mentioned, the <strong>README.md</strong> file is an excellent starting point for getting familiar with this new module.</p> <p>The <strong>Samples</strong> folder provides several scripts and csv file examples to demonstrate how this library can be best used. These examples illustrate the variety of functionality in the library, including connecting to the platform, onboarding devices, generating API credentials, interacting with iLO, etc.</p> <p>The <strong><a href="https://github.com/HewlettPackard/POSH-HPEGreenLake/issues">Issues</a></strong> tab is a feature commonly found on websites that use version control systems such as Git, GitHub, or Bitbucket. It allows users to report issues, request new features, or provide feedback related to a project. When users click on one of the <strong>Get Started</strong> buttons, they are redirected to a form where they can enter details about their issue or suggestion. This information is then stored as a new issue in the project's issue tracker, where developers can see it and take action accordingly. The <strong>Issues</strong> tab is a valuable tool for effective communication between developers and users, allowing for a more transparent and collaborative development process.</p> <p><img src="/img/github_-glcp_issues.png" alt=""></p> <h1>Getting started with the HPE GreenLake PowerShell library</h1> <p><strong>Install-Module</strong> cmdlet is a common way to install PowerShell modules from online repositories. The cmdlet downloads and installs the module and any associated dependencies that it may have. To use the <strong>Install-Module</strong> cmdlet, you can simply open a PowerShell console (or a PowerShell ISE console or VS Code), and run the following command:</p> <pre><code class="language-powershell">> Install-Module HPEGreenLake </code></pre> <p>This will download and install the module from the official PowerShell Gallery repository. If this is your first time installing a module from the PowerShell Gallery, it will ask you to confirm whether you trust the repository or not. You can type <strong>Y</strong> and press <strong>Enter</strong> to continue with the installation.</p> <blockquote> <p><strong>Note</strong>: You must have an internet connection to install the module from the PowerShell Gallery.</p> </blockquote> <blockquote> <p><strong>Note</strong>: At the time of writing this blog, the HPE GreenLake PowerShell library has no dependencies. In other words, the library does not require the installation of any other software or modules to function properly.</p> </blockquote> <p>There could be several issues you may encounter while using the <strong>Install-Module</strong> cmdlet in PowerShell, some of which are:</p> <ul> <li><strong>Insufficient permissions</strong>: You may need administrative privileges to install modules. If you do not have sufficient privileges, you can run your PowerShell client as an administrator or use: <strong>Install-Module HPEGreenLake -Scope CurrentUser</strong></li> <li><strong>Blocked security protocols</strong>: Sometimes, the security protocols built into PowerShell can prevent the installation process. This usually happens when the PowerShell execution policy is set to "Restricted". If <strong>Get-ExecutionPolicy</strong> shows Restricted, you may need to run <strong>Set-ExecutionPolicy RemoteSigned</strong></li> </ul> <p>To find all cmdlets in a module that can be used with a specific resource, you can use the <strong>Get-Command</strong> cmdlet along with the <strong>-Module</strong> parameter to specify the name of the module.</p> <pre><code class="language-powershell">> Get-Command -Module HPEGreenLake </code></pre> <p>In this first release, about 50 cmdlets are available with the HPE GreenLake module. The table below describes the main resources and features that are supported:</p> <p><img src="/img/lj-key-supported-resources.png" alt=""></p> <p>In PowerShell, cmdlet names are constructed using a verb-noun format. The verb describes the action that the cmdlet performs (Get, Set, Remove, Invoke, etc.), and the noun specifies the object that the cmdlet acts upon. For example, the <strong>Get-HPEGLUserRole</strong> retrieves information about the user role in the HPE GreenLake platform. Note that object resource with the HPE GreenLake library always starts with <strong>HPEGL&#x3C;resource_name></strong> (GL for GreenLake).</p> <p><strong>Get-Help</strong> is an important cmdlet in PowerShell as it provides detailed information about a specific cmdlet or function. To get the full help details for the <strong>Get-HPEGLUserRole</strong> cmdlet, you can use the following command:</p> <pre><code class="language-powershell">> Get-Help Get-HPEGLUserRole -Full </code></pre> <p>To view the detailed examples of how to use a particular cmdlet, you can use the <strong>Get-Help</strong> cmdlet along with the <strong>-Examples</strong> parameter followed by the name of the cmdlet. Here's an example command that you can use to get the examples of <strong>Get-HPEGLUserRole</strong> cmdlet:</p> <pre><code class="language-powershell">> Get-Help Get-HPEGLUserRole -Examples </code></pre> <p>This will display all the available examples for the <strong>Get-HPEGLUserRole</strong> cmdlet in a list format. You can review the examples and use them according to your requirements.</p> <h1>Connection to the HPE GreenLake platform</h1> <p>The connection to the HPE GreenLake platform is done using the <strong>Connect-HPEGL</strong> cmdlet.</p> <blockquote> <p><strong>Note</strong>: The library currently only supports single-factor authentication. Multi-factor authentication (MFA) and SAML Single Sign-On are not supported.</p> </blockquote> <blockquote> <p><strong>Note</strong>: This limitation means that users who use SAML single sign-on with the HPE GreenLake platform (this applies to all HPE employees) cannot use their corporate email credentials when logging in via the <strong>Connect-HPEGL</strong> cmdlet.</p> </blockquote> <blockquote> <p><strong>Note</strong>: While waiting for SAML Single Sign-On support, the temporary solution is to add a secondary email into your HPE GreenLake account. Just go to the HPE GreenLake GUI and use the <strong>Invite Users</strong> card in <strong>Manage</strong> / <strong>Identity &#x26; Access</strong> to send an invitation to a non-corporate email address. Once you receive the email, accept the invitation and you will be directed to the HPE GreenLake interface where you can set a password. Once this is done, you can use this email address and password to log in with <strong>Connect-HPEGL</strong>.</p> </blockquote> <blockquote> <p><strong>Note</strong>: To interact with the HPE GreenLake platform through this library, you must possess at least the <em><strong>Observer</strong></em> built-in role in the <em><strong>HPE GreenLake platform</strong></em> application. This role only grants view privileges. However, if you need to make modifications, then either the <em><strong>Operator</strong></em> (with view and edit privileges) or the <em><strong>Administrator</strong></em> (with view, edit, and delete privileges) built-in roles are necessary. If none of these built-in roles are suitable, you can also create your own custom role that meets your role-based access control requirements.</p> </blockquote> <p>To begin with, it is recommended that you create a credentials object that includes your HPE GreenLake user's email and password:</p> <pre><code class="language-powershell">> $GLCP_userName = "Username@domain.com" > $GLCP_password = "xxxxxxxxxxxxxxxx" > $secpasswd = ConvertTo-SecureString -String $GLCP_password -AsPlainText -Force > $credentials = New-Object System.Management.Automation.PSCredential ($GLCP_userName, $secpasswd) </code></pre> <p>Then using the <strong>Connect-HPEGL</strong> cmdlet, you can connect to the platform:</p> <pre><code class="language-powershell">> Connect-HPEGL -Credential $credentials </code></pre> <p>If you have multiple company accounts, you can add the <strong>-CompanyName</strong> parameter to connect to the appropriate company account as follows:</p> <pre><code class="language-powershell">> Connect-HPEGL -Credential $credentials -CompanyName "HPE Mougins" </code></pre> <p>After successfully authenticating to the HPE GreenLake platform, the <em>[HPEGreenLake.Connection]</em> object is returned to the caller and (at the same time) is added to the global session tracker <strong>$HPEGreenLakeSession</strong>.</p> <p><img src="/img/lj-picture10.png" alt=""></p> <p>To display the full content of this global variable in the console, use:</p> <pre><code class="language-powershell">> $HPEGreenLakeSession | format-List </code></pre> <p><img src="/img/lj-picture6.png" alt=""></p> <p>This object contains the following properties:</p> <ul> <li><strong>session</strong>: Web session object containing information about the HPE GreenLake session, including cookies and credentials</li> <li><strong>oauth2AccessToken</strong>: OAuth2 access token string returned by the API after successful authentication</li> <li><strong>oauth2IdToken</strong>: OAuth2 ID token string returned by the API after successful authentication</li> <li><strong>oauth2RefreshToken</strong>: OAuth2 refresh token string returned by the API after successful authentication</li> <li><strong>userName</strong>: Email address that was authenticated with HPE GreenLake</li> <li><strong>customerId</strong>: HPE GreenLake customer ID</li> <li><strong>companyName</strong>: Name of the company account</li> <li><strong>oauth2TokenCreation</strong>: OAuth2 token creation datetime value</li> <li><strong>oauth2TokenCreationEpoch</strong>: Unix time since creation of the OAuth2 token</li> <li><strong>userSessionIdleTimeout</strong>: HPE GreenLake user session timeout in minutes</li> <li><strong>apiCredentials</strong>: Collection of application API credentials created during the session</li> </ul> <p>Each API credential object contains the following properties:</p> <ul> <li><strong>credential_name</strong>: Name of the API credential</li> <li><strong>application_name</strong>: Name of the application using this credential</li> <li><strong>ccs_region</strong>: Region of the application using this credential</li> <li><strong>application_instance_id</strong>: Instance ID of the application using this credential</li> <li><strong>client_id</strong>: Client ID of the API credential</li> <li><strong>client_secret</strong>: Client Secret of the API credential</li> <li><strong>connectivity_endpoint</strong>: Connectivity endpoint of the application instance</li> </ul> <p>The <strong>apiCredentials</strong> property is only filled in when using <strong>New-HPEGLAPIcredential</strong> during a session.</p> <p>All properties in this object are important. <strong>Session</strong> stores what the library uses to make all the calls in the cmdlets. You can open a session using:</p> <pre><code class="language-powershell">> $HPEGreenLakeSession.session </code></pre> <p>The output is as follows:</p> <p><img src="/img/lj-picture7.png" alt=""></p> <p>Note that in the headers, an <strong>Authorization, Bearer &#x3C;token></strong> header is defined.</p> <p>This token is a JSON Web Token (JWT). Pronounced "Jot", this type of token is used in the OAuth 2.0 standard for authentication and authorization purposes with the HPE GreenLake platform. OAuth 2.0 is a secure and standardized authorization framework designed to enable third-party applications to access resources and data without the need for the user to reveal their login credentials to the third-party application or API.</p> <p>The JWT typically contains data about the token (such as its type and signature algorithm), statements about the user (such as their ID, name, email address, or role) and an expiration time. Finally, it includes a signature that is used to verify that the token has not been altered during transmission and that it was issued by a trusted source.</p> <p>It's important to keep in mind that the access token will only be valid for a period of 2 hours, after which it will expire and no longer be usable. Note that the library has a built-in function that runs in the background to refresh the access token before it expires. The function is triggered whenever a cmdlet is executed. However, if your session has already expired due to prolonged inactivity (defined by the session timeout in the HPE GreenLake user preferences) you will need to reconnect to the platform with <strong>Connect-HPEGL</strong>.</p> <p>The <strong>Connect-HPEGL</strong> cmdlet handles the entire authorization process, which includes authorizing with the third-party identity provider and collecting various tokens. Once all the necessary tokens have been obtained, a session is created by utilizing bearer authentication with the HPE GreenLake platform Authentication (AuthN) service API using the <em>id_token</em> present in the payload to confirm your identity.</p> <p>When the session is created, CCS sets a unique session ID in a cookie that the library stores in <strong>$HPEGreenLakeSession.session.cookies</strong> so that all subsequent requests can use it. Other cookies are also added to maintain the session state between the PowerShell session and the AuthN service API. For each request from the HPE GreenLake library, these cookies are sent back to the CCS API, which allows the cloud platform to identify the user and grant various permissions.</p> <h1>Interaction with the HPE GreenLake API</h1> <h2>Main Cmdlets for roles, permissions, and restrictions</h2> <p>In HPE GreenLake, you have the ability to manage users and their permissions to services and resources within your account. The management of these permissions is done through Roles which are composed of a set of permissions that provide access to users in your HPE GreenLake account.</p> <p>Additionally, you can restrict access to specific resources by creating custom resource groupings using Resource Restriction Policies.</p> <p>The main cmdlets for managing users and their access to HPE GreenLake resources are as follows:</p> <ul> <li> <p>To manage users:</p> <ul> <li><strong>Get-HPEGLUser</strong>: to get users and their activity status and roles</li> <li><strong>Send-HPEGLUserInvitation</strong>: to invite users to your account</li> <li><strong>Remove-HPEGLUser</strong>: to delete users from your account</li> </ul> </li> <li> <p>To view and manage user roles and permissions:</p> <ul> <li><strong>Get-HPEGLRole</strong>: to view the role (= group of permissions) that you can specify and assign to users in your HPE GreenLake account</li> <li><strong>Get-HPEGLUserRole</strong>: to view user roles and permissions</li> <li><strong>Set-HPEGLUserRole</strong>: to set user roles and permissions</li> <li><strong>Remove-HPEGLUserRole</strong>: to remove user roles and permissions</li> </ul> </li> <li> <p>To define Resource Restriction Policies and manage access to resources:</p> <ul> <li><strong>Get-HPEGLResourceRestrictionPolicy</strong>: to view resource restriction policies in your HPE GreenLake account</li> <li><strong>New-HPEGLResourceRestrictionPolicy</strong>: to set resource restriction policies in your HPE GreenLake account</li> <li><strong>Remove-HPEGLResourceRestrictionPolicy</strong>: to remove resource restriction policies in your HPE GreenLake account</li> </ul> </li> </ul> <h2>Managing devices with the HPE GreenLake PowerShell library</h2> <p>In HPE GreenLake, devices are the resource for compute, network and storage devices that you can onboard on the platform in your HPE GreenLake account. Once onboarded, devices can be viewed, assigned to an application instance, applied to a subscription, and tagged.</p> <p>In the library, the main cmdlet to get information on devices is <strong>Get-HPEGLDevice</strong>.</p> <pre><code class="language-powershell">> Get-HPEGLdevice </code></pre> <p>As in the GUI, the bare cmdlet (without parameters) returns a subset of resources in a page. A maximum of 100 devices are displayed by default. The <strong>-Nolimit</strong> parameter can be used to display all available devices, but this may result in a longer response time.</p> <p>The cmdlets in this library usually generate formatted objects when they are displayed on the console to enhance readability and ease of understanding. As an example, if you enter:</p> <pre><code class="language-powershell">> Get-HPEGLdevice -Limit 10 </code></pre> <p><img src="/img/lj-picture8.png" alt=""></p> <p>The generated output is "formatted". To get the full view of a formatted object in PowerShell, you can use the <strong>Format-List</strong> cmdlet (or <strong>fl</strong> its alias). This cmdlet allows you to display all the properties and values of an object in a list form:</p> <pre><code class="language-powershell">> Get-HPEGLdevice -Limit 10 | fl </code></pre> <p><img src="/img/lj-picture9.png" alt=""></p> <p>Several other parameters are available such as Tags, Archived, Stats, Devicetype, Serialnumber, etc. You can use the help to get the complete list, and try them out at your convenience.</p> <h2>Adding devices to the HPE GreenLake platform</h2> <p>To manage devices using HPE GreenLake, one of the initial steps is to onboard the device onto the platform. When using the GUI, this step is typically done manually. One of the major advantages of using a library that supports automation is the ability to streamline and fully automate processes such as device onboarding. With automation, you can significantly increase efficiency, reduce errors, and save time and resources that would otherwise be spent on manual tasks.</p> <p>The process of onboarding devices on the platform actually requires several steps that must first be performed on the HPE GreenLake platform side. Here are the different steps and commands that need to be executed:</p> <p><img src="/img/lj-device-onboarding.png" alt=""></p> <p><strong>Add-HPEGLDeviceCompute</strong> is the first cmdlet that you can use to simply add a Compute device to the HPE GreenLake console. The sole purpose of this cmdlet is to add compute(s) with the specified tags to the platform and nothing beyond that. The corresponding cmdlets for adding storage and network devices are <strong>Add-HPEGLDeviceStorage</strong> and <strong>Add-HPEGLDeviceNetwork</strong>.</p> <p>It is worth noting that a CSV file can be utilized to add multiple devices to the platform by containing device information such as serial number, part number, subscription key, MAC address and tags depending on the type of device you wish to add. You can use this CSV file as an input for the pipeline of these cmdlets, as in this example with <strong>Add-HPEGLDeviceCompute</strong>:</p> <pre><code class="language-powershell">> Import-Csv Compute_Devices.csv | Add-HPEGLDeviceCompute </code></pre> <p>In this case, the content of the csv file to define the Compute must use the following format:</p> <p><em>SerialNumber, PartNumber, Tags</em><br> <em>WGX2380BLC, P55181-B21, Country=US State=PACA App=RH</em><br> <em>AZX2380BLD, P55182-B21, State=Texas Role=production</em><br> <em>7LKY23D9LM, P54277-B21</em></p> <p>Note that for 7LKY23D9LM, no tag is assigned in this example.</p> <p>Tags are optional with Compute but highly recommended. They are particularly useful when creating resource restriction policies. They must meet the string format: "<strong>&#x3C;Name>=&#x3C;Value> &#x3C;Name>=&#x3C;Value></strong>" such as "<strong>Country=US State=TX App=Grafana</strong>" or "<strong>Country=US</strong>".</p> <p><strong>Add-HPEGLDeviceComputeFullService</strong> is a much more advanced cmdlet than the previous one. This specific command has the ability to perform all mandatory steps of Compute onboarding, including both the HPE GreenLake platform side and server side procedures, as described in the following diagram:</p> <p><img src="/img/lj-device-onboarding-full-service.png" alt=""></p> <p>A single command is sufficient to implement the entire onboarding process. The steps include:</p> <ol> <li>Connect to HPE GreenLake using the default account or the one provided using the <strong>-GLCompanyName</strong> parameter.</li> <li>Set the automatic assignment of subscriptions with the first Compute Enhanced subscription found that has not expired and has available subscription seats.</li> <li>Onboard each device to the HPE GreenLake platform account.</li> <li>Set optional server tags if defined with the <strong>-Tags</strong> parameter.</li> <li>Associate each device with the defined application instance (subscription is automatically assigned to each device as per auto assignment configuration).</li> <li>Set each iLO to use a web proxy if defined via the different web proxy parameters.</li> <li>Connect each iLO to the HPE Compute Ops Management instance.</li> </ol> <blockquote> <p><strong>Note</strong>: This cmdlet is involved in the configuration of iLOs so the prerequisite for running this cmdlet is to have network access to both the Internet (to access the HPE GreenLake platform) and the iLO network (which is usually a private network).</p> </blockquote> <p>This time, the content of the csv file should contain information regarding iLOs which includes their IP addresses, login credentials (an administrator account is a requisite), and tags (if applicable). This is the only data needed to feed the pipeline. The format should be as follows:</p> <p><em>IP, Username, Password, Tags</em><br> <em>192.168.3.193, Administrator, password, Country=FR City=Mougins App=InfluxDB Owner=LJ</em><br> <em>192.168.3.191, Administrator, password, Country=US State=Texas Role=Production Owner=LJ</em><br> <em>192.168.3.205, demo, password</em></p> <p>All Compute data that is required by the HPE GreenLake platform to complete the Compute onboarding is collected and managed by the cmdlet in the background using RedFish iLO calls.</p> <p>You can use <strong>Get-Help</strong> to learn more about this cmdlet and particularly look at the different examples:</p> <pre><code class="language-powershell">> Help Add-HPEGLDeviceComputeFullService -full </code></pre> <p><strong>Add-HPEGLDeviceStorageFullService</strong> and <strong>Add-HPEGLDeviceNetworkFullService</strong> offer the same full onboarding service for storage and networking devices.</p> <p>Overall, the ability to fully automate device onboarding is a significant advantage that can lead to many benefits for organizations of all sizes.</p> <p>Note that there are <a href="https://github.com/HewlettPackard/POSH-HPEGreenLake/tree/master/Samples">samples</a> available on GitHub repository that can help in learning how to use the full device onboarding cmdlets and how to build a csv file for each device type.</p> <h2>Managing applications with the HPE GreenLake PowerShell library</h2> <p>The main cmdlets for managing applications in the HPE GreenLake platform are the followings:</p> <ul> <li> <p>To manage applications:</p> <ul> <li><strong>Get-HPEGLApplication</strong>: to get all applications available to you on your HPE GreenLake platform, including the ones that are provisioned into your account and the ones that are not</li> </ul> </li> <li> <p>To add and remove applications:</p> <ul> <li><strong>Add-HPEGLApplication</strong>: to provision an application in a new region</li> </ul> <blockquote> <p><strong>Note</strong>: A HPE GreenLake region refers to the geographical location where the HPE GreenLake services are hosted and provided from. This can vary depending on the customer's location and which HPE GreenLake application you are using. Customers can choose the region that best suits their needs in terms of location, availability, and compliance requirements.</p> </blockquote> <ul> <li><strong>Remove-HPEGLApplication</strong>: to delete an application instance</li> </ul> <blockquote> <p><strong>Note</strong>: This cmdlet has a high-impact and irreversible action. This action permanently deletes all data of the application instance and cannot be undone once the process has started. For example, removing the Compute Ops Management US-West instance would remove all user data, all devices, all server settings, all server groups, all API credentials, etc. It's great to hear that this cmdlet has a confirmation prompt before deleting an application instance. This feature can help prevent accidental deletions and provide an extra layer of protection for critical data in applications.</p> </blockquote> </li> <li> <p>To assign and unassign devices to applications:</p> <ul> <li><strong>Set-HPEGLDeviceApplication</strong>: to attach devices to an application instance</li> </ul> <blockquote> <p><strong>Note</strong>: Assigning devices to an application instance is the process of attaching devices to an application in a region, so that these devices become visible and managed in the application instance by users.</p> </blockquote> </li> <li> <p>To create and delete API application credentials:</p> <ul> <li><strong>New-HPEGLAPIcredential</strong>: to create an API credential for an application instance</li> </ul> <blockquote> <p><strong>Note</strong>: With the HPE GreenLake platform, developers can make API calls on any application instance as long as they have the API credentials, which consist of a client ID, client secret and connectivity endpoint.</p> <p><strong>Note</strong>: When API credentials are created, they are automatically stored in the global variable <strong>$HPEGreenLakeSession.apiCredentials</strong>. This global variable is accessible as long as the PowerShell console is active and <strong>Disconnect-HPEGL</strong> has not been used. In other words, as long as your session is active. This data is sensitive because it contains all you need to make API calls with an application API such as Compute Ops Management, Data Services Cloud Console, etc. To get a more complete example of how to deeply interact with Compute Ops Management, see <a href="https://github.com/HewlettPackard/POSH-HPEGreenLake/blob/master/Samples/Interaction-with-COM_Sample.ps1">Interaction-with-COM_Sample</a> script sample.</p> <p><strong>Note</strong>: To store the API credentials beyond the duration of the session, the cmdlet provides a <strong>-Location</strong> parameter. This parameter can be used with <strong>New-HPEGLAPIcredential</strong> to save the API credentials in a directory and the <strong>-Encrypt</strong> parameter can be utilized to encrypt the API credentials before exporting the JSON file into the designated <em>Location</em> directory.</p> </blockquote> <ul> <li><strong>Remove-HPEGLAPIcredential</strong>: to delete an API credential of an application instance</li> </ul> <blockquote> <p><strong>Note</strong>: Once API credentials are deleted, access to the application instance API is lost permanently. This is because API credentials are used to authenticate and authorize access to the API. When the credentials are deleted, the corresponding API client ID is also invalidated, which means that any requests made using those credentials will be rejected by the API server.</p> </blockquote> </li> </ul> <h2>Managing subscriptions with the HPE GreenLake PowerShell library</h2> <p>The main cmdlets for managing subscriptions in the HPE GreenLake platform are the followings:</p> <ul> <li> <p>To get information about subscriptions:</p> <ul> <li><strong>Get-HPEGLDeviceSubscription</strong>: to get information about your device subscriptions available in your HPE GreenLake account. Several parameters can be used to display the subscriptions with available quantity, the subscriptions that are expired or not expired, etc. For example, you can combine parameters to obtain only the subscription keys that have not expired and for which there are still licenses available: <strong>Get-HPEGLDeviceSubscription -Notexpired -Available</strong></li> </ul> </li> <li> <p>To manage device subscriptions:</p> <ul> <li><strong>Add-HPEGLDeviceSubscription</strong>: to add device subscriptions to the HPE GreenLake account</li> <li><strong>Set-HPEGLDeviceAutoSubscription</strong>: to set automatic subscription assignment. This feature automatically assigns a subscription to any supported device that is added to the HPE GreenLake platform</li> <li><strong>Remove-HPEGLDeviceAutoSubscription</strong>: to remove an automatic assignment of subscriptions</li> </ul> </li> <li> <p>To apply a subscription key to devices:</p> <ul> <li><strong>Set-HPEGLDeviceSubscription</strong>: to apply a subscription key to one or more devices</li> </ul> <blockquote> <p><strong>Note</strong>: When the auto device subscription is not supported or not enabled for the type of device you use, you need to manually apply a subscription key.</p> </blockquote> <ul> <li><strong>Remove-HPEGLDeviceSubscription</strong>: to detach devices from a subscription key</li> </ul> </li> </ul> <h1>Summary</h1> <p>In this blog, I have introduced you to the primary cmdlets that are provided by this new PowerShell library. However, there are more cmdlets available and much more to explore. Nevertheless, you now possess the necessary knowledge to begin testing and to develop your own PowerShell scripts to start automating tasks so you can experience greater productivity and efficiency.</p> <p>If you encounter any issues during your testing, it's highly recommended that you open a <a href="https://github.com/HewlettPackard/POSH-HPEGreenLake/issues">New issue</a> on the library’s GitHub page. This can help improve the library and potentially benefit other users who may be facing similar issues.</p> <p>For general questions, or need to discuss a topic that doesn't need to be tracked in the issue tracker, please join the GitHub Discussions for the project: <a href="https://github.com/HewlettPackard/POSH-HPEGreenLake/discussions/">Join the discussion</a></p> <h1>Want more?</h1> <ul> <li><a href="https://developer.greenlake.hpe.com/">HPE GreenLake Developer Portal</a></li> <li>To learn more about the HPE GreenLake platform, refer to the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00120892en_us">HPE GreenLake Edge to Cloud Platform User Guide</a></li> </ul>https://developer.hpe.com/integrating-istio-and-spire/https://developer.hpe.com/integrating-istio-and-spire/Tue, 25 Apr 2023 09:54:00 GMT<p>This blog demonstrates how to integrate Istio and SPIRE to enable advanced analysis and visualization of the service mesh.</p> <h2>Istio</h2> <p>Istio is an <strong>open-source service mesh</strong> that provides a uniform and efficient way to secure, connect, and monitor services. Istio automatically manages load balancing for HTTP, gRPC, WebSocket, and TCP traffic. For details, see <a href="https://istio.io/latest/about/service-mesh/">The Istio service mesh</a>.</p> <p><img src="https://istio.io/latest/docs/ops/deployment/architecture/arch.svg" alt=""></p> <h2>SPIRE</h2> <p>SPIRE (SPIFFE Runtime Environment) is a production-ready implementation of the SPIFFE (Secure Production Identity Framework for Everyone) specification that performs node and workload attestation to securely issue cryptographic identities to workloads running in heterogeneous environments.</p> <p><img src="/img/spire.png" alt=""></p> <p>SPIRE can be configured as a source of cryptographic identities for Istio workloads through an integration with Envoy’s SDS (Secret Discovery Service) API. This integration with SPIRE provides flexible attestation options not available with the default Istio identity management while harnessing Istio’s powerful service management.</p> <p><img src="/img/picture1.jpg" alt=""></p> <p>Once integrated, SPIRE issued certificates to workloads can be used for communication between different trust domains or between two different clusters also.</p> <p>In this blog post, we will show you the steps you can use to install Istio and SPIRE on the same cluster and how to deploy a sample application using SPIRE-issued identities.</p> <h3>Step 1: Creating your own cluster</h3> <p><strong>1.1</strong> Go to <a href="https://client.greenlake.hpe.com/session/hub/choose">HPE Greenlake </a>client page.</p> <p><strong>1.2</strong> Login to your tenant in HPE GreenLake Central and navigate to the HPE GreenLake for Private Cloud Enterprise dashboard. Click on <strong>Containers</strong> to launch into the containers dashboard.</p> <p><img src="/img/picture2.png" alt=""></p> <p><strong>1.3</strong> You will notice a page similar to the one shown below. Click <strong>Create cluster</strong> to create a new cluster, or you can also choose from the already created clusters. Ensure that you choose a cluster that does not have Istio pre-deployed, since this exercise will deploy SPIRE and Istio together.</p> <p><img src="/img/picture4.png" alt=""></p> <p><strong>1.4</strong> After clicking Create cluster, give a name and description to your cluster and identify the type of cluster. In our case, we have chosen a large type.</p> <p><img src="/img/picture5.png" alt=""></p> <p><strong>1.5</strong> Obtain Kubeconfig for your cluster and launch a Web terminal to access your cluster to handle steps found later in this post.</p> <p>From the <strong>Containers</strong> main page, click <strong>Launch Service Console</strong> to launch the HPE Ezmeral Runtime Enterprise. Open Kubectl, which allows you to enter commands to communicate with your cluster.</p> <p>If Kubectl is not installed, download Kubectl and the HPE Kubectl Plugin from the <strong>Dashboard</strong>.</p> <p>For more information, see** <a href="https://docs.containerplatform.hpe.com/55/reference/kubernetes/tenant-project-administration/Dashboard__Kubernetes_TenantProject_Administrator.html">Dashboard - Kubernetes Tenant/Project Administrator</a>** in the HPE Ezmeral Runtime Enterprise documentation.</p> <h3>Step 2: Install SPIRE</h3> <p>Get the quickstart yaml file using <strong><a href="https://raw.githubusercontent.com/cxteamtrials/caas-trials-content/main/services/spire/spire-quickstart.yaml">this link</a></strong> and copy that into your cluster and apply it by using kubectl.</p> <pre><code class="language-shellsession">kubectl apply -f spire-quickstart.yaml </code></pre> <p>This will install SPIRE into your cluster, along with two additional components: the SPIFFE CSI Driver and the SPIRE Kubernetes <strong>Controller manager</strong> which facilitate the registration of workloads and establishment of federation relationships.</p> <p>Verify installation of SPIRE by checking if all pods are running and containers within them are up. Specifically, you should look for the agent and SPIRE server.</p> <p><strong>Note:</strong> The number of agents depends on number of nodes you are working with. Here, we are working with three worker nodes, so three agents are assigned for each node.</p> <p>Use the command given below, and you will get the output as shown.</p> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ kubectl get pods -n spire NAME READY STATUS RESTARTS AGE spire-agent-5tlck 3/3 Running 2 (31d ago) 31d spire-agent-gnwbj 3/3 Running 1 (31d ago) 31d spire-agent-mghnw 3/3 Running 2 (31d ago) 31d spire-server-574474c7dc-42kln 2/2 Running 4 (4d1h ago) 31d </code></pre> <h3>Step 3: Install Istio</h3> <h4>Download the latest release:</h4> <p>You can download the latest release using the official Istio repository or just copy the following command (which would do the same thing for you).</p> <pre><code class="language-shellsession">curl -L https://istio.io/downloadIstio | sh - </code></pre> <p>For details, reach out to <strong><a href="https://istio.io/latest/docs/setup/getting-started/#download">ISTIO download page</a></strong>.</p> <p>Get into the Istio directory and set the path by command:</p> <pre><code class="language-shellsession">export PATH=$PWD/bin:$PATH </code></pre> <p>After exporting, get out of directory.</p> <pre><code class="language-shellsession">cd .. </code></pre> <p><strong>Note:</strong> In the future, a case might occur when your cluster does not recognize istioctl. In this case, export the path again after getting into the Istio directory.</p> <h4><strong>Install Istio with patches:</strong></h4> <p>After deploying SPIRE into your environment and verifying that all deployments are in Ready state, install Istio with custom patches for the Ingress-gateway as well as for Istio-proxy.</p> <p>Get the istio-spire-config patch using <strong><a href="https://raw.githubusercontent.com/cxteamtrials/caas-trials-content/main/services/istio/release-1.17/spire/spire-patch.yaml">this link</a></strong> and copy that patch into your cluster. Install that patch using following command.</p> <pre><code class="language-shellsession">istioctl install -f istio-spire-config.yaml </code></pre> <p>This will share the spiffe-csi-driver with the Ingress Gateway and the sidecars that are going to be injected on workload pods, granting them access to the SPIRE Agent’s UNIX Domain Socket.</p> <ul> <li> <h4>Patching Istio-Ingress gateways</h4> <p>If you receive the error shown below, your ingress-gateway is not patched yet and is not being registered onto the server.</p> <p><img src="/img/patch-error-ingress.jpg" alt=""></p> <p>For patching, the first step is to get and apply one of SPIRE controller manager’s <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRD (Custom Resource Definition)</a> ClusterSPIFFEID. It is a cluster-wide resource used to register workloads with SPIRE. The ClusterSPIFFEID can target all workloads in the cluster or can be optionally scoped to specific pods or namespaces via label selectors.</p> <p>Create a ClusterSPIFFEID CRD to generate registration entries in SPIRE server for all workloads with the label <strong><code>spiffe.io/spire-managed-identity: true</code></strong>.</p> <p>Get the ClusterSPIFFEID used by us for this demo using <strong><a href="https://raw.githubusercontent.com/cxteamtrials/caas-trials-content/main/services/spire/clusterspiffeid-example.yaml">this link</a></strong>, copy that into your cluster, and apply it.</p> <pre><code class="language-shellsession">kubectl apply -f cluster-spiffeID-crd.yaml </code></pre> <p><strong>Note:</strong> You can create your own custom clusterSPIFFEID CRD with your own match label and own selector. For now, we have created a simple CRD with one pod selector and one match label.</p> <p>Now, simply patch the ingress-gateway with spiffe.io/spire managed-identity: true label.</p> <p>This will register your ingress-gateway pod into the server.</p> <pre><code class="language-shellsession">kubectl patch deployment istio-ingressgateway -n istio-system -p '{"spec":{"template":{"metadata":{"labels":{"spiffe.io/spire-managed-identity": "true"}}}}}' </code></pre> <p>After patching, confirm that your ingress-gateway pod, istiod, and all their containers work.</p> </li> </ul> <h2>Step 4: Deploying Sample Application</h2> <p>Now that our SPIRE and Istio are integrated, the identities to workloads must be issued by SPIRE.</p> <p>For our case, we will create a namespace “bookinfo” and will add a label <strong>“spiffe.io/spire-managed-identity: true”</strong> to it. Then, we will create a new ClusterSPIFFEID CRD with <strong>namespace selector</strong> with match label as “spiffe.io/spire-managed-identity: true.”</p> <p>When the new workload is added to this namespace or any other namespace that has the lable mentioned above, it will now automatically get registered in the server.</p> <p><strong>4.1</strong> Create a new namespace.</p> <pre><code class="language-shellsession">kubectl create namespace &#x3C;insert-namespace-name-here> </code></pre> <p><strong>4.2</strong> Add a lable to it, using the same one that you have used for the clusterSPIFFEID.</p> <pre><code class="language-shellsession">kubectl label namespaces &#x3C;namespace_name> spiffe.io/spire-managed-identity=true </code></pre> <p><strong>4.3</strong> Enable istio-injection for this namespace so that any new pods that are created in that namespace will automatically have a sidecar added to them. You can achieve this by just adding another label in similar fashion.</p> <pre><code class="language-shellsession">kubectl label namespace &#x3C;namespace_name> istio-injection=enabled --overwrite </code></pre> <p>After all edits to your namespace, the namespace should look similar as shown below.</p> <p><strong>Note:</strong> If you want to, you can edit further using the following command. But take care that your resulting yaml is not invalid. You can validate your yaml using any online validator available.</p> <pre><code class="language-shellsession">kubectl edit ns &#x3C;namespace_name> </code></pre> <pre><code class="language-yaml">apiVersion: v1 kind: Namespace metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"name":"backend"},"name":"bookinfo"}} creationTimestamp: "2023-03-21T06:23:38Z" labels: istio-injection: enabled kubernetes.io/metadata.name: bookinfo spiffe.io/spire-managed-identity: "true" name: bookinfo resourceVersion: "6116523" uid: 6193a0b7-8455-46bd-a456-797ef69c045a spec: finalizers: - kubernetes status: phase: Active </code></pre> <p><strong>4.4</strong> Create and apply a ClusterSPIFFEID CRD with namespace selector.</p> <p>Copy the clusterSPIFFEID from <strong><a href="https://raw.githubusercontent.com/cxteamtrials/caas-trials-content/main/services/spire/clusterspiffeid-example.yaml">this link</a></strong> and just change the selector to <strong>namespace selector</strong>. Make sure that the correct match label is there like shown below.</p> <pre><code class="language-yaml">apiVersion: spire.spiffe.io/v1alpha1 kind: ClusterSPIFFEID metadata: name: bookinfo spec: spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}" namespaceSelector: matchLabels: spiffe.io/spire-managed-identity: "true" </code></pre> <p>After editing your clusterSPIFFEID, apply it using kubectl.</p> <pre><code class="language-shellsession">kubectl apply -f &#x3C;your_clusterSPIFFEID_name> </code></pre> <p><strong>4.5</strong> After successfully creating namespace and applying CRD, deploy your application in the namespace you created. But before you deploy your application, the workloads will need to have the SPIFFE CSI Driver volume be able to access the SPIRE Agent socket. To accomplish this, we can leverage the SPIRE pod annotation template:</p> <pre><code class="language-yaml">annotations: inject.istio.io/templates: "sidecar,spire" </code></pre> <p>You can patch it to the workload or just add this to your deployment manifest at <strong>{spec:{template:{metadata:{ annotation:}}}}</strong> as shown below.</p> <p><img src="/img/sidecar-patch.png" alt=""></p> <p>You can get the sample bookinfo application manifest from <strong><a href="https://raw.githubusercontent.com/cxteamtrials/caas-trials-content/main/services/istio/release-1.16/samples/bookinfo/bookinfo.yaml">this link</a></strong>.</p> <p><strong>Note:</strong> This manifest is annotation free, so you need to add annotation to its deployments by following the steps shown above.</p> <p>After editing the manifest, apply it in a newly created namespace.</p> <pre><code class="language-shellsession">kubectl apply -f bookinfo.yaml -n &#x3C;namespace_name> </code></pre> <p>Verify all workloads and services you just deployed are running and up.</p> <pre><code class="language-shellsession">kubectl get all -n &#x3C;namespace_name> </code></pre> <p>You will get output as shown below if everything is working fine.</p> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ kubectl get all -n bookinfo NAME READY STATUS RESTARTS AGE pod/details-v1-f8957ccb4-7vdgw 2/2 Running 0 37d pod/productpage-v1-cfb4bc854-5km2l 2/2 Running 0 37d pod/ratings-v1-65cd6fbcd8-s9jnc 2/2 Running 0 37d pod/reviews-v1-55f769fb78-czh7j 2/2 Running 0 37d pod/reviews-v2-6b7c798cc8-wkpxg 2/2 Running 0 37d pod/reviews-v3-695c7f59db-nzwwk 2/2 Running 2 (34d ago) 37d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/details ClusterIP 10.111.38.161 &#x3C;none> 9080/TCP 37d service/productpage ClusterIP 10.102.189.161 &#x3C;none> 9080/TCP 37d service/ratings ClusterIP 10.105.7.153 &#x3C;none> 9080/TCP 37d service/reviews ClusterIP 10.106.49.246 &#x3C;none> 9080/TCP 37d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/details-v1 1/1 1 1 37d deployment.apps/productpage-v1 1/1 1 1 37d deployment.apps/ratings-v1 1/1 1 1 37d deployment.apps/reviews-v1 1/1 1 1 37d deployment.apps/reviews-v2 1/1 1 1 37d deployment.apps/reviews-v3 1/1 1 1 37d NAME DESIRED CURRENT READY AGE replicaset.apps/details-v1-f8957ccb4 1 1 1 37d replicaset.apps/productpage-v1-cfb4bc854 1 1 1 37d replicaset.apps/ratings-v1-65cd6fbcd8 1 1 1 37d replicaset.apps/reviews-v1-55f769fb78 1 1 1 37d replicaset.apps/reviews-v2-6b7c798cc8 1 1 1 37d replicaset.apps/reviews-v3-695c7f59db 1 1 1 37d </code></pre> <p>Once everything is up, all workloads would get registered under the SPIRE server.</p> <p><strong>4.6</strong> You can verify the registration of workloads using the following command:</p> <pre><code class="language-shellsession">kubectl exec &#x3C;spire-server_pod_name> -n spire -c spire-server -- ./bin/spire-server entry show </code></pre> <p>Verify that every workload with same label as clusterSPIFFEID CRD’s match label is registered in the server.</p> <p><img src="/img/server-entries.png" alt=""></p> <p><strong>4.7</strong> Verify that the certificate issuer of workloads is SPIRE using the following commands for each workload.</p> <pre><code class="language-shellsession">istioctl proxy-config secret &#x3C;pod_name> -n &#x3C;namespace_name> -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem </code></pre> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ openssl x509 -in chain.pem -text | grep SPIRE Subject: C = US, O = SPIRE, x500UniqueIdentifier = e2f9c35b9198e1824373e874b13287d0 </code></pre> <p>You should also check that everything is the same for the ingress-gateway pod in Istio-system namespace and verify that your deployed workloads and ingress-gateway has the same issuer.</p> <h4>Step 5: Open the application to outside traffic</h4> <p>The Bookinfo application is deployed but not accessible from the outside. To make it accessible, you need to create an Istio Ingress Gateway, which maps a path to a route at the edge of your mesh.</p> <p><strong>5.1</strong> Associate this application with the Istio gateway:</p> <pre><code class="language-shellsession">kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml -n bookinfo </code></pre> <p><strong>5.2</strong> Ensure that there are no issues with the configuration:</p> <pre><code>k8s-spiffe-integ-master-7j7fh-m67q9:~ # istioctl analyze -n bookinfo ✔ No validation issues found when analyzing namespace: bookinfo. </code></pre> <p><strong>5.3</strong> Execute the following command to determine if your Kubernetes cluster is running in an environment that supports external load balancers:</p> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ kubectl get svc istio-ingressgateway -n istio-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-ingressgateway LoadBalancer 10.105.191.32 172.16.17.5 15021:30189/TCP,80:30392/TCP,443:30566/TCP 32d </code></pre> <p>If the EXTERNAL-IP value is set, your environment has an external load balancer. If not, then set the external load balancer first then follow further steps.</p> <p>For this cluster, we are using metallb.</p> <p><strong>5.4</strong> Download and install Kiali dashboard and Prometheus.</p> <p><strong>Install Kiali:</strong></p> <p><strong><a href="https://kiali.io/">Kiali</a></strong> is an observability console for Istio with service mesh configuration and validation capabilities. It helps you understand the structure and health of your service mesh by monitoring traffic.</p> <pre><code class="language-shellsession">kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.17/samples/addons/kiali.yaml </code></pre> <p><strong>Install Prometheus:</strong></p> <p><strong><a href="https://prometheus.io/">Prometheus</a></strong> is an open-source monitoring system and time series database. You can use Prometheus with Istio to record metrics that track the health of Istio and of applications within the service mesh.</p> <pre><code class="language-shellsession">kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.17/samples/addons/prometheus.yaml </code></pre> <p><strong>5.5</strong> After setting up the ingress gateway and bookinfo gateway, we will view the dashboard later on in this post. To ensure you'll be able to do this, you'll need to make these setting changes in your system proxy status.</p> <p>Go to <strong>Settings > Network > Proxy status ></strong> Turn Use a <strong>proxy server On</strong>. In the exceptions field, add your external IP address of kiali and ingressgateway service.</p> <p>You can get IPs of these services by following command:</p> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ kubectl get svc -n istio-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-ingressgateway LoadBalancer 10.105.191.32 172.16.17.5 15021:30189/TCP,80:30392/TCP,443:30566/TCP 32d istiod ClusterIP 10.101.27.65 &#x3C;none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 32d kiali LoadBalancer 10.103.14.197 172.16.17.6 20001:32116/TCP,9090:31950/TCP 32d prometheus ClusterIP 10.98.101.102 &#x3C;none> 9090/TCP 32d </code></pre> <p><img src="/img/manual_proxy.png" alt=""></p> <p><strong>Format:</strong> <code>http://{external ip};</code></p> <p><strong>Note:</strong> Your kiali service might be of ClusterIP type, so to get the external IP for this service, you first need to edit the service type to LoadBalancer.</p> <ul> <li> <p>Use the following command to edit the service, then edit the service type.</p> <pre><code class="language-shellsession">kubectl edit svc kiali -n istio-system </code></pre> <p>Edit the service type <strong>{spec: {type:LoadBalancer}}</strong> as shown below</p> <p><img src="/img/service_edit.png" alt=""></p> </li> </ul> <p><strong>5.6</strong> Set the ingress IP and ports:</p> <pre><code class="language-shellsession">export INGRESS_NAME=istio-ingressgateway export INGRESS_NS=istio-system export INGRESS_HOST=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.status.loadBalancer.ingress[0].ip}') export INGRESS_PORT=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.spec.ports[?(@.name=="http2")].port}') </code></pre> <p><strong>5.7</strong> Export and Set GATEWAY_URL:</p> <pre><code class="language-shellsession">export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT </code></pre> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ echo "$GATEWAY_URL" 172.16.17.5:80 </code></pre> <p>Curl into the productpage using gateway URL using following command.</p> <pre><code class="language-shellsession">k8s-spiffe-integ-master-7j7fh-m67q9:~ curl -v http://$GATEWAY_URL/productpage * Uses proxy env variable no_proxy == 'localhost,127.0.0.1,10.96.0.1,172.16.5.41,172.16.5.42,172.16.5.43,172.16.5.44,172.16.5.45,172.16.5.46,172.16.5.40,glhc-caas.glhc-hpe.local,.glhc-hpe.local,glhc-caas.customer.hpe.net,172.16.17.20,172.16.17.21,172.16.17.22,172.16.5.47,gl-pulpnode.glhc-hpe.local,gl-pulpnode,10.96.0.1,10.192.0.0/12,10.96.0.0/12,.svc,.cluster.local,.default.svc,.customer.hpe.net,172.16.17.23,172.16.17.30,gl-cp-gw-node2.glhc-hpe.local,gl-cp-gw-node1.glhc-hpe.local,172.16.17.50' * Uses proxy env variable http_proxy == 'http://172.16.0.250:8080' * Trying 172.16.0.250:8080... * TCP_NODELAY set * Connected to 172.16.0.250 (172.16.0.250) port 8080 (#0) > GET http://172.16.17.5:80/productpage HTTP/1.1 > Host: 172.16.17.5 > User-Agent: curl/7.66.0 > Accept: */* > Proxy-Connection: Keep-Alive > </code></pre> <p>You can generate traffic on the product page by just reaching out to the shown http URL.</p> <p><strong>Note:</strong> Before reaching out to this page and kiali in further step, ensure that you have followed step 5.5 properly.</p> <p><strong>5.9</strong> <strong>Kiali Dashboard</strong></p> <p>Generate traffic on the product page and observe the graphs on the Kiali dashboard. Reach out to the kiali dashboard in your browser by just copying the external IP from above and http into that IP and port.</p> <pre><code class="language-shellsession">http://&#x3C;kiali_external_ip>:&#x3C;port> </code></pre> <p>After reaching kiali dashboard, generate traffic on product page and simultaneously, view and analyse traffic on kiali using various graphs and visualising methods.</p> <p><strong>App Graph:</strong></p> <p><img src="/img/app_graph.png" alt=""></p> <p><strong>Service Graph:</strong></p> <p><img src="/img/service_graph-1.png" alt=""></p> <p><img src="/img/service_graph-22.png" alt=""></p> <p>The graph below shows services communication, and the lock here symbolises <strong>mTls protocol</strong>.</p> <p>We hope that this blog has helped you in integrating Istio and SPIRE from scratch, getting SPIRE issued identities for your sample application workloads, and setting up Kiali on your cluster for better visualisation of your service mesh.</p>https://developer.hpe.com/integrating-python-3-with-fips-enabled-openssl-3-1/https://developer.hpe.com/integrating-python-3-with-fips-enabled-openssl-3-1/Sun, 23 Apr 2023 13:41:41 GMT<h2>Introduction</h2> <p>Federal Information Processing Standard (FIPS) are a set of encryption algorithms and are mandatory in all the computer systems and software used by non-military American government agencies, government contractors, and the vendors who work with the agencies. Whenever new software is developed, it needs to be FIPS-compliant. Thus, there is a need to enable Python 3 with FIPS, but the default Python 3 binary package comes without FIPS, as shown in the screenshot below:</p> <p><img src="/img/openssl-before.jpg" alt=""></p> <p>This blog will explain step-by-step how to integrate the Python 3 with the FIPS enabled OpenSSL 3.1 on Microsoft Windows so that any new software compiled out of it is FIPS-compliant.</p> <h2>Steps for Microsoft Windows</h2> <p><strong>Step 1:</strong> Download the OpenSSL source from <code>http://www.openssl.org</code> and the Python 3 source from <code>http://www.python.org</code>.</p> <p><strong>Step 2:</strong> Install NASM and Perl Software and add to PATH</p> <p><strong>Step 3:</strong> Install these 2 Perl modules using the commands below.</p> <blockquote> <pre><code class="language-bash">cpan -i Text::Template cpan -i Test::More </code></pre> </blockquote> <p><strong>Step 4:</strong> Create a directory structure like what's shown below.</p> <blockquote> <pre><code class="language-bash">mkdir "C:\SSLout\SSL" mkdir "C:\SSLout\DLL\x64\Release" mkdir "C:\SSLout\Lib" </code></pre> </blockquote> <p><strong>Step 5:</strong> Unzip the <code>openssl-3.1.0.tar.gz</code> file into <code>C:\work\openssl-3.1.0</code></p> <p><strong>Step 6:</strong> Build the OpenSSL module using <code>VC++ 2015 x64 the Native Tools</code> Command prompt.</p> <blockquote> <pre><code class="language-bash">cd openssl-3.1.0 Perl Configure VC-WIN64A --prefix=C:\SSLout\DLL\x64\Release -openssldir=c:\SSLout\SSL enable-fips nmake nmake install_sw nmake install_ssldirs nmake install_docs nmake install_fips </code></pre> </blockquote> <p>The OpenSSL 3.1 binaries are generated in the <code>C:\SSLout\DLL\x64\Release\bin</code>.</p> <p>This is shown as shown below:</p> <p><img src="/img/openssl_directory_structure.jpg" alt=""></p> <p><strong>Step 7:</strong> Unzip the <code>Python-3.11.2.tgz</code> to <code>C:\work\Python3.11.2</code> and go into</p> <blockquote> <pre><code class="language-bash">cd C:\work\Python3.11.2\PCBuild </code></pre> </blockquote> <p>Run the <code>get_externals.bat</code> file. This will fetch the dependencies from the internet into the <code>externals</code> directory.</p> <blockquote> <pre><code class="language-bash">get_externals.bat </code></pre> </blockquote> <p>Now create the <code>openssl-bin-3.1.0</code> directory under the <code>externals</code> directory as shown below:</p> <blockquote> <pre><code class="language-bash">cd externals mkdir openssl-bin-3.1.0 </code></pre> </blockquote> <p>The directory structure will look like this as shown below:</p> <p><img src="/img/openssl_directory_structure2.jpg" alt=""></p> <p><strong>Step 8:</strong> Under the <code>externals</code> directory, create an <code>amd64</code> depending on the CPU architecture and copy the files to it as shown below:</p> <blockquote> <pre><code class="language-bash">copy C:\SSLout\DLL\x64\Release\lib\*.lib C:\work\Python3.11.2\PCBuild\externals\openssl-bin-3.1.0\amd64 copy C:\SSLout\DLL\x64\Release\bin\*.* C:\work\Python3.11.2\PCBuild\externals\openssl-bin-3.1.0\amd64 copy C:\SSLout\DLL\x64\Release\include C:\work\Python3.11.2\PCBuild\externals\openssl-bin-3.1.0\amd64 copy C:\SSLout\DLL\x64\Release\include\openssl\applink.c C:\work\Python3.11.2\PCBuild\externals\openssl-bin-3.1.0\amd64\include copy C:\SSLout\DLL\x64\Release\lib\ossl-modules\*.* C:\work\Python3.11.2\PCBuild\externals\openssl-bin-3.1.0\amd64 </code></pre> </blockquote> <p><strong>Step 9:</strong> Modify <code>PCbuild/openssl.props</code> as shown below:</p> <p><img src="/img/openssl_settings.jpg" alt=""></p> <p><strong>Step 10:</strong> Open <code>PCbuild/python.props</code> and change the entries as shown below:</p> <p><img src="/img/openssl_settings2.jpg" alt=""></p> <p><strong>Step 11:</strong> Open the Python Solution under <code>PCbuild/pcbuild.sln</code> in VS 2015. Change the Linker settings of <code>_hashlib</code> and <code>_ssl</code> projects as shown below:</p> <p><img src="/img/openssl_vs_settings.jpg" alt=""></p> <p><strong>Step 12:</strong> Now, build <code>_hashlib.pyd</code> and <code>_ssl.pyd</code> in VS 2015.</p> <p><strong>Step 13:</strong> Copy these built .pyd files from <code>\PCbuild\amd64\</code> to the Python binary installation directory <code>C:\python311\DLLs</code>.</p> <p><strong>Step 14:</strong> Start Python binary installtion and use these commands to check the OpenSSL 3.1 version as shown below:</p> <p><img src="/img/openssl-after.jpg" alt=""></p> <p><strong>Step 15:</strong> Now, to run Python 3 with a FIPS enabled OpenSSL 3.1, create <code>openssl.cnf</code> and <code>fipsmodule.cnf</code> file using the below content in <code>C:\Python311</code> directory. By using this config file, FIPS will be enabled by default.</p> <blockquote> <h4>openssl.cnf:</h4> </blockquote> <blockquote> <pre><code class="language-bash">config_diagnostics = 1 openssl_conf = openssl_init .include .\fipsmodule.cnf [openssl_init] providers = provider_sect alg_section = algorithm_sect [provider_sect] fips = fips_sect legacy = legacy_sect base = base_sect default = default_sect [base_sect] activate = 1 [legacy_sect] activate = 1 [default_sect] activate = 1 [algorithm_sect] default_properties = fips=yes </code></pre> <h4>fipsmodule.cnf:</h4> <pre><code class="language-bash">[fips_sect] activate = 1 conditional-errors = 1 security-checks = 1 module-mac = D4:64:00:E3:CE:34:EE:CE:58:32:12:08:21:6D:64:FD:E3:A6:D4:F0:E6:38:3D:2C:0C:40:1B:50:C8:8F:39:A3 </code></pre> </blockquote> <p><strong>Step 16:</strong> Now, open the <code>command</code> window as an <code>Administrator</code> and execute the following command as shown below:</p> <blockquote> <pre><code class="language-bash">set OPENSSL_CONF=C:\Python311\openssl.cnf </code></pre> </blockquote> <p><strong>Step 17:</strong> To verify Python is enabled with FIPS, both the client and server need to be in FIPS mode. The client Windows OS needs to be enabled with FIPS using these 2 steps:</p> <blockquote> <ol> <li>Open <code>gpedit.msc</code> on run menu and navigate to <code>Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options</code> and enable the <code>System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing</code> setting.</li> <li>Open <code>regedit</code> on run menu and go to <code>HKLM\System\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy\Enabled</code> and set Enabled to 1.</li> </ol> </blockquote> <p><strong>Step 18:</strong> To verify Python 3 is enabled with FIPS, run the following commands as shown below:</p> <p><img src="/img/openssl_fips_algo.jpg" alt=""></p> <p>Note that the list of crypto algorithms available are more than the crypto algorithms guaranteed. But all the available algorithms can be used if the Server where the Client SSL is connecting to, is also configured in FIPS mode.</p> <p>The living example for the OpenSSL Server is HPE iLO.</p> <p>Voila!!... Now the Python 3.11 is integrated with the OpenSSL 3.1 which is enabled with FIPS in Windows Platform. This Python 3 installation can be used to develop the applications which are OpenSSL 3 and FIPS enabled!!</p> <h2>Summary</h2> <p>In this blog, I have covered the following steps regarding integrating Python 3 with FIPS enabled OpenSSL 3.1:</p> <ul> <li>Download the required packages.</li> <li>Compile OpenSSL 3 along with FIPS enabled.</li> <li>Make the required changes and compile the Python along with the OpenSSL 3.1 binaries copied as an external dependency.</li> <li>Copy the newly generated binaries to the Python 3 installation directory.</li> <li>Test the OpenSSL 3 version in Python 3 and also verify if it is FIPS enabled.</li> </ul> <p>I hope this blog is useful to the entire developer community!! Make sure you check out our other blog posts on <a href="https://developer.hpe.com/blog/">HPE DEV</a> for more useful tutorials.</p>https://developer.hpe.com/hpe-community-hackathon-builds-stem-excitement-for-middle-school-students/https://developer.hpe.com/hpe-community-hackathon-builds-stem-excitement-for-middle-school-students/Fri, 07 Apr 2023 16:39:31 GMT<p><img src="/img/hpe20160525019_800_0_72_rgb.jpg" alt=""></p> <p>A Hewlett Packard Enterprise (HPE) employee resource group called S.W.A.G. (Supporting Women’s Aspirations Group) is hosting a hackathon for middle school students in the San Jose area on May 4th, 2023. Working in partnership with <a href="https://www.curatedpathways.org/">Curated Pathways to Innovation™</a> (CPI), they are expected to deliver an exceptional experience to around 100 students. The day will be filled with fun activities meant to raise students’ awareness of the latest advancements in technology and their potential in playing a part in its development. Many volunteers will be helping with the planning and execution of this hackathon, and a fun time is expected to be had by all.</p> <p>As a part of the hackathon, students will be required to address a specific community issue and present their solution to the problem. Children will collaborate with one another, looking at community issues such as homelessness, domestic violence and teen depression/suicide and find ways to leverage technology to address them. They will do so using a hackathon platform developed by Curated Pathways to Innovation in conjunction with HPE. This platform forms the foundation of a STEM (Science, Technology, Engineering and Mathematics) curriculum followed by the schools participating in this event. Their students already use this platform in their school, so they’re very familiar with how to work with it.</p> <h4>A day filled with activities</h4> <p>Upon arrival at the facility, students will be greeted by S.W.A.G. team members along with HPE executives, Manju Abraham, Praveena Patchipulusu, and Lloyd Santy. They’ll then be given a brief overview of the process, including the schedule, logistics, and judging criteria. Once complete, the students will be divided into teams to begin work on their chosen topic. Between 10 and 12pm students will rotate between taking a tour of the HPE Customer Innovation Center (CIC) and working on their community problem, breaking it down into solvable pieces, coming up with solutions, and preparing their presentations.</p> <p>As a part of their tour of the HPE CIC, students will be introduced to many of the amazing products HPE produces and how they probably already experience these products as part of their everyday activities – from bringing real-time location context to mobile apps in stores to immersive and connected experiences at theme parks.</p> <p><img src="/img/innovation-center-800-px.jpg" alt=""></p> <p>A special focal point of the CIC tour will be at a kiosk housing a model of a Galactic Starcruiser where the students will learn about HPE's partnership with Disney to bring about one of the most immersive experiences ever at Star Wars: Galaxy's Edge at the Walt Disney World Resort. Considering the date of this event, it certainly seems to be an appropriate tour stop. We can already imagine students calling out greetings of "May the 4th be with you". In the CIC, students will also learn about our work with auto manufacturers to bring greater safety to vehicles and how supercomputers are used to help unlock unlimited supplies of clean energy, potentially sparking some ideas for their projects. (Learn more about the <a href="https://www.hpe.com/us/en/about/virtual-customer-innovation-center.html">HPE Customer Innovation Center</a>.)</p> <p>From 12-1:30pm the teams will present their solutions, have lunch, and participate in some fun activities that include games and a movie. Afterwards, the finalists will be selected and students will move to the outdoor lawn for the Award Ceremony.</p> <p>Hackathon organizers hope that the event will help students recognize the importance of giving back to your community as well as the fun involved in technology development. And perhaps, given a memorable experience, they may even consider working for HPE someday.</p> <h4>Meet the organizers and the technology behind it all</h4> <p>S.W.A.G. (Supporting Women’s Aspirations Group) is a group of HPE employees who volunteer to make a difference in the lives of women in technology. HPE employee resource groups, like S.W.A.G., contribute to making HPE an employer of choice by supporting the company’s inclusion, equity, and diversity goals of hiring and supporting the best talent, providing a workplace that values the unique contributions of its people, and fostering an inclusive mindset that drives innovation, building a strong reputation in the marketplace. For HPE employees interested in participating as a volunteer for this year’s event, please reach out to the S.W.A.G. core team at <a href="mailto:swag-core@hpe.com">swag-core@hpe.com</a>.</p> <p>Curated Pathways to Innovation is a game-changing model that drives an impactful approach to engaging students in STEM education. It was launched as a partnership between YWCA Silicon Valley, Hewlett Packard Enterprise, Santa Clara University and Purdue University as an IT solution developed to help young women and underrepresented minorities navigate their educational journey, with an emphasis on computing careers. A web-based app, Curated Pathways acts as a virtual guidance counselor and uses gamification to engage with students.</p>https://developer.hpe.com/open-sourcing-workshops-on-demand-part3-understanding-the-backend/https://developer.hpe.com/open-sourcing-workshops-on-demand-part3-understanding-the-backend/Wed, 05 Apr 2023 09:29:49 GMT<p>In the previous <a href="https://developer.hpe.com/blog/open-sourcing-workshops-on-demand-part2-deploying-the-backend/">article</a> of this series, I explained how to deploy the backend server of our Workshops-on-Demand infrastructure.</p> <p>In this article, I will dig into details on the backend server. I will cover the inner workings of the registration process, explaining how a workshop is deployed on the backend server. Even though it takes only a few minutes for the backend server to deploy a workshop, there are many processes taking place in the background, which I will cover here.</p> <p>As a reminder, here is a diagram showing the different parts of the Workshops-on-Demand infrastructure. In this article, I will once again focus on the backend server side and, more precisely, on the JupyterHub server, where all the automation takes place.</p> <p><img src="/img/wod-blogserie3-archi3.png" alt="" title="Workshops-on-Demand Architecture"></p> <h4>Backend server / workshops deployment lifecycle</h4> <p>The following picture depicts what happens on the backend server when a participant registers for a workshop. If you remember from the first article, upon registration the frontend sends instructions to the backend server through a procmail API call so the latter can proceed with the workshop preparation and deployment. Once these tasks are completed, it provides the API-DB server with the relevant information</p> <p>Let's now look in details what is really happening on the backend server's side:</p> <p><img src="/img/wod-blogserie3-create.png" alt="" title="backend server CREATE workflow"></p> <p>0- The procmail API: This is a mail parsing process allowing the backend server to retrieve the relevant information in order to perform appropriate actions. As with any API, it uses verbs to perform actions. In our case, we leverage <strong>CREATE</strong>, <strong>CLEANUP</strong>, <strong>RESET</strong> and <strong>PURGE</strong>.</p> <p>If you need more info on procmail usage, check this <a href="https://wiki.archlinux.org/title/Procmail%3E">page</a>.</p> <p>Take a look at the following template of the <code>.procmailrc</code> file that will be expanded at setup time.</p> <pre><code class="language-shellsession">MAILDIR=$HOME/.mail # You'd better make sure it exists DEFAULT=$MAILDIR/mbox LOGFILE=$MAILDIR/from :0b #* ^From.*{{ WODSENDER }}.* # \/ defines what will be matched in $MATCH * ^Subject: *CREATE \/[1-9]+.* | {{ SCRIPTDIR }}/procmail-action.sh CREATE $MATCH :0b #* ^From.*{{ WODSENDER }}.* # \/ defines what will be matched in $MATCH * ^Subject: *CLEANUP \/[1-9]+.* | {{ SCRIPTDIR }}/procmail-action.sh CLEANUP $MATCH :0b #* ^From.*{{ WODSENDER }}.* # \/ defines what will be matched in $MATCH * ^Subject: *RESET \/[1-9]+.* | {{ SCRIPTDIR }}/procmail-action.sh RESET $MATCH :0b #* ^From.*{{ WODSENDER }}.* # \/ defines what will be matched in $MATCH * ^Subject: *PURGE student\/[1-9]+.* | {{ SCRIPTDIR }}/procmail-action.sh PURGE $MATCH </code></pre> <p>The <code>From:</code> is important as <code>.procmailrc</code> checks that the sender is the configured one from the frontend server. During the install process, the <code>WODSENDER</code> parameter ID referring to this. Any mail from any other sender but the configured one is not processed.</p> <p>This API is actually based on a script <code>procmail-action.sh</code>. This script defines the different actions linked to the verbs passed through the API calls via <code>.procmailrc</code></p> <p>Let's start with a <strong>CREATE</strong> scenario looking at the very first lines of the <code>procmail</code> log file.</p> <pre><code>From xyz@hpe.com Wed Mar 1 15:10:41 2023 Subject: CREATE 401 825 frederic.passeron@hpe.com Folder: /home/wodadmin/wod-backend/scripts/procmail-action.sh CREATE 14 </code></pre> <p>In <code>Subject:</code>, look for the API verb <strong>CREATE</strong> followed by <strong>student id,</strong> <strong>participant id</strong> and finally the registered <strong>participant email.</strong></p> <p>Here are the values respectively:</p> <ul> <li>student id: 401</li> <li>participant id: 825</li> <li>participant email: <a href="mailto:frederic.passeron@hpe.com">frederic.passeron@hpe.com</a></li> </ul> <p>In order to work properly, <code>procmail-action.sh</code> needs to source 3 files:</p> <p>1- <code>wod.sh</code></p> <p>2- <code>random.sh</code></p> <p>3- <code>functions.sh</code></p> <p><code>wod.sh</code> sets a large number of variables: This script is generated at install time as it leverages variables defined at setup time.</p> <pre><code class="language-shellsession"># This is the wod.sh script, generated at install # # Name of the admin user export WODUSER=wodadmin # Name of the WoD machine type (backend, api-db, frontend, appliance) export WODTYPE=backend # This main dir is computed and is the backend main dir export WODBEDIR=/home/wodadmin/wod-backend # BACKEND PART # The backend dir has some fixed subdirs # wod-backend (WODBEDIR) # |---------- ansible (ANSIBLEDIR) # |---------- scripts (SCRIPTDIR defined in all.yml not here to allow overloading) # |---------- sys (SYSDIR) # |---------- install # |---------- conf # |---------- skel # export ANSIBLEDIR=$WODBEDIR/ansible export SYSDIR=$WODBEDIR/sys # PRIVATE PART # These 3 dirs have fixed names by default that you can change in this file # they are placed as sister dirs wrt WODBEDIR # This is the predefined structure for a private repo # wod-private (WODPRIVDIR) # |---------- ansible (ANSIBLEPRIVDIR) # |---------- notebooks (WODPRIVNOBO) # |---------- scripts (SCRIPTPRIVDIR) # PWODBEDIR=`dirname $WODBEDIR` export WODPRIVDIR=$PWODBEDIR/wod-private export ANSIBLEPRIVDIR=$WODPRIVDIR/ansible export SCRIPTPRIVDIR=$WODPRIVDIR/scripts export SYSPRIVDIR=$WODPRIVDIR/sys export WODPRIVNOBO=$WODPRIVDIR/notebooks WODPRIVINV="" # Manages private inventory if any if [ -f $WODPRIVDIR/ansible/inventory ]; then WODPRIVINV="-i $WODPRIVDIR/ansible/inventory" export WODPRIVINV fi # AIP-DB PART export WODAPIDBDIR=$PWODBEDIR/wod-api-db # FRONTEND PART export WODFEDIR=$PWODBEDIR/wod-frontend # These dirs are also fixed by default and can be changed as needed export WODNOBO=$PWODBEDIR/wod-notebooks export STUDDIR=/student # export ANSPLAYOPT="-e PBKDIR=staging -e WODUSER=wodadmin -e WODBEDIR=/home/wodadmin/wod-backend -e WODNOBO=/home/wodadmin/wod-notebooks -e WODPRIVNOBO=/home/wodadmin/wod-private/notebooks -e WODPRIVDIR=/home/wodadmin/wod-private -e WODAPIDBDIR=/home/wodadmin/wod-api-db -e WODFEDIR=/home/wodadmin/wod-frontend -e STUDDIR=/student -e ANSIBLEDIR=/home/wodadmin/wod-backend/ansible -e ANSIBLEPRIVDIR=/home/wodadmin/wod-private/ansible -e SCRIPTPRIVDIR=/home/wodadmin/wod-private/scripts -e SYSDIR=/home/wodadmin/wod-backend/sys -e SYSPRIVDIR=/home/wodadmin/wod-private/sys" export ANSPRIVOPT=" -e @/home/wodadmin/wod-private/ansible/group_vars/all.yml -e @/home/wodadmin/wod-private/ansible/group_vars/staging" </code></pre> <p><code>random.sh</code> Exports the randomly generated password.</p> <p><code>functions.sh</code> Is a library of shell functions used by many scripts, among which can be found <code>procmail-action.sh</code>. Details are shown below.</p> <p>4- <code>procmail-action.sh</code> Calls the necessary functions and scripts to perform the <strong>CREATE</strong> operation.</p> <p>5- <code>get_session_token()</code> This function retrieves the necessary token to make an API call to the api-db server.</p> <p>6- <code>get_workshop_name()</code> This function extracts the workshop name from the mail body. In the body, one will find the workshop name. For example, <strong>WKSHP-API101</strong></p> <p>7- <code>get_workshop_id()</code> From the workshop name, the function <code>get_workshop_id()</code> will get the workshop's ID from the api-db server.</p> <p>This ID will be used later to get some of the workshop's specifics through additional API calls to the api-db server.</p> <ul> <li>Does the workshop require the use of the student password as a variable?</li> <li>Does the workshop require LDAP authentication?</li> <li>Does the workshop require a compiled script?</li> </ul> <p>8- <code>teststdid()</code> This function checks the student ID provided by procmail API is valid: For each workshop, a dedicated student range is allocated. This function exits when the student ID is not in the correct range.</p> <p>9- <code>generate_randompwd()</code> This function creates a random password for a user. It is used both for local and LDAP users' passwords. If the workshop requires an LDAP authentication (<code>get_ldap_status()</code> functions will return this information) then another function is used to update the LDAP server with the password for the given student (<code>update_ldap_passwd()</code>)</p> <p>The generated password will be sent back to the api-db server so that the frontend server can then send an email to allow participant to connect to the workshop.</p> <p>10- <code>erase_student()</code> This function erases all the content from the allocated student's home directory. You want to make sure that the home directory is not compromised. You want to start clean.</p> <p>11- <code>get_compile_status()</code> This function will check if the workshop needs some scripts to be compiled. For instance, if you need to authenticate against a private cloud portal and you don't want your participants to see the credentials, make sure to check the relevant box in the workshop table of the database. This compile feature will compile the authentication scripts into an executable that cannot be edited.</p> <p>12- If an appliance is needed for the workshop, then the following script is called: <code>create-&#x3C;WKSHP>.sh</code> .This will prepare the appliance (deploying a docker image on it for instance) and setup user env on the appliance accordingly (ssh keys, skeletons)</p> <p>For instance, <code>create-WKSHP-ML101.sh</code> will perform the following tasks in order to prepare the appliance for the workshop: It will start by reseting the appliance with the <code>reset-&#x3C;WKSHP>.sh</code> script. Then, it calls a second script aiming at preparing a generic appliance <code>create-appliance.sh</code>. Once done with these two, it moves on with the proper customization of the appliance for the given student.</p> <p>See details below.</p> <pre><code class="language-shellsession">#!/bin/bash set -x source {{ SCRIPTDIR }}/functions.sh export RTARGET={{ hostvars[inventory_hostname]['IP-WKSHP-ML101'] }} # Start by cleaning up stuff - do it early as after we setup .ssh content {{ SCRIPTDIR }}/reset-$ws.sh {{ SCRIPTDIR }}/create-appliance.sh NAME=mllab TMPDIR=/tmp/$NAME.$stdid mkdir -p $TMPDIR # Define local variables echo wid=$wid APPMIN=`get_range_min $wid` echo stdid=$stdid echo APPMIN=$APPMIN mlport=$(($stdid-$APPMIN+{{ hostvars[inventory_hostname]['MLPORT-WKSHP-ML101'] }})) mlport2=$(($stdid-$APPMIN+{{ hostvars[inventory_hostname]['MLPORT2-WKSHP-ML101'] }})) httpport=$(($stdid-$APPMIN+{{ hostvars[inventory_hostname]['HTTPPORT-WKSHP-ML101'] }})) cat > $TMPDIR/dockerd-entrypoint.sh &#x3C;&#x3C; EOF export HTTPPORT tini -g -- start-notebook.sh &#x26; sleep 3 jupyter lab list | tail -1 | cut -d'=' -f2 | cut -d' ' -f1 > {{ STUDDIR }}/student$stdid/mltoken sleep infinity EOF cat > $TMPDIR/Dockerfile &#x3C;&#x3C; EOF FROM ${NAME}:latest USER root COPY dockerd-entrypoint.sh /usr/local/bin/ ENTRYPOINT /usr/local/bin/dockerd-entrypoint.sh RUN mkdir -p {{ STUDDIR }}/student$stdid RUN useradd student$stdid -u $stdid -g 100 -d {{ STUDDIR }}/student$stdid RUN chown student$stdid:users {{ STUDDIR }}/student$stdid # Unlock the account RUN perl -pi -e "s|^student$stdid:!:|student$stdid:\$6\$rl1WNGdr\$qHyKDW/prwoj5qQckWh13UH3uE9Sp7w43jPzUI9mEV6Y1gZ3MbDDMUX/1sP7ZRnItnGgBEklmsD8vAKgMszkY.:|" /etc/shadow # In case we need sudo #RUN echo "student$stdid ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers WORKDIR {{ STUDDIR }}/student$stdid USER student$stdid ENV NB_USER student$stdid ENV NB_UID $stdid ENV HTTPPORT $httpport RUN git clone https://github.com/snowch/ml-101 {{ STUDDIR }}/student$stdid/ RUN /opt/conda/bin/jupyter-nbconvert --clear-output --inplace {{ STUDDIR }}/student$stdid/*.ipynb EOF # Look at https://stackoverflow.com/questions/34264348/docker-inside-docker-container # and http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ # For security consider using https://github.com/nestybox/sysbox cat > $TMPDIR/docker-compose.yml &#x3C;&#x3C; EOF version: '3.5' services: $NAME$stdid: image: $NAME$stdid build: . #privileged: true ports: - "$httpport:8888" - "$mlport:4040" - "$mlport2:4041" # volumes: # - /var/run/docker.sock:/var/run/docker.sock EOF cat > $TMPDIR/launch-$NAME &#x3C;&#x3C; EOF #!/bin/bash cd $TMPDIR docker-compose up --build -d EOF </code></pre> <p>13- The <code>copy_folder yml</code> playbook is now executed to deploy the notebooks and scripts necessary for the participant to run the workshop. Remember that the participant got a student (with a dedicated student id. For instance: student41) allocated to him/her at the time of the registration. This student id is picked from a range that is allocated for the workshop. The admin decides on the maximum capacity it allocates to a given workshop. <code>copy_folder.yml</code>: This is historically one of very first playbooks we used and therefore a very important one. It performs the necessary actions to deploy and personnalize (by substituting Ansible variables) the selected notebook to the appropriate student home folder</p> <p>14- In the certain cases, some post deployment actions are needed. For instance, you may want to git clone some repository to leverage some data stored there. This can only occur when done with the deployment. Therefore, a <code>post-copy-&#x3C;WKSHP.sh</code> is called.</p> <p>15- Finally, the workshop is now ready to be used by the participant. The backend therefore, needs to inform the frontend of this. To do so, it will perform two API calls:</p> <ul> <li>The first API call will update the password data for the participant's allocated student.</li> <li>The second API call will update the participant's allocated student's status to active.</li> </ul> <p>These changes will trigger the frontend web portal application to send a second email to the participant. This email will contain the necessary information for the participant to connect to its notebooks environment. The participant will then run the workshop. For each workshop, a dedicated time window is allocated. Some workshops will take longer to be run than others. The time windows varies from 2 to 4 hours maximum.The system knows how to set it up so that it will time out. This means that once the participant hits the register button on the frontend web portal, the clock starts ticking.</p> <p>Some background checks take place on the web portal to verify time spent since the registration to a given workshop. As a consequence, a reminder email is sent an hour before the workshop times out. When the bell rings at the end of the class, a new procmail API call is made to the backend server ordering a <strong>CLEANUP</strong> action. The participant can also trigger this action by registering to a new workshop before the end of the current one. He/She will have to provide the necessary information to the frontend web portal in order to end the current workshop.</p> <p>Let's see what is happening on the backend server to perform this <strong>CLEANUP</strong> scenario.</p> <p><img src="/img/wod-blogserie3-cleanup.png" alt="" title="backend server CLEANUP workflow"></p> <p>As you can see, it does not differ much from the <strong>CREATE</strong>. We still need to gather data to interact with the proper workshop from the right student. The <code>.procmail.rc</code> is providing us with this information. Then, the automation kicks in through the <code>procmail-action-sh</code> script.</p> <p>The verb is now <strong>CLEANUP</strong>. As a consequence, step 4 is now <strong>CLEANUP</strong>.</p> <p>Nothing changes from 5 to 9.</p> <p>10- <code>get_wod_completion_ratio()</code> Allows us to retrieve information through a simple computing of the numbers of notebooks cells executed thoughout the different exercices of the workshop a ratio. This enables us to see how much of the workshop is actually run. Participants are asked to fill out a form in a conclusion notebook which is present in every student's workshop's folder.</p> <p>This completion ratio script provides us this data and we store it in our database.</p> <p>11- API call to send the completion ratio figure to the database. This can be later queried to build up a nice reporting dashboard as explained here by my colleague Didier Lalli, in the following <a href="https://developer.hpe.com/blog/open-source-elasticsearch-helped-us-globally-support-virtual-labs/">article</a>.</p> <p>12- <code>erase-student()</code>: Now that we have extracted the necessary data from the student's notebooks, we can perform a cleanup of the student folder.</p> <p>13- <code>cleanup_processes_student()</code>: On top of cleaning up the student folder, we also kill all the allocated student's processes.</p> <p>14- <code>cleanup-&#x3C;workshop>.sh</code>: If any appliance is involved, this task will perform the necessary cleanup processes on the appliance.</p> <p>15- Finally, just like for the creation of a workshop process, we need to tell the frontend that the cleanup is now done. Therefore, several API calls are made to update tables in the database. The new student password is recorded. We also generate a new password at the cleanup phase to prevent unregistered logins. The student status is set to inactive. The capacity figure is incremented by one to make the seat available again.</p> <p>As for the <strong>CREATE</strong> phase, the regular checks occuring on the frontent web portal will get these data and trigger the final email to the participant thanking him for his particpation.</p> <p>Now let's look at the <strong>RESET</strong> scenario.</p> <p><img src="/img/wod-blogserie3-reset.png" alt="" title="backend server RESET workflow"></p> <p>You may wonder about the differences between CLEANUP and RESET. CLEANUP only takes care of students whereas RESET takes care of a larger scope. Let me explain...<strong>CLEANUP</strong> only takes care of student whereas <strong>RESET</strong> takes care of a larger scope.</p> <p>When a <strong>CLEANUP</strong> occurs, it deals with the participant's student workshop and home directory (the workshop directory belonging to the home directory). It cleans up workshop content, ssh keys, skeletons. The <strong>RESET</strong> will delete leftovers from the workshop's exercices. For instance, when one runs the <a href="https://developer.hpe.com/hackshack/workshop/24">Kubernetes 101</a> workshop, he is creating microservices, he's scaling them, and should at the end of the workshop run some <code>kubectl delete</code> commands to clean up everything. However, some participants may forget to run these clean up steps. And the admin needs to make sure that the next participant who will be assigned the same student environment gets a fresh one. Therefore, some measures have to be taken. These measures take place when a reset flag is associated to the workshop in the database.</p> <p>During the <strong>CLEANUP</strong> phase, a check is actually performed to test the presence of this flag through a simple API call on the frontend API-DB server. If the workshop has a reset flag then a dedicated <code>reset-WKSHP.sh</code> script is called and performs the necessary tasks. In the case of Kubernetes 101, it will wipe out any leftovers from the student. In some other cases, it will launch a revert to snapshot script on a virtual machine.</p> <p>Finally, let's consider the <strong>PURGE</strong> scenario.</p> <p><img src="/img/wod-blogserie3-purge.png" alt="" title="backend server PURGE workflow"></p> <p>In a perfect world, we would have covered here what one would somehow expect from any kind of API (GET, PUT, DELETE = CREATE, CLEANUP and RESET). But, this is unfortunately not the case. Even though we did our best to harden the deployment automation, failures might occur. Issues could occur at many different levels. From a backhoe loader cutting an internet line on the very morning of a starting event (preventing you from accessing your remote labs) to unplanned power cuts, or misconfigured power redundancy in pdus assignment, there are many examples possible of human factor related issues. As a result, the Jupyterhub server or an appliance might become unreachable, and the automation of the workshop's deployment might fail.</p> <p>In these very cases, you need to be able to cleanup the mess quickly.</p> <p><img src="/img/wod-blogserie3-failure-purge.png" alt="" title="backend server PURGE / Failure workflow"></p> <ul> <li>frontend - backend communication issues</li> <li>JupyterHub server failure</li> <li>JupyterHub server - appliance server communication issues</li> </ul> <p>The <strong>PURGE</strong> scenario is therefore triggered on Workshops-on-Demand' deployment failures.</p> <p>At the registration time, when the participant hits the register button on the frontend web portal, an entry is automatically created in the database for him. It associates the participant to a student and a workshop. It also registers the date and start time of the workshop, sets the participant status to 'welcome' in the database and a first email is sent to the participant from the frontend web portal welcoming him to the Workshop-on-Demand and stating to him that within a few minutes a second email will be sent along with the necessary information (credentials and url) to connect to the workshop's environment.</p> <p>If for any reason, the deployment of the workshop fails and as a consequence, no API call is made back to the frontend from the backend, the frontend could remain stuck forever and so would the participant. To overcome this, we implemented a check on the frontend web portal to test this welcome status. In a normal scenario, this welcome status gets updated within less than 3 minutes. If the status is not updated within 10 minutes, we consider that something went wrong during the deployment and as a result, a <strong>PURGE</strong> scenario is initiated to clean up both the backend and the frontend sides of the related registration. Of course, depending of the backend sanity, some actions could also fail. But from our experience, both frontend and backend are really reliable.</p> <p>Considering now the most common case: the backend server and frontend servers can communicate but the JupyterHub server has issues to communicate with appliances. In terms of tasks associated to the <strong>PURGE</strong> scenario, you can see that we kept the minimal as there should not be much to clean up on the backend server. Simply consider that it is a <strong>CLEANUP</strong> scenario without any workshop deployment.</p> <p>We call the same tasks to begin with as we still need student ID and workshop ID.</p> <p>We then initiate :</p> <p>9- <code>generate_randompwd()</code>: We always update the student's password for security reasons.</p> <p>10- <code>erase-student()</code>: We perform a cleanup of the student folder.</p> <p>11- API calls to update tables in the database. The new student password is recorded. We also generate a new password at the <strong>PURGE</strong> phase to prevent unregistered logins. The student status is set to inactive. The capacity figure is incremented by one to make the seat available again.</p> <p>An email is then sent to the participant explaining to him that we encountered an issue with the deployment and that we apologize for this. The same email is sent to the admin so he can work on the issue.</p> <p>Now, you should have a clearer view of what is really happening in the background when one registers for a workshop. You can see that I have uncovered many scripts to explain step by step all the stages of a workshop's deployment process. But there is more to be explained. It is obvious that the main function of the backend server is to deploy and run workshops. Nevertheless, as any other server, it cannot live without maintenance.</p> <p>This subject will be at the core of my next article where I will detail how one needs to manage and work with this server on a daily basis. What we usually call Day 2 operations.</p> <p>If we can be of any help in clarifying any of this, please reach out to us on <a href="https://slack.hpedev.io/">Slack</a>. Please be sure to drop back at <a href="https://developer.hpe.com/blog">HPE DEV</a> for a follow up on this. Check out also the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a>! Willing to collaborate with us? Contact us and let's build together some more workshops! Stay tuned!</p>https://developer.hpe.com/2023-April-03/https://developer.hpe.com/2023-April-03/Mon, 03 Apr 2023 13:00:00 GMThttps://developer.hpe.com/creating-dscc-block-storage-resources-using-dscc-ansible-playbooks/https://developer.hpe.com/creating-dscc-block-storage-resources-using-dscc-ansible-playbooks/Thu, 30 Mar 2023 14:02:46 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my previous <a href="https://developer.hpe.com/blog/automating-operations-on-dscc-using-ansible-playbooks/">blog post</a> , I provided an introduction to Ansible playbooks for HPE GreenLake Data Services Cloud Console and how to use them. In this post, I will show you how to create an Ansible playbook to create block storage resources.</p> <p><strong>Use case:</strong></p> <p>The objective of this use case is to provision a Primera volume from scratch using Data Services Cloud Console APIs. This use case covers the creation of the following resources in the following order:</p> <ol> <li>Create a host</li> <li>Create a host group</li> <li>Create a volume set</li> <li>Create volumes</li> <li>Update the volume set with the volumes created</li> <li>Export the volume set to the host created</li> </ol> <p><strong>Let’s look at the Ansible playbook resource-wise</strong></p> <p>Variables in Ansible are similar to variables in any programming language that can be declared and used anywhere in the script. In this use case, certain variables will be used in the playbook and these are declared under the ‘vars’ section. One important variable to watch for here is the “config” variable that provides credentials for the APIs to authenticate the request and the host URL to which the request will be sent. The greenlake_config.json file that is mentioned in the value of the config variable will have the clientID, client secret, and the host URL of the Data Services Cloud Console instance.</p> <pre><code class="language-yaml">vars: - config: "{{ playbook_dir }}/greenlake_config.json" - name: Greenlake DSCC volumeset - system_id: 2M29510B8L - initiators: [] - host_name: host_from_ansible - host_group_name: hostGroupFromAnsible - operating_system: RHE Linux - workload: ORACLE_LOG - volume_set_name: ansibleVolumeSet - volume_name: ansibleVolume </code></pre> <p>Contents of greenlake_config.json:</p> <pre><code class="language-json">{ "host": "https://us1.data.cloud.hpe.com", "client_id": "009e17ef-c356-4066-8b66-4ee10e1128b1", "client_secret": "53a9dd94d61311ec9b7e3ad2f9067b26" } </code></pre> <p><strong>Creation of host</strong></p> <p>A host represents a physical server and a host group represents a collection of physical servers. In the context of the Data Services Cloud Console API, a host is a definition for a group of initiators that belong to a single server and a host group is a group of initiators across servers.</p> <p>To create a host, initiator information is needed and this can be obtained by a GET call to get the initiator resources. One of the initiators can be used in the creation of the host. This means that the creation of a host requires two calls: one to get the initiator and another to create a host. Along with this, you must mention the name of the host, operating system, and a “user_created” flag must be set to true. Variable names will be mentioned between two curly braces and double quotes as shown in the below code snippet.</p> <pre><code class="language-yaml"> - name: Get GreenLake Host Initiators greenlake_host_initiator_facts: config: "{{ config }}" - debug: var=host_initiators - set_fact: initiators='{{ initiators + [item.id] }}' loop: "{{ host_initiators }}" when: item.protocol == 'FC' and item.hosts|length &#x3C;= 0 - debug: var=initiators - name: Create GreenLake DSCC Host greenlake_host: config: "{{ config }}" state: present data: initiator_ids: - "{{initiators.0}}" name: "{{host_name}}" operating_system: "{{operating_system}}" user_created: True - debug: var=hosts </code></pre> <p><strong>Creation of host group</strong></p> <p>A host group can be created using the host created in the previous step. The tasks of this Ansible playbook will be executed sequentially, so the creation of the host group task executes only after the host creation task is successfully completed.</p> <p>Provide the name of the host group, host Id, and make sure that the “user_created” flag is set to true.</p> <pre><code class="language-yaml"> - name: Create GreenLake DSCC Host Group greenlake_host_group: config: "{{ config }}" state: present data: name: "{{host_group_name}}" hostIds: - "{{hosts.0.id}}" user_created: True - debug: var=host_groups </code></pre> <p><strong>Creation of volume set</strong></p> <p>The device type represents whether it’s a Primera or Nimble volume. If it is 1, then it is a Primera\Alletra 9k volume and if it is 2, then it is a Nimble\Alletra 6k volume. Mandatory parameters are app_set_name, app_set_importance, app_set_type (which indicates what kind of workload is required, like Oracle Database). Optional parameters are commented on in the below snippet.</p> <blockquote> <p>Note: The request body for Nimble volume creation may vary, so please refer to the documentation for specifics.</p> </blockquote> <pre><code class="language-yaml"> - name: Create GreenLake DSCC Volume Set greenlake_volumeset: config: "{{ config }}" device_type: 1 system_id: "{{system_id}}" state: present data: # app_set_business_unit: "HPE" # app_set_comments: "Edit" app_set_importance: "MEDIUM" app_set_name: "{{volume_set_name}}" # name: "ansible_volume_set_1" app_set_type: "{{workload}}" # members: ["ansible-vol1", "ansible-vol2"] - debug: var=volume_sets </code></pre> <p><strong>Creation of volume (Primera)</strong></p> <p>The mandatory fields are the name, size, and user CPG. Other parameters are optional.</p> <pre><code class="language-yaml"> - name: Create GreenLake DSCC volume greenlake_volume: config: "{{ config }}" system_id: "{{system_id}}" state: present data: comments: "Ansible library test" count: 2 #Optional data_reduction: True #Optional name: "{{volume_name}}" size_mib: 16384 snap_cpg: "SSD_r6" #Optional snapshot_alloc_warning: 10 #Optional user_alloc_warning: 10 #Optional user_cpg: "SSD_r6" - debug: var=volumes </code></pre> <p><strong>Updating the volume set</strong></p> <p>The volume set will be updated with the volumes created earlier. For this, provide the id of the volume set and the names of the volumes created. Since these are created dynamically during the playbook execution, these are referred to as variables.</p> <pre><code class="language-yaml"> - name: Update GreenLake DSCC Volume Set with volumes greenlake_volumeset: config: "{{ config }}" device_type: 1 system_id: "{{system_id}}" state: present data: id: "{{volume_sets.0.id}}" add_members: - "{{volume_name}}.0" - "{{volume_name}}.1" - debug: var=volume_sets </code></pre> <p><strong>Exporting the volume set to the host group</strong></p> <p>Now, the volume set created needs to be exported to the host group created earlier. Provide the system id, host group id, and the volume set id in the form of variables.</p> <pre><code class="language-yaml"> - name: Export GreenLake DSCC Volume Set from system greenlake_volumeset: config: "{{ config }}" device_type: 1 system_id: "{{system_id}}" state: export data: id: "{{volume_sets.0.id}}" host_group_ids: - "{{host_groups.0.id}}" - debug: var=volume_sets </code></pre> <p>I hope you found this blog post helpful. This is just a sample use case one can implement to create resources on HPE GreenLake Data Services Cloud Console. Admins/users can come up with multiple use cases that can be used to manage the Data Services Cloud Console resources, like cleaning the resources, monitoring the resources, etc. Keep an eye out for new blogs and videos that are on track to be released related to the automation of Data Services Cloud Console operations.</p>https://developer.hpe.com/meet-our-new-hpe-dev-team-representative-at-hpe-technology-and-solution-summit-2023/https://developer.hpe.com/meet-our-new-hpe-dev-team-representative-at-hpe-technology-and-solution-summit-2023/Thu, 30 Mar 2023 13:29:10 GMT<p><img src="/img/tss2023.png" alt="" title="HPE TECHNOLOGY AND SOLUTIONS SUMMIT 2023"></p> <p>HPE Technology and Solution Summit 2023 (HPE TSS 2023) goes physical again this year in sunny Barcelona. After being relegated to virtual sessions these last couple of times, you will once again enjoy the pleasure of being able to engage with an HPE Developer Community representative in person. This time, all our sessions will be held by our new team member, Mathieu Losmede.</p> <p>HPE TSS is the largest HPE technical event that is held in the Europe, Middle East, and Africa (EMEA) geography. Whether you are an HPE Presales or Partner, you will have the opportunity to attend technical breakout, Hands-on Labs, as well as strategic keynotes, and also finally get to meet your peers in person!</p> <p>The decision to hold this event at a physical location came late this year. As a result, the HPE Developer Community team will be unable to bring the complete, well-loved Hack Shack to the event. And we will only be able to spare one member of our team to attend and run our sessions. However, we will be offering a number of sessions, hands-on labs, and a few surprises.</p> <p><strong>Meet our new team member: Mathieu Losmede</strong></p> <p>Mathieu joined the HPE Developer Community team last summer. He was present at Discover 2022 with us at Las Vegas in the Hack Shack. You can see him between Patrick and myself on the following picture. He will be the face of the HPE Developer Community team in Barcelona.</p> <p><img src="/img/fred-6-b-3-512-.jpg" alt=""></p> <p>Mathieu came in with a strong HPE GreenLake background. He was part of the HPE GreenLake Cloud Service (aka Greenlake Central) SRE - Devops team. As our team's focus has recently shifted more heavily towards the HPE GreenLake Cloud Platform, we were very fortunate to have him join our group.</p> <p>Mathieu will be all over the place in Barcelona. One might wonder whether he managed to discover the secret of ubiquity. He will be delivering several technical breakouts either on his own or in partnership with other business units (BU) representatives (HPE GreenLake, Storage, Compute, and Aruba). He will also be co-presenting in other BUs' sessions. On top of these, he will deliver three hands-on labs based on our well-known <a href="https://developer.hpe.com/hackshack/workshops">Workshops-on-Demand</a> project.</p> <p><img src="/img/tss2023-wod.png" alt="" title="Workshops-on-Demand"></p> <p>Look for the HPE Developer Community track in the HPE TSS 2023 agenda to get details about all our sessions.</p> <p><strong>Techies just wanna have fun!</strong></p> <p>As always,the HPE Developer Community team brings in new ways to learn in a fun way. This year at TSS 2023 in Barcelona, we propose you to participate to a treasure hunt. Do not worry, no long walking needed there, we won't be sending you looking for clues in the city of Barcelona. All you will be needing is an internet browser and a few minutes. The treasure hunt will be available from the HPE TSS mobile app. It will start on Monday the 22nd of May at 12.00 pm CEST and end on Thursday the 25th of May at 10.00 CEST.</p> <p>Three winners will be selected. The winners will receive their prizes on Thursday, May 25th on the show floor at the HPE GreenLake booth at 12:15pm CEST. Please read the Terms and Conditions for participation in the HPE Developer Community Treasure Hunt found <a href="https://developer.hpe.com/hackshack/hpetss2023-treasurehunt-terms-conditions/">here</a>.</p> <p><img src="/img/echodotgen3.png" alt="" title="Amazon Echo Dot3"></p> <p>If you are going to TSS 2023, make sure to sign up for the HPE Developer Community sessions. As you know, we value the importance of this technical event. We expect you do as well and appreciate that you have regularly rated our sessions highly. We look forward to reconnecting with you there. As does Mathieu :-).</p>https://developer.hpe.com/automating-operations-on-dscc-using-ansible-playbooks/https://developer.hpe.com/automating-operations-on-dscc-using-ansible-playbooks/Wed, 29 Mar 2023 14:00:40 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Automation is one of the top trends in technology and the pace of automation is accelerating with more companies opting for developing fully automated systems. Automation reduces time, effort, cost, and manual errors while increasing efficiency and productivity. Gone are those days when many complex coding skills were required to implement automation. Now, there are many low-code tools available in the market, like Ansible, that make automation easier.</p> <p>In this blog post, I am excited to be able to introduce the Ansible playbooks for HPE GreenLake Data Services Cloud Console and show you how to use them. Along with the <a href="https://github.com/HewlettPackard/greenlake-data-services-python">Python SDK</a> for HPE GreenLake Data Services Cloud Console, these playbooks should help you in your efforts to automate HPE GreenLake Data Services through an infrastructure-as-code approach.</p> <p>Ansible is an open-source IT automation tool that automates provisioning, configuration management, application deployment, and many other IT processes. Two main features that make Ansible the best choice for automation are:</p> <ul> <li>Ansible does not require any programming.</li> <li>Idempotence is offered as a built-in feature of many of the Ansible modules. This means the result of performing a task once is the same as performing it multiple times without any intervening actions.</li> </ul> <p><strong>Why do we need Ansible playbooks for HPE GreenLake Data Services Cloud Console?</strong></p> <p>Ansible helps the users/admins automate the deployment of resources and applications without the manual overhead of creating everything from scratch. These playbooks can be configured with conditions, variables, and tasks. Currently, simple playbooks, like performing CRUD operations on the resources, are available. These playbooks can be considered basic building blocks and can be reused to build simple-to-complex use cases.</p> <p><strong><a href="https://github.com/HewlettPackard/greenlake-data-services-ansible">Ansible Modules for HPE GreenLake Data Services Cloud Console</a>:</strong></p> <p>The following Ansible modules are currently available for Data Services Cloud Console. You can use the samples given or customize it.</p> <p><em>greenlake_audit_events_facts</em> - Get details of audit events</p> <p><em>greenlake_host</em> - Manage host</p> <p><em>greenlake_host_facts -</em> Get details of a host</p> <p><em>greenlake_host_group -</em> Manage host group</p> <p><em>greenlake_host_group_facts -</em> Get details of a host group</p> <p><em>greenlake_host_initiator_facts -</em> Get details of a host initiator</p> <p><em>greenlake_storage_system_facts -</em> Get details of a storage system</p> <p><em>greenlake_volume -</em> Manage volume</p> <p><em>greenlake_volume_facts -</em> Get details of a volume</p> <p><em>greenlake_volumeset -</em> Manage volume set</p> <p><em>greenlake_volumeset_facts -</em> Get details of a volume set</p> <p><strong>Prerequisites to use these Ansible playbooks:</strong></p> <ol> <li>Any machine with Python 3.8 or newer installed. This includes Red Hat, Debian, CentOS, macOS, any of the BSDs, and so on. (Microsoft Windows is not supported - see Note below)</li> <li>The latest and most stable version of Ansible must be installed. (current version is 2.9)</li> <li>HPE GreenLake Data Services Python SDK. (Installation procedure is mentioned below)</li> <li>Cloning the GitHub repo that has these playbooks</li> <li>Setup ANSIBLE_LIBRARY and ANSIBLE_MODULE_UTILS environment variables</li> </ol> <blockquote> <p>Note: If you are using Windows 10, then an Ubuntu terminal called Windows Subsystem for Linux(WSL) can be used.</p> </blockquote> <p><strong>Installing Ansible on Ubuntu</strong></p> <ol> <li> <p>Make sure your system’s package index is up to date. Refresh the package index with the following command:</p> <pre><code class="language-shell">$sudo apt update </code></pre> </li> <li> <p>Next, install Ansible on Ubuntu with the command:</p> <pre><code class="language-shell">$sudo apt install ansible </code></pre> </li> </ol> <p>The installation will prompt you to press Y to confirm, with the rest of the installation process being automated.</p> <ol start="3"> <li> <p>Check the version of Ansible by using the following command:</p> <pre><code class="language-shell">$ansible --version </code></pre> </li> </ol> <p>For other operating systems, please refer to the official <a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html">Ansible documentation</a>.</p> <p><strong>Installing the HPE GreenLake Data Services Python SDK</strong></p> <p>These Ansible playbooks use the Python libraries of the HPE GreenLake Python SDK. Install the Python SDK using the steps mentioned <a href="https://github.com/HewlettPackard/greenlake-data-services-python#installation--usage">here</a>.</p> <p><strong>Cloning the Data Services Cloud Console GitHub repo</strong> To clone the repo, execute the following command on the machine where you installed Ansible:</p> <pre><code class="language-shell">git clone https://github.com/HewlettPackard/greenlake-data-services-ansible.git </code></pre> <p>This repo mainly consists of two folders:</p> <ol> <li><strong>Libraries</strong>: These are Python libraries that will be used by the Ansible playbooks to perform CRUD operations on Data Services Cloud Console resources.</li> <li><strong>Examples</strong>: This folder consists of sample Ansible playbooks that perform CRUD operations on Data Services Cloud Console resources. With these examples, one can start building use cases.</li> </ol> <p><strong>Setting up environment variables</strong></p> <p>Set the ANSIBLE_LIBRARY variable to the greenlake-data-services-ansible/library directory of the repo -</p> <p><code>ANSIBLE_LIBRARY=/home/admin/greenlake-data-services-ansible/library</code></p> <p>Set the ANSIBLE_MODULE_UTILS to the module_utils directory under greenlake-data-services-ansible/library directory -</p> <p><code>ANSIBLE_MODULE_UTILS=/home/admin/greenlake-data-services-ansible/library/module_utils</code></p> <p><strong>Usage</strong></p> <p>Let’s have a look at an example used to perform Data Services Cloud Console operations on a host using an Ansible playbook. For any Ansible playbook to be used, an inventory file is required. The Ansible inventory file defines the hosts and groups of hosts on which commands, modules, and tasks in a playbook operate. In this example, we are calling REST APIs from our local machine. This file can be placed anywhere and the path of this file can be given during the Ansible playbook execution. Create an inventory file, name it “hosts” (the name of the file can be anything), and update it with the following details:</p> <p><code>[localhost] 127.0.0.1 ansible_connection=local ansible_python_interpreter=/home/admin/ansible-env/bin/python</code></p> <p>The IP address given is localhost, which means operations are performed on the local host. Provide the Python interpreter location on your system. Name this file “hosts”.</p> <p>Let's look at a sample of the host file. Ansible playbooks contain a list/array of tasks that will be performed sequentially. The below code snippet has the following tasks.</p> <ol> <li> <p><strong>Creation</strong> of host</p> <p>In this block, the input request parameters are provided under the section ‘data’. In this sample, only required fields for the REST API call are provided such as name, initiator_ids, user_created flag, and operating system.</p> </li> <li> <p><strong>Update</strong> a host</p> <p>In an update request, one can update the name, and change the initiators. These input parameters can be provided under the ‘data’ section.</p> </li> <li> <p><strong>Delete</strong> a host</p> <p>To delete a host, all you need to provide is the name of the host.</p> <pre><code class="language-yaml">--- - hosts: all vars: - config: "{{ playbook_dir }}/greenlake_config.json" - name: "Greenlake DSCC Host" tasks: - name: Create GreenLake DSCC Host greenlake_host: config: "{{ config }}" state: present data: initiator_ids: - "b015d393e2274592a37cc7a579c8b0ca" name: "hostAnsibleTest" operating_system: "RHE Linux" user_created: True # new_name: "hostAnsibleNameUpdated1" - debug: var=hosts - name: Update GreenLake DSCC Host Name greenlake_host: config: "{{ config }}" state: present data: initiator_ids: - "b015d393e2274592a37cc7a579c8b0ca" name: "hostAnsibleTest" operating_system: "RHE Linux" user_created: True new_name: "hostAnsibleTestUpdated" - name: Delete GreenLake DSCC Host greenlake_host: config: "{{ config }}" state: absent data: name: "hostAnsibleTestUpdated" - debug: var=hosts </code></pre> </li> </ol> <p>To execute the Ansible playbook, execute the following command:</p> <pre><code class="language-shell">$ansible-playbook examples/greenlake_host.yaml --i ../hosts </code></pre> <p>This is what the result of a playbook execution looks like.</p> <pre><code class="language-shell">(ansible_env) anusha@MUCILUR2I5:~/greenlake-data-services-ansible$ ansible-playbook examples/greenlake_host.yaml -i ../hosts [WARNING]: Found variable using reserved name: name PLAY [all] ******************************************************************************************************************************************************************************************************** TASK [Gathering Facts] ******************************************************************************************************************************************************************************************** ok: [127.0.0.1] TASK [Create GreenLake DSCC Host] ********************************************************************************************************************************************************************************* changed: [127.0.0.1] TASK [debug] ****************************************************************************************************************************************************************************************************** ok: [127.0.0.1] => { "hosts": [ { "associated_links": [ { "resourceUri": "/api/v1/initiators?filter=hostId in (63e338328779477481d44d906866b60b)", "type": "initiators" } ], "associated_systems": null, "comment": null, "console_uri": "/data-ops-manager/host-initiators/63e338328779477481d44d906866b60b", "contact": null, "customer_id": "eb00678a466b11ec94d66ec0ab988305", "edit_status": "Not_Applicable", "fqdn": null, "generation": 1655290314, "host_groups": [], "id": "63e338328779477481d44d906866b60b", "initiators": [ { "address": "c3:33:ff:58:5f:19:00:1e", "id": "b015d393e2274592a37cc7a579c8b0ca", "ip_address": null, "name": "Host Path C333FF585F19001E (1:3:2)", "protocol": "FC", "systems": [ "2M29510B8N", "2M29510B8L" ] } ], "ip_address": null, "location": null, "marked_for_delete": false, "model": null, "name": "hostAnsibleTest", "operating_system": "RHE Linux", "persona": null, "protocol": null, "subnet": null, "systems": [ "2M29510B8N", "2M29510B8L" ], "type": "host-initiator", "user_created": true } ] } TASK [Update GreenLake DSCC Host Name] **************************************************************************************************************************************************************************** changed: [127.0.0.1] TASK [Delete GreenLake DSCC Host] ********************************************************************************************************************************************************************************* changed: [127.0.0.1] TASK [debug] ****************************************************************************************************************************************************************************************************** ok: [127.0.0.1] => { "hosts": [ { "associated_links": [ { "resourceUri": "/api/v1/initiators?filter=hostId in (63e338328779477481d44d906866b60b)", "type": "initiators" } ], "associated_systems": null, "comment": null, "console_uri": "/data-ops-manager/host-initiators/63e338328779477481d44d906866b60b", "contact": null, "customer_id": "eb00678a466b11ec94d66ec0ab988305", "edit_status": "Update_In_Progress", "fqdn": null, "generation": 1655290320, "host_groups": [], "id": "63e338328779477481d44d906866b60b", "initiators": [ { "address": "c3:33:ff:58:5f:19:00:1e", "id": "b015d393e2274592a37cc7a579c8b0ca", "ip_address": null, "name": "Host Path C333FF585F19001E (1:3:2)", "protocol": "FC", "systems": [ "2M29510B8N", "2M29510B8L" ] } ], "ip_address": null, "location": null, "marked_for_delete": false, "model": null, "name": "hostAnsibleTestUpdated", "operating_system": "RHE Linux", "persona": null, "protocol": null, "subnet": null, "systems": [ "2M29510B8N", "2M29510B8L" ], "type": "host-initiator", "user_created": true } ] } PLAY RECAP ******************************************************************************************************************************************************************************************************** 127.0.0.1 : ok=6 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>These Ansible playbooks can be used to fetch details as well. Under the examples folder, there are files with the suffix ‘_facts’ which specify that it is used to get the details of a resource. For example, take a look at the ‘greenlake_host_facts.yaml’. This playbook is used to get the details of the host.</p> <pre><code class="language-yaml">--- - hosts: all vars: - config: "{{ playbook_dir }}/greenlake_config.json" - name: "Get Hosts" tasks: - name: Get GreenLake Hosts greenlake_host_facts: config: "{{ config }}" # id: "fbc6f4e700154bff8cdfebaf10c3b965" # params: # limit: 10 # int | Number of items to return at a time (optional) # offset: 5 # int | The offset of the first item in the collection to return (optional) - debug: var=hosts </code></pre> <p>In this playbook, you have an option to add filters to it like ID, limit, offset, etc., or else you can get the details of all hosts available. Use the following command to execute the Ansible playbook:</p> <pre><code class="language-shell">$ansible-playbook examples/greenlake_host_facts.yaml --i ../hosts </code></pre> <p>The output looks like this:</p> <pre><code class="language-shell">(ansible_env) anusha@MUCILUR2I5:~/greenlake-data-services-ansible$ ansible-playbook examples/greenlake_host_facts.yaml -i ../hosts [WARNING]: Found variable using reserved name: name PLAY [all] ******************************************************************************************************************************************************************************************************** TASK [Gathering Facts] ******************************************************************************************************************************************************************************************** ok: [127.0.0.1] TASK [Get GreenLake Hosts] **************************************************************************************************************************************************************************************** ok: [127.0.0.1] TASK [debug] ****************************************************************************************************************************************************************************************************** ok: [127.0.0.1] => { "hosts": [ { "associated_links": [ { "resourceUri": "/api/v1/initiators?filter=hostId in (e1b09ecf837b46cca365044bf7237bc3)", "type": "initiators" }, { "resourceUri": "/api/v1/host-initiator-groups?filter=hostId in (e1b09ecf837b46cca365044bf7237bc3)", "type": "host-groups" } ], "associated_systems": [ "2M29510B8N", "2M29510B8L" ], "comment": null, "console_uri": "/data-ops-manager/host-initiators/e1b09ecf837b46cca365044bf7237bc3", "contact": null, "customer_id": "eb00678a466b11ec94d66ec0ab988305", "edit_status": "Not_Applicable", "fqdn": null, "generation": 1653561376, "host_groups": [ { "id": "0fb8a5f6616c4465a77a43cb7841e105", "marked_for_delete": false, "name": "HostGroup-01", "systems": [ "2M29510B8N", "2M29510B8L" ], "user_created": true } ], "id": "e1b09ecf837b46cca365044bf7237bc3", "initiators": [ { "address": "c3:33:ff:58:5f:19:00:04", "id": "1b6b97741b21480bb916ca3b8c0a2c6e", "ip_address": null, "name": "Host Path C333FF585F190004 (0:3:1)", "protocol": "FC", "systems": [ "2M29510B8N", "2M29510B8L" ] }, { "address": "c3:33:ff:58:5f:19:00:24", "id": "1bce3ffd6a0d4b74a8520ddc55e2a1eb", "ip_address": null, "name": "Host Path C333FF585F190024 (1:3:1)", "protocol": "FC", "systems": [ "2M29510B8N", "2M29510B8L" ] } ], "ip_address": null, "location": null, "marked_for_delete": false, "model": null, "name": "Host-01", "operating_system": "Ubuntu", "persona": null, "protocol": null, "subnet": null, "systems": [ "2M29510B8N", "2M29510B8L" ], "type": "host-initiator", "user_created": true } ] } PLAY RECAP ******************************************************************************************************************************************************************************************************** 127.0.0.1 : ok=3 changed=0 unreacha-ble=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>In the current release, only critical resources are supported. New resources will be added moving forward. This is just the beginning. In the future, there are many use cases to cover.</p> <p>In this blog, I gave you a preview of the Ansible playbooks for HPE GreenLake Data Services Cloud Console. This SDK is a Beta version with lots of rooms for improvement. I would urge you to try out the features and keep us engaged with your feedback so that we come out with a better version for the GA. Also, look out for other posts in the HPE Developer blog that are related to Data Services Cloud Console.</p>https://developer.hpe.com/introducing-python-sdk-for-dscc/https://developer.hpe.com/introducing-python-sdk-for-dscc/Wed, 29 Mar 2023 13:45:50 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>In my previous <a href="https://developer.hpe.com/blog/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/">blog</a>, I discussed how to generate a Software Development Kit (SDK) from the HPE GreenLake Data Services Cloud Console Open API spec using a third-party tool called OpenAPI generator. Today, I am going to talk more in detail on the Python SDK generated using the tool. As we progress, we will discuss the implementation aspects, and then take up a use-case to better understand how to use the SDK.</p> <p><strong>APIs and SDKs</strong></p> <p>APIs are a list of functions libraries that are used to communicate with a web service (via secure HTTP), whereas an SDK is a development kit that facilitates the usage of these APIs. APIs enable any software developer to create business opportunities by leveraging the capabilities provided by the API to extend their application whereas the SDK facilitates the process for developers. An SDK is a collection of software tools and programs for a specific application platform that allows developers to manipulate the functions supported by the service. These can be considered as a wrapper on the top of the APIs, making the code consumable by the application.</p> <p>One thing to note in the context of HPE GreenLake Data Services Cloud Console is that the user always gets access to the latest Data Services Cloud Console API version through the SDKs. How? The SDK is designed and deployed (using CI/CD pipelines such as Jenkins) in such a way that with every new release of the Data Services Cloud Console Open API spec, the SDKs get updated automatically. Thus, keeping it up-to date without any manual intervention. This also reduces time which is spent waiting for updates with newer features.</p> <p><strong>Introducing Python SDK for HPE GreenLake Data Services Cloud Console</strong></p> <p>Due to the wide adoption of Python, and for the Python lovers out there who did not have an option to achieve their automation goals, we have the Python SDK available now. You can access the SDK on this <a href="https://github.com/HewlettPackard/greenlake-data-services-python">github </a>page.</p> <p>This SDK contains the following:</p> <ul> <li>Documentation (under docs folder)</li> <li>Code libraries (under greenlake_data_services folder)</li> <li>Test file (under test folder)</li> <li>README file</li> <li>The Python libraries that are required to run this SDK (requirements.txt &#x26; test-requirements.txt)</li> </ul> <p><strong>Requirements</strong>:</p> <p>Python (>=3.5) is required to run the scripts. Run the following command to install all the required packages that the SDK is dependent on:</p> <pre><code class="language-shell">pip install –r requirements.txt </code></pre> <p>Example usage:</p> <p>Let us consider Audits as an example. Audit events are a collection of tasks performed by users. The below code snippet uses a GET method to fetch the details of audit events, like task ID, user email, state, etc.</p> <p>The sample code is provided in the <a href="https://github.com/HewlettPackard/greenlake-data-services-python/blob/main/docs/AuditEventApi.md">documentation</a> of this resource. Take the sample code and replace the BEARER_TOKEN with the access token. Generate the access token as mentioned in this <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">blog</a>.</p> <p>Save the file as GetAudits.py</p> <pre><code class="language-python">import time import greenlake_data_services from pprint import pprint from greenlake_data_services.api import audit_api from greenlake_data_services.model.audit_bad_request import AuditBadRequest from greenlake_data_services.model.audit_internal_server_error import AuditInternalServerError from greenlake_data_services.model.audit_results import AuditResults from greenlake_data_services.model.audit_service_unavailable import AuditServiceUnavailable from greenlake_data_services.model.audit_user_forbidden import AuditUserForbidden from requests_oauthlib import OAuth2Session from requests.auth import HTTPBasicAuth from oauthlib.oauth2 import BackendApplicationClient CLIENT_SECRET = "36f6d0fa92ea11ecae650a4cf4dda9cf" CLIENT_ID = "33a7fd43-4d63-41d5-8a10-1494eb5430c9" client = BackendApplicationClient(CLIENT_ID) oauth = OAuth2Session(client=client) auth = HTTPBasicAuth(CLIENT_ID, CLIENT_SECRET) token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth) access_token = token["access_token"] # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure Bearer authorization (JWT): JWTAuth configuration = greenlake_data_services.Configuration( access_token = access_token, host = "https://us1.data.cloud.hpe.com" ) # Enter a context with an instance of the API client with greenlake_data_services.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = audit_api.AuditApi(api_client) filter = "filter_example" # str | Filter criteria - e.g. state eq Failure and occurredAt gt 2020-09-08T16:51:33Z (optional) limit = 1 # int | The number of results to return (optional) offset = 1 # int | The number of results to skip (optional) sort = "sort_example" # str | A comma separated list of properties to sort by, followed by a direction indicator (\"asc\" or \"desc\"). If no direction indicator is specified the default order is ascending. - e.g. state,version desc. Currently only support sorting by 1 property per request (optional) select = "select_example" # str | A list of properties to include in the response. Currently only support returning of all fields. (optional) try: # GET audit-events api_response = api_instance.audit_events_get(limit=limit, offset=offset)#filter=filter, , sort=sort, select=select) pprint(api_response) except greenlake_data_services.ApiExc eption as e: print("Exception when calling AuditApi->audit_events_get: %s\n" % e) </code></pre> <p>And run the script.</p> <pre><code class="language-python">$python GetAudits.py </code></pre> <p>The output of scripts are in JSON format. For reference, there are example scripts available for all the resources listed on the Data Services Cloud Console API spec. Check them out on the SDK, after which you can customize it per your use-case.</p> <pre><code class="language-json">{'items': [{'associated_resource': {'id': '/api/v1/tasks/3b0139a1-478b-4a24-9811-9a1e072b5744', 'name': 'Delete [Vol-01.1]', 'type': 'tasks'}, 'code': '', 'context_id': '18337cba-c12f-4c16-a1c2-755471de8ed1', 'customer_id': 'eb00678a466b11ec94d66ec0ab988305', 'id': '9794158b-fd2e-4cb6-bbf6-c9df3867d035', 'loggedAt': datetime.datetime(2022, 6, 8, 10, 12, 31, tzinfo=tzutc()), 'message': 'Parent Task : Delete [Vol-01.1] - Completed', 'occurred_at': '2022-06-08T10:12:31Z', 'permission': '', 'scope': '', 'source': '/api/v1/storage-systems/device-type1/2M29510B8L/volumes/0c24e55f12e5609ca0e2de527dfa8426', 'source_ip_address': 'fleet-gql-data-graph:4000', 'state': 'Success', 'task_id': '', 'unique_id': 'audit.events+2+24511', 'user_email': 'anusha.y@hpe.com', 'version': 1}], 'page_limit': 1, 'page_offset': 1, 'total': 6978} </code></pre> <p><strong>Next steps</strong></p> <p>Now that you have access to the Python SDK for HPE GreenLake Data Services Cloud Console, use it to create automation for any use-case that requires the use of Data Services Cloud Console APIs, right from your console. In <a href="https://developer.hpe.com/blog/automating-operations-on-dscc-using-ansible-playbooks/">my next blog</a>, I talk about how to use Ansible playbooks to achieve your automation goals for Data Services Cloud Console. Stay tuned!</p>https://developer.hpe.com/announcing-chapel-1-30-0/https://developer.hpe.com/announcing-chapel-1-30-0/Fri, 24 Mar 2023 00:29:50 GMT<p>External Blog</p>https://developer.hpe.com/bare-metal-provisioning-on-hpe-greenlake-using-terraform/https://developer.hpe.com/bare-metal-provisioning-on-hpe-greenlake-using-terraform/Mon, 20 Mar 2023 18:30:57 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <h1>Introduction</h1> <p>The HPE GreenLake for Private Cloud Enterprise: Bare Metal service offering enables you to create dedicated compute instances deployed on a physical IT infrastructure, facilitating on-demand scalability, convenience, and agility as a cloud service.</p> <p>Using bare metal service, you can create compute-instances provisioned with specific operating systems, network connections, one or more public SSH keys, and optional network-attached storage volumes. The service can be accessed via GUI as well as via public APIs, enabling developers to use an Infrastructure-as-Code tool to build, change, and manage infrastructure in a consistent and repeatable way.</p> <p>The HPE GreenLake Terraform provider <strong>hpegl</strong> by HPE GreenLake provides Infrastructure-as-Code support for HPE GreenLake Cloud Services. Using the <strong>hpegl</strong> Terraform provider, you can automate the management of your infrastructure. You can provision OS on bare metal, spin up virtual machines and bring up a Kubernetes cluster; starting right from bare metal all the way up the stack to your desired configurations and applications.</p> <p>In this blog post, I will walk you through the steps required to use the HPE GreenLake Terraform Provider to deploy and further manage bare metal compute instances.</p> <h1>Preparing an Infrastructure-as-Code implementation</h1> <h2>Setting up an API client for access</h2> <p>You need an API client to authenticate against HPE GreenLake.</p> <p>Follow the below steps for an API client creation:</p> <ol> <li>From the HPE GreenLake platform, launch the HPE GreenLake Central console for the appropriate tenant. Under the settings icon on the tenant Dashboard page, select the <strong>User Management</strong> option.</li> </ol> <p><img src="/img/greenlake_console_usermanagement.png" alt="User Management" title="User Management"></p> <ol start="2"> <li>Under the API Clients tab, click on <strong>Create API Client</strong>.</li> </ol> <p><img src="/img/greenlake_console_createapiclient.png" alt="Create API Client" title="Create API Client"></p> <ol start="3"> <li>Enter a <strong>Name</strong> (mandatory field) and <strong>Description</strong> (optional) for the API client, and click on <strong>Create</strong> button.</li> </ol> <p><img src="/img/greenlake_conolse_apiclient_create.png" alt="Create API Client" title="Create API Client"></p> <ol start="4"> <li>Ensure you make a note of the <strong>Issuer</strong>, <strong>Client ID</strong>, and <strong>Client Secret</strong> before clicking on the <strong>Close</strong> button. These details will be exported as environment variables in the next section.</li> </ol> <p><img src="/img/greenlake_conolse_apiclient_created.png" alt="API Client Created" title="API Client Created"></p> <ol start="5"> <li> <p>In the API Clients page, select the newly created client, and click on <strong>Create Assignment</strong> button.</p> </li> <li> <p>Assign the roles <strong>BMAAS Access Viewer</strong> and <strong>BMAAS Access Project Contributor</strong> on the <strong>Space: Default.</strong></p> </li> </ol> <p><img src="/img/greenlake_console_createbmaasassignment.png" alt="BMaaS Roles" title="BMaaS Roles"></p> <p>The API client is now ready to be used to run the Terraform resources.</p> <h2>Select the Compute group ID</h2> <p>A compute group is a logical grouping of bare metal resources (compute instances, networks, SSH keys, etc.) that a team of cloud consumers can consume. You must specify the compute group ID to interact with the bare metal service resources. You can create a new compute group or select the existing one from the HPE GreenLake console. Make a note of the compute group ID because you need it to set a variable.</p> <p><strong>Note: Compute group is an AKA Project.</strong></p> <ol> <li>Navigate to HPE GreenLake for Private Cloud Services card -> Bare Metal -> Compute Groups.</li> </ol> <p><img src="/img/compute_group_list.png" alt="BMaaS Compute Groups" title="BMaaS Compute Groups"></p> <ol start="2"> <li>Click on the desired compute group and then extract the ID from the browser URL seen at that time.<br> This will be exported in the environment variable <strong>HPE_METAL_PROJECT_ID</strong> in the later section.</li> </ol> <p><img src="/img/compute_group_id.png" alt="Compute group ID" title="Compute group ID"></p> <h2>Install Terraform</h2> <p>Next, get your system ready to run Terraform. In case this has not been done yet:</p> <ol> <li> <p>Download and install Terraform, version v0.13 or later.<br> For more information, see <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">https://learn.hashicorp.com/tutorials/terraform/install-cli</a>.</p> </li> <li> <p>Verify the installation with <strong>terraform -help.</strong></p> </li> </ol> <p>At this point, you are ready to start building your infrastructure description file.</p> <h1>Deploy Compute Instance</h1> <h2>Select Terraform provider with bare metal service configurations</h2> <ol> <li> <p>Export the following environment variables on your setup.</p> <p>Export the Tenant ID:</p> <pre><code class="language-shell">export HPEGL_TENANT_ID="&#x3C;Tenant ID> </code></pre> <p>Export the API Client credentials that you obtained when you created an API Client within HPE GreenLake Central:</p> <pre><code class="language-shell">export HPEGL_USER_ID="&#x3C;API Client ID>" export HPEGL_USER_SECRET="&#x3C;API Client Secret>" export HPEGL_IAM_SERVICE_URL="&#x3C;Issuer URL>" </code></pre> <p>Export the Compute group ID:</p> <pre><code class="language-shell"># compute group/project ID export HPEGL_METAL_PROJECT_ID="&#x3C;Compute group ID>" </code></pre> <p>Export bare metal service REST URL:</p> <pre><code class="language-shell"># Production Environment: https://client.greenlake.hpe.com/api/metal # Integration Environment: https://client.greenlake.hpe-gl-intg.com/api/metal # local development: http://localhost:3002 export HPEGL_METAL_REST_URL="&#x3C;Metal Service REST Base URL>" </code></pre> </li> <li> <p>Configure the Terraform provider.</p> <p>Create an empty folder and put a file in it called <strong>main.tf</strong> with the following contents:</p> <p><strong>main.tf:</strong></p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "HPE/hpegl" version = ">= 0.3.12" } } } provider "hpegl" { # metal block for configuring bare metal resources. metal { } } </code></pre> <p>This tells Terraform that you are going to be using HPE/hpegl as your provider and you are using the bare metal service.</p> </li> </ol> <h2>Write resource configuration for compute instance creation</h2> <p>To deploy compute instance, you need to use the <strong>hpegl_metal_host</strong> terraform resource.</p> <p><strong>Note:</strong></p> <ul> <li>compute instance is AKA host.</li> <li>compute instance type is AKA machine size.</li> </ul> <p>The <strong>hpegl_metal_host</strong> resource supports many different arguments, but these are the required ones:</p> <ul> <li>networks – List of networks. The list must always include any networks marked as '<strong>required</strong>' for that location.</li> <li>machine_size – Compute instance type.</li> <li>ssh – List of SSH keys to push to the host.</li> <li>location – The data center location of where the compute instance will be provisioned.</li> <li>Image – OS image in the form flavor@version.</li> </ul> <p>You can also check the documentation <a href="https://registry.terraform.io/providers/HPE/hpegl/latest/docs/resources/metal_host">here</a> to see all the required and optional fields.</p> <p>Your next step with the TF file is to query the HPE GreenLake provider to collect the above-required information for creating a host. For this, you will use the Terraform data statements.</p> <h3>Querying for available OS images</h3> <p>In order to list the available OS images for OS, add the below data statements in your Terraform file <strong>main.tf</strong>:</p> <pre><code class="language-hcl">data "hpegl_metal_available_images" "ubuntu" { # select anything that looks like ubuntu:20.04 filter { name = "flavor" values = ["(?i)ubuntu"] } filter { name = "version" values = ["20.04*"] } } </code></pre> <p>The OS image list can be fetched using the following statements in <strong>main.tf</strong>:</p> <pre><code class="language-hcl">locals { ubuntu_image = format("%s@%s", data.hpegl_metal_available_images.ubuntu.images[0].flavor, data.hpegl_metal_available_images.ubuntu.images[0].version) } </code></pre> <h3>Querying for other available resources</h3> <p>For this, you should use <strong>hpegl_metal_available_resources</strong> data resource. For example, the following statements show how to retrieve and store the available SSH keys in a local variable.</p> <p>Append to main.tf:</p> <pre><code class="language-hcl"># query available resources. data "hpegl_metal_available_resources" "available" { } # using one of the available SSH keys. locals { ssh_keys = data.hpegl_metal_available_resources.available.ssh_keys[0].name } </code></pre> <p>Using a similar technique, you can retrieve the rest of the data you need - networks, machine size, etc.</p> <p>Append to main.tf:</p> <pre><code class="language-hcl"># choosing a location that has at least one machine available. locals { location = ([for msize in data.hpegl_metal_available_resources.available.machine_sizes : msize.location if msize.quantity > 0])[0] } # Listing required networks for this location. locals { networks = ([for net in data.hpegl_metal_available_resources.available.networks : net.name if net.host_use == "Required" &#x26;&#x26; net.location == local.location]) } # choosing machine size/Compute Instance Type to deploy OS. locals { machine_size = ([for msize in data.hpegl_metal_available_resources.available.machine_sizes : msize.name if msize.location == local.location]) } </code></pre> <p><a href="https://registry.terraform.io/providers/HPE/hpegl/latest/docs/data-sources/metal_available_resources">Here</a> you can get information about each of the bare metal data statements supported by the <strong>hpegl</strong> provider.</p> <h3>Create Compute instance</h3> <p>The last step is for you to define a <strong>hpegl_metal_host</strong> terraform resource in the file to request a new compute instance (host):</p> <pre><code class="language-hcl">resource "hpegl_metal_host" "demo_host" { name = "demo-host-1" image = local.ubuntu_image machine_size = local.machine_size[0] ssh = [ local.ssh_keys ] networks = local.networks location = local.location description = "Simple Host Deployment Demo" } </code></pre> <h4>Initialize Terraform</h4> <p>Before you can use Terraform, you will have to initialize it from the configuration file we have created. In the same directory as the <strong>main.tf</strong> file you created, run the command: <strong>terraform init</strong>.</p> <p><img src="/img/bmaas_terraform_init.png" alt="terraform init" title="terraform init"></p> <h4>Validate and View the Terraform Execution Plan</h4> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run the command: <strong>terraform plan</strong>.</p> <p><img src="/img/bmaas_terraform_plan.png" alt="terraform plan" title="terraform plan"></p> <h4>Apply the Terraform Execution Plan</h4> <p>The command you need to use is now: <strong>terraform apply</strong>. This will rerun the plan command, then prompt you to confirm before it starts applying what’s in the plan:</p> <p><img src="/img/bmaas_terraform_apply_1.png" alt="terraform apply" title="terraform apply"></p> <p><img src="/img/bmaas_terraform_apply_2.png" alt="terraform apply" title="terraform apply"></p> <h2>Advanced Example</h2> <p>The above example shows how to deploy a compute instance from pre-existing resources. Below is another code sample demonstrating compute instance deployment using dynamic resources and additional optional configurations with <strong>hpegl_metal_host.</strong></p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "HPE/hpegl" version = ">= 0.3.12" } } } provider "hpegl" { # metal block for configuring bare metal resources. metal { } } locals { location = "USA:CO:FTC" } resource "hpegl_metal_ssh_key" "newssh_1" { name = "newssh_1" public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCv03o//GEQ9/6eI1qZleyBbSndg0n5AkcKVnf5D4fEjwkWrtSIJEnROqJddEAn2XYALAk9x1AcB4Nue3q4tDG17VeK3ODo0+9Dx0LYqUTawnFWmo4X80QKr658Jmt7Enmnk5x2IrUDcNwAzALVellkBbwq7QbYUu1swSycNlNhSfGizqo/lQCNIHXyeRQ8oJxOuZkbiturXHZL389blIrTeUo53xmwE1TolVS8QzZRN8ve1GjFvpC5dl6orzi6LXDcrDcbZaxlrW+YQqyaipFRAw1DyTalrfpqxtq/Y9+Elz5xgCnUaepHN6ha/k81wtI2rySHga6pMOcJKlxaRS5OfzdrWh7oi2tEAaiq2y3pTr9hROQ2OGcMNU5gxbVU2ymeXdHVsAHMCmyKvQe0g0/fJzmNA/excogFCWDN7Spy9s2V39IbEKttyXjD/dpave7re9eFzYHA1CBEnNjMuvJj0H4tnpAETdQ6UbnjbE4JYn5eKGvnJ2w1JTfSdMK8nMcxqo4HfHWuLFuntCV9GAlWIVIvJn1pYisY8kEOtN5w6QrLTfsei96/TfssAsfhrDrVtgcgNU3EvZlC6Uaaly7D0ISFeufsxkPswu+jGNUJvGEqDiqvt05lSEZWS5viR/TOROTlicaGN9dhez/fqHcj5cnuoK1pmibK5GT7/Yf1Gw== user1@quattronetworks.com" } resource "hpegl_metal_network" "newpnet_1" { name = "newpnet_1" description = "New private network 1 description" location = local.location ip_pool { name = "npool" description = "New IP pool description" ip_ver = "IPv4" base_ip = "10.0.0.0" netmask = "/24" default_route = "10.0.0.1" sources { base_ip = "10.0.0.3" count = 10 } dns = ["10.0.0.50"] proxy = "https://10.0.0.60" no_proxy = "10.0.0.5" ntp = ["10.0.0.80"] } } resource "hpegl_metal_host" "demo_advance" { count = 0 name = "demo-advance-1" image = "ubuntu@20.04-20210713" machine_size = "G2i" ssh = [hpegl_metal_ssh_key.newssh_1.id] networks = ["Public", hpegl_metal_network.newpnet_1.name] network_route = "Public" network_untagged = hpegl_metal_network.newpnet_1.name location = local.location description = "Hello from Terraform" # Attaching tags labels = { "purpose" = "devops" } } </code></pre> <h2>Cleaning up resources </h2> <p>When you no longer need the resources created via Terraform, destroy the resources using the <strong>terraform destroy</strong> command. This will automatically use the HPE GreenLake provider to clean the infrastructure in HPE GreenLake.</p> <p><img src="/img/bmaas_terraform_cleanup.png" alt="terraform destroy" title="terraform destroy"></p> <h1>Summary</h1> <p>In this blog, I covered how to provision a compute instance with Terraform provider for HPE GreenLake using bare metal resources. I also showed you advanced usage of hpegl resource statements to deploy a compute instance with dynamic resources.<br> I hope you found this information interesting and helpful in helping you get started with the HPE GreenLake Terraform provider. You can also go through the below links to understand more about the HPE GreenLake Terraform provider.</p> <ul> <li><a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code – Part 1</a></li> <li><a href="https://developer.hpe.com/blog/kubernetes-cluster-as-code-part-2/">Kubernetes Cluster as Code – Part 2</a></li> <li><a href="https://registry.terraform.io/providers/hpe/hpegl/latest">Learn more about the HPE GreenLake Terraform provider</a></li> </ul>https://developer.hpe.com/custom-docker-images-for-emr-serverless/https://developer.hpe.com/custom-docker-images-for-emr-serverless/Thu, 09 Mar 2023 11:10:16 GMT<p>EMR serverless is the latest addition from the AWS to offer out of box support for Elastic MapReduce paradigm with auto scaling and pay as you go model.</p> <p>One often needs to build custom images in EMR serverless since the application uses specialized libraries that don't come with an EMR serverless base image. An explicit use case would be the requirements found with Delta Lake over S3 or specific Python modules, such as boto modules, database access libraries (i.e. pgcopy), etc. While there exists the option to have them installed at run time (i.e. when the image is running in a container and a custom script is used to install it), in production settings, it is encouraged to prebundle all the required libraries, modules, and jars. Dynamic installations are typically discouraged due to access permissions and internet connectivity issues.</p> <p>Further, getting prebundled assembly jars with all the dependencies is feasible in certain programming such as a scala/java, but with widely used Python the concept of creating assembly modules is missing. To address this, one option for all Python based modules/libraries that are required by the application code is to be prebundle them into custom images. We are going to show you a fairly quick and easy way to do this for AWS.</p> <p>In this article we take two scenarios, one where the application expects the data jars to be in specific location for its execution as in case of Delta Lake and another where we install specific Python modules using pip install both of which are bundled into a custom docker image that the EMR Serverless uses.</p> <p>Delta Lake (<a href="https://delta.io">https://delta.io</a>) is an open-source storage framework that enables building a lake house architecture. One of the important features supported by Delta Lake is the checkpoint management for batch/streaming jobs that uses AWS S3 as a data lake. The most common approach in using Delta Lake is via the command line running the spark-submit command - something like <code>"sparkSubmitParameters": "--packages io.delta:delta-core_2.12:1.2.1"</code>. This dynamically pulls the Delta Lake (via maven project) when the spark jobs run (ref. <a href="https://docs.delta.io/latest/quick-start.html#pyspark-shell">https://docs.delta.io/latest/quick-start.html#pyspark-shell</a> ). The other approach is to use <strong>pip install delta-spark==2.2.0</strong> when Python is used. However, both these approaches in production might not be possible, as they require an active internet connection and necessary permissions on the production machine to download/install a library at run time in the EMR Serverless application.</p> <p>Another use case is the installation of all custom libraries (in the form of jars etc.) used by the code. In this article, we will provide you with step-by-step instructions on how to build custom images, addressing both of the above scenarios using the ECR to register our Docker image. We will also show you how to use the Docker image in the EMR Serverless.</p> <p><em>Step 1:</em>  Pre-requisites to build a custom Docker image</p> <p><em>Step 2:</em>  Identify the base EMR Serverless Docker image, packages/jars that need to be installed/downloaded, and prepare the custom Docker image.</p> <p><em>Step 3:</em>  Push the Docker image to the ECR</p> <p><em>Step 4:</em> Create the EMR Serverless applications by using one of the following approaches:</p> <p>(a) Via the AWS command line (AWS CLI)</p> <p>(b) Via EMR Serverless settings in the EMR management console</p> <h2>Step 1: Prerequisites to build a custom Docker image:</h2> <ul> <li> <p>The latest version of the Docker client</p> </li> <li> <p>The latest version of AWS CLI</p> </li> <li> <p>Verify the access to any dependent repository from where resources (modules, code etc) might be downloaded.</p> </li> </ul> <h2><strong>Step 2: Sample Docker file used to create custom EMR Serverless image:</strong></h2> <p><em># Refer AWS EMR documentation for the release version tag</em></p> <p><em><code>FROM public.ecr.aws/emr-serverless/spark/emr-6.9.0:20221108</code></em></p> <p><em>USER root</em></p> <p><em># Dependent JAR files</em></p> <p><em><code>RUN curl -O</code><a href="https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.2.0/delta-core_2.12-2.2.0.jar"><code>https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.2.0/delta-core_2.12-2.2.0.jar</code></a></em>``</p> <p><code>RUN curl -O https://repo1.maven.org/maven2/io/delta/delta-storage/2.2.0/delta-storage-2.2.0.jar</code></p> <p><em># The base emr-image sets WORKDIR to /home/hadoop, hence the JAR files will be downloaded under /home/hadoop.</em></p> <p><em># Then these jars will be copied to /usr/lib/spark/jars which was set as SPARK_HOME by EMR base image.</em></p> <p><em><code>RUN cp /home/hadoop/delta-core_2.12–2.2.0.jar /usr/lib/spark/jars/</code></em><br> <em><code>RUN cp /home/hadoop/delta-storage-2.2.0.jar /usr/lib/spark/jars/</code></em></p> <p><em># EMRS will run the image as hadoop</em></p> <p><em><code>USER hadoop:hadoop</code></em></p> <h2><strong>Step 3: Pushing the Docker image to the</strong> Amazon Elastic Container Registry <strong>(AWS ECR):</strong></h2> <p>Command to authenticate Docker to an AWS ECR public registry assuming region as us-east-1. As the EMR Serverless base Docker image is present in the ECR public registry, run the command shown below before building your customized Docker file.</p> <p><em>aws ecr-public get-login-password — region us-east-1 — profile &#x3C;profile_name_in_your_aws_credentials_file> | docker login — username AWS — password-stdin public.ecr.aws</em></p> <p>Command to build Docker image:</p> <p><em>docker build -t local_docker_image_name:tag -f &#x3C;docker_file_name> <code>.</code></em></p> <p>Command to tag locally built Docker image in order to push to the AWS ECR private registry:</p> <p><em>docker tag local_docker_image_name:tag <a href="http://aws_account_id.dkr.ecr.region.amazonaws.com">aws_account_id.dkr.ecr.region.amazonaws.com</a>**<a href="http://718515174980.dkr.ecr.us-east-1.amazonaws.com/emr-serverless-ci-examples:emr-serverless-6.9.0-V1">/docker_image_name</a>:tag</em></p> <p>Command to authenticate Docker to an AWS ECR private registry assuming region as us-east-1. Run the command shown below before pushing the Docker image to AWS ECR private registry.</p> <p><em><strong><code>aws ecr get-login-password — region us-east-1 — profile &#x3C;profile_name_in_your_aws_credentials_file> | docker login — username AWS — password-stdin</code><a href="http://aws_account_id.dkr.ecr.region.amazonaws.com"><code>aws_account_id.dkr.ecr.region.amazonaws.com</code></a></strong></em>``</p> <p> <strong>Note:</strong></p> <p>  1. When the --profile option is not provided, credentials will be picked from the profile name <default> in the ~/.aws/credentials file. If you are using access credentials for a different user, then include the profile section in ~/.aws/credentials files.</p> <p>2.  Example content of ~/.aws/credentials file:          </p> <p>[default]<br> aws_access_key_id =<br> aws_secret_access_key =</p> <p>[testprofile]<br> aws_access_key_id=<br> aws_secret_access_key=</p> <p>Command to push Docker image to the AWS ECR private registry:</p> <p>    <em>docker push <a href="http://aws_account_id.dkr.ecr.region.amazonaws.com">aws_account_id.dkr.ecr.region.amazonaws.com</a>**<a href="http://718515174980.dkr.ecr.us-east-1.amazonaws.com/emr-serverless-ci-examples:emr-serverless-6.9.0-V1">/docker_image_name:</a>tag</em></p> <h2><strong>Step 4:  EMR Serverless using the custom Docker image created:</strong></h2> <p>There are two approaches for EMR Serverless to use the custom image created:</p> <p>(a) via the AWS CLI</p> <p>(b) Via EMR Serverless settings in the EMR management console</p> <h3><strong>4a) Via the AWS CLI command to create simple EMR Serverless application using custom Docker image:</strong></h3> <p>AWS CLI reference for EMR Serverless application management: <a href="https://docs.aws.amazon.com/cli/latest/reference/emr-serverless/index.html">https://docs.aws.amazon.com/cli/latest/reference/emr-serverless/index.html</a></p> <p><em>aws — region <region> emr-serverless create-application</em> <br> <em>— release-label emr-6.9.0</em> <br> <em>— type “SPARK”</em> <br> <em>— name emr-application-1</em> <br> <em>— image-configuration ‘{ “imageUri”: “<your AWS account ID>.dkr.ecr.<region>.<a href="http://amazonaws.com/emr-serverless-ci-examples:emr-serverless-ci-ml">amazonaws.com/&#x3C;ecr_registry_name:</a>image_name>” }’</em></p> <p><strong>Note:</strong> Image URI can be copied from the AWS ECR registry.</p> <h3><strong>4b)</strong> Via EMR Serverless settings in the EMR management console</h3> <h5>EMR Serverless custom image settings section in the AWS EMR Serverless management console**:**</h5> <p><img src="/img/customimagesettings-in-emrserverless.png" alt="EMR Serverless Management Console Docker section"></p> <h5>Browse and select custom EMR Serverless image from the AWS ECR private registry in the same region:</h5> <p><img src="/img/selectionofimagefromecr_registry.png" alt="EMR Serverless Management Console to choose the Docker image"></p> <h3><strong>Summary:</strong></h3> <p>This article provides the reader with mechanisms to build custom Docker image where custom libraries especially Python modules need to be bundled and used in EMR Serverless. We use two use cases with Python modules and Delta Lake libraries on how one can build this Docker image. This should help all developers using Python as their software language for EMR Serverless to pre-bundle the Python libraries for the production environment. If you found this blog post helpful, we recommend you read more on the topic by referencing the documentation below:</p> <ul> <li> <p><strong><a href="https://aws.amazon.com/blogs/big-data/add-your-own-libraries-and-application-dependencies-to-spark-and-hive-on-amazon-emr-serverless-with-custom-images/">Steps to build custom Docker image for EMR Serverless</a></strong></p> </li> <li> <p><strong><a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html">How to pull Docker image from AWS ECR registry</a></strong></p> </li> <li> <p><strong><a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html">How to authorize AWS ECR registry with Docker client</a></strong></p> </li> <li> <p><strong><a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html">How to push Docker image to AWS ECS registry</a></strong></p> </li> <li> <p><strong><a href="https://docs.aws.amazon.com/cli/latest/reference/emr-serverless/index.html">AWS EMR Serverless CLI reference</a></strong></p> </li> <li> <p><strong><a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/docker-custom-images-steps.html">How to customize image for multi CPU architecture</a></strong></p> </li> </ul> <h2><strong>About the Authors:</strong></h2> <p><img src="/img/niranjan_2.jpg" alt=""></p> <p>D R Niranjan is a Senior software/Cloud application developer at HPE, experienced in developing software and cloud applications for HPE Servers and Storage systems. He is technically skilled in Python, Java, Scala, Spark programming, AWS, Dockers and Kubernetes, PgSQL/RDBMS. You can have a look at his LinkedIn profile here: <a href="https://www.linkedin.com/in/niranjan-d-r/">https://www.linkedin.com/in/niranjan-d-r/</a></p> <p><img src="/img/sagar-pic_2.jpg" alt="" title="sagar-nyamagouda@hpe.com"></p> <p>Sagar Nyamagouda holds a B.E(Information Science and Engineering) from BMS College of Engineering (BMSCE), Bengaluru and M.Tech in Software Systems from BITS Pilani. He is an experienced R&#x26;D Engineer working on Big Data Technologies and building AI/ML pipelines to give real time insights to customers. An AI/ML enthusiast, he is currently working with HPE enabling advanced insights for Data Services in HPE GreenLake Cloud Platform. LinkedIn Profile: <a href="http://www.linkedin.com/in/sagarny">www.linkedin.com/in/sagarny</a></p> <p><img src="/img/chirag_2.jpg" alt="" title="chirag.talreja@hpe.com"></p> <p>A graduate from BITS Pilani, Pilani, Chirag is currently working as a cloud developer in HPE. He has over 6 years of experience in designing, developing, and implementing complex data processing pipelines. He is experienced in microservice architecture, big data tech stack like Apache Spark , Spark Streaming , SQL/NoSQL databases, Kafka, Core Java, Scala, and Python and has a good understanding of the AWS platform. His LinkedIn profile is <a href="https://www.linkedin.com/in/chiragtalreja29/">https://www.linkedin.com/in/chiragtalreja29/</a></p> <p><img src="/img/chinmay_2jpg.jpg" alt="" title="chinmay.chaturvedi@hpe.com"></p> <p>Chinmay currently works at HPE as a cloud engineer. He has expertise in various Big Data Processing technologies, including the building of ML/AI platforms, data engineering and processing (most recently related to Spark, AWS, SQL, etc.). Please view his LinkedIn profile here: <a href="https://www.linkedin.com/in/chinmay-chaturvedi-707886138">https://www.linkedin.com/in/chinmay-chaturvedi-707886138</a></p> <p><a href="https://www.linkedin.com/in/chinmay-chaturvedi-707886138"></a></p> <p><img src="/img/whatsapp-image-2021-11-02-at-10.01.24-am-1-.jpg" alt=""></p> <p>Kalapriya Kannan currently works with HPE on cloud enablement of storage analytics. She holds a Ph.D from IISc. She has authored around 60 peer reviewed international conference papers and over a 100 disclosure submissions for patent filing for which she has been granted 65 patents. Her interests are in the area of distributed and parallel systems and currently she is working in processing of big data for analytical insights. Her linkedIn profile: <a href="https://www.linkedin.com/in/kalapriya-kannan-0862b55b">https://www.linkedin.com/in/kalapriya-kannan-0862b55b</a></p> <p><img src="/img/download.png" alt=""></p> <p>Roopali has 19 years of work experience in systems software development in the areas of operating system, file system and cloud technologies. She has played a number of various roles, starting from developer to lead expert and product owner. In her current role, she is responsible for functional delivery and people management of Data Observability Analytics Project.</p>https://developer.hpe.com/create-a-cold-tiered-volume-and-use-hpe-ezmeral-data-fabric-object-store-as-remote-target/https://developer.hpe.com/create-a-cold-tiered-volume-and-use-hpe-ezmeral-data-fabric-object-store-as-remote-target/Mon, 06 Mar 2023 01:05:32 GMT<h2>Introduction</h2> <h3>What is data tiering and why do it?</h3> <p>Much of your data needs to be retained, either to meet regulatory requirements or because it still has value. Data tiering is useful for data that is not accessed frequently but needs to be retained to be stored in a more resource-efficient and cost-effective manner.</p> <p>The most frequently accessed file data can be thought of as a "hot data tier, which uses normal file storage. Data used less often can be moved to low-cost storage alternatives in different ways, depending on the relative frequency of access. Some data is rarely accessed or modified but needs to be archived for future projects, for verification purposes in audits, or to meet regulatory requirements. This "cold" data could be tiered to low-cost object storage in the same data storage system or in a remote storage system, such as remote object storage.</p> <p>In HPE Ezmeral Data Fabric, you can create a cold-tiered volume, set corresponding storage policies, and periodically offload the data in the volume to the remote object storage. The remote object storage can be the object storage of AWS, GCP, Azure and other public clouds, or an object storage service compatible with Minio. Of course, you can also use the Object Store of HPE Ezmeral Data Fabric as a remote target. This article will demonstrate how to create a cold-tiered volume and configure another HPE Ezmeral Data Fabric Object Store as the remote target for offloading. At the same time, I will also demonstrate how to create an account, IAM user, and bucket in the HPE Ezmeral Data Fabric Object Store, and use the AWS CLI to perform a <strong>put object operation</strong> on this bucket through the above configuration.</p> <h3><a href="#advantages-of-using-data-tiering"></a>Advantages of using data tiering</h3> <p>HPE Ezmeral Data Fabric provides a rule-based automated tiering functionality that allows you to seamlessly integrate with:</p> <ul> <li>Low-cost storage as an additional storage tier in the data fabric cluster for storing file data that is less frequently accessed ("warm" data) in an erasure-coded volume.</li> <li>3rd party cloud object storage as an additional storage tier in the data fabric cluster to store file data that is rarely accessed or archived ("cold" data).</li> </ul> <p>In this way, valuable on-premise storage resources can be used for more active or "hot" file data and applications, while "warm" and/or "cold" file data can be retained at minimal cost for compliance, historical, or other business reasons. HPE Ezmeral Data Fabric provides consistent and simplified access to and management of the data.</p> <h3><a href="#advantages-of-using-object-store"></a>Advantages of using Object Store</h3> <p>HPE Ezmeral Data Fabric Object Store is a native object storage solution that efficiently stores objects and metadata for optimized access.</p> <p>Underlying each Object Store bucket is a volume. Every bucket created in an Object Store account is automatically associated with a volume. You can snapshot or mirror a bucket volume for disaster recovery.</p> <p>If you create an account in Object Store, specify the erasure coding scheme (ecscheme) in the storage_class. All buckets created in the account inherit the ecscheme. Underlying volumes are automaticall tiered in such a way that data in a bucket volume can be offloaded to a back-end volume to reclaim storage space.</p> <p>Some potential Object Store use cases include:</p> <ul> <li>Archive data and build on-premises applications, or migrate to cloud-native applications.</li> <li>Store media for operational use; reduce costs of storing globally distributed media, such as music, video, and images.</li> <li>Run analytics on data with tools like Apache Spark, Apache Drill, Presto, and S3 Select to gain valuable insights into customers, operations, or markets.</li> <li>Maintain Spark Delta Lake time travel information. You can time travel to see different versions of the data when Object Store is configured as a data lake for Spark Delta Lake.</li> <li>Store ML model data and share the ML models in real-time with downstream applications.</li> <li>Publish S3 events to HPE Ezmeral Data Fabric Streams.</li> </ul> <h2><a href="#install-and-configure-object-store"></a>Install and configure Object Store</h2> <p>Regarding the installation of Object Store, I recommend you use the <a href="https://docs.datafabric.hpe.com/72/MapRInstaller.html">Installer</a>.</p> <p>When using the Installer to install HPE Ezmeral Data Fabric 7.0.0 or higher, you must enable security. Even for POC environments, I recommend you enable and configure basic security.</p> <p>If you have ever used the ecological components of Apache Hadoop, or other commercial big data suites, then I think you already have the basic concepts of authentication, authorization, audit, and encryption of the Hadoop ecosystem.</p> <p>The rationale for the security of HPE Ezmeral Data Fabric is the same as an open-source Hadoop ecosystem. For example, SASL (MapR-SASL, Kerberos) is used for authentication, Ranger is used for authorization, and TLS is used for encryption.</p> <p>Use the Installer to install HPE Ezmeral Data Fabric, which can automatically create a series of TLS-related certificates, and automatically configure core components and various HPE Ezmeral Ecosystem Pack components to enable security.</p> <h3><a href="#post-installation-configuration-for-object-store"></a>Post-installation configuration for Object Store</h3> <p>Some post-installation steps must be performed before you can use the HPE Ezmeral Data Fabric Object Store. You should refer to this document - <a href="https://docs.datafabric.hpe.com/72/AdvancedInstallation/Enabling_object_store.html">Enabling the HPE Ezmeral Data Fabric Object Store</a> for post-installation configuration.</p> <p>For the above document, I have a few supplementary notes, which should make your configuration smoother.</p> <h4>1. About "keytool -noprompt -importcert" command</h4> <p>Please do not use this 👇 command in the original document:</p> <pre><code class="language-shell">${JAVA_HOME}/bin/keytool -noprompt -importcert -file /opt/mapr/conf/ca/chain-ca.pem -alias maprca -keystore ${JAVA_HOME}/lib/security/cacerts -storepass &#x3C;store_password> </code></pre> <p>Instead, use the following command:</p> <pre><code class="language-shell">${JAVA_HOME}/bin/keytool -noprompt -importcert -file /opt/mapr/conf/ca/chain-ca.pem -alias maprca -cacerts -storepass &#x3C;store_password> </code></pre> <p>If you use the Installer to install HPE Ezmeral Data Fabric on a fresh OS, then Installer will automatically install JDK 11, then "-storepass" password is "changeit".</p> <p>There is another place in the documentation where keytool is used. The command is as follows:</p> <pre><code class="language-shell">${JAVA_HOME}/bin/keytool -noprompt -importcert -file /opt/mapr/conf/ca/chain-ca.pem -alias mosscert -keystore ${JAVA_HOME}/lib/security/cacerts -storepass changeit </code></pre> <p>You don't need to execute <code>keytool</code> twice to import the same <strong>CA certificate</strong> file.</p> <p>I suggest you change "maprca" to something more recognizable.</p> <p>For example, if you named the cluster "edf-cluster-a.mycompany.com" when you installed the HPE Ezmeral Data Fabric cluster, then you can use the following keytool command:</p> <pre><code class="language-shell">${JAVA_HOME}/bin/keytool -noprompt -importcert -file /opt/mapr/conf/ca/chain-ca.pem -alias edf-clustera-ca -keystore ${JAVA_HOME}/lib/security/cacerts - storepass &#x3C;cacerts_truststore> </code></pre> <p>The file - /opt/mapr/conf/ca/chain-ca.pem, is a self-signed TLS certificate file created by the Installer when configuring wire-level encryption for the cluster.</p> <p>Since it's a self-signed TLS certificate, the client (application, or your browser) will not be able to trust the TLS certificate of the server when accessing the HPE Ezmeral Data Fabric server. This is because the CA certificate used by the self-signed TLS certificate is not publicly trusted.</p> <p>In any scenario where a self-signed TLS certificate is used, you need to import the self-signed CA certificate into the OS system.</p> <p>You can use the following command to look at things like CA certificates in your JVM:</p> <pre><code class="language-shell">keytool -list -v -cacerts </code></pre> <p>You should now see something like the following:</p> <pre><code class="language-dart">Alias name: digicertassuredidrootca Creation date: Feb 2, 2023 Entry type: trustedCertEntry Owner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US Issuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US ... </code></pre> <p>This is a recognized CA certificate issued by Digicert.</p> <p>After you use <code>keytool</code> to import /opt/mapr/conf/ca/chain-ca.pem into the JVM, use keytool -list -v -cacerts to see something similar to the following:</p> <pre><code class="language-dart">Alias name: edf-clustera-ca Creation date: Feb 16, 2023 Entry type: trustedCertEntry Owner: CN=MapR Engineering Signing CA, OU=MapR Engineering Signing CA, O=MapR, DC=hpecorp, DC=net Issuer: CN=MapR Engineering Root CA, OU=MapR Engineering Root CA, O=MapR, DC=hpecorp, DC=net ... </code></pre> <h4>2. About client software options</h4> <p>The documentation describes several client software that you can use to interact with the HPE Ezmeral Object Store. I would like to add the following clarification:</p> <p>When using the <code>mc</code> command-line tool or the <code>s3cmd</code> command-line tool to interact with HPE Ezmeral Data Fabric Object Store without completing the configuration of "<a href="https://docs.datafabric.hpe.com/72/AdvancedInstallation/Enabling_object_store.html#concept_isb_53h_5bb__section_nnp_kr2_bvb">Enabling S3 Virtual-Host-Style Requests</a>", some commands will not work properly.</p> <p>In this case, for management operations, such as creating accounts, IAM accounts, buckets, etc., I recommend you use the <code>mc</code> command line tool.</p> <p>For object listing, getting, putting, and deleting operations, I recommend you use <code>AWS CLI</code>.</p> <h4>3. Cannot revert to HTTP mode after enabling HTTPS.</h4> <p>If the Object Store was installed using the Installer, the Object Store would also have security enabled, including HTTPS.</p> <p>For example, although you can see the description below from HTTP Access to Object Store, this configuration change alone does not change Object Store to HTTP mode.</p> <blockquote> <p>To revert to http access, comment out the moss.certs.dir=/opt/mapr/conf line in the /opt/mapr/conf/moss.conf file.</p> </blockquote> <p>☝from here - <a href="https://docs.datafabric.hpe.com/72/AdvancedInstallation/Enabling_object_store.html#concept_isb_53h_5bb__section_bg1_zd1_vsb">HTTP Access to Object Store</a> Additionally, there is nothing else in HPE Ezmeral Data Fabric's documentation on how to modify Object Store's TLS mode. You may be able to find out how to turn off HTTPS from this document. 👉 <a href="https://docs.datafabric.hpe.com/72/MapROverview/Object-Store-signed-certs.html">Using Custom Signed Certificates with Object Store</a></p> <h3><a href="#create-a-bucket-in-hpe-ezmeral-data-fabric-object-store-and-upload-some-objects"></a>Create a bucket in HPE Ezmeral Data Fabric Object Store and upload some objects</h3> <p>In order to create a bucket, you need to create an account, an IAM user, and finally a bucket in sequence. While creating IAM user and bucket, you also need to prepare access policy control for who (IAM User) can perform which (list, get, put, etc.) operations on which bucket.</p> <p>You can read the following document - <a href="https://docs.datafabric.hpe.com/72/MapROverview/object_store_account_management.html">Entities and Resources</a> first to gain a deeper understanding of the entity model of HPE Ezmeral Data Fabric Object Store.</p> <p>Below, I will demonstrate how to create the above-required entities.</p> <h4>Create an account</h4> <p>I choose to use the <a href="https://docs.datafabric.hpe.com/72/ReferenceGuide/mc-commands-overview.html">mc</a> command line tool to create an account.</p> <p>Before using the mc command line for the first time, you need to create an alias for your administrator. An alias contains an access endpoint, such as "<a href="https://s3-us-west-1.amazonaws.com">https://s3-us-west-1.amazonaws.com</a>", which is an Amazon AWS S3 endpoint; another example is "<a href="http://10.10.88.198:9000">http://10.10.88.198:9000</a>", which is a Minio endpoint. An alias also contains the access key and secret key used by your administrator or IAM User.</p> <p><strong>1. You first use the "<a href="https://docs.datafabric.hpe.com/72/ReferenceGuide/mc-alias-list.html">mc alias list</a>" command to view the default alias in the following systems.</strong></p> <p>❗Note: If you are using self-signed TLS certificates or installed the cluster via Installer, you have to copy <ins>/opt/mapr/conf/ca/chain-ca.pem</ins> to <ins>~/.mc/certs/CAs/</ins> on the node running <code>mc</code>. The reason for this step is the same as why you imported the self-issued CA to the keytool of the JVM earlier, <code>mc</code> also needs to import the self-issued CA to communicate with the S3server of the Object Store.</p> <pre><code class="language-shell">sudo -E -u mapr /opt/mapr/bin/mc alias list </code></pre> <p>Sample output:</p> <pre><code>gcs URL : https://storage.googleapis.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v2 Path : dns local URL : http://localhost:9000 AccessKey : SecretKey : API : Path : auto play URL : https://play.min.io AccessKey : XXXXXXXXXXXXXXXXXXXX SecretKey : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX API : S3v4 Path : auto s3 URL : https://s3.amazonaws.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v4 Path : dns </code></pre> <p><strong>2. Generate S3 keys to authenticate your administrator</strong></p> <p>The cluster administrator (typically the mapr user) must authenticate to the Object Store cluster and generate S3 keys (accessKey and secretKey) on the default Object Store account. Perform this operation before performing any CLI operations in Object Store.</p> <p>If the cluster is secure, use <a href="https://docs.datafabric.hpe.com/72/SecurityGuide/ThemaprloginUtility.html">maprlogin</a> to authenticate the cluster administrator, and then generate the keys:</p> <pre><code class="language-shell">maprcli s3keys generate -domainname primary -accountname default -username mapr -json </code></pre> <p>🗒Note: An Object Store cluster has a domain, accounts, buckets, users, and access policies associated with it. Installing Object Store in a cluster provides a primary domain and a default account.</p> <p>Sample output:</p> <pre><code class="language-json">{ "timestamp":1676472096994, "timeofday":"2023-02-15 10:41:36.994 GMT+0800 PM", "status":"OK", "total":1, "data":[ { "accesskey":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "secretkey":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } ] } </code></pre> <p>🗒Note: If you encounter any problem when generating the S3 keys, refer to this page: <a href="https://docs.datafabric.hpe.com/72/MapROverview/object-store-get-started.html#object-store-get-started__section_x4h_w13_4tb">Generate S3 Keys to Authenticate Users and Applications</a>.</p> <p><strong>3. Use the <code>mc alias set</code> command to create an alias for admin user</strong></p> <pre><code class="language-shell">mc alias set s3-admin-alias https://`hostname -f`:9000 {ACCESS_KEY} {SECRET_KEY} --api "s3v4" --path "off" --json </code></pre> <p>🗒Note: "s3-admin-alias" is the value of the alias parameter, you define it. "https://<code>hostname -f</code>:9000" is the endpoint of the Object Store service. Here, I'm running the command on the node that is running the S3server. After created an alias, you would find the information is appended into <ins>$HOME/.mc/config.json</ins>.</p> <p><strong>4. Create an account</strong></p> <p>I chose to use the Object Store Web GUI to create an account.</p> <p>Refer to this document - <a href="https://docs.datafabric.hpe.com/72/MapROverview/create-account.html#create_account__section_fsm_1hn_nrb">Using the Object Store Interface</a>.</p> <p>Create an account using Object Store Web GUI - 1 <a href="https://ibb.co/SyW9MKK"><img src="https://i.ibb.co/XLNvKzz/Object-Store-Create-Account-1.png" alt="Object-Store-Create-Account-1" border="0"></a></p> <p>Enter the following in "Default Bucket Place":</p> <pre><code class="language-json">{ "Version": "2012-10-17", "Statement": [ { "Sid": "GrantAdminPutPermissions", "Effect": "Allow", "Principal": "arn:primary:default:user:mapr", "Action":"s3:PutObject", "Resource":"arn:aws:s3:::${bucket}/*" }, { "Sid":"GrantAnonymousReadPermissions", "Effect":"Allow", "Principal": "*", "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::${bucket}/*"] } ] } </code></pre> <p>HPE Ezmeral Data Fabric Object Store is an on-premises object storage service compatible with Minio. Some concepts such as Domain and Default Account do not exist in public cloud object storage services such as AWS S3. But the policy for bucket and IAM user is compatible with the policy in public cloud object storage. For the Bucket Policy here, you can refer to AWS S3 <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html">Bucket policy examples</a> and HPE Ezmeral Data Fabric Object Store document - <a href="https://docs.datafabric.hpe.com/72/MapROverview/object-store-policies.html">Access Policies</a>.</p> <p>After creating the user account, you can use the below command to view it:</p> <pre><code class="language-shell">sudo -u mapr /opt/mapr/bin/mc admin account list {ADMIN_ALIAS} domain=primary --json </code></pre> <p>Sample output:</p> <pre><code class="language-json">{ "name": "default", "id": 0, "admin": "mapr", "labelname": "default", "minrepl": 2, "desiredrepl": 3, "usercount": 2 } { // # 👇 The account you just created. "name": "s3test", "id": 1, "admin": "mapr", "def_bucket_policy": { "Version": "2012-10-17", "Statement": [ { "Sid": "GrantAdminPutPermissions", "Effect": "Allow", "Principal": { "AWS": [ "arn:primary:default:user:mapr" ] }, "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::${bucket}/*" ] }, { "Sid": "GrantAnonymousReadPermissions", "Effect": "Allow", "Principal": { "AWS": [ "*" ] }, "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::${bucket}/*" ] } ] }, "size": 22871, "labelname": "default", "topology": "/data/default-rack", "minrepl": 1, "desiredrepl": 1, "usercount": 1, "bucketcount": 1 } </code></pre> <h4>Create an IAM User in the non-default account just created</h4> <p>In step 4, you created a non-default account named "s3test". In HPE Ezmeral Data Fabric Object Store, you must create a non-default account to create an IAM user, and you should use the IAM User to operate buckets.</p> <pre><code class="language-shell">sudo -u mapr /opt/mapr/bin/mc admin user add s3-admin-alias s3-test-iam_user account=s3test domain=primary </code></pre> <p>🗒Note: "s3-admin-alias" is the admin alias you created in step-3, and "s3-test-iam_user" is the IAM User name. For more information, refer to: <a href="https://docs.datafabric.hpe.com/72/MapROverview/create-IAM-user.html">Create IAM Users</a>.</p> <p>Next, you create an IAM policy for the IAM User - s3-test-iam_user.</p> <pre><code class="language-shell">cat &#x3C;&#x3C;'EOF' > ./PolicyPublicRead.json { "Version":"2012-10-17", "Statement": [ { "Sid":"GrantAnonymousReadPermissions", "Effect":"Allow", "Principal": "*", "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::${bucket}/*"] } ] } EOF </code></pre> <pre><code class="language-shell">mc admin policy add s3-admin-alias PolicyPublicRead ./PolicyPublicRead.json account=s3test domain=primary </code></pre> <p>🗒Note: "PolicyPublicRead" is the IAM Policy's name.</p> <p>You can also use the Object Store Web GUI to create the IAM Policy, like shown in the following screenshot👇.</p> <p><a href="https://ibb.co/xsH1nYH"><img src="https://i.ibb.co/n0C7XBC/Object-Store-Create-IAMPolicy-1.png" alt="Object-Store-Create-IAMPolicy-1" border="0"></a></p> <p>Let's create another IAM policy named "GrantbucketOperations"👇. You will associate these 2 IAM Policies to the IAM User - "s3-test-iam_user" later.</p> <pre><code class="language-json">{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:CreateBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::s3-test-iam-user-bucket/*" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::s3-test-iam-user-bucket/*" } ] } </code></pre> <p>To associate an IAM Policy to an IAM User:</p> <pre><code class="language-shell">sudo -u mapr /opt/mapr/bin/mc admin policy set s3-admin-alias PolicyPublicRead users='s3-test-iam_user' account='s3test' domain='primary' sudo -u mapr /opt/mapr/bin/mc admin policy set s3-admin-alias GrantBucketOperations users='s3-test-iam_user' account='s3test' domain='primary' </code></pre> <h4>Create a bucket for the IAM user</h4> <p>First, you need to generate the access key and secret key for the IAM User - "s3-test-iam_user".</p> <pre><code class="language-shell">sudo -u mapr maprcli s3keys generate -domainname primary \ -accountname s3test \ -username 's3-test-iam_user' </code></pre> <p>☝Then you would get the access key and secret key for the IAM User - "s3-test-iam_user".</p> <pre><code class="language-shell">sudo -u mapr mc alias set s3-test-iam_user-alias https://`hostname -f`:9000 \ {ACCESS_KEY} \ {SECRET_KEY} \ --api "s3v4" --path "off" --json sudo -u mapr mc mb --account s3test --ignore-existing --disable-versioning --json s3-test-iam_user-alias/s3-test-iam-user-bucket </code></pre> <p>☝Now you have created a bucket named "s3-test-iam-user-bucket" using the IAM user - "s3-test-iam_user". Because "s3-test-iam_user" is inside account - "s3test", the bucket will be also placed under account - "s3test".</p> <p>To list buckets using the <code>mc</code> command:</p> <pre><code class="language-shell">/opt/mapr/bin/mc ls --account s3test --versions --recursive --summarize --json s3-test-iam_user-alias </code></pre> <p>Sample output:</p> <pre><code class="language-json">{ "status": "success", "type": "folder", "lastModified": "2023-02-16T15:22:16+08:00", "size": 23897893980, "key": "s3-test-iam-user-bucket/", "etag": "", "url": "https://m2-maprts-vm197-172.mip.storage.hpecorp.net:9000/", "versionOrdinal": 1 } { "totalObjects": 1, "totalSize": 23897893980 } </code></pre> <h4>Install the AWS CLI and put an file into the bucket</h4> <p>To install the AWS CLI, refer to this Amazon AWS document 👉 <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">Installing or updating the latest version of the AWS CLI</a>.</p> <p>Then, create a profile for the IAM user:</p> <pre><code class="language-shell">export AWS\_CA\_BUNDLE=/opt/mapr/conf/ca/chain-ca.pem aws configure --profile s3-test-iam_user-ray-2-objstor </code></pre> <p>❗Note: Before using the AWS CLI, remember to export the environment AWS_CA_BUNDLE=/opt/mapr/conf/ca/chain-ca.pem. Otherwise AWS CLI cannot communicate with S3server because S3server is using self-signed TLS certificates.</p> <p>After inputting the above command, the AWS CLI will ask you to input the access key and secret key. After the profile is created, the information will be stored at <ins>$HOME/.aws/config</ins> and <ins>$HOME/.aws/credentials</ins>.</p> <p>Use the below command to list buckets:</p> <pre><code class="language-shell">aws s3api list-buckets --endpoint-url https://`hostname -f`:9000 --profile s3-test-iam_user-ray-2-objstor </code></pre> <p>Use the below command to put a file into the bucket:</p> <pre><code class="language-shell">aws s3api put-object --bucket s3-test-iam-user-bucket --key 'testdir/s3-test-iam-user-dir/hpe-cp-rhel-release-5.5.1-3083.bin' --body 'downloads/hpe-cp-rhel-release-5.5.1-3083.bin' --endpoint-url https://m2-maprts-vm197-172.mip.storage.hpecorp.net:9000 --profile s3-test-iam_user-ray-2-objstor </code></pre> <p>🗒Note: "s3-test-iam-user-bucket" is the bucket's name which you created before. "testdir/s3-test-iam-user-dir/hpe-cp-rhel-release-5.5.1-3083.bin" is the path which you want to put into the bucket. The part of "testdir/s3-test-iam-user-dir/" indicates it's under this directory, if the directory doesn't exist, it will be created. "downloads/hpe-cp-rhel-release-5.5.1-3083.bin" is the local file path which you want to put into the bucket.</p> <h2>Create an cold-tiered volume and offlad to remote Object Store</h2> <p>So, now we are going to create a volume on another cluster and configure the cold-tier remote target for this volume. Then we will manually offload the data in this olume to the remote HPE Ezmeral Data Fabric Object Store.</p> <h3>Create a cold-tiered volume via a web GUI</h3> <p>First, you will need to log into MCS, and enter the following positions in turn at the top of the screen: Data --> Volumes👇</p> <p><a href="https://ibb.co/1Zg5RXN"><img src="https://i.ibb.co/2q9QcMr/Create-Cold-Tier-Volume-1.png" alt="Create-Cold-Tier-Volume-1" border="0"></a></p> <p>Then click "Create Volume" at the top of the screen👇.</p> <p><a href="https://ibb.co/T8SV1Mt"><img src="https://i.ibb.co/cFHGwhQ/Create-Cold-Tier-Volume-2.png" alt="Create-Cold-Tier-Volume-2" border="0"></a></p> <p>Fill in the necessary information, you can refer to this document 👉 <a href="https://docs.datafabric.hpe.com/72/ClusterAdministration/data/volumes/CreateVols.html">Creating a Volume</a>.</p> <p>Turn on the "Data Tiering" switch and select "Remote Archiving(Cold)". Refer to the figure below to fill in the remote target information:</p> <p><a href="https://ibb.co/cyy3cWt"><img src="https://i.ibb.co/3TTdyxW/Create-Cold-Tier-Volume-3.png" alt="Create-Cold-Tier-Volume-3" border="0"></a></p> <ul> <li>URL: The host where the S3server of the remote HPE Ezmeral Data Fabric cluster is located, and the port number is the default port 9000 of the S3server.</li> <li>Bucket: The bucket created for IAM User in previous steps.</li> <li>Access key and secret key: The keys of the IAM User created in the previous step.</li> </ul> <h3>Configure the CA certificate of the remote Object Store for the MAST Gateway of the local cluster</h3> <p>You should remember that in the earlier steps, we configured the CA certificate of the Object Store's self-signed TLS certificate for the JDK keystore as well as the mc command line tool and the AWS CLI.</p> <p>Now, you will also need to configure this self-signed CA root certificate for MAST Gateway so that it can communicate with the remote Object Store.</p> <p>Refer to this document - <a href="https://docs.datafabric.hpe.com/72/StorageTiers/ConfigMASTGateway.html">Configuring the MAST Gateway Service</a>, and set the value of "mastgateway.curl.cainfo" in the configuration file.</p> <p>You need to find <ins>/opt/mapr/conf/ca/chain-ca.pem</ins> from a host of the Object Store cluster first, and copy it to the MAST Gateway node. For the convenience of management, you can rename it appropriately and configure it as the value of "mastgateway.curl.cainfo".</p> <h3>Use the maprcli volume offload command to manually offload data</h3> <p>Now you can place some data in the cold-tiered volume you just created. I put a 5.6GB file in it.</p> <p>Then, you can use the following command to manually trigger the offload of the entire olume.</p> <pre><code class="language-shell">maprcli volume offload -ignorerule true -name {VOLUME_NAME} </code></pre> <p>Then, you can use the following command to monitor the offload status.</p> <pre><code class="language-shell">watch 'maprcli volume tierjobstatus -name {VOLUME_NAME} -json' </code></pre> <p>🗒Note: the <code>watch</code> will execute the following string as a command every 2 seconds.</p> <p>When the offload is complete, you will see the following output.</p> <pre><code class="language-json">{ "timestamp": 1676564889008, "timeofday": "2023-02-17 12:28:09.008 GMT+0800 AM", "status": "OK", "total": 1, "data": [ { "offload": { "state": "Success", "progress": "100%", "startTime": "2023-02-17 00:22:47.352 GMT+0800", "endTime": "2023-02-17 00:27:00.014 GMT+0800", "offloadedDataSize": "5697.702 MB", "gateway": "10.163.173.99:8660" } } ] } </code></pre> <h2>Summary</h2> <p>In this article, I have demonsrated how to create a bucket in HPE Ezmeral Data Fabric Object store and upload data using the AWS CLI command line tool. Then, I showed you how to create a cold-tiered volume and configure it to use the remote Object Store as a remote target. Finally, I showed you how to manually trigger a volume data offload to verify that the whole work is populated. I hope this article was helpful to you. Catch you next time!</p>https://developer.hpe.com/2023-March-03/https://developer.hpe.com/2023-March-03/Fri, 03 Mar 2023 13:00:00 GMThttps://developer.hpe.com/open-sourcing-workshops-on-demand-part2-deploying-the-backend/https://developer.hpe.com/open-sourcing-workshops-on-demand-part2-deploying-the-backend/Wed, 01 Mar 2023 17:23:03 GMT<p>In the first <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">article</a> of this series, I described the reasons behind the decision to open source our Workshops-on-Demand (WoD) project and gave you a comprehensive picture of the project's overall infrastructure. In this second article, I will cover the backend part of the project and explain how to deploy it.</p> <p>The overall infrastructure can run on physical servers or VMs. We usually designate one server for the frontend and a second server for the backend. You could also decide to separate every single component of each side.</p> <p><img src="/img/howto-wod-5.png" alt=""></p> <h2>How to deploy your own backend...</h2> <p>As explained in the previous <a href="https://developer.hpe.com/blog/willing-to-build-up-your-own-workshops-on-demand-infrastructure/">article</a>, the project is split into multiple repositories from the architectural and public / private aspects. The architecture is divided between the frontend and backend. The project admins will need to decide whether they are willing to develop and propose public-only content to the participants or add any proprietary and private content.</p> <p>I will start with the simpliest scenario: A public-only approach. Then we will dive into the specificities related the private approach.</p> <h3>Public-only Deployment: No private backend nor private workshops</h3> <p><strong>Important Note:</strong></p> <p><strong>This part is compulsory for any type of deployment. Public only or public + private.</strong></p> <p>First, you need a repository to clone. The Workshops-on-Demand GitHub projects can be found <a href="https://github.com/Workshops-on-Demand/">here</a>. We have packaged the solution in several Github repos. Each repository handles a specific role in the overall architecture.</p> <p>Here's a quick look at what can be found in each:</p> <p><img src="/img/wod-blogserie2-repos.png" alt="" title="WOD Repositories"></p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-notebooks">wod-notebooks</a>:</strong> Public Workshops-on-Demand based on Jupyter Notebooks.</p> <ul> <li>You can test them live at <a href="https://hackshack.hpedev.io/workshops">https://hackshack.hpedev.io/workshops</a></li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-install">wod-install</a>:</strong> Installer part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-backend">wod-backend</a>:</strong> Back-end part of our Workshops-on-Demand setup.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend">wod-frontend</a>:</strong> Frontend part of the Workshops-on-Demand project.</p> <ul> <li>Based on NGINX and NodeJS technologies, it provides the participtants' Registration Portal used to enable booking of the workshops.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db">wod-api-db</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>Open API 3.0 based api used to manage the Workshops-on-Demand project. It also provides a database hosting the different status of participants, workshops, and students.</li> </ul> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-private">wod-private</a>:</strong> Example Private configuration for Workshops-on-Demand (WoD).</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-frontend-private">wod-frontend-private</a>:</strong> Private Frontend part of the Workshops-on-Demand project.</p> <p><strong><a href="https://github.com/Workshops-on-Demand/wod-api-db-private">wod-api-db-private</a>:</strong> Workshops-on-Demand registration portal application</p> <ul> <li>This provide examples for creating your own cutomization layer on top of the public standard WoD Backend / wod Notebooks content. Do not put any confidential data here as this is a public repository!</li> </ul> <p><strong>Note</strong>: There are now 7 repositories available for now.</p> <p><img src="/img/wod-blogserie2-2repos.png" alt="" title="Workshops-on-Demand repositories"></p> <p>It provides:</p> <ul> <li> <p>An Installer allowing you to install either Backend, Api-DB server, or Frontend using a single line of command.</p> </li> <li> <p>A complete JupyterHub server with some addons (additional Jupyterhub kernels, Ansible galaxies, and PowerShell libraries) on your system, ready to host Workshops-on-Demand that you can find here.</p> </li> <li> <p>A postfix server used for the procmail API</p> </li> <li> <p>An Ansible engine to allow automation</p> </li> <li> <p>A fail2ban service</p> </li> <li> <p>An Admin user to manage everything</p> </li> <li> <p>A set of scripts to handle different tasks such as:</p> <ul> <li>Notebooks deployment</li> <li>Jupyterhub compliancy</li> <li>Users compliancy</li> <li>Security Management</li> </ul> </li> </ul> <h4>Backend server preparation:</h4> <p>Before cloning the backend repository, you will need to prepare the server that will host the backend features. When ready, you will proceed with the cloning and then the installation process.</p> <h5>Prerequesites:</h5> <p>In order to setup the backend server, you will need:</p> <ul> <li>A fresh OS install on physical / virtualized server running Ubuntu 24.04 or Centos 7.9 leveraging any deployment mechanism of your choice.(e.g. iLO, vagrant, etc.). You may even use this vagrant file to automatically generate a complete setup leveraging vagrant, libvirt and QEMU/KVM.</li> <li>A Linux account with sudo priviledges on your Linux distro. Name it <code>install</code></li> </ul> <p><strong>Note</strong>: In order to support 100 concurrent users, you need:</p> <ul> <li>2 cpus or more machine</li> <li>128 GB of RAM</li> <li>500 GB of storage</li> </ul> <p>We are currently using an HPE ProLiant DL360 Gen10 server on our different production sites.</p> <p>When done with OS installation and preparation</p> <ul> <li>From the WoD-backend server (aka JupyterHub server), as the install user, you will need to clone the wod-install repo first.</li> </ul> <pre><code class="language-shellsession">install$ git clone https://github.com/Workshops-on-Demand/wod-install.git install$ cd wod-install/ </code></pre> <ul> <li>Examine default installation parameters and adapt when necessary accordingly. Files are self-documented.</li> </ul> <p>Look at the following files within ansible/group_vars directory.</p> <ul> <li><code>all.yml</code> file</li> </ul> <pre><code class="language-shellsession">vi all.yml --- # We create fixed user accounts to provide an isolated execution environment to run the jupyter notebooks # They are called studentXXX where XXX is set between USERMIN and USERMAX defined below potentially with the addition of an offset (UIDBASE) for their uid/gid # Their home directory is located under /student and is thus named /student/studentXXX # Corresponding JupyterHub accounts are also created # # USERMIN indicates the starting ID of the Linux and Jupyter user account range # USERMIN: 1 # # USERMAX indicates the ending ID of the Linux and Jupyter user account range # USERMAX: 20 # # UIDBASE is the offset used to create the Linux user account IDs # Example when creating user 35 with UIDBASE of 2000, the uid created is 2035 # UIDBASE: 2000 # # GIDBASE is the offset used to create the Linux group IDs # Example: When creating user 35 with GIDBASE of 2000, the gid created is 2035 # GIDBASE: 2000 # # Set CLEAN to true if you want all Liunx &#x26; Jupyter user accounts to be removed before ansible check # CLEAN: false # # VAULTPWD is the password used to manage the Ansible vault # VAULTPWD: VeryComplexPasswd1234! # # NOCHECKSSH are ssh options used to dialog with appliances # By default, avoid checking Host keys and Host file as they may change on a regular base # NOCHECKSSH: -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null # # Branding management - Use if you want to customize Logo and Notebooks branding # BRANDING: "WoD Developer" BRANDINGWOD: "WoD Developer" BRANDINGLOGO: "" BRANDINGURL: "https://wod.io" # # Survey management - Use if you want to ask for feedback on your workshops - Look at existing conclusion notebooks SURVEYURL: TBD SURVEYCHALURL: TBD # # JPHUB is the directory used to install the JupyterHub stack (a Python venv) # JPHUB: /opt/jupyterhub # # # These variables are defined in Ansible playbooks. Do not change without knowing what you are doing. # STUDDIR: "{{ ansible_env.STUDDIR }}" WODBEDIR: "{{ ansible_env.WODBEDIR }}" WODPRIVDIR: "{{ ansible_env.WODPRIVDIR }}" WODNOBO: "{{ ansible_env.WODNOBO }}" WODAPIDBDIR: "{{ ansible_env.WODAPIDBDIR }}" WODFEDIR: "{{ ansible_env.WODFEDIR }}" SCRIPTDIR: "{{ WODBEDIR }}/scripts" ANSIBLEDIR: "{{ WODBEDIR }}/ansible" # This is the predefined structure for a private repo WODPRIVNOBO: "{{ WODPRIVDIR }}/notebooks" SCRIPTPRIVDIR: "{{ WODPRIVDIR }}/scripts" ANSIBLEPRIVDIR: "{{ WODPRIVDIR }}/ansible" </code></pre> <ul> <li><code>wod-backend</code> file</li> </ul> <pre><code class="language-shellsession">vi wod-backend # # These variables are located lower in the ansible tree to allow different values required for different backends while keeping a single frontend # # BASESTDID is the offset used to create users in the DB. It is required that each backend has a different non overlapping value. # Overlap is defined by BASESTDID + USERMAX (from all.yml) # # Example: # for student 35 in location A having BASESTDID to 0 the user is create as id 35 # for student 35 in location B having BASESTDID to 2000 the user is create as id 2035 # There is no overlap as long as you do not create more than 2000 users which should be the value of USERMAX in that case. # # This is different from the offset UIDBASE used for Linux uid # BASESTDID: 0 # # POSTPORT is the Postfix Port on which the smtp service is listening to receive API mail requests from the frontend # POSTPORT: "10025" # # In case you are using an LDAP server to use, flag as such the corresponding workshops in the DB and use the following values: # LDAPSRVNAME: ldap.example.org LDAPDMN: example.org LDAPPWD: MotDePasseLDAPCompliquéAussi123!!!## LDAPPORT: "389" # # For various existing public WoDs - These are needed. Adapt but do not remove! # SSHPORT-WKSHP-Docker101: 14101 SSHPORT-WKSHP-Ansible101: 16001 HTTPPORT-WKSHP-Docker101: 14151 HTTPPORT-WKSHP-Ansible101: 16051 HTTPPORT-WKSHP-Spark101: 17161 HTTPPORT-WKSHP-Concourse101: 19061 HTTPPORT-WKSHP-ML101: 18061 HTTPPORT-WKSHP-DataVisu101: 22161 CONCOURSEPORT-WKSHP-Concourse101: 19001 CONCOURSEPORT2-WKSHP-Concourse101: 19031 IP-WKSHP-DataVisu101: x.y.z.t IP-WKSHP-Concourse101: x.y.z.t IP-WKSHP-Docker101: x.y.z.t IP-WKSHP-Ansible101: x.y.z.t IP-WKSHP-Spark101: x.y.z.t IP-WKSHP-ML101: x.y.z.t IP-WKSHP-StackStorm101: x.y.z.t SPARKPORT-WKSHP-Spark101: 17101 SPARKPORT2-WKSHP-Spark101: 17131 MLPORT-WKSHP-ML101: 18101 MLPORT2-WKSHP-ML101: 18031 DATAVISUPORT1-WKSHP-DataVisu101: 22101 DATAVISUPORT2-WKSHP-DataVisu101: 22131 </code></pre> <ul> <li><code>wod-system</code> file</li> </ul> <pre><code class="language-shellsession">vi wod-system # # Backend API management # # Do not change, as the port is fixed in the JupyterHub install # WODBEAPIURL: http://{{ WODBEFQDN }}:8000 # # Replace with a random one - TODO Do that automatically at install time # WODBETOKEN: 2c0246e2c8564dc6ac7b12c544b25d77 # # You may want to use these variables if you have an OPNSense server as a security FW and are allowing http comm internally # #OPNSENSEKEY: #OPNSENSESEC: #OPNSENSEIP: #OPNSENSEPORT: # # Front-end API management # # Do not change, as the port is fixed in the JupyterHub install # WODFEAPIURL: https://{{ WODAPIDBFQDN }}/api # # Adapt to your setup - Used by installer to setup the frontend # WODFEAPIUSER: moderator WODFEAPIPWD: MotDePasseCompliquéAussi125!!!## </code></pre> <p>See the example below for a backend server.</p> <h3>Backend installation process:</h3> <p>The installation process is handled by a dedicated repo : wod-install. This repo needs to be cloned on every single machine constituting the wod architecture.</p> <p>The wod installer allows through a single command to install either a backend, an api-db server, or a frontend.</p> <p><a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/INSTALL.md#for-private-based-workshops-on-demand-private-backend--private-workshops-or-if-you-need-to-modify-defaults"></a>Once you are done with the files, you can can proceed with the installation itself. The installation is based on a common install script <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/install/install.sh">install.sh </a>that allows the deployment of the different parts of the solution. It can be called as follows:</p> <p><code>install.sh [-h][-t type][-g groupname][-b backend][-f frontend][-a api-db][-e external][-u user][-s sender]</code></p> <p><code>-h</code> provides the help</p> <p><code>install.sh</code> performs the following tasks:</p> <ul> <li> <p>Calls the <code>install-system-&#x3C;&#x3C; distribution name >>.sh</code> script</p> </li> <li> <p>Installs minimal required (<code>ansible, git, jq, openssh server, npm</code>)</p> </li> <li> <p>Creates an admin user as defined upper (default is <code>wodadmin</code>) with sudo rights</p> </li> <li> <p>Calls the <code>install-system-common.sh</code> script that performs the following tasks:</p> <ul> <li>Cleanup</li> <li>Github repos cloning (leveraging install.repo file) : public Backend and public Private repos</li> <li>Create ssh keys for wodadmin</li> <li>Creates GROUPNAME variables</li> <li>Creates Ansible inventory files</li> </ul> </li> <li> <p>Calls the <code>install_system.sh</code> script with the type (backend, frontend, etc..) that performs the following tasks:</p> <ul> <li>Install the necessary stack based on selected type</li> <li>Create a <code>wod.sh</code> script in <code>wod-backend</code> directory to be used by all other scripts</li> <li>Source the <code>wod.sh</code> file</li> <li>Setup Ansible-galaxies (<code>community.general</code> and <code>posix</code>)</li> <li>Setup Ansible and call the playbook <code>install_&#x3C;type>.yml</code> followed by the <code>ansible\_check\_&#x3C;type>.yml</code></li> </ul> </li> </ul> <p>At the end of the installation process:</p> <ul> <li>you will have a JupyterHub server running on port 8000</li> <li>You will get a new <code>wodadmin</code> user (Default admin)</li> <li>You will get a set of 20 students (Default value)</li> </ul> <p>All playbooks are self-documented. Please check for details.</p> <p>We leave it to you to handle the necessary port redirection and SSL certificates management when needed. In our case, we went for a simple yet efficient solution based on an OPNSense Firewall along with a HAProxy setup to manage ports'redirection, HTTP to HTTPS Redirection, SSL Certificates. The backend also includes a Fail2ban service for login security management.</p> <p>At this point, you should be able to access your JupyterHub environment with a few pre-installed set of kernels like <code>Bash, Python, ansible, ssh, PowerShell</code>.</p> <p>You can then start developing new notebooks for your public based environment. And if you don't know how to do this, I will explain how in my next article</p> <p>If you need to develop private content that cannot be shared with the wider Open Source Community because of dedicated IP, the next section in this article will explain how to handle this.</p> <h3><strong>How to handle private-content based Workshops-on-Demand</strong></h3> <h4><em>(private backend + private workshops on top of default public backend and notebooks)</em></h4> <p>The principle remains similar, with a few differences explained below.</p> <ul> <li> <p>You will start by forking the following public private <a href="https://github.com/Workshops-on-Demand/wod-private.git">repo</a> on Github under your own Github account (we will refer to it as <code>Account</code>).</p> </li> <li> <p>Next, clone the forked repo.</p> </li> <li> <p>Edit the <code>all.yml</code> and <code>&#x3C;groupname></code> files to customize your setup. This variable <code>&#x3C;groupname></code> defines possible backend server in your environement. By default, the project comes with a sample working file named <code>production</code> in <code>ansible/group-vars</code>. But you could have multiple. In our case, we have defined <code>sandbox</code>, <code>test</code>, <code>staging</code> and several <code>production</code> files, all defining a different backend environment. These files will be used to override the default values specified by the public version delivered as part of the default public installation.</p> </li> <li> <p>Commit and push changes to your repo.</p> </li> <li> <p>Create an <code>install.priv</code> file located in <code>install</code> directory when using a private repo (consider looking at <a href="https://github.com/Workshops-on-Demand/wod-backend/blob/main/install/install.repo">install.repo</a> file for a better understanding of the variables).</p> </li> <li> <p>Define the WODPRIVREPO and WODPRIVBRANCH variables as follows:</p> <ul> <li><code>WODPRIVBRANCH="main"</code></li> <li><code>WODPRIVREPO="git@github.com:Account/Private-Repo.git wod-private"</code></li> </ul> </li> </ul> <p><strong>Note:</strong> When using a token</p> <p>Please refer to the following <a href="https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token">url</a> to generate a <code>token</code> file in <code>install</code> directory of WoD-backend:</p> <ul> <li> <p>Edit the <code>install.priv</code> file located in <code>install</code> directory of WoD-backend:</p> <ul> <li>Create line before variable declaration: <code>token=`cat $EXEPATH/token` </code></li> <li>Use the token in the url WODPRIVREPO="git clone <a href="https://user:$token@github.com/Account/wod-private.git%C2%A0wod-private">https://user:$token@github.com/Account/wod-private.git wod-private</a>"</li> </ul> </li> </ul> <p>You are now ready to perform the installation again to support a private repository.</p> <p>Please note that this setup phase can be concurrent with the public setup phase. Indeed, the install script should detect the presence of the private repository owing to the presence of the install.priv file. It will automatically adjust the different scripts and variables to add the relevant content. It will actually overload some of the variables with private ones.</p> <p>You now have a working Workshops-on-Demand backend server in place. Congratulations! The next article in the series will help you better understand the lifecycle of the backend server. How does a workshop registration work from the backend server 's side? How do you manage this server on a daily basis? How and when do you need to update it ? All these questions will be answered in the next article. And from there, we will move to the frontend side of things and finally to a workshop's creation process.</p> <p>If you need support for this installation process, use our dedicated <a href="https://hpedev.slack.com/archives/C01B60X8SSD">slack channel</a>.</p> <p>Please be sure to check back <a href="https://developer.hpe.com/blog">HPE Developer blog site</a> to read all the articles in this series. Also, check out the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a> <a href="https://developer.hpe.com/hackshack/replays/42">Data Visualization 101</a> is now available! Stay tuned for additional Workshops-on-Demand in our catalog.</p>https://developer.hpe.com/a-guide-to-enable-managed-istio-service-mesh-in-a-kubernetes-cluster-on-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/a-guide-to-enable-managed-istio-service-mesh-in-a-kubernetes-cluster-on-hpe-greenlake-for-private-cloud-enterprise/Thu, 16 Feb 2023 13:36:32 GMT<h2>Introduction</h2> <p>In this blog post, we demonstrate how an end user can deploy a containerized application or a managed service on a Kubernetes-based container stack using the cluster add-on feature provided by <strong>HPE GreenLake for Private Cloud Enterprise: Containers</strong> and then access it over an external network or internet. The containers service evaluates the user’s environment and makes add-ons available to the user so that they can add the containerized application or managed service to the cluster as required.</p> <p>For those of you who may be unfamiliar with the term, a <strong>Service mesh</strong> is a network of microservices that consist of distributed applications and communications between those applications. It is a dedicated infrastructure layer that facilitates service-to-service communications routed through the proxy, ensuring secure communication.</p> <p><strong>Istio</strong> is an open-source service mesh that provides a platform for distributed applications that includes API integrations with logging, telemetry, or policy systems. It provides a uniform and more efficient way to secure, connect, and monitor services. Istio automatically manages load balancing for HTTP, gRPC, WebSocket, and TCP traffic. For details, see <strong><a href="https://istio.io/latest/about/service-mesh/">The Istio service mesh</a></strong>.</p> <h2>Enabling Istio service mesh add-on from a cluster</h2> <h3>Step-1: Create a Kubernetes cluster from the containers page</h3> <p>To create a cluster, you must have been assigned the roles of <strong>Private Cloud Cluster Owner</strong> and <strong>Private Cloud Widget Viewer</strong>.</p> <ul> <li>From the <strong>Containers</strong> main page, under the <strong>Clusters</strong> tab, click <strong>Create Cluster</strong>.</li> <li>In the <strong>Create Cluster</strong> form, provide the cluster name '<strong>hpe</strong>', and select the standard cluster blueprint. The new cluster appears in the list of clusters.</li> </ul> <p><img src="/img/clustermainpage-2.png" alt="" title="Clusters view"></p> <p>As indicated above, there are multiple clusters deployed in parallel for multiple purposes. For the <strong>Istio</strong> service mesh add-on enablement/deployment in our example, we are using a cluster created with the name "<strong>hpe</strong>".</p> <p><img src="/img/clusterhpeview.png" alt="" title="Cluster &#x27;hpe&#x27; view"></p> <h3>Step-2: Enabling an add-on from a cluster</h3> <ul> <li>On the <strong>Containers</strong> main page, click a cluster row to open the cluster details screen.</li> <li>On the cluster details screen, click the <strong>Add-ons</strong> tab.</li> <li>Click <strong>Enable add-on</strong> if no add-ons are enabled or click <strong>Enable another add-on</strong>.</li> </ul> <p><img src="/img/blankaddonpage.png" alt="" title="Add-ons view"></p> <ul> <li>In the <strong>Enable Addon</strong> wizard, select the <strong>Istio-service-mesh</strong> add-on and click <strong>Next</strong>.</li> </ul> <p><img src="/img/istioaddonpage-11.png" alt="" title="Select Add-on view"></p> <ul> <li>Provide the values for the fields that appear for the selected add-on, read and accept the license agreement, and click <strong>Enable</strong>.</li> </ul> <p><img src="/img/istioaddonpage-22.png" alt="" title="Selected Add-on Istio-service-mesh view"></p> <ul> <li>After successful add-on enablement, add-on status will get updated to '<strong>succeeded</strong>'.</li> </ul> <p><img src="/img/istioaddongreenstatus.png" alt="" title="Add-ons view"></p> <ul> <li>View the details of the add-on that you just enabled.</li> </ul> <p><img src="/img/istioaddondetailspage.png" alt="" title="Add-on Istio-service-mesh overview"></p> <h3>Step-3: Launching the Kiali dashboard - the console for Istio service mesh</h3> <p><strong>Kiali</strong> is an open-source project that provides observability for the Istio service mesh.</p> <p>From the <strong>Overview</strong> tab, click the <strong>KialiURL</strong> link and use the <strong>Kiali token</strong>.</p> <p>The <strong>Kiali dashboard</strong> launches in a new web page.</p> <p><strong>Note</strong>: The URL for the Kiali console might be different in your environment.</p> <p><img src="/img/kiali-console.png" alt="" title="Kiali console view"></p> <p><strong>Note</strong>: To take advantage of the Istio features, pods in the mesh must be running an Istio sidecar proxy. Injection of the proxy can be done either on a per-pod basis or at namespace level. To enable side car injection, refer to the <a href="https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/">setup instructions</a>. For information about using Kiali, see the <strong><a href="https://kiali.io/docs/">Kiali documentation</a></strong>.</p> <h3>Step-4: Download scoped kubeconfig from the container platform page</h3> <ul> <li>From the <strong>Clusters</strong> tab, select the '<strong>hpe</strong>' Kubernetes cluster and click <strong>Launch Service Console</strong>. This will direct you to the container platform page.</li> <li>Click on Download <strong>kubeconfig</strong>.</li> </ul> <p><strong>Note</strong>: Launching the service console from HPE GreenLake Central is configured through SAML SSO and adds a session token to the kubeconfig file. You will need to download the kubeconfig file again if you want to continue to access the cluster when the session token expires after an hour.</p> <h3>Step-5: Deploying a sample Istio application: Bookinfo</h3> <p>This procedure follows the standard Istio documentation to deploy a sample application. To know more about Bookinfo Application, see the <strong><a href="https://istio.io/latest/docs/examples/bookinfo/">Istio documentation</a></strong>.</p> <p><strong>Use the following commands to create the namespace and label for Istio sidecar proxy injection to deploy the application in the bookinfo namespace</strong>.</p> <pre><code class="language-shellsession">$ kubectl create namespace bookinfo namespace/bookinfo created $ kubectl label namespace bookinfo istio-injection=enabled namespace/bookinfo labeled $ kubectl get namespace bookinfo --show-labels NAME STATUS AGE LABELS bookinfo Active 105s gl.hpe.com/namespaceid=10d70074-0c2b-4221-804e-1437ed1842ca,hpe.com/cluster=stub,hpe.com/namespacetype=Tenant,hpe.com/tenant=bookinfo,hpe.com/version=6.2.0,hpecp.hpe.com/hpecptenant=hpecp-tenant-106,istio-injection=enabled,kubernetes.io/metadata.name=bookinfo,serving.kubeflow.org/inferenceservice=enabled </code></pre> <p><strong>Deploy the Bookinfo application using the YAML manifest file i.e. services/istio/release-1.16/samples/bookinfo/bookinfo.yaml from the</strong> <a href="https://github"></a><a href="https://github.com/cxteamtrials/caas-trials-content">Github repository</a>.</p> <pre><code class="language-shellsession">$ kubectl apply -f bookinfo.yaml -n bookinfo service/details created serviceaccount/bookinfo-details created deployment.apps/details-v1 created service/ratings created serviceaccount/bookinfo-ratings created deployment.apps/ratings-v1 created service/reviews created serviceaccount/bookinfo-reviews created deployment.apps/reviews-v1 created deployment.apps/reviews-v2 created deployment.apps/reviews-v3 created service/productpage created serviceaccount/bookinfo-productpage created deployment.apps/productpage-v1 created </code></pre> <p><strong>Confirm all pods and services are deployed successfully</strong>.</p> <pre><code class="language-shellsession">$ kubectl get pods,services -n bookinfo NAME READY STATUS RESTARTS AGE details-v1-698b5d8c98-qglhw 2/2 Running 0 6m17s productpage-v1-bf4b489d8-bkpdm 2/2 Running 0 6m17s ratings-v1-5967f59c58-28kc5 2/2 Running 0 6m17s reviews-v1-9c6bb6658-mw2df 2/2 Running 0 6m17s reviews-v2-8454bb78d8-p4h9d 2/2 Running 0 6m17s reviews-v3-6dc9897554-g7xqh 2/2 Running 0 6m17s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE details ClusterIP 10.98.141.15 &#x3C;none> 9080/TCP 14m productpage ClusterIP 10.104.123.90 &#x3C;none> 9080/TCP 6m45s ratings ClusterIP 10.108.60.57 &#x3C;none> 9080/TCP 6m46s reviews ClusterIP 10.106.208.181 &#x3C;none> 9080/TCP 14m </code></pre> <p><strong>Configure the service to access the application outside of the cluster</strong>.</p> <ul> <li>Edit the deployed service <strong>productpage</strong>.</li> <li>Change service type from ClusterIP to <strong>NodePort</strong>.</li> <li>Add the label <strong>hpecp.hpe.com/hpecp-internal-gateway=true</strong>. The service will be automatically mapped/exposed to a <strong>Container platform gateway host</strong> with an assigned port.</li> </ul> <pre><code class="language-shellsession">$ kubectl describe svc productpage -n bookinfo Name: productpage Namespace: bookinfo Labels: app=productpage hpecp.hpe.com/hpecp-internal-gateway=true service=productpage Annotations: hpecp-internal-gateway/9080: epicgw.customer.hpe.net:10072 Selector: app=productpage Type: NodePort IP Family Policy: SingleStack IP Families: IPv4 IP: 10.104.123.90 IPs: 10.104.123.90 Port: http 9080/TCP TargetPort: 9080/TCP NodePort: http 31766/TCP Endpoints: 10.192.3.181:9080 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HpeCp 21s hpecp-agent Created HPECP K8S service </code></pre> <p><strong>Confirm the application is accessible from outside the cluster</strong>.</p> <p>The Bookinfo application <strong>productpage</strong> can be accessed in the browser by typing the URL <strong><a href="http://epicgw.customer.hpe.net:10072">http://epicgw.customer.hpe.net:10072</a></strong></p> <p>Note : The above URL might be different in your environment. You can form the URL by referencing annotations from the <strong>productpage</strong> service.</p> <p><img src="/img/bookinfo-productpage.png" alt="" title="Bookinfo application default view"></p> <p><img src="/img/bookinfo-productpage-normal-user.png" alt="" title="Bookinfo application productpage view"></p> <h2>Monitor the sample application using the Kiali dashboard</h2> <p>Enter <strong>bookinfo</strong> into the field Filter by Namespace. The Kiali overview screen displays the details about the namespace bookinfo. It shows that 4 applications are running in the <strong>namespace bookinfo</strong> with no inbound traffic.</p> <p><img src="/img/kiali-bookinfo.png" alt="" title="Kiali overview"></p> <p>In the <strong>Graph</strong> tab from the left navigation menu, after selecting the <strong>namespace bookinfo</strong>, the screen shows an overview topology of the Bookinfo application.</p> <p><img src="/img/kiali-console-graph.png" alt="" title="Kiali graph view"></p> <p>In the <strong>Applications</strong> tab from the left navigation menu, after selecting the <strong>namespace bookinfo</strong>, the screen shows application details of the Bookinfo application.</p> <p><img src="/img/kiali-console-applications.png" alt="" title="Kiali applications view"></p> <p>In the <strong>Workloads</strong> tab from the left navigation menu, after selecting the <strong>namespace bookinfo</strong>, the screen shows deployment details of the Bookinfo application.</p> <p><img src="/img/kiali-console-workloads.png" alt="" title="Kiali workloads view"></p> <p>In the <strong>Services</strong> tab from the left navigation menu, after selecting the <strong>namespace bookinfo</strong>, you can check all the services details of the Bookinfo application.</p> <p><img src="/img/kiali-console-services.png" alt="" title="Kiali services view"></p> <h2>Summary</h2> <p>You can find the GitHub repository that hosts demo code <strong><a href="https://github.com/cxteamtrials/caas-trials-content">here</a></strong>.</p> <p>We hope that this blog post has provided you with enough information for you to get started deploying containerized application or a managed service <strong>i.e. Istio service mesh</strong> on a Kubernetes-based container stack using the cluster add-on feature provided by <strong>HPE GreenLake for Private Cloud Enterprise: Containers</strong>.</p> <p>To view more articles and tutorials on the use of the HPE GreenLake for Private Cloud Enterprise, refer to the <strong><a href="https://developer.hpe.com/">HPE Developer Community blog</a></strong>.</p>https://developer.hpe.com/smart-use-cases-for-smartsim/https://developer.hpe.com/smart-use-cases-for-smartsim/Mon, 13 Feb 2023 08:45:53 GMT<p><a href="https://developer.hpe.com/platform/smartsim/home">SmartSim</a> is an open source library that aims to bridge the divide between traditional numerical simulations and data science tools. SmartSim started as a project at Cray before Cray was acquired by Hewlett Packard Enterprise (HPE). SmartSim was later open sourced under the BSD-2 license.</p> <p><a href="https://www.hpe.com/us/en/compute/hpc/hpc-software.html">High performance computing software</a> (HPC) takes advantage of parallel computing techniques. A large portion of the traditional HPC applications are written in C, C++, or Fortran whereas data science toolkits are often written in Python and NumPy (which is used for scientific computing in Python). This can make it difficult to bridge the gap between the data science world and HPC. SmartSim is a software library aimed at providing the glue between these two worlds.</p> <p>SmartSim enables users to use machine learning models inside their existing Fortran/C/C++ simulations. It does so by allowing them to push and pull data to/from these applications and an in-memory, highly performant database. Because this database also supports the storage of ML models, SmartSim also allows coders to enable online training of machine learning models and use those models to make predictions inside of that same code. In essence, SmartSim tries to take the world of traditional HPC applications and bring it together with modern data science tools.</p> <p>With SmartSim, you get an infrastructure library component and a client library component. The infrastructure library helps users set up, configure, run and monitor simulations from a Python script. The SmartSim infrastructure library basically provides a lightweight API for users to create what we call experiments. These experiments may be a combination of traditional HPC applications and other data science tools. SmartSim’s infrastructure library automatically deploys the infrastructure on the compute resources that’s needed to accomplish whatever that experiment describes (e.g., having a tensor flow backend to do online inference).</p> <p>The client library that's a part of SmartSim enables users to embed this machine learning capability inside of traditional applications. The library is available in four languages, Python, C, C++ and Fortran. It provides a very simple API that allows the transfer of data between the traditional applications and the database. The simplicity of the API allows users to rapidly integrate their application with only a minimal amount of modification to the original source code (usually less than 10 lines).</p> <p>One of the early use cases that the designers focused on was using SmartSim to enable better predictions inside of a global ocean model. The <a href="https://www.gfdl.noaa.gov/mom-ocean-model/">MOM6 (Modular Ocean Model)</a> global ocean model has an algorithm that calculates a term called Eddy Kinetic Energy (EKE) which governs the strength of turbulence. It turns out that the parameterization is not very accurate. So, using SmartSim, we trained a machine learning model on a high fidelity run to learn the mapping from a coarsened version of the quantities (similar to those in the low fidelity run) to the true EKE. Not only did that machine learning model improve the accuracy by at least 20%, but there was no visible degradation in performance, even though this simulation required about 1.6 million inferences per second.</p> <p>SmartSim is great for users who have very specialized domain expertise in things like computational fluid dynamics or molecular dynamics; but don’t necessarily have the knowledge set to deploy the machine learning infrastructure for high performance. SmartSim removes that technical barrier for users so that they can spend more time experimenting with combining scientific simulations and ML.</p> <p>For more information on SmartSim, please reference the <a href="https://developer.hpe.com/platform/smartsim/home">SmartSim page</a> found on the <a href="https://developer.hpe.com/">HPE Developer Community portal</a>. You can also access <a href="https://github.com/CrayLabs/SmartSim">SmartSim’s GitHub page</a>.</p>https://developer.hpe.com/2023-February-01/https://developer.hpe.com/2023-February-01/Wed, 01 Feb 2023 13:00:00 GMThttps://developer.hpe.com/learn-what-you-can-do-with-hpe-data-services-cloud-console-api-in-just-3-minutes/https://developer.hpe.com/learn-what-you-can-do-with-hpe-data-services-cloud-console-api-in-just-3-minutes/Fri, 27 Jan 2023 14:17:03 GMT<p><a href="https://developer.hpe.com/greenlake/data-services-cloud-console/home/">HPE Data Services Cloud Console</a>, available through the <a href="https://developer.hpe.com/greenlake/hpe-greenlake-cloud-platform/home/">HPE GreenLake edge-to-cloud platform</a>, is a Software-as-a-service (SaaS) based cloud console application that delivers a suite of cloud data services that enable a unified data operations as a service for storage infrastructure, simplify storage and data management, and bring the cloud experience to wherever data lives. Data Services Cloud Console also offers a unified and fully programmable API that enables developers to automate data infrastructure management.</p> <p>If you’re looking for a quick way to discover everything you can do with the HPE GreenLake Data Services Cloud Console API using popular tools that doesn’t require programming such as Postman, this blog post is definitely for you.</p> <p>As you know, one of the benefits of working within a community is the ability to take advantage of open collaboration, sharing hints, tools, and resources. This is exactly what I am doing here. This post helps you get started with the Data Services Cloud Console API for HPE GreenLake for Block Storage cloud data service by taking advantage of the Postman collection contributed by one of our HPE Developer Community members.</p> <blockquote> <p><strong>Note:</strong> This blog post assumes you have created an <a href="https://console.greenlake.hpe.com/">HPE GreenLake account</a> and joined your account to your company account (also called an <em><strong>organization</strong></em>). You also got assigned appropriate roles and permissions by the administrator for your organization in order to access HPE data services resources (for example storage arrays and volumes) through the Data Services Cloud Console application instances. A Data Service Cloud Console application instance is a service cluster running in one of the HPE regions.</p> </blockquote> <h3>Data Services Cloud Console and REST API</h3> <p>Data Services Cloud Console supports a set of REST APIs that allows users to integrate HPE Data Services Cloud Console with their custom applications. By using <a href="https://oauth.net/2/">OAuth 2.0 protocol</a> to authenticate and authorize applications, secure and time-limited (<em><strong>120 minutes</strong></em>) access to HPE data services are provided via an <strong>access token</strong>. The token ensures that client API requests access HPE data services for the requested operation securely and according to the authorization granted to the user who created it.</p> <blockquote> <p><strong>Note:</strong> You can find the Data Services Cloud Console API documentation <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">here</a> and in the help section of the <a href="https://console.greenlake.hpe.com/">HPE GreenLake Cloud Platform</a>.</p> </blockquote> <p>The REST APIs support standard HTTP request methods (GET, POST, PATCH, PUT and DELETE). A HTTP request is made by providing a specific HPE regional connectivity endpoint for the Data Service Cloud Service application instance, HTTP request method, access token and data payload. The HTTP response for these requests are returned in a JSON format.</p> <p>Currently, there are three HPE regional Data Services Cloud Console application instance endpoints:</p> <ul> <li> <p>EU Central</p> <ul> <li><a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></li> </ul> </li> <li> <p>AP Northeast</p> <ul> <li><a href="https://jp1.data.cloud.hpe.com">https://jp1.data.cloud.hpe.com</a></li> </ul> </li> <li> <p>US West</p> <ul> <li><a href="https://us1.data.cloud.hpe.com">https://us1.data.cloud.hpe.com</a></li> </ul> </li> </ul> <p>HPE GreenLake Cloud Platform allows developers to make API calls on a particular regional Data Services Cloud Console customer instance. Using the API functionality in the HPE GreenLake Cloud Platform graphical user interface (GUI), developers can create their <strong>API client application</strong> credentials. The credentials consist of a <em>ClientID-ClientSecret</em> pair that represents the permissions granted to the user who creates the API client application credentials. The credentials are then used to generate and refresh expired OAuth based access token. Once the token is generated or refreshed, it can be used as an <strong>authorization bearer token</strong> to make further secure REST API calls to protected HPE data services resources via the regional Data Services Cloud Console application instance.</p> <p>You can refer to <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">this blog post</a> to learn how to create API client application credentials for your specific regional Data Services Cloud Console application instance. Make sure to copy the <em>ClientID</em> and <em>ClientSecret</em> values to a safe location as you will need them to generate the access token via a REST API call, which is necessary to complete the following steps.</p> <p><strong>Ready? Let’s get started!</strong></p> <h3>Step 1 – Sign in to your Postman account</h3> <p>You can sign in to your Postman account either from the <a href="https://identity.getpostman.com/login">web app</a> or from the desktop app. If you don’t have a Postman account already, you can sign up for a Postman account <a href="https://identity.getpostman.com/signup">here</a> or download the desktop app <a href="https://www.postman.com/downloads/">here</a>.</p> <h3>Step 2 – Copy the existing HPE GreenLake Data Services Cloud Console API public collection</h3> <p>Upon log in to your Postman account, from the <em><strong>Search</strong></em> bar, look for the public collection "<strong>Data Services Cloud Console API</strong>" and select the public collection from our community contributor, <a href="mailto:mark.van.silfhout@hpe.com">Mark van Silfhout</a>, as shown below:</p> <p><img src="/img/search-dscc-collection-postman-figure1.png" alt="The Data Services Cloud Console API public collection" title="The Data Services Cloud Console API public collection"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 1: The Data Services Cloud Console API public collection.</span></p> <p>Fork the collection to make a copy of it in your Postman workspace. This allows you to work with your own copy of the collection and perform changes without affecting the parent collection. Select the <em><strong>more actions</strong></em> icon (the ellipsis) next to the collection, then select <em><strong>Create a fork</strong></em>. When you fork the public collection, you can choose to watch the original collection to be notified about changes made to the parent collection. This allows you to pull updates from the parent collection into your forked copy, should the parent collection be updated.</p> <p>Alternatively, you can export the public collection locally to your local storage as JSON file and import it as a new collection in your Postman workspace.</p> <blockquote> <p><strong>Note:</strong> You must sign in to your Postman account to create a fork or export the public collection. To fork a collection within a public workspace, you must also enable your public profile in your Postman profile settings.</p> </blockquote> <h3>Step 3 – Set your Data Services Cloud Console collection variables</h3> <p>The Data Services Cloud Console API collection built by Mark makes use of collection variables that are available throughout the REST API requests in the collection. Select the collection in your workspace and then select the <em><strong>Variables</strong></em> tab as shown below.</p> <p><img src="/img/variables-dscc-collection-postman-figure2.png" alt="HPE Data Services Cloud Console collection variables" title="HPE Data Services Cloud Console collection variables"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 2: HPE Data Services Cloud Console collection variables.</span></p> <p>Define the <em>current value</em> of the collection variables to match your Data Services Cloud Console context:</p> <ul> <li> <p><strong>baseUrl</strong>: This variable defines the base URL of the REST API requests. It should match the regional endpoint of your Data Services Cloud Console application instance where your storage devices are registered.</p> </li> <li> <p><strong>ClientId</strong> and <strong>ClientSecret</strong>: these should be set with the value of your Client Application API credentials you previously created using the HPE GreenLake Cloud Platform GUI. These variables are used to request an OAuth access token by authenticating with the authorization server referenced in the <strong>sso_URI</strong> variable.</p> </li> <li> <p><strong>sso_URI</strong>: This variable is the URI of the OAuth authorization server. If your organization has set up their own HPE GreenLake SAML Single Sign-On (SSO) authorization server to create an access token, replace the current default value with your SSO URI. Otherwise keep the value for this variable as currently set to <em>sso.common.cloud.hpe.com/as/token.oauth2</em>.</p> </li> <li> <p><strong>BearerToken:</strong> Do not edit this variable. Keep the value field empty. The collection variable BearerToken will be set automatically upon successful execution of the <em><strong>GetToken</strong></em> API call as explained in the next step.</p> </li> </ul> <h3>Step 4 – Acquire an OAuth access token as your session bearer token</h3> <p>Data Services Cloud Console API uses a bearer token as an authorization type to ensure that all REST API requests access authorized data services securely. So you first need to obtain a token from the OAuth authorization server before you can make any REST API calls to your regional Data Services Cloud Console application instance. To do so, proceed as follows:</p> <ul> <li> <p>From your collection, generate the token using the <em><strong>GetToken</strong></em> API call from the <em><strong>GetToken-Using-Variables</strong></em> folder.</p> </li> <li> <p>Verify you get a status code of 200 for a successful response with the token value in the response body.</p> </li> <li> <p>Check that the token value has been automatically defined for the collection variable <em><strong>BearerToken</strong></em>.</p> </li> </ul> <p>The <em>GetToken</em> API call has defined a script in the <em><strong>Tests</strong></em> tab to programmatically set the collection variable BearerToken as shown in the picture below. The programmatically defined token is then used to authenticate any subsequent REST API calls.</p> <p><img src="/img/tests-capturebearertoken-dscc-collection-postman-figure3.png" alt="Defining collection variables programmatically in script" title="Defining collection variables programmatically in script"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 3: Defining collection variables programmatically in script.</span></p> <blockquote> <p><strong>Note:</strong> Access bearer token expires after 120 minutes. Run the <em>GetToken</em> API request again to refresh the token before or after it expires.</p> </blockquote> <h3>Step 5 – Make subsequent secure REST API calls</h3> <p>The client REST API requests are authenticated by presenting the access token as the authorization bearer token to the regional Data Services Cloud Console application instance. The instance validates the access token, and if valid, serves the request.</p> <p>Pick one REST API call from the <em><strong>storage-systems</strong></em> folder to <em><strong>Get all storage systems</strong></em> registered with your regional Data Services Cloud Console application instance.</p> <p>As shown in the two pictures below, all REST API requests in the collection will inherit the authorization bearer token that is specified at the collection level.</p> <p><img src="/img/authorization-dscc-collection-postman-figure4.png" alt="Authorization type (bearer token) specified at the collection level" title="Authorization type (bearer token) specified at the collection level"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 4: Authorization type (bearer token) specified at the collection level.</span></p> <p><img src="/img/authorization-inherited-dscc-collection-postman-figure5.png" alt="REST API request with authorization type inherited from parent collection" title="REST API request with authorization type inherited from parent collection"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 5: REST API request with authorization type inherited from parent collection.</span></p> <p>As depicted in the figure below, the API supports several query parameters (click on Params tab in the request) depending on the resource type, such as filter (filter the set of resources returned), limit (maximum number of records to return), offset (resource offset to start the response from) and sort (order in which to return the resources in the collection).</p> <p><img src="/img/getallstoragesystems-dscc-collection-postman-figure6.png" alt="REST API request with query parameter to search for storage array of type HPE Alletra 9060" title="REST API request with query parameter to search for storage array of type HPE Alletra 9060"></p> <p><span style="color:grey; font-family:Arial; font-size:1em">Figure 6: REST API request with query parameter to search for storage array of type HPE Alletra 9060.</span></p> <p>The query parameters are indicated after the question mark (“?”) in the REST API URL. Select and adjust the query parameters according to your environment. Make sure to refer to <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">the API documentation</a> to understand the query parameters that can be used for each HPE Data Services Cloud Console API request.</p> <p>Finally click the <strong>Send</strong> button. You will get a JSON representation of the storage system resources registered based on the query parameters specified. Here, I am getting the list of storage systems of type “<em>HPE Alletra 9060</em>”.</p> <pre><code class="language-json">{ "items": [ { "systemWWN": "2FF70002AC07EB29", "name": "s3294", "id": "4UW0003294", "description": "System s3294", "mgmtIp": "16.182.41.5", "softwareVersions": "9.5.0", "model": "HPE Alletra 9060", "productFamily": "deviceType1", "state": "NORMAL", "callhomeStatus": "ENABLED_NORMAL", "resourceUri": "/api/v1/storage-systems/device-type1/4UW0003294", "upSince": 1670726169000, "fqdn": "s3294.mip.storage.hpecorp.net", "capacityDetail": { "volumeSpace": 21092875, "snapSpace": 1676167, "totalUsedSpace": 22769042 }, "associatedLinks": [ { "type": "storage-pools", "resourceUri": "/api/v1/storage-systems/4UW0003294/storage-pools" }, { "type": "volumes", "resourceUri": "/api/v1/storage-systems/4UW0003294/volumes" }, { "type": "swupdatestatus", "resourceUri": "/api/v1/storage-systems/4UW0003294/swupdate/status" } ], "arrayList": null, "collectionStatus": { "metricStatus": { "status": "NORMAL" }, "configStatus": { "status": "NORMAL" }, "overAllStatus": "NORMAL" }, "lastConnectedTime": 1674674119, "connectionStatus": "CONNECTED", "customerId": "c93de31c382811ecb478320c1ce21c93", "generation": 1674674119, "type": "storage-system", "tierType": "STORAGE_TIER_9000_NVME", "tierName": "HPE Alletra 9000" } ], "total": 1, "pageLimit": 50, "pageOffset": 0, "requestUri": "https://us1.data.cloud.hpe.com/api/v1/storage-systems?filter=model%20eq%20%22HPE%20Alletra%209060%22" </code></pre> <p><strong>That’s it. No more than 3 minutes!</strong></p> <h2>Summary</h2> <p>This blog gives you a great example on how to obtain the access token using an API call and should help you get started with the Data Services Cloud Console REST API for the <em>HPE GreenLake for Block Storage</em> data service using Postman. Additional HPE data services APIs will be published in accordance to the expansion of cloud data services delivered through the HPE Data Services Cloud Console. Make sure you stay tuned for any update of Mark’s public Postman collection for Data Services Cloud Console API.</p> <p>If you have more time, I invite you to further explore the rest of the collection on your own, while adjusting the query parameters to match your Data Services Cloud Console context. Also, don’t hesitate to provide Mark with feedback on his very convenient collection.</p> <p>Any questions on HPE GreenLake Data Services Cloud Console API? Please join the <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a> and start a discussion in our <a href="https://hpedev.slack.com/archives/C02D6H623JP">#hpe-greenlake-data-services-cloud-console</a> channel.</p>https://developer.hpe.com/willing-to-build-up-your-own-workshops-on-demand-infrastructure/https://developer.hpe.com/willing-to-build-up-your-own-workshops-on-demand-infrastructure/Tue, 24 Jan 2023 12:35:46 GMT<h1>A bit of background</h1> <p><a href="https://developer.hpe.com/hackshack/workshops/">The Workshops-on-Demand program </a> has been an important asset for the HPE Developer Community for the last 2 years. If you are interested in learning more on the genesis of the project, check the following <a href="https://developer.hpe.com/blog/from-jupyter-notebooks-as-a-service-to-hpe-dev-workshops-on-demand/">blog</a>.</p> <p>This program allows us to deliver free hands-on workshops. We use these during HPE sponsored events, like HPE Discover and HPE Technical Symposium Summit, as well as open source events, like Open Source Summit and KubeCon, to promote API/automation-driven solutions along with some 101-level coding courses. By the end of 2022, more than 4000 had registered for our workshops.</p> <p>If you are interested in creating your own training architecture to deliver workshops, this project is definitely for you. It would allow you to create, manage content and deliver it in an automated and very efficient way. Once ready, the infrastructure can deliver workshops in matter of minutes!</p> <p>Trying to build a similar architecture on your own is obviously possible, but we integrated so much automation around the different components like the JupyterHub server deployment along with multiple pre installed kernels, user management and much more. When leveraging our project, one can actually get a proper public-based environment in 2 hours.</p> <p>This very first article will set the stage for the following blog articles where I will explain how to setup your own Workshops-on-Demand infrastructure.</p> <h2>Why would we consider open sourcing our Workshops-on-Demand?</h2> <p>Firstly, if you read carefully the messaging on our <a href="https://developer.hpe.com/">homepage</a> , you will find words like sharing and collaborating. This is part of the HPE Developer team's DNA.</p> <p>Secondly, the project is based on open source technologies like Jupyter and Ansible. It felt natural that the work we did leveraging these should also benefit the open source community.</p> <p>We have, actually, shared the fundamentals of the project thoughout the HPE Developer Community, and to a wider extent, the Open Source Community through different internal and external events. And the feedback has always been positive. Some people found the project very appealing. Originally, and long before even thinking of open sourcing the project, when we really started the project development, people were mainly interested in the content and not necessarily in the infrastructure. The students wanted to be able to reuse some of the notebooks. And in a few cases, they also asked for details about the infrastructure itself, asking about the notebooks delivery mechanism and other subjects like the <a href="https://www.youtube.com/watch?v=zZm6ObQATDI">procmail API</a>.</p> <p>Early last year, we were contacted by an HPE colleague who was willing to replicate our setup in order to deliver notebooks to its AI/ML engineers. His aim was to provide a simple, central point from which he could deliver Jupyter Notebooks, that would later be published on the Workshops-on-Demand infrastructure frontend portal, allowing content to be reused and shared amongst engineers. While, over time, we had worked a lot on automating content delivery and some parts of the infrastructure setup, we needed now to rework and package the overall solution to make it completely open source and reusable by others.</p> <p>As a result of our work on that project, over the course of 2022 we started to open source the Workshops-on-Demand program. As a project developed within the confiines of Hewlett Packard Enterprise (HPE), we had a number of technical, branding, and legal hurdles we needed to overcome in order to achieve this.</p> <h4>Legal side of things</h4> <p>From a legal standpoint, we needed to go through the HPE OSRB (Open Source Review Board) to present the project that we wanted to open source. We were asked to follow a process that consisted of four steps:</p> <p><img src="/img/wod-osrb1.png" alt="HPE OSRB Process" title="HPE OSRB process"></p> <p>As the project did not contain any HPE proprietary software, as it is based on open source technologies like Ansible and Jupyter, the process was quite straightforward. Besides, HPE did not want to exploit commercially the generated intellectual property. We explained to the OSRB that the new architecture of the solution would allow the administrator of the project to separate public content from private content, with private content being a proprietary technology.</p> <p>This had a huge influence on the future architecture of the project that originally did not allow it. In our case, for instance, any workshop related to an HPE technology like HPE Ezmeral, would fall into the private part of the project, and therefore, would not appear on the public github repository that we had to create for the overall project distribution.</p> <h4>Technical side of things</h4> <p>From a technical standpoint, as mentioned above, we had to make sure to separate the public only content from any possible private content. We started by sorting the different workshops (public vs private based). We also had to sort the related scripts that come along with the workshops. Going through this process, we found out that some of the global scripts had to be reworked as well to support any future split of public and private content. Similarly, we had to address any brand specifics, parameterizing instead of hardcoding variables as it was in the first version.</p> <p>This took us a few months and we are now ready to share with you the result of this work. In this first blog, I will focus on the architecture side of the Workshops-on-Demand project.</p> <p>Further blog articles will help you setup your own architecture.</p> <h2>The How</h2> <h2>Understand the architecture first</h2> <p>The Workshops-on-Demand concept is fairly simple. The following picture gives you a general idea of how the process works.</p> <p><img src="/img/howto-wod-6.png" alt="Workshops-on-Demand Concepts 1" title="Workshops-on-Demand Concepts 10000 feet view"></p> <p>Now that you understand the basic principle, let's look at the details. The image below shows what happens at each stage from a protocol standpoint.</p> <p><img src="/img/howto-wod-10.png" alt="" title="Workshops-on-Demand Architecture Diagram"></p> <h3>The Register Phase</h3> <p><strong>1.</strong> Participants start by browsing a frontend web server that presents the catalog of available workshops. They then select one and register for it by entering their email address, first and last names, as well as their company name. Finally, they accept the terms and conditions and hit the register button.</p> <p>As the register button is clicked, the frontend server performs a series of actions.</p> <p><strong>2.</strong> Assigns a student (i.e student401) from the dedicated workshop range to the participant. Every workshop has a dedicated range of students assigned to it.</p> <p>Here is a screenshot of the workshop table present in the frontend database server showing API 101 workshops details.</p> <p><img src="/img/howto-wod-2.png" alt="Workshops Table from Frontend DB server" title="Workshops Table from Frontend DB server"></p> <ul> <li> <p>Frederic Passeron gets assigned a studentid "student397" for workshop "API101".</p> <p>Here are the details of the participant info when registered to a given workshop.</p> </li> </ul> <p><img src="/img/howto-wod-3.png" alt="Customers Table from Frontend DB server" title="Customers Table from Frontend DB server"></p> <p><strong>3.</strong> An initial email is sent to participants from the frontend server welcoming them to the workshop and informing them that the deployment is ongoing and that a second email will arrive shortly providing the necessary information required to log onto the workshop environment.</p> <p><strong>4.</strong> At the same time, the frontend server sends the necessary orders through a procmail API call to the backend server. The mail sent to the backend server contains the following details:</p> <ul> <li>Action Type ( CREATE, CLEANUP, RESET)</li> <li>Workshop ID</li> <li>Student ID</li> </ul> <p><strong>5.</strong> The backend server recieves the order and processes it by parsing the email recieved using the procmail API. the procmail API automates the management of the workshops.</p> <p>Like any API, it uses verbs to perform tasks.</p> <ul> <li>CREATE to deploy a workshop</li> <li>CLEANUP to delete a workshop</li> <li>RESET to reset associated workshop's resource</li> </ul> <p><strong>CREATE subtasks:</strong></p> <ul> <li><strong>6.</strong> It prepares any infrastructure that might be required for the workshop (Virtual Appliance, Virtual Machine, Docker Container, LDAP config, etc.).</li> <li>It generates a random Password for the allocated student.</li> <li>It deploys the workshop content on the jupyterhub server in the dedicated student home directory (Notebooks files necessary for the workshops).</li> <li><strong>7.</strong> It sends back the confirmation of the deployment of the workshop, along with the student's required details (i.e password), through API Calls to the frontend server.</li> </ul> <p><strong>8.</strong> The frontend server tables will be updated in the following manner:</p> <ul> <li>The customers table shows an active status for the participant row. The password field has been updated.</li> <li>The workshop table also gets updated. The capacity field decrements the number of available seats.</li> <li>The student tables gets updated as well by setting the allocated student to active</li> </ul> <p><strong>9.</strong> The frontend server sends the second email to each participant providing them with the details to connect to the workshop environment.</p> <h3>The Run Phase</h3> <p><strong>10.</strong> From the email, the particpant clicks on the start button. it will open up a browser to the JupyterHub server and directly open the readme first notebook, presenting the workshop's flow.</p> <p>Participants will go through the different steps and labs of the workshop connecting to the necessary endpoints and leveraging the different kernels available on the JupyterHub server.</p> <p>Meanwhile, the frontend server will perform regular checks on how much time has passed. Depending on the time allocation (from 2 to 4 hours) associated with the workshop, the frontend server will send a reminder email usually a hour before the end of the time allocated. The time count actually starts when participants hit the register for the workshop button. It is mentioned in the terms and conditions.</p> <p>Finally, when the time is up, the frontend server sends a new order to the backend to perform either CLEANUP or RESET action for the dedicated studentid.</p> <p><strong>RESET subtasks:</strong></p> <ul> <li>It resets any infrastructure that was required for the workshop (Virtual Appliance, Virtual Machine, Docker Container, LDAP config, etc..).</li> <li>It generates a random password for the allocated student.</li> <li>It deletes the workshop content on the JupyterHub server in the dedicated student home directory (Notebooks files necessary for the workshop).</li> <li>It sends back the confirmation of the CLEANUP or RESET of the workshop along with the student details (i.e password) through API Calls to the frontend server.</li> </ul> <p>The frontend server tables will be updated in the following manner:</p> <ul> <li>The customers table will show an inactive status for the participant row. The password field has been updated.</li> <li>The Workshop table gets also updated. The capacity field increment the number of available seats.</li> <li>The student tables gets updated as well by setting the allocated student to inactive.</li> <li>The frontend sends the final email to the participant.</li> </ul> <h3>The React and Reward Phase:</h3> <ul> <li> <p>A final email thanks the students for their participation. It provides a link to an external survey and encourages the participants to share their achievement <a href="https://developer.hpe.com/blog/become-a-legend/">badge</a> on social media like linkedin or twitter.</p> <p>Et voila! <a href="https://developer.hpe.com/blog/become-a-legend/"></a></p> </li> </ul> <p>With this very first article, I wanted to set the stage for the following articles where I will explain how to setup your own Workshops-on-Demand infrastructure. We will start by looking at the "JupyterHub" side of things. I will detail how to set it up depending on your use case (Public only vs Public and Private). Then, I will move to the workshop development part; from the notebook development to the automation that needs to come along with it in order to be properly integraged into the overall solution. Finally, the last article will cover the frontend's side. It will show you how to deploy it and more. I may also cover the appliance management aspect in another dedicated article.</p> <p>Please be sure to drop back at <a href="https://developer.hpe.com/blog">HPE DEV</a> for a follow up on this. Check out also the Hack Shack for new <a href="https://developer.hpe.com/hackshack/workshops">workshops</a>! We are currently working on different subjects like Data Visualization 101 or HPE GreenLake for Compute Operations Management API 101. Stay tuned!</p>https://developer.hpe.com/living-the-open-source-culture-at-hpe/https://developer.hpe.com/living-the-open-source-culture-at-hpe/Fri, 20 Jan 2023 20:38:44 GMT<p>Here at Hewlett Packard Enterprise (HPE), we like to say that “open collaboration is in HPE’s DNA”. We believe <a href="https://www.hpe.com/us/en/open-source.html">open source technologies and communities</a> can help deliver innovative solutions securely and at scale.  Certainly, open source has a lot to do with the technologies we deliver. But, importantly, it also has a lot to do with how we conduct business on a daily basis, like connecting across teams and business units to collaborate, play, and learn together. </p> <p>HPE has an Open Source Program Office where we share projects and ideas during our weekly open source office hours – discussions in these sessions have even inspired some inner-sourcing projects.  We have an “Open for Challenge” team that competes with and against other company teams in our wellness challenges (right now, we’re tracking our steps to put open source fans at HPE on our company-wide leader board). But, my favorite collaboration activity is our Open Source &#x26; Tech Bookclub. Any HPE employee can join our internal Slack channel dedicated to discussing and exploring everything from legal tomes to cyberpunk fiction. </p> <p>2022 was our inaugural year where we covered the following selections:</p> <ul> <li><em>The Cathedral &#x26; the Bazaar:</em> Musings on Linux and Open Source by an Accidental Revolutionary by Eric S. Raymond (1999). I re-read this seminal treatise for the bookclub and was struck by just how much of ESR’s insights still hold true 20+ years later.</li> <li><em>The Phoenix Project</em> by Gene Kim (2018) is so engaging and fast-paced that it is practically a DevOps beach read.</li> <li><em>The Business and Economics of Linux and Open Source</em> by Martin Fink (2003). Martin, originally part of HPE’s parent company HP, actually dropped by for our discussion which added a unique perspective to the topics covered in this book.</li> <li><em>The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography</em> by Simon Singh (2000) is one of my favorite books on cryptography – the author uses fantastically accessible analogies, includes a treasure hunt and makes quantum cryptography as understandable as possible without an advanced degree.</li> <li><em>The Making and Maintenance of Open Source</em> by Nadia Eghbal (2020) tackles one of the more recent issues in the sustainability of open source projects. This topic was meaty enough that we held a live happy hour for this discussion!</li> </ul> <p>Once the bookclub had a baseline established, we held an active channel-wide vote on which books to expand our club into for the second year.  Here are our winners for 2023:</p> <ul> <li><em>Life 3.0: Being Human in the Age of Artificial Intelligence</em> by Max Tegmark - (2017) – I’ve only read the first chapter so far but it poses lots of provocative questions about the future.</li> <li><em>Open Source Law, Policy and Practice - Second Edition</em>, Edited by Amanda Brock (2022) (this one is available for free download from <a href="https://global.oup.com/academic/product/open-source-law-policy-and-practice-9780198862345?cc=us&#x26;lang=en&#x26;">Oxford University Press</a>). This one is jam-packed with articles by some of the leading minds in the open source world.</li> <li><em>The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race</em> by Walter Isaacson – I haven’t started this one yet but I think it will be fascinating.</li> <li><em>Snow Crash</em> by Neal Stephenson – the most amazing thing about this cyberpunk novel is that it was envisioned in 1992.</li> </ul> <p>I’m looking forward to the discussions on all these books both in our virtual channel as well as our scheduled live discussions. I’m also looking forward to our virtual globe-trotting in the wellness step challenge. If you would like to work and play with us, consider checking out our <a href="https://careers.hpe.com/us/en">job board</a> or contributing to one of our <a href="https://www.hpe.com/us/en/open-source.html">featured open source projects</a>!</p>https://developer.hpe.com/intro-to-core-api-part-1-integer-incrementing/https://developer.hpe.com/intro-to-core-api-part-1-integer-incrementing/Thu, 19 Jan 2023 14:37:21 GMT<p>External Blog</p>https://developer.hpe.com/developing-an-open-source-metadata-framework-to-enhance-the-ml-pipeline-experience/https://developer.hpe.com/developing-an-open-source-metadata-framework-to-enhance-the-ml-pipeline-experience/Sun, 15 Jan 2023 18:21:10 GMT<p>In this blog series, you’ll meet some of the open source experts working with the HPE Developer Community team. In this post, I’ll be interviewing Annmary Justine K who works in the HPE AI Research Lab on foundations and techniques for data centric trustworthy AI. As an AI Expert, she has been looking into areas like data provenance, data versioning, metadata tracking and lineages. Prior to her work in HPE’s AI research lab, she worked in HPE Storage, focused on various products ranging from file, block, and object level storage. She has published multiple research papers and patent disclosures in her 15 years of industry experience.</p> <p>Annmary began her open source journey by contributing to the Hierarchical Data Format (HDF) standard, a set of file formats designed to store and organize large amounts of data, especially developed for supercomputing applications.  Originally developed at the U.S. National Center for Supercomputing Applications, it is now supported by the non-profit HDF Group, whose mission is to ensure continued development in this area. Annmary’s work was originally focused on HDF5 and then she went on to create Common Metadata Framework.</p> <h3>How did you first get involved with contributing to open source?</h3> <p>My journey with open source started with the Hierarchical Data Format project. Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data and is used extensively in scientific data applications. I started by writing a versioning connector for HDF5 that creates separate sub files for each dataset created in HDF5 and mounts these sub-files as external links in the main file. It enables the versioning of HDF5 files at a dataset boundary.  This work exposed me to the need for integrated versioning and metadata management for data and AI pipelines and the discovery of this gap led to the creation of the Common Metadata Framework (CMF).</p> <h3>Tell me a little more about this project.</h3> <p>Common Metadata Framework is a metadata tracking library for machine learning (ML) pipelines. Its power lies in its ability to track distributed lineages for pipelines and to track lineages from different experiment variants. It provides abstractions to support datacentric AI /ML development and allows you to gain better insights into the versions of data and artifacts. The information can be exported, i.e. via Openlineage. As such, it is not locked into the platform and can be shared from this platform to others for metadata sharing and collaborative development. It provides a framework to exchange metadata among different team members and enables a Git-like model for the sharing of metadata, making it unique from the other experiment management frameworks in the market. It enables users to push their local metadata to the remote repository, where it is merged to create the global metadata and pulls metadata from the global metadata to the local, to create a local view, which would contain only the metadata of interest.</p> <p>The information collected by CMF can provide interesting insights about your AI pipelines. For example, it can help identify what could be the sphere of influence of a corrupted dataset. What other downstream artifacts or models were affected by a tainted dataset? How did your model perform for a particular subset of data versus another subset? What other experiments were executed with the dataset and which of these provided the best results? What different intermediate artifacts were created while developing a particular model? These are only some of the questions the information gathered can answer.</p> <p>These insights not only help in terms of traceability and reproducibility of pipelines. They can also be used to optimize the performance of the pipelines. Pipelines can be optimized for their execution time, execution metrics, carbon footprint etc. The insights can also be used to provide other valuable additions, like providing high-quality representative data from huge volumes of unlabeled datasets.</p> <h3>How did you get involved with this project?</h3> <p>The HPE AI research lab started CMF from the ground up when the group realized that there was a gap in the ecosystem for a platform that enables the exchange of AI/ML metadata. Reproducibility in AI/ML pipelines are possible only when one knows the source code version, the data version and hyper parameters being used. Tracking this metadata enables the creation of trustworthy models. Although there are many metadata tracking tools in the field, we realized that there is no single framework that enables the integrated tracking of all of these while also enabling the ability to freely share this metadata. That is when we came up with CMF.  We have built CMF on top of existing open source tools (e.g. DVC and ML Metadata) so that we didn’t end up reinventing the wheel and instead provided added value.</p> <h3>What excites you about your role here?</h3> <p>Being the creator of a new open source project was challenging. Working in open source enables you to don multiple hats at the same time. The role also allows you to stretch beyond your technical skills. It provides opportunities to present your work, collaborate with teams outside of the company, and work towards creating value not just for one company but society at large.  </p> <h3>What things are you working on right now in regards to this project?</h3> <p>We have just scratched the surface with Common Metadata Framework. The metadata collected from AI pipelines enables further optimization and we are building intelligence for that.  We are working on multiple use cases found in the science community. One specific use case is the High Energy Physics Particle Tracking Pipeline. Deep learning-based pattern recognition methods are now regularly used for High energy Physics (HEP) particle tracking. The leading techniques involve successive filtering of detector impact cloud points from sensors in collider experiments to isolate possible trajectories of exotic particles. This involves metric learning using fully-connected multi-layer perceptron’s, binary classifiers for edge filtering and a graph neural network (GNN) to improve purity of selected points. This work faces numerous challenges, including large datasets and a large parameter space with cascaded inputs/outputs making optimization difficult. Distributed algorithm development and multiple compute environments (HPC, on-prem &#x26; cloud) add to the complexity. CMF has been applied in scenarios like this to capture metadata for the entire experiment population by capturing data lineage, metrics, network architecture and hyperparameters. We are working towards optimizing this pipeline to provide a faster turnaround time for experiments and improve model performance.</p> <h3>Do you have any advice you’d like to pass on?</h3> <p>When working with open source it is important to realize that the road ahead is long and may not provide instant results. Adoption may come only after consistent effort and building a lot of tools around it. You need to continue to persevere and work towards your goals incrementally. </p> <p>Open source also offers the opportunity to collaborate and work together with a broader community. It is important to work in tandem with other tools in the ecosystem so that, together, you create a more enhanced ecosystem.</p>https://developer.hpe.com/greenlake-platform-infrastructure-monitoring-with-apache-skywalking/https://developer.hpe.com/greenlake-platform-infrastructure-monitoring-with-apache-skywalking/Fri, 13 Jan 2023 08:52:15 GMT<h1><strong>Overview</strong></h1> <p>Modern applications are predominantly distributed microservices-based applications that are flexible for scalability and developed using varying technology stacks. These applications are deployed on a highly available infrastructure composed of a multitude of compute instances (bare metal/virtual machine/container). It is imperative to monitor, in near real time, how the compute resources are utilized, so that necessary proactive actions are initiated to keep the applications in a consistently acceptable state. Application Performance Management (APM) tools help to centrally monitor distributed applications and/or compute resources.</p> <p>Apache SkyWalking is a popular, open-source application performance monitoring (APM) tool used to collect, analyze, aggregate, and visualize data from services and cloud-native infrastructures. It provides an easy way to maintain a clear view of your distributed systems, even across clouds. You can use it with HPE GreenLake for Private Cloud Enterprise, the managed, pay-per-use Private Cloud as a Service (PCaaS) solution to enhance monitoring and observability of your workloads.</p> <p>Because HPE GreenLake for Private Cloud Enterprise provides you with a flexible, self-service environment where you can implement a hybrid approach and run different kinds of bare metal, virtual machine, and containerized workloads, you can take advantage of additional apps and tools like SkyWalking to customize your situation and meet your specific needs.</p> <p>In this post, I will focus on how you can use SkyWalking and OpenTelemetry or OTel for short, to do infrastructure host monitoring while taking advantage of how HPE GreenLake provides you with a consistent cloud experience across all your applications and data, whether on- or off-premises.</p> <h1><strong>Let's get started</strong></h1> <h2>Definitions</h2> <ul> <li><strong>Monitoring</strong> is tooling or a technical solution that allows teams to watch and understand the state of their systems. Monitoring is based on gathering predefined sets of metrics or logs.</li> <li><strong>Observability</strong> is tooling or a technical solution that allows teams to actively debug their system. Observability is based on exploring properties and patterns not defined in advance.</li> </ul> <h2><strong>Pre-requisites</strong></h2> <ul> <li>An active service subscription to HPE GreenLake Private Cloud Enterprise.</li> <li>VM and/or Bare metal instances are provisioned for the workloads and you are able to log in to the console.</li> <li>SkyWalking instance has been setup either on premises or in Hyperscalers. Ensure that inbound ingress traffic is allowed on port 11800 for SkyWalking instance.</li> </ul> <h2><strong>Steps to configure monitoring</strong></h2> <h3>Step 1- Monitoring host metric with node exporter</h3> <p>Log in to the console of machine to be monitored.</p> <p>Next, download and run the node exporter using the snippet code below.</p> <pre><code class="language-shell">wget https://github.com/prometheus/node_exporter/releases/download/v1.4.0-rc.0/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz tar xvfz node_exporter-1.4.0-rc.0.linux-amd64.tar.gz rm -rf node_exporter-1.4.0-rc.0.linux-amd64.tar.gz cd node_exporter-1.4.0-rc.0.linux-amd64 cp node_exporter /usr/sbin cat&#x3C;&#x3C;EOF >nodeexporter.service [Unit] Description=Service to start the node_exporter on the node [Service] Type=Simple ExecStart=/usr/sbin/node_exporter StandardOutput=syslog StandardError=syslog SyslogIdentifier=%n [Install] WantedBy=multi-user.target EOF mv nodeexporter.service /etc/systemd/system systemctl daemon-reload systemctl enable nodexporter.service systemctl start nodexporter.service </code></pre> <p>Check that the metrics endpoint shows metrics exported from the infrastructure node.</p> <pre><code class="language-shell">curl &#x3C;http://localhost:9100/metrics> </code></pre> <p><img src="/img/node_metrics.png" alt="" title="Node Metrics"></p> <h3>Step 2- Setup OpenTelemetry (OTel) collector</h3> <p>OTel provides a set of standardized vendor-agnostic SDKs, APIs, and tools for ingesting, transforming, and sending data to an observability back-end. You can install OTel collector in the same host, which exposes metrics, or we can create central VM/Bare metal instance to receive telemetry information from several infrastructure instances.</p> <p>Install OTel collector.</p> <pre><code class="language-shell">sudo wget &#x3C;https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.54.0/otelcol_0.54.0_linux_amd64.deb> sudo dpkg -i otelcol_0.54.0_linux_amd64.deb systemctl status otelcol # By default, the otelcol systemd service will be started with the --config=/etc/otelcol/config.yaml option after installation. </code></pre> <p>Create a new OTel configuration file (say /etc/otelcol/otel-collector-config.yaml) to link the metrics from host nodes to SkyWalking Observability Analysis Platform (OAP). The orange boxes shown in the images below must be modified to point to the correct infrastructure hosts and SkyWalking DNS or IP address.</p> <p><img src="/img/otel_collector_configuration.png" alt="" title="OTeL Configuration"></p> <p>Change the OTel configuration to the newly created OTel configuration file.</p> <pre><code class="language-shell">cd /etc/otelcol sudo nano otelcol.conf # Change the path to the new configuration file you just created in the # above step and save. sudo systemctl restart otelcol sudo journalctl -u otelcol </code></pre> <h3>Step 3- Validate the infrastructure host metrics are seen in the SkyWalking UI</h3> <p><img src="/img/skywalking_vm_monitoring.png" alt="" title="SkyWalking UI with Infrastructure details"></p> <h1><strong>Conclusion</strong></h1> <p>Infrastructure and application level monitoring gives a holistic picture of service reliability. In this post, I have shown you how easily you can set up compute infrastructure resource monitoring, running in on- or off-premises, using OpenTelemetry and SkyWalking. CPU, Memory, and Disk usage trends showin in the APM tool give the monitoring team early insights into potential system issues that should be addressed before they impact customer applications. Additionally, APM tools often provide trace, logs, metrics monitoring along with alerting, which can be used in conjunction with infrastructure monitoring tools to provide highly reliable services for customers.</p> <h1><strong>Reference</strong></h1> <p><a href="https://skywalking.apache.org/docs/main/v9.3.0/en/concepts-and-designs/overview/">Apache SkyWalking</a></p> <p><a href="https://cloud.google.com/architecture/devops/devops-measurement-monitoring-and-observability">Monitoring and observability</a></p> <p><a href="https://prometheus.io/docs/guides/node-exporter/">Prometheus Node exporter</a></p> <p><a href="https://opentelemetry.io/">OpenTelemetry</a></p>https://developer.hpe.com/set-up-apache-skywalking-for-k8s-monitoring-in-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/set-up-apache-skywalking-for-k8s-monitoring-in-hpe-greenlake-for-private-cloud-enterprise/Wed, 11 Jan 2023 20:19:50 GMT<h2>Why is Kubernetes monitoring so important?</h2> <p><a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a> delivers a modern private cloud to support your app workloads running in any combination across your edges, colocations, and data centers. It contains one HPE service, called <a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Private Cloud Enterprise: Containers</a>, which provides an enterprise-grade container management service using open source Kubernetes. It allows customers to create a Kubernetes cluster, view details about existing clusters, and launch the service console.</p> <p>Though Kubernetes dramatically simplifies application deployment in containers and across clouds, it adds a new set of complexities for managing, securing and troubleshooting applications. Container-based applications are dynamic and they are being designed using microservices, where the number of components is increased by an order of magnitude.</p> <p>To ensure Kubernetes security, it requires self-configuration that is typically specified in code, whether Kubernetes YAML manifests, Helm charts, or templating tools. Properly configuring for workloads, clusters, networks, and infrastructure is crucial for averting issues and limiting the impact if a breach occurs. Dynamic provisioning via Infrastructure as code, automated configuration management and orchestration also add to monitoring and troubleshooting complexity.</p> <p>Since Kubernetes is widely used for processing customer workloads, the non-availability of both workloads and the cluster itself, from misconfiguration of core components to network connectivity issues in Kubernetes, can adversely impact productivity, business continuity and user experience. To avoid this, enterprises must closely monitor the status of the objects managed and operations performed by Kubernetes, proactively capture abnormalities, and resolve them well before end-users notice.</p> <p>Kubernetes monitoring is critical to managing application performance, service uptime and troubleshooting. However, it presents a challenge for a traditional, static monitoring approach, emphasizing the need for real time monitoring. Having a good application performance monitoring (APM) tool is becoming essential for Kubernetes monitoring.</p> <p>In <a href="https://developer.hpe.com/blog/get-started-with-application-performance-monitoring-tools-overview/">my first blog post</a>, I walked through some of the best APM tools, described their key features and discussed their strengths and weaknesses in detail. In this blog post, I choose one APM tool, <em>Apache SkyWalking</em>, and describe in detail how to set it up in HPE GreenLake for Private Cloud Enterprise for monitoring a Kubernetes cluster.</p> <h2>Apache SkyWalking</h2> <p><a href="https://skywalking.apache.org/">Apache SkyWalking</a> is an open source application performance monitoring (APM) tool, especially designed for microservices, cloud native, and container-based architectures.</p> <p>Apache SkyWalking is lightweight and scalable. It can be easily set up as a <em>self-managed</em> APM tool within an on-premises data center. This avoids leasing customer data to third party services and matches well with the strict security parameters of HPE GreenLake for Private Cloud environment.</p> <h2>Prerequisites</h2> <p>Before starting, make sure you have the following:</p> <ul> <li>A Kubernetes cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em>, together with the <em>HPE kubectl plugin</em> and the <em>kubeconfig</em> file of the Kubernetes cluster. You can download them from the launched service console in HPE GreenLake for Private Cloud Enterprise. To simplify the setup process, you can export the environment variable <code>KUBECONFIG</code> and point it to the downloaded <em>kubeconfig</em> file.</li> <li>The <em><a href="https://helm.sh/docs/intro/install/">Helm</a></em> CLI tool, version 3.8.1 or later</li> </ul> <p>With your user access setup, you should have access to permissions that can create and update the following resources in the Kubernetes cluster:</p> <ul> <li>Kubernetes Service Account(s)</li> <li>Kubernetes Roles &#x26; RoleBindings</li> </ul> <h2>Set up Apache SkyWalking for Kubernetes monitoring</h2> <p>Apache SkyWalking leverages the Kubernetes <a href="https://github.com/kubernetes/kube-state-metrics">kube-state-metrics</a> service for collecting metrics data from Kubernetes cluster. It then leverages the <em>OpenTelemetry</em> collector to transfer the Kubernetes metrics to the <em>OpenTelemetry</em> receiver in the Apache SkyWalking Observability Analysis Platform (OAP) for Kubernetes monitoring.</p> <p><img src="/img/otel-collector.png" alt=""></p> <h3>Deploy Apache SkyWalking</h3> <p>In this blog post, I will take the approach to setting up the Apache SkyWalking as a <em>self-managed</em> APM tool within the Kubernetes cluster created in HPE GreenLake for Private Cloud Enterprise. This mainly takes into account matching with the strict security parameters of HPE GreenLake for Private Cloud environment.</p> <p>To start, install Apache SkyWalking using Helm charts with <em>elasticsearch</em> as storage:</p> <pre><code class="language-markdown">$ git clone https://github.com/apache/skywalking-kubernetes $ cd skywalking-kubernetes/chart $ helm repo add elastic https://helm.elastic.co $ helm dep up skywalking $ kubectl create ns skywalking $ helm install skywalking skywalking –n skywalking \ --set oap.image.tag=9.2.0 \ --set oap.storageType=elasticsearch \ --set ui.image.tag=9.2.0 \ --set elasticsearch.imageTag=7.1.1 \7 --set elasticsearch.persistence.enabled=true \ --set elasticsearch.sysctlInitContainer.enabled=false \ --set oap.env.SW_OTEL_RECEIVER=default \ --set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="k8s-cluster\,k8s-service\,k8s-node" </code></pre> <p>After running the above commands, the Apache SkyWalking is installed to the Kubernetes cluster's namespace <em>skywalking</em>. It creates the <em>elasticsearch</em> as the <code>StatefulSet</code> resource, running a pod on each worker node. It runs the Apache SkyWalking OAP with replicas as 2 to provide high availability.</p> <p>The last two options, <em>oap.env.SW_OTEL_RECEIVER=default</em> &#x26; <em>oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="k8s-cluster,k8s-service,k8s-node"</em>, enable the <em>OpenTelemetry</em> receiver and define the metrics for the Kubernetes service, service instance and endpoint. It requires Apache SkyWalking OAP to have access to the Kubernetes API server to query the metadata.</p> <p>You can check the detailed Apache SkyWalking installation by typing the following <em>kubectl</em> command:</p> <pre><code class="language-markdown">$ kubectl get all -n skywalking NAME READY STATUS RESTARTS AGE pod/elasticsearch-master-0 1/1 Running 0 8m7s pod/elasticsearch-master-1 1/1 Running 0 8m7s pod/elasticsearch-master-2 1/1 Running 0 8m7s pod/skywalking-es-init-m9t5c 0/1 Completed 0 8m7s pod/skywalking-oap-7f757c7668-nq2cz 1/1 Running 0 8m8s pod/skywalking-oap-7f757c7668-q8z7m 1/1 Running 0 8m8s pod/skywalking-ui-549dc5989f-jq9b9 1/1 Running 0 8m8s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-master ClusterIP 10.110.35.173 &#x3C;none> 9200/TCP,9300/TCP 8m5s service/elasticsearch-master-headless ClusterIP None &#x3C;none> 9200/TCP,9300/TCP 8m5s service/skywalking-oap ClusterIP 10.108.29.84 &#x3C;none> 11800/TCP,12800/TCP 8m5s service/skywalking-ui ClusterIP 10.102.186.131 &#x3C;none> 80/TCP 8m5s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/skywalking-oap 2/2 2 2 8m6s deployment.apps/skywalking-ui 1/1 1 1 8m6s NAME DESIRED CURRENT READY AGE replicaset.apps/skywalking-oap-7f757c7668 2 2 2 8m9s replicaset.apps/skywalking-ui-549dc5989f 1 1 1 8m9s NAME READY AGE statefulset.apps/elasticsearch-master 3/3 8m5s NAME COMPLETIONS DURATION AGE job.batch/skywalking-es-init 1/1 7m27s 8m6s </code></pre> <p>You can edit the deployed SkyWalking UI service <em>skywalking-ui</em> and change its type from <em>ClusterIP</em> to <em>NodePort</em>. The service will be automatically mapped to the gateway host with an assigned port.</p> <pre><code class="language-markdown">$ k edit service/skywalking-ui -n skywalking $ k describe service/skywalking-ui -n skywalking Name: skywalking-ui Namespace: skywalking Labels: app=skywalking app.kubernetes.io/managed-by=Helm chart=skywalking-4.2.0 component=ui heritage=Helm hpecp.hpe.com/hpecp-internal-gateway=true release=skywalking Annotations: hpecp-internal-gateway/80: gl2-caas.gl-hpe.local:10037 meta.helm.sh/release-name: skywalking meta.helm.sh/release-namespace: skywalking Selector: app=skywalking,component=ui,release=skywalking Type: NodePort IP: 10.102.186.131 Port: &#x3C;unset> 80/TCP TargetPort: 8080/TCP NodePort: &#x3C;unset> 32748/TCP Endpoints: 10.192.7.25:8080 Session Affinity: None External Traffic Policy: Cluster Events: &#x3C;none> </code></pre> <p>As shown in the <em><strong>Annotations</strong></em> section of the service description above, the SkyWalking UI can then be accessed in your browser by typing the address <em>gl2-caas.gl-hpe.local:10037</em>:</p> <p><img src="/img/sw-ui-k8s.png" alt=""></p> <h3>Deploy kube-state-metrics service</h3> <p>The Kubernetes <em>kube-state-metrics</em> service will be deployed to listen to the Kubernetes API server and generate metrics about the state of the Kubernetes objects.</p> <pre><code class="language-markdown">$ helm install kube-state-metrics -n skywalking prometheus-community/kube-state-metrics </code></pre> <h3>Set up <em>OpenTelemetry</em> collector</h3> <p>The <em>OpenTelemetry</em> collector needs to be installed and set up to transfer the Kubernetes metrics to <em>OpenTelemetry</em> receiver from the SkyWalking OAP server. I use the standard Docker image <em>otel/opentelemetry-collector:0.50.0</em> to deploy the <em>OpenTelemetry</em> collector to the Kubernetes cluster.</p> <h4>Set up role-based access control (RBAC)</h4> <p>Kubernetes RBAC is a key security control to ensure that cluster users and workloads have access only to resources required to execute their roles. It is important to ensure that, when designing permissions for cluster users, the cluster administrator understands the areas where privilege escalation could occur, to reduce the risk of excessive access leading to security incidents.</p> <p>To set up RBAC, you create a <em>Service Account</em>, a <em>ClusterRole</em>, and connect the two with a <em>ClusterRoleBinding</em>.</p> <h5>1. Create a YAML file <em>otel-sa-kubernetes-monitor.yaml</em> for the service account:</h5> <pre><code class="language-markdown">apiVersion: v1 kind: ServiceAccount metadata: name: otel-sa-kubernetes-monitor </code></pre> <h5>2. Create a YAML file <em>otel-role-kubernetes-monitor.yaml</em> for the cluster roles:</h5> <pre><code class="language-markdown">apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-role-kubernetes-monitor rules: - apiGroups: [ "" ] resources: # @feature: kubernetes-monitor; permissions to read resources - "endpoints" - "pods" - "services" - "nodes" - "nodes/metrics" - "nodes/proxy" verbs: [ "get", "watch", "list" ] </code></pre> <h5>3. Create a YAML file <em>otel-role-binding-kubernetes-monitor.yaml</em> to bind the service account with the cluster roles:</h5> <pre><code class="language-markdown">apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-role-binding-kubernetes-monitor roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otel-role-kubernetes-monitor subjects: - kind: ServiceAccount name: otel-sa-kubernetes-monitor namespace: skywalking </code></pre> <h5>4. Deploy the service account, the cluster role and the cluster rolebinding:</h5> <pre><code class="language-markdown">$ kubectl apply -f otel-sa-kubernetes-monitor.yaml -n skywalking $ kubectl apply -f otel-role-kubernetes-monitor.yaml -n skywalking $ kubectl apply -f otel-role-binding-kubernetes-monitor.yaml.yaml -n skywalking </code></pre> <h4>Deploy <em>OpenTelemetry</em> collector</h4> <h5>1. Create a YAML file <em>otel-collector-config.yaml</em> to set the <em>OpenTelemetry</em> config to scrape the Kubernetes metrics:</h5> <pre><code class="language-markdown">apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-conf-kubernetes-monitor labels: app: otel-kubernetes-monitor data: otel-collector-config: | service: pipelines: metrics: receivers: [ prometheus ] exporters: [ opencensus,logging ] exporters: opencensus: endpoint: "skywalking-oap.skywalking.svc.cluster.local:11800" tls: insecure: true logging: loglevel: debug receivers: prometheus: config: scrape_configs: # @feature: kubernetes-monitor; configuration to scrape Kubernetes Nodes metrics - job_name: 'kubernetes-cadvisor' scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [] target_label: cluster replacement: cfe-iac-clu - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor - source_labels: [instance] separator: ; regex: (.+) target_label: node replacement: $$1 action: replace # @feature: kubernetes-monitor; configuration to scrape Kubernetes Endpoints metrics - job_name: kube-state-metrics metrics_path: /metrics kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] regex: kube-state-metrics replacement: $$1 action: keep - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [] target_label: cluster replacement: cfe-iac-clu </code></pre> <h5>2. Create a YAML file <em>otel-collector-deploy.yaml</em> for the <em>OpenTelemetry</em> collector deployment:</h5> <pre><code class="language-markdown">apiVersion: apps/v1 kind: Deployment metadata: name: otel-deployment-kubernetes-monitor labels: app: otel-kubernetes-monitor spec: replicas: 1 selector: matchLabels: app: otel-kubernetes-monitor template: metadata: labels: app: otel-kubernetes-monitor annotations: sidecar.istio.io/inject: "false" spec: serviceAccountName: otel-sa-kubernetes-monitor containers: - name: otel-kubernetes-monitor image: otel/opentelemetry-collector:0.50.0 command: - "/otelcol" - "--config=/conf/otel-collector-config.yaml" volumeMounts: - name: otel-collector-config-vol-kubernetes-monitor mountPath: /conf volumes: - name: otel-collector-config-vol-kubernetes-monitor configMap: name: otel-collector-conf-kubernetes-monitor items: - key: otel-collector-config path: otel-collector-config.yaml </code></pre> <h5>3. Deploy the OpenTelemetry collector:</h5> <pre><code class="language-markdown">$ kubectl apply -f otel-collector-config.yaml -n skywalking $ kubectl apply -f otel-collector-deploy.yaml -n skywalking </code></pre> <pre><code class="language-markdown">$ kubectl get all -n skywalking -l app=otel-kubernetes-monitor NAME READY STATUS RESTARTS AGE pod/otel-deployment-kubernetes-monitor-798cdd8486-gz885 1/1 Running 0 93d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/otel-deployment-kubernetes-monitor 1/1 1 1 96d NAME DESIRED CURRENT READY AGE replicaset.apps/otel-deployment-kubernetes-monitor-798cdd8486 1 1 1 96d </code></pre> <p>After all setup steps are finished, the Kubernetes metrics will show up in the SkyWalking UI, under the <em>Kubernetes</em> tab:</p> <p><img src="/img/sw-k8s-clu.png" alt=""></p> <p>You can check the Kubernetes overview from the SkyWalking UI:</p> <p><img src="/img/sw-k8s-overview.png" alt=""></p> <p>And the Kubernetes nodes:</p> <p><img src="/img/sw-k8s-node.png" alt=""></p> <p>The Kubernetes worker node:</p> <p><img src="/img/sw-k8s-node-instance.png" alt=""></p> <p>And the Kubernetes services:</p> <p><img src="/img/sw-k8s-svc.png" alt=""></p> <h2>Conclusion</h2> <p>In this blog post, I discussed the challenges in Kubernetes monitoring and why it’s important for Kubernetes monitoring in HPE GreenLake for Private Cloud Enterprise. I then took the Apache SkyWalking as the application performance monitoring (APM) tool and showed the detailed process of setting it up, as a <em>self-managed</em> environment in HPE GreenLake for Private Cloud Enterprise for monitoring a Kubernetes cluster. It provides a way to gain the visibility of the objects and operations performed by Kubernetes, and to resolve issues in the cluster.</p>https://developer.hpe.com/set-up-apache-skywalking-for-k8s-and-vm-monitoring-in-hpe-greenlake-private-cloud/https://developer.hpe.com/set-up-apache-skywalking-for-k8s-and-vm-monitoring-in-hpe-greenlake-private-cloud/Tue, 10 Jan 2023 07:31:55 GMT<h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a> delivers a modern private cloud to support your app workloads with bare metal, containers, and virtual machines (VMs) running in any combination across your edges, colocations, and data centers. It combines self-service resource access for developers with consumption and performance transparency for IT. Within this modern application environment, having a robust application performance monitoring (APM) tool is becoming essential. It can help IT professionals to ensure that deployed applications meet the performance, reliability and valuable user experience required by developers, partners and customers.</p> <p>In <a href="https://developer.hpe.com/blog/get-started-with-application-performance-monitoring-tools-overview/">my first blog post</a>, I walked through some of the best APM tools, describing their key features, strengths, and weaknesses. In this blog post, using one of the APM tools I covered in my previous post, I will describe, in detail, the process of how to set it up in HPE GreenLake for Private Cloud Enterprise for monitoring and alerting customer applications.</p> <h2>Apache SkyWalking</h2> <p><a href="https://skywalking.apache.org/">Apache SkyWalking</a> is an open source APM tool with capabilities for monitoring, tracing and diagnosing distributed systems. It’s especially designed for microservices, cloud native and container-based architectures.</p> <p>Apache SkyWalking provides a list of agents to be used for building <em>Java</em>, <em>.NET Core</em>, <em>PHP</em>, <em>NodeJS</em>, <em>Golang</em>, <em>LUA</em>, <em>Rust</em> and <em>C++</em> apps. It provides tracing, metrics analysis, alerting, service mesh observability and visualization.</p> <p>Apache SkyWalking is lightweight and scalable. It can be easily set up as a <em>self-managed</em> APM tool within an on-premises data center. This avoids leasing customer data to third party services and matches well with the strict security parameters of HPE GreenLake for Private Cloud environment.</p> <h2>Set up Apache SkyWalking for application monitoring</h2> <p>I will take you through setting up Apache SkyWalking as a <em>self-managed</em> APM tool within the Kubernetes cluster created in HPE GreenLake for Private Cloud Enterprise. By setting up the APM tool within this environment, it can benefit from the security features of HPE GreenLake.</p> <h3>Prerequisites</h3> <p>Before starting, make sure you have the following requirements:</p> <ul> <li>A Kubernetes cluster, being provisioned in HPE GreenLake for Private Cloud Enterprise</li> <li>The <em>kubectl</em> CLI tool, version 1.23 or later, together with the <em>kubeconfig</em> files for accessing the Kubernetes cluster</li> <li>The <em><a href="https://helm.sh/docs/intro/install/">Helm</a></em> CLI tool, version 3.8.1 or later</li> </ul> <h3>Deploy Apache SkyWalking</h3> <p>Install Apache SkyWalking using Helm charts with <em>elasticsearch</em> as storage:</p> <pre><code class="language-markdown">$ git clone https://github.com/apache/skywalking-kubernetes $ cd skywalking-kubernetes/chart $ helm repo add elastic https://helm.elastic.co $ helm dep up skywalking $ kubectl create ns skywalking $ helm install skywalking skywalking –n skywalking \ --set oap.image.tag=9.2.0 \ --set oap.storageType=elasticsearch \ --set ui.image.tag=9.2.0 \ --set elasticsearch.imageTag=7.17.1 \ --set elasticsearch.persistence.enabled=true \ --set elasticsearch.sysctlInitContainer.enabled=false </code></pre> <p>After running the above commands, Apache SkyWalking is installed on the Kubernetes cluster's namespace <em>skywalking</em>. The option <em>elasticsearch.persistence.enabled=true</em> in the above Helm install command creates the <em>elasticsearch</em> as the <code>StatefulSet</code> object, running a pod on each worker node. The command runs the Apache SkyWalking Observability Analysis Platform (OAP) with replicas as 2 to provide high availability.</p> <p>It should be noted that the last option <em>elasticsearch.sysctlInitContainer.enabled=false</em> in the above Helm install command is necessary. Otherwise, the command will try to set up <em>vm.max_map_count</em> using a privileged container during <em>elasticsearch</em> installation. Running privileged containers leaves a large chance that an attacker will be able to run code as root. There is one <em>PodSecurityPolicy</em>, <strong><code>psp-privileged-container</code></strong>, which has been pre-deployed in K8s clusters to deny privileged container running in the HPE GreenLake for Private Cloud Enterprise environment. This policy will fail the Helm install.</p> <p>You can check the detailed Apache SkyWalking installation by typing the following <em>kubectl</em> command:</p> <pre><code class="language-markdown">$ kubectl get all -n skywalking NAME READY STATUS RESTARTS AGE pod/elasticsearch-master-0 1/1 Running 0 8m7s pod/elasticsearch-master-1 1/1 Running 0 8m7s pod/elasticsearch-master-2 1/1 Running 0 8m7s pod/skywalking-es-init-m9t5c 0/1 Completed 0 8m7s pod/skywalking-oap-7f757c7668-nq2cz 1/1 Running 0 8m8s pod/skywalking-oap-7f757c7668-q8z7m 1/1 Running 0 8m8s pod/skywalking-ui-549dc5989f-jq9b9 1/1 Running 0 8m8s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-master ClusterIP 10.110.35.173 &#x3C;none> 9200/TCP,9300/TCP 8m5s service/elasticsearch-master-headless ClusterIP None &#x3C;none> 9200/TCP,9300/TCP 8m5s service/skywalking-oap ClusterIP 10.108.29.84 &#x3C;none> 11800/TCP,12800/TCP 8m5s service/skywalking-ui ClusterIP 10.102.186.131 &#x3C;none> 80/TCP 8m5s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/skywalking-oap 2/2 2 2 8m6s deployment.apps/skywalking-ui 1/1 1 1 8m6s NAME DESIRED CURRENT READY AGE replicaset.apps/skywalking-oap-7f757c7668 2 2 2 8m9s replicaset.apps/skywalking-ui-549dc5989f 1 1 1 8m9s NAME READY AGE statefulset.apps/elasticsearch-master 3/3 8m5s NAME COMPLETIONS DURATION AGE job.batch/skywalking-es-init 1/1 7m27s 8m6s </code></pre> <p>You can edit the deployed SkyWalking UI service <em>skywalking_ui</em> and change its type from <em>ClusterIP</em> to <em>NodePort</em>. The service will be automatically mapped to a gateway host with an assigned port.</p> <pre><code class="language-markdown">$ k edit service/skywalking-ui -n skywalking $ k describe service/skywalking-ui -n skywalking Name: skywalking-ui Namespace: skywalking Labels: app=skywalking app.kubernetes.io/managed-by=Helm chart=skywalking-4.2.0 component=ui heritage=Helm hpecp.hpe.com/hpecp-internal-gateway=true release=skywalking Annotations: hpecp-internal-gateway/80: gl2-caas.gl-hpe.local:10037 meta.helm.sh/release-name: skywalking meta.helm.sh/release-namespace: skywalking Selector: app=skywalking,component=ui,release=skywalking Type: NodePort IP: 10.102.186.131 Port: &#x3C;unset> 80/TCP TargetPort: 8080/TCP NodePort: &#x3C;unset> 32748/TCP Endpoints: 10.192.7.25:8080 Session Affinity: None External Traffic Policy: Cluster Events: &#x3C;none> </code></pre> <p>As shown in the <em><strong>Annotations</strong></em> section of the service description above, the SkyWalking UI can then be accessed in the browser by typing the address <em>gl2-caas.gl-hpe.local:10037</em>:</p> <p><img src="/img/sw-ui.png" alt=""></p> <h3>Deploy a sample application: <em>SpringBoot</em></h3> <p>As my first demo application, I will create a <em>SpingBoot</em> Web app that provides a REST endpoint <strong>/message</strong> to print some nice message. Then, I will generate a <em>jar</em> package using the <a href="https://maven.apache.org/what-is-maven.html">Apache Maven</a> command <em>mvn</em> with the <em>pom.xml</em> file:</p> <pre><code class="language-markdown">├── Dockerfile ├── Dockerfile.agentless ├── pom.xml ├── README.md ├── src │ ├── main │ └── test └── target ├── springboot-k8s-demo.jar └── test-classes $ cat pom.xml &#x3C;?xml version="1.0" encoding="UTF-8"?> &#x3C;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> &#x3C;modelVersion>4.0.0&#x3C;/modelVersion> &#x3C;parent> &#x3C;groupId>org.springframework.boot&#x3C;/groupId> &#x3C;artifactId>spring-boot-starter-parent&#x3C;/artifactId> &#x3C;version>2.6.1&#x3C;/version> &#x3C;relativePath/> &#x3C;!-- lookup parent from repository --> &#x3C;/parent> &#x3C;groupId>com.javatechie&#x3C;/groupId> &#x3C;artifactId>springboot-k8s-demo&#x3C;/artifactId> &#x3C;version>0.0.1-SNAPSHOT&#x3C;/version> &#x3C;name>springboot-k8s-demo&#x3C;/name> &#x3C;description>Demo project for Spring Boot&#x3C;/description> &#x3C;properties> &#x3C;java.version>1.8&#x3C;/java.version> &#x3C;/properties> &#x3C;dependencies> &#x3C;dependency> &#x3C;groupId>org.springframework.boot&#x3C;/groupId> &#x3C;artifactId>spring-boot-starter-web&#x3C;/artifactId> &#x3C;/dependency> &#x3C;dependency> &#x3C;groupId>org.springframework.boot&#x3C;/groupId> &#x3C;artifactId>spring-boot-starter-test&#x3C;/artifactId> &#x3C;scope>test&#x3C;/scope> &#x3C;/dependency> &#x3C;/dependencies> &#x3C;build> &#x3C;plugins> &#x3C;plugin> &#x3C;groupId>org.springframework.boot&#x3C;/groupId> &#x3C;artifactId>spring-boot-maven-plugin&#x3C;/artifactId> &#x3C;/plugin> &#x3C;/plugins> &#x3C;finalName>springboot-k8s-demo&#x3C;/finalName> &#x3C;/build> &#x3C;/project> $ mvn compile $ mvn package $ ls target/springboot-k8s-demo.jar target/springboot-k8s-demo.jar </code></pre> <p>You can see the <em>jar</em> package <em>springboot-k8s-demo.jar</em> is created in the <em>target</em> folder. By building a <em>Docker</em> image using this generated <em>jar</em> file, the <em>SpringBoot</em> web app can be easily deployed as a containerized application in the Kubernetes cluster.</p> <p>Apache SkyWalking provides a list of agents for instrumenting applications. A specific agent per programming language can be used to build a corresponding service that collects application data and exports them to the SkyWalking OAP server.</p> <p><img src="/img/sw-agents.png" alt=""></p> <p>In order to monitor the sample <em>SpringBoot</em> web app from Apache SkyWalking, download the Java agent and rebuild the image.</p> <p><img src="/img/java-agent.png" alt=""></p> <p>Here is the <em>Dockerfile</em> for building the image with the Java agent:</p> <pre><code class="language-markdown">$ cat Dockerfile FROM adoptopenjdk:11-jre-hotspot # copy extracted java agent folder from the downloaded apache skywalking archive ADD agent /opt/agent # copy the app jar file EXPOSE 8080 ADD target/springboot-k8s-demo.jar /app/springboot-k8s-demo.jar WORKDIR /app ENTRYPOINT ["java","-javaagent:/opt/agent/skywalking-agent.jar=agent.namespace=default,agent.service_name=springboot-k8s-app,collector.backend_service=skywalking-oap.skywalking.svc.cluster.local:11800, plugin.jdbc.trace_sql_parameters = true,profile.active=true","-jar","/app/springboot-k8s-app.jar"] </code></pre> <h3>Monitor SpringBoot application from SkyWalking UI</h3> <p>After building the Docker image <em>guopingjia/springboot-k8s-demo:pce</em> and pushing it to the <em>DockerHub</em> registry, deploy the <em>SpringBoot</em> web app in the Kubernetes cluster with the <em>deployment.yaml</em> manifest file:</p> <pre><code class="language-markdown">$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: springboot-k8s-demo spec: selector: matchLabels: app: springboot-k8s-demo replicas: 1 template: metadata: labels: app: springboot-k8s-demo spec: containers: - name: springboot-k8s-demo image: docker.io/guopingjia/springboot-k8s-demo:pce imagePullPolicy: IfNotPresent ports: - containerPort: 8080 $ kubectl apply -f deployment.yaml </code></pre> <p>Upon web app deployment, the built-in Java agent will start collecting application data and posting it to the SkyWalking OAP. All the application metrics will be available in the SkyWalking UI, under the <em>General Service</em> tab:</p> <p><img src="/img/java-app.png" alt=""></p> <p>Below is the <em>SpringBoot</em> web app topology map:</p> <p><img src="/img/java-app-map.png" alt=""></p> <h3>Deploy a multi-tier application</h3> <p>As my second demo application, I will deploy a multi-tier <em>music</em> application, available as part of <a href="https://github.com/apache/skywalking-showcase">Apache SkyWalking showcase application</a>. This multi-tier music application consists of a frontend app server and its UI, a backend gateway service, a recommendation service and songs service, together with an <em>H2</em> database. Each microservice is implemented with a different programming language, e.g. <em>NodeJS</em>, <em>React</em>, <em>Java Spring</em>, <em>Python</em>, etc.</p> <p><img src="/img/multl-tier-app-music.png" alt=""></p> <p>In order to monitor this multi-tier <em>music</em> application from Apache SkyWalking, you need to pick up the SkyWalking agent per programming language and rebuild the corresponding service to collect and send application metrics to the SkyWalking OAP server.</p> <p>You can rebuild the <em>Docker</em> image per service using the agent version file <em><strong>Dockerfile</strong></em> in each service's folder from the multi-tier <em>music</em> application repo:</p> <pre><code class="language-markdown">├── app │   ├── Dockerfile │   ├── Dockerfile.agentless │   ├── Makefile │   ├── package.json │   ├── package-lock.json │   ├── server │   └── ui ├── gateway-service │   ├── build.gradle │   ├── Dockerfile │   ├── Dockerfile.agentless │   ├── gradle │   ├── gradle.properties │   ├── gradlew │   ├── gradlew.bat │   ├── Makefile │   ├── settings.gradle │   └── src ├── recommendation-service │   ├── Dockerfile │   ├── Dockerfile.agentless │   ├── Makefile │   ├── requirements.txt │   └── src └── songs-service ├── build.gradle ├── Dockerfile ├── Dockerfile.agentless ├── gradle ├── gradle.properties ├── gradlew ├── gradlew.bat ├── Makefile ├── settings.gradle └── src </code></pre> <p>After image files are rebuilt with the agents, the multi-tier <em>music</em> application can be deployed in the Kubernetes cluster:</p> <pre><code class="language-markdown">$ envsubst &#x3C; resources.yaml | kubectl create -f - service/gateway created deployment.apps/gateway-deployment created service/songs created deployment.apps/songs-deployment created service/rcmd created deployment.apps/recommendation-deployment created service/app created deployment.apps/app-deployment created deployment.apps/loadgen-deployment created </code></pre> <h3>Monitor multi-tier application from SkyWalking UI</h3> <p>When the multi-tier <em>music</em> app gets deployed, the agents built with each microservice will start collecting application data and post it to the SkyWalking OAP. The multi-tier <em>music</em> application metrics will be available in the SkyWalking UI, under the <em>General Service</em> tab:</p> <p><img src="/img/multl-tier-app.png" alt=""></p> <p>Below is the multi-tier <em>music</em> application topology map:</p> <p><img src="/img/multl-tier-app-map.png" alt=""></p> <p>You can also check the following multi-tier <em>music</em> application trace page. It's very helpful when you debug any performance issue in the application.</p> <p><img src="/img/sw-app-trace.png" alt=""></p> <h3>Application alerting</h3> <p>Apache SkyWalking provides an alerting mechanism to measure application performance according to a list of pre-defined metrics, e.g., <em>service_resp_time</em>, <em>database_access_resp_time</em>, and <em>service_sla</em>. It will trigger alerting when some metrics reach pre-defined thresholds. You can define new metrics using Observability Analysis Language (OAL) or customize the existing metrics with new thresholds.</p> <p>Here you can see the alarms page from SkyWalking UI showing all the triggered alerts for deployed multi-tier <em>music</em> application:</p> <p><img src="/img/sw-app-alarms.png" alt=""></p> <p>The alarms page shows <em>Successful rate of service agent::app is lower than 80% in 2 minutes of last 10 minutes</em>. It indicates an issue from the frontend app server in the multi-tier <em>music</em> application.</p> <p>Apache SkyWalking configures the alerting using a collection of alerting rules located in <em>/skywalking/config/alarm-settings.yml</em> from the SkyWalking OAP pod. You can check the content by running the following command:</p> <pre><code class="language-markdown">$ kukectl exec pod/skywalking-skywalking-helm-oap-bfb57fbf8-5g7k7 -n skywalking -it -- cat /skywalking/config/alarm-settings.yml </code></pre> <p>When comparing it to the output of the <em>alarm-settings.yml</em>, you can see the alerts from the alarms page are triggered by the following metric alerting rule <em>service_sla</em>:</p> <pre><code class="language-markdown"> service_sla_rule: # Metrics value need to be long, double or int metrics-name: service_sla op: "&#x3C;" threshold: 8000 # The length of time to evaluate the metrics period: 10 # How many times after the metrics match the condition, will trigger alarm count: 2 # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period. silence-period: 3 message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes </code></pre> <p>You can go to the frontend app by clicking the service <code>agent::app</code> from SkyWalking UI Service page. From the below service <code>agent::app</code> overview page, it shows <strong>Success Rate 66.66%</strong>.</p> <p><img src="/img/sw-svc-app-overview.png" alt=""></p> <p>You may check further the service's trace page and try to figure out the root cause for this issue.</p> <p><img src="/img/sw-svc-app-trace.png" alt=""></p> <h2>Conclusion</h2> <p>In this blog post, I used the Apache SkyWalking application performance monitoring (APM) tool and explained in detail how to set it up as a <em>self-managed</em> environment in HPE GreenLake for Private Cloud Enterprise to be used for monitoring and alerting applications. Using the instrumentation of multiple supported agents from Apache SkyWalking, the application workloads can be easily monitored through the integrated Apache SkyWalking UI, with a nice application topology map, tracing details and real-time alarms for any application performance issues.</p> <p>In <a href="https://developer.hpe.com/blog/set-up-apache-skywalking-for-k8s-monitoring-in-hpe-greenlake-for-private-cloud-enterprise/">the next blog post of the series</a> , I will show you how to use the Apache SkyWalking APM tool for monitoring of Kubernetes clusters provisioned on HPE GreenLake for Private Cloud Enterprise.</p>https://developer.hpe.com/2023-January-09/https://developer.hpe.com/2023-January-09/Mon, 09 Jan 2023 13:00:00 GMThttps://developer.hpe.com/get-started-with-application-performance-monitoring-tools-overview/https://developer.hpe.com/get-started-with-application-performance-monitoring-tools-overview/Mon, 09 Jan 2023 08:53:23 GMT<h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a> delivers a modern private cloud to support your app workloads with bare metal, containers, and virtual machines (VMs) running in any combination across your edges, colocations, and data centers. It combines self-service resource access for developers with consumption and performance transparency for IT. Within this modern application environment, having a robust application performance monitoring (APM) tool is becoming essential. It can help IT professionals ensure that deployed applications meet the performance, reliability and valuable user experience required by developers, partners and customers.</p> <p>This blog post will give an overview of the existing application performance monitoring (APM) tools, their key features, and their deployment and pricing models. It will provide some guidance on your effort in various APM feature evaluation and product selection. I have written it to help you analyze which parts of your stack require the most monitoring and select APM tools that align with the monitoring needs of your applications and operational environment.</p> <h2>Application Performance Monitoring (APM)</h2> <p>The continued availability and appropriate performance of an application are essential to a company’s ability to maintain uninterrupted business processes. This prevents unnecessary business disruptions and enhances customer satisfaction. To ensure this, many enterprises use application performance monitoring (APM). APM relies on a collection of tools and processes used to track the performance of applications and analyze the reports to spot anomalies and performance-related issues.</p> <p>Modern application architectures can be complex, involving large numbers of services and distributed systems located across multiple networks and physical locations, including the cloud. These environments can be challenging to monitor.</p> <p>APM tools collect data through metrics, traces and logs to measure performance and identify potential problems. They help collect application data across a broader range of environments and performs sophisticated analytics on data patterns to provide insights on large and complex environments. As the business impact of outages rises day by day, more and more businesses are likely to spend money on the best APM tool that matches their monitoring needs.</p> <p>There are a broad range of APM tools to choose from, some specifically dedicated to APM tasks and others with APM functionality built into a broader array of features. Some of the most popular APM tools can be deployed as a Software-as-a-Service (SaaS) solution within a public cloud, or on-premises within a private cloud, or even across a hybrid environment. While you can find a number of APM tools that have a ton of features, covering most use cases, these can also come with premium pricing attached. Choosing a good APM tool that best fits both your monitoring needs and your budget is a challenge.</p> <h2>APM Tools: open-source vs commercial vendor</h2> <p>Open source-based APM tools offer a lot of freedom for users, since users can access and customize the tool's source code for their project-specific needs. They also allow for self-hosting, which can help in the context of tightening data protection laws and removing the privacy and security concerns customers may have to put customer data going to third-party services. Open source APM tools also often offer a vibrant community of active developers who might provide helpful plugins and tips.</p> <p>However, many SaaS-based commercial APM tools offer free or reasonably priced bundles. Open source is rarely <strong>really</strong> free, and many commercial SaaS solutions offer better, more reliable, and reasonably priced APM tools. You should spend time and effort in APM feature evaluation and make the best decision that saves cost and works well with your stack.</p> <h2>APM Tools Overview</h2> <p>From the <a href="https://www.gartner.com/reviews/market/application-performance-monitoring-and-observability">Gartner Research</a>, it provides the reviews and ratings of existing commercial vendor based application performance monitoring and observability tools in the market. In this blog post, I will take a look at some of those APM tools. I also choose some open-source based APM tools to look at in this blog post.</p> <h3><strong>Splunk</strong></h3> <p><img src="/img/splunk.png" alt=""></p> <p><a href="https://www.splunk.com/">Splunk</a> is an extensible data platform that offers a range of solutions for analytics, monitoring and security to identify data patterns, provide metrics and diagnose problems. It delivers real-time monitoring and alerting for all environments, on-premises, hybrid or multicloud.</p> <p>Key features:</p> <ul> <li>Collects data from virtually any source and location</li> <li>Converts logs into metrics and analyzes and correlates data to create real-time visualizations and dashboards</li> <li>Provides a policy-based mechanism to reserve system resources for workload collection</li> <li>Provides a search processing language for both simple searches and advanced data exploration</li> <li>Provides thresholds for monitoring events and proactively warns of potential problems when data passes the threshold</li> <li>Pushes alerts to notify regarding critical events and impending conditions in real-time</li> <li>Analyzes metrics and events data with visualizations like bar charts, reference lines, scatter plots and column charts</li> <li>Offers outlier and anomaly detection and predictive analytics using machine learning toolkit</li> <li>Supports open source algorithms and creates custom machine learning models to help operationalize data</li> </ul> <p>Splunk is more focused on monitoring and analyzing data generated from various machines, converting them so that they can be analyzed by developers. It takes more of a log management approach that makes it ideal for managing and monitoring the large amount of data generated from the devices running on the network. It’s great for analyzing the huge number of log files generated by enterprise systems. It eliminates the need for IT to spend hours trawling through all the logs looking for performance issues. Splunk integrates data streams from a huge number of sources. It supports a wide range of data formats, and <em>.xml</em>, <em>.csv</em> and <em>.json</em> files are all supported. This is important if a company needs data stream integration from multiple data formats.</p> <p>Splunk is a much broader platform and toolset geared for a heavy duty large enterprise. It offers a breadth of management by providing a wide range of products. Splunk bundles similar tools together and offers them as two different types of platforms, <em>Splunk Cloud</em> and <em>Splunk Enterprise</em>. Splunk Cloud can be hosted on the cloud server. The entire set of the configurations, as well as the maintenance, is completely done by Splunk. Splunk Enterprise can be maintained by the data center and users need to just style up the entire hardware infrastructure. The integration of Splunk Cloud and Splunk Enterprise provides end-to-end full-stack coverage across hybrid cloud environments.</p> <p>Apart from being regarded as a <em>Visionary</em> in the latest Gartner Magic Quadrant for APM and Observability, Splunk also has been named as a <em>Leader</em> in the latest <a href="https://www.splunk.com/en_us/blog/security/2022-gartner-magic-quadrant-for-siem-splunk-named-a-leader-for-the-9th-consecutive-year.html">Gartner Magic Quadrant for Security Information and Event Management (SIEM)</a>. Splunk includes more than <em>2300</em> out-of-the-box integrations for comprehensive tech stack visibility. In <a href="https://stackshare.io/splunk">StackShare community</a>, Splunk has been mentioned in <em>79</em> company stacks and <em>437</em> developer stacks. It belongs among the founding members and the number one contributor to <em>OpenTelemetry</em>. Splunk APM supports open, vendor-neutral instrumentation, allowing for even more flexibility.</p> <p>Splunk has a reputation for being expensive. It’s not a low-cost option. Upselling beyond APM, e.g., adding <em>SIEM</em> module and real-time monitoring, can send the budget even higher. It’s very important to determine what you really need and what you can dispense with if you decide to go with Splunk.</p> <h3><strong>New Relic</strong></h3> <p><img src="/img/new-relic.png" alt=""></p> <p><a href="https://newrelic.com/">New Relic</a> is a SaaS-based observability platform that includes APM as one of its key services. Organizations can trace dependencies across their distributed applications to detect anomalies, address errors, optimize performance and improve the customer experience. The product offers visibility into the application stack, from back-end APIs to the user devices.</p> <p>Key features:</p> <ul> <li>Provides flexible instrumentation and dashboarding to collect data to meet the unique needs of specific applications and industries</li> <li>Guides appropriate engineer responses and helps them to the most important performance abnormalities using multiple techniques including AI and ML algorithms</li> <li>Correlates application performance to end-user experience through real-user monitoring and synthetic monitoring</li> <li>Connects application and infrastructure performance to explore the problem</li> <li>Uses multiple data types to count and measure every single request to have performance visibility down to the method level</li> <li>Supports real-time error analysis with on-demand diagnostic tools</li> <li>Integrates with various DevOps tools for incident response, logging and configuration management</li> <li>Supports important cloud service instrumentation</li> <li>Handles spikes in traffic with SaaS based architecture</li> </ul> <p>New Relic is an <em>all-in-one</em> application performance tool that lets you see performance from the end user's experience, through servers and down to the line of application code. It addresses not only APM, but also infrastructure, user monitoring and performance analytics for desktop, web and mobile applications. It offers premium features such as real-time monitoring for mobile, web and cloud application performance. It has a personalized dashboard that keeps track of all monitoring, as well as other activity and application performance. It customizes dashboards and enables alerts with real-time tracking.</p> <p>New Relic has been graded as a <em>Leader</em> in the latest <a href="https://newrelic.com/blog/nerd-life/gartner-magic-quadrant-22">Gartner Magic Quadrant for APM and Observability</a>. In <a href="https://stackshare.io/new-relic#stacks">StackShare community</a>, New Relic has a broader approval rating, being mentioned in <em>11589</em> company stacks and <em>7841</em> developer stacks. New Relic is very strong in using community resources for learning the application, training of users, and troubleshooting issue via self-serve. It provides support through blogs, meetups, and social media channels. New functions, such as anomaly detection in logs, greater support for Microsoft Azure and AWS integration, data exploration, correlation, browser monitoring, instrumentation and <em>AIOps</em>, keep being added and supported. New Relic has outstanding capabilities in reporting and dashboard, user interaction performance, and multicloud resource view. Its <em>OpenTelemetry</em> capabilities and contributions place it ahead of many of other APM tools.</p> <p>New Relic uses the freemium pricing strategy. It’s free to use with its most generous free tier that include <em>100GB</em> data ingest per month for unlimited basic users and 1 free full platform user, with the default data retention of 8 days and up. It then starts its <em>Standard</em> plan at <em>$0.30/GB</em> beyond, based on the amount of data you want to send to New Relic. Based on the number of users and their permissions, the <em>Standard</em> plan offers <em>$49/month</em> for core users, and <em>$99/month</em> for up to 5 full platform users. New Relic offers the <em>Pro</em> and <em>Enterprise</em> plans for teams with more than 5 users and those who have advanced security and support needs. For some premium features, such as real-time application performance monitoring, New Relic is more expensive than other SaaS solutions. You should be careful when deciding whether the additional price for a particular feature is worth it for your company.</p> <h3><strong>Datadog</strong></h3> <p><img src="/img/datadog.png" alt=""></p> <p><a href="https://www.datadoghq.com/">Datadog</a> is a monitoring, security and analytics platform for cloud applications. It brings together end-to-end traces, metrics, and logs to make applications, infrastructure and third-party services entirely observable.</p> <p>Key features:</p> <ul> <li>Aggregates metrics and events across the full DevOps stack with more than 600 built-in integrations</li> <li>Provides full visibility into modern applications for monitoring, troubleshooting and optimizing application performance</li> <li>Analyzes and explores log data in context for troubleshooting and alerting</li> <li>Monitors proactively the user experience in a single platform</li> <li>Correlates frontend performance with business impact</li> <li>Visualizes traffic flow in cloud-native environments</li> <li>Builds real-time interactive dashboards</li> <li>Gets alerted and notified on critical issues</li> <li>Instruments applications with new integrations</li> </ul> <p>Datadog is a SaaS-based application that focuses on cloud monitoring and security with public, private and hybrid options. It takes an infrastructure monitoring approach geared toward analytics and application performance, and it is praised especially for its infrastructure and security monitoring features. Datadog handles the entire DevOps and SRE workflow, including the complete incident management and <em>SIEM</em>. With its built-in security monitoring capabilities, Datadog is able to send observational data to its Cloud SIEM product. It makes the incident management fairly easy by declaring and managing incidents from events and monitors. Users can create incidents, rank them by severity, manage incident resolution by assigning responsible users and teams, and send basic email and notifications. Some of the features provided by Datadog, such as the real-time alerts and automated reports, bring amazing advantages to organizations. New features, such as network monitoring, security analysis, <em>AIOps</em>, business analytics, and incident management, keep being added and supported in Datadog. It offers a much broader applicability both in terms of APM capabilities and monitoring other areas such as infrastructure, device, server, database, and log management.</p> <p>Datadog has been graded as a <em>Leader</em> in the latest <a href="https://www.datadoghq.com/resources/gartner-magic-quadrant-apm-observability-2022/">Gartner Magic Quadrant for API and Observability</a>. In <a href="https://stackshare.io/datadog">StackShare community</a>, Datadog has been mentioned in <em>1271</em> company stacks and <em>6360</em> developer stacks. It supports community APIs and extensions to integrate with existing IT infrastructure. Datadog is a major contributor to <em>OpenTelemetry</em>. Its learning platform offers web-based coding labs. It enables new users to get hands-on experience in a simulated environment and plunges users into the workflow from the start.</p> <p>The Datadog interface offers extensive functionality and supports further customization to dashboards and interfaces. With many supported features, it could be difficult for new users who may be overwhelmed by the number of options. They definitely need to take their time to fully understand its features and how to maximize the benefits of those services. In the beginning, it can be hard to track the log data, and create and manage the customer dashboards. Datadog can work with a wide array of data formats and sources. However, it’s not a platform that can deal with a large number of information sources. Data formats, such as <em>.xml</em>, <em>.csv</em> and <em>.json</em>, are not supported.</p> <p>Datadog prices out at around <em>$15 per user</em>. It has an open pricing policy with published prices. Its pricing per-month options include per host, per million events, and per GB of analyzed log files. As a SaaS-based tool, Datadog offers generally low prices.</p> <h3><strong>Dynatrace</strong></h3> <p><img src="/img/dynatrace.png" alt=""></p> <p><a href="https://www.dynatrace.com/">Dynatrace</a> is a software-intelligence monitoring platform offering various tools focused on monitoring modern infrastructures and distributed applications, user experience, and business intelligence.</p> <p>Key features:</p> <ul> <li>Provides a single agent to automatically discover, instrument and collect monitoring metrics for all types of entities in application environment</li> <li>Ingests metric data and events into its AI engine and provides code-level visibility and root-cause answers for applications</li> <li>Uses an interactive topology map to visualize the dynamic relationships among all application components across every tier</li> <li>Supports automated remediation through integration with any CI/CD tools</li> <li>Monitors cloud environments, virtual machines, network, process, host, server-side service, mobile app and real user</li> <li>Discovers and monitors dynamic microservice workloads running in containers</li> <li>Monitors message queues to gain visibility into microservice communications</li> <li>Provides full front-to-back observability ensuring every application is available, functional, and efficient across every channel for the best customer experiences</li> </ul> <p>Dynatrace is an <em>all-in-one</em> platform that monitors the application performance, the underlying infrastructure and the experience of the end users, thanks to its integrated AI engine. Dynatrace deployment is fairly straightforward. The initial setup process offers sufficient onboarding support for deploying the agent based on the environment. It supports configuring the agent from its Web UI. This makes the setup of log monitoring and APM relatively seamless. Dynatrace’s documentation offers sufficient support to deploy, set up and tweak the agent. <a href="https://university.dynatrace.com/">Dynatrace University</a> is available directly from the UI via a link in the user settings drop-down menu.</p> <p>Dynatrace can be deployed either as a SaaS solution with its data being retained in the cloud, or as a <em>self-managed</em> solution that allows customers to maintain control of where their data resides, whether in the cloud or on-premises. This deployment model can really help in the context of tightening data protection laws in the customer's environment.</p> <p>Dynatrace has been named as a <em>Leader</em> in the latest <a href="https://www.dynatrace.com/monitoring/gartner-magic-quadrant-for-application-performance-monitoring-observability">Gartner Magic Quadrant for APM and Observability</a>. In the latest Gartner Critical Capabilities report, Dynatrace has obtained the highest scores in <em>4</em> of <em>6</em> use cases, ranked as #1 IT Operations, Digital Experience Monitoring (DEM), DevOps/AppDev and SRE/Platform Operations. Dynatrace is a major contributor to <em>OpenTelemetry</em>. Its roadmap for <em>OpenTelemetry</em> also puts it ahead of many of other APM tools.</p> <p>Dynatrace offers minimal alerting, but almost no problem/incident management features out-of-the-box. The third-party incident management and status page solutions must be integrated. Dynatrace has no capabilities in the area of federated, hierarchical, or edge <em>AI/ML</em>.</p> <p>Dynatrace offers a full-stack pricing model, starting at <em>$74/month</em> per 8 GB per host. It also offers individual product pricing models, such as infrastructure monitoring, digital experience monitoring, application security and open ingestion, etc. Each of those pricing models works as an add-on and is not included in the full-stack. They are charged with additional cost.</p> <h3><strong>Elastic</strong></h3> <p><img src="/img/elastic.png" alt=""></p> <p><a href="https://www.elastic.co/">Elastic</a> is a distributed search and analytics solution.</p> <p>Key features:</p> <ul> <li>Operates in a distributed environment with scalability and resiliency</li> <li>Allows full control over data, users and cluster operations with a variety of management tools, such as snapshots, index lifecycle, data tiers, data streams</li> <li>Protects data with a list of security features, such as <em>keystore</em>, encrypted communications, RBAC, IP filtering, security realms, SSO and audit logging</li> <li>Supports customized and reliable alerting and notification integration with any other third-party systems</li> <li>Allows to work with data using various language clients, Elasticsearch <em>DSL</em> and <em>SQL</em>, and <em>REST APIs</em></li> <li>Extends Elasticsearch functionality with various plugins and integrations</li> <li>Runs and manages Elasticsearch across public cloud, private cloud and Kubernetes using <em>Elastic Cloud</em>, <em>Elastic Cloud Enterprise</em> and <em>Elastic Cloud on Kubernetes</em></li> <li>Ingests any data type using language clients, ingest nodes, lightweight shippers or <em>Logstash</em></li> <li>Enriches raw data using a variety of analyzers, tokenizer, filters, and enrichment options</li> <li>Supports document storage, time series analysis and metrics, and geospatial analytics</li> <li>Provides full-text search capabilities with its inverted index, tunable relevance scoring and advanced query <em>DSL</em></li> <li>Finds data relationships through aggregations and graph exploration and creates alerts</li> <li>Models and automates the analysis of time series data, combines alerting and inference using machine learning</li> </ul> <p>Elastic builds and maintains the <em>Elastic Stack</em>, an <em>all-in-one</em> platform built upon the proven <em>Elasticsearch, Logstash, and Kibana (ELK) Stack</em> for the logs, metrics, and application trace data with a multitude of out-of-the-box integrations. Elastic Stack is the foundation for its primary solutions, <em>Elastic Enterprise Search</em>, the fleet of search solutions, <em>Elastic Observability</em>, the solution for unified visibility across logs, metrics and APM data, and <em>Elastic Security</em>, the solution that unifies endpoint protection and <em>SIEM</em>. You can easily deploy any of these solutions as a managed service with Elastic Cloud, with one stack powering three solutions.</p> <p>Elastic has been named as a <em>Visionary</em> in the latest <a href="https://www.elastic.co/explore/devops-observability/2022-gartner-magic-quadrant-apm/">Gartner Magic Quadrant for APM and Observability</a>. It has a modern initial interface that users can take advantage out of the box. It provides a lot of very powerful tools for data ingestion, data enrichment, data analysis and various plugins and open source integrations, from years of development and community input. Elastic has good capabilities across reporting and dashboards, user interaction performance, multicloud resource view, predictive analysis, and intelligent data push. It’s easy to use, but a bit of a hassle to configure and maintain. Since Elastic is based on open source code, it requires technical skills in open source and it has quite high threshold to get over to understand how the system works and how to configure it properly.</p> <p>Elastic offers a 14-day free trial of the <em>Standard</em> plan without requiring credit card details. After this, users can choose from 4 paid subscription plans. The <em>Standard</em> plan starts at <em>$95/month</em>, and it provides access to core security features and solutions including APM. The <em>Gold</em> plan adds custom plugins, while the <em>Platinum</em> plan offers advanced security features and machine learning support. It also includes endpoint detection and response, protection, and event collection capabilities. The <em>Enterprise</em> plan adds additional enterprise features, such as searchable snapshots, <em>Elastic Maps</em> server and data retention for security related data, and raises its cost to <em>$175/month</em>.</p> <h3><strong>Prometheus</strong></h3> <p><img src="/img/prometheus.png" alt=""></p> <p><a href="https://prometheus.io/">Prometheus</a> is an open source system monitoring and alerting toolkit and time series database originally developed by <a href="https://soundcloud.com/">SoundCloud</a>.</p> <p>Key features:</p> <ul> <li>Implements a multi-dimensional data model with time series being identified by metric name and a set of key-value pairs</li> <li>Provides a flexible query language <em>PromQL</em> to leverage the dimensionality</li> <li>Stores time series in memory and on local disk in an efficient custom format with no dependency on distributed storage</li> <li>Records metrics in real time via a pull model over HTTP</li> <li>Allows slicing and dicing of collected time series data to generate ad-hoc graphs, tables, and alerts</li> <li>Supports of pushing time series via an intermediary gateway</li> <li>Discovers targets via service discovery or static configuration</li> <li>Supports multiple modes for visualizing data using a built-in expression browser, <em>Grafana</em> integration and a console template language</li> </ul> <p>Prometheus joined <a href="https://www.cncf.io">Cloud Native Computing Foundation (CNCF)</a> and became its second hosted project after Kubernetes. It has managed to obtain a large and vibrant community of contributors and users ever since. Prometheus is good and focuses mainly on application metric monitoring. In order to have a seamless experience with both metrics and traces that are required by APM, you can integrate Prometheus with other open source tracing tools, such as <a href="https://www.jaegertracing.io/">Jaeger</a>. However, since Jaeger lacks sophisticated capabilities for analyzing and segmenting all of a user's trace data, it has only some support for certain data filtering. Experience with such integration may not be great.</p> <p>Prometheus is an open source tool with <em>46K</em> GitHub stars and <em>7.7K</em> Github forks. In <a href="https://stackshare.io/prometheus">StackShare community</a>, Prometheus has been mentioned in <em>852</em> company stacks and <em>1962</em> developer stacks. Since it is free, Prometheus certainly wins on pricing. However, full functionality of Prometheus demands skills in open source and competence in <em>Apache</em> based applications. Without those required skills and experience, the Prometheus interface can be difficult to master, and some others even find it difficult to set it up and scale.</p> <p>Prometheus is maintained by volunteers, not by a company. It relies on other open source tools for security. Fixing security issues in Prometheus is done on a <em>best-effort</em> basis. Prometheus strives to release security fixes within 7 days for its key components <em>alertmanager</em>, <em>node exporter</em>, <em>blackbox exporter</em> and <em>pushgateway</em>, etc.</p> <h3><strong>Apache SkyWalking</strong></h3> <p><img src="/img/skywalking.png" alt=""></p> <p><a href="https://skywalking.apache.org/">Apache SkyWalking</a> is an open source APM tool with capabilities for monitoring, tracing and diagnosing distributed system. It’s especially designed for microservices, cloud native and container-based architectures.</p> <p>Key features:</p> <ul> <li>Provides metrics analysis of services, service instances and endpoints with distributed tracing, log collecting and metrics collecting and customization</li> <li>Supports root cause analysis with profiling the code on the runtime by in-process agent, <em>eBPF</em> profiler and network profiler</li> <li>Provides dependency analysis of service instances and endpoints</li> <li>Supports service topology map analysis</li> <li>Detects slow services and endpoints and provides performance optimization</li> <li>Detects slow <em>SQL</em> statement for database performance monitoring</li> <li>Provides message queue performance and consuming latency monitoring</li> <li>Starts tracing from browser for browser performance monitoring</li> <li>Supports infrastructure monitoring for Kubernetes and Linux</li> <li>Supports alerting using rules in both observability analysis language and metric analysis language</li> </ul> <p>Apache SkyWalking provides a list of agents to be used for building <em>Java</em>, <em>.NET Core</em>, <em>PHP</em>, <em>Node.js</em>, <em>Golang</em>, <em>LUA</em>, <em>Rust</em> and <em>C++</em> apps. It supports to integrate and collect data from multiple sources, including <em>Prometheus</em>, <em>OpenTelemetry</em> and <em>Zabbix</em> for metrics and logs, <em>Zipkin</em> for traces. It provides tracing, metrics analysis, alerting, service mesh observability and visualization.</p> <p>Apache SkyWalking is an open source tool with <em>21K</em> GitHub stars and <em>6K</em> GitHub forks. In <a href="https://stackshare.io/apache-skywalking#stacks.">StackShare community</a>, Apache SkyWalking does not yet have as great a share of the market, only being mentioned in 12 developer stacks. However, Apache SkyWalking has more than <em>600</em> contributors on GitHub and thousands of contributions every year. All the agents for application instrumentation have been actively maintained.</p> <p>Apache SkyWalking is the first open source project that initialized and implemented an <a href="https://www.envoyproxy.io/docs/envoy/v1.18.2/api-v2/service/accesslog/v2/als.proto">Envoy Access Log Service (ALS)</a> based solution to provide observability on the service mesh, no matter the architecture or language. Since service mesh provides full control of the routed <em>RPC</em>, including <em>HTTP</em> and <em>TCP</em>, this observation solution is much easier to be added without language-specific technology. With this solution, users can get the application service topology map, metrics graph, request details and error message with a very nice visualization. This integration solution can be extremely important for monitoring and visualizing applications that consist of many microservices running across on-premises, cloud-based or hybrid environments.</p> <p>Apache SkyWalking is lightweight and scalable, and it supports alerting and visualization. It can be easily set up as a <em>self-managed</em> APM tool within an on-premises data center. This avoids leasing customer data to third-party services and removes the restricted security restriction in the user's environment.</p> <h2>Conclusion</h2> <p>In this blog post, I hope I gave you a closer look at some of the best APM tools that are out there, both open-source and commercial vendor based. In it, I have listed the key features of each APM tool and discussed each tool's strengths and weaknesses. The importance of a good APM solution is now indisputable. All it takes is to pick the right one based on your monitoring needs for your applications.</p> <p>This blog post is the first of three in a series. In <a href="https://developer.hpe.com/blog/set-up-apache-skywalking-for-k8s-and-vm-monitoring-in-hpe-greenlake-private-cloud/">the second post of the series</a>, I will show you the detailed process on how to set up the Apache SkyWalking APM tool for monitoring and alerting of customer applications deployed on Kubernetes cluster provisioned on HPE GreenLake for Private Cloud Enterprise. In <a href="https://developer.hpe.com/blog/set-up-apache-skywalking-for-k8s-monitoring-in-hpe-greenlake-for-private-cloud-enterprise/">the third post of the series</a>, I will expand on setting up Apache SkyWalking APM tool to monitor the infrastructure of Kubernetes clusters deployed on HPE GreenLake for Private Cloud Enterprise.</p> <h2>Reference</h2> <ul> <li><a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a></li> <li><a href="https://www.splunk.com/">Splunk</a></li> <li><a href="https://newrelic.com/">New Relic</a></li> <li><a href="https://www.datadoghq.com/">Datadog</a></li> <li><a href="https://www.dynatrace.com/">Dynatrace</a></li> <li><a href="https://www.elastic.co/">Elastic</a></li> <li><a href="https://prometheus.io/">Prometheus</a></li> <li><a href="https://skywalking.apache.org/">Apache SkyWalking</a></li> </ul>https://developer.hpe.com/announcing-chapel-1-29-0/https://developer.hpe.com/announcing-chapel-1-29-0/Sat, 07 Jan 2023 18:07:06 GMT<p>External Blog</p>https://developer.hpe.com/monitor-application-performance-across-hybrid-cloud-environment-using-apache-skywalking-and-service-mesh/https://developer.hpe.com/monitor-application-performance-across-hybrid-cloud-environment-using-apache-skywalking-and-service-mesh/Fri, 06 Jan 2023 06:42:26 GMT<h2>Introduction</h2> <p>In <a href="https://developer.hpe.com/blog/how-to-deploy-application-across-hybrid-clouds-beyond-hpe-greenlake-for-private-cloud-enterprise/">my previous blog post</a>, I covered the detailed process of deploying the complex Online Boutique application in a hybrid cloud environment, across the public EKS cluster from AWS to the private Kubernetes cluster in <a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a>. This hybrid cloud model amplifies the benefits of both private and public clouds and allows for more seamless integration across technical barriers. It enables the enterprise to rely on the security of on-premises data centers while taking advantage of the agility of managing the front-end of an application in the public cloud. This model is becoming increasingly popular as more businesses and enterprises shift toward cloud-based computing.</p> <p>However, this evolution presents challenges for monitoring, as applications and services are inherently more distributed in this environment. Having a good application performance monitoring (APM) tool is becoming essential within the hybrid cloud environment. It can consolidate performance metrics and troubleshoot data for assets across the hybrid cloud environment into a single application. It makes it easier to track metrics and allows the enterprise to resolve problems quickly.</p> <p>For this blog post, I chose Apache SkyWalking as an APM tool and describe how to set up it, as a <em>self-hosted</em> APM tool, for monitoring and alerting the application performance across the hybrid cloud environment. I took the service mesh as an auto instrumentation mechanism for application monitoring, without adding any manual process to instrument existing applications.</p> <h2>Apache SkyWalking</h2> <p><a href="https://skywalking.apache.org/">Apache SkyWalking</a> is an open source APM tool that can monitor, trace, and diagnose distributed systems. It’s especially designed for microservices, cloud native and container-based architectures.</p> <p>Apart from a list of supported agents to be used for instrumenting applications, Apache SkyWalking implements an <a href="https://www.envoyproxy.io/docs/envoy/v1.18.2/api-v2/service/accesslog/v2/als.proto">Envoy Access Log Service (ALS)</a> based solution to provide observability on the service mesh under Kubernetes environment, no matter the architecture or language. A service mesh provides a mesh of <em>Layer 7</em> proxies that manage network traffic between services. It supports application observability at the platform layer, instead of the application layer, by abstracting away how inter-process and service-to-service communications being handled in Kubernetes. Using a list of implemented analyzers, e.g., <em>k8s-mesh</em> and <em>mx-mesh</em>, Apache SkyWalking can receive and analyze the detailed access logs of all requests, both <em><strong>HTTP</strong></em> and <em><strong>TCP</strong></em>, emitted from Envoy ALS. With this solution, users could get the application service topology map, metrics graph, request details and error message with a very nice visualization. This observation solution is much easier to be added without language-specific technology. It can be extremely important for monitoring and visualizing applications that consist of many microservices running across on-premises, cloud-based or hybrid environments.</p> <p>Apache SkyWalking is lightweight and scalable. It can be easily set up as a <em>self-hosted</em> APM tool within a hybrid cloud environment, without any additional external resources for hosting the tool. This can help in the context of a resource-constrained environment and remove the privacy and security concerns customers may have in putting customer data that goes out to third-party services.</p> <h2>Set up Apache SkyWalking</h2> <h3>Prerequisites</h3> <p>Before you start, make sure you have the following required elements:</p> <ul> <li>A public Kubernetes cluster from one of the public cloud providers such as <em>AWS</em>, <em>Microsoft Azure</em> or <em>Google</em>. For the purposes of this case study blog post, one EKS cluster, named <em>eks-cfe-public</em> from AWS, is used. However, it works if you choose a cluster from other providers.</li> <li>A private Kubernetes cluster, named <em>eks-pce-clu-1</em> provisioned in HPE GreenLake for Private Cloud Enterprise.</li> <li>The <em>kubectl</em> CLI tool, version 1.23 or later, together with the <em>kubeconfig</em> files for accessing both public and private clusters.</li> <li>The <em><a href="https://helm.sh/docs/intro/install/">Helm</a></em> CLI tool, version 3.8.1 or later.</li> <li>The <a href="https://istio.io/latest/docs/reference/commands/istioctl/">istioctl</a> CLI tool, version 1.16.0 or later. Use the <a href="https://istio.io/latest/docs/setup/install/istioctl/">istioctl Installation</a> to install this CLI tool to your local development environment. The <em>istioctl</em> client will be used for installing and set up <em>Istio</em> service mesh.</li> <li>The <a href="https://skupper.io/">Skupper</a> CLI tool, the latest version 1.2.0. Use the <a href="https://skupper.io/start/#step-1-install-the-skupper-command-line-tool-in-your-environment">Skupper Installation</a> to install this CLI tool to your local development environment.</li> </ul> <h3>Deploy Apache SkyWalking to AWS EKS cluster</h3> <p>Install Apache SkyWalking using Helm charts with <em>elasticsearch</em> as storage to the public AWS EKS cluster:</p> <pre><code class="language-markdown">$ git clone https://github.com/apache/skywalking-kubernetes $ cd skywalking-kubernetes/chart $ helm repo add elastic https://helm.elastic.co $ helm dep up skywalking $ kubectl create ns skywalking $ helm install skywalking skywalking –n skywalking \ --set oap.image.tag=9.2.0 \ --set ui.image.tag=9.2.0 \ --set oap.storageType=elasticsearch \ --set elasticsearch.imageTag=7.17.1 \ --set elasticsearch.persistence.enabled=true \ --set values.telemetry.v2.enabled=true \ --set oap.envoy.als.enabled=true \ --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=mx-mesh \ --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=mx-mesh </code></pre> <p>Above the Helm commands, install the Apache SkyWalking to the namespace <em>skywalking</em> of the AWS EKS cluster. It uses the <em>elasticsearch</em> as the storage type and creates it as a <em>StatefulSet</em> resource, running a pod on each worker node. It installs the Apache SkyWalking Observability Analysis Platform (OAP) with replicas being set as 2 to ensure high availability. The installation enables the Envoy Access Log Service (ALS) and specifies the ALS analyzer as <em>mx-mesh</em>.</p> <p>It should be noted that Apache SkyWalking also supports the ALS analyzer <em>k8s-mesh</em>, which uses the metadata from Kubernetes cluster to analyze the logs. It requires the SkyWalking OAP server to access the Kubernetes API server to get information of pods, services and service endpoints. This works only for monitoring a single cloud environment. In the case of a hybrid cloud, you need to use the ALS analyzer <em>mx-mesh</em>, which uses the Envoy metadata exchange mechanism to get the service names. It's required for monitoring applications deployed in the hybrid cloud environment.</p> <p>You can check the detailed Apache SkyWalking installation by typing the following <em>kubectl</em> command:</p> <pre><code class="language-markdown">$ kubectl get all -n skywalking NAME READY STATUS RESTARTS AGE pod/elasticsearch-master-0 1/1 Running 0 6h34m pod/elasticsearch-master-1 1/1 Running 0 6h34m pod/skywalking-oap-init-n92hp 1/1 Completed 0 88s pod/skywalking-skywalking-helm-oap-bfb57fbf8-27frm 1/1 Running 0 92s pod/skywalking-skywalking-helm-oap-bfb57fbf8-djzw5 1/1 Running 0 52s pod/skywalking-skywalking-helm-ui-7776f4854d-nvkds 1/1 Running 0 6h34m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-master ClusterIP 172.20.171.140 &#x3C;none> 9200/TCP,9300/TCP 6h34m service/elasticsearch-master-headless ClusterIP None &#x3C;none> 9200/TCP,9300/TCP 6h34m service/skywalking-skywalking-helm-oap ClusterIP 172.20.155.177 &#x3C;none> 11800/TCP,12800/TCP 6h34m service/skywalking-skywalking-helm-ui ClusterIP 172.20.205.170 &#x3C;none> 80/TCP 6h34m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/skywalking-skywalking-helm-oap 2/2 2 2 6h34m deployment.apps/skywalking-skywalking-helm-ui 1/1 1 1 6h34m NAME DESIRED CURRENT READY AGE replicaset.apps/skywalking-skywalking-helm-oap-bfb57fbf8 2 2 2 96s replicaset.apps/skywalking-skywalking-helm-ui-7776f4854d 1 1 1 6h34m NAME READY AGE statefulset.apps/elasticsearch-master 2/2 6h34m NAME COMPLETIONS DURATION AGE job.batch/skywalking-oap-init 1/1 31s 94s </code></pre> <p>Change the service types for both service <em>skywalking-skywalking-helm-oap</em> and <em>skywalking-skywalking-helm-ui</em> from <em>ClusterIP</em> to <em>LoadBalancer</em>. With the built-in support of Elastic Load Balancing (ELB) in AWS EKS cluster, the externally accessible load balancing host names will be created for those two services in the Apache SkyWalking installation.</p> <pre><code class="language-markdown">$ kubectl edit service/skywalking-skywalking-helm-ui -n skywalking $ kubectl edit service/skywalking-skywalking-helm-oap -n skywalking $ k get svc -n skywalking -l app=skywalking NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE skywalking-skywalking-helm-oap LoadBalancer 172.20.155.177 afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com 11800:30642/TCP,12800:32269/TCP 38d skywalking-skywalking-helm-ui LoadBalancer 172.20.205.170 a2dea6e89216444e28ed29ef48c0b0fa-951983485.us-east-2.elb.amazonaws.com 80:31740/TCP 38d </code></pre> <p>The Apache SkyWalking UI can be made (publicly) accessible using the assigned URL <a href="http://a2dea6e89216444e28ed29ef48c0b0fa-951983485.us-east-2.elb.amazonaws.com/">http://a2dea6e89216444e28ed29ef48c0b0fa-951983485.us-east-2.elb.amazonaws.com/</a>. The Apache SkyWalking OAP server ELB host name with its port <em>11800</em>, <strong>afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com:11800</strong>, will be used in the following configuration to send the application metrics.</p> <h3>Install <em>Istio</em> service mesh</h3> <p>Install <em>istio</em> service mesh using <em>istioctl</em> with Envoy Access Log Service enabled:</p> <pre><code class="language-markdown">$ kubectl create ns istio-system $ istioctl install -y --set profile=demo \ --set meshConfig.enableEnvoyAccessLogService=true \ --set meshConfig.defaultConfig.envoyAccessLogService.address=afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com:11800 \ --set meshConfig.defaultConfig.envoyMetricsService.address=afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com:11800 \ --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[0]=.*' </code></pre> <p>Run the above commands on both the AWS EKS cluster and the private Kubernetes cluster. It installs the <em>istio</em> to the namespace <em>istio-system</em> of the clusters. It explicitly enables the Envoy Access Log Service (ALS), with both <em>envoyAccessLogService.address</em> and <em>envoyMetricsService.address</em> settings pointing to the Apache SkyWalking OAP server ELB host name and its port <em>11800</em>.</p> <p>You can check the detailed <em>istio</em> installation by typing the following <em>kubectl</em> command:</p> <pre><code class="language-markdown">$ kubectl get all -n istio-system NAME READY STATUS RESTARTS AGE pod/istio-egressgateway-78fb5cf46-djxv2 1/1 Running 0 38d pod/istio-ingressgateway-77b9d69b74-499vf 1/1 Running 0 38d pod/istiod-67fcb675b5-2dhjw 1/1 Running 0 38d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/istio-egressgateway ClusterIP 172.20.224.42 &#x3C;none> 80/TCP,443/TCP 38d service/istio-ingressgateway LoadBalancer 172.20.249.41 a17641fd9b6564b02ab3cc5faeb51e7a-1241343104.us-east-2.elb.amazonaws.com 15021:30506/TCP,80:31341/TCP,443:31933/TCP,31400:31131/TCP,15443:32118/TCP 38d service/istiod ClusterIP 172.20.95.232 &#x3C;none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 38d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/istio-egressgateway 1/1 1 1 38d deployment.apps/istio-ingressgateway 1/1 1 1 38d deployment.apps/istiod 1/1 1 1 38d NAME DESIRED CURRENT READY AGE replicaset.apps/istio-egressgateway-78fb5cf46 1 1 1 38d replicaset.apps/istio-ingressgateway-77b9d69b74 1 1 1 38d replicaset.apps/istiod-67fcb675b5 1 1 1 38d </code></pre> <h3>Deploy Online Boutique application</h3> <p>Please refer to my blog post <a href="https://developer.hpe.com/blog/how-to-deploy-application-across-hybrid-clouds-beyond-hpe-greenlake-for-private-cloud-enterprise/">here</a> on how to deploy the Online Boutique application across the public AWS EKS cluster and the private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise. The <em>Skupper</em> CLI tool is used for deploying the Virtual Application Network (VAN) and creating connection between the private Kubernetes cluster and the public AWS EKS cluster.</p> <p>After deploying the Online Boutique application, you can verify deployment by checking the <em>Skupper</em> status from the deployment namespace <em>boutique</em>:</p> <ul> <li>from private Kubernetes cluster:</li> </ul> <pre><code class="language-markdown">$ skupper status --namespace boutique Skupper is enabled for namespace "boutique" with site name "pce-private" in interior mode. It is connected to 1 other site. It has 10 exposed services. </code></pre> <ul> <li>from AWS EKS cluster:</li> </ul> <pre><code class="language-markdown">$ skupper status --namespace boutique Skupper is enabled for namespace "boutique" with site name "aws-public" in interior mode. It is connected to 1 other site. It has 10 exposed services. The site console url is: https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8080 The credentials for internal console-auth mode are held in secret: 'skupper-console-users‘ </code></pre> <p>The Skupper console shows the connections from the public AWS EKS cluster to the private Kubernetes cluster and the hybrid deployment of the Online Boutique application:</p> <p><img src="/img/skupper-apps-monitor.png" alt=""></p> <p>The Online Boutique UI can be accessed through deployed frontend service URL from public AWS EKS cluster:</p> <p><img src="/img/online-boutique-frontend-monitor.png" alt=""></p> <h3>Online Boutique application monitoring using ALS</h3> <h4>1 Add the label <em>istio-injection=enabled</em> to namespace <em>boutique</em></h4> <p>Run the following command, from both the AWS EKS cluster and the private Kubernetes cluster, to add the label <em>istio-injection=enabled</em> to the namespace <em>boutique</em>, in which the Online Boutique application is deployed. This will enable <em>istio</em> injection for this namespace:</p> <pre><code class="language-markdown">$ kubectl label namespace boutique istio-injection=enabled $ kubectl get ns boutique --show-labels NAME STATUS AGE LABELS boutique Active 3d istio-injection=enabled,kubernetes.io/metadata.name=boutique </code></pre> <h4>2 Restart Online Boutique application deployment</h4> <p>Run the following command from both the AWS EKS cluster and the private Kubernetes cluster:</p> <pre><code class="language-markdown">$ kubectl rollout restart deployment </code></pre> <p>This command will restart all the deployments in the namespace <em>boutique</em> by terminating the related pods and re-creating them. Since the namespace has the label <em>istio-injection=enabled</em> added, the newly created pods will be injected automatically with the <em>istio-proxy</em> container. The <em>PROXY_CONFIG</em> in the <em>istio-proxy</em> container contains the configuration of <em>envoyAccessLogService.address</em> and <em>envoyMetricsService.address</em>, both pointing to the Apache SkyWalking OAP server URL:</p> <pre><code class="language-markdown"> istio-proxy: Container ID: docker://2e0fa0a4e290a0138e59fe11e4bd4cdaffa329e5b780e9ed227089bb10660c73 Image: docker.io/istio/proxyv2:1.16.0 Image ID: docker-pullable://istio/proxyv2@sha256:f6f97fa4fb77a3cbe1e3eca0fa46bd462ad6b284c129cf57bf91575c4fb50cf9 Port: 15090/TCP Host Port: 0/TCP Args: proxy sidecar --domain $(POD_NAMESPACE).svc.cluster.local --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --log_output_level=default:info --concurrency 2 State: Running Started: Fri, 02 Dec 2022 16:34:23 +0100 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 1Gi Requests: cpu: 10m memory: 40Mi Readiness: http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30 Environment: JWT_POLICY: third-party-jwt PILOT_CERT_PROVIDER: istiod CA_ADDR: istiod.istio-system.svc:15012 POD_NAME: productcatalogservice-77589df479-p2p2c (v1:metadata.name) POD_NAMESPACE: boutique (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) SERVICE_ACCOUNT: (v1:spec.serviceAccountName) HOST_IP: (v1:status.hostIP) PROXY_CONFIG: {"envoyAccessLogService":{"address":"afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com:11800"},"envoyMetricsService":{"address":"afd4a163a65c74af5ad732fcf86b7dff-261027448.us-east-2.elb.amazonaws.com:11800"},"proxyStatsMatcher":{"inclusionRegexps":[".*"]}} ISTIO_META_POD_PORTS: [ {"containerPort":3550,"protocol":"TCP"} ] ISTIO_META_APP_CONTAINERS: server ISTIO_META_CLUSTER_ID: Kubernetes ISTIO_META_INTERCEPTION_MODE: REDIRECT ISTIO_META_WORKLOAD_NAME: productcatalogservice ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/boutique/deployments/productcatalogservice ISTIO_META_MESH_ID: cluster.local TRUST_DOMAIN: cluster.local``` </code></pre> <p>It should be noted that, in order to avoid collecting ALS logs for the Skupper deployments (e.g., <em>skupper-router</em> &#x26; <em>skupper-service-controller</em> deployed to the same namespace that are used for multicloud communication and hybrid application deployment), we can add the annotation <em>sidecar.istio.io/inject: false</em> to those <em>Skupper</em> deployments. After restart deployments, the <em>Skupper</em> deployments will not inject the <em>istio-proxy</em> to their pods.</p> <h4>3 Monitor Online Boutique application</h4> <p>After following these steps, the Online Boutique application metrics will be visible from the Apache SkyWalking UI, under the <em>Service Mesh</em> tab:</p> <p><img src="/img/sw-app-svc.png" alt=""></p> <p>The SkyWalking UI <em>Topology</em> page will show the Online Boutique application topology map:</p> <p><img src="/img/sw-app-map.png" alt=""></p> <p>You can check the service <em>Overview</em> and <em>Endpoint</em> pages per service, e.g., <em>frontend</em>:</p> <p><img src="/img/sw-app-overview.png" alt=""></p> <p><img src="/img/sw-app-endpoint.png" alt=""></p> <p>You can also check all the services from the <em>Data Plane</em> tab under <em>Service Mesh</em> that are observed through <em>Envoy Metrics Service</em> setup:</p> <p><img src="/img/sw-app-svc-dataplane.png" alt=""></p> <h3>Online Boutique application alerting</h3> <p>Apache SkyWalking provides an alerting mechanism to measure application performance according to a list of pre-defined metrics, e.g., <em>service_resp_time</em>, <em>service_instance_resp_time</em>, and <em>service_sla</em>. It will trigger alerting when some metrics reach its pre-defined thresholds.</p> <p>Apache SkyWalking configures the alerting using a collection of alerting rules located in <em>/skywalking/config/alarm-settings.yml</em> from the SkyWalking OAP pod. You can check the content by running the following command:</p> <pre><code class="language-markdown">$ kukectl exec pod/skywalking-skywalking-helm-oap-bfb57fbf8-5g7k7 -n skywalking -it -- cat /skywalking/config/alarm-settings.yml </code></pre> <p>You can define new metrics by adding a new entry to the file using the SkyWalking observability analysis language (OAL), or customize the existing metrics with new thresholds.</p> <p>Below is the alarms page from the SkyWalking UI showing all the triggered alerts for a deployed Online Boutique application:</p> <p><img src="/img/sw-alert.png" alt=""></p> <p>The alarms page shows an alert * Response time of service instance frontend-549fd9954f-lvnsv of frontend is more than 1000ms in 2 minutes of last 10 minutes *.</p> <p>This alert is triggered by the following metric alerting rule for the metric <em>service_instance_resp_time</em>:</p> <pre><code class="language-markdown"> service_instance_resp_time_rule: metrics-name: service_instance_resp_time op: ">" threshold: 1000 period: 10 count: 2 silence-period: 5 message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes </code></pre> <p>It indicates an issue from the <em>frontend</em> service instance in Online Boutique application. You can check the service trace page further to figure out the root cause of this issue.</p> <h2>Conclusion</h2> <p>This blog post discussed the challenges in hybrid cloud monitoring and described the process of setting up Apache SkyWalking as a <em>self-hosted</em> APM tool for application performance monitoring across a hybrid cloud environment. Instead of taking the manual instrumentation mechanism to rebuild the application with various agents to collect and send the application metrics, we used an auto instrumentation approach with service mesh in the setup. This Envoy Access Log Service (ALS) based approach did not require any change to the deployed applications and the setup process showed that it was very easy to configure it for hybrid cloud monitoring. This integration solution can be extremely important for monitoring and visualizing an application that consists of many microservices running across hybrid cloud environments.</p>https://developer.hpe.com/hpe-joins-the-linux-foundation’s-spdx-workgroup-as-an-spdx-supporter/https://developer.hpe.com/hpe-joins-the-linux-foundation’s-spdx-workgroup-as-an-spdx-supporter/Thu, 05 Jan 2023 17:52:22 GMT<p>Hewlett Packard Enterprise (HPE) is pleased to have joined <a href="https://www.linuxfoundation.org/projects">The Linux Foundation®</a> <a href="https://spdx.dev/">Software Package Data Exchange (SPDX)®</a> workgroup as an SPDX Supporter.</p> <p>HPE has chosen <a href="https://spdx.dev/about/">SPDX</a> for its software bills of materials (SBOMs) to communicate an ingredient list for its software products. SBOMs written in SPDX-format are easy to understand … even by a human. The real value in using a standardized format like SPDX, however, will be in the creation of automation tooling to consume and report back on the information contained in SBOMs, especially when matching them up to vulnerability databases. Using SPDX, HPE will be able to adapt its reporting for a variety of initiatives, including [US Executive Order #14028](<a href="https://www.federalregister.gov/documents/2021/05/17/2021-10460/improving-the-nations-cybersecurity(May)">https://www.federalregister.gov/documents/2021/05/17/2021-10460/improving-the-nations-cybersecurity(May)</a> as well as global initiatives aligned with software security and supply chain management. Furthermore, since SPDX has been <a href="https://www.iso.org/standard/81870.html">approved</a> by the International Organization for Standardization (ISO), it will be easier to cross-collaborate with external suppliers who also choose this standard format.</p> <p>SPDX also helps with a broader issue at the intersection of <em>Open Source Street</em> and <em>Security Avenue</em>. Various public security feeds/databases use different names to describe the same packages/components. By supporting the inclusion of alternative identifiers (e.g., PURL, SWID, etc.), SPDX provides an entry point for teams to correlate the SBOM information they receive from a supplier with a larger number of public security feeds or even cross-correlate with SBOMs created using the other popular SBOM specifications. Finally, HPE supports SPDX’s efforts in <a href="https://github.com/spdx/license-list-XML/blob/main/DOCS/license-inclusion-principles.md">reviewing licenses for inclusion</a>. SPDX’s legal group serves an important function in identifying and cataloguing commonly used open source licenses with short identifiers to aid in license identification for open source projects.</p> <p>HPE is committed to software security, and we want to provide our customers with confidence in our products by using a format that serves both security and license identification initiatives.</p>https://developer.hpe.com/how-to-deploy-application-across-hybrid-clouds-beyond-hpe-greenlake-for-private-cloud-enterprise/https://developer.hpe.com/how-to-deploy-application-across-hybrid-clouds-beyond-hpe-greenlake-for-private-cloud-enterprise/Thu, 05 Jan 2023 07:52:05 GMT<h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/private-cloud-enterprise.html">HPE GreenLake for Private Cloud Enterprise</a> delivers a modern private cloud to support your app workloads with bare metal, containers, and virtual machines (VMs) running in any combination across your edges, colocations, and data centers. It combines self-service resource access for developers with consumption and performance transparency for IT.</p> <p>This blog post shows you how to deploy a complex application that consists of multiple microservices as a hybrid app that spans both a public AWS EKS cluster and a private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise. By using a hybrid cloud solution, you can combine the compliance benefits of a private cloud in HPE GreenLake for Private Cloud Enterprise environment with the scalability and connectivity of the public cloud. You can rely on the security of finely tuned, on-premises data centers while turning to the agility of cloud computing to manage the front end of an application in the public cloud. Using HPE GreenLake for Private Cloud Enterprise, you can optimize resource allocation, save costs, and improve overall productivity and performance in the process.</p> <h2>Prerequisites</h2> <p>Before you start, make sure you have the following required components:</p> <ul> <li>A public Kubernetes cluster from one of the public cloud providers such as <em>AWS</em>, <em>Microsoft Azure</em> or <em>Google</em>. For the purposes of the use case being highlighted in this blog post, a single EKS cluster, named <em>eks-cfe-public</em> from AWS, is being used. However, it works if you choose a cluster from other providers.</li> <li>A private Kubernetes cluster, named <em>eks-pce-clu-1</em> provisioned in HPE GreenLake for Private Cloud Enterprise.</li> <li>The <em>kubectl</em> CLI tool, version 1.23 or later, together with the <em>kubeconfig</em> files for accessing both the public AWS EKS cluster and private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise. To simplify the setup process, you can start two terminal sessions in your environment, export the environment variable <code>KUBECONFIG</code> in each session and point it to the <em>kubeconfig</em> file for accessing the public AWS EKS cluster and the private Kubernetes cluster, respectively.</li> <li>The <a href="https://skupper.io/">Skupper</a> CLI tool, the latest version 1.2.0. Use the <a href="https://skupper.io/start/#step-1-install-the-skupper-command-line-tool-in-your-environment">Skupper Installation</a> to install this CLI tool to your environment. The <em>Skupper</em> CLI tool works with the same environment setup for <em>kubectl</em> for accessing the public AWS EKS cluster and private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise. Some options, e.g., <em>--kubeconfig</em>, <em>--context</em>, and <em>--namespace</em>, can be used explicitly in <em>Skupper</em> for using a specific <em>kubeconfig</em> file and context or accessing a Kubernetes namespace.</li> </ul> <h2>Online Boutique</h2> <p><a href="https://github.com/GoogleCloudPlatform/microservices-demo">Online Boutique</a> is a cloud-first microservices demo application. It consists of an <em>11-tier</em> microservices application. The application is a web-based e-commerce app where users can browse items, add them to the cart, and purchase them. This demo app has been used widely for demonstrating various technologies. It’s easy to deploy and it works on any Kubernetes cluster.</p> <p>This blog post will use the <em>Online Boutique</em> as the demo application, deploying it across the public AWS EKS cluster and the private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise using <em>Skupper</em>.</p> <p><img src="/img/apps.png" alt=""></p> <h2>Skupper</h2> <p><a href="https://skupper.io/">Skupper</a> is a <em>Layer 7</em> service interconnect. It enables secure communication across multiple Kubernetes clusters through a Virtual Application Network (VAN). The VAN connects the applications and services in multiple clusters into a virtual network so that they can communicate with each other as if they were all running in the same site. VANs are able to provide connectivity across the hybrid cloud because they operate at Layer 7 (the application layer). They use Layer 7 application routers to route communication between Layer 7 application addresses.</p> <p>With <em>Skupper</em>, your application can span multiple cloud providers, data centers, and regions with no VPNs or special firewall rules.</p> <h2>Deploy Online Boutique application</h2> <p>Clone the <a href="https://github.com/GoogleCloudPlatform/microservices-demo">Online Boutique</a> GitHub repo to your local environment:</p> <pre><code class="language-markdown">$ git clone https://github.com/GoogleCloudPlatform/microservices-demo.git $ cd microservices-demo/release/ </code></pre> <p>From the manifests file <em>kubernetes-manifests.yaml</em> in the folder, create the following 3 manifests files:</p> <ul> <li> <p><strong>k8s-manifests-deploy-private.yaml</strong>, including the following 3 <em>Deployment</em> manifests:</p> <ul> <li><em>emailservice</em></li> <li><em>paymentservice</em></li> <li><em>shippingservice</em></li> </ul> </li> <li> <p><strong>k8s-manifests-deploy-public.yaml</strong>, including the following 7 <em>Deployment</em> manifests:</p> <ul> <li><em>frontend</em></li> <li><em>recommendationservice</em></li> <li><em>productcatalogservice</em></li> <li><em>checkoutservice</em></li> <li><em>cartservice</em></li> <li><em>currencyservice</em></li> <li><em>redis-cart</em></li> <li><em>adservice</em></li> </ul> </li> <li> <p><strong>k8s-manifests-service-public.yaml</strong>, including the following 2 <em>Service</em> manifests:</p> <ul> <li><em>frontend</em></li> <li><em>frontend-external</em></li> </ul> </li> </ul> <h3>Deploy application microservices to AWS EKS cluster</h3> <p>Create the namespace <em>boutique</em> in the AWS EKS cluster and then deploy 7 <em>Deployment</em> and 2 <em>Service</em> resources to the namespace:</p> <pre><code class="language-markdown">$ kubectl create ns boutique $ kubectl config set-context --current --namespace boutique $ kubectl apply -f k8s-manifests-deploy-public.yaml deployment.apps/recommendationservice created deployment.apps/frontend created deployment.apps/productcatalogservice created deployment.apps/checkoutservice created deployment.apps/cartservice created deployment.apps/currencyservice created deployment.apps/redis-cart created deployment.apps/adservice created $ kubectl apply -f k8s-manifests-service-public.yaml service/frontend created service/frontend-external created $ kubectl get svc frontend ClusterIP 172.20.103.129 &#x3C;none> 80/TCP 40s &#x3C;none> frontend-external LoadBalancer 172.20.16.223 a52d7c861c01c4466803a44373bc11dc-1387384363.us-east-2.elb.amazonaws.com 80:31482/TCP 40s &#x3C;none> </code></pre> <h3>Deploy application microservices to private Kubernetes cluster</h3> <p>Similarly, create the namespace <em>boutique</em> in the private Kubernetes cluster running on HPE GreenLake for Private Cloud Enterprise and then deploy 3 <em>Deployment</em> resources to the namespace:</p> <pre><code class="language-markdown">$ kubectl create ns boutique $ kubectl config set-context --current --namespace boutique $ kubectl apply -f k8s-manifests-deploy-private.yaml deployment.apps/emailservice created deployment.apps/paymentservice created deployment.apps/shippingservice created </code></pre> <h3>Deploy Virtual Application Network</h3> <p>Define the Virtual Application Network using <em>Skupper</em> on both the public AWS EKS cluster and private Kubernetes cluster:</p> <h4>1. In the public AWS EKS cluster, deploy the <em>aws-public</em> application router.</h4> <pre><code class="language-markdown">$ kubectl config set-context --current –namespace boutique $ skupper init --site-name aws-public Waiting 115 seconds for LoadBalancer IP or hostname... Waiting 111 seconds for LoadBalancer IP or hostname... Waiting 108 seconds for LoadBalancer IP or hostname... Skupper is now installed in namespace 'boutique'. Use 'skupper status' to get more information. $ skupper status Skupper is enabled for namespace "boutique" with site name "aws-public" in interior mode. It is connected to 1 other site. It has 10 exposed services. The site console url is: https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8080 The credentials for internal console-auth mode are held in secret: 'skupper-console-users' </code></pre> <h4>2. In the private Kubernetes cluster, deploy the <em>pce-private</em> application router.</h4> <pre><code class="language-markdown">$ kubectl config set-context --current –namespace boutique $ skupper init --ingress none --site-name pce-private Skupper is now installed in namespace 'boutique'. Use 'skupper status' to get more information. $ skupper status Skupper is enabled for namespace "boutique" with site name "pce-private" in interior mode. It is not connected to any other sites. It has no exposed services </code></pre> <h4>3. In the public AWS EKS cluster, create a connection token for connection.</h4> <pre><code class="language-markdown">$ skupper token create ~/aws-public-token.yaml Token written to /home/guoping/aws-public-token.yaml </code></pre> <h4>4. In the private Kubernetes cluster, define the connections to the public AWS EKS cluster.</h4> <pre><code class="language-markdown">$ skupper link create ~/aws-public-token.yaml Site configured to link to https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8081/d2e35a8c-6654-11ed-bf10-000c295724b5 (name=link1) Check the status of the link using 'skupper link status'. $ skupper link status Links created from this site: ------------------------------- Link link1 is active Currently active links from other sites: ---------------------------------------- There are no active links $ skupper status Skupper is enabled for namespace "boutique" with site name "pce-private" in interior mode. It is connected to 1 other site. It has no exposed services. </code></pre> <h4>5. In the public AWS EKS cluster, verify connectivity has been established.</h4> <pre><code class="language-markdown">$ skupper status Skupper is enabled for namespace "aws-boutique" with site name "aws-public" in interior mode. It is connected to 1 other site. It has no exposed services. The site console url is: https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8080 The credentials for internal console-auth mode are held in secret: 'skupper-console-users' </code></pre> <p>From the <em>Skupper</em> console URL at <strong><a href="https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8080">https://aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com:8080</a></strong>, you can see the connections from the public AWS EKS cluster and the private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise:</p> <p><img src="/img/skupper-status.png" alt=""></p> <h3>Expose application microservices to Virtual Application Network</h3> <h4>1. In the private Kubernetes cluster, expose 3 services:</h4> <pre><code class="language-markdown">$ skupper expose deployment emailservice --address emailservice --port 5000 --protocol http2 --target-port 8080 deployment emailservice exposed as emailservice $ skupper expose deployment paymentservice --address paymentservice --port 50051 --protocol http2 --target-port 50051 deployment paymentservice exposed as paymentservice $ skupper expose deployment shippingservice --address shippingservice --port 50051 --protocol http2 --target-port 50051 deployment shippingservice exposed as shippingservice </code></pre> <h4>2. In the public AWS EKS cluster, expose 7 services:</h4> <pre><code class="language-markdown">$ skupper expose deployment productcatalogservice --address productcatalogservice --port 3550 --protocol http2 --target-port 3550 deployment productcatalogservice exposed as productcatalogservice $ skupper expose deployment recommendationservice --address recommendationservice --port 8080 --protocol http2 --target-port 8080 deployment recommendationservice exposed as recommendationservice $ skupper expose deployment checkoutservice --address checkoutservice --port 5050 --protocol http2 --target-port 5050 deployment checkoutservice exposed as checkoutservice $ skupper expose deployment cartservice --address cartservice --port 7070 --protocol http2 --target-port 7070 deployment cartservice exposed as cartservice $ skupper expose deployment currencyservice --address currencyservice --port 7000 --protocol http2 --target-port 7000 deployment currencyservice exposed as currencyservice $ skupper expose deployment adservice --address adservice --port 9555 --protocol http2 --target-port 9555 deployment adservice exposed as adservice $ skupper expose deployment redis-cart --address redis-cart --port 6379 --protocol tcp --target-port 6379 deployment redis-cart exposed as redis-cart </code></pre> <h3>Access Online Boutique application</h3> <p>From the <em>Skupper</em> console, you can see all the deployed services to the public AWS EKS cluster and the private Kubernetes cluster:</p> <p><img src="/img/skupper-apps.png" alt=""></p> <p>From the public AWS EKS cluster, check all the deployed services.</p> <pre><code class="language-markdown">$ kubectl get svc -n boutique NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE adservice ClusterIP 172.20.183.120 &#x3C;none> 9555/TCP 40d cartservice ClusterIP 172.20.255.202 &#x3C;none> 7070/TCP 40d checkoutservice ClusterIP 172.20.146.32 &#x3C;none> 5050/TCP 40d currencyservice ClusterIP 172.20.244.103 &#x3C;none> 7000/TCP 40d emailservice ClusterIP 172.20.136.4 &#x3C;none> 5000/TCP 28h frontend ClusterIP 172.20.103.129 &#x3C;none> 80/TCP 40d frontend-external LoadBalancer 172.20.16.223 a52d7c861c01c4466803a44373bc11dc-1387384363.us-east-2.elb.amazonaws.com 80:31482/TCP 40d paymentservice ClusterIP 172.20.244.25 &#x3C;none> 50051/TCP 28h productcatalogservice ClusterIP 172.20.147.163 &#x3C;none> 3550/TCP 40d recommendationservice ClusterIP 172.20.83.157 &#x3C;none> 8080/TCP 40d redis-cart ClusterIP 172.20.179.232 &#x3C;none> 6379/TCP 40d shippingservice ClusterIP 172.20.16.129 &#x3C;none> 50051/TCP 28h skupper LoadBalancer 172.20.111.44 aea867abf6fb6413d8f577652da564c1-130946084.us-east-2.elb.amazonaws.com 8080:31907/TCP,8081:30027/TCP 40d skupper-router LoadBalancer 172.20.182.70 acaedc6978d3b453b8555d6dead90943-1598691456.us-east-2.elb.amazonaws.com 55671:30272/TCP,45671:32499/TCP 40d skupper-router-local ClusterIP 172.20.175.145 &#x3C;none> 5671/TCP 40d skupper-router-local ClusterIP 172.20.249.51 &#x3C;none> 5671/TCP 35m </code></pre> <p>The <em>Online Boutique</em> application can be accessed from the assigned LoadBalancing host name <strong>a52d7c861c01c4466803a44373bc11dc-1387384363.us-east-2.elb.amazonaws.com</strong>:</p> <p><img src="/img/online-boutique-frontend.png" alt=""></p> <p>You can start shopping by adding items to the shopping cart, creating your shipping address and choosing the payment method. Please note that both the payment and the shipping services are running from the private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise.</p> <p><img src="/img/online-boutique-payment.png" alt=""></p> <p>You can then place an order to complete your shopping.</p> <p><img src="/img/online-boutique-order.png" alt=""></p> <h2>Next Steps</h2> <p>This blog post described the process of deploying the <em>Online Boutique</em> application as a hybrid app across both a public EKS cluster in AWS and a private Kubernetes cluster in HPE GreenLake for Private Cloud Enterprise environment.</p> <p>Running applications and services in this hybrid cloud environment is becoming increasingly popular as more businesses and enterprises shift toward cloud-based computing. This model can amplify the benefits of both private and public clouds and allows for more seamless integration across technical barriers.</p> <p>In <a href="https://developer.hpe.com/blog/monitor-application-performance-across-hybrid-cloud-environment-using-apache-skywalking-and-service-mesh/">my next blog post of the series</a>, I will show you how to install and set up the Apache SkyWalking application performance monitoring tool to monitor the deployed application in such a hybrid cloud environment as this. It helps to reduce management complexity and deliver operational insights for more informed business practices, and protect your most valuable user data.</p>https://developer.hpe.com/getting-started-with-the-hpe-greenlake-compute-ops-management-api-in-just-3mn/https://developer.hpe.com/getting-started-with-the-hpe-greenlake-compute-ops-management-api-in-just-3mn/Wed, 04 Jan 2023 09:57:13 GMT<style> ul li{ font-size:25px; } </style> <style> table { display: block; width: max-content !important; max-width: 100%; overflow: auto; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; } td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: left !important; font-weight: normal !important; padding: 10px !important; } thead tr:first-child td { -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none; border:1px solid grey; text-align: center !important; padding: 20px !important; font-weight: bold !important; } </style> <h2>Apply the power of a Developer Community</h2> <p>HPE GreenLake provides APIs for the different applications hosted on the HPE GreenLake Cloud Platform to do what can be done in the GUI through programmatic techniques. One of these applications is HPE GreenLake for Compute Ops Management (COM). HPE GreenLake for Compute Ops Management automates and transforms complex and time-consuming compute management operations into a simplified experience across edge-to-cloud. You can find its <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/">API documentation</a> on the <a href="https://developer.greenlake.hpe.com/">HPE GreenLake API portal</a>.</p> <p>However, if you are time constrained and prefer to just poke around instead of browsing through pages of documentation, this blog post is for you. One of the benefits of working within a community is the ability to take advantage of open collaboration, sharing hints, tools, and resources. In order to discover the API capabilities more rapidly, you can use a (free) tool, such as Postman, and leverage some of the Postman collections already created by members of the HPE Developer Community. If you are not yet familiar with Postman, no problem! In this post, I will provide you with step-by-step instructions.</p> <blockquote> <p>Note: You can follow these instructions with the Postman native application as well as the SaaS version of Postman. </p> </blockquote> <p>Ready? Let’s start the stopwatch, now!</p> <p><img src="/img/stopwatch1.jpg" alt=""></p> <h2>Step 1 [optional] - Get yourself a Postman account</h2> <p>If you don’t have a Postman account already, you can request a free one at: <a href="https://identity.getpostman.com/signup">https://identity.getpostman.com/signup</a></p> <h2>Step 2 - Log into Postman</h2> <p>Log into your Postman account. From the search bar, look for the collection “Compute Ops Management”  and select from the list, the one from our Community contributor, Lionel Jullien’s (<a href="mailto:lio@hpe.com">lio@hpe.com</a>) public workspace. </p> <blockquote> <p>Note: You can access it directly from <a href="https://www.postman.com/jullienl/workspace/lionel-jullien-s-public-workspace/collection/991177-a2b4838f-3e9d-4047-b02f-f813b73a6724?ctx=documentation">here</a>.</p> </blockquote> <h2>Step 3 - Fork the existing HPE GreenLake Compute Ops management collection</h2> <p>You can fork Lionel’s COM collection into your Postman workspace, which is very similar to a GitHub fork operation. This will copy the content of the collection to your workspace, and maintain a link to the source, which would allow you to get updates (if Lionel does update his collection), or contribute to Lionel’s collection by opening a Pull Request (PR) if you have made changes to the collection. After you forked his collection, you can work on your own copy in your Postman workspace.</p> <blockquote> <p>Note: You can also export the collection and import it into your own workspace.</p> </blockquote> <h2>Step 4 - Prepare your COM environment</h2> <p>The COM collection built by Lionel makes use of environment variables which can be grouped in a Postman environment. You will have to define and initialize these environment variables before using calls from the Postman collection. You can view Lionel’s Postman environment in his workspace, and either fork his environment or recreate it from scratch by creating a new environment in your workspace and setting up the following variables:</p> <ul> <li>ClientID</li> <li>ClientSecret</li> <li>url</li> </ul> <blockquote> <p>Note: Be careful, as Postman variables are case-sensitive. Also, you can set the type of variable ClientSecret to secret, so the value of the variable is only shown upon request.</p> </blockquote> <h2>Step 5 - Set your COM environment</h2> <p>You now need to initialize the 3 environment variables that you created in your Postman workspace. In order to do this you need to use the HPE GreenLake Cloud Platform Graphical User Interface (GUI). Please refer to <a href="https://developer.hpe.com/blog/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/">this blog post</a> to create an API Access in your own HPE GreenLake for Compute Ops Management instance. You do not need to generate an access token from the GUI, as you will do it via the API in the next step.</p> <p>From the GUI, make note of your Client ID, your Client Secret and the Connectivity Endpoint that corresponds to your region. Then, in your Postman environment, set a value to the 3 variables:</p> <table> <thead> <tr> <th>Variable name</th> <th>Initial Value</th> </tr> </thead> <tbody> <tr> <td>ClientID</td> <td>Your client ID</td> </tr> <tr> <td>ClientSecret</td> <td>Your client Secret</td> </tr> <tr> <td>url</td> <td>Your Connectivity Endpoint</td> </tr> </tbody> </table> <blockquote> <p>Note: Make sure that there isn’t any space or carriage return at the end of the variable values and to save your environment. </p> </blockquote> <h2>Step 6 - Obtaining a session token</h2> <p>Open your Postman workspace and locate the COM collection you have forked (or imported). Now, generate a session token using the <strong>Create session</strong> call from the <strong>0_session</strong> folder</p> <blockquote> <p>Note:  Before pressing Send, make sure that you have selected your environment in the upper right corner drop down. By default, it is set to <strong>No Environment</strong>.</p> </blockquote> <p>Once the call returns, make sure you have a Status of 200 OK and check the JSON response body:</p> <pre><code class="language-json">{ "access_token": "eyJhbGciOiJSUzI1NiIsImtpZCI6IlRFak8tZEJPbThxUDlqRUlxdVE5aXVKX09HTSIsInBpLmF0bSI6ImRlejAifQ.eyJjbGllbnRfaWQiOiJmYWE5ZDZjMi04MjdjLTQzMWYtYTI2My1kNGE1YzY5YjYwZjIiLCJpc3MiOiJodHRwczovL3Nzby5jb21tb24uY2xvdWQuaHBlLmNvbSIsImF1ZCI6ImV4dGVybmFsX2FwaSIsInN1YiI6ImNvbS5kZW1vdXNlckBnbWFpbC5jb20iLCJ1c2VyX2N0eCI6ImIwMDQxNjk0NDM5MjExZWM5MWQ2YzZhZGQyMDZhMjQ3IiwiYXV0aF9zb3VyY2UiOiJjY3NfdG9rZW5fbWFuYWdlbWVudCIsInBsYXRmb3JtX2N1c3RvbWVyX2lkIjoiMzQ2NTJmZjAzMTc3MTFlYzliYzA5Njg3MjU4MGZkNmQiLCJpYXQiOjE2NzEwOTM2ODUsImFwcGxpY2F0aW9uX2luc3RhbmNlX2lkIjoiNjM3ZjA5MzgtMTg4Mi00NzNiLTkyNDQtNGNkMDFkMWExNjY4IiwiZXhwIjoxNjcxMTAwODg1fQ.ajV-eV98TrdZPtQAIynOW_9zrF0HaZo_g8eBdoxjHldEvXlxyomcpT3ElI_Ke2AZAGKDDQB9zihy0bfplSgnMg1yBoH7r2Ih4PJXm-lprFQZPF9dApDvv39sPu-VJZU2RijCGPp5fDqzbcF-37zbCzdhihdYsAnQE3VcGd8xKRTkH8JZVe3Rg22_ndzZOqTR3SAeVcFGuI3PN3r2mJ5GyxlT8ckt_QsUHrxtYPEVwZnOwRFtrT7JB-Ht3vJB8wJJwXdyIadr64gunV3UusMjxMzYk4RpvdMjRuANhhBUAHxdA3Mmnq3MXtnSLu_hPGz2MUzoXO9IPMI9Csq3q5G8Rg", "token_type": "Bearer", "expires_in": 7199 } </code></pre> <p>This token is known as a Bearer token and it is valid for 7200 seconds (120 minutes or 2 hours). It will expire after that period and will need to be regenerated by running a Create session call again if you have exceeded that time.</p> <p>There is javascript code executed at the end of the call to automatically capture the value of the session token and store it in a new environment variable (called <strong>COM_Token</strong>) that is visible in your Postman environment. You can check this code in the Tests tab of the API call:</p> <pre><code class="language-js">const responseJson = pm.response.json(); // Set environment variables pm.environment.set("COM_Token", responseJson.access_token); // Display in console. To see the console, go to View / Show Postman Console console.log("COM Token:", pm.environment.get("COM_Token"));  // Display in Test Results pm.test("COM session token = " + pm.environment.get("COM_Token")); </code></pre> <p>This code allows you to run any subsequent call from the collection using that saved token (until it expires).</p> <h2>Step 7 - Collecting API versions</h2> <p>The COM API is an aggregation of multiple independent APIs. To help you find what the latest version of each component is, the collection provides a little “trick” in <strong>2- Get the resource API versions from the API reference</strong> from the <strong>0_session</strong> folder. Call this to populate API versions of each of the component, and before using any other calls from the collection:</p> <p><img src="/img/apicalls.png" alt=""></p> <blockquote> <p>Note: This is not using an API, but parsing the content of a documentation web page, so it might be subject to changes in the future.</p> </blockquote> <h2>Step 8 - Exploring the API</h2> <p>Now that you have a valid token and have identified the API endpoints, you are ready to explore the rest of the COM API using this handy collection. Pick one call from the <strong>Servers</strong> folder to <strong>List of all ProLiant DL325 Gen10 Plus</strong> and hit Send. You will get the following JSON response (provided you have some of these servers in your environment).</p> <pre><code class="language-json">{     "offset": 0,     "count": 1,     "total": 1,     "items": [         {             "id": "P18606-B21+CZJ11105MN",             "type": "compute-ops/server",             "platformFamily": "PROLIANT",             "resourceUri": "/compute-ops/v1beta2/servers/P18606-B21+CZJ11105MN",             "name": "Geneva-CIC-amd16",             "createdAt": "2022-11-07T09:42:20.767907+00:00",             "updatedAt": "2022-12-14T17:20:43.292718+00:00",             "generation": 27,             "hardware": {                 "serialNumber": "CZJ11105MN",                 "model": "ProLiant DL325 Gen10 Plus",                 "uuid": "36383150-3630-5A43-4A31-313130354D4E",                 "productId": "P18606-B21",                 "powerState": "ON",                 "indicatorLed": "LIT",                 "health": {                     "summary": "OK",                     "healthLED": "OK",                     "fans": "OK",                     "fanRedundancy": "REDUNDANT",                     "liquidCooling": "NOT_PRESENT",                     "liquidCoolingRedundancy": "NOT_PRESENT",                     "memory": "OK",                     "network": "OK",                     "powerSupplies": "OK",                     "powerSupplyRedundancy": "REDUNDANT",                     "processor": "OK",                     "storage": "OK",                     "temperature": "OK",                     "bios": "OK",                     "smartStorage": "OK"                 },                 "bmc": {                     "mac": "B4:7A:F1:B0:4C:44",                     "ip": "10.4.25.209",                     "hostname": "hdp-sto-amd16-ilo.hpintelco.org"                 }             },             "state": {                 "managed": true,                 "connected": true,                 "connectedModifiedAt": "2022-12-14T17:20:43.333272+00:00",                 "subscriptionState": "SUBSCRIBED",                 "subscriptionTier": "Enhanced",                 "subscriptionExpiresAt": "2027-04-21T17:55:00+00:00"             },             "firmwareInventory": [                 {                     "name": "iLO 5",                     "version": "2.70 May 16 2022",                     "deviceContext": "System Board"                 },                 {                     "name": "System ROM",                     "version": "A43 v2.56 (02/10/2022)",                     "deviceContext": "System Board"                 },                 {                     "name": "Intelligent Platform Abstraction Data",                     "version": "10.1.0 Build 37",                     "deviceContext": "System Board"                 },                 {                     "name": "System Programmable Logic Device",                     "version": "0x11",                     "deviceContext": "System Board"                 },                 {                     "name": "Power Management Controller Firmware",                     "version": "1.0.8",                     "deviceContext": "System Board"                 },                 {                     "name": "Power Supply Firmware",                     "version": "1.00",                     "deviceContext": "Bay 1"                 },                 {                     "name": "Power Supply Firmware",                     "version": "1.00",                     "deviceContext": "Bay 2"                 },                 {                     "name": "Redundant System ROM",                     "version": "A43 v2.54 (12/03/2021)",                     "deviceContext": "System Board"                 },                 {                     "name": "Intelligent Provisioning",                     "version": "3.52.37",                     "deviceContext": "System Board"                 },                 {                     "name": "Power Management Controller FW Bootloader",                     "version": "1.1",                     "deviceContext": "System Board"                 },                 {                     "name": "HPE Smart Storage Energy Pack 1 Firmware",                     "version": "0.70",                     "deviceContext": "Embedded Device"                 },                 {                     "name": "Mellanox Network Adapter - 88:E9:A4:02:8C:30",                     "version": "16.32.10.10",                     "deviceContext": "PCI-E Slot 1"                 },                 {                     "name": "HPE SN1600Q 32Gb 2p FC HBA",                     "version": "1.75.07",                     "deviceContext": "PCI-E Slot 2"                 },                 {                     "name": "HPE Smart Array P408i-a SR Gen10",                     "version": "5.00",                     "deviceContext": "Storage Slot 12"                 },                 {                     "name": "10/25Gb 2-port SFP28 BCM57414 OCP3 Adapter",                     "version": "219.0.144.0",                     "deviceContext": "OCP 3.0 Slot 10"                 },                 {                     "name": "Embedded Video Controller",                     "version": "2.5",                     "deviceContext": "Embedded Device"                 },                 {                     "name": "480GB 6G SATA SSD",                     "version": "HPG1",                     "deviceContext": "Slot=12:Port=1I:Box=1:Bay=1"                 },                 {                     "name": "480GB 6G SATA SSD",                     "version": "HPG1",                     "deviceContext": "Slot=12:Port=1I:Box=1:Bay=2"                 }             ],             "softwareInventory": \[],             "lastFirmwareUpdate": null,             "host": {                 "osName": "None",                 "osVersion": "None",                 "hostname": "Geneva-CIC-amd16",                 "osType": null,                 "osDescription": "None"             },             "firmwareBundleUri": null,             "tags": {},             "biosFamily": "A43",             "processorVendor": "AMD EPYC 7352 24-Core Processor                ",             "autoIloFwUpdate": true,             "serverGeneration": "GEN_10         }     ] } </code></pre> <p>That’s it. No more than 3 minutes!</p> <p><img src="/img/stopwatch2.jpg" alt=""></p> <p>If you have more time, feel free to explore the rest of the collection on your own. Also, don’t hesitate to provide Lionel with feedback on his very convenient collection.</p> <p>Any questions on HPE GreenLake for Compute Ops Management? Please join the HPE Developer Slack Workspace (<a href="https://slack.hpedev.io/">https://slack.hpedev.io/</a>) and start a discussion in our <a href="https://hpedev.slack.com/archives/C03QTQWC213">#hpe-greenlake-compute-ops-management </a>channel.</p>https://developer.hpe.com/2022-December-05/https://developer.hpe.com/2022-December-05/Mon, 05 Dec 2022 13:00:00 GMThttps://developer.hpe.com/deploying-mongodb-application-on-hpe-greenlake-for-containers/https://developer.hpe.com/deploying-mongodb-application-on-hpe-greenlake-for-containers/Thu, 01 Dec 2022 15:59:42 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Containers is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>In this blog post, we demonstrate how an end user can deploy a containerized, stateful MongoDB application on a Kubernetes-based container stack provided by HPE GreenLake for Containers and then access it over an external network or internet. In this scenario, the Kubernetes cluster gets configured using HPE CSI driver, along with the default storage class.</p> <h2>An overview of HPE GreenLake for Containers</h2> <p>HPE GreenLake for Containers provides a pay-as-you-go, Kubernetes-based container-optimized stack that is delivered as a service to help you operationalize your containers at scale.</p> <p>HPE GreenLake for Containers provides a standardized approach for cluster creation using cluster blueprints with user roles and role-based access controls as defined in the HPE GreenLake for Containers interface to the created clusters and applications. It also provides you with a dashboard view that displays the status of all Kubernetes services and resource utilization across all clusters.</p> <p>HPE manages your environment and will contact you before any upgrades are made to let you know the planned date and time, along with any pre-upgrade requirements.</p> <p>The HPE GreenLake for Containers service:</p> <ul> <li>Is an HPE-designed, implemented, owned, and operated private cloud that is deployed at a customer site</li> <li>Is offered as a consumption-based service that allows customers to better align costs to outcomes</li> <li>Supports Kubernetes on VMware vSphere</li> <li>Supports HPE Nimble and Alletra Storage arrays which provides persistent storage for containerized workloads</li> </ul> <p>You can launch HPE GreenLake for Containers using the HPE GreenLake for Private Cloud Enterprise card on the HPE GreenLake Central Dashboard. From the Private Cloud Enterprise main page, click Containers to create clusters and blueprints, view details about existing clusters, and launch the HPE Ezmeral Runtime Environment (Containers page).</p> <h2>HPE GreenLake for Containers: Machine blueprint layout for Kubernetes cluster node(s)</h2> <p>HPE GreenLake for Containers uses machine blueprints to define the infrastructure details for the worker nodes used in a cluster.</p> <p>Predefined blueprints are provided when the service is provisioned, and you can create your own custom machine blueprints.</p> <p>A machine blueprint includes the:</p> <ul> <li>Machine provider</li> <li>Operating system image and version</li> <li>Number of vCPU cores and amount of memory in the node</li> <li>Compute instance types</li> <li>Storage instance types</li> </ul> <p><img src="/img/image-1.png" alt=""></p> <p><img src="/img/image-2.png" alt=""></p> <p><img src="/img/image-3.png" alt=""></p> <h2>HPE GreenLake for Containers: Cluster blueprint layout for Kubernetes cluster</h2> <p>HPE GreenLake for Containers uses cluster blueprints to define the cluster layout and other infrastructure details used to create a cluster.</p> <p>Predefined blueprints are provided when the service is provisioned. You can copy and modify the predefined blueprints or create your own custom cluster blueprints.</p> <p>A cluster blueprint includes the:</p> <ul> <li>Cluster provider</li> <li>Version of Kubernetes to deploy on the cluster</li> <li>Default storage class</li> <li>Control plane nodes and worker nodes</li> </ul> <p><img src="/img/image-4.png" alt=""></p> <p><img src="/img/image-5.png" alt=""></p> <h2>HPE GreenLake for Containers: MongoDB application deployment on Kubernetes cluster</h2> <p>MongoDB is an open source, NoSQL database that provides support for JSON-styled, document-oriented storage systems. It supports a flexible data model that enables you to store data of any structure, and provides a rich set of features, including full index support, sharding, and replication.</p> <p>Below is the preferred cluster configuration for a MongoDB in-memory database workload.</p> <p><img src="/img/image-6.png" alt=""></p> <p><img src="/img/image-7.png" alt=""></p> <h3>Step-1: Create a Kubernetes cluster from containers page</h3> <p>To create a cluster, you must have been assigned the roles of <strong>Private Cloud Cluster Owner</strong> and <strong>Private Cloud Widget Viewer</strong>.</p> <ul> <li>From the Containers main page, under the Clusters tab, click Create Cluster.</li> <li>In the Create Cluster form, provide the cluster name 'hpe', and select the standard cluster blueprint. The new cluster appears in the list of clusters.</li> </ul> <p><img src="/img/image-8.png" alt=""></p> <p><img src="/img/image-9.png" alt=""></p> <p>As indicated above, there are multiple clusters deployed in parallel for multiple purposes. For the MongoDB application deployment in our example, the cluster will be created with the name "hpe".</p> <p><img src="/img/image-10.png" alt=""></p> <h3>Step-2: Download scoped kubeconfig from Container platform page</h3> <ul> <li>From the Clusters tab, select the 'hpe' Kubernetes cluster and click <strong>Launch Service Console</strong>. This will direct you to the container platform page.</li> <li>Click on Download kubeconfig.</li> </ul> <p><img src="/img/image-11.png" alt=""></p> <p><img src="/img/image-12.png" alt=""></p> <p>Note: Launching HPE Ezmeral Runtime Enterprise from HPE GreenLake Central is configured through SAML SSO and adds a session token to the kubeconfig file. You will need to download the kubeconfig file again if you want to continue to access the cluster when the session token expires after an hour.</p> <h3>Step-3: View the 'hpe' Kubernetes cluster environment details</h3> <p>Get Kubernetes <strong>cluster version.</strong></p> <pre><code class="language-shellsession">$ kubectl version --short Client Version: v1.20.0 Server Version: v1.20.11-hpe-2 </code></pre> <p>Get Kubernetes <strong>cluster nodes.</strong></p> <pre><code class="language-shellsession">$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-hpe-master-d6xv8-254v7.glhc-hpe.local Ready control-plane,master 71d v1.20.11-hpe-2 172.16.17.115 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 k8s-hpe-master-d6xv8-8fxxz.glhc-hpe.local Ready control-plane,master 71d v1.20.11-hpe-2 172.16.17.110 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 k8s-hpe-master-d6xv8-jjrpc.glhc-hpe.local Ready control-plane,master 71d v1.20.11-hpe-2 172.16.17.114 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 k8s-hpe-worker-qscr4-89n67.glhc-hpe.local Ready worker 71d v1.20.11-hpe-2 172.16.17.109 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 k8s-hpe-worker-qscr4-fp8px.glhc-hpe.local Ready worker 71d v1.20.11-hpe-2 172.16.17.116 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 k8s-hpe-worker-qscr4-l95j4.glhc-hpe.local Ready worker 71d v1.20.11-hpe-2 172.16.17.113 &#x3C;none> SUSE Linux Enterprise Server 15 SP2 5.3.18-150200.24.115-default containerd://1.5.1-hpe-1 </code></pre> <p>Get Kubernetes cluster <strong>default storage class.</strong></p> <pre><code class="language-shellsession">$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gl-sbc-glhcnimblestor (default) csi.hpe.com Delete Immediate true 69d gl-sbc-hpe csi.hpe.com Delete Immediate true 69d gl-sbp-glhcnimblestor csi.hpe.com Delete Immediate true 69d hpe-hdd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 69d hpe-nvme-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 69d hpe-ssd-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 69d </code></pre> <p>Get all resources available under <strong>hpe-storage</strong> namespace i.e. HPE CSI driver, Snapshot controller.</p> <pre><code class="language-shellsession">$ kubectl get all -n hpe-storage NAME READY STATUS RESTARTS AGE pod/hpe-csi-controller-7c6f876494-vrd49 9/9 Running 0 69d pod/hpe-csi-node-cpmkg 2/2 Running 0 69d pod/hpe-csi-node-m2f75 2/2 Running 0 69d pod/hpe-csi-node-m9mj9 2/2 Running 0 69d pod/nimble-csp-db7c7bb65-c5wrk 1/1 Running 0 69d pod/primera3par-csp-6f999b8d76-wtd4n 1/1 Running 0 69d pod/snapshot-controller-0 1/1 Running 0 69d pod/snapshot-controller-64b98b668f-rdfwr 1/1 Running 0 32d pod/snapshot-controller-64b98b668f-tdzz9 1/1 Running 0 32d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alletra6000-csp-svc ClusterIP 10.109.228.208 &#x3C;none> 8080/TCP 69d service/alletra9000-csp-svc ClusterIP 10.107.227.232 &#x3C;none> 8080/TCP 69d service/nimble-csp-svc ClusterIP 10.102.45.17 &#x3C;none> 8080/TCP 69d service/primera3par-csp-svc ClusterIP 10.104.57.79 &#x3C;none> 8080/TCP 69d NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/hpe-csi-node 3 3 3 3 3 &#x3C;none> 69d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hpe-csi-controller 1/1 1 1 69d deployment.apps/nimble-csp 1/1 1 1 69d deployment.apps/primera3par-csp 1/1 1 1 69d deployment.apps/snapshot-controller 2/2 2 2 32d NAME DESIRED CURRENT READY AGE replicaset.apps/hpe-csi-controller-7c6f876494 1 1 1 69d replicaset.apps/nimble-csp-db7c7bb65 1 1 1 69d replicaset.apps/primera3par-csp-6f999b8d76 1 1 1 69d replicaset.apps/snapshot-controller-64b98b668f 2 2 2 32d NAME READY AGE statefulset.apps/snapshot-controller 1/1 69d </code></pre> <h3>Step-4: Create a namespace on 'hpe' Kubernetes cluster for MongoDB deployment</h3> <p>Create namespace with name <strong>mongo.</strong></p> <pre><code class="language-shellsession">$ kubectl create ns mongo namespace/mongo created $ kubectl get ns NAME STATUS AGE argocd Active 71d default Active 71d ezctl Active 71d gatekeeper-system Active 71d hpe Active 71d hpe-externalclusterinfo Active 71d hpe-ldap Active 71d hpe-logzio Active 71d hpe-metering Active 71d hpe-nodesvc Active 71d hpe-operations Active 50d hpe-secure Active 71d hpe-security Active 71d hpe-snow Active 71d hpe-storage Active 71d hpe-system Active 71d hpe-templates-compute Active 71d hpecp Active 71d hpecp-bootstrap Active 71d hpecp-falco Active 71d kd-apps Active 71d kd-mlops Active 71d kube-node-lease Active 71d kube-public Active 71d kube-system Active 71d kubernetes-dashboard Active 71d mongo Active 6s opsramp-agent Active 71d </code></pre> <p>Switch to new namespace i.e. <strong>mongo</strong> in the current context.</p> <pre><code class="language-shellsession">$ kubectl config set-context --current --namespace=mongo Context "caas-dev-3-hpe-hpe-ashish-kumar@hpe.com" modified. $ kubectl config view --minify --output 'jsonpath={..namespace}'; echo mongo $ kubectl get pods No resources found in mongo namespace. </code></pre> <h3>Step-5: Deploy MongoDB</h3> <p>Deploy the MongoDB application using YAML file i.e. <strong>services/mongodb/install-mongo.yaml</strong> from <strong><a href="https://github.com/cxteamtrials/caas-trials-content">https://github.com/cxteamtrials/caas-trials-content</a></strong> location.</p> <pre><code class="language-shellsession">$ kubectl create -f install-mongo.yaml service/mongo created statefulset.apps/mongo created $ cat install-mongo.yaml --- apiVersion: v1 kind: Service metadata: name: mongo labels: name: mongo hpecp.hpe.com/hpecp-internal-gateway: "true" # Expose the service on ERE Gateway spec: ports: - protocol: TCP port: 27017 targetPort: 27017 selector: role: mongo --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mongo spec: selector: matchLabels: app: mongo serviceName: "mongo" # Number of initial MongoDB pods to deploy - default is 1 if not set replicas: 3 podManagementPolicy: Parallel template: metadata: labels: app: mongo spec: terminationGracePeriodSeconds: 10 containers: - name: mongo image: mongo env: - name: POD_IP_ADDRESS valueFrom: fieldRef: fieldPath: status.podIP command: - "mongod" - "--bind_ip_all" - "--replSet" - rs0 - "--oplogSize" - "128" ports: - containerPort: 27017 volumeMounts: - name: mongo-persistent-storage mountPath: /data/db - name: mongo-sidecar image: cvallance/mongo-k8s-sidecar env: - name: MONGO_SIDECAR_POD_LABELS value: "app=mongo" volumeClaimTemplates: - metadata: name: mongo-persistent-storage spec: accessModes: [ "ReadWriteOnce" ] storageClassName: gl-sbc-glhcnimblestor resources: requests: storage: 50Gi </code></pre> <h3>Step-6: Validate MongoDB deployment</h3> <p>Get MongoDB related Kubernetes resources like; pod, service, pvc and validate the deployment.</p> <pre><code class="language-shellsession">$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mongo-0 2/2 Running 0 18h 10.192.3.58 k8s-hpe-worker-qscr4-89n67.glhc-hpe.local &#x3C;none> &#x3C;none> mongo-1 2/2 Running 0 18h 10.192.3.59 k8s-hpe-worker-qscr4-89n67.glhc-hpe.local &#x3C;none> &#x3C;none> mongo-2 2/2 Running 0 18h 10.192.4.208 k8s-hpe-worker-qscr4-l95j4.glhc-hpe.local &#x3C;none> &#x3C;none> $ kubectl get all NAME READY STATUS RESTARTS AGE pod/mongo-0 2/2 Running 0 18h pod/mongo-1 2/2 Running 0 18h pod/mongo-2 2/2 Running 0 18h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mongo ClusterIP 10.96.180.195 &#x3C;none> 27017/TCP 18h NAME READY AGE statefulset.apps/mongo 3/3 18h $ kubectl get svc $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mongo ClusterIP 10.96.180.195 &#x3C;none> 27017/TCP 18h $ kubectl describe svc mongo Name: mongo Namespace: mongo Labels: hpecp.hpe.com/hpecp-internal-gateway=true name=mongo Annotations: &#x3C;none> Selector: role=mongo Type: ClusterIP IP Families: &#x3C;none> IP: 10.96.180.195 IPs: 10.96.180.195 Port: &#x3C;unset> 27017/TCP TargetPort: 27017/TCP Endpoints: &#x3C;none> Session Affinity: None Events: &#x3C;none> $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mongo-persistent-storage-mongo-0 Bound pvc-bc7faca6-6cd6-4796-acc1-864716c98f86 50Gi RWO gl-sbc-glhcnimblestor 18h mongo-persistent-storage-mongo-1 Bound pvc-1ef8ef09-c858-4862-989c-d246308518b4 50Gi RWO gl-sbc-glhcnimblestor 18h mongo-persistent-storage-mongo-2 Bound pvc-12044e23-8535-4b3e-bc02-a99819b9753b 50Gi RWO gl-sbc-glhcnimblestor 18h $ kubectl describe pvc mongo-persistent-storage-mongo-0 Name: mongo-persistent-storage-mongo-0 Namespace: mongo StorageClass: gl-sbc-glhcnimblestor Status: Bound Volume: pvc-bc7faca6-6cd6-4796-acc1-864716c98f86 Labels: app=mongo Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: csi.hpe.com Finalizers: [kubernetes.io/pvc-protection] Capacity: 50Gi Access Modes: RWO VolumeMode: Filesystem Used By: mongo-0 Events: &#x3C;none> $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-12044e23-8535-4b3e-bc02-a99819b9753b 50Gi RWO Delete Bound mongo/mongo-persistent-storage-mongo-2 gl-sbc-glhcnimblestor 19h pvc-193a0dbf-79ed-43da-864d-6dfc8673b502 1Gi RWO Delete Bound hpe-metering/prometheus-managed-kube-prometheus-st-prometheus-db-prometheus-managed-kube-prometheus-st-prometheus-0 gl-sbc-hpe 71d pvc-1ef8ef09-c858-4862-989c-d246308518b4 50Gi RWO Delete Bound mongo/mongo-persistent-storage-mongo-1 gl-sbc-glhcnimblestor 19h pvc-4dabe263-101d-4fcd-be49-6c6a7a97d51a 500Gi RWO Delete Bound ashish-mongo-deploy/nimblepod-500 gl-sbc-hpe 66d pvc-68265149-eaf7-4239-982d-917f34f704c4 100Gi RWO Delete Bound ashish-mongo-deploy/nimblepod-100 gl-sbc-hpe 66d pvc-76062c79-819d-4b6e-af6b-207df5e008ee 100Gi RWO Delete Bound ashish-mongo-deploy/ashish-mongo-persistent-storage-mongo-1 gl-sbc-hpe 66d pvc-86360549-ac45-4ab7-b01e-426c8d3b0d33 5Gi RWO Delete Bound ashish-hpe-mongo-deploy/ashish-test-pvc gl-sbc-hpe 2d20h pvc-8a565d78-a0f3-4db1-b577-c88699c3aa41 100Gi RWO Delete Bound ashish-mongo-deploy/ashish-mongo-persistent-storage-mongo-2 gl-sbc-hpe 66d pvc-96b24e79-9c05-4c6b-8fcf-754d70dad152 30Gi RWO Delete Bound ashish-hpe-mongo-deploy/ashish-test-pvc2 gl-sbc-hpe 47h pvc-a8e578e8-a213-4607-912e-f89da2203cfd 100Gi RWO Delete Bound ashish-mongo-deploy/ashish-mongo-persistent-storage-mongo-0 gl-sbc-hpe 66d pvc-aced3b34-1d46-427a-a315-a90bdb687561 2Gi RWO Delete Bound hpe-metering/metering-agent gl-sbc-hpe 51d pvc-b4f5173e-ad34-4811-8722-c5f1ced1cafd 200Gi RWO Delete Bound ashish-mongo-deploy/nimblepod-200 gl-sbc-hpe 66d pvc-bc7faca6-6cd6-4796-acc1-864716c98f86 50Gi RWO Delete Bound mongo/mongo-persistent-storage-mongo-0 gl-sbc-glhcnimblestor 19h pvc-e27f5ae0-0f02-4690-b24a-2c65f9c51b1f 1000Gi RWO Delete Bound ashish-mongo-deploy/nimblepod-1000 gl-sbc-hpe 66d pvc-e800beac-56c4-46df-8bf7-4374e311133e 30Gi RWO Delete Bound ashish-hpe-mongo-deploy/ashish-mongo-persistent-storage-mongo-2 gl-sbc-hpe 47h pvc-e8aed33b-dc99-41bc-b301-2e2d3ca81b8d 30Gi RWO Delete Bound ashish-hpe-mongo-deploy/ashish-mongo-persistent-storage-mongo-0 gl-sbc-hpe 47h pvc-ee3146e2-ad77-4418-93b8-a94cd30e63ad 10Gi RWO Delete Bound ashish-mongo-deploy/nimblepod-10 gl-sbc-hpe 66d pvc-eee0e074-3284-40f5-914e-88842899efc2 5Gi RWO Delete Bound ashish-mongo-deploy/ashish-test-pvc gl-sbc-hpe 69d pvc-f42472d1-f8f2-4ac4-8955-c37664d7cf8b 30Gi RWO Delete Bound ashish-hpe-mongo-deploy/ashish-mongo-persistent-storage-mongo-1 gl-sbc-hpe 47h $ kubectl describe pv pvc-bc7faca6-6cd6-4796-acc1-864716c98f86 Name: pvc-bc7faca6-6cd6-4796-acc1-864716c98f86 Labels: &#x3C;none> Annotations: pv.kubernetes.io/provisioned-by: csi.hpe.com Finalizers: [kubernetes.io/pv-protection external-attacher/csi-hpe-com] StorageClass: gl-sbc-glhcnimblestor Status: Bound Claim: mongo/mongo-persistent-storage-mongo-0 Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 50Gi Node Affinity: &#x3C;none> Message: Source: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: csi.hpe.com FSType: xfs VolumeHandle: 067ac428a6431c485b00000000000000000000044d ReadOnly: false VolumeAttributes: accessProtocol=iscsi allowOverrides=nfsResources,nfsNamespace cloneOf= createSnapshot=false dedupeEnabled=true description=hpe destroyOnDelete=true encrypted=false folder=caas-pvs fsType=xfs hostEncryption=false limitIops=-1 limitMbps=-1 performancePolicy=default pool=default storage.kubernetes.io/csiProvisionerIdentity=1663147353961-8081-csi.hpe.com syncOnDetach=false targetScope=volume thick=false volumeAccessMode=mount Events: &#x3C;none>  </code></pre> <h3>Step-7: Configure MongoDB primary and secondary replica</h3> <p>Get the IP address of each mongo pod.</p> <pre><code class="language-shellsession">$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mongo-0 2/2 Running 0 44m 10.192.3.58 k8s-hpe-worker-qscr4-89n67.glhc-hpe.local &#x3C;none> &#x3C;none> mongo-1 2/2 Running 0 44m 10.192.3.59 k8s-hpe-worker-qscr4-89n67.glhc-hpe.local &#x3C;none> &#x3C;none> mongo-2 2/2 Running 0 44m 10.192.4.208 k8s-hpe-worker-qscr4-l95j4.glhc-hpe.local &#x3C;none> &#x3C;none> </code></pre> <p>Expose mongo as a service.</p> <pre><code class="language-shellsession">kubectl expose pod/mongo-0 --type="NodePort" --port 27017 kubectl expose pod/mongo-1 --type="NodePort" --port 27017 kubectl expose pod/mongo-2 --type="NodePort" --port 27017 </code></pre> <p>Describe each mongo service and get the port details on which service has been exposed over ERE Gateway.</p> <pre><code class="language-shellsession">$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mongo ClusterIP 10.96.180.195 &#x3C;none> 27017/TCP 54m mongo-0 NodePort 10.97.222.217 &#x3C;none> 27017:30946/TCP 23m mongo-1 NodePort 10.98.166.198 &#x3C;none> 27017:32648/TCP 19m mongo-2 NodePort 10.101.219.63 &#x3C;none> 27017:31216/TCP 20m $ kubectl describe svc mongo-0 Name: mongo-0 Namespace: mongo Labels: app=mongo controller-revision-hash=mongo-7648bd99c8 hpecp.hpe.com/hpecp-internal-gateway=true statefulset.kubernetes.io/pod-name=mongo-0 Annotations: hpecp-internal-gateway/27017: epicgw.customer.hpe.net:10030 Selector: app=mongo,controller-revision-hash=mongo-7648bd99c8,statefulset.kubernetes.io/pod-name=mongo-0 Type: NodePort IP Families: &#x3C;none> IP: 10.97.222.217 IPs: 10.97.222.217 Port: &#x3C;unset> 27017/TCP TargetPort: 27017/TCP NodePort: &#x3C;unset> 30946/TCP Endpoints: 10.192.3.58:27017 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HpeCp 25m hpecp-agent Created HPECP K8S service $ kubectl describe svc mongo-1 Name: mongo-1 Namespace: mongo Labels: app=mongo controller-revision-hash=mongo-7648bd99c8 hpecp.hpe.com/hpecp-internal-gateway=true statefulset.kubernetes.io/pod-name=mongo-1 Annotations: hpecp-internal-gateway/27017: epicgw.customer.hpe.net:10031 Selector: app=mongo,controller-revision-hash=mongo-7648bd99c8,statefulset.kubernetes.io/pod-name=mongo-1 Type: NodePort IP Families: &#x3C;none> IP: 10.98.166.198 IPs: 10.98.166.198 Port: &#x3C;unset> 27017/TCP TargetPort: 27017/TCP NodePort: &#x3C;unset> 32648/TCP Endpoints: 10.192.3.59:27017 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HpeCp 22m hpecp-agent Created HPECP K8S service $ kubectl describe svc mongo-2 Name: mongo-2 Namespace: mongo Labels: app=mongo controller-revision-hash=mongo-7648bd99c8 hpecp.hpe.com/hpecp-internal-gateway=true statefulset.kubernetes.io/pod-name=mongo-2 Annotations: hpecp-internal-gateway/27017: epicgw.customer.hpe.net:10035 Selector: app=mongo,controller-revision-hash=mongo-7648bd99c8,statefulset.kubernetes.io/pod-name=mongo-2 Type: NodePort IP Families: &#x3C;none> IP: 10.101.219.63 IPs: 10.101.219.63 Port: &#x3C;unset> 27017/TCP TargetPort: 27017/TCP NodePort: &#x3C;unset> 31216/TCP Endpoints: 10.192.4.208:27017 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HpeCp 23m hpecp-agent Created HPECP K8S service </code></pre> <p>Install '<strong>mongosh</strong>' client locally for shell interaction with MongoDB. <strong>mongosh</strong> will be accessing the mongo cluster from outside the Kubernetes cluster. You can download package <strong>services/mongodb/mongosh-1.6.0-win32-x64.zip</strong> from <strong><a href="https://github.com/cxteamtrials/caas-trials-content">https://github.com/cxteamtrials/caas-trials-content</a></strong> location.</p> <p>Extract the package and set the <strong>mongosh</strong> client bin path.</p> <pre><code class="language-shellsession">$ export PATH=$PATH:/c/Ashish/mongosh-1.6.0-win32-x64/bin/ </code></pre> <p>Connect to MongoDB service over ERE Gateway through <strong>mongosh</strong> client.</p> <pre><code class="language-shellsession">$ mongosh --host epicgw.customer.hpe.net --port 10030 Current Mongosh Log ID: 637f7622aaa80bc199cfcb06 Connecting to: mongodb://epicgw.customer.hpe.net:10030/?directConnection=true&#x26;appName=mongosh+1.6.0 Using MongoDB: 6.0.3 Using Mongosh: 1.6.0 For mongosh info see: https://docs.mongodb.com/mongodb-shell/ ------ The server generated these startup warnings when booting 2022-11-24T12:48:39.726+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted 2022-11-24T12:48:39.727+00:00: You are running this process as the root user, which is not recommended 2022-11-24T12:48:39.728+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options] 2022-11-24T12:48:39.728+00:00: vm.max_map_count is too low ------ ------ Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc). The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product improvements and to suggest MongoDB products and deployment options to you. To enable free monitoring, run the following command: db.enableFreeMonitoring() To permanently disable this reminder, run the following command: db.disableFreeMonitoring() ------ test> </code></pre> <p>Initialize MongoDB replicaset.</p> <pre><code class="language-shellsession">test> rs.initiate() { info2: 'no configuration specified. Using a default configuration for the set', me: 'mongo-0:27017', ok: 1 } rs0 [direct: other] test> </code></pre> <p>Register mongo-0 pod as primary replica.</p> <pre><code class="language-shellsession">rs0 [direct: other] test> var cfg = rs.conf();cfg.members[0].host="10.192.3.58:27017";rs.reconfig(cfg) { ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669297890, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669297890, i: 1 }) } </code></pre> <p>Validate that mongo-0 pod is registered as primary replica.</p> <pre><code class="language-shellsession">rs0 [direct: other] test> rs.status() { set: 'rs0', date: ISODate("2022-11-24T13:52:34.600Z"), myState: 1, term: Long("1"), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long("2000"), majorityVoteCount: 1, writeMajorityCount: 1, votingMembersCount: 1, writableVotingMembersCount: 1, optimes: { lastCommittedOpTime: { ts: Timestamp({ t: 1669297950, i: 1 }), t: Long("1") }, lastCommittedWallTime: ISODate("2022-11-24T13:52:30.843Z"), readConcernMajorityOpTime: { ts: Timestamp({ t: 1669297950, i: 1 }), t: Long("1") }, appliedOpTime: { ts: Timestamp({ t: 1669297950, i: 1 }), t: Long("1") }, durableOpTime: { ts: Timestamp({ t: 1669297950, i: 1 }), t: Long("1") }, lastAppliedWallTime: ISODate("2022-11-24T13:52:30.843Z"), lastDurableWallTime: ISODate("2022-11-24T13:52:30.843Z") }, lastStableRecoveryTimestamp: Timestamp({ t: 1669297900, i: 1 }), electionCandidateMetrics: { lastElectionReason: 'electionTimeout', lastElectionDate: ISODate("2022-11-24T13:49:50.807Z"), electionTerm: Long("1"), lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, numVotesNeeded: 1, priorityAtElection: 1, electionTimeoutMillis: Long("10000"), newTermStartDate: ISODate("2022-11-24T13:49:50.829Z"), wMajorityWriteAvailabilityDate: ISODate("2022-11-24T13:49:50.840Z") }, members: [ { _id: 0, name: '10.192.3.58:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 3836, optime: { ts: Timestamp({ t: 1669297950, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T13:52:30.000Z"), lastAppliedWallTime: ISODate("2022-11-24T13:52:30.843Z"), lastDurableWallTime: ISODate("2022-11-24T13:52:30.843Z"), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp({ t: 1669297790, i: 2 }), electionDate: ISODate("2022-11-24T13:49:50.000Z"), configVersion: 2, configTerm: 1, self: true, lastHeartbeatMessage: '' } ], ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669297950, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669297950, i: 1 }) } rs0 [direct: primary] test> </code></pre> <p>Add mongo-1 pod as secondary replica.</p> <pre><code class="language-shellsession">rs0 [direct: primary] test> rs.add("10.192.3.59:27017") { ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669298574, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669298574, i: 1 }) } </code></pre> <p>Validate that mongo-1 pod is registered as secondary replica.</p> <pre><code class="language-shellsession">rs0 [direct: primary] test> rs.status() { set: 'rs0', date: ISODate("2022-11-24T14:03:52.376Z"), myState: 1, term: Long("1"), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long("2000"), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 2, writableVotingMembersCount: 2, optimes: { lastCommittedOpTime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, lastCommittedWallTime: ISODate("2022-11-24T14:03:50.861Z"), readConcernMajorityOpTime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, appliedOpTime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, durableOpTime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, lastAppliedWallTime: ISODate("2022-11-24T14:03:50.861Z"), lastDurableWallTime: ISODate("2022-11-24T14:03:50.861Z") }, lastStableRecoveryTimestamp: Timestamp({ t: 1669298619, i: 4 }), electionCandidateMetrics: { lastElectionReason: 'electionTimeout', lastElectionDate: ISODate("2022-11-24T13:49:50.807Z"), electionTerm: Long("1"), lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, numVotesNeeded: 1, priorityAtElection: 1, electionTimeoutMillis: Long("10000"), newTermStartDate: ISODate("2022-11-24T13:49:50.829Z"), wMajorityWriteAvailabilityDate: ISODate("2022-11-24T13:49:50.840Z") }, members: [ { _id: 0, name: '10.192.3.58:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 4514, optime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T14:03:50.000Z"), lastAppliedWallTime: ISODate("2022-11-24T14:03:50.861Z"), lastDurableWallTime: ISODate("2022-11-24T14:03:50.861Z"), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp({ t: 1669297790, i: 2 }), electionDate: ISODate("2022-11-24T13:49:50.000Z"), configVersion: 4, configTerm: 1, self: true, lastHeartbeatMessage: '' }, { _id: 1, name: '10.192.3.59:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 58, optime: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, optimeDurable: { ts: Timestamp({ t: 1669298630, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T14:03:50.000Z"), optimeDurableDate: ISODate("2022-11-24T14:03:50.000Z"), lastAppliedWallTime: ISODate("2022-11-24T14:03:50.861Z"), lastDurableWallTime: ISODate("2022-11-24T14:03:50.861Z"), lastHeartbeat: ISODate("2022-11-24T14:03:52.308Z"), lastHeartbeatRecv: ISODate("2022-11-24T14:03:50.818Z"), pingMs: Long("0"), lastHeartbeatMessage: '', syncSourceHost: '10.192.3.58:27017', syncSourceId: 0, infoMessage: '', configVersion: 4, configTerm: 1 } ], ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669298630, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669298630, i: 1 }) } rs0 [direct: primary] test> </code></pre> <p>Add mongo-2 pod as secondary replica.</p> <pre><code class="language-shellsession">rs0 [direct: primary] test> rs.add("10.192.4.208:27017") { ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669298829, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669298829, i: 1 }) } </code></pre> <p>Validate that mongo-2 pod is registered as secondary replica.</p> <pre><code class="language-shellsession">rs0 [direct: primary] test> rs.status() { set: 'rs0', date: ISODate("2022-11-24T14:08:05.339Z"), myState: 1, term: Long("1"), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long("2000"), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: { lastCommittedOpTime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, lastCommittedWallTime: ISODate("2022-11-24T14:08:00.869Z"), readConcernMajorityOpTime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, appliedOpTime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, durableOpTime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, lastAppliedWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastDurableWallTime: ISODate("2022-11-24T14:08:00.869Z") }, lastStableRecoveryTimestamp: Timestamp({ t: 1669298870, i: 1 }), electionCandidateMetrics: { lastElectionReason: 'electionTimeout', lastElectionDate: ISODate("2022-11-24T13:49:50.807Z"), electionTerm: Long("1"), lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1669297790, i: 1 }), t: Long("-1") }, numVotesNeeded: 1, priorityAtElection: 1, electionTimeoutMillis: Long("10000"), newTermStartDate: ISODate("2022-11-24T13:49:50.829Z"), wMajorityWriteAvailabilityDate: ISODate("2022-11-24T13:49:50.840Z") }, members: [ { _id: 0, name: '10.192.3.58:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 4767, optime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T14:08:00.000Z"), lastAppliedWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastDurableWallTime: ISODate("2022-11-24T14:08:00.869Z"), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp({ t: 1669297790, i: 2 }), electionDate: ISODate("2022-11-24T13:49:50.000Z"), configVersion: 6, configTerm: 1, self: true, lastHeartbeatMessage: '' }, { _id: 1, name: '10.192.3.59:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 311, optime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, optimeDurable: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T14:08:00.000Z"), optimeDurableDate: ISODate("2022-11-24T14:08:00.000Z"), lastAppliedWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastDurableWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastHeartbeat: ISODate("2022-11-24T14:08:05.090Z"), lastHeartbeatRecv: ISODate("2022-11-24T14:08:05.094Z"), pingMs: Long("0"), lastHeartbeatMessage: '', syncSourceHost: '10.192.3.58:27017', syncSourceId: 0, infoMessage: '', configVersion: 6, configTerm: 1 }, { _id: 2, name: '10.192.4.208:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 56, optime: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, optimeDurable: { ts: Timestamp({ t: 1669298880, i: 1 }), t: Long("1") }, optimeDate: ISODate("2022-11-24T14:08:00.000Z"), optimeDurableDate: ISODate("2022-11-24T14:08:00.000Z"), lastAppliedWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastDurableWallTime: ISODate("2022-11-24T14:08:00.869Z"), lastHeartbeat: ISODate("2022-11-24T14:08:05.104Z"), lastHeartbeatRecv: ISODate("2022-11-24T14:08:03.615Z"), pingMs: Long("0"), lastHeartbeatMessage: '', syncSourceHost: '10.192.3.59:27017', syncSourceId: 1, infoMessage: '', configVersion: 6, configTerm: 1 } ], ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1669298880, i: 1 }), signature: { hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0), keyId: Long("0") } }, operationTime: Timestamp({ t: 1669298880, i: 1 }) } rs0 [direct: primary] test> </code></pre> <h2>HPE GreenLake for Containers: Data Science with MongoDB deployed on a Kubernetes cluster</h2> <p>The below steps illustrate an interactive Jupyter notebook adding records and querying MongoDB deployed on the "hpe" Kubernetes cluster.</p> <p><img src="/img/image-13.png" alt=""></p> <p>Create a Database instance 'glcaasmongodemo' and sample collection 'glsamplecollection' and add document records in database.</p> <p><img src="/img/image-14.png" alt=""></p> <p>Add multiple documents to sample collection.</p> <p><img src="/img/image-15.png" alt=""></p> <p>Perform query operation on sample collection.</p> <p><img src="/img/image-16.png" alt=""></p> <h2>HPE GreenLake for Containers: Demo Summary</h2> <p>You can find the GitHub repository that hosts demo code <a href="https://github.com/cxteamtrials/caas-trials-content">here</a>.</p> <p>We hope that this blog post has provided you with enough information for you to get started deploying a containerized, stateful MongoDB application using HPE GreenLake for Containers. To view more articles and tutorials on the use of HPE GreenLake for Containers, refer to the <a href="https://developer.hpe.com/">HPE Developer Community blog</a>.</p>https://developer.hpe.com/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-python-and-powershell/https://developer.hpe.com/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-python-and-powershell/Thu, 01 Dec 2022 12:28:58 GMT<p>HPE GreenLake for Compute Ops Management automates and transforms complex and time-consuming compute management operations into a simplified experience across edge-to-cloud. Whether you are an IT OPS or a DEV OPS engineer, you know that automation is the key to success. And today’s automation relies heavily on APIs and how one can interact easily with them. So, let us show you how to leverage the API provided so you, too, can take advantage of what HPE GreenLake for Compute Ops Management can provide. In this blog post, we will cover how to do this using Python and PowerShell.</p> <p><img src="/img/greenlake-com-ops-api-curl1.png" alt="figure1"></p> <h1><strong>HPE GreenLake for Compute Ops Management REST API</strong></h1> <p>HPE GreenLake for Compute Ops Management provides a Restful API to customers who want to manage their devices programmatically or through a command line interface. The API enables customers to invoke operations or tasks such as list devices, see device details, device health, manage a device's firmware and much more.</p> <p>To learn more about HPE GreenLake for Compute Ops Management API methods and resources, you can refer to <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">the API reference documentation</a>.</p> <p>Here are some of the operations you can do with the HPE GreenLake for Compute Ops Management API:</p> <ul> <li>Obtain the list of servers in your account</li> <li>Obtain the list of firmware bundles</li> <li>Obtain the list of job templates</li> <li>Perform a firmware update</li> <li>Run a carbon emissions report</li> <li>Power on/off/restart a server</li> </ul> <p><img src="/img/greenlake-com-ops-api-pyt-pow9.png" alt="figure2"></p> <p>In a previous <a href="https://developer.hpe.com/blog/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-curl-and-postman">blog</a>, we covered some of basics of this API using cURL or Postman. In this post, we would like to share with you a few examples of REST API calls that can be made through simple Python and PowerShell code examples.</p> <p>The HPE GreenLake for Compute Ops Management REST API uses the OAuth 2.0 HPE GreenLake authentication flow, where a limited lifetime access token is provided in the header of each REST API request as the authorization bearer. The process to generate this necessary access token is well described in the blog post written by <strong>Nisha Thomas</strong>, entitled <a href="https://developer.hpe.com/blog/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/">How to use an API access token for HPE GreenLake for Compute Ops Management</a>. If you are not familiar with this token generation and usage,  we would strongly advise you to read it as it represents the very first and important steps to be performed prior to getting the chief benefits described above.</p> <ul> <li>Sign up for an HPE Account</li> <li>Connect to HPE GreenLake</li> <li>Create API Client Credentials</li> <li>Generate an Access Token based on your credentials and relevant connectivity endpoint (HPE GreenLake for Compute Ops Management in our case)</li> </ul> <p>Once the access token is generated, it can be used to make API calls to perform any HTTP method against the desired HPE GreenLake for Compute Ops Management resource. To do this, you simply add the access token to the HTTP header with the keyword "Authorization: Bearer {token}". The name “Bearer authentication” can be understood as “give access to the bearer of this token”.</p> <p>There are several ways to invoke the API:</p> <ul> <li>Using REST directly or from JavaScript</li> <li>Using cURL</li> <li>Using Python</li> <li>Using Go</li> <li>Using PowerShell</li> <li>Using Ansible, Terraform, etc.</li> </ul> <p>The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">HPE GreenLake for Compute Ops Management API Reference site</a> leverages OpenAPI conformant documentation that provides a complete explanation of the operations supported by the Unique Resource Identifiers <a href="">(URIs)</a>as well as sample requests and responses.</p> <h1>Using PowerShell</h1> <p>From a terminal application, start a PowerShell environment and set the variables that will be used later.</p> <pre><code class="language-powershell"># Store our Compute Ops Manager API endpoint $endpoint = "https://us-west2-api.compute.cloud.hpe.com" # Initialize an empty headers array $headers = @{} # Add a header for authorization, using the Access Token generated in task 2 $headers["Authorization"] = "Bearer_access_token_here" </code></pre> <p><img src="/img/greenlake-com-ops-api-pyt-pow1.png" alt="figure3"></p> <p>Our first request:</p> <pre><code class="language-powershell">$response = Invoke-webrequest "$endpoint/compute-ops/v1beta2/servers?limit=2" -Method GET -Headers $headers </code></pre> <p>When the request is successful, the response should display 200:</p> <p><img src="/img/greenlake-com-ops-api-pyt-pow2.png" alt="figure4"></p> <pre><code class="language-powershell">$response.StatusCode </code></pre> <p>The JSON data returned by the request is in $response.Content but it is rather hard to read as is, I can prettify it a bit with:</p> <pre><code class="language-powershell">$response.Content | ConvertFrom-Json | Format-List </code></pre> <p>Now, the request returned 2 objects (count : 2) and the actual server details are in the items array property.</p> <p><img src="/img/greenlake-com-ops-api-pyt-pow3.png" alt="figure5"></p> <p>The API supports several query parameters depending on the resource type, such as filter, limit (maximum number of records to return), offset (resource offset to start the response from) and sort (order in which to return the resources in the collection).</p> <p>Filters provide the ability to limit the resources that take part in the action of a REST call. When a REST call includes a filter, the GET or DELETE action is restricted to a response that meets the filter requirements. Filters are specified by using the query parameter 'filter'.</p> <p>A simple example of filtering follows:</p> <p><strong>GET <URI>?filter=powerState eq 'Off'</strong></p> <p>This example shows a simple filter. The resources returned by the query are limited to results with the attribute powerState and the value Off.</p> <p>To use a filter on a nested property name, the '/' separator can be specified as follows:</p> <p><strong>GET /compute-ops/v1beta2/servers?filter=hardware/model eq 'ProLiant DL365 Gen10 Plus'</strong></p> <p>To test this filter, I run the following command:</p> <pre><code class="language-powershell">$response = Invoke-webrequest "$endpoint/compute-ops/v1beta2/servers?filter=contains(hardware/model,'DL365')" -Method GET -Headers $headers </code></pre> <p>I start by checking the request was successful as before:</p> <pre><code class="language-powershell">$response.StatusCode </code></pre> <p>It should display 200.</p> <p>Then I parse the result with the following command:</p> <pre><code class="language-powershell">$response.Content | ConvertFrom-Json | select count </code></pre> <p>it should display 4, the number of servers with 'DL365' in the hardware model in my environment.</p> <p>Now I can look at the actual content:</p> <p><strong>($response.Content | ConvertFrom-Json).items</strong></p> <p><img src="/img/greenlake-com-ops-api-pyt-pow4.png" alt="figure6"></p> <p>Refer to the following <a href="https://developer.hpe.com/blog/">blog</a> for more information on filter query syntax and API references.</p> <h1>Using Python</h1> <p>From a Python 3 environment: At the Python interpreter prompt >>>, I import 2 modules to make HTTP requests and to work with JSON data</p> <pre><code class="language-python">import requests import json </code></pre> <p>I need to set some variables that I will use later:</p> <pre><code class="language-python"># Store our Compute Ops Manager API endpoint endpoint = "https://us-west2-api.compute.cloud.hpe.com" # Add a header for authorization, using the Access Token generated in task 2 headers = {"Authorization": "Bearer " + "paste_access_token_here"} </code></pre> <p><img src="/img/greenlake-com-ops-api-pyt-pow5.png" alt="figure8"></p> <p>I can now send my first request:</p> <pre><code class="language-python">response = requests.get(url=endpoint + '/compute-ops/v1beta2/servers?limit=2', headers=headers) </code></pre> <p>Once again, I check if the request was successful, the response status code should display 200:</p> <pre><code class="language-python">print(response.status_code) </code></pre> <p>In order to display the JSON data returned by the request in a human-readable form:</p> <pre><code class="language-python">print(json.dumps(response.json(), indent=2)) </code></pre> <p><img src="/img/greenlake-com-ops-api-pyt-pow6.png" alt="figure 9"></p> <p>I see the request returned 2 objects <strong>(count : 2)</strong> and the actual server details are in the <strong>items</strong> array property.</p> <p>The API supports several query parameters depending on the resource type, such as filter, limit (maximum number of records to return), offset (resource offset to start the response from) and sort (order in which to return the resources in the collection).</p> <p>Filters provide the ability to limit the resources that take part in the action of a REST call. When a REST call includes a filter, the GET or DELETE action is restricted to a response that meets the filter requirements. Filters are specified by using the query parameter 'filter'.</p> <p>A simple example of filtering follows:</p> <p><strong>GET <URI>?filter=powerState eq 'Off'</strong></p> <p>This example shows a simple filter. The resources returned by the query are limited to results with the attribute powerState and the value Off.</p> <p>To use a filter on a nested property name, the '/' separator can be specified as follows:</p> <p><strong>GET /compute-ops/v1beta2/servers?filter=hardware/model eq 'ProLiant DL365 Gen10 Plus'</strong></p> <p>I can test with the following command:</p> <pre><code class="language-python">response = requests.get(url= endpoint + "/compute-ops/v1beta2/servers?filter=contains(hardware/model,'DL365')", headers=headers) </code></pre> <p>First check if the request was successful as before</p> <pre><code class="language-python">print(response.status_code) </code></pre> <p>It should display 200.</p> <p>Now, display the number of objects in the response.</p> <pre><code class="language-python">print(response.json()\['count']) </code></pre> <p>It should display 4.</p> <p><img src="/img/greenlake-com-ops-api-pyt-pow7.png" alt="figure 10"></p> <p>To display the JSON data returned by the request in a human-readable form, type the following:</p> <pre><code class="language-python">print(json.dumps(response.json(), indent=2)) </code></pre> <p><img src="/img/greenlake-com-ops-api-pyt-pow8.png" alt="figure 11"></p> <p>Refer to previous sections for more information on filter query syntax and API references.</p> <h2>What’s next?</h2> <p>In this blog post, we covered how to get started with HPE GreenLake for Compute Ops Management REST API and explained how to post simple API calls through Python and PowerShell. In some of our future <a href="https://developer.hpe.com/hackshack/workshops">Workshops-on-Demand</a>, you will have the opportunity to run live experiments on these examples. Stay tuned for more details!</p> <p><a href="">Learn more about HPE GreenLake</a></p> <p>Learn more about the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/guide">HPE GreenLake for Compute Ops Management REST API</a></p> <p>This <a href="https://github.com/jullienl/HPE-Compute-Ops-Management">GitHub repository</a> hosts many script samples for the Compute Ops Management API including PowerShell, Python, Ansible playbooks and others.</p> <p>Find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog">HPE Developer blog</a>.</p>https://developer.hpe.com/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-curl-and-postman/https://developer.hpe.com/explore-the-hpe-greenlake-for-compute-ops-management-rest-api-using-curl-and-postman/Wed, 30 Nov 2022 12:36:05 GMT<p>HPE GreenLake for Compute Ops Management automates and transforms complex and time-consuming compute management operations into a simplified experience across edge-to-cloud. Whether you are an IT OPS or a DEV OPS engineer, you know that automation is the key to success. And today’s automation relies heavily on APIs and how one can interact easily with them. So, let us show you how to leverage the API provided so you, too, can take advantage of what HPE GreenLake for Compute Ops Management can provide. In this blog post, we will cover how to do this using cURL and Postman.</p> <p><img src="/img/greenlake-com-ops-api-curl1.png" alt="blog figure1"></p> <h1>HPE GreenLake for Compute Ops Management REST API</h1> <p>HPE GreenLake for Compute Ops Management provides a Restful API to customers who want to manage their devices programmatically or through a command line interface. The API enables customers to invoke operations or tasks such as list devices, see device details, device health, manage a device's firmware and much more.</p> <p>To learn more about HPE GreenLake for Compute Ops Management API methods and resources, you can refer to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API reference documentation</a>.</p> <p>Here are some of the operations you can do with the HPE GreenLake for Compute Ops Management API:</p> <ul> <li>Obtain the list of servers in your account</li> <li>Obtain the list of firmware bundles</li> <li>Obtain the list of job templates</li> <li>Perform a firmware update</li> <li>Run a carbon emissions report</li> <li>Power on/off/restart a server</li> </ul> <p><img src="/img/greenlake-com-ops-api-curl24.png" alt="blog figure1"></p> <p>We would like to share with you today a few examples of REST API calls that can be made through simple <a href="https://curl.se/">cURL</a> commands or <a href="https://www.postman.com/">Postman</a> examples.</p> <p>The HPE GreenLake for Compute Ops Management REST API uses the OAuth 2.0 HPE GreenLake authentication flow, where a limited lifetime access token is provided in the header of each REST API request as the authorization bearer. The process to generate this necessary access token is well described in the blog post written by <strong>Nisha Thomas</strong>, entitled <a href="https://developer.hpe.com/blog/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/">How to use an API access token for HPE GreenLake for Compute Ops Management</a>. If you are not familiar with this token generation and usage,  we would strongly advise you to read it as it represents the very first and important steps to be performed prior to getting the chief benefits described above.</p> <ul> <li>Sign up for an HPE Account</li> <li>Connect to HPE GreenLake</li> <li>Create API Client Credentials</li> <li>Generate an Access Token based on your credentials and relevant connectivity endpoint (HPE GreenLake for Compute Ops Management in our case)</li> </ul> <p>Once the access token is generated, it can be used to make API calls to perform any HTTP method against the desired HPE GreenLake for Compute Ops Management resource. To do this, you simply add the access token to the HTTP header with the keyword "Authorization: Bearer {token}". The name “Bearer authentication” can be understood as “give access to the bearer of this token”.</p> <p>There are several ways to invoke the API:</p> <ul> <li>Using REST directly or from JavaScript</li> <li>Using cURL</li> <li>Using Python</li> <li>Using Go</li> <li>Using PowerShell</li> <li>Using Ansible, Terraform, etc.</li> </ul> <p>The <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">HPE GreenLake for Compute Ops Management API Reference site</a> leverages OpenAPI conformant documentation that provides a complete explanation of the operations supported by the Unique Resource Identifiers <a href="">(URIs)</a>as well as sample requests and responses.</p> <p>If you are wondering which to use, cURL is probably a good choice if you like command line interfaces, Postman if you prefer graphical user interfaces, and PowerShell or Python if you are really into programming.</p> <h1>Using cURL</h1> <p>From the command prompt, use a simple cURL command like:</p> <pre><code class="language-shell">curl -X GET https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?limit=2 -H "Authorization:Bearer ,&#x3C; access_token_here>”** </code></pre> <p>Note that you must use the correct connectivity endpoint according to the region where HPE GreenLake for Compute Ops Management is deployed. Currently, these are the connectivity endpoints for the possible HPE GreenLake for Compute Ops Management regions:</p> <ul> <li><a href="https://eu-central1-api.compute.cloud.hpe.com">EU Central</a>  </li> <li><a href="https://eu-central1-api.compute.cloud.hpe.com">AP Northeast</a></li> <li><a href="https://us-west2-api.compute.cloud.hpe.com">US West</a></li> </ul> <p>The request uses the <strong>GET</strong> method for the <strong>/compute-ops/v1beta2/servers</strong> resource to obtain the list of the available servers as described in the API reference.</p> <p><img src="/img/greenlake-com-ops-api-curl25.png" alt="blog figure3"></p> <p>The command uses a header parameter with keyword <strong>"Authorization:Bearer {token}</strong>"</p> <p><strong>?limit=2</strong> at the end of the URL is a query parameter to limit the response to a maximum of two servers as documented in the request pane of the <em>servers</em> resource:</p> <p><img src="/img/greenlake-com-ops-api-curl19.png" alt="blog figure4"></p> <p>As a result of the command, the API response lists two compute servers that are on-boarded and assigned to the corresponding application for the user's account.</p> <p><img src="/img/greenlake-com-ops-api-curl15.png" alt="blog figure4"></p> <p>To see the API response code, add -I at the end of the command:</p> <pre><code class="language-shell">curl -X GET https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?limit=2 -H "Authorization:Bearer &#x3C;your access_token_here>” –I </code></pre> <p><img src="/img/greenlake-com-ops-api-curl2.png" alt="blog figure5"></p> <p>The response code displays <strong>200 OK.</strong></p> <p>This status response code indicates that the request has succeeded. The complete list of response codes for each resource in the API reference is shown below:</p> <p><img src="/img/greenlake-com-ops-api-curl26.png" alt="blog figure6"></p> <p>The JSON response provided by the API is not formatted, which makes it difficult to read and understand the JSON content, but a tool like <a href="https://stedolan.github.io/jq/">jq</a> can prettify the content. Just add “| jq” at the end of the command to get a better visual display.</p> <p><img src="/img/greenlake-com-ops-api-curl4.png" alt="blog figure7"></p> <p>The API supports several query parameters depending on the resource type, such as filter, limit (maximum number of records to return), offset (resource offset to start the response from) and sort (order in which to return the resources in the collection).</p> <p>Filters provide the ability to limit the resources that take part in the action of a REST call. When a REST call includes a filter, the GET or DELETE action is restricted to a response that meets the filter requirements. Filters are specified by using the query parameter 'filter'.</p> <p>Here is a simple example of filtering:</p> <p><strong>GET <URI>?filter=powerState eq 'Off'</strong></p> <p>This example shows a simple filter. The resources returned by the query are limited to results with the attribute <strong>powerState</strong> and the value <strong>Off</strong>.</p> <p>To use a filter on a nested property name, the '<strong>/</strong>' separator can be specified as follows:</p> <p><strong>GET /compute-ops/v1beta2/servers?filter=hardware/model eq 'ProLiant DL365 Gen10 Plus'**</strong><br> {**</p> <p>    <strong>"offset": 0,</strong></p> <p>    <strong>"count": 20,</strong></p> <p>    <strong>"total": 20,</strong></p> <p>    <strong>"items": [</strong></p> <p>        <strong>{</strong></p> <p>            <strong>"id": "P39368-B21+CN70421C51",</strong></p> <p>            <strong>"type": "compute-ops/server",</strong></p> <p>            <strong>"platformFamily": "PROLIANT",</strong></p> <p>            <strong>"resourceUri": "/compute-ops/v1beta2/servers/P39368-B21+CN70421C51",</strong></p> <p>            <strong>"name": "HPE-HOL56",</strong></p> <p>            <strong>"createdAt": "2022-04-29T12:35:35.265978+00:00",</strong></p> <p>            <strong>"updatedAt": "2022-10-25T19:54:36.572565+00:00",</strong></p> <p>            <strong>"generation": 292,</strong></p> <p>            <strong>"hardware": {</strong></p> <p>                <strong>"serialNumber": "CN70421C51",</strong></p> <p>                <strong>"model": "ProLiant DL365 Gen10 Plus",</strong></p> <p>                <strong>"uuid": "33393350-3836-4E43-3730-343231433531",</strong></p> <p>                <strong>"productId": "P39368-B21",</strong></p> <p>                <strong>"powerState": "ON",</strong></p> <p>                <strong>"indicatorLed": "OFF",</strong></p> <p>                <strong>"health": {</strong></p> <p>                  **  "summary": "OK",**</p> <p>                  **  "healthLED": "OK",**</p> <p>                  **  "fans": "OK",**</p> <p>                  **  "fanRedundancy": "REDUNDANT",**</p> <p>                  **  "liquidCooling": "NOT_PRESENT",**</p> <p>                  **  "liquidCoolingRedundancy": "NOT_PRESENT",**</p> <p>                  **  "memory": "OK",**</p> <p>                  **  "network": "UNKNOWN",**</p> <p>                  **  "powerSupplies": "OK",**</p> <p>                  **  "powerSupplyRedundancy": "NOT_PRESENT",**</p> <p>                  **  "processor": "OK",**</p> <p>                  **  "storage": "OK",**</p> <p>                  **  "temperature": "OK",**</p> <p>                  **  "bios": "OK",**</p> <p>                  **  "smartStorage": "OK"**</p> <p>                <strong>},</strong></p> <p>                <strong>"bmc": {</strong></p> <p>                  **  "mac": "B4:7A:F1:4E:9E:92",**</p> <p>                  **  "ip": "172.30.231.116",**</p> <p>                  **  "hostname": "None"**</p> <p>The following cURL command includes the filter</p> <pre><code class="language-shell">curl -X GET “https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?filter=contains(hardware/model,'DL365')” -H "Authorization:Bearer &#x3C;your access_token_here>" </code></pre> <p><img src="/img/greenlake-com-ops-api-curl5.png" alt="blog figure8"></p> <p>Contains is another filter option to return resources where associated <strong>hardware/model</strong> contains <strong>DL365.</strong> The result of the command shows 4 server resources whose model name contains DL365.</p> <p>The filter query syntax supports a richer set of filters than the single operation in the previous example. Filtering syntax is broken down by Operations, Logic, and Types. In the previous example, <a href=""></a>the operation was 'eq' for equality. Most comparison operations require the evaluated property name to the left of the operator and a literal to the right.</p> <p><img src="/img/greenlake-com-ops-api-curl16.png" alt="blog figure9"></p> <p>To learn more about Filtering, see <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/guide/#filtering">the HPE GreenLake for Compute Ops Management guide.</a></p> <p>The HPE GreenLake for Compute Ops Management API supports many resources that are described in the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API reference</a> document, as illustrated below:</p> <p><img src="/img/greenlake-com-ops-api-curl17.png" alt="blog figure10"></p> <p>Unique Resource Identifiers <a href="">(URIs)</a>  are used to identify a resource. A URI is a full API path ending in an identification number.</p> <p>For example:</p> <ul> <li>compute-ops/v1beta2/servers/{serverId}</li> <li>compute-ops/v1beta1/reports/{id}/data</li> </ul> <p><strong>v1beta1</strong>, <strong>v1beta2</strong> in the URI is the version of the resource that is being accessed.</p> <p>One can invoke the common HTTP <a href=""></a>methods, like GET, POST, PUT, PATCH, and DELETE, on resources in the HPE GreenLake for Compute Ops Management API as shown below for the <strong>filters</strong> resource:</p> <p><img src="/img/greenlake-com-ops-api-curl23.png" alt="blog figure11"></p> <p>Refer to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API reference</a> site for a complete list of <a href=""></a>methods supported by each API resource.</p> <h1>Using Postman</h1> <p>Postman is a tool designed to build and use APIs.</p> <p>To get started, “create a request”, as shown below:</p> <p><img src="/img/greenlake-com-ops-api-curl7.png" alt="blog figure12"></p> <p>In the request URL field, enter the endpoint URL: </p> <p><a href=""></a><strong><a href="https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?limit=2">https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?limit=2</a></strong></p> <p>This is the base connectivity endpoint you have seen in earlier, followed by the resource URI you want to query <strong>/compute-ops/v1beta2/servers</strong> as described in the API reference table.</p> <p><img src="/img/greenlake-com-ops-api-curl25.png" alt="blog figure13"></p> <p>In order to limit the output as documented in the request pane of the <em>servers</em> resource:</p> <p>We use query parameter <strong>?limit=2</strong></p> <p><img src="/img/greenlake-com-ops-api-curl19.png" alt="blog figure14"></p> <p><img src="/img/greenlake-com-ops-api-curl20.png" alt="blog figure15"></p> <p>In the <em>Authorization</em> tab, choose  <strong>Bearer Token</strong> in the <em>Type</em> drop-down menu and paste the access token that was generated earlier in the Token field. Postman will generate the appropriate Authorization header when you send the request.</p> <p><img src="/img/greenlake-com-ops-api-curl8.png" alt="blog figure16"></p> <p>Hit the <strong>Send</strong> button to get a <strong>200 OK</strong> status response indicating success and a JSON body with the details of two compute servers on-boarded and assigned to the corresponding application for the user's account.</p> <p><img src="/img/greenlake-com-ops-api-curl9.png" alt="blog figure17"></p> <p>The complete list of response codes is documented for each resource in the API reference table shown below:</p> <p><img src="/img/greenlake-com-ops-api-curl21.png" alt="blog figure18"></p> <p>The API supports several query parameters depending on the resource type, such as filter, limit (maximum number of records to return), offset (resource offset to start the response from) and sort (order in which to return the resources in the collection).</p> <p>Filters provide the ability to limit the resources that take part in the action of a REST call. When a REST call includes a filter, the GET or DELETE action is restricted to a response that meets the filter requirements. Filters are specified by using the query parameter 'filter'.</p> <p>A simple example of filtering follows:</p> <p><strong>GET <URI>?filter=powerState eq 'Off'</strong></p> <p>This example shows a simple filter. The resources returned by the query are limited to results with the attribute <strong>powerState</strong> and the value <strong>Off</strong>.</p> <p>To use a filter on a nested property name, the '<strong>/</strong>' separator can be specified as follows:</p> <p><strong>GET /compute-ops/v1beta2/servers?filter=hardware/model eq 'ProLiant DL365 Gen10 Plus'</strong></p> <p>To test this filter, we enter the following URL in the Request field:</p> <p><strong><a href="https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?filter=contains(hardware/model,&#x27;DL365&#x27;)">https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers?filter=contains(hardware/model,'DL365')</a></strong></p> <p><img src="/img/greenlake-com-ops-api-curl22.png" alt="blog figure19"></p> <p>The Authorization tab should still have your Access Token (an access token is valid for 2 hours).</p> <p>We hit the <strong>Send</strong> button.</p> <p>The request should indicate success (Status is 200 OK) and, in this example, the response shows 4 server resources whose model name contains DL365.</p> <p><img src="/img/greenlake-com-ops-api-curl10.png" alt="blog figure20"></p> <p>The filter query syntax supports a richer set of filters than the single operation in the previous example. Filtering syntax is broken down by Operations, Logic, and Types.</p> <p>In the previous example<a href="">,</a>  the operation was 'eq' for equality. Most comparison operations require the evaluated property name to the left of the operator<a href=""></a> and a literal to the right.</p> <p><img src="/img/greenlake-com-ops-api-curl16.png" alt="blog figure21"></p> <p>To learn more about Filtering, see the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/guide/#filtering">HPE GreenLake for Compute Ops Management guide</a>.</p> <p>Unique Resource Identifiers (URIs) are used to identify a resource. A URI is a full API path ending in an identification number.</p> <p>For example:</p> <ul> <li>compute-ops/v1beta2/servers/{serverId}</li> <li>compute-ops/v1beta1/reports/{id}/data</li> </ul> <p><strong>v1beta1</strong>, <strong>v1beta2</strong> in the URI is the version of the resource that is being accessed.</p> <p>You can invoke the common HTTP methods, like GET, POST, PUT, PATCH, and DELETE,on resources in the HPE GreenLake for Compute Ops Management API as shown below for the <strong>filters</strong> resource:</p> <p>Refer to the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API reference</a> site for a complete list of methods supported by each API resource.</p> <p>Finally, a Postman collection of executable API requests for the HPE GreenLake for Compute Ops Management API can be found <a href="https://www.postman.com/jullienl/workspace/lionel-jullien-s-public-workspace/collection/991177-10c5377d-892b-4612-9e81-23d75d6c2f0d?ctx=documentation">HERE</a>.</p> <h1>What’s next?</h1> <p>In this blog post, we covered how to get started with the HPE GreenLake for Compute Ops Management REST API, explained how to post simple API calls through cURL commands, and showed you how to leverage Postman to achieve the same results.  In our next article, we will show how similar calls can be performed using Python or PowerShell.</p> <p>Learn more about <a href="https://www.hpe.com/hpe/greenlake">HPE GreenLake</a></p> <p>Learn more about the <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/guide">HPE GreenLake for Compute Ops Management REST API</a></p> <p>Find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog">HPE Developer blog</a>.</p>https://developer.hpe.com/example-of-mapreduce-with-ezmeral-data-fabric-mapr-database-binary-table/https://developer.hpe.com/example-of-mapreduce-with-ezmeral-data-fabric-mapr-database-binary-table/Mon, 14 Nov 2022 15:56:03 GMT<h2>Introduction</h2> <p>No matter what kind of application you want to develop, the underlying layer will need a database. Nowadays, applications based on big data are generally divided into two types: BI (Business Intelligence) and AI (Artificial Intelligence). Naturally, both of these two types of applications require a big data system as a base. In traditional applications, we use RDBMS as the database, but in big data systems, we need to use NoSQL databases.</p> <h3>Advantages of using HPE Ezmeral Data Fabric database</h3> <p>Let's first look at the position of the HPE Ezmeral Data Fabric database in the HPE Ezmeral Data Fabric software stack. Since the bottom layer of HPE Ezmeral Data Fabric database is the file system, you get the benefits of better performance, simpler management, and ease of use.</p> <p><img src="/img/system_architecture_position-hpe-edf_database.png" alt="HPE EDF Database is based on file system" title="Position of the database in HPE EDF stack"></p> <p>In my personal experience, I have found this to be true. For example, when using open source Apache Hadoop, due to its design principles, in order to fully realize the performance of Hadoop, it is necessary to merge small files before putting them in. With HPE Ezmeral Data Fabric file system, it's not necessary to care about whether small files need to be merged due to the existence of the logical unit volume within the file system. Another advantage of using HPE Ezmeral Data Fabric file system is that it provides the widely used protocol interface - NFS - allowing you to mount the HPE Ezmeral Data Fabric file system as an NFS file system on your PC. That is to say, you can mount the HPE Ezmeral Data Fabric file system as an NFS file system on your PC. This is something that Hadoop and other peer commercial software cannot do.</p> <p>I have some additional thoughts on important unique advantages of HPE Ezmeral Data Fabric database. The first thing that comes to my mind is, the simplicity of the product.</p> <p>For example, if you are using products in the Apache Hadoop ecosystem, or a commercial version of a big data system like Cloudera, you need to install and maintain the NoSQL service included in it separately. One often comes into contact with HBase, Mongo, DB, etc. But in HPE Ezmeral Data Fabric, you don't need to deploy HBase and MongoDB separately, because these two different types of NoSQL systems have been integrated in HPE Ezmeral Data Fabric core as HPE Ezmeral Data Fabric database. From the process level, you only see one MFS process.</p> <p>When you are using HBase, you need to consider the HBase Master and Region Server processes, as well as the underlying Hadoop Namenode and Datanode processes. HPE Ezmeral Data Fabric database includes two different types of NoSQL database systems, namely: Binary Table and JSON Table, which correspond to open source HBase and MongoDB respectively. Now, only one software process can be seen, which is MFS. When you use a completely open source big data technology stack or other commercial big data platforms, you will still see a bunch of processes. So you can see one of the biggest differences: simplicity.</p> <p>Though a column-oriented NoSQL database, such as HBase, is a bit outdated in design when compared with a document-oriented NoSQL database, such as MongoDB, I did not have enough information on a better replacement in order to write this article. I did find a demo of Spark based on HBase in the HPE Developer portal entitled <a href="https://developer.hpe.com/blog/spark-streaming-with-hbase/">Spark Streaming with HBase</a> that I was able to use.</p> <p>As stated, the advantages of using HPE Ezmeral Data Fabric database focus around simplicity. The advantages of the HPE Ezmeral Data Fabric file system include improved performance, ease of use, and ease of maintenance. Now let me show you how to use them with a Binary Table and MapReduce to apply these advantages when dealing with huge data volumes in a distributed storage and computing system. As for why we use this kind of NoSQL products, I don’t think I need to go into details here. This is the same as why Hadoop, a big data file system, was born. Simply put, it is because we need to build a distributed storage and computing system. In order to complete the analysis and computation tasks of huge data volumes on cheap commercial computers. In addition, the main reason for me to write this article is: I did not find a demo article in the HPE Developer portal that introduces the use of Binary Table and MapReduce together, so I would like to add such an example.</p> <h2>Getting started using MapReduce with HPE Ezmeral Data Fabric</h2> <p>In this article, I will show you how to create the Development Environment for HPE Ezmeral Data Fabric on Linux, Microsoft Windows and Apple Mac in a single-node cluster based on Docker containers. It will provide you with a choice of different HPE Ezmeral Data Fabric versions, based on how it integrates into HPE Ezmeral Ecosystem Packs. This way you can quickly create an HPE Ezmeral Data Fabric environment on your work computer. Then, using this as an example, I'll demonstrate a MapReduce application that uses HPE Ezmeral Data Fabric's database binary table as the backend service. I will create the table using the HBase shell command line tool customized for the HPE Ezmeral Data Fabric database and do aggregation operations using a MapReduce application. Let's get started!</p> <h3>Prerequisite: Create a Development Environment for HPE Ezmeral Data Fabric</h3> <p>Before you get started, you'll need to set up a Development Environment for HPE Ezmeral Data Fabric. To get some hints on how to do this, refer to the 👉 <a href="https://developer.hpe.com/blog/getting-started-with-spark-on-mapr-sandbox/">Getting Started with Spark on MapR Sandbox</a> blog post. I recommend that you read the latest official documentation, 👉 <a href="https://docs.datafabric.hpe.com/70/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">Development Environment for HPE Ezmeral Data Fabric</a>, before setting up your environment.</p> <p>All you really need to do is follow the instructions in the documentation. Be aware that the documentation will instruct you to install Docker Desktop on a Mac, but you don't really have to install Docker Desktop. It's fine to install the Docker Engine using a standard Linux distribution.</p> <p><strong>It's worth noting that installing Docker Desktop in Windows won't work.</strong> I tried getting it to work through the following actions: I first installed WSL2 (Windows Subsystem Linux 2) then installed Docker Desktop for Windows and integrated with WSL2, and then ran the HPE Ezmeral Data Fabric Development Environment install script, but it still failed.</p> <p>So I ended up installing VMWare on my Windows PC, creating a CentOS8 VM, and ran the HPE Ezmeral Data Fabric Development Environment setup script in the VM. This approach proved feasible. Also, you can always choose the version of the development environment you want to deploy. You just need to change the tag of the Docker image.</p> <p>After running the setup script, you'll have an HPE Ezmeral Data Fabric cluster up and running. But in order to run a MapReduce application, you need to install the YARN framework first. Starting from release of HPE Ezmeral Data Fabric 6.2.0, the YARN framework decoupled from the core platform and exists as an ecosystem component as part of HPE Ezmeral Ecosystem Packs. So if you are running a Development Environment newer than 6.2.0 (including 6.2.0), you need to complete the steps found in <a href="https://docs.datafabric.hpe.com/70/AdvancedInstallation/InstallingHadoop.html">Installing Hadoop and YARN</a>.</p> <h2>Using MapReduce on an HPE Ezmeral Data Fabric database binary table</h2> <p>HPE Ezmeral Data Fabric database binary table is equivalent to the HPE Ezmeral Data Fabric version of Apache HBase, but its technical implementation is different from HBase. This is, of course, because the bottom layer of HPE Ezmeral Data Fabric database is the HPE Ezmeral Data Fabric File Store. For users, there is almost no difference between using HPE Ezmeral Data Fabric database binary table and using HBase.</p> <p>Now, let's imagine that we want to build a user notifications service. Since HBase does not support any operations that go across rows or across tables, in order to implement operations such as <strong>joins</strong> and <strong>group by</strong> in RDBMS, we will use MapReduce to complete some data analysis tasks.</p> <h3>Create a binary table</h3> <p><strong>Note</strong>: <em>The commands in this article are all executed as the user "mapr", which is by default the admin user of the Development Environment for HPE Ezmeral Data Fabric. You can also use the "root" user to create the table and run the application, but if you don't modify the ACEs of the table, the "mapr" user would be not able to see the data in the table.</em></p> <p>We are going to use the <a href="https://docs.datafabric.hpe.com/70/ReferenceGuide/HBaseShellforMapR-DB.html">hbase shell</a> to create a <a href="https://docs.datafabric.hpe.com/70/MapR-DB/intro-binary-tables.html">binary table</a> inside the HPE Ezmeral Data Fabric database. To be able to use the <code>hbase shell</code>, you will first need to install the <strong><a href="https://docs.datafabric.hpe.com/70/AdvancedInstallation/InstallingHBase-client-node.html?hl=mapr-hbase">mapr-hbase</a></strong> package.</p> <p>For data management convenience, I'll show you how to create a volume for the binary table. The commands are as follows:</p> <pre><code class="language-bash">sudo -u mapr maprcli volume list -output terse -columns volumename,volumetype,actualreplication,localpath,mounted,mountdir,logicalUsed,used,nameContainerDataThresholdMB,nameContainerSizeMB,needsGfsck maprcli volume create -name test.binarytable1 -path /testbinarytable1volume # sudo -u mapr maprcli volume mount -name test.binarytable1 -path /testbinarytable1volume sudo -u mapr hadoop fs -ls -d -h /testbinarytable1volume sudo -u mapr hadoop mfs -ls /testbinarytable1volume </code></pre> <p>☝ The volume's name is <em>test.binarytable1</em> and it will be mounted as <ins>/testbinarytable1volume/</ins> in the HPE Ezmeral Data Fabric file system.</p> <p>On any node where the mapr-hbase package is installed, enter the command: "hbase shell" to enter the HBase shell interface.</p> <p>👇 Execute the following command on the HBase shell interface:</p> <pre><code>create '/testbinarytable1volume/notifications','attributes','metrics' </code></pre> <p>In this example, the table name is <ins>/testbinarytable1volume/notifications</ins> and the "attributes" and "metrics" are column families.</p> <p><strong>Note</strong>: In HPE Ezmeral Data Fabric database binary table, the table name is by default a path in the file system. You can change the style of the table name to be like something that would be found in Apache HBase. Refer to 👉 <a href="https://docs.datafabric.hpe.com/70/UpgradeGuide/.MappingTableNamespace-HBase-DBbinary_2.html">Mapping to HBase Table Namespaces</a> for details.</p> <h3>Build and run the MapReduce application</h3> <h4>MapReduce application source code</h4> <p>For your convenience, I have put the code on Github: <a href="https://github.com/aruruka/Example-MapReduce-With-EzmeralDataFabricMapR-DatabaseBinaryTable">Example-MapReduce-With-EzmeralDataFabricMapR-DatabaseBinaryTable</a>. You can download it and compile it using Visual Studio Code. This MapReduce application is simple. It follows this logic:</p> <ol> <li>Get data from the source table: <ins>/testbinarytable1volume/notifications</ins>.</li> <li>Output aggregated data to the target table: <ins>/testbinarytable1volume/summary</ins>.</li> </ol> <p>This is basically a variation of "Word Count". This MapReduce application simply aggregates the number of rows which contains a column called "type" in the "attributes" column family. For example, there may be comment type, promotion type, friend-request type, etc. in this table. Then the app will count how many rows of comment type, promotion type, friend-request type are there.</p> <h4>How to run it on HPE Ezmeral Data Fabric</h4> <p>When ready, you can run the MapReduce application on your one-node cluster of the Development Environment for HPE Ezmeral Data Fabric. Refer to the following commands:</p> <p>First, use this command to create a new table for storing the counter number:</p> <p>👇 Execute the following command on the HBase shell interface:</p> <pre><code>create '/testbinarytable1volume/summary','metrics' </code></pre> <p>Then, run the MapReduce application via <code>yarn jar</code> command:</p> <pre><code class="language-bash">sudo -u mapr \ yarn jar ./target/original-hbase-example-1.0-SNAPSHOT.jar com.shouneng.learn.mapReduce.Main \ -libjar ./hbase-server-1.4.13.200-eep-810.jar,hbase-client-1.4.13.200-eep-810.jar,hadoop-common-2.7.6.200-eep-810.jar,hbase-common-1.4.13.203-eep-810.jar </code></pre> <p>You should see the following output:</p> <pre><code class="language-markdown">yarn jar ./target/original-hbase-example-1.0-SNAPSHOT.jar com.shouneng.learn.mapReduce.Main WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.hbase.util.UnsafeAvailChecker (file:/opt/mapr/hadoop/hadoop-2.7.6/share/hadoop/common/hbase-common-1.4.13.200-eep-810.jar) to method java.nio.Bits.unaligned() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hbase.util.UnsafeAvailChecker WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 22/09/18 13:09:58 WARN mapreduce.TableMapReduceUtil: The addDependencyJars(Configuration, Class&#x3C;?>...) method has been deprecated since it is easy to use incorrectly. Most users should rely on addDependencyJars(Job) instead. See HBASE-8386 for more details. 22/09/18 13:09:58 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 22/09/18 13:09:58 INFO mapreduce.TableMapReduceUtil: Configured mapr.hbase.default.db maprdb 22/09/18 13:09:59 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 22/09/18 13:09:59 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 22/09/18 13:09:59 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to m2-maprts-vm99-173.mip.storage.hpecorp.net/10.163.173.99:8032 22/09/18 13:09:59 INFO client.ConnectionFactory: mapr.hbase.default.db unsetDB is neither MapRDB or HBase, set HBASE_MAPR mode since mapr client is installed. 22/09/18 13:09:59 INFO client.ConnectionFactory: ConnectionFactory receives mapr.hbase.default.db(unsetDB), set clusterType(HBASE_MAPR), user(mapr), hbase_admin_connect_at_construction(false) 22/09/18 13:09:59 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 22/09/18 13:10:00 INFO client.ConnectionFactory: mapr.hbase.default.db unsetDB is neither MapRDB or HBase, set HBASE_MAPR mode since mapr client is installed. 22/09/18 13:10:00 INFO client.ConnectionFactory: ConnectionFactory receives mapr.hbase.default.db(unsetDB), set clusterType(HBASE_MAPR), user(mapr), hbase_admin_connect_at_construction(false) 22/09/18 13:10:00 INFO util.RegionSizeCalculator: Region size calculation disabled for MapR tables /testbinarytable1volume/notifications 22/09/18 13:10:00 INFO mapreduce.JobSubmitter: number of splits:1 22/09/18 13:10:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1657816483829_0017 22/09/18 13:10:00 INFO mapreduce.JobSubmitter: Executing with tokens: [] 22/09/18 13:10:01 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - org.apache.hadoop.yarn.security.MapRTicketManager 22/09/18 13:10:01 INFO impl.YarnClientImpl: Submitted application application_1657816483829_0017 22/09/18 13:10:01 INFO mapreduce.Job: The url to track the job: http://m2-maprts-vm99-173.mip.storage.hpecorp.net:8088/proxy/application_1657816483829_0017/ 22/09/18 13:10:01 INFO mapreduce.Job: Running job: job_1657816483829_0017 22/09/18 13:10:09 INFO mapreduce.Job: Job job_1657816483829_0017 running in uber mode : false 22/09/18 13:10:09 INFO mapreduce.Job: map 0% reduce 0% 22/09/18 13:10:17 INFO mapreduce.Job: map 100% reduce 0% 22/09/18 13:10:22 INFO mapreduce.Job: map 100% reduce 100% 22/09/18 13:10:22 INFO mapreduce.Job: Job job_1657816483829_0017 completed successfully 22/09/18 13:10:22 INFO mapreduce.Job: Counters: 59 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=282992 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 MAPRFS: Number of bytes read=347 MAPRFS: Number of bytes written=170 MAPRFS: Number of read operations=38 MAPRFS: Number of large read operations=0 MAPRFS: Number of write operations=42 Job Counters Launched map tasks=1 Launched reduce tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5632 Total time spent by all reduces in occupied slots (ms)=8865 Total time spent by all map tasks (ms)=5632 Total time spent by all reduce tasks (ms)=2955 Total vcore-seconds taken by all map tasks=5632 Total vcore-seconds taken by all reduce tasks=2955 Total megabyte-seconds taken by all map tasks=5767168 Total megabyte-seconds taken by all reduce tasks=9077760 DISK_MILLIS_MAPS=2816 DISK_MILLIS_REDUCES=3930 Map-Reduce Framework Map input records=4 Map output records=4 Map output bytes=55 Map output materialized bytes=0 Input split bytes=181 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=65 Reduce input records=4 Reduce output records=2 Spilled Records=8 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=209 CPU time spent (ms)=4600 Physical memory (bytes) snapshot=1240059904 Virtual memory (bytes) snapshot=9576202240 Total committed heap usage (bytes)=1421869056 HBase Counters BYTES_IN_REMOTE_RESULTS=0 BYTES_IN_RESULTS=0 MILLIS_BETWEEN_NEXTS=0 NOT_SERVING_REGION_EXCEPTION=0 NUM_SCANNER_RESTARTS=0 NUM_SCAN_RESULTS_STALE=0 REGIONS_SCANNED=0 REMOTE_RPC_CALLS=0 REMOTE_RPC_RETRIES=0 ROWS_FILTERED=0 ROWS_SCANNED=0 RPC_CALLS=0 RPC_RETRIES=0 Shuffle Errors IO_ERROR=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 </code></pre> <p><strong>Note</strong>: If you encountered an issue indicating that some of the HBase related Java packages are missing, you can simply copy the following packages from 📁<ins>/opt/mapr/hbase/hbase-{VERSION}/lib/</ins> to 📁<ins>/opt/mapr/hadoop/hadoop-{VERSION}/share/hadoop/common/</ins>.</p> <ul> <li>hbase-client-*.jar</li> <li>hbase-server-*.jar</li> <li>hbase-protocol-*.jar</li> <li>hbase-hadoop2-compat-*.jar</li> <li>hbase-hadoop-compat-*.jar</li> <li>hbase-metrics-*.jar</li> <li>hbase-metrics-api-*.jar</li> <li>hbase-shaded-gson-*.jar</li> <li>hbase-shaded-htrace-*.jar</li> <li>metrics-core-*.jar</li> <li>hbase-common-*.jar</li> </ul> <p>You may run into this issue because the HBase related packages are considered as third-party libraries to the Hadoop system. For more information, refer to👉: <a href="https://docs.datafabric.hpe.com/70/DevelopmentGuide/Manage3rdPartyLibsForMapReduce.html">Install the third-party libraries on each node that runs the program</a>.</p> <h2>Glossary</h2> <details> <summary>HPE Ezmeral Data Fabric (aka. MapR)</summary> <p>HPE Ezmeral Data Fabric is a platform for data-driven analytics, ML, and AI workloads. The platform serves as a secure data store and provides file storage, NoSQL databases, object storage, and event streams. The patented filesystem architecture was designed and built for performance, reliability, and scalability. 📖<a href="https://docs.datafabric.hpe.com/70/index.html">Documentation website</a></p> </details> <details> <summary>HPE Ezmeral Ecosystem Packs</summary> <p>This is a software collection package that includes computing scheduling frameworks and computing engines of common Hadoop ecosystems such as YARN, Spark, Drill, and Hive, as well as a service suite for monitoring performance indicators and logs of HPE Ezmeral Data Fabric. Users of HPE Ezmeral Data Fabric will use these customized Spark, Drill, Hive and other software to complete computing, analysis tasks and machine learning tasks.</p> </details> <h2>Summary</h2> <p>In this blog post, I introduced the advantages of HPE Ezmeral Data Fabric database compared to similar products. I also demonstrated a MapReduce application based on HPE Ezmeral Data Fabric database binary table. HPE Ezmeral Data Fabric database binary table is equivalent to a better alternative to Apache HBase.</p> <p>I hope this blog post can help you quickly develop applications based on HPE Ezmeral Data Fabric database binary table. Please note that developing an application based on HPE Ezmeral Data Fabric database binary table is essentially the same as developing an application based on Apache HBase, but some of the specific steps are different. For example, in the step of creating a binary table, in addition to the method of using the <code>habse shell</code> demonstrated in this article, you can still use <code>maprcli</code>, a command line tool proprietary to HPE Ezmeral Data Fabric. Moreover, the experience of developing a MapReduce application based on HPE Ezmeral Data Fabric database binary table is also applicable to other computing engines, such as Spark.</p> <p>Stay tuned to the <a href="https://developer.hpe.com/blog">HPE Developer blog</a> for more interesting posts and tutorials on HPE Ezmeral Data Fabric.</p>https://developer.hpe.com/how-citizen-developers-can-build-award-winning-enterprise-apps-with-low-code-development-platforms/https://developer.hpe.com/how-citizen-developers-can-build-award-winning-enterprise-apps-with-low-code-development-platforms/Fri, 11 Nov 2022 13:58:12 GMT<p>I was not hired to be a developer. My background is in mechanical engineering and patent law and I joined Hewlett Packard Enterprise to help manage its large patent portfolio.  Sure, I took a couple of classes on coding in college, but who doesn’t these days? With all the focus around STEM in schools, a new generation of talent is learning how to build web and mobile apps, just as previous generations learned how to construct imaginative structures with Lego.</p> <p>It has never been easier for individuals or small teams to build apps through the use of low code platforms like Microsoft Power Apps and Power Automate. I was first introduced to these platforms a few years ago, and with the power they provided me, I am now proud to call myself a “Citizen Developer.” These tools enabled me to lead a small, self-organized cross-department team to create Idea Matchmaker, a mobile app that allows team members to easily connect and self-organize around interesting projects—a sort of “dating app” for ideas .</p> <h3>A match-making app for ideas</h3> <p>As a world-wide technology company, Hewlett Packard Enterprise (HPE) houses a lot of talent. Ideas can crop up at anytime from anywhere. But ideas need more than simple recognition to come to fruition – they need resources with the appropriate skill sets to make them a reality. As you can imagine, in such a large company, it’s not always easy to build the right team to carry out your bold ideas.</p> <p>A few years ago, I was part of a team within HPE’s patent department tasked with figuring out an easy way to gather feedback from the company’s vast technical community. This would allow the department to better protect its key inventions with patents and give team members a stronger voice in the company’s innovation strategy. We built a custom message board for each invention and were happy to discover that, in addition to commenting on the merit of the invention itself, team members would often suggest improvements and offer to collaborate. It was then that we realized how powerful a crowdsourcing platform designed to connect people could be.</p> <p>This inspired us to create Idea Matchmaker, an app that allows HPE team members to easily post and find small projects from other team members that they would like to work on. It’s based on the simple design and user experience for dating and friend-finding apps like Bumble and Tinder, but adapted for HPE project matchmaking. The app works on both mobile and desktop platforms and includes a recommendation engine that leverages a bit of artificial intelligence to assist with matchmaking. With Idea Matchmaker, HPE team members can easily pass or connect on projects based on their interest level. The app began as a submission to an HPE company-wide competition, called Innovation Quest, and won first place in the User Experience category. Winning the Innovation Quest garnered the app a lot of attention and it’s now being deployed company wide.</p> <h3>How we did it: the power of Power Apps</h3> <p>Building an application outside of the traditional software development streams within a company can be somewhat intimidating. There are a lot of challenges, like setting up cloud-based automation, dealing with security issues, implementing access controls, and navigating data compliance issues. Any “enterprise” app worth building will likely include sensitive or confidential data, which will limit how you can store and access your data outside of your company’s network. It’s not like you can just create a GoDaddy website and put your company’s sensitive information on it. Fortunately, with Microsoft Power Apps, everything can live within your existing Office 365 environment, which will allow you to easily build a secure and compliant end-to-end solution.</p> <p>Microsoft Power Apps authenticates using your existing Office 365 credentials, so you don’t need to worry about setting up your own Single Sign-On (SSO) workflow. You can also easily connect to data stored in your existing SharePoint sites, and can create surprisingly robust data tables using SharePoint lists. This allows you to rapidly iterate on your ideas without worrying about the time and effort required to secure and spend a budget on database infrastructure. If your data happens to be stored in other databases, like a SQL server or MySQL database, you can connect to that, too (with a premium connector license). This allows you to easily build proof-of-concept apps to validate solutions before investing the big bucks in scaling things up.</p> <p>The Microsoft Power Apps development environment is accessible via your browser, so you don’t need to install any desktop applications. And you won’t need to set up a Virtual Machine to virtualize your development environment. You can just open the site and start building your app, which I liken to a combination of creating PowerPoint slides, using formulas in Excel, with the bonus of being able to initiate events outside of the app from the app itself (e.g., click a button in the app to send an e-mail, create a document, or pull data from a database).</p> <p>Using low-code development platforms means you can focus on things like screen layouts and components of your app rather than dwelling on code syntax. For example, you can easily change the color of a button in your app by selecting from a list of colors on a side panel. You don’t have to navigate to the relevant section of the code and try to remember whether the color parameter is called “bgcolor” or “background-color”.</p> <p>Logic in a low-code app like Microsoft Power Apps feels similar to adding a formula to a cell in Excel, with simple if/then logic to define basic app logic (e.g., “If X, the button should be green, if not the button should be grey”). For adventurous low-code developers, you can also choose to make your entire app layout “responsive”, which will allow your app to automatically resize to fit the dimensions of a user’s browser window. Responsive design can add a great deal of complexity for citizen developers. If it’s not necessary for your specific use case, you can also use a fixed aspect ratio layout, which will allow your app to scale up or down similar to a PowerPoint slide deck. I often use a fixed aspect ratio to quickly build the first few versions of an app, and then switch to a responsive design once all the stakeholders approve of the overall design.</p> <h3>Enhance your skills and get things done faster</h3> <p>It’s important to remember that platforms like this aren’t just for the non-coder. They can help seasoned developers as well. Remember—skills are the new currency. Most developers do not have expertise in all aspects of bringing an app to life (e.g., authentication, front-end, back-end, databases, APIs, etc.). Low-code tools can allow a single developer to quickly put together a serviceable app rather than incurring the overhead of building a full team of developers.</p> <p>Another thing to keep in mind is how low-code/no-code platforms can save you time and money. If you’re not a developer, you might need to create a statement of work for a third-party vendor to create an app for you. This can be incredibly expensive—on the order of hundreds of thousands of dollars for medium-complexity apps. If you can do it yourself, you don’t have to find the right vendor, set up all those meetings, follow up, etc. You can get things done much faster, even if it’s just automating some existing process you have.</p> <p>I really enjoyed creating Idea Matchmaker and I’m always looking for ways now to take advantage of low-code platforms. If you’re interested in being a Citizen Developer, you might want to check out this <a href="https://hpe.zoom.us/webinar/register/4716663493942/WN_8jlRM9SaRKmbT3r1CDNtDw">HPE Developer Munch &#x26; Learn session</a> where I and a couple of colleagues get more into what HPE is doing with platforms like this today.</p>https://developer.hpe.com/2022-November-01/https://developer.hpe.com/2022-November-01/Tue, 01 Nov 2022 13:00:00 GMThttps://developer.hpe.com/galadriel-a-spire-federation-alternative/https://developer.hpe.com/galadriel-a-spire-federation-alternative/Mon, 31 Oct 2022 19:57:46 GMT<p>SPIRE (SPIFFE Runtime Environment) provides a workload identity management solution that allows organizations to establish attestable and verifiable mTLS (Mutual Transport Layer Security) connections among services. These capabilities support the concept of explicit trust beyond the perimeters of cloud providers or data centers. Its platform-agnostic and pluggable architecture gives organizations the flexibility to orchestrate services based on zero trust principles in highly heterogeneous environments. </p> <p>SPIRE provides functionality to federate with other entities following the SPIFFE(Secure Production Identity Framework for Everyone) federation specification. It allows workloads to authenticate to peer SPIRE systems on a separate node or environment that may be governed by the same organization or a separate organization. The SPIFFE community and SPIRE users have identified the need to have a scalable alternative to the current federation method. It is difficult to set up and maintain SPIRE federation in large, heterogeneous SPIRE deployments due to the numerous steps involved and issues with keeping the configuration up to date. Galadriel is an alternative approach to SPIRE federation that allows large SPIRE federation deployments to be sustainable, scalable, and secure. Galadriel offers SPIRE users a flexible, multi-tenant, and API driven solution that gets them a step closer to explicit trust.</p> <h2>SPIRE concepts</h2> <p>In SPIRE, a trust domain is a discrete zone in which workloads can mutually authenticate each other without the need to set up SPIRE federation. A trust domain typically corresponds to a distinct computing environment, administrative department, or similar division.</p> <p>SPIRE attests identities by using properties of the cloud provider or on-premises equipment.  In both cases, it maintains a set of key-pairs that represent the trust domain and its boundaries. While the private keys are kept secure, the public keys are published in files called “trust bundles,” which are necessary for validating the X.509 certificates or JWTs (JSON Web Token) that are used to establish secure communications. </p> <h3>How does the current SPIRE federation work?</h3> <p>Let’s assume that two SPIRE Servers (A and B) of two different trust domains will enter in a federated relationship. Thus, they will access each other's trust bundles to validate certificates for mTLS connections among workloads that cross trust domains. To configure the trust bundle exchange, the following must occur.</p> <p>First, SPIRE Server A exposes its trust bundle via an end-point API. This end-point is accessed by SPIFFE federation counterparts (SPIRE B) and configured in the SPIRE Server A’s configuration file. SPIRE Server B exposes its trust bundle in the same manner.</p> <p>Second, each SPIRE server must be configured to retrieve trust bundles from each other. SPIRE Server A “maps” SPIRE Server B’s federation end-point and defines a federated relationship with that server (this can be done via the federation API or via the server’s configuration file). SPIRE Server B sets the same configuration for A.</p> <p>Third, federation is bootstrapped in Servers A and B and both must be enabled to fetch trust bundles. They must authenticate the SPIFFE identity of each other to initiate the exchange.</p> <p>Fourth, registration entries are created in both servers defining which workloads are federated, so they can establish mTLS across trust domains. </p> <h3>Challenges with current SPIRE federation</h3> <p>Even though SPIRE allows the management of federation configurations dynamically via an API, there are several limitations to the current SPIRE approach that do not allow for large federation use cases.</p> <p>First, existing SPIRE federation options require a secure, public endpoint to serve the federation data, either through Web PKI (leveraging publicly trusted certificate authorities) or SPIFFE authentication. This entails a substantial administrative hassle and, in some cases, may be impossible, such as for on-premises use cases due to the security issue of exposing a public endpoint.</p> <p>Second, federating many-to-many relationships requires manual creation of federation entries. For example, federating five trust domains to each other requires 20 administrator actions (each domain needs four new configuration changes to federate with the four others). More administrator actions are required when trust points change URLs or relationships change.</p> <p>A third limitation relates to the lifecycle and auditing of federated relationships. SPIRE does not provide a mechanism to manage and track the lifecycle of federated relationships. Current techniques rely solely on manual configuration changes, and do not allow for observability of trust bundle exchange.  </p> <h2>Enter Galadriel</h2> <p>Galadriel is a new open source effort initiated by HPE that extends the existing federation authorization techniques from SPIRE by centralizing the management and exchange of trust bundles. It collects trust bundles from SPIRE Servers, routes them, and presents them to other SPIRE Servers and cloud resources. Galadriel aspires to provide an all-in-one solution for managing and auditing external relationships for SPIRE implementations.</p> <p>Galadriel introduces two main components: a server and harvester. These components facilitate the exchange of SPIRE-generated trust bundles among multiple SPIRE Servers. The exchange is enforced via relationship rules established at the Galadriel server level, and the harvesters transfer bundles to and from SPIRE Servers via a central hub. All relationships are mutually consented to and trust bundles are validated before being consumed. The harvester takes a “proxy” role and serves as a middle layer for the configuration of the federation in SPIRE and the trust bundle exchange.</p> <p>This new federation approach removes the need to have a public end-point configured in the SPIRE Server, streamlines the steps to establish federation, and creates the visibility and maintainability needed to manage federation at scale. </p> <p>An initial release of Galadriel is planned for the first half of 2023. For more detailed design information about Galadriel, check out the design proposal <a href="https://docs.google.com/document/d/1nkiJV4PAV8Wx1oNvx4CT3IDtDRvUFSL8/edit?usp=sharing&#x26;ouid=104807789400318304424&#x26;rtpof=true&#x26;sd=true">RFC - SPIFFE/SPIRE Federation</a>  presented to the SPIRE community.</p> <p>To know more about HPE's contributions to SPIFFE and SPIRE, visit the <a href="https://developer.hpe.com/platform/spiffe-and-spire-projects/home">HPE - SPIFFE/SPIRE</a> website.</p>https://developer.hpe.com/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/https://developer.hpe.com/how-to-use-an-api-access-token-for-hpe-greenlake-for-compute-ops-management/Thu, 27 Oct 2022 17:16:38 GMT<p>Common identity frameworks and protocols use token-based strategies to secure access to applications and resources. OAuth 2.0 is one of the most popular, using access tokens and refresh tokens to allow an application to access resources hosted by other servers on behalf of a user. The Compute Ops Management REST API uses the OAuth 2.0 HPE GreenLake authentication flow, where a limited lifetime access token is provided in the header of each REST API request as the authorization bearer. The access token is associated with a subject (person or service) and retains all the same permissions and privileges as the subject. In this blog post, I will discuss the essential steps required to generate this access token.</p> <h2>HPE GreenLake steps to obtain the access token for Compute Ops Management</h2> <p>Start the process by logging in and authenticating into <a href="https://console.greenlake.hpe.com/">HPE GreenLake</a>, which is authenticated by the Identity Provider (validated through username, password, Single Sign-On, or Multi-Factor Authentication). The prerequisites are that Compute Ops Management is provisioned/added to your account and you must be assigned a role associated with performing the intended operation.</p> <p>To get started, you need to create the API client credentials for the specific Compute Ops Management application instance, which is used to generate the access token. Once the token is generated, you can make further API calls.</p> <h3>Configuring API Client Credentials</h3> <p>To configure your API Client Credentials, perform the following steps:</p> <ol> <li>Click on the <strong>Manage</strong> link on the header on HPE GreenLake</li> <li>Select the <strong>API</strong> tile</li> <li>Click the <strong>Create Credentials</strong> link. The <strong>Create Credentials</strong> screen displays:</li> </ol> <p><img src="/img/glcp_manage.png" alt="GreenLake manage link" title="GreenLake manage link"></p> <p><img src="/img/glcp_api.png" alt="GreenLake API link" title="GreenLake API link"></p> <p><img src="/img/glcp_create_cred.png" alt="GreenLake Create Credential Button" title="GreenLake Create Credential Button"></p> <ol start="4"> <li>Select the <strong>Application</strong> you want to access.</li> <li>Provide a Credential Name.</li> <li>Click the <strong>Create Credentials</strong> button to continue.</li> </ol> <p><img src="/img/glcp_create_cred_dialog.png" alt="GreenLake Create Credential Dialog" title="GreenLake Create Credential Dialog"></p> <ol start="7"> <li>The <strong>Credentials Created</strong> screen displays your credentials.</li> <li>Next, you must copy the <strong>Client Secret</strong> to a safe and secure location. HPE GreenLake does not store your <strong>Client Secret</strong>. Select the <strong>copy icon</strong> to save your information.</li> <li>Click the <strong>Close</strong> button to continue.</li> </ol> <p><img src="/img/glcp_create_cred_copy.png" alt="GreenLake Copy Credential" title="GreenLake Copy Credential"></p> <h2>Generating an access token</h2> <p>Once you have created credentials, you can view their details on the API page. This token has a limited lifespan and will expire after 120 minutes. Using this method, you will need to return to the page to generate another token after it has expired.</p> <ol> <li>Click the arrow next to the credential name to display the credential details. It allows you to <strong>Generate Access Token</strong>.</li> <li>Click <strong>Generate Access Token</strong> to continue. The Generated Access Token screen displays.</li> </ol> <p><img src="/img/glcp_generate_token.png" alt="Generate GreenLake Access Token" title="Generate GreenLake Access Token"></p> <ol start="3"> <li>Enter your <strong>Client Secret</strong> and click the <strong>Create Access Token</strong> button.</li> <li>The <strong>Access Token Created</strong> screen displays your <strong>Access Token</strong>. <strong>Note:</strong> Since access tokens are not stored, HPE GreenLake recommends you make a copy of your access token and keep it in a safe location.</li> <li>Click the <strong>Close</strong> button when you are finished.</li> </ol> <p><strong>Note:</strong> This access token is referenced later as “&#x3C;copy_access_token_here>” when demonstrating an API call.</p> <p><img src="/img/glcp_create_access_token.png" alt="Create GreenLake Access Token" title="Create GreenLake Access Token"></p> <p><img src="/img/glcp_copy_token.png" alt="Copy GreenLake Access Token" title="Copy GreenLake Access Token"></p> <h3>Resetting your client secret</h3> <p>There may be a time when you want to reset your client secret for security purposes or if you did not copy the client secret. You can recreate it by using the Reset option. Resetting the client secret will invalidate all tokens associated with the Client ID and secret.</p> <ol> <li>Click the ellipsis next to the <strong>Generate Access Token</strong> button to reset your client secret.</li> <li>Click the <strong>Reset Client Secret</strong> link. The client secret is recreated with a new value.</li> </ol> <p><img src="/img/glcp_reset_token.png" alt="Reset GreenLake Access Token" title="Reset GreenLake Access Token"></p> <h1>How to use the access token</h1> <p>You can embed the access token in the REST API request to perform the HTTP method against the desired Compute Ops Management resource to obtain the response. Note that you must use the correct connectivity endpoint according to the region where Compute Ops Management is deployed. Currently, these are the connectivity endpoints for the possible regions:</p> <ul> <li><strong>EU Central</strong> - <a href="https://eu-central1-api.compute.cloud.hpe.com">https://eu-central1-api.compute.cloud.hpe.com</a></li> <li><strong>AP Northeast</strong> - <a href="https://ap-northeast1-api.compute.cloud.hpe.com">https://ap-northeast1-api.compute.cloud.hpe.com</a></li> <li><strong>US West</strong> - <a href="https://us-west2-api.compute.cloud.hpe.com">https://us-west2-api.compute.cloud.hpe.com</a></li> </ul> <p><img src="/img/glcp_endpoint.png" alt="GreenLake API Endpoint" title="GreenLake API Endpoint"></p> <p>The access token must be added to the header "Authorization: Bearer " for any REST API request. The name “Bearer authentication” can be understood as “giving access to the bearer of this token.” The following example uses the GET method for the resource servers to obtain a list of available servers.</p> <h2>How to use the access token - cURL method</h2> <p>Next is a curl command (run as a console command), that will use the generated access token to make a call against the API Endpoint to list servers in my account:</p> <p><strong>curl -X GET <a href="https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers">https://us-west2-api.compute.cloud.hpe.com/compute-ops/v1beta2/servers</a> -H "Accept:application/json" -H "Authorization:Bearer &#x3C;copy_access_token_here>”</strong></p> <p>Response - List of compute servers onboarded and assigned to the corresponding application for your customer account</p> <pre><code class="language-json">{ "offset":0, "count":1, "total":1, "items":[ { "id":"P07595-B21+MXQ1140XVX", "type":"compute-ops/server", "platformFamily":null, "resourceUri":"/compute-ops/v1beta2/servers/P07595-B21+MXQ1140XVX", "name":"MXQ1140XVX", "createdAt":"2022-09-15T18:46:21.488619+00:00", "updatedAt":"2022-09-15T18:46:21.488619+00:00", "generation":1, "state":{ "managed":true, "connected":false, "connectedModifiedAt":null, "subscriptionState":"REQUIRED", "subscriptionTier":null, "subscriptionExpiresAt":null } } ] } </code></pre> <h2>How to use the access token - POSTMAN</h2> <p>To execute the REST API using the Postman tool, the access token needs to be copied to the <strong>Bearer Token</strong> section of the <strong>Authorization</strong> tab.</p> <p><img src="/img/glcp_postman.png" alt="GreenLake API Call with POSTMAN" title="GreenLake API Call with POSTMAN"></p> <h1>Deleting client credentials</h1> <p>Deleting the client ID and secret will invalidate all tokens associated with the ID and secret.</p> <ol> <li>Click the ellipsis next to the <strong>Generate Access Token</strong> button to delete your client credentials.</li> <li>Select the <strong>Delete Credentials</strong> link. <strong>Note:</strong> If a user is deleted from HPE GreenLake, any tokens or client IDs generated and associated with any applications owned by this user will no longer be valid.</li> </ol> <p><img src="/img/glcp_delete_cred.png" alt="Delete GreenLake Client Credential" title="Delete GreenLake Client Credential"></p> <p>I hope this blog post, in giving you an example of how to obtain the access token from HPE GreenLake and use it with the Compute Ops Management REST API, helps you make the most of your as-a-Service infrastructure. You may also wish to read the blog post on the Compute Ops Management REST API for the essential steps required to further explore the API to get even more out of it.</p>https://developer.hpe.com/reflections-on-helping-linux-become-enterprise-ready/https://developer.hpe.com/reflections-on-helping-linux-become-enterprise-ready/Wed, 26 Oct 2022 15:17:41 GMT<p>In this blog series, you’ll meet some of the open source experts working with the HPE Developer Community team. In this post, I’ll be interviewing Suparna Bhattacharya who works in the HPE AI Research Lab on foundations and techniques for data centric trustworthy AI. She is an HPE Fellow and also the architect for Project Data Map, an innovation project in the CTO. Suparna was a Linux kernel contributor between 2000 and 2007 during the period when Linux was evolving into an enterprise OS.</p> <p>Some of the contributions Suparna made to Linux were in the areas of filesystems, IO and RAS (reliability, availability, and serviceability), helping to establish Linux as an enterprise-grade OS. During her tenure as a contributor, Suparna learned many things she was able to apply throughout her engineering career and shared them with me here.</p> <p><strong>Suparna, can you set the stage for me as to your involvement with contributing to the Linux kernel?</strong></p> <p>Back in 2000, there were a number of companies interested in helping Linux evolve into an operating system that could be used in enterprise-scale environments. I was working at IBM at the time and looking for ways I could contribute to help in that effort. My specific project was intended to look at serviceability capabilities, i.e. RAS features. But I wound up contributing in several areas that made their way into the kernel over the course of several years (2000-2007), including:</p> <ul> <li>Aspects of block I/O</li> <li>Asynchronous file system I/O</li> <li>ext4 file system</li> <li>pre-emptible RCU</li> <li>RAS features, including kprobes and kdump</li> </ul> <p><strong>You’ve likened your experience to bridging Lilliput and Brobdingnag, places found in Gulliver’s Travels by Jonathan Swift. Can you explain what you mean by that?</strong></p> <p>Up until that point, Linux had predominantly been used on relatively smaller systems. The challenge now was to make it work well on much larger systems in order to support the enterprise. It felt like trying to build a city where both the Lilliputians could live alongside the Brobdingnagians. You had to be careful that, when you introduced code to support the larger systems, you didn’t trample on the Lilliputians.  In other words, our challenge was to advance the scalability, responsiveness and dependability of Linux for enterprise workloads without introducing additional overhead for non-enterprise users.</p> <p><strong>Can you give an example?</strong></p> <p>Because I had experience working on file systems, my interest was piqued in relation to the Linux IO stack and the file system stack. I noticed some developments coming out of the community to expand the block IO subsystem to support larger IO requests, which is what is needed for databases. A group of developers came up with this new abstraction called kiobuf that intrigued me. I thought to myself, well, okay, if you have this, then you may need to do a few additional things when the IO request completes. So I tried building a little something on to that, adding new techniques so you could work when the IO request completed and trigger callbacks. I came up with a mechanism for doing that, and as I was discussing it on some of the IRC channels, an experienced kernel developer encouraged me to <a href="https://lkml.indiana.edu/hypermail/linux/kernel/0101.3/1273.html">post it</a> as an RFC so I could get feedback from the community.</p> <p>In doing so, I inadvertently triggered a whole <a href="https://lore.kernel.org/lkml/CA2569E6.0051970D.00@d73mta03.au.ibm.com/#r">firestorm</a> of emails around kiobuf itself between several stalwarts in the kernel development community. I wasn’t even trying to do anything to change the kiobuf layout. I was merely suggesting the addition of some new features on top of it. But apparently I had drawn the attention of Linus Torvalds to the kiobufs, descriptors that were being proposed for large IO systems and databases. He expressed his concern that, in the process of trying to support these bigger systems, we were making it more difficult for anything that required low-latency handling, like floppy disks and smaller devices.</p> <p><strong>What was the fallout from the email firestorm?</strong></p> <p>Then, <a href="https://www.landley.net/kdocs/ols/2004/ols2004v1-pages-51-62.pdf">Jens Axboe</a> started looking at how to rewrite the block IO sub-system. The challenge was to be able to support these larger IO requests while at the same time supporting diverse workloads and devices at the low end.</p> <p>Leaning on some of my past experience, I studied his re-write as he posted his patches and came up with notes on the block IO evolution that captured the design issues and tradeoffs. Some of these “bionotes” eventually became part of the block IO design documentation. While I was writing this up, a few others pointed out that there were still places where the new system was inefficient for certain kinds of IO devices, and I had to come up with some patches to fix that. It’s kind of funny how, although my original objective was to get Linux to the point where it worked better for the enterprise, I found myself contributing patches to make it better so that changes we were making ensured it still worked efficiently for smaller systems .</p> <p>Still, I found that writing up these design issues and design documentation and then contributing patches was a very nice way to get deeply involved with the community.</p> <p><strong>What did you learn from this?</strong></p> <p>This issue of potentially fixing one aspect for one set of users while introducing challenges for others led me to the understanding that, wherever you make a change, it’s better to make minimal changes and not make huge changes in one shot. Each contribution lays the groundwork for someone else to add more to it, so you can evolve the code carefully.</p> <p>Traditionally, we’ve always been used to building systems in a modular fashion. And Linux is no exception. That’s why you can work on one part of the subsystem without breaking things in other areas. But modularity can also lead to very bloated subsystems. So that’s why you want to proceed carefully, making small changes, sharing it with the community, get feedback, etc.  Don’t make one huge change all at once.</p> <p><strong>Were you able to get those RAS features in that you wanted?</strong></p> <p>It took a few years to get RAS capabilities accepted into Linux. Part of it, as I pointed out earlier, had to do with how difficult it was to support large enterprises while ensuring that we didn’t make the kernel unusable for desktop or low-end systems. But another part of it had to do with a long-held belief amongst the contributor community that you should be able to look at the code and understand why it failed, helping you identify the root cause, and not have to rely on injecting things into the kernel that might be considered intrusive.</p> <p>For example, a key piece of RAS is serviceability. If something goes wrong in an enterprise environment, you need the service engineers to be able to figure out what went wrong, and the only way they can tell is by inserting probes, like running a trace utility, doing a crash dump, etc. There was quite a lot of apprehension about incorporating these features into the mainline kernel because they were perceived as intrusive and polluting the elegance of kernel code. Many also worried that one might look at the debugger and apply “band-aid” fixes but wouldn’t really get at the root cause. It wasn’t that the community was against including these capabilities… they just didn’t want them done in a way that made the kernel messy or complicated.</p> <p>Eventually, we were able to add in kernel dynamic probes, aka kprobes, a dynamic tracing capability. The idea here was that, on a running kernel, you could put a probe in at any point and collect information instead of using a static trace. It took a long time to be incorporated; we posted our first implementation in August of 2000 and it took four years to get into the main line. I also worked on a crash dump feature, where again I learned that a minimalist approach is the best approach.</p> <p><strong>Tell me how the kprobes work emphasized the need for minimalism to you.</strong></p> <p>I call this my journey from Modularity to Minimalism, because the first time we tried to implement it, we tried to make it do everything. We wanted it to have this probing capability, but also ensure some safety restrictions and wrote an in-kernel interpreter to allow users to write these probe functions in a new language, and all of it eventually amounted to 6,000 lines of kernel code. It was all very well thought out and we expected it would be well-received by the community. But it wasn’t. It wasn’t that folks didn’t like the capability or the techniques. It was just that they felt that it really didn’t belong in the kernel. It was too much.</p> <p>A kernel developer in Australia, <a href="https://kernel.org/doc/ols/2006/ols2006v2-pages-109-124.pdf">Rusty Russell</a>, helped us out by suggesting we create just a simple patch that let you put a probe on the kernel function and register the output from that. We would isolate that tiny little patch and post it on the kernel mailing list. This brought our 6000 lines of code down to 550. It took care of the main issues to ensure correctness but didn’t attempt to address a whole lot of different situations. We also came up with a simple trick to enable developers to easily probe function arguments.</p> <p>After that, it received a lot better reception. There were other things that still needed to be added to it, and they eventually were. What I learned through doing this was that sometimes you need to go backwards in order to move forward. We had to undo a lot of the work and just tease out the very minimal elements. But that’s what led to success, because then folks were willing to adopt it and build upon it. After it was adopted, others added on Perf probes, System tap, eBPF, and a lot of other things.</p> <p><strong>What about the crash dump feature – what was the difficulty there?</strong></p> <p>Crash dump had been integrated into every other OS I had worked with, but it wasn’t in Linux. I started working with this project called Linux Kernel Crash Dump, which started to add that capability and I made a lot of contributions there, but again, it was pretty hard to get into the mainline kernel.</p> <p>The critical need here was a reliable and flexible crash dump mechanism that would support a wide-range of target devices and environments, while at the same time being easy to use and maintain. We first considered modular solutions. These provided a pluggable framework that was flexible, but the problem was that it was just too complex. It took 5000 lines of code just to capture a crash dump, and that code was being executed when the kernel was already in deep trouble.</p> <p>So there was this intrinsic challenge – we needed something simple and minimalist but most attempts to address the needs of diverse environments would make everything complex. What I learned here was that, when you have conflicting design goals like this, it’s a great opportunity for innovation.</p> <p>In the end, we asked ourselves “What is the minimal base for a crash dump?” You would need to save registers and invoke a safe soft reboot to a kernel in reserved memory while preserving a core file “view” of the crashed kernel’s memory.  This would make it possible to reuse existing utilities, such as cp to save a copy of the file (to create a disk dump) or scp and ftp (to create a network dump). When we looked at it that way, we achieved more with less; we got better flexibility and wider applicability, with barely 1000 lines of code (over kexec).</p> <p><strong>Working on Linux really made an impact on you, didn’t it?</strong></p> <p>It was a very exciting time, probably the most enjoyable phase of my entire career. We had no idea back then of how significant and ubiquitous Linux would be in the years to come – some refer to it as “the software that runs the world”.  I learned that participating in open source communities can bring a deep satisfaction from making contributions that can have a lasting impact, while also instilling lasting lessons for how one approaches their work.</p> <p>For example, one of the most important lessons I learned was that, when addressing design goals for new features, conventional approaches are not always good enough. You have to rethink and challenge long held beliefs to gracefully cope with evolving needs. New challenges are always emerging.</p> <p>And finally, I learned the power of simplicity, learning to do more with less. You want to reduce things down to the most essential features and avoid unnecessary complexity. Understanding these things has helped me to bring about change more easily in each and every engineering organization I’ve been a part of throughout my career.</p>https://developer.hpe.com/spiffe-spire-graduates-enabling-greater-security-solutions/https://developer.hpe.com/spiffe-spire-graduates-enabling-greater-security-solutions/Mon, 24 Oct 2022 18:38:25 GMT<blockquote> <p>Late September 2022, the Cloud Native Computing Foundation® (CNCF)<a href="https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/"> announced the graduation of the SPIFFE and SPIRE projects</a>, joining an elite group of only 18 graduated CNCF projects, like HELM and Kubernetes. Being designated a graduated project means that the project is considered stable, with enough governance processes around it, and ready to be widely deployed in production.</p> <h3>A Secure Production Identity Framework for Everyone</h3> <p>SPIFFE (Secure Production Identity Framework for Everyone) is designed to work within dynamic and heterogeneous environments to provide a means to securely authenticate workloads using a Zero Trust system. These workloads could be anything from a physical or virtual node to an individual process. By providing a secure identity to every workload, it removes the need for shared secrets and provides a foundation for higher level platform-agnostic security controls.</p> <p>Rising from the need to better secure today’s cloud-native ecosystem, SPIFFE and SPIRE resolved a fundamental issue – a need for a standardized, cryptographic, platform-agnostic identity foundation to help secure services across heterogeneous cloud production environments. SPIFFE and SPIRE moved from the CNCF Sandbox to the Incubator in 2020, and has grown significantly since, being contributed to by many leading technology companies, including VMWare, Uber, ByteDance, Anthem, Transferwise, IBM, and Hewlett Packard Enterprise (HPE).</p> <h3>How HPE puts SPIFFE/SPIRE into action</h3> <p>As a technology company, leading and influencing meaningful innovation in the Open Source community is important to us, as it provides key capabilities that our customers desire and helps us design solutions that meet their needs. SPIFFE and SPIRE are important components of enabling seamless zero trust infrastructures and have become integral parts of a number of HPE projects, including the following:</p> <h4>HPE Cray Exascale Super Computer Management</h4> <p>SPIFFE/SPIRE has shown that it can scale and how it’s being used by the Cray System Management (CSM) tooling underscores that super power. HPE Cray Exascale Super Computers are big – data center-sized big – comprised of tens of thousands of compute nodes and high-performance storage. In late 2018, development work began on the Cray CSM, a solution enabling system administrators to manage these large-scale supercomputers leveraging the architecture and advances of hyper-scalers and cloud providers, based on open source code.</p> <p>When both Scytale and Cray were acquired by HPE in 2019, the synergies found in their technologies started the engineering groups working together. They were quickly able to identify how important and helpful SPIFFE/SPIRE could be to the CSM development effort. Given how CSM helps customers to go beyond the traditional and enable new services, deploy a broad range of workloads, and drive towards an as-a-service experience, the integration of SPIRE makes these implementations even more secure.</p> <p>Today, SPIRE is an integral part of CSM and running in production at facilities including <a href="https://www.hpe.com/us/en/newsroom/press-release/2021/04/us-department-of-energys-los-alamos-national-laboratory-expands-collaboration-with-hewlett-packard-enterprise-on-new-supercomputer-design-to-advance-scientific-research.html">Los Alamos National Lab</a> (Lawrence Berkeley National Lab – <a href="https://www.nersc.gov/">National Energy Research Scientific Computing Center</a>), <a href="https://www.hpe.com/us/en/newsroom/press-release/2020/10/hewlett-packard-enterprise-wins-160m-contract-to-power-one-of-the-worlds-fastest-supercomputers-based-in-finland-to-bolster-europes-research-in-science-and-unlock-economic-growth.html">EuroHPC JU’s LUMI</a>, the <a href="https://www.hpe.com/us/en/newsroom/press-release/2021/04/swiss-national-supercomputing-centre-hewlett-packard-enterprise-and-nvidia-announce-worlds-most-powerful-ai-capable-supercomputer.html">Swiss National Supercomputer Center (CSCS)</a> and the <a href="https://www.hpcwire.com/2021/04/22/microsoft-to-provide-worlds-most-powerful-weather-climate-supercomputer-for-uks-met-office/">UK Met weather service</a>, with more coming. It is also expected to play a role in enabling federated systems interaction in various scenarios both in HPC and AI/ML operations in many HPE Cray Exascale facilities and beyond.</p> <h4>HPE GreenLake</h4> <p>As our customers transition to an everything-as-a-service consumption model, they must deal with situations where applications can straddle multiple data centers, multiple clouds, 3rd party managed service providers and edge locations. A fundamental but complex problem with this digital transformation is how these physically distributed software systems can reliably authenticate to each other and securely communicate at scale, especially over untrusted networks.  </p> <p>As part of shared security responsibilities with our customers to ensure customers can trust to host their data on HPE GreenLake Edge-to-Cloud solutions, HPE must ensure HPE GreenLake Edge-to-Cloud Platform adheres to zero trust security best practices. HPE GreenLake Edge-to-Cloud Platform leverages SPIFFE and SPIRE to ensure all micro-services running inside the platform, and any external micro-services that the platform needs to talk to, are continuously attested and issued short-lived cryptographic identities (SPIFFE IDs) to establish secure communication with mutual TLS (mTLS) encryption. This protects the platform by eliminating long-lived secrets and minimizing any impact from credential exfiltration attacks.</p> <p>The SPIFFE/SPIRE cryptographic identities enable other opportunities for use with HPE GreenLake Edge-to-Cloud Platform as well. They offer a global identity foundation that could be leveraged for a broad spectrum of platform eco-system tools and services, such as a service mesh, policy framework, secrets managers, identity and access management (IAM) frameworks, and software supply chain integrity tools. By bootstrapping authentication to authorization tools, one could address the “credential zero” or “secure introduction” problem by eliminating the need for workloads to securely store the credentials needed to authenticate to the secret store.</p> <h3>Driving towards the best secured edge-to-cloud platform</h3> <p>The examples noted above are just some of the ways HPE is using this innovative technology to provide for a better and more seamlessly secured environment. There’s a lot more going on. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE Developer blog</a> to learn about different SPIFFE/SPIRE implementations and other news in this area. You might also want to check out the book, <a href="https://spiffe.io/book/">Solving the Bottom Turtle</a>, for interesting and detailed information about SPIFFE and SPIRE.</p> </blockquote>https://developer.hpe.com/exploring-the-role-of-the-citizen-developer/https://developer.hpe.com/exploring-the-role-of-the-citizen-developer/Mon, 24 Oct 2022 13:33:55 GMT<style> ul li{ font-size:25px; list-style-position: outside; margin-bottom:8px; } </style> <p><img src="/img/hpe20160720045_1600_0_72_rgb-1200-x-675.jpg" alt=""></p> <p>A topic I’ve been reading about a lot lately is the rise of the Citizen Developer.</p> <p>Citizen Developers leverage low-code/no-code (LCNC) platforms to build applications through visual representations without having to have spent years developing deep coding skills. They’ve become more plentiful as more and more businesses rely on software applications to help automate their processes, helping organizations do more with less.</p> <p>In essence, Citizen Developers are just non-IT employees who use technology to make work easier and more efficient using platforms designed to create programs using intuitive interfaces and a mouse click, as opposed to having to write lines and lines of code. Enabling Citizen Developers can save companies a lot of money by removing the constraint on IT for low-complexity innovation and helping everyone become more self-sufficient and efficient.</p> <p>At last week’s [SlashData’s DevRe[X] Summit](<a href="https://www.devrelxsummit.com/">https://www.devrelxsummit.com/</a>), I learned that the usage of low/no code has doubled in the last 2 years, with 57% of developers reporting the use of LCNC tools.</p> <p><img src="/img/usage-of-low-no-code-copy.jpg" alt="Usage of low/no code"></p> <p>You can check the full session from Christina Voskoglou, which the data comes from, <a href="https://youtu.be/Up-N5x6NCps">here</a>.</p> <p>That is an interesting statistic, as it seems to indicate that these tools are being used not only by Citizen Developers, but are also making their way into “real” developers’ toolboxes (for building prototypes, front end development, etc.).</p> <p>The reason for this rise in popularity is that there are many pros to the Citizen Developer movement:</p> <ul> <li>Low-code/no-code platforms allow non-developers to build simple apps with little IT investment, while freeing up corporate developers to focus on key applications.</li> <li>They provide pre-existing libraries of components for quick software builds.</li> <li>It expands the ability to innovate within the enterprise.</li> <li>The platforms allow one to start small, then customize and improve as you go, again without corporate developer interaction.</li> </ul> <p>As you can see, there are a lot of benefits to be had for an organization. However, if you decide you want to provide your employees with low-code/no-code platforms and tools, there are a few things you’ll want to make sure corporate IT is involved with:</p> <ul> <li>You might need to provide training on simple development concepts and on LCNC tools you selected, and deployed for employees.</li> <li>You will need to watch over the data being used by Citizen Devs, to make sure no compliance or privacy rules are broken. Most likely you will need to implement some policy restrictions, which might represent a governance challenge.</li> <li>Enabling Citizen Devs can represent a risk to IT sprawl and shadow IT, if not carefully managed.</li> <li>It may represent a security risk, as in most case these Citizen Dev-built apps are being used and developed on personal devices, mobile devices and non-corporate Wi-Fi connections.</li> </ul> <h3>Read more about it</h3> <p>One article I recently found was very interesting: <a href="https://thenewstack.io/how-low-code-platforms-can-help-cloud-native-developers/"> How Low-Code Platforms Can Help Cloud Native Developers</a>. Vinothini Raju, who wrote a <a href="https://developer.hpe.com/blog/autopilot-kubernetes-deployments-on-hpe-ezmeral-runtime-enterprise/">blog post</a> for the <a href="https://developer.hpe.com/">HPE Developer Community</a> last year, reminds us that the rise of no-code platforms isn’t surprising, given that humans have a tendency to process visuals much faster than text. In her article, she explores low-code/no-code platforms for cloud-native deployments. It’s a fascinating article that points out why these platforms are such a boon to cloud-native development. It’s a short read and I highly recommend it.</p> <h3>Hear more about it</h3> <p>HPE is exploring low-code/no-code options as well. In the <a href="https://hpe.zoom.us/webinar/register/8716666192015/WN_8jlRM9SaRKmbT3r1CDNtDw">HPE Developer November 16th Munch &#x26; Learn</a>, Jeffrey Fougere, an Innovation Strategist at HPE, Colin Lue King, Talent Management Manager at HPE, and Richard Kerridge, Learning Technology Architecture &#x26; Strategy Manager at HPE will come to talk on this subject. They’ll introduce session attendees to some of the popular platforms, like Microsoft Power Aps and Power Automate, and explain how HPE employees themselves have been able to build enterprise-level apps and automations with little-to-no budget. This is a session you’ll not want to miss. Even full-time developers can benefit from learning about these technologies to create full-stack solutions outside of their areas of expertise. There will be a replay posted on the <a href="https://developer.hpe.com/campaign/munch-and-learn">Munch &#x26; Learn Technology Talks</a> calendar page once it’s ready.</p> <p>What’s your experience with low-code/no-code solutions? Other community members could learn a lot from your experience. If you’re a citizen developer, consider <a href="https://developer.hpe.com/contribute">contributing a blog post</a> on the subject. The HPE Developer Community team would love to hear more about it.</p>https://developer.hpe.com/how-to-monitor-hpe-compute-ops-management/https://developer.hpe.com/how-to-monitor-hpe-compute-ops-management/Wed, 19 Oct 2022 18:21:24 GMT<style>ul li{ font-size:26px;padding-bottom: 0.5em;}</style> <style> i{ color:grey;font-family:'Courier New';font-size:22px; } </style> <p>The purpose of this blog post is to describe how to generate Grafana dashboards to monitor any HPE Compute infrastructure managed by HPE GreenLake for Compute Ops Management.</p> <h1>Grafana Dashboards</h1> <p>IT infrastructure metrics visualization is critical for health monitoring, prediction, and capacity planning. It provides a powerful way of viewing infrastructure utilization, revealing issues and helping maintain uninterrupted services.</p> <p>Grafana’s time-series graphs are the perfect enabler for IT infrastructure optimization. They can assist administrators in monitoring temperature changes, network traffic performance, power consumption, and much more. They can be used to compare data over time to note trends and detect issues, allowing administrators to make any necessary adjustments and prevent downtime.</p> <p>The following picture shows a typical HPE infrastructure dashboard with different panels generated from HPE Compute Ops Management:</p> <p><img src="/img/2022-10-21-11_14_38-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <h1>HPE GreenLake for Compute Ops Management REST API</h1> <p>HPE GreenLake for Compute Ops Management provides a northbound RESTful <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API </a>that supports many operations. All the data you can get from the Compute Ops Management API can be leveraged to create beautiful and instructive Grafana dashboards.</p> <p>To take advantage of this, simply use a generic Grafana plugin that can handle REST requests, parse JSON responses and generate tables. With this solution, we greatly reduce the complexity of the solution which in principle requires a database like Prometheus or InfluxDB. In this post, I will show you how to do it without a database...</p> <p>HPE GreenLake for Compute Ops Management REST API uses the OAuth 2.0 authentication based on the client credential, which generates a limited lifetime access token.</p> <p>The access token is a long string in the form of a JSON Web Token that is signed using RS256 algorithm. The access token must be added into the HTTP header with keyword "Authorization: Bearer {token}" for any REST API request.</p> <p>For information about how to create API client credentials and how to generate an access token for HPE GreenLake for Compute Ops Management, refer to <a href="https://developer.greenlake.hpe.com/docs/greenlake/guides/public/authentication/authentication/">this web page</a>.</p> <p>Only a few resource metrics are currently supported by Compute Ops Management via the RESTful API, but things will change quickly in the coming months. Today, the only metric available is the carbon footprint report but many other resources are available to create nice Grafana dashboards such as data related to the number of servers, health of servers, service packs, groups, etc.</p> <h1>Grafana Infinity plugin</h1> <p>There are several Grafana plugins that support data collection via the REST API (e.g. Infinity, <a href="https://grafana.com/grafana/plugins/simpod-json-datasource/">JSON</a>, <a href="https://grafana.com/grafana/plugins/marcusolsson-json-datasource/">JSON API</a>) but <a href="https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/">Infinity </a>has the great advantage of offering an advanced query language that is essential for manipulating JSON data into a suitable format that Grafana can understand. This language is called <a href="https://sriramajeyam.com/grafana-infinity-datasource/wiki/uql/">UQL</a>, Infinity's unstructured query language.</p> <p>UQL is not simple at first glance, but I will provide examples in this blog. With UQL, you can customize the results you need regardless of the JSON format returned by the API.</p> <p>A UQL query can be formed with a list of commands joined by <code>|</code>. Most of the time, fields are referenced in double quotes and string values are referenced in single quotes as shown below:</p> <p><img src="/img/2022-10-21-11_47_58-hpe-software-%E2%80%8E-onenote-for-windows-10.png" alt=""></p> <p>The following diagram describes the different components of the solution:</p> <p><img src="/img/2022-10-19-16_15_11-lj-synergy-composable-fabric.pptx-powerpoint.png" alt=""></p> <h2>Pros and Cons about this solution</h2> <p>As with any solution, there are both Pros and Cons to using it.</p> <p>Pros:</p> <ul> <li>A lightweight solution that only requires Grafana and an easily installable plugin</li> <li>Supports collecting metrics from any API</li> <li>Cross-platform support, all components can be installed on Microsoft Windows or Linux</li> </ul> <p>Cons:</p> <ul> <li>Cannot create a time series Grafana visualization with non-time series data you may retrieve from an API (This would require the use of a database, like Prometheus or InfluxDB)</li> <li>Requires in-depth knowledge of the API, authentication, and methods</li> <li>Requires knowledge of the UQL language to manipulate JSON data</li> </ul> <h1>Configuration</h1> <h2>Prerequisites</h2> <ul> <li>Grafana must be installed, started, and enabled</li> <li>HPE GreenLake for Compute Ops Management API client credentials are required (this consists of a <em>client ID</em> and a <em>client secret</em>)</li> </ul> <h2>Infinity plugin installation</h2> <p>From an SSH session on the Grafana server, enter:<br> > <i>grafana-cli plugins install yesoreyeram-infinity-datasource</i></p> <p>Then restart the Grafana service:<br> > <i>service grafana-server restart</i></p> <p>For more details on how to install the Infinity plugin, you can check out the <a href="https://github.com/yesoreyeram/grafana-infinity-datasource">Infinity GitHub repository</a>.</p> <h2>Grafana configuration</h2> <p>To launch the Grafana User Interface, open a web browser and navigate to <strong>http://&#x3C;grafana_IP or DNS name>:3000/</strong></p> <p>Note: The default HTTP port that Grafana listens to is 3000 unless you have configured a different port.</p> <p>Click on the gear icon on the side menu and click <strong>Data Sources</strong>.</p> <p><img src="/img/lj-grafana-com-picture1.png" alt=""></p> <p>Search for Infinity from the data source list.</p> <p><img src="/img/lj-grafana-com-picture2.png" alt=""></p> <p>Add <strong>Infinity</strong> as a new data source and name it <strong>Infinity-COM</strong>.</p> <p><img src="/img/2022-10-19-17_05_15-infinity-com_-settings-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>Leave all other settings as default and click <strong>Save &#x26; test</strong>.</p> <p>Then to create a new Dashboard, click on the Dashboards icon on the side menu and select <strong>New dashboard</strong>.</p> <p><img src="/img/lj-grafana-com-picture3.png" alt=""></p> <p>The next steps consist in creating variables and panels for this dashboard. I will describe in this blog each step with a lot of details to give you an overview of the methods I use.</p> <blockquote> <p><strong>Important note</strong>: You have the option to import the dashboard directly from my <a href="https://github.com/jullienl/HPE-Compute-Ops-Management/tree/main/Grafana">GitHub repository</a> using the JSON file provided, which contains everything you need (layout, variables, styles, data sources, queries, etc.). Once imported in Grafana, the same dashboard that is described in detail below with its different panels will be immediately available, it includes carbon emissions reports, server information and health, firmware bundles, groups information, etc. Refer to the README.md for more details.</p> </blockquote> <p>Going back to the manual configuration, the next step is to create Grafana variables for this new dashboard. These will be used to simplify the API authentication process and the REST requests you will define later.</p> <p>Click on <strong>Dashboard settings</strong>:</p> <p><img src="/img/lj-grafana-com-picture4.png" alt=""></p> <p>Then select <strong>Variables</strong> then <strong>Add variables</strong>:</p> <p><img src="/img/lj-grafana-com-picture5.png" alt=""></p> <p>Three variables are required:</p> <ol> <li> <p>A variable for the HPE GreenLake for Compute Ops Management API endpoint URL:</p> <p>Endpoints are the host URLs that you will submit your API requests to. Compute Ops Management has unique endpoints in specific regions. Which region is used depends on which region the devices were onboarded into via the HPE GreenLake Cloud Platform.</p> <p>Use the following list to identify your application endpoint:</p> <ul> <li>US West: <a href="https://us-west2-api.compute.cloud.hpe.com/">https://us-west2-api.compute.cloud.hpe.com/</a></li> <li>EU Central: <a href="https://eu-central1-api.compute.cloud.hpe.com/">https://eu-central1-api.compute.cloud.hpe.com/</a></li> <li>AP NorthEast: <a href="https://ap-northeast1-api.compute.cloud.hpe.com/">https://ap-northeast1-api.compute.cloud.hpe.com/</a></li> </ul> <p>Create a new variable using the following parameters:</p> <ul> <li>Name: <strong>url</strong></li> <li>Type: <strong>Custom</strong></li> <li>Value: <em>&#x3C;Your endpoint URL></em></li> </ul> <p><img src="/img/2022-10-19-17_41_55-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <p>Note: If you have devices in different regions, you can define your variable with the values of the three endpoints separated by a comma.</p> <p><img src="/img/2022-10-24-11_22_41-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>Note: When multiple endpoints values are defined, the <em>url</em> variable is listed in a drop-down list at the top of the dashboard. This is very useful if you want to quickly adjust the visualization of the different regions.</p> <p><img src="/img/2022-10-24-11_24_26-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> </li> <li> <p>A variable to generate the access token for the API authentication:</p> <p>Compute Ops Management REST API uses the OAuth 2.0 authentication based on the client credential, which generates a limited lifetime access token. So, the variable must be created using:</p> <ul> <li>Name: <strong>session</strong></li> <li>Type: <strong>Query</strong></li> <li>Hide: <strong>Variable</strong></li> <li>Data source: <strong>Infinity-COM</strong></li> <li>Query Type: <strong>Infinity</strong></li> <li>URL: <strong><a href="https://sso.common.cloud.hpe.com/as/token.oauth2">https://sso.common.cloud.hpe.com/as/token.oauth2</a></strong></li> </ul> <p><img src="/img/2022-10-24-11_51_57-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>Click then on <strong>HTTP method, Query param, Headers</strong> and use the following parameters:</p> <ul> <li>Method: <strong>POST</strong></li> <li>Body: <strong>grant_type=client_credentials&#x26;client_id=</strong>&#x3C;your-client-ID><strong>&#x26;client_secret=</strong>&#x3C;your-client-secret></li> </ul> <p>Note: <em>&#x3C;your-client-ID></em> and <em>&#x3C;your-client-secret></em> are the Compute Ops Management API client credentials generated from the HPE GreenLake Cloud Platform (GLCP).</p> <img src="/img/2022-10-19-18_07_06-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /> <p>And add in the Headers tab:</p> <ul> <li>Header Name: <strong>Content-type</strong></li> <li>Header Value: <strong>application/x-www-form-urlencoded</strong></li> </ul> <p><img src="/img/lj-grafana-com-picture6.png" alt=""></p> <p>When everything is properly configured, the access token generated by the Compute Ops Management API should be displayed in the <em>Preview of values</em> section at the bottom of the page:</p> <p><img src="/img/lj-grafana-com-picture7.png" alt=""></p> </li> <li> <p>A variable for the carbon footprint report ID:</p> <p>I use a variable for the carbon footprint report ID, because each time a new report is generated, a new <code>id</code> is created. So, by using a variable, I can fetch the last report <code>id</code> and be sure that all my CO2 report API requests will be successful.</p> <p>For this variable, use the following parameters:</p> <ul> <li>Name: <strong>reportID</strong></li> <li>Type: <strong>Query</strong></li> <li>Hide: <strong>Variable</strong></li> <li>Data source: <strong>Infinity-COM</strong></li> <li>Query Type: <strong>Infinity</strong></li> <li>URL: <strong>${url}/compute-ops/v1beta1/reports</strong></li> <li>Column 1: <strong>reportDataUri</strong></li> </ul> <p><img src="/img/2022-10-24-11_53_38-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>Note: <code>${variablename}</code> is the general syntax for calling a variable in Grafana. So <code>${url}</code> used in the URL field calls the <em>url</em> variable you defined earlier. Same for <code>${session}</code> below in the header value, it calls the access token generated by the API.</p> <ul> <li>Method: <strong>GET</strong></li> </ul> <p><img src="/img/2022-10-19-18_35_09-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <ul> <li>Header Name: <strong>Authorization</strong></li> <li>Header Value: <strong>Bearer ${session}</strong></li> </ul> <p>&#x3C;img</p> </li> </ol> <p>src="/img/2022-10-19-18_34_48-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <p>Once completed, the URI of the carbon footprint report should be displayed in the <em>Preview of values</em> section:</p> <p><img src="/img/2022-10-19-19_02_23-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <p>The variables and parameters of the dashboard are now complete. The warning on the <code>reportID</code> variable below is expected because this variable is not yet used by any panel. This will be corrected once <code>reportID</code> is used in a query.</p> <p><img src="/img/2022-10-19-19_17_15-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>Click on <strong>Save dashboard</strong> and give it an appropriate name.</p> <h2>Creating Grafana panels</h2> <p>The last step of the configuration is to create panels for this new dashboard.</p> <p>To create a first panel, click on the <strong>Add panel</strong> icon in the upper menu bar and click <strong>Add a new panel</strong>.</p> <p><img src="/img/2022-10-19-19_41_45-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <p>The following sections describe different panels you can generate in Grafana with HPE Compute Ops Management. They are provided as examples and the parameters can of course be modified to suit your needs. Each section lists the required parameters, provides API information, and methods and includes the UQL queries (if needed) to transform the JSON data.</p> <h3>Carbon footprint report (all servers)</h3> <p>Analyzing carbon emissions can help you understand the impact of your servers on the environment. Use the carbon footprint report to view the estimated carbon emissions generated by powering all the servers in a Compute Ops Management application region. The report displays the following information for the past seven days:</p> <ul> <li>Estimated total carbon emissions for all servers</li> <li>Estimated daily carbon emissions for all servers</li> <li>Estimated total carbon emissions for each server</li> </ul> <p>The report does not include estimates of the embedded carbon footprint from manufacturing and distribution of the servers.</p> <blockquote> <p><strong>Important note</strong>: Carbon footprint information is only available after running a report from the HPE GreenLake Cloud Platform GUI and, be aware, there is currently no API method to generate a report, so this process can only be manual and via the GUI. The carbon footprint report, when run, collects data from the last seven days and Compute Ops Management only saves the most recent report. As a result, the visualization in Grafana will be limited to seven days prior to the report run date and to get an up-to-date graph, it is necessary to run a new report from the HPE GreenLake GUI.</p> </blockquote> <h4>Panel overview</h4> <p><img src="/img/2022-10-19-20_06_34-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <h4>API request</h4> <p><code>get /compute-ops/v1beta1/reports/{id}/data</code></p> <h4>API response</h4> <pre><code class="language-{"> "id": "843023bd-9412-46c2-8ac2-a3691f657fdb", "type": "compute-ops/report-data", "name": "Carbon Footprint Report (All Servers)", "request": { "reportDataStartAt": "2022-02-04T01:04:20+00:00", "reportDataEndAt": "2022-02-11T01:04:20+00:00", "reportType": "CARBON_FOOTPRINT" }, "series": [ { "name": "Carbon Emissions", "type": "CO2_EMISSIONS", "units": "kgCO2e", "unitsDisplayName": "kilograms of carbon dioxide equivalent", "subject": { "displayName": "1M512501AB", "id": "875765-S01+1M512501AB", "type": "SERVER" }, "summary": { "avg": 6.4, "sum": 6.4 }, "bucketDurationInSec": 86961.3, "expectedSamplesPerBucket": 289.9, "buckets": [ { "timestamp": "2019-08-24T14:15:22Z", "value": 6.4, "numSamples": 233, "noValueReason": null, "extrapolated": 8 } ] } ] } </code></pre> <h4>Panel configuration</h4> <ul> <li> <p>Data source: <strong>Infinity-COM</strong><br /></p> </li> <li> <p>Type: <strong>UQL</strong> <br /></p> </li> <li> <p>Format: <strong>Time Series</strong> <br /></p> </li> <li> <p>URL: <strong>${url}${reportID}</strong> <br /></p> </li> <li> <p>Method: <strong>GET</strong> <br /></p> </li> <li> <p>Header name: <strong>Authorization</strong></p> </li> <li> <p>Header value = <strong>Bearer ${session}</strong></p> </li> <li> <p>UQL:<br> <strong>parse-json</strong><br> <strong>| jsonata "series[subject.type = 'TOTAL']"</strong><br> <strong>| scope "buckets"</strong><br> <strong>| project "Timestamp"=todatetime("timestamp"), "Carbon Emissions (kgCO2e)"="value"</strong></p> <p>Description of the UQL commands:</p> <ul> <li><strong>parse-json</strong>: command to instruct UQL to parse the response as JSON</li> <li><strong>jsonata</strong>: command to select the object representing the carbon emission report for all servers available in the <code>series</code> array</li> <li><strong>scope</strong>: command to set <code>buckets</code> as the output data</li> <li><strong>project</strong>: command to create a table with two columns (<em>Timestamp</em> and <em>Carbon Emissions (kgCO2e)</em>)</li> </ul> <p>Note: JSONata is an open-source expression language that is used for querying and transforming JSON data. You can refer to the following <a href="https://www.stedi.com/docs/mappings/jsonata-cheatsheet">JSONata Cheatsheet</a> for tons of examples on how to manipulate JSON data.</p> </li> <li> <p>Visualization: <strong>Time Series</strong></p> <ul> <li>Unit: <strong>Number</strong></li> </ul> </li> </ul> <p><img src="/img/2022-10-19-20_07_50-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <h3>Carbon footprint report (each server)</h3> <p>This report displays the estimated total carbon emissions for each server.</p> <h4>Panel overview</h4> <p><img src="/img/2022-10-21-11_23_31-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <h4>API request</h4> <p><code>get /compute-ops/v1beta1/reports/{id}/data</code></p> <h4>API response</h4> <pre><code class="language-{"> "id": "843023bd-9412-46c2-8ac2-a3691f657fdb", "type": "compute-ops/report-data", "name": "Carbon Footprint Report (All Servers)", "request": { "reportDataStartAt": "2022-02-04T01:04:20+00:00", "reportDataEndAt": "2022-02-11T01:04:20+00:00", "reportType": "CARBON_FOOTPRINT" }, "series": [ { "name": "Carbon Emissions", "type": "CO2_EMISSIONS", "units": "kgCO2e", "unitsDisplayName": "kilograms of carbon dioxide equivalent", "subject": { "displayName": "1M512501AB", "id": "875765-S01+1M512501AB", "type": "SERVER" }, "summary": { "avg": 6.4, "sum": 6.4 }, "bucketDurationInSec": 86961.3, "expectedSamplesPerBucket": 289.9, "buckets": [ { "timestamp": "2019-08-24T14:15:22Z", "value": 6.4, "numSamples": 233, "noValueReason": null, "extrapolated": 8 } ] } ] } </code></pre> <h4>Panel configuration</h4> <ul> <li> <p>Data source: <strong>Infinity-COM</strong><br /></p> </li> <li> <p>Type: <strong>UQL</strong> <br /></p> </li> <li> <p>Format: <strong>Table</strong> <br /></p> </li> <li> <p>URL: <strong>${url}${reportID}</strong> <br /></p> </li> <li> <p>Method: <strong>GET</strong> <br /></p> </li> <li> <p>Header name: <strong>Authorization</strong></p> </li> <li> <p>Header value = <strong>Bearer ${session}</strong></p> </li> <li> <p>UQL:<br> <strong>parse-json</strong><br> <strong>| scope "series"</strong><br> <strong>| project "Servers"="subject.displayName", "Carbon Emissions"="summary.sum"</strong></p> <p>Description of the UQL commands:</p> <ul> <li><strong>parse-json</strong>: command to instruct UQL to parse the response as JSON</li> <li><strong>scope</strong>: command to set <code>series</code> as the output data</li> <li><strong>project</strong>: command to create a table with two columns (<em>Servers</em> and <em>Carbon Emissions</em>)</li> </ul> </li> <li> <p>Override1: Fields with name = <strong>Carbon Emissions</strong> / Cell display Mode = <strong>LCD Gauge</strong></p> </li> <li> <p>Visualization: <strong>Table</strong></p> <ul> <li>Unit: <strong>kgCO2e</strong></li> <li>Color scheme: <strong>Green-Yellow-Red (by value)</strong><br /></li> </ul> </li> </ul> <p><img src="/img/2022-10-19-20_26_28-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <h3>Server health and information panel</h3> <p>This panel displays the health and information for each server. The information you want to display in the panel is very flexible, many properties are available in servers resource, such as server model, serial number, model, power status, etc.</p> <h4>Panel overview</h4> <p><img src="/img/2022-10-21-11_24_26-hpe-com-using-infinity-uql-native-api-calls-grafana-%E2%80%94-mozilla-firefox.png" alt=""></p> <h4>API request</h4> <p><code>get /compute-ops/v1beta2/servers/</code></p> <h4>API response</h4> <pre><code class="language-{"> "id": "P39886-B21+CN70490RXS", "type": "compute-ops/server", "platformFamily": "PROLIANT", "resourceUri": "/compute-ops/v1beta2/servers/P39886-B21+CN70490RXS", "name": "HPE-HOL17", "createdAt": "2022-03-25T16:22:22.373640+00:00", "updatedAt": "2022-10-19T20:18:08.466399+00:00", "generation": 515, "hardware": { "serialNumber": "CN70490RXS", "model": "ProLiant DL360 Gen10 Plus", "uuid": "38393350-3638-4E43-3730-343930525853", "productId": "P39886-B21", "powerState": "OFF", "indicatorLed": "OFF", "health": { "summary": "WARNING", "healthLED": "WARNING", "fans": "OK", "fanRedundancy": "REDUNDANT", "liquidCooling": "NOT_PRESENT", "liquidCoolingRedundancy": "NOT_PRESENT", "memory": "OK", "network": "UNKNOWN", "powerSupplies": "OK", "powerSupplyRedundancy": "NOT_PRESENT", "processor": "OK", "storage": "OK", "temperature": "OK", "bios": "WARNING", "smartStorage": "OK" }, "bmc": { "mac": "B4:7A:F1:54:71:68", "ip": "172.30.231.79", "hostname": "None" } }, "state": { "managed": true, "connected": true, "connectedModifiedAt": "2022-10-18T17:38:10.624330+00:00", "subscriptionState": "SUBSCRIBED", "subscriptionTier": "Enhanced", "subscriptionExpiresAt": "2027-01-31T19:11:00+00:00" }, "firmwareInventory": [ { "name": "iLO 5", "version": "2.72 Sep 04 2022", "deviceContext": "System Board" }, , { "name": "8 SFF 12G x1SAS UBM2 BC BP", "version": "1.20", "deviceContext": "Slot=12:Port=2I:Box=1" } ], "softwareInventory": [], "lastFirmwareUpdate": { "status": "OK", "attemptedAt": "2022-10-19T20:12:03.401750Z", "firmwareInventoryUpdates": [ { "name": "U46_1.64_08_11_2022.fwpkg", "version": "1.64_08-11-2022", "status": "OK" }, &#x3C;...> { "name": "cp053854.exe", "version": "5.32", "status": "OK" } ] }, "host": { "osName": "None", "osVersion": "None", "hostname": "HPE-HOL17", "osType": null, "osDescription": "None" }, "firmwareBundleUri": "/v1/firmware-bundles/ea469b39ed5106434169397999e816bf", "tags": {}, "biosFamily": "U46", "processorVendor": "Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz", "autoIloFwUpdate": true } </code></pre> <h4>Panel configuration</h4> <ul> <li>Data source: <strong>Infinity-COM</strong><br /></li> <li>Type: <strong>JSON</strong> <br /></li> <li>Format: <strong>Table</strong> <br /></li> <li>URL: <strong>${url}/compute-ops/v1beta2/servers?limit=100</strong> <br /></li> <li>Method: <strong>GET</strong> <br /></li> <li>Header name: <strong>Authorization</strong></li> <li>Header value = <strong>Bearer ${session}</strong><br /></li> <li>Column 1: <strong>name</strong> as <strong>Name</strong></li> <li>Column 2: <strong>hardware.serialNumber</strong> as <strong>Serial Number</strong></li> <li>Column 3: <strong>hardware.model</strong> as <strong>Model</strong></li> <li>Column 4: <strong>hardware.health.summary</strong> as <strong>Health</strong></li> <li>Column 5: <strong>hardware.powerState</strong> as <strong>Power State</strong></li> <li>Column 6: <strong>hardware.bmc.ip</strong> as <strong>iLO IP</strong></li> <li>Column 7: <strong>lastFirmwareUpdate.status</strong> as <strong>Last FW Update Status</strong></li> </ul> <p><img src="/img/2022-10-20-17_20_28-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <ul> <li>Override1: Fields with name = <strong>Health</strong> / Cell display Mode = <strong>Color text</strong></li> <li>Override2: Fields with name = <strong>Power State</strong> / Cell display Mode = <strong>Color text</strong></li> <li>Override3: Fields with name = <strong>Last FW Update Status</strong> / Cell display Mode = <strong>Color text</strong></li> </ul> <p><img src="/img/2022-10-20-17_23_49-.png" /></p> <ul> <li>Visualization: <strong>Table</strong></li> <li>Value mappings: <ul> <li><strong>OK</strong>: Green</li> <li><strong>WARNING</strong>: Orange</li> <li><strong>ERROR</strong>: Red</li> <li><strong>ON</strong>: Green</li> <li><strong>OFF</strong>: Red</li> </ul> </li> </ul> <p><img src="/img/2022-10-20-17_22_27-hpe-com-using-infinity-uql-native-api-calls-grafana-—-mozilla-firefox.png" /></p> <p>Many other panels using other <a href="https://developer.greenlake.hpe.com/docs/greenlake/services/compute-ops/public/openapi/compute-ops-latest/overview/">API resources</a> can be generated, for example for firmware bundles, groups, activities, etc. And now you have all the basics to get started and create the panels you need in your environment.</p> <p>This concludes this blog post. I hope you find it useful and should you have any feedback, please send me a <a href="mailto:lio@hpe.com">message</a>.</p>https://developer.hpe.com/open-source-contributor-helps-istio-integrate-with-spire/https://developer.hpe.com/open-source-contributor-helps-istio-integrate-with-spire/Wed, 12 Oct 2022 15:24:29 GMT<p><img src="/img/max-lambrecht-istio-with-spire-1200-x-675.jpg" alt=""></p> <p>In this blog series, you’ll get to meet some of the open source experts working with the HPE Developer Community team. In this post, I’ll be interviewing Max Lambrecht, a software engineer working with the HPE Security Engineering team on Zero Trust Security. Max has been an open source contributor to both the SPIFFE/SPIRE and Istio projects; most importantly, enabling their integration.</p> <h2>How did you get involved with open source?</h2> <p>I first became involved with open source in 2018 when I joined Scytale and began working on SPIFFE and SPIRE. Scytale was subsequently acquired by Hewlett Packard Enterprise (HPE) in 2020. In 2021, there were some conversations within HPE engineering about integrating Istio with SPIRE, and a new team was created to research and implement that integration. With my background in SPIFFE and SPIRE, and the experience I had with Istio, the project looked like a good fit for me and a great opportunity to contribute to open source.</p> <h2>What exactly is Istio?</h2> <p>Istio is by far the most popular service mesh in the world. The Istio service mesh helps developers cope with the challenges of distributed or microservices architectures. It provides a dedicated infrastructure layer through which you can add capabilities, like observability, traffic management, and security, to modern microservice-based applications without adding them to your code.</p> <p>The Istio project was started by teams from Google and IBM in partnership with the Envoy teams from Lyft. It’s a huge project with many contributors, including several large companies, from around the world. In April of 2022, Istio was donated to the Cloud Native Computing Foundation, the open-source organization which SPIFFE and SPIRE are also part of.</p> <h2>Why was it so important for you to help integrate Istio with SPIRE?</h2> <p>Over the last four or five years, probably just after SPIRE started getting attention from the open-source community and companies started using it, there was interest in using SPIRE as the certificate provider for services running on an Istio deployment, enabling all the powerful capabilities of SPIRE for software identities within that environment.</p> <p>Service meshes are becoming more and more relevant as the Zero Trust security model gains traction and the perimeter-based model proves to be untenable in a world of hybrid clouds and increasingly complex microservices architectures.</p> <p>To this scenario, SPIRE brings strongly attested identities that are automatically delivered in heterogeneous environments. SPIRE enables multiple mechanisms to assert the identity of a machine and a service running on it. This adds a strong level of security to service meshes, as identities are key in securing communication across services. SPIFFE/SPIRE is a foundational building block of a zero-trust infrastructure.</p> <p>By bringing together these two technologies, Istio, a powerful service mesh, and SPIRE, a toolchain of APIs for delivering strong identities, I was able to help provide a way to greatly increase the security posture of services deployments.</p> <h2>How did the project progress?</h2> <p>The project involved several different activities that ranged from understanding Istio’s complex code base to testing several approaches to achieve integration with SPIRE, and then implementing a proof of concept of the selected approach, documenting it, and presenting it to people from the Istio community (Istio users, Istio maintainers). We discussed the potential solutions during Istio maintainers' calls and received useful feedback that helped us improve the solution. Once we came to an agreement on how to do it, we proceeded by putting together a pull request to submit the implementation into Istio. It was received with very good feedback, and finally accepted and merged. It was a big achievement for the team, for HPE, and for the SPIFFE/SPIRE and Istio communities. It’s something that people in both communities had been looking forward to for years.</p> <h2>What do you think the future will look like?</h2> <p>In my opinion, I think this will bring about a significant increase in the adoption of SPIFFE and SPIRE, as it will attract contributors to these projects, and they will continue to work on the development of tools to automate the deployment and configuration of SPIRE integrated with Istio.</p> <p>And given the <a href="https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/">recent graduation of SPIFFE and SPIRE as CNCF projects</a>, I foresee many will be attracted even more to adopting these projects.</p> <h2>Is there anything else you’d like to share with our readers?</h2> <p>The humble words of wisdom I’d like to leave with readers is this: Contribute back to the community, contribute to open source, find a project that you find interesting and/or one that can be useful to others. Look for tasks that you can take on and engage with the project’s community in Slack. You’ll grow as a professional, you’ll grow your network, you’ll grow your skills, and ultimately feel a huge sense of accomplishment. Sometimes you’ll have to overcome impostor syndrome. We all experience it from time to time. But that’s a good sign... it means that you are getting involved in things that can make you grow and learn.</p>https://developer.hpe.com/how-to-monitor-hpe-oneview-infrastructure-with-grafana-metrics-dashboards-and-influxdb/https://developer.hpe.com/how-to-monitor-hpe-oneview-infrastructure-with-grafana-metrics-dashboards-and-influxdb/Wed, 12 Oct 2022 10:00:35 GMT<style>ul li{ font-size:26px;padding-bottom: 0.5em;}</style> <style> i{ color:grey;font-family:'Courier New';font-size:22px; } </style> <p>The purpose of this blog post is to describe how to generate Grafana dashboards using InfluxDB and PowerShell scripts to monitor any HPE Compute infrastructure managed by HPE OneView.</p> <h1>Grafana Dashboards</h1> <p>IT infrastructure metrics visualization is critical for health monitoring, prediction, and capacity planning. It provides a powerful way of viewing infrastructure utilization, revealing issues and helping maintain uninterrupted services.</p> <p>Grafana’s time-series graphs are the perfect enabler for IT infrastructure optimization. They can assist administrators in monitoring temperature changes, network traffic performance, power consumption, and much more. They can be used to compare data over time to note trends and detect issues, allowing administrators to make any necessary adjustments and prevent downtime.</p> <p>The following picture shows a typical HPE infrastructure dashboard with Synergy frame, compute, and interconnect metrics:</p> <p><img src="/img/image001.png" alt=""></p> <h1>HPE OneView metric resources</h1> <p>Multiple metrics resources are supported by HPE OneView through the API, including CPU, memory, power consumption, temperature, health, capacity data for some resources like enclosures, interconnects and server hardware. Network statistics and network throughput are also available for each uplink and downlink ports in "interconnects" such as HPE Virtual Connect modules.</p> <p>The following table provides the resource metrics that are accessible through the HPE OneView RESTful API:</p> <table> <thead> <tr> <th align="left"><strong>Server hardware Metrics</strong></th> <th align="left"><strong>URI</strong></th> </tr> </thead> <tbody> <tr> <td align="left">Ambient Temperature</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=AmbientTemperature</em></td> </tr> <tr> <td align="left">Cpu Average Frequency</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=CpuAverageFreq</em></td> </tr> <tr> <td align="left">Cpu Utilization</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=CpuUtilization</em></td> </tr> <tr> <td align="left">Average Power</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=AveragePower</em></td> </tr> <tr> <td align="left">Peak Power</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=PeakPower</em></td> </tr> <tr> <td align="left">Power Cap</td> <td align="left"><em>/rest/server-hardware/{id}/utilization?fields=PowerCap</em></td> </tr> </tbody> </table> <table> <thead> <tr> <th align="left"><strong>Enclosures Metrics</strong></th> <th align="left"><strong>URI</strong></th> </tr> </thead> <tbody> <tr> <td align="left">Ambient Temperature</td> <td align="left"><em>/rest/enclosures/{id}/utilization?fields=AmbientTemperature</em></td> </tr> <tr> <td align="left">Average Power</td> <td align="left"><em>/rest/enclosures/{id}/utilization?fields=AveragePower</em></td> </tr> <tr> <td align="left">Peak Power</td> <td align="left"><em>/rest/enclosures/{id}/utilization?fields=PeakPower</em></td> </tr> </tbody> </table> <table> <thead> <tr> <th align="left"><strong>Interconnect Metrics</strong></th> <th align="left"><strong>URI</strong></th> </tr> </thead> <tbody> <tr> <td align="left">Statistics for the specified port name on an interconnect</td> <td align="left"><em>/rest/interconnects/{id}/statistics/portname</em></td> </tr> <tr> <td align="left">Interconnect cpu and memory utilization data</td> <td align="left"><em>/rest/interconnects/{id}/utilization</em></td> </tr> </tbody> </table> <p>HPE OneView metrics are enabled by default. For HPE Virtual Connect network statistics, the Utilization Sampling settings defined in the logical interconnect group controls the data collection rate and sample interval value. By default, the HPE Virtual Connect module sampling rate is 12 samples per hour, as shown in the following figure:</p> <p><img src="/img/image002.png" alt=""></p> <h1>InfluxDB Time-series database</h1> <p>My decision to use InfluxDB with Grafana metrics dashboards to monitor HPE OneView infrastructure was made for several reasons. First, InfluxDB can be installed on both Microsoft Windows and Linux, it is an open-source tool, and many useful modules are available to create entries in a database. Other options include Prometheus, which can also be used to collect and record measurements in real time.</p> <p>To collect HPE OneView metrics, I use a PowerShell script. This script collects utilization statistics of defined resources from the HPE OneView API periodically and continually transmits the metrics to InfluxDB via its REST API. The timestamped metrics data is saved into the InfluxDB time series database that Grafana uses to generate the graphs.</p> <p>The script is an independent process that must run continuously.</p> <p>The following diagram describes the different components of the solution:</p> <p><img src="/img/image003.png" alt=""></p> <h2>Pros and Cons about this solution</h2> <p>As with any solution, there are both Pros and Cons to using it.</p> <p>Pros:</p> <ul> <li>Enables time-series graphs for the HPE OneView Server hardware utilization statistics and HPE Virtual Connect modules utilization statistics</li> <li>Supports collecting metrics from any API (simply requires the appropriate PowerShell script for the collection)</li> <li>Provides a flexible solution using widely used and cross-platform scripting language</li> <li>Cross-platform support, all components can be installed on Microsoft Windows or Linux</li> </ul> <p>Cons:</p> <ul> <li>Requires development of PowerShell scripts if the examples provided do not meet your needs</li> <li>Requires in-depth knowledge of the language, API, authentication, and methods</li> </ul> <h1>Configuration</h1> <h2>Prerequisites</h2> <ul> <li>Grafana and InfluxDB must be installed, started, and enabled</li> <li>A firewall rule must be created to allow TCP port 8086 (used by InfluxDB API, by default)</li> <li>PowerShell Core for Linux must be installed if a Linux server is used to run the PowerShell scripts</li> <li>HPE OneView PowerShell library 6.60 or later must be used</li> </ul> <h2>Configure InfluxDB http authentication</h2> <p>By default, all security features are disabled in InfluxDB, so it is recommended to set up authentication by creating an <em>admin</em> user.</p> <p>To launch the influx command line interface (CLI), type:<br> > <i>influx</i></p> <p>Then create a user with an authentication password:<br> > <i><em>CREATE USER admin WITH PASSWORD 'P@ssw0rd' WITH ALL PRIVILEGES</em></i></p> <p>Once created, authenticate using:<br> > <i><em>auth</em></i><br> username: <i><em>admin</em></i><br> password: <i>********</i></p> <p>To enable the http authentication, you need to modify the InfluxDB configuration file. Go to the <strong>[http]</strong> section of <strong>/etc/influxdb/influxdb.conf</strong> and change the <strong>auth-enabled</strong> value to <strong>true.</strong></p> <p>[http]<br> auth-enabled = <i><em>true</em></i></p> <p>Once modified, restart the InfluxDB service:<br> > <i><em>sudo systemctl restart influxdb</em></i></p> <h2>PowerShell scripts for HPE OneView metrics collection</h2> <p>PowerShell scripts to collect metrics from the HPE OneView API can be found in my GitHub repository <a href="https://github.com/jullienl/HPE-Synergy-OneView-demos/tree/master/Powershell/Grafana%20Metrics">here</a>.</p> <p>Two distinct scripts are available, one for the interconnect metrics and one for compute, enclosure, and server profile metrics.</p> <p><img src="/img/image004.png" alt=""></p> <p>For each script, it is important to provide all the required variables for HPE OneView and InfluxDB.</p> <p><img src="/img/image005.png" alt=""></p> <p>For <em>Grafana-Interconnect-monitoring.ps1</em>, at the beginning of the script you need to provide the interconnect module names and port IDs that you would like to monitor using a hash table format:</p> <p><img src="/img/picture1.png" alt=""></p> <p>Note that the interconnect modules and port names can be found in the HPE OneView UI (in the Interconnects menu):</p> <p><img src="/img/image006.png" alt=""></p> <p>For <em>Grafana-Server_Enclosure-monitoring.ps1</em>, you need to provide at the beginning of the script, the resource names (server hardware or server profile or enclosure) and utilization (CPU, power, or temperature) that you want to monitor using a hash table format:</p> <p><img src="/img/image007.png" alt=""></p> <p>The names of the resources that need to be provided can be easily identified in the corresponding menus of the HPE OneView user interface.</p> <p>These scripts are written to collect metrics continually. They can be run in background on a Linux system using a crontab configuration or on a Microsoft Windows one, using Task Scheduler.</p> <h3>How to run the scripts on a Microsoft Windows machine?</h3> <p>The following commands can be used to schedule both jobs on a Microsoft Windows machine:<br> > <i><em>$trigger = New-JobTrigger -AtStartup -RandomDelay 00:00:30</em></i></p> <p>> <i><em>Register-ScheduledJob -Trigger $trigger -FilePath "...\Grafana-Server_Enclosure-monitoring.ps1" -Name GrafanaServerEnclosureMonitoring</em></i></p> <p>> <i><em>Register-ScheduledJob -Trigger $trigger -FilePath "...\Grafana-Interconnect-monitoring.ps1" -Name GrafanaInterconnectMonitoring</em></i></p> <p>You can check the job schedule by typing:<br> > <i><em>Get-ScheduledJob</em></i></p> <p><img src="/img/image008.png" alt=""></p> <p>Alternatively, launch Windows Task Scheduler, by pressing Windows + R keys on your keyboard to run a command, and enter:<br> > <i><em>taskschd.msc</em></i></p> <p><img src="/img/image009.png" alt=""></p> <p>As I am using an "at startup" trigger, it is required to restart the server in order to run the scripts.</p> <p>Restart the server and confirm that scripts are executed. Once restarted, you can run on a Microsoft Windows machine:<br> > <i><em>Get-job</em></i></p> <p><img src="/img/image010.png" alt=""></p> <h3>How to run the scripts on a Linux machine?</h3> <p>PowerShell can be installed on different Linux distributions today and it works perfectly, using a crontab configuration would make the scripts run in background. This allows using only one Linux machine to host all components (i.e., Grafana, InfluxDB and the execution of the PowerShell scripts).</p> <p>To learn more, you can refer to <a href="https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.2">this article</a>.</p> <p>The Linux repositories proposed by Microsoft can be found <a href="https://packages.microsoft.com/config/">here</a>.</p> <p><img src="/img/image011.png" alt=""></p> <p>On a RHEL/CentOS virtual machine, you can use the following steps:</p> <ul> <li>Add the Microsoft package repository:</li> </ul> <p>> <i><em>curl <a href="https://packages.microsoft.com/config/centos/8/prod.repo">https://packages.microsoft.com/config/centos/8/prod.repo</a> | sudo tee /etc/yum.repos.d/microsoft.repo</em></i></p> <ul> <li>Run the PowerShell installation:<br> > <i><em>yum install powershell</em></i></li> <li>Copy the script files to the Linux system and set the execution permission on both files:<br> > <i><em>chmod +x Grafana-Interconnect-monitoring.ps1</em></i><br> > <i><em>chmod +x Grafana-Server_Enclosure-monitoring.ps1</em></i></li> <li>Open the crontab configuration:<br> > <i><em>crontab -e</em></i></li> <li>Add two configurations, one for each script with a startup execution after a sleep time: <ul> <li><em>@reboot sleep 30 &#x26;&#x26; pwsh -File ".../Grafana-Interconnect-monitoring.ps1"</em></li> <li><em>@reboot sleep 30 &#x26;&#x26; pwsh -File ".../Grafana-Server_Enclosure-monitoring.ps1"</em></li> </ul> </li> <li>Restart the Linux machine to trigger the execution:<br> > <i><em>shutdown -r now</em></i></li> </ul> <h3>How to ensure that the scripts have started successfully?</h3> <p>First, to make sure that the scripts have started, you can check that the databases have been created using the InfluxDB tool.</p> <p>Connect to the server running InfluxDB and <em>launch the InfluxDB CLI</em>:<br> > <i><em>influx</em></i></p> <p>Authenticate using your InfluxDB credentials:<br> > <i><em>auth</em></i></p> <p>Display existing databases:<br> > <i><em>show databases</em></i></p> <p>If both databases defined in the script are listed, then both scripts have started successfully:</p> <p><img src="/img/image012.png" alt=""></p> <p>To verify that the metrics are successfully collected, open one of the databases and check the data content as shown below:<br> > <i><em>use ov_icm_db</em></i><br> > <i><em>show measurements</em></i></p> <p><img src="/img/image013.png" alt=""></p> <p>The measurements listed here correspond to the metrics (ports or resources) defined in the PowerShell scripts.</p> <p>Open one of the measurements to verify that the metric data is coming in:<br> > <i><em>SELECT * FROM "Frame3-Interconnect3-Q1"</em></i></p> <p><img src="/img/image014.png" alt=""></p> <p>The data shows that the collection of metrics has started and that everything is running fine.</p> <h2>Adding InfluxDB data sources in Grafana</h2> <p>Now that InfluxDB is configured and the data is collected, you can add the databases (created by the two scripts) to Grafana as new InfluxDB data sources.</p> <p>Once that is completed, any dashboard you create in Grafana will have access to the metric data collected.</p> <p>To launch the Grafana IU, open your web browser and navigate to <strong>http://&#x3C;grafana_IP or DNS name>:3000/</strong></p> <p>Note: The default HTTP port that Grafana listens to is 3000 unless you have configured a different port.</p> <p>Click on the gear icon on the side menu and click <strong>Add data Sources</strong>.</p> <p><img src="/img/image015.png" alt=""></p> <p>Select <strong>InfluxDB</strong> from the data source list.</p> <p><img src="/img/image016.png" alt=""></p> <p>For server and enclosure metrics, enter a data source name, e.g., <strong>InfluxDB-OV-Server-Metrics</strong></p> <p>Add the InfluxDB URL; by default it is <strong><a href="http://localhost:8086/">http://localhost:8086</a></strong></p> <p>Add the database name that you defined in <em>Grafana-Server_Enclosure-monitoring.ps1</em></p> <p><img src="/img/image017.png" alt=""></p> <p>Provide the InfluxDB admin username and password.</p> <p><img src="/img/image018.png" alt=""></p> <p>Once completed, click on the <strong>Save &#x26; Test</strong> button.</p> <p>If no error is returned, a "Data source is working" message is displayed.</p> <p><img src="/img/image019.png" alt=""></p> <p>Now repeat the same Add data source procedure for the Interconnect metrics, this time using the data source name <strong>InfluxDB-OV-Interconnect-Metrics</strong></p> <p><img src="/img/image020.png" alt=""></p> <p>Again, using the database name you defined in <em>Grafana-Interconnect-monitoring.ps1</em></p> <p><img src="/img/image021.png" alt=""></p> <p>Once this is done, click on the <strong>Save &#x26; Test</strong> button and make sure the data source is working.</p> <p><img src="/img/image022.png" alt=""></p> <p>You can then click on the <strong>Back</strong> button to return to the Data sources configuration window.</p> <p>You should now have two new Grafana data sources corresponding to the two InfluxDB databases generated by the two PowerShell scripts.</p> <p><img src="/img/image023.png" alt=""></p> <p>This completes the configuration of the Grafana data source.</p> <p>You are now ready to access your InfluxDB time series databases in Grafana.</p> <h2>Creating the Grafana dashboard</h2> <p>A Grafana dashboard can aggregate one or more panels using multiple sources. Thus, you can create a single dashboard with server/enclosure and interconnect metrics panels.</p> <p>Click on the Dashboards icon on the side menu and click <strong>New dashboard.</strong></p> <p><img src="/img/image024.png" alt=""></p> <p>Click on <strong>Add a new panel</strong> to create a panel to visualize the first HPE Virtual Connect module metrics.</p> <p>In Data source, select <strong>Influxdb-OV-Interconnect-Metrics</strong></p> <p><img src="/img/image025.png" alt=""></p> <p>For <strong>Query options</strong>, it is recommended to set <strong>5m</strong> as the minimum query interval to match the HPE OneView API metrics sampling value of the interconnect interfaces (see below).</p> <p><img src="/img/image026.png" alt=""></p> <p>Then, you need to define a query for each port you specified in the PowerShell script (in the <strong>$Ports</strong> variable) for this interconnect module name, as shown below:</p> <p><img src="/img/image027.png" alt=""></p> <p>For example, to set the Q1 port, you need to click on <strong>select measurement</strong> next to <strong>FROM</strong>:</p> <p><img src="/img/image028.png" alt=""></p> <p>A list of all the measurements available in the database is displayed in the drop-down menu, as seen below:</p> <p><img src="/img/image029.png" alt=""></p> <p>Select <strong>Q1,</strong> then click on <strong>field (value)</strong> in the <strong>SELECT</strong> row to select the value you want to display:</p> <p><img src="/img/image030.png" alt=""></p> <p>A list of all measurement values recorded in the database displays in the drop-down menu:</p> <p><img src="/img/image031.png" alt=""></p> <p>Select <strong>receiveKilobytesPerSec</strong> for example.</p> <p>You can then set the alias as <strong>Q1</strong> to replace the default metric name and get a clear legend label on the chart.</p> <p><img src="/img/image032.png" alt=""></p> <p><img src="/img/image033.png" alt=""></p> <p>The metric points should already appear on the graph.</p> <p>Further, you can specify a panel title in the right-side menu using the interconnect name you selected, like <strong>VC Frame3-Interconnect3</strong> (in our example).</p> <p><img src="/img/image034.png" alt=""></p> <p>And for a better visualization, you can select <strong>Always</strong> for <strong>Connect null values</strong> and <strong>Never</strong> for <strong>Show points</strong> in the <strong>Graph styles</strong> section.</p> <p><img src="/img/image035.png" alt=""></p> <p>And finally, set the unit data rate you selected in the <strong>SELECT</strong> row. Scroll down to the <strong>Standard options</strong> section and in <strong>Unit</strong> , select <strong>Data rate</strong> and click on <strong>kilobytes/sec</strong>.</p> <p><img src="/img/image036.png" alt=""></p> <p>Rendering should display as follows:</p> <p><img src="/img/image037.png" alt=""></p> <p>This completes the configuration of the first port query.</p> <p>You will need to click on the <strong>+ Query</strong> button for the other ports and repeat the same query configuration (as previously described) for all the ports defined in the PowerShell script.</p> <p><img src="/img/image038.png" alt=""></p> <p><img src="/img/image039.png" alt=""></p> <p>Once all queries have been defined, you can save the panel using the <strong>Save</strong> button in the upper right corner. Type a name for the newly created dashboard like <strong>HPE OneView Metrics</strong>.</p> <p><img src="/img/image040.png" alt=""></p> <p>You can now duplicate this panel to create another one for the second HPE Virtual Connect module. Click on the panel's context menu, select <strong>More</strong> , then <strong>Duplicate</strong>.</p> <p>Click on the duplicated panel's context menu then select <strong>Edit</strong>.</p> <p>Change the panel title with the name of your second Virtua Connect module like <strong>VC Frame3-Interconnect6 Metrics</strong></p> <p>Then modify each query by selecting the ports on the second interconnect module that you want to monitor.</p> <p><img src="/img/image041.png" alt=""></p> <p>Click <strong>Save</strong> then <strong>Apply</strong>.</p> <p>The dashboard now displays two panels, one for each HPE Virtual Connect module that was defined in <em>Grafana Interconnect monitoring.ps1</em></p> <p><img src="/img/image042.png" alt=""></p> <p>The next step consists in creating panels to display Compute and Frame metrics.</p> <p>Click on the <strong>Add panel</strong> button on the upper bar and select <strong>Add a new panel</strong>.</p> <p>Select the <strong>InfluxDB-OV-Server-Metrics</strong> data source, then select the resource you want to monitor.</p> <p><img src="/img/image043.png" alt=""></p> <p>Select the measurement you need:</p> <p><img src="/img/image044.png" alt=""></p> <p>Then:</p> <ul> <li>Add a panel title</li> <li>Use <strong>5m</strong> for the Min interval query options</li> <li>Select the graph styles options <ul> <li>Connect null values: <strong>Always</strong></li> <li>Show points: <strong>Never</strong></li> </ul> </li> <li>Select the correct unit corresponding to the measurement type <ul> <li>Power: Energy / <strong>Watt</strong></li> <li>Temperature: Temperature / <strong>Celsius</strong></li> <li>CPU: simply type <strong>GHz</strong></li> </ul> </li> <li>Add meaningful alias names to reflect reported measurement</li> </ul> <p>Add any additional measurement you need, using another query.</p> <p>Here is an example of a frame panel with power and temperature metrics defined:</p> <p><img src="/img/image045.png" alt=""></p> <p>For panels with two different types of measurements (Watt and Celsius) as seen above, you need to define two Y-axes. One for the temperature and one for the power consumption.</p> <p>Select <strong>Overrides</strong> at the top of the right-side menu, then click on <strong>Add field override</strong> :</p> <p><img src="/img/image046.png" alt=""></p> <p>After that, select the following override properties:</p> <ul> <li>Fields with name: <strong>Temperature</strong></li> <li>Axis placement: <strong>Right</strong></li> <li>Standard options > Unit: Energy / <strong>Watt (W)</strong></li> </ul> <p><img src="/img/image047.png" alt=""></p> <p>When completed, the panel displays the two Y-axis:</p> <p><img src="/img/image048.png" alt=""></p> <p>You can then click on <strong>Save,</strong> then <strong>Apply</strong> buttons to return to the Grafana dashboard. An additional panel to monitor the temperature and power consumption of a frame is displayed.</p> <p><img src="/img/image049.png" alt=""></p> <p>Next, you can add as many panels as you have resources defined in your PowerShell scripts.</p> <p>This concludes this blog post. I hope you find it useful and should you have any feedback, please send me a <a href="mailto:lio@hpe.com">message</a>.</p>https://developer.hpe.com/mellanox-switch-automation-welcome-to-2012/https://developer.hpe.com/mellanox-switch-automation-welcome-to-2012/Fri, 07 Oct 2022 22:54:50 GMT<p><img src="/img/rick-kaufmann-network-automation-1200-x-675.jpg" alt=""></p> <p>The Mellanox 2010 ethernet switch is a curious thing. It is only half the width of a normal 19-inch rack mount-switch. It has 18 10/25G ports. This is perfect when you need two switches for high-availability deployments but only a handful of ports are required. The fact that they are only half-width you only need 1U of a data center rack. I believe because of its features and price, it makes an excellent candidate for HPE dHCI deployments.</p> <p>I was recently involved with a proof-of-concept for a very large customer who would be using Mellanox switches in over 200 locations. My job was to configure the network to support all the different connections the deployment required. I quickly calculated that 400 different switches would need to be configured. That would be a lot of typing on a terminal using the switch command line. Fortunately, I love to automate things and maintain consistency in all my switch configuration files.</p> <p>You never know when you'll run into the same issue, so let me help you out here by showing you how I handled it and gifting you the application I developed to take care of instances like this. First, let's take a look at the architecture so you can better understand how I did it."</p> <p><img src="/img/network_760x435.png" alt="" title="network"></p> <p>Within the architecture, there are only five networks; the iLO (or Out-of-Band management), the VMware Management network, the home of the vCenter, a VM production network, and two ISCSI data networks</p> <p>My ultimate goal was to develop an application to speak directly to the switches via the REST API. Without having access to the hardware, it was a bit of a challenge to develop the Python bindings. To solve this challenge, I would need to create a text file which contains the configuration statements for the switch. Lucky for me, this is super easy using Flask and Jinja2 templates.  </p> <p>Looking at the switch configuration file, most of the information is static. Very little of it is dynamic. There are only 17 variables that need to be collected before they can be applied to a Jinja2 template. So, the solution for me was to use a Flask form and pass the variables over to Jinja2 to create the necessary configuration file. </p> <p>Writing the Python code can be quickly accomplished if you can see the results of the API requests. It contains valuable information that gives guidance as to the specific structure of the information the switch sends back. Once this information is understood, Flask can process the information and apply the Jinja2 template. Here's a handy link to the Flask documentation.<a href="https://flask.palletsprojects.com/en/2.2.x/">https://flask.palletsprojects.com/en/2.2.x/</a></p> <p>Flask is a web server written in python, it makes it super easy to build elegant user interfaces. Jinja2 is a templating solution that allows placing variable information in a text file. Jinja2 will look for the “double mustache” or “{{ x }}”. This tells JInja2 where the variable information gets placed in the file. If I had a variable like “ip_address” which was 137.162.0.98 and a configuration like this: ip address {{ ip_address }}, Jinja2 would resolve the text to something like this; ip address 137.162.0.98.  You can learn more about Jinja2 by following this link: <a href="https://jinja.palletsprojects.com/en/3.1.x/"></a><a href="https://svn.python.org/projects/external/Jinja-2.1.1/docs/_build/html/index.html">https://svn.python.org/projects/external/Jinja-2.1.1/docs/_build/html/index.html</a></p> <p><img src="/img/form_597x384.png" alt="" title="Mellon"></p> <p>I timed it and it only takes about three minutes to fill out the form and hit the generate button. The application I designed generates two configuration files, which can be secure copied to the Mellanox switch.</p> <p>With this application there are two options; one will generate a pair of configuration files, and the other will generate configuration files for many different sites or locations.</p> <p>In the main directory of the application, there is a comma separated variable file, known as a CSV. The title is mellanox_config.csv. Using Microsoft Excel, open the CSV file and you will see three columns. By changing the values in column “C”, you can generate a pair of configuration files. Below is an example of the contents of the file. At first it won't look as nice as the example. You will have to stretch out the columns and shade them to your liking (totally optional)</p> <p><img src="/img/pair.png" alt="" title="Pair of switches"></p> <p>If there are many sites to generate, then the bulk file can be used.</p> <p><img src="/img/bulk.png" alt="" title="Bulk sites"></p> <p>A video tutorial can be found on my personal blog here: <a href="https://www.techworldwookie.com/automation-for-the-sake-of-automating/"></a><a href="https://www.techworldwookie.com/automation-for-the-sake-of-automating">https://www.techworldwookie.com/automation-for-the-sake-of-automating</a></p> <p>The application runs in Docker and docker-compose needs to be installed on the host system. I just use Docker Desktop and everything becomes quite portable. The application can be found over on my GitHub account, here: <a href="https://github.com/xod442/Mellon"></a><a href="https://github.com/xod442/Mellon">https://github.com/xod442/Mellon</a></p> <p>If I can ever get access to a physical switch, I will finish the API version of this application. Until then, I give you Mellon. You can still do it the hard way if you like, but I find using Mellon gives me more time to code.</p>https://developer.hpe.com/2022-October-07/https://developer.hpe.com/2022-October-07/Fri, 07 Oct 2022 13:00:00 GMThttps://developer.hpe.com/catch-hpe-at-kubecon-na-2022/https://developer.hpe.com/catch-hpe-at-kubecon-na-2022/Tue, 04 Oct 2022 15:47:12 GMT<p>This October 24-28, the Cloud Native Computing Foundation will once again be hosting its flagship conference – <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/">KubeCon+CloudNativeCon</a> – as a hybrid event. Take this opportunity to meet with thousands of cloud-native leaders, including gold-sponsor Hewlett Packard Enterprise (HPE), as the community gathers to further the education and advancement of cloud native computing. This is an important event for HPE as it offers us the opportunity to further educate the community on our expertise and support of open source and cloud native technologies.</p> <h3>Have some Kubist fun and learn something in the process</h3> <p>HPE subject matter experts representing SPIFFE/SPIRE, HPE Storage, Determined AI, HPE Ezmeral, Zerto and HPE PointNext will be available in person to answer questions and offer demos at Booth #G13. Come to:</p> <ul> <li> <p>Learn about the cloud-native data infrastructure portfolio from HPE and how to accelerate delivery of the most demanding stateful workloads on Kubernetes.</p> </li> <li> <p>Get advice on advanced use cases involving data management, business transformations, and why persistent storage matters for app developers.</p> </li> <li> <p>Discuss fully open-source machine learning (ML) model training. Learn how to get started, discuss current trends in the ML ecosystem, and join the community.</p> </li> <li> <p>Understand how HPE Pointnext can help you in your multi-cloud journey and transform monolithic to containerized, cloud-native workloads.</p> </li> </ul> <p>You’re going to want to stay and pay attention to these talks, because our magical master of ceremonies will be quizzing you at the end. Be the first one to correctly answer the questions and you’ll come away with either a Rubik’s Cube or one of our cool wooden puzzles.</p> <p>Each day, we’ll also be giving away a Raspberry Pi Starter Kit. Booth visitors who get scanned in the booth will be given a raffle ticket and, at the end of the last presentation of each day, we will draw the winner of the prize.</p> <h3>Join us in celebrating the graduation of SPIFFE/SPIRE!</h3> <p>Contributing alongside the many leading technology companies who were a part of this project, like VMWare, Uber, ByteDance, Anthem, Transferwise, IBM, etc., HPE is really excited that SPIFFE/SPIRE has just been designated a Graduated project of CNCF. Leading and influencing meaningful innovation in the Open Source community is important to us, as it provides key capabilities that our customers desire and helps us design solutions that meet their needs.</p> <p>Make sure you stop by the booth to talk with some of our SPIFFE/SPIRE experts, like Andrew Jessup, to learn more about what it can do for you. We’ll be giving out several copies of <em><strong><a href="https://thebottomturtle.io/Solving-the-bottom-turtle-SPIFFE-SPIRE-Book.pdf">Solving the Bottom Turtle</a></strong></em> as part of our celebration.</p> <h3>Ahoy, developers… join virtually and seek out your cache!</h3> <p>As part of our virtual presence, we’ll be hosting several Office Hours designed for developers, data scientists, and data/ML engineers to hear more about what HPE offers for them. <a href="https://kubecon-cloudnativecon-na.com/virtual-exhibitor/?v4b4342b0f72f3260e37d74de68eab433fee0c641d933e76d52be7eb34b211371656f07b5a54b2e522db3ac7b27c7d555=068E4EDFD2D581C2C838CEEFD5F655754F523A6864305A1184BF01DD527B60FB0740D3F17B594A1732C891BBBA3213A8">Tune in</a> for:</p> <ul> <li> <p>The HPE Developer Community team’s session on <em><strong>What? Not yet an HPE Developer Community member? There’s treasure to be found!</strong></em>, guiding you through the rich set of resources HPE offers developers through our web portal.</p> </li> <li> <p>The HPE Ezmeral engineering group’s talk on <em><strong>HPE Ezmeral Early Access</strong></em>, a beta program that allows developers to try new products before they’re released. With exemplary training and support provided by HPE, developers will get hands-on experience with new features and capabilities.</p> </li> </ul> <p>The HPE Developer Community team will also be hosting a Virtual Treasure Hunt. This is a popular game that we’ve held at several past events that gets updated with new questions every time. Twelve players who complete the Virtual Treasure Hunt will be selected as winners of a $50 e-gift card to be redeemed at the HPE-branded merchandise e-store. Completing the Virtual Treasure Hunt requires participants to explore the HPE Developer Community’s rich ecosystem and submit their entry via the <a href="https://bit.ly/kubecon-na-2022-hpedev-treasure-hunt">Virtual Treasure Hunt form</a> during the event. Refer to the giveaway <a href="https://developer.hpe.com/hackshack/treasurehunt-terms-conditions/">Terms and Conditions</a> for complete program details.</p> <h3>We’ve always got something going on in this space</h3> <p>HPE has been a proud sponsor of KubeCon+CloudNativeCon for years. We’re excited to be able to share with you the key activities HPE is involved with in the areas of Kubernetes and other open source technologies. Make sure you stop by our booth, either virtually or physically. We’re always happy to engage with you.</p>https://developer.hpe.com/hpe-greenlake-delivers-nvidia-ai-as-a-service/https://developer.hpe.com/hpe-greenlake-delivers-nvidia-ai-as-a-service/Thu, 29 Sep 2022 17:14:38 GMT<p>Building the infrastructure required for AI solutions can be an expensive and daunting undertaking. That’s why an as-a-service option that supports customer flexibility makes so much sense. With NVIDIA AI Enterprise now available through HPE GreenLake in select countries, developers can quickly get started building high-performing AI solutions. They can do so easily by using the NVIDIA AI Enterprise software suite – an end-to-end, cloud-native suite of AI and data analytics software that can be deployed anywhere, from the data center to the cloud.</p> <p>Announced this past June, the service offers NVIDIA AI Enterprise on NVIDIA-Certified HPE ProLiant DL380 and DL385 servers running VMware vSphere with Tanzu on a pay-per-use basis that can be tailored to the needs of the organization. Customers can select from predefined packages that include management of infrastructure and software and add on optional AI advisory and solution workshops.</p> <p>Check out this blog post by Nicola Sessions, senior manager of product marketing for AI software at NVIDIA, entitled <a href="https://blogs.nvidia.com/blog/2022/06/28/hpe-greenlake-edge-to-cloud/">NVIDIA Teams with HPE to Take AI From Edge to Cloud</a>, to learn more about the NVIDIA AI Enterprise on HPE GreenLake solution.</p>https://developer.hpe.com/beta-test-our-new-hpe-greenlake-for-data-fabric/https://developer.hpe.com/beta-test-our-new-hpe-greenlake-for-data-fabric/Fri, 23 Sep 2022 20:32:16 GMT<p>The HPE Ezmeral Early Access program is a new program designed to ensure new and exciting products are constantly developed. Through this beta program, we are giving you the opportunity to test out products and provide feedback to the engineering team. With exemplary training and support provided by HPE, you’ll get hands-on experience with new features and capabilities. One of our newest, exciting projects revolves around HPE Ezmeral Data Fabric, as it is about to start a new journey in its delivery of an enterprise-grade data fabric for hybrid and multi-cloud enterprises, bringing enterprise-grade data fabric to the as-a-Service world.</p> <p><strong>Chasing that single-source-of-truth</strong></p> <p>As we look back on software technology evolution, particularly around the area of analytics, we see a major challenge and theme recurring around the establishment of a single-source-of-truth. Firstly, there’s EIS (Executive Information Systems), which were usually OLAP (online analytical processing) cubes (multi-dimensional arrays of data), professing a single point to capture, aggregate and share analytic data. Then, when OLAP cube scalability became too limiting, ROLAP (Relational Online Analytical Processing) became the preferred solution. By definition, ROLAP brought the relational database into analytic solutions and added technologies and terms such as query and reporting, the data mart, and data warehouse (with their star and snowflake schemas), all including the single-source-of-truth value proposition in their range of benefits to the enterprise.</p> <p>Next came the data lake, predominantly driven by HDFS (Hadoop Distributed File System) taking mass, unstructured data storage to significantly higher scales. Structured use cases started to fold into HDFS with new iterations of columnar stores providing scale, with some structure, typically over HDFS / data lakes.</p> <p>While this was going on, clouds were forming – however clouds posed a greater threat to the single-source-of-truth. Clouds wanted your data to reside outside your premises and made it hard (and/or expensive) to share. For enterprises that deploy across multiple clouds, and likely on-prem and at the edge, this meant new islands of data and multiple versions of the truth.</p> <p>Today we have returned to the single data lake as one solution to the many versions of the truth issue. The single date lake is reminiscent of past approaches: bring all data into one location to standardize, normalize, aggregate and then distribute the results. A second approach, a federated approach, is to leave the data in place and retrieve only what is requested when it is requested, possibly with some query optimization and intelligent caching thrown in to minimize the impact on performance.</p> <p><strong>Solving the issue with data fabric</strong></p> <p>HPE Ezmeral Data Fabric started and grew up in the big data world. It has long been trusted as an enterprise-grade data fabric managing data at scale for mission-critical applications across industries and use cases. Users today enjoy a single global name space that provides one point to review all attached data assets across data fabric deployments. Multi-modal data access delivers on the need to build an application once and deploy anywhere there is a data fabric. In addition, the multi-modal nature of the data fabric allows the development of stream-based, object-based apps and file-based apps against the same data without duplicating that data – very much a single version of the truth. Additionally, and if needed, the data fabric is good at moving data securely and optimally from deployment to deployment. </p> <p>The upcoming release of HPE GreenLake for Data Fabric brings the same enterprise-grade data fabric to the as-a-Service world. The installation, configuration and management of deployments is no longer a customer responsibility. The data fabric comes installed, running and is maintained by HPE. Customer-facing management tasks are elevated to the level of creation of storage buckets and volumes, managing user quotas, sharing of entry points with fellow application developers and overall monitoring of usage, all driven through a central UI that can manage all your HPE GreenLake for Data Fabric deployments.</p> <p><strong>Beta test it yourself</strong></p> <p>With the upcoming HPE GreenLake for Data Fabric beta, we are targeting use cases in the hybrid and multi-cloud world, for example those who want to write an application once and run it on any deployment target using a consistent set of multi-modal APIs. It is for those who want to build a data plane that gives a consistent view of data assets across clouds via a single global name space and those that want to synchronize data across clouds securely and efficiently. The as-a-Service tooling allows the management of all clusters from a single interface, further simplifying management of your data plane. We will post additional resources on the <a href="https://developer.hpe.com/">HPE Developer Community portal</a> that show example use cases. We will also provide you with <a href="https://hpedev.slack.com/archives/C044E295003">a Slack channel</a> where you can connect with us to answer any questions you have.</p> <p>HPE will begin the beta of this new service in November this year (2022). Register today to learn more about how you can participate in this program. Beta testers, can deploy initially into AWS then other public clouds as well as on-prem hardware. Join us to learn more in our upcoming <a href="https://hpe.zoom.us/webinar/register/3716641878854/WN_xLR2ynonSi6SojUswkVmRw">HPE GreenLake for Data Fabric webinar</a>. We look forward to learning about your use cases.</p>https://developer.hpe.com/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/https://developer.hpe.com/meet-open-source-grommet-lead-developer-and-architect-shimrit-yacobi/Thu, 22 Sep 2022 18:09:48 GMT<p><img src="/img/shimi-yacobi-375.jpg" alt=""></p> <p>In this open source contributor series, you’ll get to meet members of the HPE Developer Community who aim to advance the way people live and work by contributing to open source projects.</p> <p>A software engineering graduate from the Middle East, Shimrit Yacobi has a long history with <a href="https://v2.grommet.io/">Grommet</a>, an open source project led by HPE. Grommet is a React-based component library and framework that helps developers and designers create web applications. Shimi, as she prefers to be called, describes herself as a passionate React.js developer focused on making the vision of a design-driven user experience a reality.</p> <p>In addition to her numerous code contributions, including building and designing new components and features for the library, Shimi was the lead developer and architect and spent many years acting as the Grommet Community Manager. As a part of this role, she connected with HPE product teams, developers from the community, designers, and UX research to understand requirements and use cases. She continues to work with them today to augment the Grommet framework to meet customer needs while maintaining a tidy and clean API, architecture and implementation.</p> <h2>In your own words, can you define Grommet for me?</h2> <p><a href="https://v2.grommet.io/">Grommet</a> is an open source UI development and design framework that simplifies the way web applications are built by providing a package of commonly used interface elements from which developers and designers can choose to use. It’s both a framework and a component library.</p> <h2>Can you tell me a little more about your role in regards to this project?</h2> <p>I tend to live and breathe this framework. I work on ideating and developing new components, solving complex problems with simple solutions, and collaborating with teams on development and architecture requirements. An important aspect of my job is acting as a mentor; encouraging HPE and outside developers and designers to join the Grommet community, embrace the collaborative culture, and be part of a family while making contributions and providing constant feedback as we grow. I also provide in-house and community consulting to Grommet users on best practices, solution optimization, and how to build responsive applications using Grommet features like accessibility and modularity. This close collaboration with the Grommet Community ensures that Grommet is always on the cutting edge of technology.</p> <h2>How did you get involved with Grommet?</h2> <p>I first encountered Grommet working as a developer in the HPE Hyperconverged systems group. I was on one of the development teams that had been encouraged to use Grommet. I was able leverage the framework quite well to win an HPE Hackathon project, while also contributing to the open source code. The team was a lot of fun to work with and encouraged me to work with them.</p> <h2>I hear that Grommet is pretty popular. Just how popular is it and why do you think that is?</h2> <p>Grommet has over 120,000 downloads a month, over 300 contributors, and continues to be adopted as one of the premier ReactJS frameworks both within HPE and throughout the open source community. Another indication of its popularity is that we have more than 4,500 members on our <a href="https://slack-invite.grommet.io/">Grommet Slack Channel</a>.</p> <p>I think one of the things that makes Grommet so popular is that it simplifies the way web applications are built. Grommet does this by providing a package of commonly used interface elements from which developers and designers can choose to use. It provides powerful themes and tools that allow web developers to customize the color, type, component elements and layout according to their needs. It also provides support for Web Content Accessibility Guidelines (WCAG), making it great for accessibility.</p> <h2>What other open source projects have you been involved with?</h2> <p>I was the lead developer and architect for the HPE Design System, an open source library of elements consisting of working code, best practices, and design resources. It features human-centered guidelines, and engages with a vibrant community of contributors. One of the things I like most about it is that it enables experiences to be crafted with uncompromising integrity.</p> <p>One of the things that makes the HPE Design System so powerful is that it has what we refer to as a human-centered design, crafted upon user research and the real needs of customers. In addition, it’s a system designed so that all those working on it can collaborate really well. It provides a common language for designers, developers, and stakeholders, allowing everyone to really understand each other, allowing for more precise specification of UX requirements.</p> <h2>Is there anything else you’d like to share with our readers?</h2> <p>Advocating for women in tech and overall diversity are passions of mine. I was the HPE Employee Resources Group (ERG) Co-Chair of HPE Northern Colorado Women’s Network, a place where I could contribute to improve cultural agendas for the company and make HPE a better place to work. I am also an avid facilitator of the Culture Catalyst group for Diversity and inclusion.</p> <p>To learn more about the open source projects that are of particular interest to HPE, please visit our <a href="https://www.hpe.com/us/en/open-source.html">website</a>. Interested in exploring what HPE offers for developers and data scientists? Check out our <a href="https://developer.hpe.com/">HPE Developer Community site</a> for a ton of articles, workshops, tutorials, and other resources.</p>https://developer.hpe.com/chapel-scales-new-heights-with-1-28-0/https://developer.hpe.com/chapel-scales-new-heights-with-1-28-0/Tue, 20 Sep 2022 12:29:26 GMT<p>Exciting news for the Chapel community this month – Chapel 1.28.0 has just been released! Chapel, a productive parallel programming language that scales from laptops to supercomputers, now features improved GPU support, a new Communication module that supports low-level puts and gets, and improvements to the interface and other aspects of the code. This open source project is also being highlighted this month at the AnitaB.Org 2022 Grace Hopper Celebration where Michelle Strout and Lydia Duncan of the HPE High Performance Computing and AI group ran a Choose Your Own Adventure project during the Open Source Day, where developers could come to learn more about Chapel and how to contribute to the project.</p> <p>As Chapel technical lead Brad Chamberlain pointed out in the announcement, many of the changes in Chapel 1.28.0 have been done to improve the language and libraries, thus stabilizing core features in preparation for its forthcoming 2.0 release. As an example, working with non-default numerical types, or combinations of numerical types, has been made simpler, more intuitive, and more consistent with other scientific languages. Implicit conversion and resolution rules have been updated to better preserve bit-widths and ensure operations combining disparate types generate natural, rather than surprising, results or any types of errors.</p> <p>Adding to the GPU support that’s been added in the past few releases, Chapel 1.28.0 now permits a wider variety of parallel loop computations to be generated as GPU kernels. A new GPU utility module has also been added, providing the ability to assert that a computation will execute on the GPU as intended (or generate an explanation as to why it would not). The module also adds support for printing simple messages from within GPU loops.</p> <p>A new Communication module has also been introduced in Chapel 1.28.0. This module supports put/get routines to perform explicit data transfers between locales (e.g. nodes in a cluster or on a supercomputer) for users who want to exert more control over low-level communication in their code. You’ll also find many other notable improvements to the ‘chpldoc’ and ‘mason’ tools, which respectively create documentation from commented Chapel code and provide packaging support for Chapel.</p> <p>Here are some of the main highlights of the release:</p> <ul> <li>Significantly improved behavior for numerical ops on small/mixed types</li> <li>A new Communication module for performing low-level puts/gets</li> <li>Expanded idioms that can run on GPUs and a new GPU utility module</li> <li>Improvements to ‘chpldoc’ and ‘mason’</li> <li>Simplified and improved selection between overloaded routines</li> <li>Stabilized language and library improvements</li> <li>Reduced average compilation times using the LLVM support library</li> <li>Improved portability to ARM-based Macs</li> </ul> <p>For complete details on this new release, please refer to the following resources:</p> <ul> <li><a href="https://chapel.discourse.group/t/announcing-chapel-1-28-0/16129">Release announcement</a></li> <li><a href="https://github.com/chapel-lang/chapel/blob/release/1.28/CHANGES.md">Release changes list version 1.28.0</a></li> <li><a href="https://twitter.com/ChapelLanguage">Chapel Language Twitter page</a></li> <li><a href="https://www.facebook.com/ChapelLanguage/">Chapel Programing Language Facebook page</a></li> </ul>https://developer.hpe.com/2022-September-08/https://developer.hpe.com/2022-September-08/Thu, 08 Sep 2022 13:00:00 GMThttps://developer.hpe.com/attend-nvidia-gtc-–-the-developer-conference-for-the-era-of-ai/https://developer.hpe.com/attend-nvidia-gtc-–-the-developer-conference-for-the-era-of-ai/Thu, 25 Aug 2022 17:01:25 GMT<p>Interested in artificial intelligence and high performance application development? Hewlett Packard Enterprise (HPE) is as well and is excited to announce its participation as a Diamond sponsor of NVIDIA GTC, September 19-22. GTC is the developer conference for the era of AI and the metaverse, exploring development, design and simulation, robotics, and applied real-time technologies. It’s an online event where students, developers, researchers, creators, IT decision-makers, and business leaders gather to learn how to shape our world with the power of AI, computer graphics, data science, and more.</p> <p><a href="https://www.nvidia.com/gtc/?ncid=ref-spo-810249">Register for NVIDIA GTC</a> now.</p> <p>Hear from industry leaders from around the world through over 200+ curated conference talks—including technical, business strategy, and getting-started sessions designed to accelerate tech careers. GTC will provide networking opportunities to connect with HPE and NVIDIA experts.</p> <h3>Accelerate your AI with these HPE-sponsored GTC sessions</h3> <p>Catch our two HPE-sponsored GTC sessions, which focus on our partnership to bring you advanced technologies to accelerate your AI efforts. Find and register for them in the <a href="https://www.nvidia.com/gtc/session-catalog/?tab.catalogallsessionstab=16566177511100015Kus#/">session catalog</a>. NVIDIA CEO and founder Jensen Huang’s inspiring keynote is also something not to miss, so make sure you attend this event on September 20 at 8:00 a.m. PT. Click <a href="https://www.nvidia.com/en-us/gtc/keynote/">here</a> to watch once it’s available. (Registration isn’t required to view the keynote.)</p> <h4>Accelerate AI by teaming up with the right crew and the best infrastructure (<a href="https://register.nvidia.com/flow/nvidia/gtcfall2022/attendeeportal/page/sessioncatalog?tab.catalogallsessionstab=16566177511100015Kus&#x26;search=A41399">A41399</a>)</h4> <p>Hear from Evan Sparks, Vice President of AI and HPC at HPE, as he describes the complete canonical stack required to accelerate AI–starting with the best-of-breed servers and GPU accelerators and optimized development stacks, such as HPE Machine Learning Development Environment. In this on-demand session, you’ll learn about tips and pitfalls to avoid that will help you speed up your AI projects. You’ll also hear about Champollion, an AI supercomputer built by HPE in collaboration with NVIDIA, which was recently added to the TOP500 list of the most powerful non-distributed computer systems.</p> <h4>Accelerating AI Edge to Cloud with NVIDIA and HPE GreenLake (<a href="https://register.nvidia.com/flow/nvidia/gtcfall2022/attendeeportal/page/sessioncatalog?tab.catalogallsessionstab=16566177511100015Kus&#x26;search=A41396">A41396</a>)</h4> <p>In this Simulive session, HPE subject-matter experts will cover how to combine NVIDIA with HPE GreenLake to accelerate AI edge-to-cloud services. Join in and hear Jeroen Bronkhorst, HPE GreenLake Cloud Services Portfolio Manager for AI/ML, and Terry Chiang, Data Scientist / MLOPs Product Manager for HPE Ezmeral, discuss today’s increased focus on ML model life-cycle management and the new hardware accelerators and optimized open-source tooling that’s popping up at every step of the AI model life cycle. You’ll learn about the latest innovations in data acquisition and preparation, model development and training, and model deployment and inference enabled by HPE, NVIDIA, and partner ecosystems that can be delivered as an on-premises cloud service through HPE GreenLake.</p> <h3>Attend our Watch Party!</h3> <p>NVIDIA and HPE will also be teaming up for a <a href="https://www.nvidia.com/gtc/session-catalog/?search=WP41326b&#x26;search=WP41326b%2C+WP41326b&#x26;tab.catalogallsessionstab=16566177511100015Kus#/session/1660839912631001SdVF">Watch Party</a>, an interactive session hosted by NVIDIA to interact with other attendees via chat while you watch a GTC session together. Join Ilene Carpenter, Earth Sciences Segment Manager at HPE, and Karthik Kashinath, AI-HPC and engineering lead of Earth-2 at NVIDIA, as they host [NVIDIA's Earth-2: Progress and Challenges Towards Building a Digital Twin of the Earth for Weather and Climate [A41326]](<a href="https://urldefense.com/v3/**https:/www.nvidia.com/gtc/session-catalog/*/session/1657569242361001HePe**;Iw!!NpxR!iaGDB93N3IrNqbhcsQjMZFXHeOcgNN7I_xUjpU7_9Eh33OXYyNZxg-m4prRZJ7moWNukqYfYTiOFxe5qb25cmQ$">https://urldefense.com/v3/**https:/www.nvidia.com/gtc/session-catalog/*/session/1657569242361001HePe**;Iw!!NpxR!iaGDB93N3IrNqbhcsQjMZFXHeOcgNN7I_xUjpU7_9Eh33OXYyNZxg-m4prRZJ7moWNukqYfYTiOFxe5qb25cmQ$</a>). The <a href="https://www.nvidia.com/gtc/session-catalog/?search=WP41326b&#x26;search=WP41326b%2C+WP41326b&#x26;tab.catalogallsessionstab=16566177511100015Kus#/session/1660839912631001SdVF">Watch Party</a> will be held on Thursday, September 22, at 4:30-6:00 p.m. CET. Hear about NVIDIA’s recently launched Earth-2 initiative, which aims to build digital twins of the Earth to address one of the most pressing challenges of our time – climate change. Join us and learn how NVIDIA’s Earth-2 digital twin continues to advance what is possible in the application of AI to meteorology.</p> <p>Don't miss the chance to connect with AI developers and innovators at GTC, September 19-22, 2022. <a href="https://www.nvidia.com/gtc/?ncid=ref-spo-810249">Register for NVIDIA GTC</a> today! And join the <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.nvidia.com%2Fdeveloper-program&#x26;data=05%7C01%7Cgpiana%40nvidia.com%7Cd3a1422c83b745facca208da84896a6d%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637968022969870756%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&#x26;sdata=VDyF2%2Fs42oE4sDRdu61f6HZpgiJK%2FNFoMYjioPRvGjs%3D&#x26;reserved=0">NVIDIA Developer Program</a> for free and exclusive access to SDKs, technical documentation, peer, and domain expert help. NVIDIA offers tools and training to accelerate AI, HPC, and graphics applications.</p>https://developer.hpe.com/deploy-stateful-mongodb-application-on-kubernetes-clusters-in-hpe-greenlake-for-containers-1/https://developer.hpe.com/deploy-stateful-mongodb-application-on-kubernetes-clusters-in-hpe-greenlake-for-containers-1/Tue, 16 Aug 2022 14:02:17 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Containers is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Containers</a>, one of the HPE GreenLake cloud services available on the HPE GreenLake Central platform, allows customers to open the Clusters screen to create a cluster, view details about existing clusters, and launch the HPE GreenLake for Container service console. It provides an enterprise-grade container management service using open source Kubernetes.</p> <p>This blog post describes the process of deploying a stateful MongoDB application on the created Kubernetes clusters in HPE GreenLake for Containers. Using the Kubernetes StatefulSet and Headless Service, together with the pre-provisioned persistent volumes, the MongoDB application can be deployed as a Replica Set that provides redundancy, fault tolerance and high availability. This MongoDB application deployment can be used for working with a large amount of data and a high number of workloads in customer production environments.</p> <h2>MongoDB Application</h2> <p><a href="https://www.mongodb.com/">MongoDB</a> is a general-purpose, document-based NoSQL database program that is a popular alternative to traditional databases. The MongoDB data model can represent any kind of object structures that can have properties or even be nested for multiple levels. Unlike the relational databases, MongoDB can store large amounts of data without requiring a logical category or schema. Therefore, it takes up the challenge of focusing on scalability and the speed of queries. MongoDB is free, open-source software. You can download the MongoDB packages, set them up and configure them at no expense.</p> <p>MongoDB can be deployed as a standalone instance that is running as a single <em>mongod</em> server. It can also be deployed as a <a href="https://www.mongodb.com/docs/manual/core/replica-set-architectures">Replica Set</a> with multiple running <em>mongod</em> instances that maintain the same data set. The Replica Set contains several data-bearing nodes. Of those data-bearing nodes, one and only one member is deemed the <em>Primary</em> node, while the other nodes are deemed <em>Secondary</em> nodes.</p> <p><img src="/img/mongodb-replica-set.png" alt=""></p> <p>The standalone MongoDB instance deployment is suitable for testing and some aspects of development. However, it’s not adequate for production use. The MongoDB Replica Set deployment ensures multiple replicas running at any given time, which provides redundancy, fault tolerance and high availability. It is recommended for production environments, such as HPE GreenLake for Containers. This blog post will focus on deploying the MongoDB application as a Replica Set.</p> <h2>Deploy MongoDB Application</h2> <h3>Requirements</h3> <p>A Kubernetes cluster can be created using either HPE GreenLake for Containers GUI, or through Infrastructure as Code (IaC) with <a href="https://registry.terraform.io/providers/HPE/hpegl/0.2.2">HPE GreenLake Terraform Provider</a> as explained in the blog post <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code</a>. By launching to the HPE GreenLake for Containers service console, you can download the <strong>kubectl</strong> binary, together with the <em>kubeconfig</em> file, and set it up to access the Kubernetes cluster using kubectl CLI. The administrative access will be configured from the created Kubernetes cluster in HPE GreenLake for containers, which allows you to set up the Kubernetes RBAC for MongoDB application deployment.</p> <p>Sample view of HPE GreenLake for Containers Clusters screen:</p> <p><img src="/img/containers-clusters.png" alt=""></p> <p>Sample view of HPE GreenLake for Containers service console:</p> <p><img src="/img/containers-service-console.png" alt=""></p> <h3>Set up Role-Based Access Control (RBAC)</h3> <p>Kubernetes RBAC is a key security control to ensure that cluster users and workloads have access only to resources required to execute their roles. It is important to ensure that, when designing permissions for cluster users, the cluster administrator understands the areas where privilege escalation could occur, to reduce the risk of excessive access leading to security incidents.</p> <p>To set up RBAC, you create a Service Account, a ClusterRole, and connect the two with a Cluster RoleBinding.</p> <h4>1. Create a YAML file <em>sa.yaml</em> for the service account:</h4> <pre><code class="language-markdown">apiVersion: v1 kind: ServiceAccount metadata: name: mongo-account </code></pre> <h4>2. Create a YAML file <em>clusterrole.yaml</em> for the cluster roles:</h4> <pre><code class="language-markdown">apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mongo-role rules: - apiGroups: [""] resources: ["deployments"] verbs: ["list", "watch"] - apiGroups: [""] resources: ["services"] verbs: ["*"] - apiGroups: [""] resources: ["pods"] verbs: ["get","list", "watch"] </code></pre> <h4>3. Create a YAML file <em>clusterrolebinding.yaml</em> to bind the service account with the cluster access roles:</h4> <pre><code class="language-markdown">apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mongo_role_binding subjects: - kind: ServiceAccount name: mongo-account roleRef: kind: ClusterRole name: mongo-role apiGroup: rbac.authorization.k8s.io </code></pre> <h3>Create the StatefuleSet Deployment</h3> <p>To deploy MongoDB as a Replica Set with multiple pods, a Kubernetes StatefulSet deployment will be required. The data persistence setup can be done with a <em>VolumeClaimTemplate</em> in the StatefulSet deployment.</p> <p>It should be noted that the Kubernetes Deployment works fine if you have only one single MongoDB replica being deployed. In case multiple replicas are running, which is required in the production environment, there will be some issues using Kubernetes Deployment. Developers have to handle the concurrent read-write of the same data in the deployment.</p> <h4>1. Create a YAML file <em>namespace.yaml</em> for the namespace:</h4> <pre><code class="language-markdown">apiVersion: v1 kind: Namespace metadata: name: mongodb </code></pre> <h4>2. Create a YAML file <em>statefulset.yaml</em> for MongoDB StatefulSet deployment:</h4> <pre><code class="language-markdown">apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb-replica spec: serviceName: pce-mongodb replicas: 1 selector: matchLabels: app: mongo template: metadata: labels: app: mongo selector: mongo spec: terminationGracePeriodSeconds: 30 serviceAccount: pce-mongo-account containers: - name: mongodb image: docker.io/mongo:4.2 command: ["/bin/sh"] args: ["-c", "mongod --replSet=rs0 --bind_ip_all"] resources: limits: cpu: 100m memory: 200Mi requests: cpu: 100m memory: 100Mi ports: - name: mongo-port containerPort: 27017 volumeMounts: - name: mongo-data mountPath: /data/db volumeClaimTemplates: - metadata: name: mongo-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi </code></pre> <h3>Set Up the Headless Service</h3> <p>A Kubernetes Headless Service does not allocate an IP address or forward traffic. It’s used for creating a service grouping. Clients can connect to the pods of a Headless Service by connecting to the service’s DNS name. The DNS returns the pods’ IPs and the client can connect directly to the pods instead via the service proxy. The Headless Service can be used for deploying the stateful applications such as MongoDB, while still providing the benefits of a service definition for taking care of the pod restart.</p> <h4>Create a YAML file <em>service.yaml</em> for the headless service:</h4> <pre><code class="language-markdown">apiVersion: v1 kind: Service metadata: name: mongodb labels: app: mongo spec: ports: - port: 27017 protocol: TCP targetPort: 27017 selector: app: mongo clusterIP: None </code></pre> <h3>Deploy MongoDB Application with Kustomize</h3> <p>With all created YAML files, MongoDB application will be deployed using <a href="https://kustomize.io/">Kustomize</a>.</p> <h4>1. Move all YAML files to the folder '<em>base</em>':</h4> <pre><code class="language-markdown">├── base │ ├── clusterrolebinding.yaml │ ├── clusterrole.yaml │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── pvc.yaml │ ├── sa.yaml │ ├── service.yaml │ └── statefulset.yaml </code></pre> <p>The file <em>kustomization.yaml</em> lists all YAML files in its <em>resources</em> section:</p> <pre><code class="language-markdown">apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - namespace.yaml - sa.yaml - clusterrole.yaml - clusterrolebinding.yaml - statefulset.yaml - service.yaml </code></pre> <h4>2. Create another folder '<em>overlays</em>' and a sub-folder '<em>production</em>':</h4> <pre><code class="language-markdown">└── overlays └── production ├── kustomization.yaml └── patch-statefulset.yaml </code></pre> <p>The contents of the file <em>kustomization.yaml</em>:</p> <pre><code class="language-markdown">apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base commonLabels: env: production namePrefix: pce- namespace: pce-mongodb patches: - ./patch-statefulset.yaml </code></pre> <p>The contents of the file <em>patch-statefulset.yaml</em>:</p> <pre><code class="language-markdown">apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb-replica spec: replicas: 3 </code></pre> <p>This YAML file will patch the original StatefulSet and deploy the MongoDB application as a three-member replica set.</p> <h4>3. Deploy MongoDB with the following command:</h4> <pre><code class="language-markdown">$ kubectl apply -k overlays/production namespace/pce-mongodb created serviceaccount/pce-mongo-account created clusterrole.rbac.authorization.k8s.io/pce-mongo-role created clusterrolebinding.rbac.authorization.k8s.io/pce-mongo_role_binding created service/pce-mongodb created statefulset.apps/pce-mongodb-replica created </code></pre> <p>MongoDB application is deployed as Replica Set to the namespace '<em>pce-mongodb</em>' using the service account '<em>pce-mongo-account</em>'. Running the following command, you should see three specified replicas pods are all in <strong>running</strong> status:</p> <pre><code class="language-markdown">$ kubectl get all -n pce-mongodb NAME READY STATUS RESTARTS AGE pod/pce-mongodb-replica-0 1/1 Running 0 30s pod/pce-mongodb-replica-1 1/1 Running 0 19s pod/pce-mongodb-replica-2 1/1 Running 0 11s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/pce-mongodb ClusterIP None &#x3C;none> 27017/TCP 30s NAME READY AGE statefulset.apps/pce-mongodb-replica 3/3 30s </code></pre> <h2>Set up MongoDB Replication Host</h2> <p>After the MongoDB application is deployed with all the replica pods in running states, you need to set up MongoDB replication.</p> <h4>1. Connect to the MongoDB pod <em>pce-mongodb-replica-0</em>:</h4> <pre><code class="language-markdown">$ kubectl exec -it pce-mongodb-replica-0 -n pce-mongodb -- mongo MongoDB shell version v4.2.21 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&#x26;gssapiServiceName=mongodb Implicit session: session { "id" : UUID("5d7b5a72-ff37-4ead-b864-965a551dc966") } MongoDB server version: 4.2.21 Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see https://docs.mongodb.com/ Questions? Try the MongoDB Developer Community Forums https://community.mongodb.com --- Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc). The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product improvements and to suggest MongoDB products and deployment options to you. To enable free monitoring, run the following command: db.enableFreeMonitoring() To permanently disable this reminder, run the following command: db.disableFreeMonitoring() --- > </code></pre> <h4>2. Initiate the replication:</h4> <pre><code class="language-markdown">> rs.initiate() { "info2" : "no configuration specified. Using a default configuration for the set", "me" : "pce-mongodb-replica-0:27017", "ok" : 1 } rs0:SECONDARY> </code></pre> <h4>3. Add the pod as the <em>Primary</em> server to the replication configuration:</h4> <pre><code class="language-markdown">rs0:SECONDARY> var cfg = rs.conf() rs0:PRIMARY> cfg.members[0].host="pce-mongodb-replica-0.pce-mongodb:27017" pce-mongodb-replica-0.pce-mongodb:27017 rs0:PRIMARY> rs0:PRIMARY> rs.reconfig(cfg) { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1658309061, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1658309061, 1) } rs0:PRIMARY> </code></pre> <h4>4. Add the second MongoDB pod <em>pce-mongodb-replica-1</em> to the replication configuration:</h4> <pre><code class="language-markdown">rs0:PRIMARY> rs.add("pce-mongodb-replica-1.pce-mongodb:27017") { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1658309117, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1658309117, 1) } rs0:PRIMARY> </code></pre> <h4>5. Continue to add the third MongoDB pod <em>pce-mongodb-replica-2</em> to the replication configuration:</h4> <pre><code class="language-markdown">rs0:PRIMARY> rs.add("pce-mongodb-replica-2.pce-mongodb:27017") { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1658309148, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1658309148, 1) } rs0:PRIMARY> </code></pre> <h4>6. Verify MongoDB replication status:</h4> <pre><code class="language-markdown">rs0:PRIMARY> rs.status() { "set" : "rs0", "date" : ISODate("2022-07-20T09:26:19.014Z"), "myState" : 1, "term" : NumberLong(1), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "majorityVoteCount" : 2, "writeMajorityCount" : 2, "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "lastCommittedWallTime" : ISODate("2022-07-20T09:26:13.146Z"), "readConcernMajorityOpTime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "readConcernMajorityWallTime" : ISODate("2022-07-20T09:26:13.146Z"), "appliedOpTime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "durableOpTime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "lastAppliedWallTime" : ISODate("2022-07-20T09:26:13.146Z"), "lastDurableWallTime" : ISODate("2022-07-20T09:26:13.146Z") }, "lastStableRecoveryTimestamp" : Timestamp(1658309133, 1), "lastStableCheckpointTimestamp" : Timestamp(1658309133, 1), "electionCandidateMetrics" : { "lastElectionReason" : "electionTimeout", "lastElectionDate" : ISODate("2022-07-20T09:22:33.048Z"), "electionTerm" : NumberLong(1), "lastCommittedOpTimeAtElection" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "lastSeenOpTimeAtElection" : { "ts" : Timestamp(1658308953, 1), "t" : NumberLong(-1) }, "numVotesNeeded" : 1, "priorityAtElection" : 1, "electionTimeoutMillis" : NumberLong(10000), "newTermStartDate" : ISODate("2022-07-20T09:22:33.139Z"), "wMajorityWriteAvailabilityDate" : ISODate("2022-07-20T09:22:33.224Z") }, "members" : [ { "_id" : 0, "name" : " pce-mongodb-replica-0.pce-mongodb:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 459, "optime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2022-07-20T09:26:13Z"), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1658308953, 2), "electionDate" : ISODate("2022-07-20T09:22:33Z"), "configVersion" : 4, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "pce-mongodb-replica-1.pce-mongodb:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2022-07-20T09:26:13Z"), "optimeDurableDate" : ISODate("2022-07-20T09:26:13Z"), "lastHeartbeat" : ISODate("2022-07-20T09:26:18.007Z"), "lastHeartbeatRecv" : ISODate("2022-07-20T09:26:18.010Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "pce-mongodb-replica-0.pce-mongodb:27017", "syncSourceHost" : "pce-mongodb-replica-0.pce-mongodb:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 4 }, { "_id" : 2, "name" : "pce-mongodb-replica-2.pce-mongodb:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 31, "optime" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1658309173, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2022-07-20T09:26:13Z"), "optimeDurableDate" : ISODate("2022-07-20T09:26:13Z"), "lastHeartbeat" : ISODate("2022-07-20T09:26:18.007Z"), "lastHeartbeatRecv" : ISODate("2022-07-20T09:26:18.536Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "pce-mongodb-replica-1.pce-mongodb:27017", "syncSourceHost" : "pce-mongodb-replica-1.pce-mongodb:27017", "syncSourceId" : 1, "infoMessage" : "", "configVersion" : 4 } ], "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1658309173, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1658309173, 1) } rs0:PRIMARY> </code></pre> <p>The <strong>members</strong> section of the status output shows three replicas. The pod <em>pce-mongodb-replica-0</em> is listed as the <strong>Primary</strong> replica, while the other two pods, <em>pce-mongodb-replica-1</em> &#x26; <em>pce-mongodb-replica-2</em>, are listed as the <strong>Secondary</strong> replicas.</p> <p>The MongoDB replica set deployment is set up and ready to operate now. You can download and install the <a href="https://www.mongodb.com/docs/compass">MongoDB Compass</a> and set up an external connection to connect to the MongoDB application deployed in the Kubernetes cluster. Using the powerful MongoDB Compass GUI, you can query, aggregate, and analyze the MongoDB data from the MongoDB deployment.</p> <h2>Scale MongoDB Application</h2> <p>If you want to add another replica set to the MongoDB deployment to scale up the MongoDB application, you can run the following kubectl command:</p> <pre><code class="language-markdown">$ kubectl scale statefulset &#x3C;name> -n &#x3C;namespace> --replicas=&#x3C;number of replicas> </code></pre> <p>Then, follow up with <em>Step 1</em> from the previous section to connect to the first MongoDB pod and repeat <em>Step 5</em> to add the new replica pod.</p> <p>To scale down the MongoDB application, you can simply run the command <strong>rs.remove()</strong> to remove the replica pod from the replication configuration.</p> <h2>Summary</h2> <p>This blog post describes the detailed process used to deploy and set up a stateful MongoDB application as a Replica Set deployment on Kubernetes clusters in HPE GreenLake for Containers. Kubernetes provides the ability to persist the state, deploy the stateful application, and manage and scale those applications with state. The production deployment of the MongoDB application provides redundancy, fault tolerance, high availability, and scalability. It can be used for working with a large amount of data and a high number of workloads in customer production environments.</p>https://developer.hpe.com/mlops-–-deploying-an-ml-model-in-greenlake-platform-mlops-service/https://developer.hpe.com/mlops-–-deploying-an-ml-model-in-greenlake-platform-mlops-service/Mon, 08 Aug 2022 07:48:49 GMT<h2>Overview</h2> <p>HPE GreenLake Central is an advanced software-as-a-service platform that provides you with a consistent cloud experience for all your applications and data on-premises or off-premises. It provides you with insights and controls to manage your hybrid IT estate, complementing your use of public clouds and data centers. HPE GreenLake Central gives you the ability to choose where and how to place your workloads and data, and through the services you purchase enables you to monitor security, capacity, resource utilization, and costs.</p> <p>HPE GreenLake for ML Ops is an on-premises, enterprise-grade ML service, enabling developers and data scientists to rapidly build, train, and deploy ML models from pilot to production, at any scale.</p> <p>This preconfigured solution comprises an optimized hardware stack and is powered by HPE Ezmeral Runtime Enterprise. It provides data scientists with self-service access to a sandbox environment for prototyping and testing, to eliminate IT provisioning delays, ensure repeatability, and accelerate time-to-value. As a fully managed solution, the HPE GreenLake for ML Ops offering frees IT from routine infrastructure management tasks.</p> <h2>Machine learning Lifecycle</h2> <p><img src="/img/blog_3.png" alt="ML Life cycle" title="ML Life cycle"></p> <p>The ML project life cycle can generally be divided into three main stages: data preparation, model creation, and deployment. All three of these components are essential for creating quality models that will bring added value to your business. This process is <em>cyclical</em> because the insights gained from the existing model will help define the next model to be deployed.</p> <p>In this article, I will focus on optimal model identified after data preparation and building model. More specifically, as an example, I will show you how to deploy a sample ONNX model available in Kubernetes native MinIO Object storage, using Triton Inference Server.</p> <p>Triton Inference server supports deployment of any AI model from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton delivers optimized performance for many query types, including real time, batched, ensembles and audio/video streaming. To learn more about Triton Inference server, refer to the References section at the end of the post.</p> <p>HPE GreenLake for ML Ops platform allows customers to host their favorite cloud native applications, like MLflow, Minio, etc.</p> <h2>Pre-requisites</h2> <ul> <li> <p>An active service subscription to HPE GreenLake for ML Ops</p> </li> <li> <p>An ML Ops project that has been created by an ML Ops admin for which the user is able to launch to HPE Ezmeral Runtime Enterprise through an ML Ops Project Member role</p> </li> <li> <p>Available access credentials to any S3 based object storage</p> </li> <li> <p>Triton Inference Server container image accessible either through on-prem registry (eg Harbor) or public registry accessible.</p> </li> </ul> <h3>Steps to deploy a model</h3> <p><strong>Step 1:</strong> Validation connection to Kubernetes cluster</p> <ol> <li> <p>Click HPE GreenLake for ML Ops card in the HPE GreenLake Central <strong>Dashboard,</strong> which shows the number of previously created projects.</p> <p><img src="/img/blog_2.png" alt="Select ML Ops Project" title="Select ML Ops Project"></p> </li> <li> <p>Click link “Launch ML Operations Console”</p> </li> <li> <p>Click “Dashboard” on left navigation of “Ezmeral Container Platform”</p> </li> <li> <p>Download “kubectl”, “HPE kubectl plugin”, and “kubeconfig”</p> </li> <li> <p>Set environment variable KUBECONFIG to point to the kubeconfig file</p> </li> <li> <p>Validate connectivity to the cluster using command “kubectl get no”</p> </li> </ol> <p><strong>Step 2:</strong> Place the model in S3 object storage</p> <ol> <li> <p>Place the model and configuration file for Triton Inference server in object storage</p> <p><img src="/img/blog_1.png" alt="object storage for model" title="object storage for model"></p> </li> <li> <p>Sample configuration file for the model as shown below:</p> </li> </ol> <pre><code class="language-markdown">name: "braintumor_onnx" platform: "onnxruntime_onnx" max_batch_size : 0 input [   {     name: "input_1"     data_type: TYPE_FP32     dims: \[ 100, 128, 128, 2 ]   } ] output [   {     name: "conv2d_22"     data_type: TYPE_FP32     dims: \[ -1, 128, 128, 4 ]   } ] </code></pre> <p><strong>Step 3:</strong> Create a namespace and secret for object storage credentials using the commands shown below:</p> <pre><code class="language-shell">kubectl create namespace triton kubectl create secret generic minio_cred –from-literal=AWS_ACCESS_KEY_ID=&#x3C;specify_access_key> --from-literal=AWS_SECRET_ACCESS_KEY=&#x3C;specify_secret_access_key> -n triton </code></pre> <p><strong>Step 4:</strong> Create a deployment to host the model and check the pods and services are running</p> <pre><code class="language-yaml">--- apiVersion: apps/v1 kind: Deployment metadata:   labels:     app: triton   name: triton   namespace: triton spec:   replicas: 1   selector:     matchLabels:       app: triton   template:     metadata:       labels:         app: triton     spec:       containers: - image: nvcr.io/nvidia/tritonserver:21.10-py3         name: tritonservercont         command: ["/bin/bash"]         args: ["-c", "/opt/tritonserver/bin/tritonserver --model-repository=s3://https://&#x3C;objectstoreurl.com:port>/sample/models --strict-model-config=false"]         env: - name: AWS_ACCESS_KEY_ID             valueFrom:               secretKeyRef:                 name: minio-cred                 key: AWS_ACCESS_KEY_ID - name : AWS_SECRET_ACCESS_KEY             valueFrom:               secretKeyRef:                 name: minio-cred                 key: AWS_SECRET_ACCESS_KEY         ports: - containerPort: 8000             name: http - containerPort: 8001             name: grpc - containerPort: 8002             name: metrics         volumeMounts: - mountPath: /dev/shm           name: dshm         resources:           requests:             cpu: 2             memory: 8Gi             nvidia.com/gpu: 1           limits:             cpu: 2             memory: 16Gi             nvidia.com/gpu: 1       volumes: - name: dshm         emptyDir:           medium: Memory       securityContext:         runAsUser: 1000         fsGroup: 1000       nodeSelector:         gl.hpe.com/instance-type: GL-GP-MLi-Metal --- apiVersion: v1 kind: Service metadata:   name: triton   namespace: triton   labels:     hpecp.hpe.com/hpecp-internal-gateway: "true" spec:   type: NodePort   selector:     app: triton   ports: - protocol: TCP       name: http       port: 8000       nodePort: 30850       targetPort: 8000 - protocol: TCP       name: grpc       port: 8001       nodePort: 30851       targetPort: 8001 - protocol: TCP       name: metrics       nodePort: 30852       port: 8002       targetPort: 8002 </code></pre> <p>          <strong>Notes:</strong></p> <ul> <li>In the above YAML, replace <a href="objectstoreurl.com:port">objectstoreurl.com:port</a> and model path with actual object store URL, model sub path. Model refers to location where model and associated configuration is placed.</li> <li>Node selector “gl.hpe.com/instance-type: GL-GP-MLi-Metal” is used to place the workload in inference cluster</li> <li>Check that the pods and service are running using the commands shown below:</li> </ul> <pre><code class="language-shell">kubectl get po –n triton kubectl get svc –n triton </code></pre> <ul> <li>Service labels <strong>hpecp.hpe.com/hpecp-internal-gateway: "true"</strong> is placed to get a gateway endpoint to access the Triton Inference server outside the cluster. To find the endpoint, run this command:</li> </ul> <pre><code class="language-shell">kubectl describe svc &#x3C;service_name> -n triton </code></pre> <ul> <li>Check the metrics endpoints for models hosted in Triton Inference server are accessible:</li> </ul> <pre><code class="language-shell">kubectl describe svc &#x3C;service_name> -n triton </code></pre> <h2>Next steps</h2> <p>Once the production model is hosted, then any application can perform inference on the model hosted by Triton inference server. For more on Triton client libraries, refer to the section below.</p> <h2>References</h2> <ul> <li> <p><a href="https://github.com/triton-inference-server/server">Triton Inference server</a></p> </li> <li> <p><a href="https://github.com/triton-inference-server/client">Triton Client</a></p> </li> <li> <p><a href="https://min.io/">MinIO</a></p> </li> <li> <p><a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=HPE-GreenLake-for-ML-Ops.html">HPE GreenLake for ML Ops documentation</a></p> </li> <li> <p><a href="https://docs.containerplatform.hpe.com/54/reference/HPE_Ezmeral_Container_Platform.html">HPE Ezmeral Runtime Enterprise Documentation</a></p> </li> </ul>https://developer.hpe.com/2022-August-04/https://developer.hpe.com/2022-August-04/Fri, 05 Aug 2022 13:00:00 GMThttps://developer.hpe.com/insights-from-a-hack-shack-staffers-diary/https://developer.hpe.com/insights-from-a-hack-shack-staffers-diary/Thu, 21 Jul 2022 16:48:50 GMT<p>After two years of only being able to meet virtually, HPE Developer Community members were excited to be able to connect with one another in-person at HPE Discover 2022 in the Hack Shack. It was an amazing event with more than 8,000 visitors. The event focused mostly on HPE GreenLake and all the 70+ new services that have been made available, from private cloud for the enterprise to data analytics (try some of them here in the<a href="https://testdrive.greenlake.hpe.com/"> HPE GreenLake Test Drive</a>).</p> <p>It was great to once again meet up with colleagues and customers face-to-face. I arrived early this year in order to help with the Hack Shack setup over the weekend. Our CEO, Antonio Neri, stopped by the booth while we were doing setup. He gladly agreed to take a picture with some of us members of the HPE Developer Community team. (He’s the one in blue and I’m to the right.)</p> <center><img src="/img/fred-1b-3-501-pix-.jpg" alt="HPE Developer Community staff with HPE CEO, Antonio Neri"></center> <h3>Highlights of the week</h3> <p>The Hack Shack schedule was pretty intense, as usual. For those of you who may not have had the opportunity to stop by, you did miss out on quite a few cool activities. We had 13 interactive meetups focused on Open Source, how to take advantage of data using a data map, security, data protection, API integration, and infrastructure-as-code. We also hosted 5 software challenges based on our Workshop-on-Demand structure. The most popular software challenge this year was the one entitled <em><strong>VM Desired State Management in HPE GreenLake</strong></em> where one learned to manage an HPE GreenLake Private Cloud for Enterprise workload using Terraform.</p> <p>In the front yard, I met soon-to-become new members of the HPE Developer Community after sharing with them a few simple, yet efficient, words and a quick demo of our developer portal. I also helped attendees participate in our software challenges. These software challenges were enjoyed by many and will be transformed over the summer into workshops that people can take through our <a href="https://developer.hpe.com/hackshack/workshops">Workshop-on-Demand catalog</a>.</p> <center><img src="/img/fred-2-b-3-475-pix.jpg" alt="HPE Developer Community portal prominently displayed"></center> <p>Another highlight of the week were the foosball challenges we engaged in with Hack Shack attendees. I remained unbeaten at the foosball table the entire week. And I even managed to deliver our HPE Developer Community message while playing. A peculiar, but effective, tactic I used to win, I must admit.  </p> <center><img src="/img/fred-3-b-3-405-pix-.jpg" alt="Frederic remains undefeated in foosball"></center> <p>Many attendees decided to try our virtual treasure hunt. This simple scavenger-hunt style game allowed them to learn more about both the team and the different assets available in the HPE Developer Community portal. The winners were awarded Canakits and Raspberry Pi 4 computers during our Wednesday night party.</p> <center><img src="/img/fred-4-b-3-450-pix-.jpg" alt="Listening to Dr. Goh at the Hack Shack Celebration party"></center> <p>Next to the Hack Shack was a space called the Living Labs where HPE introduced the new trials experience for HPE GreenLake. The HPE Developer Community team expects to work closely in the near future with HPE GreenLake teams to provide more developer-related content for their catalog. A software challenge we developed for this Discover event – the Terraform provider for HPE GreenLake – will be one of the very first workshops on that list.</p> <h3>The introduction of the HPE GreenLake Developer portal</h3> <p>One of the most interesting announcements of the week was the introduction of the <a href="https://developer.greenlake.hpe.com/">HPE GreenLake Developer portal</a>. As HPE transitions to an as-a-service strategy, APIs and developers become more and more important. It would be illogical for HPE to offer a platform where customers would consume resources by clicking on well-designed User Interfaces. APIs must be made available and described properly so they can be leveraged to allow complete automation. Equally, providers should be developed to allow better interaction with industry-standard automation frameworks like Red Hat’s Ansible, Hashicorp Terraform, Chef, or Puppet. The HPE GreenLake developer portal aims at offering API access to customers, allowing them to integrate seamlessly with HPE GreenLake programmatically.</p> <center><img src="/img/fred-5-b-3-512-pix-.jpg" alt="One of the technical meetup sessions"> </center> <p>The evidence that Discover 2022 was a HPE GreenLake-centric event abounded. The design of the showcase floor reflected that, with beams of data feeding a central control center. As HPE shifts to this model, the personas the company addresses are also evolving. Not only do our solutions address the IT Ops administrators moving to a DevOps culture, but also the new data-driven engineers working on Data and ML Ops. All these personas develop code and need to interact seamlessly with HPE GreenLake offerings. The HPE Developer Community team is doing its best to facilitate this interaction. That’s why we will be here again next year with new opportunities to learn, content and fun activities!</p> <center><img src="/img/fred-6-b-3-512-.jpg" alt="The onsite HPE Developer Community team"> </center>https://developer.hpe.com/introducing-the-hpe-greenlake-developer-portal/https://developer.hpe.com/introducing-the-hpe-greenlake-developer-portal/Wed, 20 Jul 2022 15:42:12 GMT<p>During the executive keynote session at the HPE Discover 2022 Edge-to-Cloud Conference, Antonio Neri, President and CEO of Hewlett Packard Enterprise, announced the availability of the HPE GreenLake Developer Portal. He was joined by Fidelma Russo, HPE CTO, and Bryan Thompson, HPE VP of GreenLake Product Management, who gave a quick look at how it provides APIs and documentation to assist developers of all types integrate their apps and services with HPE GreenLake. A <a href="https://developer.greenlake.hpe.com/docs/greenlake">beta version of the portal</a> is available today, easily accessed through the <a href="https://developer.hpe.com/">HPE Developer Community portal</a>.</p> <p>Recently, I had the opportunity to talk with HPE GreenLake Platform Product Manager Navaneethan Venugopalan and HPE Distinguished Technologist Travis Tripp about this eagerly anticipated resource and review it myself. Through the portal, HPE GreenLake customers and partners can take advantage of a well-documented, secure, and scalable framework of APIs for the HPE GreenLake edge-to-cloud platform.  Starting off by providing material for the HPE GreenLake for Compute Ops Management service, contents in this resource portal will grow rapidly, so make sure you check it often.</p> <p><strong>First off, congratulations on the launch of your new portal. I'm sure our readers are very excited to hear about it. Could you give them a little more detail about the portal?</strong></p> <p>Thank you! We’re excited as well. The HPE GreenLake Developer Portal is 100% dedicated to automating integrations with HPE GreenLake. Developers can find all of the official HPE GreenLake API and integration guides there, delivered directly from the HPE GreenLake engineering teams.  It will always contain the most up-to-date and accurate guidance, because it is delivered as part of our end-to-delivery pipeline.  In fact, you will find the very same API and integrations guides there that all of the HPE GreenLake teams use themselves!</p> <p><strong>Why did you develop the portal?</strong></p> <p>We received numerous requests from our customers and partners to make this information easily available so that developers, data scientists, IT operators, MSP partners, resellers, and 3rd-party systems integrators could automate their interactions with HPE GreenLake.  The announcement of its availability at Discover 2022 was eagerly received. Our customers are excited to be able to automate common and repetitive tasks using these APIs.</p> <p><strong>Can anyone, then, access the portal?</strong></p> <p>The portal, itself, is easily accessed by anyone – no login is required. Simply visit <a href="https://developer.greenlake.hpe.com/">developer.greenlake.hpe.com</a> or go through the <a href="https://developer.hpe.com/">HPE Developer Community portal</a> to access the site. We do offer varying levels of access, though. All the HPE GreenLake APIs support role-based access and you must be authenticated and authorized to use them. Every API in HPE GreenLake is designed with zero trust in mind, meaning that in order to access the API, you must have the correct authorization.  This means that based on the role you’ve been granted in your HPE GreenLake environment will allow you to successfully invoke the relevant APIs. You can always browse the APIs to learn more, but if you happen to be an authorized partner, you’ll get access to even more partner API docs, as we publish API documentation in two categories – public and partner.</p> <p><strong>I see that the HPE GreenLake for Compute Ops Management service is available today. What operational tasks can a developer automate using the HPE GreenLake management console APIs?</strong></p> <p>Developers using HPE GreenLake for Compute Ops Management (COM) can automate bulk server lifecycle management tasks (such as updating the firmware of servers), perform power on/off operations, set up policies for establishing firmware baselines, and more. They can also perform monitoring operations to get inventory and health status, to name a few. COM has been purpose-built to be API first; pretty much any operation that can be performed via the UI can be performed via the API.</p> <p><strong>What additional APIs will you be adding in the not-too-distant future and when might we expect them?</strong></p> <p>HPE GreenLake for Containers, Storage, and Private Cloud Enterprise are some of the services that will become available in the next 4-6 months.</p> <p><strong>At the Discover event, were there any specific things customers requested?</strong></p> <p>Customers at the event asked that we ensure it is kept up-to-date with the most current and accurate information, which we are able to do considering how it is part of our end-to-delivery pipeline.  They also requested support for tools like Terraform and Ansible. They were even interested in contributing to it with open source tools and libraries. Finally, they asked for a way to provide feedback and suggestions, which we’re currently looking into.</p> <p><strong>I wish you much success with the portal and look forward to working with you and your team more closely to compliment your site with HPE GreenLake-related blog posts, workshops, and technical talks on the HPE Developer Community portal.</strong></p> <p>We do as well. As you know, the <a href="https://developer.hpe.com/platform/hpe-greenlake/home/">HPE GreenLake edge-to-cloud platform page</a> on the HPE Developer Community site already links to over a dozen blog posts and tutorials. There’s already one HPE GreenLake related <a href="https://developer.hpe.com/hackshack/workshops">Workshop-on-Demand</a>, and we expect that more will be added soon. We’ve been invited to participate in the Munch &#x26; Learn talks and Meetup sessions, and we look forward to doing that as well.</p>https://developer.hpe.com/discover-the-importance-of-hpe-greenlake-to-our-community/https://developer.hpe.com/discover-the-importance-of-hpe-greenlake-to-our-community/Mon, 18 Jul 2022 18:29:05 GMT<!--StartFragment--> <p>It was standing-room only when HPE’s CEO, Antonio Neri, gave his <a href="https://www.youtube.com/watch?v=5YocxnhKAnM">keynote</a> to kick off the HPE Discover 2022 Edge-to-Cloud Conference. The importance of the HPE GreenLake edge-to-cloud platform resounded everywhere throughout the event. But most exciting for us was when the newly refreshed HPE Developer Community portal was shown on the onstage screen. You could hear a large cheer as Antonio noted the importance of developers to HPE, specifically in relation to the HPE GreenLake platform, and mentioned the HPE Developer Community by name. </p> <p><img src="/img/hpe-dev-on-screen.jpg" alt=""></p> <p>In his <a href="https://www.youtube.com/watch?v=5YocxnhKAnM">keynote</a>, Antonio Neri announced the availability of the new HPE GreenLake Developer portal – along with new tools – created with the aim of making it easier for developers to access Application Programming Interfaces (APIs) for HPE GreenLake. By previewing this portal, Antonio showed how HPE is working to engage with developers, IT DevOps teams, and data engineers to help them harness data and facilitate its use enterprise-wide to inform business decision making.</p> <p>It was a proud moment for the team – and an important one as we expand the HPE Developer Community’s ability to help developers of all types engage on the HPE GreenLake platform.</p> <h2>Develop with the HPE GreenLake edge-to-cloud platform</h2> <p>Antonio emphasized that bringing a consistent cloud operating model across all workloads and data was the real answer to the conundrum businesses face when they ask which workloads should be put in a public cloud and which should stay on-premises. And this is where HPE GreenLake provides the solution. It’s a platform built specifically for a highly distributed enterprise designed for application modernization.</p> <p>But it’s also important to remember that HPE GreenLake is a platform for application development. He pointed out that the company is continuing to build out its growing HPE GreenLake partner ecosystem with distributors, value-added resellers, systems integrators, ISVs, and service providers. “We are placing partner capabilities at the center of our platform through our strategy to open our unique APIs to partners,” Neri told the audience. “They can choose to build, resell or manage on behalf of the customers.”</p> <p>Antonio highlighted that HPE has already shown its HPE GreenLake API partner commitment by announcing tight API integration with the cloud marketplaces of the top four global distributors – TD Synnex, Ingram Micro, Arrow Electronics and ALSO Group – making it easier for them to sell cloud services through and with the channel. Ultimately, the aim is to open the full power of the platform to partners.</p> <h2>Looking forward to next Discover</h2> <p>Traditionally, the HPE Discover event has been an event where customers and partners get key insights into HPE’s strategies and products. In recent years, it has attracted more business line managers than techies. As our strategy continues to evolve to a cloud software and services model, the important issues that will be discussed will be how to integrate with the HPE GreenLake edge-to-cloud platform. Developers take heed! This is your moment. Start talking with your managers now to make sure they budget for your ability to travel and hear what’s new at the next HPE Discover event!</p> <!--EndFragment-->https://developer.hpe.com/summary-from-the-9th-annual-chapel-implementers-and-users-workshop-chiuw-2022/https://developer.hpe.com/summary-from-the-9th-annual-chapel-implementers-and-users-workshop-chiuw-2022/Fri, 15 Jul 2022 19:31:06 GMT<p><strong>Introduction</strong></p> <p>Programming today becomes complicated by the many kinds of parallelism that exist in everything from phones to laptops to supercomputers.  The open-source Chapel parallel programming language makes parallel programming easier and more productive, while still enabling high performance that takes advantage of the wide variety of parallelism available today.</p> <p>In this post, Michelle Strout, general workshop chair, and Engin Kayraklioglu, program committee chair, summarize the highlights of the recent 9th Annual Chapel Implementers and Users Workshop (CHIUW 2022). Read on to hear about some exciting applications that are using Chapel productively, the coding day that happened the day before the workshop, updates on the project, and feedback the team received during and after the workshop.</p> <p><strong>Applications Written in Chapel</strong></p> <p>Programmers generally enjoy trying out new programming languages, but like to see example use cases before they lean on it.  CHIUW provides them with an opportunity to hear about and ask questions about many different use cases. The Chapel programming language is being used productively in a range of application domains: data science, aeronautical simulations, cosmology simulations, and quantum diagonalization to name a few. CHIUW featured the following talks on different Chapel applications:</p> <ul> <li><a href="https://youtu.be/vBxPTzIRRr0">Large-Scale and User-Friendly Exact Diagonalization in Chapel</a></li> <li><a href="https://youtu.be/uTE_RZkODOk">Recent Developments in the CHApel Multi-Physics Simulation Software</a></li> <li><a href="https://youtu.be/YrXYpgnt4rQ">UltraLight Dark Matter in Simulations: A Chapel-Powered Eigenstate Perspective</a></li> <li><a href="https://youtu.be/pstRsgMhCDA">Implementing and Optimizing Parquet I/O in Chapel</a></li> <li><a href="https://youtu.be/xI9EByv7A5M">Truss Analytics Algorithms and Integration in Arkouda</a></li> <li><a href="https://youtu.be/400jmMzdzHQ">From C and Python to Chapel as My Main Programming Language</a></li> </ul> <p><strong>Coding Day</strong></p> <p>For Coding Day, anyone interested in working one-on-one or in small groups with developers from the Chapel team at HPE could indicate their interest in an online form.  We had 7 different sessions.  Programmers interested in Chapel were able to ask questions specific to their Chapel code, interactively make changes to the code, and work through issues with Chapel developers, who were present to provide immediate assistance. In these sessions, Chapel users and developers worked on:</p> <ul> <li>Adding diagnostic support for Chapel runtime’s remote data cache</li> <li>Writing a cell-list module/library in Chapel</li> <li>Investigating porting a Dask application to Arkouda</li> <li>Implementing a Lisp interpreter in Chapel</li> <li>Optimizing distributed memory performance of a very large-scale matrix-vector multiplication</li> <li>Discussing Chapel’s nascent GPU support and going over the internals of the current implementation</li> <li>Learning Chapel in a peer-programming setting</li> </ul> <p>Next year, we plan to keep Coding Day virtual, grow it to include more small groups at non-intersecting times, and publish a schedule ahead of time.  Email Engin at <a href="mailto:engin@hpe.com">engin@hpe.com</a> if you have any thoughts about what you would like to work on or see during next year’s Chapel coding day.</p> <p><strong>Chapel Project Updates</strong></p> <p>One update we heard from Chapel developers was that the new parser was being used in the production Chapel compiler.  This is important because an often-heard complaint from Chapel users is that the compiler is too slow.  The current production compiler does whole program compilation and thus is not able to take advantage of separate and incremental compilation approaches.  The current <em>dyno</em> effort within the Chapel team is redesigning the Chapel compiler to enable separate, incremental, and in general more dynamic compilation.</p> <p>Another important update we heard was in regards to the ever-growing GPU support for Chapel.  Currently some <code>forall</code> loops in Chapel are compiled for CPUs and as GPU kernels.  Which version to run is selected at runtime.  Below is an example of Chapel code that currently runs as a GPU kernel on machines where one or more GPUs are available.</p> <p><img src="/img/chiuw22-gpu-code-example.jpg" alt="coforall gpu in here.gpus do on gpu { var A, B, C: [1..n] real; const alpha = 2.0; B = 1.0; C = 2.0; A = B + alpha + C; }" title="Chapel Code that is offloaded to all the gpus on a locale/node."></p> <p>Keep an eye on the <a href="https://chapel-lang.org/docs/technotes/gpu.html">GPU Programming Technical Note</a> for new features as GPU support in Chapel continues to expand.  More GPU support means handling an especially difficult kind of parallelism that programmers struggle with these days.</p> <p><strong>Feedback from Attendees</strong></p> <p>One user, Nelson Dias, gave a talk about moving to Chapel from the C and Python programming languages.  Nelson’s abstract states, “Chapel is a very elegant language, providing the power and speed of C and Fortran, while allowing a high degree of abstraction and expressiveness that rivals Python's. I have used it in the last two years for: calculating statistics over massive turbulence datasets, implementing models for lake evaporation in hydrology, and testing some relatively simple numerical solutions of partial differential equations.”  <a href="https://youtu.be/400jmMzdzHQ">His talk</a> details the advantages and disadvantages he found while programming in Chapel.</p> <p>In a post-workshop survey, attendees provided the following feedback to help improve CHIUW:</p> <ul> <li>Add short tutorials on how to use language features in a real language, as well as a debugging and performance analysis tutorial. (The Chapel team plans on doing these for next year's Chapel Coding Day.)</li> <li>Expand presentation topics (The CHIUW organizers can encourage new submissions from the community next year)</li> <li>Explore how people are doing performance optimizations in Chapel applications (For next year, we will encourage such submissions from the community)</li> <li>Talk more about libraries, specifically have one about parallel/distributed libraries in Chapel and another about wrapping C libraries in Chapel and lightweight Python wrappers for Chapel</li> <li>Offer more about the internals of the Chapel compiler, runtime, and libraries</li> </ul> <p>Attendees also pointed out that their favorite Chapel features included parallel iterators, domains, global view memory, separation of concerns, and multi-resolution parallel programming.</p> <p><strong>Summary</strong></p> <p>Thank you for taking the time to read this post summarizing the recent Chapel workshop that highlights applications using Chapel, Coding Day, updates for the project, and feedback the team received during and after the workshop.</p> <p>Check out all of the talk videos, slides, and submissions at the <a href="https://chapel-lang.org/CHIUW2022.html">9th Annual Chapel Implementers and Users Workshop (CHIUW 2022)</a> website.  Come interact with the open-source Chapel project at the <a href="https://chapel-lang.org/">Chapel website</a>, on <a href="https://github.com/chapel-lang/">GitHub</a>, <a href="https://stackoverflow.com/questions/tagged/chapel">StackOverflow</a>, <a href="https://www.facebook.com/ChapelLanguage">Facebook</a>, <a href="https://twitter.com/ChapelLanguage">Twitter</a>, <a href="https://chapel.discourse.group/">Discourse</a>, or <a href="https://www.youtube.com/c/ChapelParallelProgrammingLanguage">YouTube</a>.  Consider how Chapel could help you solve some of your parallel programming challenges.</p>https://developer.hpe.com/scale-kubernetes-cluster-using-hpe-greenlake-terraform-provider/https://developer.hpe.com/scale-kubernetes-cluster-using-hpe-greenlake-terraform-provider/Tue, 12 Jul 2022 14:06:02 GMT<p>The process of managing and provisioning computer data centers through machine-readable definition files, also known as Infrastructure-as-Code (IaC), offers many significant benefits. It helps to increase operational agility, simplify management, reduce errors, and save cost.</p> <p>In this post, I will explore options to declare and scale Kubernetes clusters on HPE GreenLake using the HPE GreenLake Terraform Provider.</p> <h1>Prerequisite</h1> <p>Before starting this tutorial, it is recommended that you read the blog post <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code - Part 1</a>, which includes steps for creating a Kubernetes cluster. This post expands upon that scenario by examining how to scale a cluster.</p> <h1>Verify existing Kubernetes cluster</h1> <p>After the cluster is created following the instructions found in the <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">Kubernetes Cluster as Code - Part 1 blog post</a>, launch HPE GreenLake Central console and verify that the cluster is present under the appropriate tenant.</p> <p>You should see the <strong>tf-test</strong> cluster present under <strong>Dashboard -> Manage your Private Cloud -> Containers</strong>.</p> <p><img src="/img/cluster_list.jpg" alt=""></p> <p>Below is the reference Terraform configuration file for the existing cluster.</p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_site" "blr" { name = "BLR" space_id = var.HPEGL_SPACE } data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } resource hpegl_caas_cluster test { name = "tf-test" blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id site_id = data.hpegl_caas_site.blr.id space_id = var.HPEGL_SPACE } </code></pre> <h1>Add worker node to Kubernetes cluster resource</h1> <p>You can scale the cluster by adding a worker node. The following worker-node attributes are specified to add or modify node pools in the declared Kubernetes cluster resource.</p> <ol> <li><strong>name</strong>: Fill in the name that would ideally represent each node pool.</li> <li><strong>machine_blueprint_id</strong>: Fill in the ID for the machine blueprint that is already present in HPE GreenLake Central for your tenant. Use the machine blueprint data source to retrieve the machine blueprint ID.</li> <li><strong>min_size</strong>: Add the number of minimum nodes to be present as part of this node pool. The autoscaler will not scale the nodepool below this number.</li> <li><strong>max_size</strong>: Add the number of maximum nodes to be present as part of this node pool. The autoscaler will not scale the nodepool above this number.</li> </ol> <p>Below is the reference Terraform configuration for creating the cluster with additional nodes.</p> <pre><code class="language-hcl">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_site" "blr" { name = "BLR" space_id = var.HPEGL_SPACE } data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } data "hpegl_caas_machine_blueprint" "standard_worker" { name = "standard-worker" site_id = data.hpegl_caas_site.blr.id } resource hpegl_caas_cluster test { name = "tf-test" blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id site_id = data.hpegl_caas_site.blr.id space_id = var.HPEGL_SPACE worker_nodes { name = "test-node-pool" machine_blueprint_id = data.hpegl_caas_machine_blueprint.standard_worker.id min_size = "1" max_size = "2" } } </code></pre> <p><em>Note: Machine blueprints are used to define the infrastructure details for the worker nodes used in a cluster. A machine blueprint includes the following:</em></p> <ul> <li><em>Machine provider</em></li> <li><em>Operating system image and version</em></li> <li><em>Number of vCPU cores and amount of memory in the node</em></li> </ul> <h1>Ready to Terraform plan</h1> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run <strong>terraform plan</strong></p> <pre><code class="language-shellsession">$ terraform plan hpegl_caas_cluster.test: Refreshing state... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # hpegl_caas_cluster.test will be updated in-place ~ resource "hpegl_caas_cluster" "test" { id = "a32fabb9-7c19-42d1-9a38-ebf122810c0a" name = "tf-test" # (17 unchanged attributes hidden) + worker_nodes { + min_size = "1" + max_size = "2" + machine_blueprint_id = "0ac21c99-2fdb-491d-a590-a5016690b80b" + name = "test-node-pool" } } Plan: 0 to add, 1 to change, 0 to destroy. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. </code></pre> <h1>Ready to Terraform apply</h1> <p>Terraform apply executes the actions proposed in the Terraform plan and updates the resources. Run the command <strong>terraform apply</strong> and type <strong>yes</strong> when asked to <strong>Enter a value</strong>.</p> <pre><code class="language-shellsession">$ terraform apply hpegl_caas_cluster.test: Refreshing state... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # hpegl_caas_cluster.test will be updated in-place ~ resource "hpegl_caas_cluster" "test" { id = "a32fabb9-7c19-42d1-9a38-ebf122810c0a" name = "tf-test" # (17 unchanged attributes hidden) + worker_nodes { + min_size = "1" + max_size = "2" + machine_blueprint_id = "0ac21c99-2fdb-491d-a590-a5016690b80b" + name = "test-node-pool" } } Plan: 0 to add, 1 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes hpegl_caas_cluster.test: Modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 1m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 3m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 5m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 7m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 9m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 11m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 13m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 15m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 17m10s elapsed] hpegl_caas_cluster.test: Still modifying... [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a, 19m10s elapsed] hpegl_caas_cluster.test: Modifications complete after 19m18s [id=a32fabb9-7c19-42d1-9a38-ebf122810c0a] Apply complete! Resources: 0 added, 1 changed, 0 destroyed. </code></pre> <p>From the HPE GreenLake edge-to-cloud platform, launch the HPE GreenLake Central console. Navigate to <strong>Dashboard -> Manage your Private Cloud -> Containers</strong> and select the <strong>tf-test</strong> cluster created. You will see additional nodes with <strong>Node Pool</strong> name as "<strong>test-node-pool</strong>" being created successfully.</p> <p><img src="/img/cluster_detail_page.jpg" alt=""></p> <h1>Scale Options with Auto Scaler</h1> <p>Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster nodepool within the min_size and max_size value. It increases the size of the cluster when there are pods that failed to schedule due to insufficient resources and decreases the size of the cluster when some nodes are consistently unneeded for a significant amount of time.</p> <p>The above example is specifically for adding a single worker node pool to an existing cluster. Below are all the possible options available for auto scaling.</p> <ol> <li><strong><em>Add worker node pools</em>:</strong> You can add multiple node pools by simply declaring corresponding <strong>worker_nodes</strong> in the same cluster resource.</li> <li><strong><em>Reduce worker node pools</em>:</strong> Remove <strong>worker_nodes</strong> associated with a specific node pool from the cluster resource.</li> <li><strong><em>Scaling up/down worker node</em>:</strong> Updating the <strong>min_size</strong> and <strong>max_size</strong> field increases(scale up) or decreases(scale down) the number of nodes under each node pool. If both the values are same, auto scaling will be disabled. In any case, <strong>min_size</strong> cannot be greater than <strong>max_size</strong>.</li> <li><strong><em>Increase/decrease default worker node count</em>:</strong> Every cluster by default has a worker node even if <strong>worker_nodes</strong> are not declared in the Terraform configuration. This originally comes from what's declared in the cluster blueprint. You can override and update the min_size, max_size and machine blueprint for this default worker by declaring <strong>worker_nodes</strong> with the default worker node name. </li> </ol> <p>Note: If you remove the default node pool (<strong>worker_nodes</strong> with the default worker node name) in your configuration file, the default configuration coming from the cluster blueprint shall be retained.</p> <h1>Summary</h1> <p>In this blog, I covered how to scale Kubernetes clusters with Terraform provider for HPE GreenLake. I showed you how to update an existing cluster with additional worker nodes. I also discussed several options available to increase or reduce worker nodes across different node pools for Kubernetes clusters.</p> <p>I hope you found this information interesting and useful while considering the scale of Kubernetes cluster with HPE GreenLake Terraform provider. Use the following links to understand more about Terraform and HPE GreenLake Terraform Provider.</p> <p>-      <a href="https://www.terraform.io/"> Learn more about Terraform</a></p> <p>-      <a href="https://www.hpe.com/us/en/greenlake.html"> Learn more about HPE GreenLake</a></p> <p>-      <a href="https://registry.terraform.io/providers/HPE/hpegl"> Learn more about the HPE GreenLake Terraform provider</a></p> <p>Don’t forget, you can always find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog/tag/hpe-greenlake">HPE Developer blog</a>.</p>https://developer.hpe.com/configuring-azure-ad-with-greenlake-cloud-platform-and-aruba-central/https://developer.hpe.com/configuring-azure-ad-with-greenlake-cloud-platform-and-aruba-central/Mon, 11 Jul 2022 12:04:30 GMT<p>Single sign-on (SSO) enables users to securely authenticate with multiple applications and websites by logging in only once using just one set of credentials (username and password). With SSO, the application or website that the user is trying to access relies on a trusted third party (identity provider) to verify that users are who they say they are.</p> <p>Azure Active Directory (Azure AD) is a cloud-based identity and access management service that helps you access external resources, such as Microsoft 365, the Azure portal, and thousands of other SaaS applications. Aruba Central uses Security Assertion Markup Language (SAML) identiy provider (idP) to issue authentication assertions in conjunction with a single sign-on profile. In this blog post, I'll explain the process for configuring Azure AD to authenticate users into the HPE GreenLake Cloud Platform (HPE GLCP) and Aruba Central using SAML idP.</p> <p>I'll explain the process for configuring Azure AD to authenticate users into HPE Greenlake Cloud Platform (HPE GLCP) and Aruba Central using SAML idP.</p> <p>If you're looking for the Okta version of this information, it can be found on <a href="https://www.wifi-guys.com/?p=512">WIFI-GUYS</a></p> <h2>Contents</h2> <!-- prettier-ignore-start --> <ul> <li><a href="#before-you-begin">Before you Begin</a></li> <li><a href="#terms-used-in-this-document">Terms used in this blog post</a></li> <li><a href="#steps-to-configure-ssosaml-application-in-azure-ad">Steps to Configure SSO/SAML Application in Azure AD</a></li> <li><a href="#step-1-create-an-azure-ad-enterprise-application">Step 1: Create an Azure AD Enterprise Application</a></li> <li><a href="#step-2-configure-gclp-for-saml-federation">Step 2: Configure GCLP for SAML Federation</a></li> <li><a href="#login-to-glcp-and-aruba-central-using-azure-ad">Login to HPE GLCP and Aruba Central using Azure AD</a></li> <li><a href="#using-azure-ad-mfa">Using Azure AD MFA</a></li> <li><a href="#troubleshooting">Troubleshooting</a></li> <li><a href="#appendix-generating-the-hpe_ccs_attribute">Appendix: Generating the hpe_ccs_attribute</a></li> </ul> <!-- prettier-ignore-end --> <h2>Before you Begin</h2> <p>This blog post references the following documentation:</p> <ul> <li> <p><a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=ccs-help_en_us">HPE Greenlake User Guide</a></p> </li> <li> <p><a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=a00092451en_us&#x26;page=GUID-CD81FAF8-9601-4773-899F-049A506FEE2E.html">Single sign-on (SS) authentication</a></p> </li> <li> <p><a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=a00120892en_us">HPE Greenlake Platform Guide</a></p> </li> </ul> <p>If you're looking for the Central 2.5.4 SAML integration guide, <a href="https://github.com/michaelrosejr/arubasso/tree/main/Central254">it has been moved</a>.</p> <h2>Terms used in this blog post</h2> <ul> <li><strong>CCS</strong>: Common Cloud Service</li> <li><strong>GLCP</strong>: HPE GreenLake Cloud Platform</li> <li><strong>SSO</strong>: Single Sign On</li> <li><strong>SAML</strong>: Security Assertion Markup Language</li> <li><strong>idP</strong>: Identity Providers</li> <li><strong>AD</strong>: Active Directory</li> <li><strong>MFA</strong>: Multi-Factor Authentication</li> <li><strong>MSP</strong>: Managed Service Proivder</li> <li><strong>XML</strong>: eXtensible Markup Language</li> </ul> <h2>Steps to Configure a SSO/SAML Application in Azure AD</h2> <p>To configure SSO in Aruba Central, first download the metadata file from Azure AD.</p> <ol> <li>Create an Enteprise Application in the [Azure Portal](https://portal.azure.com)</li> <li>Configure the Enterprise Application for HPE GLCP</li> <li>Download the federated metadata XML file from Enterprise Application</li> <li>Claim and configure your domain within HPE GLCP</li> <li>Upload the federated metadata XML file to HPE GLCP </li> <li>Create a recovery account</li></ol> <h2>Step 1: Create an Azure AD Enterprise Application</h2> <ul> <li> <p>Log into to the Azure portal.</p> </li> <li> <p>Click <strong>Enterprise Applications</strong> (you may need to search for it, if it's not on your menu)</p> </li> <li> <p>Click <strong>New Application</strong></p> </li> </ul> <p><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/new_app.png" alt="Image"></p> <ul> <li> <p>Click <strong>Create your own Application</strong></p> <p>Enter the name of your app. (Ex: Aruba Central USWEST 4)</p> </li> </ul> <p><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/create_app.png" alt="Image"></p> <ul> <li> <p>Select <strong>Integrate any other application you don't find in the gallery (Non-gallery)</strong></p> </li> <li> <p>Under Step 1: Assign users and groups, select the AD Group you created at the beginning of this document.</p> </li> </ul> <p><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/assign-users-groups.png" alt="Image"></p> <ul> <li> <p>Under Step 2: Set Up Single Sign-On</p> </li> <li> <p>The default setting is Disabled. Select <strong>SAML</strong></p> </li> </ul> <p><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/select-saml.png" alt="Image"></p> <ul> <li>Under Basic SAML Configuration, click <strong>Edit</strong></li> </ul> <table> <thead> <tr> <th align="center">Attribute</th> <th align="center">Values</th> </tr> </thead> <tbody> <tr> <td align="center"><strong>Identifier (Entity ID):</strong></td> <td align="center"><a href="https://sso.common.cloud.hpe.com">https://sso.common.cloud.hpe.com</a></td> </tr> <tr> <td align="center"><strong>Reply URL (Assertion Consumer Service URL):</strong></td> <td align="center"><a href="https://sso.common.cloud.hpe.com/sp/ACS.saml2">https://sso.common.cloud.hpe.com/sp/ACS.saml2</a></td> </tr> <tr> <td align="center"><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/azure-saml-ccs-urls.png" alt="azure-saml-ccs-urls" height="50%" width="50%"></td> <td align="center"></td> </tr> </tbody> </table> <ul> <li> <p>Under Attributes &#x26; Claims</p> <table> <thead> <tr> <th>Attribute</th> <th>Value</th> </tr> </thead> <tbody> <tr> <td>emailaddress</td> <td>user.givenname</td> </tr> <tr> <td>name</td> <td>user.userprincipalname</td> </tr> <tr> <td>gl_first_name</td> <td>user.givenname</td> </tr> <tr> <td>gl_last_name</td> <td>user.surname</td> </tr> <tr> <td>hpe_ccs_attribute</td> <td>See Below</td> </tr> </tbody> </table> <pre><code class="language-markdown">version_1#2fd5f97acbc211ecadc006baf610dd36:00000000-0000-0000-0000-000000000000:Account Administrator:ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator:ALL_SCOPES </code></pre> <pre><code class="language-markdown">Where the PCID (2fd5f97acbc211ecadc006baf610dd36) is your ID for HPE GLCP and App ID (683da368-66cb-4ee7-90a9-ec1964768092) for your Central cluster </code></pre> </li> </ul> <p><strong>For more details on the <code>hpe_ccs_attritube</code>, see the Appendix: <a href="#appendix-generating-the-hpe_ccs_attribute">Generating the <code>hpe_ccs_attribute</code></a></strong> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/azure-saml-custom-attributes-img1.png" alt="Image"> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/azure-saml-hpe_ccs_attribute.png" alt="Image"></p> <ul> <li>Click <strong>Download</strong> under Step 3 : Federation Metadata XML.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/azure-saml-federation-metadata-download.png" alt="azure-saml-federation-metadata-download" height="50%" width="50%"> <h2>Step 2: Configure GCLP for SAML Federation</h2> <ul> <li>Login to HPE GLCP and select Manage.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/manage.png" alt="manage" height="50%" width="50%"> <ul> <li>Select the Authentication tile.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs-authentication.png" alt="ccs_authentication" height="50%" width="50%"> <ul> <li>Claim your domain for SAML.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs-claim-domain.png" alt="claim_domain" height="50%" width="50%"> <ul> <li>Upload the <em>Federation Metadata XML</em> file from the previous section.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs-samle-azure-metadata-summry.png" alt="metadatasummary" height="70%" width="70%"> <ul> <li>Apply the following configuration settings. These should match the First and Last Name settings you set above for Azure.</li> </ul> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs-saml-config-settings-summary.png" alt="saml-settings" height="70%" width="70%"> <ul> <li> <p>Create the recovery user per the instructions.</p> </li> <li> <p>Validate the settings are correct.</p> </li> <li> <p>Save and finish the configuration.</p> </li> <li> <p>If you get an error that the SAML configuration wasn't completed using the account with the @domain.com, you'll have to log out and login again with the SAML domain and go through the above configuration again.</p> </li> </ul> <h2>Log in to HPE GLCP and Aruba Central using Azure AD</h2> <ul> <li>Once you've completed the above steps, log in to HPE Greenlake Central using your Azure AD email.</li> </ul> <p><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs_login.png" alt="ccs_login" height="40%" width="40%"><img src="/img/0b085a5aef05404e9ecdf52cb9088feb/ccs_login_saml.png" alt="ccs_login_saml" height="40%" width="40%"></p> <ul> <li>If everything is working correctly, you should have logged into HPE GLCP and find the Aruba Central application tile with an button to "Launch" the Aruba Central application.</li> </ul> <h2>Using Azure AD MFA</h2> <p>By default, Azure AD enables Multi-Factor Authentication (MFA). However, for testing and demos, it's much easier to disable MFA on your accounts. To disable MFA, please see the following documentation: <a href="https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults">What are security defaults</a></p> <h2>Troubleshooting</h2> <ul> <li> <p>There's a useful 3rd-party browser tool called: SAML Tracer</p> </li> <li> <p>This tool will allow you to verify the attributes you're sending to Central.</p> </li> <li> <p>It can be useful when configuration SAML with multiple HPE Greenlake Central accounts or domains.</p> </li> <li> <p>SAML Tracer</p> </li> </ul> <p><a href="https://chrome.google.com/webstore/detail/saml-tracer/mpdajninpobndbfcldcmbpnnbhibjmch?hl=en">Chrome</a> <a href="https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/">FireFox</a> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/firefox-saml-tracer.png" alt="Image"></p> <h2>Appendix: Generating the <code>hpe_ccs_attribute</code></h2> <p>The <code>hpe_ccs_attribute</code> is used to determine your HPE GLCP account. The format for the <code>hpe_ccs_attribute</code> is as follows:</p> <img src="/img/0b085a5aef05404e9ecdf52cb9088feb/hpe_ccs_attribute-img1.png" alt="hpe_ccs_attribute-img1" height="75%" width="75%"> <p>An example <code>hpe_ccs_attribute</code> for a single HPE GLCP and Aruba Central account would be:</p> <pre><code class="language-markdown">version_1#2fd5f97acbc211ecadc006baf610dd36:00000000-0000-0000-0000-000000000000:Account Administrator:ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator:ALL_SCOPES </code></pre> <p>or</p> <pre><code class="language-markdown">version_1#5b0ec0e8b4f411eca432ba72799953ac:00000000-0000-0000-0000-000000000000:Account Administrator:ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator:ALL_SCOPES#5b0ec0e8b4f411eca432ba72799953ac:00000000-0000-0000-0000-000000000000:Account Administrator:ALL_SCOPES </code></pre> <p>If you're a Managed Service Provider (MSP), then the <code>hpe_ccs_attribute</code> for Administrator rights to HPE GLCP and Aruba Central for all customer tenant accounts is as follows:</p> <pre><code class="language-markdown">version_1#d951f8c8c67711eca2cf9efb55836a4d:00000000-0000-0000-0000-000000000000:Account Administrator|TENANT|:ALL_SCOPES:00000000-0000-0000-0000-000000000000:Account Administrator|MSP|:ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator|TENANT| : ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator|MSP| : ALL_SCOPES </code></pre> <p>The <code>hpe_ccs_attribute</code> string for a tenant under a MSP account is shown below. Please note, you <strong>must</strong> have the SAML domain configuration configured for that tenant account using the <strong>same</strong> setting as the MSP account. To say it another way, you <strong>must</strong> go through this configuration for each tenant account under the MSP.</p> <pre><code class="language-markdown">version_1#f9ee1cdecc1611ecb00e9e24ed17d2a7:00000000-0000-0000-0000-000000000000:Observer|TENANT| :ALL_SCOPES:683da368-66cb-4ee7-90a9-ec1964768092:Aruba Central Administrator|TENANT| :ALL_SCOPES </code></pre> <h2>Conclusion</h2> <p>In this blog post, you learned how to configure Azure AD with HPE Greenlake by passing the necessary configuration and customizations using the hpe_ccs_attribute. From this point, you can create custom attributes to grant different level of access based on roles such as Read/Write or Read/Only access.</p> <p>If you have feedback to this blog post, please send me a message.</p> <p>Be sure to come back to the HPE Developer Community blog for more articles on this and other interesting subjects.</p>https://developer.hpe.com/2022-July-06/https://developer.hpe.com/2022-July-06/Wed, 06 Jul 2022 13:00:00 GMThttps://developer.hpe.com/service-mesh-security-hardening-–-using-spire-with-istio/https://developer.hpe.com/service-mesh-security-hardening-–-using-spire-with-istio/Mon, 27 Jun 2022 16:22:05 GMT<p>Building applications using microservices offers developers the ability to better scale their applications and take better advantage of public and hybrid cloud architectures. The ability to split each part of the application into independent codebases that perform one specific task means that each self-contained service can increase in size independently as its needs change, providing for this scalability. And it allows cross-functional teams to develop, test, and update services independently, leading to faster deployments and updates.</p> <p>Though there are significant advantages to a microservices architecture, it can also be much more complex to manage and secure. With the potential for hundreds of services, it’s challenging for developers to keep track of component interactions, health, performance, and security.</p> <p>In a recent SPIFFE blog post, Nathalia Satie Gomazako points out how a service mesh solves the problem of inter-service communications. By controlling service-to-service communication over a network, it allows separate parts of an application to communicate with one another. She explains how Istio is a very popular service mesh that does just this.</p> <p>Nathalia goes on to explain how SPIRE, the reference implementation of SPIFFE, the Secure Production Identity Framework for Everyone, can integrate with Istio and assist with security concerns, especially when dealing with a multi-cloud infrastructure. This integration extends Istio capabilities by allowing workloads to be identified and to get their identities by a pre-defined set of assigned attributes. With this attestation process, Istio can securely issue cryptographic identities to workloads.</p> <p>Her article, <a href="https://blog.spiffe.io/hardening-istio-security-with-spire-d2f4f98f7a63">Hardening Istio security with SPIRE</a>, is a quick 3-minute read and quite informative. I highly recommend it.</p>https://developer.hpe.com/kubernetes-cluster-as-code-part-2/https://developer.hpe.com/kubernetes-cluster-as-code-part-2/Thu, 23 Jun 2022 14:01:05 GMT<h2>Getting started</h2> <p>The process of managing and provisioning computer data centers through machine-readable definition files, also known as Infrastructure-as-Code (IaC), offers many significant benefits, like:</p> <ul> <li>Increased operational agility</li> <li>Simplified management</li> <li>Reduced errors</li> <li>Saved cost</li> </ul> <p>Using the HPE GreenLake Terraform provider, you can bring up a Kubernetes cluster starting right from the infrastructure layer and way up in the stack to set up desired configurations and applications. In the diagram below, 2 and 3 are community providers that are available and can be used in combination with HPE GreenLake TF provider to deploy applications on the Kubernetes cluster. In this blog post, I will illustrate how this can be implemented using Terraform.</p> <p>Using similar Terraform files as shown in this blog post, customers can deploy any application of their choice on the Kubernetes cluster by consuming the appropriate community providers available. With this capability, customers can customize the Kubernetes clusters based on their needs.</p> <p><img src="/img/image2022-6-20_12-36-56.png" alt=""></p> <h2>Let's recap</h2> <p>In <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">my first blog post</a>, I covered the usage of the HPE GreenLake Terraform provider to create and destroy a Kubernetes cluster and discussed how to use community providers, in combination with HPE GreenLake TF provider, to create a namespace in the Kubernetes cluster.</p> <p>In this blog post, I will focus on managing application deployments using IaC. Here, I will be deploying a Prometheus application on an existing/pre-created Kubernetes cluster. Hence, the pre-requisite to proceed would be to have a Kubernetes cluster and a namespace created. You could follow the steps mentioned in <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">my first blog post</a> to achieve this.</p> <h2>Application deployment on a Kubernetes cluster</h2> <h3>Helm provider</h3> <p>Below is the code block for adding <strong>helm</strong> community provider. Please refer to <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">my first blog post</a> for details regarding <strong>hpegl_caas_cluster</strong> data source.</p> <pre><code class="language-json">provider "helm" { kubernetes { host = yamldecode(base64decode(data.hpegl_caas_cluster.tf-test-7.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.tf-test-7.kubeconfig)).users[0].user.token insecure = true } } </code></pre> <h3>Helm-release Terraform resource for Prometheus stack deployment</h3> <p>In order to deploy Prometheus stack using the helm-release resource, the following values have to be filled in the <strong>prometheus-deploy.tf</strong> file:</p> <ol> <li>Cluster Name: Fill in the <strong>name</strong> of the pre-created cluster in <strong>hpegl_caas_cluster</strong> block. In the below example, name= "tf-test-7"</li> <li>Namespace: Fill in the appropriate <strong>namespace</strong> in the <strong>helm_release</strong> block. In the below example, namespace= "test-namespace" </li> </ol> <p><strong>prometheus-deploy.tf</strong> </p> <blockquote> <p>Note: You can name this file according to your preference. We are using prometheus-deploy.tf here for easy reference.</p> </blockquote> <pre><code class="language-json">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider "hpegl" { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_cluster" "tf-test-7" { name = "tf-test-7" space_id = var.HPEGL_SPACE } provider "helm" { kubernetes { host = yamldecode(base64decode(data.hpegl_caas_cluster.tf-test-7.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.tf-test-7.kubeconfig)).users[0].user.token insecure = true } } resource "helm_release" "prometheus-stack" { name = "prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" chart = "kube-prometheus-stack" version = "36.0.2" namespace = "test-namespace" set { name = "grafana.service.type" value = "NodePort" } } </code></pre> <h3>Initializing workspace &#x26; synchronizing infrastructure components</h3> <p>Place the <strong>prometheus-deploy.tf</strong> file in your working directory and initialize the working directory using the command: <strong>terraform init</strong></p> <pre><code class="language-markdown">$ terraform init Initializing the backend... Initializing provider plugins... - Reusing previous version of hashicorp/helm from the dependency lock file - Reusing previous version of terraform.example.com/caas/hpegl from the dependency lock file - Using previously-installed hashicorp/helm v2.5.1 - Using previously-installed terraform.example.com/caas/hpegl v0.0.1 Terraform has made some changes to the provider dependency selections recorded in the .terraform.lock.hcl file. Review those changes and commit them to your version control system if they represent changes you intended to make. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. </code></pre> <h3>Terraform ready to plan</h3> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run: <strong>terraform plan</strong></p> <pre><code class="language-markdown">$ terraform plan Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # helm_release.prometheus-stack will be created + resource "helm_release" "prometheus-stack" { + atomic = false + chart = "kube-prometheus-stack" + cleanup_on_fail = false + create_namespace = false + dependency_update = false + disable_crd_hooks = false + disable_openapi_validation = false + disable_webhooks = false + force_update = false + id = (known after apply) + lint = false + manifest = (known after apply) + max_history = 0 + metadata = (known after apply) + name = "prometheus-stack" + namespace = "test-namespace" + recreate_pods = false + render_subchart_notes = true + replace = false + repository = "https://prometheus-community.github.io/helm-charts" + reset_values = false + reuse_values = false + skip_crds = false + status = "deployed" + timeout = 300 + verify = false + version = "36.0.2" + wait = true + wait_for_jobs = false + set { + name = "grafana.service.type" + value = "NodePort" } } Plan: 1 to add, 0 to change, 0 to destroy. ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. </code></pre> <h3>Terraform ready to apply</h3> <p>Terraform apply executes the actions proposed in the Terraform plan and deploys the resources. Run <strong>terraform apply</strong> and then type yes when asked to <strong>Enter a value</strong>.</p> <pre><code class="language-markdown">$ terraform apply Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # helm_release.prometheus-stack will be created + resource "helm_release" "prometheus-stack" { + atomic = false + chart = "kube-prometheus-stack" + cleanup_on_fail = false + create_namespace = false + dependency_update = false + disable_crd_hooks = false + disable_openapi_validation = false + disable_webhooks = false + force_update = false + id = (known after apply) + lint = false + manifest = (known after apply) + max_history = 0 + metadata = (known after apply) + name = "prometheus-stack" + namespace = "test-namespace" + recreate_pods = false + render_subchart_notes = true + replace = false + repository = "https://prometheus-community.github.io/helm-charts" + reset_values = false + reuse_values = false + skip_crds = false + status = "deployed" + timeout = 300 + verify = false + version = "36.0.2" + wait = true + wait_for_jobs = false + set { + name = "grafana.service.type" + value = "NodePort" } } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes helm_release.prometheus-stack: Creating... helm_release.prometheus-stack: Still creating... [10s elapsed] helm_release.prometheus-stack: Still creating... [20s elapsed] helm_release.prometheus-stack: Still creating... [30s elapsed] helm_release.prometheus-stack: Still creating... [40s elapsed] helm_release.prometheus-stack: Still creating... [50s elapsed] helm_release.prometheus-stack: Still creating... [1m0s elapsed] helm_release.prometheus-stack: Still creating... [1m10s elapsed] helm_release.prometheus-stack: Still creating... [1m20s elapsed] helm_release.prometheus-stack: Still creating... [1m30s elapsed] helm_release.prometheus-stack: Still creating... [1m40s elapsed] helm_release.prometheus-stack: Still creating... [1m50s elapsed] helm_release.prometheus-stack: Still creating... [2m0s elapsed] helm_release.prometheus-stack: Creation complete after 2m2s [id=prometheus-stack] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. </code></pre> <p>After Terraform apply is complete, you can see the Prometheus pods deployed by running the command:</p> <pre><code class="language-bash">$ kubectl get po -n &#x3C;namespace> </code></pre> <p><img src="/img/12.png" alt=""></p> <p>You can see the Prometheus services by running the command: </p> <pre><code class="language-bash">$ kubectl get svc-n &#x3C;namespace> </code></pre> <p><img src="/img/13.png" alt=""></p> <p>You can fetch the node IP by running the command: </p> <pre><code class="language-bash">$ kubectl get nodes -o wide </code></pre> <p><img src="/img/14.png" alt=""></p> <p>Since Grafana is exposed as a NodePort, this <strong>PORT</strong>: 32424 and the node's internal IP <strong>INTERNAL-IP</strong>: 172.16.17.168, can be used to access Grafana portal as follows:</p> <pre><code class="language-markdown">https://&#x3C;INTERNAL-IP>:&#x3C;PORT> </code></pre> <p>For example, <a href="http://172.16.17.168:32424">http://172.16.17.168:32424</a></p> <p><img src="/img/14-2-.png" alt=""></p> <h2>Summary</h2> <p>In <a href="https://developer.hpe.com/blog/kubernetes-clusters-as-code-part1/">my first blog post</a>, I covered how to get started with the Terraform provider for HPE GreenLake and explained how to create a Kubernetes cluster and bring up a namespace on the Kubernetes cluster using Kubernetes community provider. In this article, I showed you how to manage application deployments on a Kubernetes cluster using Terraform.</p> <p>I hope you found this information interesting and useful in helping you get started with HPE GreenLake Terraform provider. You could also go through the below links to understand more about Terraform and HPE GreenLake Terraform Provider</p> <p>-      <a href="https://www.terraform.io/"> Learn more about Terraform</a></p> <p>-      <a href="https://www.hpe.com/us/en/greenlake.html"> Learn more about HPE GreenLake</a></p> <p>-      <a href="https://registry.terraform.io/providers/HPE/hpegl"> Learn more about the HPE GreenLake Terraform provider</a></p> <p>Don’t forget, you can always find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog/tag/hpe-greenlake">HPE Developer blog</a>.</p>https://developer.hpe.com/kubernetes-clusters-as-code-part1/https://developer.hpe.com/kubernetes-clusters-as-code-part1/Thu, 23 Jun 2022 10:15:09 GMT<h2>Getting started</h2> <p>The process of managing and provisioning computer data centers through machine-readable definition files, also known as Infrastructure-as-Code (IaC), offers many significant benefits. It helps to increase operational agility, simplify management, reduce errors, and save cost. In this post, we will explore some of the benefits of using IaC to build a Kubernetes cluster from scratch, with all the necessary configuration and core services, on HPE GreenLake using Terraform (TF). Storing Kubernetes cluster and favorable configurations as code helps in repeatability and change management.</p> <p>IaC with Kubernetes is not new. There are providers in the developer community that are quite good and well supported. Using the HPE GreenLake Terraform provider, you can bring up a Kubernetes cluster starting right from the infrastructure layer and way up in the stack to set up desired configurations and applications. For reference, see the below picture.</p> <p><img src="/img/image2022-6-20_12-36-56.png" alt=""></p> <p>HPE GreenLake TF provider brings the Kubernetes stack up on the HPE GreenLake infrastructure, and exposes credentials for other TF providers to integrate further and build the complete stack, as desired. In the diagram above, 2 and 3 are community providers that are available, which can be used in combination with HPE GreenLake TF provider.</p> <p>One of the options provided by HPE GreenLake is to make it easy for customers to order and operate a private cloud with a mix of virtual machines, containers, and physical servers. This is exactly what the private cloud service is all about. It provides access via a public API, allowing developers to use an infrastructure-as-code type of tool to automate provisioning, for example using Terraform. For customers to try out everything that is mentioned in this blog series, they should have subscribed to <strong>HPE GreenLake for private cloud enterprise</strong> that unifies Virtual Machines, Containers and Bare-metal in one cloud service.</p> <p>In this two-part blog series, I’ll share my experience as a first-time user of HPE Greenlake TF provider. This blog series aims to provide a step by step walkthrough of how to bring up a Kubernetes cluster using Terraform and how to deploy applications on a pre-created Kubernetes cluster.</p> <p>In this first part, I will focus on the pre-requisites needed prior to using HPE GreenLake TF provider and the steps to be followed to bring up a Kubernetes cluster. I will also discuss on how 3rd party community providers can be used in tandem with HPE Greenlake TF provider.</p> <p>In <a href="https://developer.hpe.com/blog/kubernetes-cluster-as-code-part-2/">the second part of this series</a>, I will illustrate how to deploy applications on a namespace in the pre-created Kubernetes cluster, using Terraform.</p> <h2>Preparing for infrastructure-as-code implementation </h2> <h3>Setting up API Client access</h3> <p>You need an API client to authenticate against HPE GreenLake.</p> <blockquote> <p>Note: You should have <strong>IAM Owner</strong> role for the appropriate tenant to proceed with API Client creation.</p> </blockquote> <p>Follow the below steps for API Client creation:</p> <ol> <li>From the HPE GreenLake platform, launch the <strong>HPE GreenLake Central console</strong> for the appropriate tenant. Under the settings icon on the tenant <strong>Dashboard</strong> page, select <strong>User Management</strong> option.</li> </ol> <p><img src="/img/dashboard.png" alt=""></p> <ol start="2"> <li>Under the <strong>API Clients</strong> tab, click on <strong>Create API Client</strong>.</li> </ol> <p><img src="/img/2.png" alt=""></p> <ol start="3"> <li> Enter a <strong>Name</strong> (mandatory field) and <strong>Description</strong> (optional) for the API client, and click on <strong>Create</strong> button.</li> </ol> <p><img src="/img/3.png" alt=""></p> <ol start="4"> <li> Ensure you make a note of the <strong>Issuer</strong>, <strong>Client ID</strong> and <strong>Client Secret</strong> before clicking on the <strong>Close</strong> button. These details will be exported as environment variables in the next section.</li> </ol> <p><img src="/img/4.png" alt=""></p> <ol start="5"> <li>In the <strong>API Clients</strong> page, select the newly created client, and click on <strong>Create Assignment</strong> button.</li> </ol> <p><img src="/img/5.png" alt=""></p> <ol start="6"> <li>Create an assignment with <strong>Role Assignment:</strong> <strong>Private Cloud Cluster Owner</strong> and <strong>Space:</strong> <strong>Default.</strong></li> </ol> <p><img src="/img/6.png" alt=""></p> <p>The API client is now ready to be used to run the Terraform resources.</p> <h3>Selecting a Terraform provider with container service configurations</h3> <h4>1. Ensure you have Terraform installed.</h4> <p>Terraform can be installed by following: <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">Terraform Installation</a></p> <p>Installation can be verified using the below command.</p> <pre><code class="language-bash">$ terraform version Terraform v1.1.9 on linux_amd64 </code></pre> <h4>2. Export the following environment variables on your machine.</h4> <p>Export the Tenant ID and Space ID:</p> <pre><code class="language-markdown">export HPEGL_TENANT_ID=&#x3C;Tenant ID> export TF_VAR_HPEGL_SPACE=&#x3C;Space ID> </code></pre> <p>Export the API client details based on what was noted down.</p> <pre><code class="language-markdown">export HPEGL_USER_ID=&#x3C;Client ID> export HPEGL_USER_SECRET=&#x3C;Client Secret> export HPEGL_IAM_SERVICE_URL=&#x3C;Issuer> </code></pre> <h4>3. Choosing the appropriate Terraform provider.</h4> <p>The first section of the Terraform file will enumerate the “providers” you rely upon for building your infrastructure, and there could be multiple providers in a single TF file. In this case here, you will have the HPE GreenLake provider referenced as hpe/hpegl (<strong>source</strong>) and the available versions to choose from (<strong>version</strong>), in the official <a href="https://registry.terraform.io/providers/HPE/hpegl/0.2.2">Terraform registry</a>.</p> <p>The first lines of your Terraform configuration file should look like this:</p> <pre><code class="language-json">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } </code></pre> <h2>Create a cluster resource</h2> <h3>Terraform data source for cluster blueprint</h3> <p>In order to use the data source available for cluster blueprint, you should add the below block in your Terraform file, and specify the cluster blueprint name. Using this data source, Terraform will fetch the cluster Blueprint ID associated with it.</p> <p>In the below block, "demo" is the cluster blueprint <strong>name</strong> provided:</p> <pre><code class="language-json">data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } </code></pre> <p>The cluster blueprint ID can now be fetched by using:</p> <pre><code class="language-json">blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id </code></pre> <h3>Terraform data source for site</h3> <p>In order to use the data source available for site, you should add the below block in your Terraform file, and specify the site name. Using this data source, Terraform will fetch the site ID associated with it.</p> <p>In the below block, "BLR" is the site <strong>name</strong> provided:</p> <pre><code class="language-json">data "hpegl_caas_site" "blr" { name = "BLR" space_id = "" } </code></pre> <p>The site ID can now be fetched by using:</p> <pre><code class="language-json">site_id = data.hpegl_caas_site.blr.id </code></pre> <h3> Terraform data source for cluster</h3> <p>In order to use the data source available for cluster, you should add the below block and provide the cluster name and space id. Using this data source, Terraform will fetch the cluster server and user token associated with it.</p> <p>In the below block, "tf-test" is the <strong>name</strong> of the pre-created Kubernetes cluster.</p> <pre><code class="language-json">data "hpegl_caas_cluster" "test" { name = "tf-test" space_id = "" } </code></pre> <p>The <strong>host</strong> (cluster server) and <strong>token</strong> (user token) can now be fetched from the cluster kubeconfig using:</p> <pre><code class="language-json">host = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).users[0].user.token </code></pre> <h3>Terraform resource for cluster</h3> <p>In order to create a Kubernetes cluster using the cluster resource, the following values should be specified in the <strong>cluster-create.tf</strong> file shown below:</p> <ol> <li>Site Name: Fill in the appropriate site <strong>name</strong> in the <strong>hpegl_caas_site</strong> block. In the below example, name= "BLR" </li> <li>Cluster Blueprint Name: Fill in the appropriate cluster blueprint <strong>name</strong> in the <strong>hpegl_caas_cluster_blueprint</strong> block. In the below example, name= "demo" </li> <li>Cluster Name: Fill in the cluster <strong>name</strong> of your choice in the <strong>hpegl_caas_cluster</strong> block. In the below example, name= "tf-test"</li> </ol> <blockquote> <p>Note: Here, the space_id is automatically set to the value specified while exporting TF_VAR_HPEGL_SPACE.</p> </blockquote> <p><strong>cluster-create.tf</strong></p> <blockquote> <p>Note: You can name this file according to your preference. We are using cluster-create.tf here for easy reference.</p> </blockquote> <pre><code class="language-json">terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_site" "blr" { name = "BLR" space_id = var.HPEGL_SPACE } data "hpegl_caas_cluster_blueprint" "bp" { name = "demo" site_id = data.hpegl_caas_site.blr.id } resource hpegl_caas_cluster test { name = "tf-test" blueprint_id = data.hpegl_caas_cluster_blueprint.bp.id site_id = data.hpegl_caas_site.blr.id space_id = var.HPEGL_SPACE } </code></pre> <p>You can get information about each of the data sources and resources mentioned above from <a href="https://github.com/HPE/terraform-provider-hpegl/tree/main/docs">Github</a>.</p> <h3>Initializing workspace &#x26; synchronizing infrastructure components</h3> <p>Place the cluster-create.tf file in your working directory and initialize the working directory using the command: <strong>terraform init</strong></p> <pre><code class="language-markdown">$ terraform init Initializing the backend... Initializing provider plugins... - Finding hpe/hpegl versions matching ">= 0.2.0"... - Installing hpe/hpegl v0.2.2... - Installed hpe/hpegl v0.2.2 (signed by a HashiCorp partner, key ID D1F277A1AC66CE3D) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. </code></pre> <h3> Terraform ready to plan</h3> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run: <strong>terraform plan</strong></p> <pre><code class="language-markdown">$ terraform plan Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # hpegl_caas_cluster.test will be created + resource "hpegl_caas_cluster" "test" { + api_endpoint = (known after apply) + appliance_name = (known after apply) + blueprint_id = "982be053-ae7d-4623-9af7-69a299e68bc9" + cluster_provider = (known after apply) + created_date = (known after apply) + default_storage_class = (known after apply) + default_storage_class_description = (known after apply) + health = (known after apply) + id = (known after apply) + k8s_version = (known after apply) + kubeconfig = (known after apply) + last_update_date = (known after apply) + machine_sets = (known after apply) + machine_sets_detail = (known after apply) + name = "tf-test" + service_endpoints = (known after apply) + site_id = "ecb6b8a4-3303-4528-96d1-42230336a9ec" + space_id = "8d5dfbc0-f996-4e45-ab34-e719588a96ca" + state = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. </code></pre> <h3>Terraform ready to apply</h3> <p>Terraform apply executes the actions proposed in the Terraform plan and deploys the resources. Run the command: <strong>terraform apply</strong> and type <strong>yes</strong> when asked to <strong>Enter a value</strong>.</p> <pre><code class="language-markdown">$ terraform apply Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # hpegl_caas_cluster.test will be created + resource "hpegl_caas_cluster" "test" { + api_endpoint = (known after apply) + appliance_name = (known after apply) + blueprint_id = "982be053-ae7d-4623-9af7-69a299e68bc9" + cluster_provider = (known after apply) + created_date = (known after apply) + default_storage_class = (known after apply) + default_storage_class_description = (known after apply) + health = (known after apply) + id = (known after apply) + k8s_version = (known after apply) + kubeconfig = (known after apply) + last_update_date = (known after apply) + machine_sets = (known after apply) + machine_sets_detail = (known after apply) + name = "tf-test" + service_endpoints = (known after apply) + site_id = "ecb6b8a4-3303-4528-96d1-42230336a9ec" + space_id = "8d5dfbc0-f996-4e45-ab34-e719588a96ca" + state = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes hpegl_caas_cluster.test: Creating... hpegl_caas_cluster.test: Still creating... [10s elapsed] hpegl_caas_cluster.test: Still creating... [1m0s elapsed] hpegl_caas_cluster.test: Still creating... [5m0s elapsed] hpegl_caas_cluster.test: Still creating... [10m0s elapsed] hpegl_caas_cluster.test: Still creating... [20m0s elapsed] hpegl_caas_cluster.test: Still creating... [25m0s elapsed] hpegl_caas_cluster.test: Still creating... [25m10s elapsed] hpegl_caas_cluster.test: Still creating... [25m20s elapsed] hpegl_caas_cluster.test: Still creating... [25m30s elapsed] hpegl_caas_cluster.test: Still creating... [25m40s elapsed] hpegl_caas_cluster.test: Still creating... [25m50s elapsed] hpegl_caas_cluster.test: Still creating... [26m0s elapsed] hpegl_caas_cluster.test: Still creating... [26m11s elapsed] hpegl_caas_cluster.test: Still creating... [26m21s elapsed] hpegl_caas_cluster.test: Still creating... [26m31s elapsed] hpegl_caas_cluster.test: Still creating... [26m41s elapsed] hpegl_caas_cluster.test: Still creating... [26m51s elapsed] hpegl_caas_cluster.test: Still creating... [27m1s elapsed] hpegl_caas_cluster.test: Still creating... [27m11s elapsed] hpegl_caas_cluster.test: Still creating... [27m21s elapsed] hpegl_caas_cluster.test: Still creating... [27m31s elapsed] hpegl_caas_cluster.test: Still creating... [27m41s elapsed] hpegl_caas_cluster.test: Still creating... [27m51s elapsed] hpegl_caas_cluster.test: Still creating... [28m1s elapsed] hpegl_caas_cluster.test: Still creating... [28m11s elapsed] hpegl_caas_cluster.test: Still creating... [28m21s elapsed] hpegl_caas_cluster.test: Still creating... [28m31s elapsed] hpegl_caas_cluster.test: Still creating... [28m41s elapsed] hpegl_caas_cluster.test: Still creating... [28m51s elapsed] hpegl_caas_cluster.test: Still creating... [29m1s elapsed] hpegl_caas_cluster.test: Creation complete after 29m1s [id=8a3396db-ae26-44fd-a128-264c357f71fb] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. </code></pre> <p>From the HPE GreenLake platform, launch the <strong>HPE GreenLake Central console</strong> for the appropriate tenant, and from the <strong>Dashboard</strong>, select <strong>Clusters</strong> to view the list of clusters. You will see <strong>tf-test</strong> cluster has been created successfully.</p> <p><img src="/img/create.png" alt=""></p> <h2>Delete a cluster resource</h2> <p>In Terraform, clean-up can be done using the destroy command. This will automatically use the HPE GreenLake provider to clean the infrastructure in HPE GreenLake. Run the following command: <strong>terraform destroy</strong></p> <pre><code class="language-markdown">$ terraform destroy hpegl_caas_cluster.test: Refreshing state... [id=8a3396db-ae26-44fd-a128-264c357f71fb] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # hpegl_caas_cluster.test will be destroyed - resource "hpegl_caas_cluster" "test" { - api_endpoint = "https://gl-caas.gl-hpe.local:10003" -> null - appliance_name = "Austin" -> null - blueprint_id = "982be053-ae7d-4623-9af7-69a299e68bc9" -> null - cluster_provider = "ecp" -> null - created_date = "2022-06-12T11:11:55Z" -> null - default_storage_class = "gl-sbc-glhc-nimblestor" -> null - health = "ok" -> null - id = "8a3396db-ae26-44fd-a128-264c357f71fb" -> null - k8s_version = "v1.20.11.hpe-2" -> null - kubeconfig = "YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGluc2VjdXJlLXNraXAtdGxzLXZlcm lmeTogdHJ1ZQogICAgc2VydmVyOiBodHRwczovL2dsLWNhYXMuZ2wtaHBlLmxvY2FsOjEwMDAzCiAgbmFtZTogZGVmYXVsdApjb250ZXh0czoKLSBjb2 50ZXh0OgogICAgY2x1c3RlcjogZGVmYXVsdAogICAgbmFtZXNwYWNlOiBkZWZhdWx0CiAgICB1c2VyOiBkZWZhdWx0CiAgbmFtZTogZGVmYXVsdApjdX JyZW50LWNvbnRleHQ6IGRlZmF1bHQKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBkZWZhdWx0CiAgdXNlcjoKICAgIH Rva2VuOiBleUpoYkdjaU9pSlNVekkxTmlJc0ltdHBaQ0k2SWpKM2NVNTRhbFpvT0RJM04xVTRRalZ0TjJ3elNUQnpVVmhxUkhCVWVrOURVRk5PT1VWRVc ExOW5ha0VpZlEuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFl XTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSnJkV0psTFhONWMzUmxiU0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpj bVYwTG01aGJXVWlPaUowWlhKeVazlkUkU2cXA1RG5od3pDZnRvX2FQMXBPZwo=" -> null - last_update_date = "2022-06-12T11:41:53Z" -> null - machine_sets = [ - { - count = 1 - machine_blueprint_id = "25d70b74-1048-4aae-8994-aa72f08cfacb" - name = "master" - os_image = "" - os_version = "" }, - { - count = 1 - machine_blueprint_id = "ebcb6ad2-b201-4c92-9dad-1bbb0b90cde3" - name = "worker" - os_image = "" - os_version = "" }, ] -> null - machine_sets_detail = [ - { - compute_type = "General Purpose" - count = 1 - machine_blueprint_id = "25d70b74-1048-4aae-8994-aa72f08cfacb" - machine_provider = "vmaas" - machine_roles = [ - "controlplane", - "etcd", ] - machines = [ - { - created_date = "2022-06-12T11:11:55Z" - health = "ok" - hostname = "172.20.20.102" - id = "4247" - last_update_date = "2022-06-12T11:41:35Z" - name = "k8s-tf-test-master-mshls-hwlgf" - state = "ready" }, ] - name = "master" - networks = [ - "VM_Production", ] - os_image = "sles-custom" - os_version = "15" - proxy = "" - size = "Large" - size_detail = [ - { - cpu = 4 - ephemeral_disk = 500 - memory = 16384 - name = "Large" - persistent_disk = 0 - root_disk = 120 }, ] - storage_type = "General Purpose" }, - { - compute_type = "General Purpose" - count = 1 - machine_blueprint_id = "ebcb6ad2-b201-4c92-9dad-1bbb0b90cde3" - machine_provider = "vmaas" - machine_roles = [ - "worker", ] - machines = [ - { - created_date = "2022-06-12T11:11:55Z" - health = "ok" - hostname = "172.20.20.121" - id = "4248" - last_update_date = "2022-06-12T11:41:35Z" - name = "k8s-tf-test-worker-58mvq-qvdch" - state = "ready" }, ] - name = "worker" - networks = [ - "VM_Production", - "iSCSI_A", - "iSCSI_B", ] - os_image = "sles-custom" - os_version = "15" - proxy = "" - size = "xLarge" - size_detail = [ - { - cpu = 8 - ephemeral_disk = 500 - memory = 32768 - name = "xLarge" - persistent_disk = 0 - root_disk = 120 }, ] - storage_type = "General Purpose" }, ] -> null - name = "tf-test" -> null - service_endpoints = [ - { - endpoint = "https://gl-caas.gl-hpe.local:10004" - name = "Kubernetes Dashboard" - namespace = "kubernetes-dashboard" - type = "system" }, ] -> null - site_id = "ecb6b8a4-3303-4528-96d1-42230336a9ec" -> null - space_id = "8d5dfbc0-f996-4e45-ab34-e719588a96ca" -> null - state = "ready" -> null } Plan: 0 to add, 0 to change, 1 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes hpegl_caas_cluster.test: Destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 10s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 1m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 2m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 3m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 4m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 5m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 6m0s elapsed] hpegl_caas_cluster.test: Still destroying... [id=8a3396db-ae26-44fd-a128-264c357f71fb, 7m40s elapsed] hpegl_caas_cluster.test: Destruction complete after 7m48s Destroy complete! Resources: 1 destroyed. </code></pre> <p>From the HPE GreenLake platform, launch the <strong>HPE GreenLake Central console</strong> for the appropriate tenant, and from the <strong>Dashboard</strong>, select <strong>Clusters</strong> to view the list of clusters. You will see <strong>tf-test</strong> cluster has been deleted.</p> <p><img src="/img/delete.png" alt=""></p> <h2>Additional 3rd party provider of your choice from community</h2> <p>You can also use a 3rd party provider of your choice from the community. In this section, we will discuss on how to create a namespace in GL CaaS cluster using Kubernetes community provider.</p> <h3>Kubernetes provider</h3> <p>Below is the code block for adding <strong>Kubernetes</strong> provider:</p> <pre><code class="language-json">provider "kubernetes" { host = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).users[0].user.token insecure = true } </code></pre> <h3>Terraform resource for namespace</h3> <p>You can create a Kubernetes namespace using the <strong>kubernetes_namespace</strong> resource and providing a namespace <strong>name</strong> of your choice. In the below example, name = "test-namespace".</p> <pre><code class="language-json">resource "kubernetes_namespace" "test-namespace" { metadata { name = "test-namespace" } lifecycle { ignore_changes = [ metadata[0].labels, ] } } </code></pre> <h3>Namespace creation using Terraform</h3> <p><strong>namespace-create.tf</strong> : Below is a complete example of using the Kubernetes provider and creating a namespace on a pre-created Kubernetes cluster.</p> <blockquote> <p>Note: You can name this file according to your preference. We are using namespace-create.tf here for easy reference.</p> </blockquote> <pre><code class="language-json"># Copyright 2020 Hewlett Packard Enterprise Development LP # Set-up for terraform >= v0.13 terraform { required_providers { hpegl = { source = "hpe/hpegl" version = ">= 0.2.2" } } } provider hpegl { caas { } } variable "HPEGL_SPACE" { type = string } data "hpegl_caas_cluster" "test" { name = "tf-test-7" space_id = var.HPEGL_SPACE } provider "kubernetes" { host = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).clusters[0].cluster.server token = yamldecode(base64decode(data.hpegl_caas_cluster.test.kubeconfig)).users[0].user.token insecure = true } resource "kubernetes_namespace" "test-namespace" { metadata { name = "test-namespace" } lifecycle { ignore_changes = [ metadata[0].labels, ] } } </code></pre> <h3>Initializing workspace &#x26; synchronizing infrastructure components</h3> <p>Place the namespace-create.tf file in your working directory and initialize the working directory using the command: <strong>terraform init</strong></p> <pre><code class="language-markdown">$ terraform init Initializing the backend... Initializing provider plugins... - Finding hpe/hpegl versions matching ">= 0.2.0"... - Installing hpe/hpegl v0.2.2... - Installed hpe/hpegl v0.2.2 (signed by a HashiCorp partner, key ID D1F277A1AC66CE3D) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. </code></pre> <h3>Terraform ready to plan</h3> <p>Terraform plan is a dry run that lets you preview the changes that Terraform plans to make to your infrastructure based on the data you provide in your Terraform file. To see this, run: <strong>terraform plan</strong></p> <pre><code class="language-markdown">$ terraform plan Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # kubernetes_namespace.test-namespace will be created + resource "kubernetes_namespace" "test-namespace" { + id = (known after apply) + metadata { + generation = (known after apply) + name = "test-namespace" + resource_version = (known after apply) + uid = (known after apply) } } Plan: 1 to add, 0 to change, 0 to destroy. ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. </code></pre> <h3>Terraform ready to apply</h3> <p>Terraform apply executes the actions proposed in the Terraform plan and deploys the resources. Run <strong>terraform apply</strong> and then type <strong>yes</strong> when asked to <strong>Enter a value</strong>.</p> <pre><code class="language-markdown">$ terraform apply Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # kubernetes_namespace.test-namespace will be created + resource "kubernetes_namespace" "test-namespace" { + id = (known after apply) + metadata { + generation = (known after apply) + name = "test-namespace" + resource_version = (known after apply) + uid = (known after apply) } } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes kubernetes_namespace.test-namespace: Creating... kubernetes_namespace.test-namespace: Creation complete after 0s [id=test-namespace] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. </code></pre> <p>You can verify the created namespace <strong>test-namespace</strong>, by running the command: <strong>kubectl get namespace</strong></p> <p><img src="/img/11.png" alt=""></p> <h2>Next up</h2> <p>In <a href="https://developer.hpe.com/blog/kubernetes-cluster-as-code-part-2/">my next blog post</a>, we will continue our discussion on deploying applications on a pre-created Kubernetes cluster.</p>https://developer.hpe.com/lift-and-transform-apps-with-hpe-csi-driver-for-kubernetes/https://developer.hpe.com/lift-and-transform-apps-with-hpe-csi-driver-for-kubernetes/Fri, 17 Jun 2022 10:26:26 GMT<p>Do you run your applications in virtual machines on VMware today using the same patterns once employed by bare-metal paradigms? It's just a bit more efficient from a resource consumption perspective, but what about the lifecycle of the application, its dependencies — including libraries, host operating systems and various mnemonics — used to keep the application up and running?</p> <p>Containerization, and Kubernetes in particular, helps businesses move away from imperative operational models that require multiple teams to perform tedious tasks on a regular basis. It allows them to adopt an agile declarative model where there's a clean separation of concerns, along with a high degree of automation and abstractions that make sense for running a high-performance technology company.</p> <p>In this blog post, we’ll discuss a methodology that can be employed for lifting and transforming legacy stateful applications running on VMware vSphere to Kubernetes using HPE Alletra, Nimble Storage or Primera and the HPE CSI Driver for Kubernetes. The destination Kubernetes cluster could potentially be running on VMware vSphere or any other hypervisor, but it’s not a requirement. As long as the Kubernetes cluster and the origin VMware vSphere environment has connectivity to the underlying array, we’re in good shape.</p> <p>The TL;DR version of this blog post is available as an instructional lightboard video <a href="https://www.youtube.com/watch?v=M7t_qPe3i5E">published on YouTube</a>. The blog post concretizes what’s being outlined in the lightboard video.</p> <p><a href="https://www.youtube.com/watch?v=M7t_qPe3i5E"><img src="/img/untitled.jpg" alt="YouTube"></a></p> <p>Still want more details? Let’s get started!</p> <h1>Origin state</h1> <p>In the following example, we’re going to use a MariaDB database with the commonly used <a href="https://github.com/datacharmer/test_db">Employees database</a> as an example workload. It provides a simple means to validate its contents and ensure that the contents are intact throughout its journey.</p> <p>The database is running in a standalone Virtual Machine (VM) hosted on a legacy NFS datastore.</p> <p><img src="/img/slide1.png" alt="MariaDB running on VMware vSphere" title="MariaDB running on VMware vSphere"></p> <p>The content is simply validated by loading a SQL statement file provided in the GitHub repo.</p> <pre><code class="language-markdown">$ mysql -u mmattsson -ppassword &#x3C; test_employees_sha.sql INFO TESTING INSTALLATION table_name expected_records expected_crc departments 9 4b315afa0e35ca6649df897b958345bcb3d2b764 dept_emp 331603 d95ab9fe07df0865f592574b3b33b9c741d9fd1b dept_manager 24 9687a7d6f93ca8847388a42a6d8d93982a841c6c employees 300024 4d4aa689914d8fd41db7e45c2168e7dcb9697359 salaries 2844047 b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f titles 443308 d12d5f746b88f07e69b9e36675b6067abb01b60e table_name found_records found_crc departments 9 4b315afa0e35ca6649df897b958345bcb3d2b764 dept_emp 331603 d95ab9fe07df0865f592574b3b33b9c741d9fd1b dept_manager 24 9687a7d6f93ca8847388a42a6d8d93982a841c6c employees 300024 4d4aa689914d8fd41db7e45c2168e7dcb9697359 salaries 2844047 b5a1785c27d75e33a4173aaa22ccf41ebd7d4a9f titles 443308 d12d5f746b88f07e69b9e36675b6067abb01b60e table_name records_match crc_match departments OK ok dept_emp OK ok dept_manager OK ok employees OK ok salaries OK ok titles OK ok computation_time 00:00:49 summary result CRC OK count OK </code></pre> <p>In the remaining tests throughout this post, only the summary results will be displayed.</p> <h1>Destination state</h1> <p>When the transition has completed, the Employees database should be running on a Kubernetes cluster using persistent storage provided by the HPE CSI Driver for Kubernetes backed by a HPE Nimble Storage array. The procedures are similar regardless of the backend (HPE Alletra and Primera included).</p> <p><img src="/img/slide2.png" alt="MariaDB running on Kubernetes" title="MariaDB running on Kubernetes"></p> <p>While the destination state in our example is a virtualized Kubernetes cluster within the same vSphere environment, that is not a requirement. It can be any hypervisor, bare-metal or otherwise as long as the destination cluster is compatible with the HPE CSI Driver for Kubernetes and any of the supported backends.</p> <h1>Initial transition</h1> <p>The first step in becoming agile with the dataset of interest (the MariaDB database in this case) is to transition the virtual machine's storage to a VMware vSphere Virtual Volume (vVol) from the legacy NFS datastore. This is done with VMware vSphere Storage vMotion that allows the live migration of a running virtual machine's file system from one storage system to another, with no downtime for the VM or service disruption.</p> <p><img src="/img/vmotion-screen-shot-2022-06-14-at-12.08.27-pm.png" alt="VMware vSphere Storage vMotion example." title="VMware vSphere Storage vMotion example."></p> <p>Another important detail is that the data that matters to the application resides in a filesystem compatible with the destination Kubernetes cluster and without partitioning schemes or volume managers. We know for a fact that MariaDB stores all its data on <code>/var/lib/mysql</code>, so let’s investigate that path.</p> <pre><code class="language-markdown">$ df -h /var/lib/mysql Filesystem Size Used Avail Use% Mounted on /dev/sda 64G 1.2G 63G 2% /var </code></pre> <p>In the output we can clearly see that the filesystem of interest resides on a disk device without a volume manager or a partitioning scheme. If this is not the case, the data needs to migrate within the VM to a new blank disk.</p> <p>Once the disk(s) have been migrated over the HPE Nimble Storage array, each individual virtual disk is represented as a standalone volume along with the VM's metadata volumes.</p> <p><img src="/img/vvol-screen-shot-2022-06-14-at-1.12.59-pm.png" alt="vVol Datastore Volumes" title="vVol Datastore Volumes"></p> <p>The 64GB volume represents the <code>/var</code> filesystem subject for transforming into a Kubernetes persistent volume.</p> <h1>Iterate and learn</h1> <p>In this next phase of the workflow, all configuration and testing are performed directly from the Kubernetes cluster. An admin needs to install the HPE CSI Driver for Kubernetes, create a <code>Secret</code> that represents the backend user or tenant and setup a <code>StorageClass</code> designed to allow the importing of volumes.</p> <p>There are several ways to <a href="https://scod.hpedev.io/csi_driver/deployment.html">deploy the HPE CSI Driver</a> but the most common method is to use Helm. Here's the gist of it.</p> <pre><code class="language-markdown">$ kubectl create ns hpe-storage $ helm install my-hpe-csi-driver hpe-storage/hpe-csi-driver --version 2.1.1-0 -n hpe-storage </code></pre> <p>Next, create a <code>Secret</code> for the backend.</p> <pre><code class="language-yaml">apiVersion: v1 kind: Secret metadata: name: hpe-backend namespace: hpe-storage stringData: serviceName: nimble-csp-svc servicePort: "8080" backend: &#x3C;Your Nimble array> username: &#x3C;Your username with privileges to the vVol folder> password: &#x3C;Your password> </code></pre> <p>A <code>StorageClass</code> needs to be specifically crafted to allow users to import volumes and snapshots from the storage array.</p> <pre><code class="language-yaml">apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: hpe-transform provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: xfs csi.storage.k8s.io/controller-expand-secret-name: hpe-backend csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage description: Volume transformed from vVol by HPE CSI Driver allowOverrides: importVolAsClone,importVolumeName,forceImport reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>The important details in the <code>StorageClass</code> are the <code>allowOverrides</code>. Here, we put the parameters we want the application admin to override once testing has started (<code>importVolAsClone</code>) and once the database is ready for the final import (<code>importVolumeName</code> and <code>forceImport</code>).</p> <p>For the final production instance, we’re going to use the <a href="https://artifacthub.io/packages/helm/bitnami/mariadb">Bitnami MariaDB Helm chart</a>. For this chart to install cleanly and be able to bring up the database, the following considerations need to be taken into account:</p> <ul> <li>There needs to be a known <code>root@localhost</code> account with a known password in the source database.</li> <li>The Bitnami chart does not support altering the <code>mountPath</code>. The source filesystem needs to have a symlink to <code>/var/lib/mysql</code> from <code>/var/lib/data</code>.</li> </ul> <p>For the initial cloning of the source database, the following <code>values.yaml</code> file is being used.</p> <pre><code class="language-yaml">image: tag: 10.3 auth: rootPassword: my-password global: imagePullSecrets: # Only needed if you're rate limited to Docker Hub - regcred primary: persistence: enabled: true storageClass: hpe-transform annotations: csi.hpe.com/importVolAsClone: virt-mm-db-1.vmdk size: 64Gi subPath: lib </code></pre> <p>The above parameters are quite self-explanatory. The <code>subPath</code> directive instructs the kubelet to mount the <code>lib</code> directory in the root of the volume. As a result, <code>/lib</code> in the volume will be mounted at <code>/bitnami/mariadb</code> and the <code>data</code> subdirectory, which holds the MariaDB databases, will be seen properly by the container binaries.</p> <p>Deploying the chart is straight forward.</p> <pre><code class="language-markdown">$ helm repo add bitnami https://charts.bitnami.com/bitnami $ helm install my-test-clone bitnami/mariadb -f values.yaml </code></pre> <p>Once the chart is deployed and the database comes up, the "Hello World" test will, of course, verify the database.</p> <pre><code class="language-markdown">$ kubectl exec -it my-test-clone-mariadb-0 -- bash I have no name!@my-test-clone-mariadb-0:/$ mysql -ummattsson -ppassword &#x3C; /bitnami/mariadb/scripts/test_employees_sha.sql INFO TESTING INSTALLATION computation_time 00:00:49 summary result CRC OK count OK </code></pre> <p>It's likely that things may not come up properly on the first couple of iterations. It needs to read logs, error messages, and course correct until the database comes up cleanly. Once it does, in a production scenario, test clients can be attached to further validate and solidify that moving the application to Kubernetes will actually work.</p> <h1>The endgame</h1> <p>Up until now, there’s been zero downtime or disruption to the source application. The final step of this project is a one-way street. Ensure clients can connect to the destination database instance before proceeding.</p> <p>While it’s harmless to leave the test clone chart and <code>PersistentVolumeClaim</code> on the cluster, we’ll remove it to avoid confusion.</p> <pre><code class="language-markdown">$ helm uninstall my-test-clone $ kubectl delete pvc/data-my-test-clone-mariadb-0 </code></pre> <p>Next, we need to prepare the <code>values-prod.yaml</code> file.</p> <pre><code class="language-yaml">image: tag: 10.3 auth: rootPassword: my-password global: imagePullSecrets: # Only needed if you're rate limited to Docker Hub - regcred primary: persistence: enabled: true storageClass: hpe-transform annotations: csi.hpe.com/importVolumeName: virt-mm-db-1.vmdk csi.hpe.com/forceImport: "true" size: 64Gi subPath: lib </code></pre> <p>The only change we need to make is the <code>annotations</code> stanza. Changing the directive from <code>importVolAsClone</code> to <code>importVolumeName</code>. We also need to “force” the import as the volume we’re importing already has residual vSphere application metadata associated with it. That will be overwritten by the HPE CSI Driver when forcing the import.</p> <p>Before deploying the new chart, the source VM needs to be shut down and removed from vSphere control. With HPE Nimble Storage and HPE Alletra 6000, it’s perfectly safe to orderly shut down the VM and “Delete from Disk” in the vCenter context menus. If you’re attempting these workflows and procedures with HPE Alletra 9000, Primera or 3PAR, only use “Remove from Inventory” to prevent disruption.</p> <p>Also, remember to have backups or volume replicas and a contingency plan for any unplanned event.</p> <p>Once the VM has been deleted from vSphere, the remnants on the storage array can be seen as offline volumes. The metadata volumes have been permanently removed.</p> <p><img src="/img/offline-screen-shot-2022-06-14-at-5.45.27-pm.png" alt="Offline HPE Storage Nimble volume" title="Offline HPE Storage Nimble volume"></p> <p>An important detail to understand is that <code>importVolumeName</code> only works for offline volumes on HPE Alletra 6000 and Nimble Storage.</p> <p>Ok, let’s import the database into its final state.</p> <pre><code class="language-markdown">$ helm install my-prod bitnami/mariadb -f values-import.yaml </code></pre> <p>Once the database is up, we can connect to the database and verify the contents yet again.</p> <pre><code class="language-markdown">$ kubectl exec -it my-prod-mariadb-0 -- bash I have no name!@my-prod-mariadb-0:/$ mysql -u mmattsson -ppassword &#x3C; /bitnami/mariadb/scripts/test_employees_sha.sql INFO TESTING INSTALLATION computation_time 00:00:48 summary result CRC OK count OK </code></pre> <p>Back on the storage array, the volume has automatically been renamed and now corresponds to the <code>PersistentVolume</code> name known to Kubernetes.</p> <p><img src="/img/pvc-screen-shot-2022-06-14-at-6.39.40-pm.png" alt="PersistentVolume hosted on HPE Nimble Storage" title="PersistentVolume hosted on HPE Nimble Storage"></p> <p>It’s also safe to move the volume out of the vVol datastore folder, as it’s a non-disruptive operation and may be carried out by a storage administrator.</p> <h1>Summary</h1> <p>While this blog only skims the surface of possibilities of performing IT transformations from thirty-year-old paradigms, it gives an idea of what tools DevOps teams have at their disposal while taking a crack at one of the biggest challenges in application modernization. The ability to iterate safely without production disruption is key. Quite frankly, I performed a dozen clone imports before I got every parameter right before the final import while writing this blog post. There is no easy button, you must put in the work!</p> <p>I’m gearing up for <a href="http://hpe.com/discover">HPE Discover 2022</a>. If you’d like to connect and talk shop, I’ll be in the HPE Developer Community Hack Shack on the main expo floor. <a href="https://developer.hpe.com/blog/don%E2%80%99t-miss-all-things-software-in-the-hack-shack-at-hpe-discover/">We have treasure hunts, hacking challenges and meetups</a>. Find my session “Get Started with Persistent Storage for Kubernetes with the HPE CSI Driver” (HSM4991) in the HPE Discover <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24991_10161&#x26;locale=en_US">content catalog</a>.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24991_10161&#x26;locale=en_US"><img src="/img/hpe-dlv-2022-csi-hackshack-revc.png" alt="Get Started with Persistent Storage for Kubernetes with the HPE CSI Driver"></a></p> <p>The team is also on the HPE Developer Community Slack. <a href="https://slack.hpedev.io/">Sign up</a> and <a href="https://hpedev.slack.com">login</a> to join the conversation on everything HPE Developer related. See you there!</p>https://developer.hpe.com/how-to-set-up-an-automation-pipeline-to-view-historical-trend-data-of-clusters-with-hpe-greenlake-for-containers/https://developer.hpe.com/how-to-set-up-an-automation-pipeline-to-view-historical-trend-data-of-clusters-with-hpe-greenlake-for-containers/Thu, 09 Jun 2022 11:05:39 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Containers is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>An HPE GreenLake Cloud Service, <a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Containers</a>, is built upon <a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a> and runs on a container-based Kubernetes orchestrated infrastructure*.* Using the Clusters module of the HPE GreenLake Central dashboard, one can perform several cluster-related operations. A cluster can be created using default or custom machine blueprints, deleted, scaled up, or scaled-down using this module. Additionally, the cluster detail page provides some interesting insights like CPU usage, memory allocation, and storage availability. It is essential to monitor these types of data so that the resource requirement can be determined without any business impact. The image below illustrates the cluster detail page.</p> <p><img src="/img/cluster-detail.jpg" alt="Cluster detail screen" title="Cluster detail screen"></p> <p>It can be useful to get historical trend data for such metrics to analyze application-specific metrics. Using past consumption data, a customer can decide if the cluster should be scaled up on some days of the year and scaled down on others. For a system administrator working on an abstract layer of application deployment, choosing the right blueprint is crucial. A graph indicating high memory usage over CPU usage might lead him to change blueprints that offer high compute resources. The purpose of this blog post is to illustrate how a person can implement the Automation Pipeline and provide observability via the use of automation tools such as Katalon (UI automation tool), CircleCI (continuous integration and continuous delivery platform), MySQL (a database provider) and Grafana (an interactive visualization web application tool that uses time-series data to create meaningful graphs).</p> <h2>How to Implement the Automation Pipeline?</h2> <p><a href="https://katalon.com/katalon-studio/">Kalalon studio </a>software uses an open-source Selenium framework to interact with web browsers with HTTP commands. CI systems can trigger remote execution of Katalon Studio scripts through Docker containers or command-line interfaces. At a scheduled time and frequency, cron jobs can be configured to check various dynamic measurements of clusters such as CPU usage, memory usage, storage capacity availability, the number of worker nodes in the cluster, the number of control planes in the cluster, the time taken to scale up a cluster node, time taken to scale down a cluster node, the time taken to create a cluster, etc.</p> <p>Katalon Studio provides HTML-based reports or console logs after execution is complete. It is possible to extract the required data in the form of a plain text-based file, such as .csv, using any script. CircleCI offers the capability to export this .csv file as an artifact inside the job. This artifact data can then be imported into the database. In order to visualize the collected data, the Grafana dashboard is helpful. This can be seen in the architecture below.</p> <p><img src="/img/architectural-diagram.jpg" alt="Architectural Diagram of technology stack" title="Architectural Diagram of technology stack"></p> <h2>How to Create a CircleCI pipeline?</h2> <p>A CircleCI pipeline includes various stages such as spinning up the environment by downloading required images and configuring test setup, preparing CircleCI environment, copyright check, cloning repository code, security checking, execution of test scripts, generating artifacts, generating results, and sending out results over email. A cron job schedules the execution of the automation pipeline.</p> <p><strong>Sample CircleCI config.yaml:</strong></p> <pre><code class="language-yaml">executors: katalon8_test_executor: docker: - image: default/katalon8:latest jobs: checkout-workspace: docker: - image: circleci/golang:latest copyright-check: docker: - image: copyright-tool steps: - run: name: Check copy right cluster-observability: executor: name: katalon8_test_executor steps: - run: name: "Creating directory for artifacts" command: mkdir /tmp/project/ - run: name: "Execute cluster metrics gathering test suite" command: xvfb-run katalonc -consoleLog -browserType="Chrome" -retry=0 -statusDelay=15 -testSuitePath="Test Suites/ClusterMetricsSuite" -executionProfile='default' -projectPath='/project/sample.prj' --config -webui.autoUpdateDrivers=true - store_test_results: path: ./tests/katalon/Reports - store_artifacts: path: /tmp/project/ workflows: cluster-observability: triggers: - schedule: cron: "0 20 * * *" filters: branches: only: - master jobs: - checkout-workspace - copyright-check - gather-cluster-metrics requires: - checkout-workspace - cluster-scale-operations requires: - checkout-workspace - update-results requires: - checkout-workspace </code></pre> <p><strong>Sample CircleCI workflow can be demonstrated as below:</strong></p> <p><img src="/img/circleci-workflow.jpg" alt="Sample CircleCI Automation pipeline" title="Sample CircleCI Automation pipeline"></p> <p><strong>A csvDataSource.csv artifact file may look like what's shown below:</strong></p> <pre><code class="language-sql">IDCluster,DateTime,BlueprintType,CPUUsage,MemoryAllocation,StorageCapacity,WorkerNodeCount,ScalingRequired,ClusterScaleUpDuration,ClusterScaleDownDuration,ClusterCreationDuration,ClusterDeletionDuration 2a0810e6-0c32-451b-933b-74fbdf86358a,2022-06-15 20:00:00,standard,50%,70%,5GB,3,N,,,, ce18d4e0-9af3-40da-8d43-266fe05d17ba,2022-06-16 20:00:00,standard,80%,70%,5GB,3,Y,01,,, 767801bd-7f1e-4a9a-804e-b560f168d968,2022-06-17 20:00:00,standard,10%,10%,5GB,4,Y,,01,, r3b185d5-c96a-49a5-b6de-13ae93c93fd4,2022-06-18 20:00:00,standard,07%,10%,5GB,2,N,,,, </code></pre> <p>As illustrated in the above data file, the UI automation run gathers data for the cluster where the application is deployed. A threshold value can be predefined for scaling requirements in the Katalon script. For example, 80% above usage should trigger scaling up, while 10% usage should trigger scaling down. Based on the first record after app deployment, CPU and Memory metrics are 50% and 70%, respectively, for which scaling is not necessary and is marked as 'N' in the data file. The CPU reached 80% usage, on the second day, which means that a scale up operation is required and is marked with 'Y' in the data file. The script for scaling up operation can also record the time it takes to become ready.</p> <h2>How to Download Artifacts from CircleCI Workflow Dynamically?</h2> <p>CircleCI offers <a href="https://circleci.com/docs/2.0/managing-api-tokens/#creating-a-personal-api-token">API Token</a> to view pipelines via an API interface. Based upon requirements, various <a href="https://circleci.com/docs/api/v2/">APIs </a>can be selected. In the <a href="https://www.postman.com/downloads/">Postman </a>tool, a user can try a combination of APIs for passing the output of one API result to the input of another API. A curl command for downloading artifacts from CircleCI API may look like the one mentioned below:</p> <pre><code class="language-shell">curl -H "Circle-Token: $TOKEN" "https://circleci.com/api/v2/project/gh/$repo/$project/$JOB_ID/artifacts" | grep -o 'https://[^"]*csvDataSource[^"]*' \ | wget --timeout=10 --verbose --header "Circle-Token: $TOKEN" --input-file - </code></pre> <h2>Grafana Dashboard Configurations</h2> <p>CircleCI build runs will collect the artifacts and those can be filled into databases like MySQL or Prometheus. In Grafana, various data source configurations are available, where the user has to configure the required data source. There are various chart options available for visual interpretation. By providing various MySQL queries, the required graph can be generated.</p> <p>A sample MySQL query to display the maximum CPU, Memory, and Storage specific to the cluster name can be written for Grafana, as mentioned below.</p> <pre><code class="language-sql">select ClusterName AS "Cluster Name", max(cast(CPU as UNSIGNED)) as "Maximum CPU Usage", max(cast(CPU as UNSIGNED)) as "Maximum Memory Usage", max(cast(CPU as UNSIGNED)) as "Maximum Storage Usage", from ClusterMetricsTable GROUP BY ClusterName; </code></pre> <p><strong>Finally, the Grafana dashboard appears as:</strong></p> <p><img src="/img/cluster-metrics-grafana.jpg" alt="SampleGrafanaDashboard" title="Sample Grafana Dashboard (Data is for illustrative purpose only. Axis are hidden)"></p> <p>Note that, all data illustrated is for understanding purposes only. No relevance to actual HPE GreenLake performance is being shown or claimed in this blog post.</p> <h2>Conclusion</h2> <p>Through this blog, users can integrate various tools and generate metrics for observability purpose. And by triggering actions based on these metrics, several objectives can be achieved, such as, selecting the right blueprint for the application or automating cluster scale-up or scale-down operations.</p>https://developer.hpe.com/benefits-of-the-platform-level-data-model-for-firmware-update-standard/https://developer.hpe.com/benefits-of-the-platform-level-data-model-for-firmware-update-standard/Tue, 07 Jun 2022 16:35:17 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/pldm/pldm_fwupd/pldm_fwupd">Server Management Portal</a></big></p> <br>https://developer.hpe.com/how-to-create-a-virtual-network-in-hpe-greenlake-for-private-cloud/https://developer.hpe.com/how-to-create-a-virtual-network-in-hpe-greenlake-for-private-cloud/Tue, 07 Jun 2022 05:34:46 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>HPE GreenLake for private cloud is designed to deliver and help manage a private cloud. Available on the HPE GreenLake Central platform, the HPE GreenLake for private cloud is</p> <ul> <li>An HPE designed, implemented, owned, and operated private cloud that is deployed at a customer site</li> <li>Offered as a consumption-based service that enables customers to better align costs to outcomes</li> <li>An intuitive self-service portal UI is used to create and manage private cloud services such as compute, storage, and network (example described in this blog)</li> </ul> <p>This blog post explains how a Customer Network Administrator can create a virtual network with a static IP pool and DHCP using NSX-T, network virtualization, and a security platform that enables the virtual cloud network in HPE GreenLake for private cloud.</p> <h2>Prerequisites</h2> <p>Access to Network Management is controlled by a user’s role.</p> <p>With the Tenant Admin user, connect to HPE GreenLake Central, locate the HPE GreenLake for private cloud dashboard widget and click the Launch icon to open the HPE GreenLake for private cloud dashboard.</p> <p>Navigate to Administration > Roles and select the role to update the permission.</p> <p>From the ACCESS column of the selected role, select FULL for the below-mentioned NSX network objects:</p> <ul> <li>Infrastructure: Networks</li> <li>Infrastructure: Network IP Pools</li> </ul> <h2>Understanding private cloud networking</h2> <p>The following illustration shows how you can use NSX objects to achieve NSX logical networking in HPE GreenLake for private cloud.</p> <p><img src="/img/fig-2.jpg" alt=""></p> <p>The network components are as follows:</p> <p><strong>VM</strong></p> <p>Tenant virtual machines (VMs) are connected to Blue and Green networks.</p> <p><strong>NSX-T Segments</strong></p> <p>NSX-T segments are layer 2 virtual domains and there are two types of segments in an NSX-T Data Center:</p> <ul> <li> <p>Overlay-backed segments(<strong>Default</strong>): This enables traffic flow between two virtual machines on different hosts. The hosts are attached to the same overlay segment and have their Layer 2 traffic carried by a tunnel between them.</p> </li> <li> <p>VLAN-backed segments: This is used for uplink traffic external to the NSX-T Data Center.</p> <p><em>Note: Raise HPE Support case to enable the backend infrastructure to support this type.</em></p> </li> </ul> <p><strong>Blue-Network, Green-Network</strong></p> <p>NSX-T segments that are attached to the tenant virtual machines and Tier1 gateway.</p> <p><strong>Tier-1 Gateway</strong></p> <p>Gateway with downlink connections to NSX-T segments and uplink connections to Tier-0 gateways using an internal transit network. Typically, a Tier-1 gateway is connected to a Tier-0 gateway in the northbound direction and to segments in the southbound direction.</p> <p><strong>Internal Transit Network</strong></p> <p>The Network enables the communication between the Tier-0 gateway and all Tier-1 gateways that are linked to it. This connectivity is established when the Tier-1 gateway is attached to the Tier-0 gateway.</p> <p><strong>Tier-0 Gateway</strong></p> <p>Gateway that processes the traffic between the logical and physical networks. A Tier-0 gateway has downlink connections to Tier1 gateways and uplink connections to the physical networks.</p> <p><strong>Ext-Net</strong></p> <p>Interface connected to Virtual Distributed Switch configured in a customer environment for enabling external connectivity from the tenant virtual machines.</p> <h2>How to create a virtual network with a static IP pool</h2> <h3>Step 1: Create IP Pool</h3> <p>Locate HPE GreenLake for private cloud card in the HPE GreenLake Central dashboard and click the Launcher icon to open the HPE GreenLake for private cloud dashboard.</p> <ul> <li> <p>Navigate to Infrastructure > Networks</p> </li> <li> <p>Click the IP Pools tab</p> </li> <li> <p>Click Add to open CREATE NETWORK POOL dialog box</p> </li> <li> <p>Configure the NSX-T IP pool parameters as follows:</p> <ul> <li><strong>Name</strong>: IP Pool Name</li> <li><strong>Pool Type</strong>: Select "Morpheus"</li> <li><strong>IP Ranges</strong>: Specify the IP pool address range by entering the STARTING ADDRESS and ENDING ADDRESS</li> </ul> </li> </ul> <p><img src="/img/fig-1.jpg" alt=""></p> <h3>Step 2: Create NSX-T Segment with Static IP Pool</h3> <p>Locate HPE GreenLake for private cloud card in the HPE GreenLake Central dashboard and click the Launcher icon to open the HPE GreenLake for private cloud dashboard.</p> <ul> <li> <p>Navigate to Infrastructure > Networks</p> </li> <li> <p>From the Networks tab, click the ADD drop-down list, select NSX-T Segment</p> </li> <li> <p>From the dialog box, configure the NSX-T segment parameters as follows. For information about additional fields that are not described here, refer <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=GUID-3DCFD624-DFE7-45A8-AFAC-BE004227C7EC.html">User Guide</a>.</p> <ul> <li><strong>Group</strong>: From the drop-down list, select an infrastructure user group to isolate the network at the group level. The default is Shared (all infrastructure groups)</li> <li><strong>Network Service</strong>: Select "NSX-T"</li> <li><strong>Name</strong>: Network Name</li> <li><strong>ACTIVE</strong>: Select to activate the network. Clear to deactivate the network</li> <li><strong>Gateway</strong>: (Optional) Enter the gateway address</li> <li><strong>Primary DNS</strong>: (Optional) Enter the primary DNS details</li> <li><strong>Secondary DNS</strong>: (Optional) Enter the secondary DNS details</li> <li><strong>Connected Gateway</strong>: (Optional) From the drop-down list, select a Tier1 gateway router</li> <li><strong>Gateway CIDR</strong>: Enter the Classless Inter-Domain Routing (CIDR) for the logical switch (example: 192.168.0.1/24)</li> <li><strong>Transport Zone</strong>: Select Overlay</li> <li><strong>Network Pool</strong>: Specify the IP Pool which was created in the prerequisites section</li> </ul> </li> </ul> <p><img src="/img/fig-3.jpg" alt=""></p> <ul> <li>Click "Save Changes"</li> </ul> <p>On successful creation, the network will list under the "Networks" tab. Use this segment for instance deployment.</p> <p><img src="/img/fig-10.jpg" alt=""></p> <h2>How to create a virtual network with DHCP</h2> <p>Locate the HPE GreenLake for private cloud card in the HPE GreenLake Central dashboard and click the Launcher icon to open the HPE GreenLake for private cloud dashboard.</p> <ul> <li> <p>Navigate to Infrastructure > Networks</p> </li> <li> <p>From the Networks tab, click the ADD drop-down list, select NSX-T Segment</p> </li> <li> <p>From the dialog box, configure the NSX-T segment parameters as follows. For information about additional fields that are not described here, refer <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=GUID-3DCFD624-DFE7-45A8-AFAC-BE004227C7EC.html">User Guide</a>.</p> <ul> <li><strong>Group</strong>: From the drop-down list, select an infrastructure user group to isolate the network at the group level. The default is Shared (all infrastructure groups)</li> <li><strong>Network Service</strong>: Select "NSX-T"</li> <li><strong>Name</strong>: Network Name</li> <li><strong>ACTIVE</strong>: Select to activate the network. Clear to deactivate the network</li> <li><strong>Gateway</strong>: (Optional) Enter the gateway address</li> <li><strong>Primary DNS</strong>: (Optional) Enter the primary DNS details</li> <li><strong>Secondary DNS</strong>: (Optional) Enter the secondary DNS details</li> <li><strong>Connected Gateway</strong>: (Optional) From the drop-down list, select a Tier1 gateway router</li> <li><strong>Gateway CIDR</strong>: Enter the Classless Inter-Domain Routing (CIDR) for the logical switch (example: 192.168.0.1/24)</li> <li><strong>Transport Zone</strong>: Select Overlay</li> </ul> </li> <li> <p>Expand 'Subnet DHCP' Section and update the below fields:</p> <ul> <li>DHCP Type: Local DHCP Server (default)</li> <li>DHCP ENABLED: Select to Enable</li> <li>DHCP Server Address: This address must not overlap the IP-ranges of the subnet, the gateway address of the subnet, or the DHCP static-binding addresses of this segment</li> <li>DHCP Ranges: Enter the DHCP ranges as comma-separated values. Entries can be in either range format (192.168.1.10-192.168.1.100) or CIDR format (192.168.10/24).</li> <li>DHCP LEASE TIME: (Optional) Enter the lease time. The default is one day.</li> </ul> </li> </ul> <p><img src="/img/fig-4.jpg" alt=""></p> <ul> <li>Click "Save Changes"</li> </ul> <p>On successful creation, the network will list under the "Networks" tab. Notice the tick mark in DHCP Column. Use this segment for instance deployment.</p> <p><img src="/img/fig-5.jpg" alt=""></p> <h2>Manage the virtual networks</h2> <p>You can manage the virtual networks from the Infrastructure > Networks page. Below is the network details page of the sample network (Green-Segment) created in the previous step.</p> <p><img src="/img/fig-6.jpg" alt=""></p> <p>Select the "Instances" tab to view the list of instances deployed using this network:</p> <p><img src="/img/fig-7.jpg" alt=""></p> <p>Select the "Host Records" tab to view the records created for every deployment on the network.</p> <p><strong>Grid View:</strong></p> <p><img src="/img/fig-8.jpg" alt=""></p> <p><strong>List View:</strong></p> <p><img src="/img/fig-9.jpg" alt=""></p> <h2>Summary</h2> <p>In this blog post, we covered how to get started with software-defined networking in HPE GreenLake for private cloud and explained the steps to create a sample virtual network with both static IP pool and DHCP. In the next article, we will cover the NSX distributed firewall feature of HPE GreenLake for private cloud and explain how to create and enforce firewall rules to restrict the network traffic to virtual machines.</p> <p>Learn more about <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=HPE-GreenLake-private-cloud-networking.html">HPE GreenLake for private cloud</a> networking</p>https://developer.hpe.com/2022-June-04/https://developer.hpe.com/2022-June-04/Fri, 03 Jun 2022 13:00:00 GMThttps://developer.hpe.com/why-open-source-is-more-than-software-the-example-of-the-linux-foundations-agstack-project/https://developer.hpe.com/why-open-source-is-more-than-software-the-example-of-the-linux-foundations-agstack-project/Wed, 01 Jun 2022 17:18:20 GMT<p>The Linux Foundation’s goal to create the greatest shared technology investment in history by enabling global open collaboration across companies, developers and users has profoundly impacted our digital heritage. Its tactics proved that using a shared development model fosters human collaboration, accelerates innovation, and notably expands adoption, with text book examples including Linux, Kubernetes, Cloud Foundry, and Jenkins. Propelled by this success, the set of open source projects under the Linux Foundation has expanded to over 400 projects.</p> <p>Now, a new Linux Foundation project, AgStack, aims to expand the definition of open source from discrete software code to also applying to a comprehensive set of open services aimed at solving one of the world’s greatest challenges – hunger – by getting data to those who need it from those who have it at the time it’s needed. AgStack ties open software and openly available data to build an open highway of information that’s collaborative, and at the same time, secure. Although AgStack is specifically aimed at agriculture, this new concept of open services can be applied to multiple industries, such as healthcare, housing, and education. It merely requires one to be open to working collaboratively.</p> <h2>Leveraging open source to improve agriculture efficiency</h2> <p>Global agriculture is a highly-complex, interrelated trillion dollar industry. Its ecosystem is massively inefficient and confronts grave challenges in the face of climate change. It employs half the world’s population and uses 70% of the world’s fresh water.  Of the thousands of stakeholders who have to work together, its small holder farmers produce at least a third <a href="https://ourworldindata.org/smallholder-food-production#:~:text=Family%20farms%20do%20produce%20around,poorest%20people%20in%20the%20world.">(from Our World Data)</a> of the entire world’s food supply and represent some of the poorest people in the world. With 30% of our food wasted and 30% of the world’s people hungry, there’s quite a bit of room for improvement.</p> <p>The AgStack project seeks to increase global agriculture efficiency through the creation, maintenance and enhancement of a free, re-usable, open and specialized digital infrastructure for data and applications. This software infrastructure is community-maintained and provides a free and secure digital “operating system” for all agriculture stakeholders to run their agriculture applications – from farmers to marketers to extension agents. AgStack essentially sits between digital agriculture applications and third-party cloud systems, and uses an openly available API to deliver the required data, models, frameworks, extensions and toolboxes to help farmers be more successful in growing crops in a sustainable manner.</p> <h2>Connecting the right data to the right people at the right time</h2> <p>Farmers are constantly grappling with weather, procurement, supply-chain, and disease-related issues.  With better data, they can make better decisions on what to plant, when to fertilize, how to irrigate, when to harvest, etc. Knowing what their particular soil requires at specific times and how to avoid pests and diseases can make or break a harvest. Even knowing market demands can help them better plan and avoid waste.</p> <p>AgStack addresses these issues by focusing on being able to deliver four pieces that interconnect:</p> <ol> <li><strong>Data.</strong> There is already a tremendous amount of data openly available today; weather data, satellite data, soil data.  Research about new varieties of seeds and fertilizer can significantly boost productivity. Near real-time information about soil types, weather conditions, market predictions, and nearby insect infestation can help farmers make better decisions on the ground. But hunting for that data requires time, expertise, and language translation in order to be useful.</li> <li><strong>Models.</strong> Many universities and research groups have developed models to predict how pests evolve, how different soils evolve, when to fertilize, etc. Again, much of this material is publically available and features permissive licensing, but you have to go off looking for it.</li> <li><strong>Frameworks.</strong> Things like user registries and asset registries are other things that can be shared that are hosted by AgStack. Some of these help identify field boundaries in such a way that protects the owners’ personal information. Others can look at them to determine if the field they’re working on has similar attributes and requires similar considerations.</li> <li><strong>Extensions and toolboxes.</strong> Extensions come from folks who share new research that can help farmers improve their practices. For example, tagged image datasets or tagged audio datasets can be extremely useful for those developing applications using machine learning algorithms. These datasets can help one determine if something is a piece of lettuce, an almond, or something else.</li> </ol> <p>It’s all about connecting the right data to the right people at the right time so the best decisions can be made.</p> <h2>Where does HPE fit in all this?</h2> <p>Being able to deliver all this from edge-to-cloud requires intensive data computation. Hewlett Packard is supporting this effort in a couple of ways. As a founding company, HPE has:</p> <ul> <li>Donated server hardware, memory, and cloud space</li> <li>Sponsored the project through HPE’s Tech for Good program, demonstrating that technology has the potential to drive real and positive change when harnessed effectively</li> <li>Openly collaborated, offering technical guidance on how to more easily and scalably provide for data motion and monitoring</li> </ul> <p>Janice Zdankus, HPE Vice President of CTO Technology Strategy and Innovation for Social Impact, whose team is involved with the AgStack project, as well as supporting projects like CGIAR and <a href="https://bigdata.cgiar.org/digital-intervention/video-enabled-extension/">DigitalGreen</a>, points out that “bringing together industry, technology, academia and government partners to solve key societal challenges, global impact can be delivered and scaled.” Her colleague, Ted Dunning, HPE CTO for Data Fabric, a member of her team and an elected member of the Board for AgStack, agrees, adding that “a modern world should make it possible for us to be generous.”</p> <h2>A road to a better tomorrow – open to all</h2> <p>AgStack Founder, Sumer Johal, joined the HPE Developer team just recently to offer his take on what makes AgStack so special. “It’s not just open source software; it’s also open services. It’s like this global road that anyone can ride on. Something that’s neutral and trusted. Something everyone can use. Something everyone will want to adopt.”</p> <p>How successful will it be? Sumer had a thought on that as well. “If you measure success by adoption and not by money, then you start looking at the world and seeing that it has large gaps in adoption that happen because everyone has a different view on how they want to make money. But it turns out, just like Linux demonstrated, that once you get away from money and look more at adoption as the currency, ironically you wind up making more money for stakeholders because more people adopt it.” AgStack has a great opportunity here. The world needs it to be successful. And it needs developers and data scientists to help it be successful by contributing to this open source initiative.</p> <p>For more information on the AgStack project, refer to the <a href="https://agstack.org/">AgStack project website</a> or read this <a href="https://www.linuxfoundation.org/press-release/linux-foundation-launches-open-source-digital-infrastructure-project-for-agriculture-enables-global-collaboration-among-industry-government-and-academia/">announcement</a> from the Linux Foundation. You can also check out this HPE Developer May 2022 Munch &#x26; Learn <a href="https://www.youtube.com/watch?v=dnhjRF5dr6M">video replay</a>.</p>https://developer.hpe.com/don’t-miss-all-things-software-in-the-hack-shack-at-hpe-discover/https://developer.hpe.com/don’t-miss-all-things-software-in-the-hack-shack-at-hpe-discover/Thu, 26 May 2022 19:22:00 GMT<p>Calling all developers, data scientists, data architects, and machine learning engineers! The HPE Developer Community is excited to welcome you to Las Vegas, June 28-30, 2022 at the Hack Shack at HPE Discover 2022, The Edge-to-Cloud-Conference. The Hack Shack is a place specifically focused on software – from development, to design, to use. There, you can connect with our experts in meetup sessions, try our unique software challenges, and go on a treasure hunt to discover a wealth of resources for open source and HPE technologies. It’s a fun place to collaborate, learn, expand your technology skills, and win prizes!</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <p><strong>Register <a href="https://attend.hpe.com/discover2022/index.cfm?iLangID=1">here</a> for HPE Discover 2022!</strong></p> </div> <p>We invite you to explore all our sessions on the <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;locale=en_US">content catalog</a> to build your own agenda. You can create your personalized agenda <a href="https://content.attend.hpe.com/go/agendabuilder.myagenda/?l=1049&#x26;locale=en_US">here</a>. To find our Hack Shack Meetup sessions in the catalogue, simply go to the search bar and type in Hack Shack. You’ll also find them listed below for easy reference. All sessions are listed in PDT.</p> <h2>Spark Your Curiosity at the Hack Shack Meetups!</h2> <p>Our Hack Shack Meetup sessions are designed to help you get your questions answered. These 30-minute discussions are led by technology experts who’ll introduce the topic, point out important features, and then engage in conversation with you as a group. You’ll also have an opportunity to connect with them after each session to go into further detail about your specific situation.</p> <h2>Discover the Developer’s Journey</h2> <p>In the Hack Shack (DEMO701), you’ll be introduced to HPE Developer, a program designed to help people like you build applications and develop integrations with HPE products and solutions. You’ll have the opportunity to connect with experts in secure connectivity, hybrid cloud, AI, and unified data analytics -- and learn more about the latest trends. You’ll also hear about unique software products that help you extract the most value from your data wherever it lives, turning insights into outcomes at the speed required to thrive in today’s complex world.</p> <p>To help you navigate the event, we’ve put together a list of recommended Hack Shack Meetup sessions to include in your agenda builder. Note: Live sessions, demos and hands-on labs in this article are listed in Pacific Daylight Time (PDT).</p> <h4>Our 13 Hack Shack Sessions</h4> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=25292_0&#x26;locale=en_US">HSM5292</a> | Hack Shack Meetup <strong>Tuesday, June 28, 12:30 p.m.</strong></p> <p><strong>Speaker:</strong> Matthew Morris (HPE) <strong>Does Your Data Lake Need a Little Chlorine? Clean Up and Modernize Your Apps</strong></p> <p>Have things become a little murky with your Hadoop solution? Join this meetup to hear how HPE Ezmeral Runtime Enterprise can modernize your applications and enable cloud-scale elasticity while tightly coupling the data at the edge with your applications running on-premises and in the cloud. You can even take advantage of cloud economics as you have it fully supported from beginning to end with HPE Professional Services and deployed as a service. Much cleaner, right?</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=25101_0&#x26;locale=en_US">HSM5101 </a>| Hack Shack Meetup <strong>Tuesday, June 28, 1:30 p.m.</strong></p> <p><strong>Speaker:</strong> Milind Bhandarkar (HPE) <strong>Interactive Analytics Workloads with PrestoDB</strong></p> <p>PrestoDB is an open source distributed SQL query engine for running interactive analytics queries against data sources of all sizes, ranging from gigabytes to petabytes. Join this interactive meetup to discuss how PrestoDB modernizes data at scale and how HPE enhancements increase scale, performance, security, and governance of PrestoDB. Learn how we integrate with a shared metadata catalog and a role-based access control system to provide a consistent view of disparate data sources.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24988_0&#x26;locale=en_US">HSM4988 </a>| Hack Shack Meetup <strong>Tuesday, June 28, 2:30 p.m.</strong></p> <p><strong>Speakers:</strong> Ted Dunning (HPE), Paul Holland (HPE) <strong>Why Open Source? Discover the Value of Open Source Project Initiatives</strong></p> <p>Wondering why open source? Let's face it, without open source, many of the products we take for granted today would never have been developed. That's because open source development encourages collaboration. Come by and let's talk about the hows and whys of open source and why HPE thinks open source is so important.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24989_0&#x26;locale=en_US">HSM4989</a>| Hack Shack Meetup <strong>Tuesday, June 28, 3:30 p.m., Wednesday, June 29, 11:00 a.m.</strong></p> <p><strong>Speaker:</strong> Max Lambrecht (HPE) <strong>SPIRE Integration with Istio for Edge-to-Cloud Security Beyond Kubernetes Microservices</strong></p> <p>Current service mesh solutions do not support leveraging service identity beyond east-west traffic to support advanced attestation capabilities for end-to-end service encryption, granular policy management, and traffic flows across distributed environments. In this talk, we'll discuss how SPIFFE and SPIRE integration with Istio complements Istio's authentication and attestation capabilities to provide more robust and flexible attestation beyond Kubernetes namespaces and service accounts.</p> <p><strong>Session ID</strong>: <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24990_0&#x26;locale=en_US">HSM4990</a> | Hack Shack Meetup <strong>Tuesday, June 28, 4:30 p.m.</strong></p> <p><strong>Speaker:</strong> Evan Sparks (HPE) <strong>Scaling Language Training to Trillion-Parameter Models on a GPU Cluster</strong></p> <p>Today, natural language processing (NLP) powers the latest conversational AI and translation apps. Join us to explore the collaborative experimentation process that machine learning teams leverage and the challenges they face while training these large-scale NLP models. See how Determined's open source deep learning training platform helps model developers train models faster and easier using tools such as resource management, fault tolerance, and model optimization.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24991_0&#x26;locale=en_US">HSM4991</a> | Hack Shack Meetup <strong>Tuesday, June 28, 5:30 p.m.</strong></p> <p><strong>Speaker:</strong> Michael Mattsson (HPE) <strong>Get Started with Persistent Storage for Kubernetes with the HPE CSI Driver</strong></p> <p>Come learn about an open source Container Storage Interface (CSI) for Kubernetes that enables the use of multiple storage backends through powerful abstractions. In this talk, we'll discuss features and capabilities available via the HPE CSI Driver and where developers can find resources to support any block storage platform with a management API.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24993_0&#x26;locale=en_US">HSM4993</a> | Hack Shack Meetup <strong>Wednesday, June 29, 9:00 a.m.</strong></p> <p><strong>Speaker:</strong> Anthony Dutra (HPE) <strong>Data Protection As Code: Introducing Zerto for Kubernetes</strong></p> <p>Zerto is a storage software solution designed for enterprise-class business continuity and disaster recovery in virtual and cloud environments. Using data protection as code, DevOps and I&#x26;O teams come together to protect apps from day one of their lifecycle. Join this meetup to discuss how Zerto's cloud-native API-first solution enables storage-agnostic backup and disaster recovery, protecting all of the application resources.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24994_0&#x26;locale=en_US">HSM4994</a> | Hack Shack Meetup <strong>Wednesday, June 29, 10:00 a.m.</strong></p> <p><strong>Speaker:</strong> Eric Soderberg (HPE) <strong>Discuss How Grommet, an HPE Open Source Project, Helps Streamline the Way You Develop Apps</strong></p> <p>Have you heard about Grommet, a powerful react component library that provides accessibility, modularity, responsiveness, and theming in a tidy package? This open source project empowers designers and developers to quickly build a responsive and accessible UX experience for the web. Come and join this discussion to hear how HPE is using Grommet in its own products and services, and how the open source nature of Grommet facilitates this.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24995_0&#x26;locale=en_US">HSM4995</a> | Hack Shack Meetup <strong>Wednesday, June 29, 12:00 p.m., Thursday, June 30, 11:00 a.m.</strong></p> <p><strong>Speaker:</strong> Glyn Bowden (HPE) <strong>Establish a Trustworthy Data Pipeline with Project Data Map</strong></p> <p>Data consumers need trusted data to make their best decisions. Data producers require tools to share data safely with those who need it. Learn how HPE is working with the open source community to leverage existing tools and resources that extract metadata with the Common Metadata Framework and Project Data Map. Help us bring disparate tools and resources together, creating a system the whole community can contribute to for the best insights from quality data, no matter where the data lives.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24996_0&#x26;locale=en_US">HSM4996</a> | Hack Shack Meetup <strong>Wednesday, June 29, 1:00 p.m.</strong></p> <p><strong>Speaker:</strong> Ted Dunning (HPE) <strong>10 Cool Open Source Projects and How You Can Contribute</strong></p> <p>Want to make your mark in the Open Source community? HPE has a long history of open collaboration and giving back. Join us in a Hack Shack discussion to learn how you can contribute to open source projects that are important to HPE, like Apache Kafka and Spark, Presto, Redfish, Grommet, AgStack, KubeDirector, and SPIFFE.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=24997_0&#x26;locale=en_US">HSM4997</a>| Hack Shack Meetup <strong>Wednesday, June 29, 2:00 p.m.</strong></p> <p><strong>Speaker:</strong> Didier Lalli (HPE) <strong>Calling all Developers, Data Scientists, Data Architects, and Machine Learning Engineers</strong></p> <p>Are you looking for the resources you need to design and build the best possible software and architecture experiences that harness the most value from your data? Join this interactive meetup and learn how to connect with the HPE Developer Community to build, communicate, and collaborate. We're all developing something. Let's do it together.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=25100_0&#x26;locale=en_US">HSM5100</a> | Hack Shack Meetup <strong>Wednesday, June 29, 3:00 p.m., Thursday, June 30, 12:00 p.m.</strong></p> <p><strong>Speaker:</strong> Navaneethan Venugopalan (HPE) <strong>Preview the New HPE GreenLake Developer Portal</strong></p> <p>Many ITOps teams are eager to see HPE expose application programmable interfaces (API) and SDKs to provide programmatic interfaces to HPE GreenLake to power their as-a-service transformation. In this session, explore the new HPE GreenLake Developer Portal, a self-service portal for developers to access published APIs of various HPE GreenLake edge-to-cloud services like Compute, Networking, and Storage.</p> <p><strong>Session ID:</strong> <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;sid=25099_0&#x26;locale=en_US">HSM5099 </a>| Hack Shack Meetup <strong>Wednesday, June 29, 4:00 p.m.</strong></p> <p><strong>Speaker:</strong> Don Wake (HPE) <strong>Accelerate Analytics with HPE Ezmeral and Apache Spark with Delta Lake</strong></p> <p>Interested in accessing data lakes directly using a lakehouse? Do you need reliable and consistent data for real-time and batch analytics with ACID transaction support? Join this interactive meetup to discuss and learn how to accelerate your analytics with HPE Ezmeral and Apache Spark with Delta Lake.</p> <h2>Work Hard, Play Hard</h2> <p>In addition to our classroom-style Hack Shack Meetup sessions, we invite you to come, relax, and participate in our fun and rewarding community activities. These include our Treasure Hunt, physical games, software challenges, and our Hack Shack Celebration Party (complete with beer and pizza!). You’ll even have the opportunity to win prizes – from HPE Developer hats and t-shirts to a hoodie or a CanaKit Raspberry Pi 4 Extreme Kit. Details below! (Please look here for <a href="https://developer.hpe.com/hackshack/hpediscover2022-participations-terms-conditions/">participation details</a>.)</p> <h4>HPE Developer Treasure Hunt</h4> <p>Want to win a new HPE Developer t-shirt or even possibly a hoodie? Participate in our scavenger hunt style Treasure Hunt! To answer the questions correctly, you’ll need to explore the physical Hack Shack in Las Vegas and scrutinize the HPE Developer portal. Those who complete the Treasure Hunt will have the opportunity to receive a t-shirt. Twelve lucky winners who answer all the questions correctly and attend our Wednesday evening Hack Shack Celebration Party will also win an HPE Developer hoodie. Look for the QR code in the booth to easily participate on your phone or use one of the provided laptops in our booth. (For complete details, please refer to the <a href="https://developer.hpe.com/hackshack/hpediscover2022-treasurehunt-terms-conditions/">Terms &#x26; Conditions</a>.)</p> <h4>Software Challenges</h4> <p>In an earlier blog post, <em><strong><a href="https://developer.hpe.com/blog/it%E2%80%99s-all-fun-and-games-at-the-hack-shack/">It’s all Fun and Games at the Hack Shack</a></strong></em>, we went into detail about our Hack Shack challenges. These role-based software challenges, which you’ll find listed in the HPE Discover 2022 catalog as session HSC5227, are available daily at the Hack Shack. Those who participate in these challenges and enter in the provided token key from June 28-30th can receive an HPE branded t-shirt. Take any of our software challenges on Tuesday, June 28th or Wednesday, June 29th and be entered for one of 6 grand prizes. One grand prize winner will be selected for each challenge. Participants must be present at the Wednesday evening 5:00-6:00pm Hack Shack Celebration to win a grand prize. (For complete details, please refer to the <a href="https://developer.hpe.com/hackshack/hpediscover2022-swchallenges-terms-conditions/">Terms &#x26; Conditions</a>.)</p> <h2>Join the Party!</h2> <p>Wednesday evening from 5:00-6:00pm, you’re invited to join us in the Hack Shack for our Celebration Party. Attendees will be treated to food (pizza, yay!) and refreshments (including beer) and get the opportunity to go home with some additional swag. Sit down, relax, and listen to our master of ceremonies, Robert Christiansen, VP of Strategy in the CTO Office at Hewlett Packard Enterprise.</p> <p>You’ll also meet our hardworking HPE Developer Community members who deliver the program, as well as our special guests, Kirk Bresniker and Dr. Eng Lim Goh. Kirk Bresniker, HPE VP and HPE Labs Chief Architect, will hand out the grand prizes for our software challenges. Dr. Eng Lim Goh, Senior VP and Chief Technologist for AI, will announce the prizes for our Treasure Hunt winners.</p> <h2>Don’t Miss Out</h2> <p>Build your agenda today and take advantage of the many things the Hack Shack offers to developers, designers, data scientists, data/ML engineers, open source advocates, and IT technologists.</p> <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css"> <div class="w3-container w3-center w3-margin-bottom"> <p><strong><a href="https://attend.hpe.com/discover2022/index.cfm?iLangID=1">Register now</a></strong></p> </div>https://developer.hpe.com/create-a-general-purpose-kubeconfig-file-in-hpe-greenlake-for-containers/https://developer.hpe.com/create-a-general-purpose-kubeconfig-file-in-hpe-greenlake-for-containers/Fri, 20 May 2022 07:02:51 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Containers is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html">HPE GreenLake for Containers</a>, one of the HPE GreenLake Cloud services, is built on the CNCF certified <a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a> and deployed as an enterprise-grade container management service using open source Kubernetes. The HPE GreenLake for Containers service is accessed and managed through one portal, called HPE GreenLake Central. The HPE GreenLake Central dashboard allows you to open the Clusters screen to create Kubernetes clusters using cluster blueprints, view details about existing clusters, and launch the HPE Ezmeral Runtime Enterprise dashboard, where you can view the status of all Kubernetes services and resource utilization across all clusters. The HPE Ezmeral Runtime Enterprise dashboard also allows you to download the kubectl binary, together with the kubeconfig file, to access and deploy applications using the client script. In this blog post, I will describe issues that are associated with this specific kubeconfig file downloaded from the dashboard and show you how to get around them.</p> <p><img src="/img/hpe-ecp-dashboard.png" alt="" title="HPE ECP Dashboard"></p> <p>The kubeconfig file that can be downloaded from the HPE Ezmeral Runtime Enterprise dashboard can introduce a number of issues:</p> <ol> <li> <p>The kubeconfig file is tied to the user who logs in to HPE GreenLake for Containers. There are many use cases that use simple scripts, such as in Bash, that call out to the kubectl or a proper client library to access the Kubernetes cluster with the kubeconfig file outside the cluster. They are not tied to any particular user. Providing a kubeconfig file that’s tied to your user is not considered to be a clean design due to the fact that each user may have different privileges, and providing the kubeconfig file with ability to access to the cluster may violate the <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">Principle of Least Privilege</a>.</p> </li> <li> <p>Since launching from HPE GreenLake Central to the HPE Ezmeral Runtime Enterprise is configured through SAML SSO, a session token is fetched and added to the kubeconfig file each time you launch the HPE Ezmeral Runtime Enterprise dashboard. With HPE GreenLake for Containers, the session token is configured to expire after an hour. You will be unable to use the downloaded kubeconfig file to access the cluster after the token expires. You will have to relaunch the dashboard and once again download the kubeconfig file.</p> </li> <li> <p>The kubeconfig file is generated to include some specific commands to show version, authenticate, and refresh the HPE Ezmeral Runtime Enterprise environment. The standard kubectl tool installed from the <a href="https://kubernetes.io/docs/tasks/tools/">official Kubernetes site</a> does not work with this kubeconfig file. You have to download the <code>HPE kubectl plugin</code>, available from the same dashboard, and use it together with the kubectl and the kubeconfig file. This causes issues due to the fact that some services and tools, such as <code>Azure DevOps</code> and <code>Jenkins</code>, use the standard kubectl to access and deploy applications to the Kubernetes clusters.</p> </li> </ol> <p>You need a way to have a kubeconfig file that is not tied to a specific user and has the right set of permissions to access the Kubernetes cluster. It should work permanently with the standard kubectl tool. This blog post walks you through the process of creating such a general-purpose kubeconfig file that allows you to access and deploy applications to the Kubernetes cluster in HPE GreenLake for Containers. By using these instructions, you will be able to create a kubeconfig file that can be used by any external scripts specific to your CI/CD pipeline setup and have them work with the Kubernetes cluster.</p> <h2>Prerequisites</h2> <p>You need to download the kubectl binary, together with the HPE kubectl plugin and the kubeconfig file, from the launched HPE Ezmeral Runtime Enterprise dashboard. The downloaded kubectl binary and its plugin need to be set up in your environment. To simplify the setup process, you should export the environment variable <code>KUBECONFIG</code> and point it to the downloaded kubeconfig file. With these setups in place, you can access the Kubernetes cluster in the HPE GreenLake for Containers.</p> <p>With your user access setup, you should have access to permissions that can create and update the following resources in the Kubernetes cluster:</p> <ul> <li>Kubernetes Service Account(s)</li> <li>Kubernetes Roles &#x26; RoleBindings</li> </ul> <h2>Setup Details</h2> <h3>Create a Kubernetes Service Account</h3> <p>You can use the following yaml manifest file to create a service account in the Kubernetes cluster. Replace the name <code>cfe-demo-sa</code> with your service account name.</p> <pre><code class="language-json"># serviceaccount.yml --- apiVersion: v1 kind: ServiceAccount metadata: name: cfe-demo-sa </code></pre> <p>Run the following commands to create the service account and verify that the service account has been created:</p> <pre><code class="language-bash">$ kubectl apply -f serviceaccount.yaml serviceaccount/cfe-demo-sa created $ kubectl get serviceaccounts cfe-demo-sa NAME SECRETS AGE cfe-demo-sa 1 24s </code></pre> <h3>Create a Role</h3> <p>After you have the service account, you can create a role with a set of permissions that represent the access rights that you want for your scripts.</p> <p>Here is the yaml manifest file to create a role. Replace the <code>cfe-demo-role</code> with your role name.</p> <pre><code class="language-json"># role.yml --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cfe-demo-role rules: - apiGroups: - "" resources: - bindings - podtemplates - replicationcontrollers - pods - services - serviceaccounts - endpoints - persistentvolumeclaims - events - configmaps - secrets - pods/exec - pods/log - pods/portforward verbs: - '*' - apiGroups: - rbac.authorization.k8s.io resources: - roles - rolebindings verbs: - '*' - apiGroups: - apps resources: - controllerrevisions - statefulsets - deployments - replicasets - daemonsets verbs: - '*' - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - '*' - apiGroups: - batch resources: - cronjobs - jobs verbs: - '*' </code></pre> <p>The <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">Using RBAC Authorization</a> section of the Kubernetes documentation provides details on how to configure the Role resource. Please check carefully the permissions for the access rights that you want for your scripts in order to comply with the Principle of Least Privilege.</p> <p>Run the following commands to create the role and verify that the role has been created:</p> <pre><code class="language-bash">$ kubectl apply -f role.yaml role.rbac.authorization.k8s.io/cfe-demo-role created $ kubectl get role cfe-demo-role NAME CREATED AT cfe-demo-role 2022-05-19T20:51:57Z </code></pre> <h3>Grant Permissions to Service Account</h3> <p>You now create the RoleBinding to bind the role to the service account.</p> <p>Here is the manifest file to create the RoleBinding that binds the role <code>cfe-demo-role</code> to the service account <code>cfe-demo-sa</code>. Replace those names with the names in your environment.</p> <pre><code class="language-json"># rolebinding.yml --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cfe-demo-rb roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: cfe-demo-role # Should match name of Role subjects: - kind: ServiceAccount name: cfe-demo-sa # Should match service account name </code></pre> <p>Run the following commands to create the rolebinding and verify that the rolebinding has been created:</p> <pre><code class="language-bash">$ kubectl apply -f rolebinding.yaml rolebinding.rbac.authorization.k8s.io/cfe-demo-rb created $ kubectl get rolebindings cfe-demo-rb NAME ROLE AGE cfe-demo-rb Role/cfe-demo-role 19s </code></pre> <h3>Extract Service Account Token</h3> <p>You can now check the secret token in the created service account and extract the token field by running the following commands. The token is a randomized string. The setup shows only a snippet of this string.</p> <pre><code class="language-bash">$ kubectl describe serviceaccounts cfe-demo-sa Name: cfe-demo-sa Namespace: cfe-demo-cluster Labels: &#x3C;none> Annotations: Image pull secrets: &#x3C;none> Mountable secrets: cfe-demo-sa-token-2zlzf Tokens: cfe-demo-sa-token-2zlzf Events: &#x3C;none> $ kubectl describe secrets cfe-demo-sa-token-2zlzf Name: cfe-demo-sa-token-2zlzf Namespace: cfe-demo-cluster Labels: &#x3C;none> Annotations: kubernetes.io/service-account.name: cfe-demo-sa kubernetes.io/service-account.uid: a467f9bd-655d-413f-ad77-5156d03d2322 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1066 bytes namespace: 16 bytes token: iIsImtpZCI6IjA2YnhmSVZrVDRGWnBab0VOYXhnWFBTTE1WWmptUm40eER </code></pre> <p>Note that if you use <code>-o yaml</code> instead of <code>describe</code> in the commands, you get a base64-encoded version of the token. You must decode it before you use it.</p> <p>If you access the Kubernetes API directly, e.g., from <code>curl</code>, you can use the token as the bearer token for the authorization header.</p> <p>However, if you have your scripts running outside the cluster that uses kubectl or a client library to access the Kubernetes cluster, you need the kubeconfig file to load the configs from. Follow the instructions in the next section to create such a kubeconfig file.</p> <h3>Create a Kubeconfig File</h3> <p>Here is a shell script to create a kubeconfig file using the secret token of the service account. Update those variables in the script header to match with your environment.</p> <pre><code class="language-bash"># create-kubeconfig.sh # Update those variables to match your environment SERVICE_ACCOUNT_NAME="cfe-demo-sa" CONTEXT=$(kubectl config current-context) NEW_CONTEXT="cfe-demo-context" TOKEN_USER="cfe-token-user" KUBECONFIG_FILE="kubeconfig-sa" # Extract service account token SECRET_NAME=$(kubectl get serviceaccount ${SERVICE_ACCOUNT_NAME} --context ${CONTEXT} -o jsonpath='{.secrets[0].name}') TOKEN_DATA=$(kubectl get secret ${SECRET_NAME} --context ${CONTEXT} -o jsonpath='{.data.token}') TOKEN=$(echo ${TOKEN_DATA} | base64 -d) #Create a general-purpose kubeconfig file kubectl config view --raw > tmp.raw kubectl --kubeconfig tmp.raw config use-context ${CONTEXT} kubectl --kubeconfig tmp.raw config view --flatten --minify > tmp.min kubectl --kubeconfig tmp.min config rename-context ${CONTEXT} ${NEW_CONTEXT} kubectl --kubeconfig tmp.min config set-credentials ${TOKEN_USER} --token ${TOKEN} kubectl --kubeconfig tmp.min config set-context ${NEW_CONTEXT} --user ${ TOKEN_USER } kubectl --kubeconfig tmp.min config view --flatten --minify > ${KUBECONFIG_FILE} # Cleanup tmp rm tmp.raw rm tmp.min </code></pre> <p>After running the script, a general-purpose kubeconfig file <code>kubeconfig-sa</code> is created. By exporting the kubeconfig file as the environment variable <code>KUBECONFIG</code>, you can access the Kubernetes cluster and check all the resources deployed in the cluster.</p> <pre><code class="language-bash">$ bash create-kubeconfig.sh Switched to context "fab-zero-cfe-demo-cluster-cfe-demo-cluster-guoping.jia@hpe.com". Context "fab-zero-cfe-demo-cluster-cfe-demo-cluster-guoping.jia@hpe.com" renamed to "cfe-demo-context". User "cfe-token-user" set. Context "cfe-demo-context" modified. $ export KUBECONFIG=kubeconfig-sa $ kubectl get all No resources found in cfe-demo-cluster namespace. </code></pre> <h2>Conclusion</h2> <p>This blog post shows you how to create a general-purpose kubeconfig file using a service account. This kubeconfig file is not tied to any specific user. It is bound with a list of permissions carefully chosen for your need to access the Kubernetes cluster. The created kubeconfig file works permanently with both the downloaded kubectl binary from HPE GreenLake for Containers dashboard and the standard one installed from the official Kubernetes site. This allows you to use it in any Kubernetes client scripts, esp., in your Kubernetes <em>CI/CD</em> pipeline.</p> <p>It should be noted that the token in a service account does not expire and is valid as long as the service account exists. You should consider to generate a new kubeconfig file by creating a new service account and then revoke access to the old service account. As one best practice for security, you should do this as often as you can. This is extremely important when you use the kubeconfig file to access the Kubernetes cluster from outside the cluster. You can refer to the <a href="https://kubernetes.io/docs/concepts/security/rbac-good-practices/">Kubernetes RBAC Good Practices</a> to understand the risks using RBAC and the general good practices to reduce the risk.</p> <p>Please check the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=HPE-GreenLake-for-containers.html">HPE GreenLake Central User Guide</a> for more details about the HPE GreenLake for Containers.</p>https://developer.hpe.com/overview-of-the-platform-level-data-model-for-redfish®-device-enablement-standard/https://developer.hpe.com/overview-of-the-platform-level-data-model-for-redfish®-device-enablement-standard/Thu, 12 May 2022 16:04:30 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/pldm/pldm_rde/pldm_rde">Server Management Portal</a>.</big></p> <br>https://developer.hpe.com/how-to-set-up-credentials-to-authenticate-container-registry-in-hpe-greenlake-for-containers/https://developer.hpe.com/how-to-set-up-credentials-to-authenticate-container-registry-in-hpe-greenlake-for-containers/Tue, 10 May 2022 08:20:45 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Containers is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p><a href="https://www.hpe.com/us/en/greenlake/containers.html"><em>HPE GreenLake for Containers</em></a>, one of HPE GreenLake Cloud Services, is an HPE-designed, implemented, owned, and operated private cloud deployed at a customer site. It is built on <a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a>, featuring open source Kubernetes container management. Customers can benefit from the use of HPE GreenLake for Containers by having a powerful underlying container-based infrastructure for both cloud-native apps and monolithic apps, and enjoy as-a-Service ease-of-use and economics. When working with Docker images, one can run into some issues, as Docker's image download rate limit can cause unexpected errors. In this blog post, I will walk you through the process of setting up a secret using the credentials for your Docker subscription in the Kubernetes cluster to avoid this issue.</p> <p>With HPE GreenLake for Containers, it is configured to use a gateway host, acting as a proxy sever, that carries client requests from deployed application service endpoints in the Kubernetes clusters. The gateway host maps the private IP endpoints of services running inside the Kubernetes clusters to external accessible IP addresses and ports. It provides better security by exposing only the gateway host IP address to the public while keeping all the others behind the firewall. However, when you create your application from a Docker image in the Kubernetes cluster, your application pods may get stuck in the error state <em><code>ErrImagePull</code></em>. Below is a sample <code>ngnix</code> application deployment to the cluster and the received error message in the pod events:</p> <pre><code class="language-markdown">$ kubectl run cfe-nginx --image=nginx pod/cfe-nginx created $ kubectl get pods NAME READY STATUS RESTARTS AGE cfe-nginx 0/1 ErrImagePull 0 5s $ kubectl describe pods cfe-nginx Name: cfe-nginx ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled &#x3C;invalid> default-scheduler Successfully assigned cfe-demo-cluster/cfe-nginx to k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Warning Failed &#x3C;invalid> kubelet, k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:19da26bd6ef0468ac8ef5c03f01ce1569a4dbfb82d4d7b7ffbd7aed16ad3eb46: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Warning Failed &#x3C;invalid> kubelet, k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Error: ErrImagePull Normal BackOff &#x3C;invalid> kubelet, k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Back-off pulling image "nginx" Warning Failed &#x3C;invalid> kubelet, k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Error: ImagePullBackOff Normal Pulling &#x3C;invalid> (x2 over &#x3C;invalid>) kubelet, k8s-cfe-demo-cluster-worker-67f75-24jmj.glhc-hpe.local Pulling image "nginx" </code></pre> <p>The above issue is caused by <a href="https://docs.docker.com/docker-hub/download-rate-limit/">the Docker policy changes for downloading images</a>. In particular, for anonymous users, the image download rate limit is set to 100 pulls per 6 hours per IP address. No matter who starts creating a new application from a Docker image, the Kubernetes cluster downloads the image as an anonymous user, which counts toward the new rate limit on the same gateway host IP. Given that other developers are using the same Kubernetes cluster, together with many <em>ArgoCD</em> jobs configured to run in the backend, the limit can be eventually reached and the <em><code>ErrimagePull</code></em> message pops up.</p> <p>To deal with this situation, the following sections show you the details on how to set up a secret in the Kubernetes cluster using the credentials for your Docker subscription. The cluster then uses the secret to authenticate to your Docker account and pull the image as an authenticated user. The image download will count against individual limit of your Docker subscription instead of the 100 downloads shared across all anonymous cluster users in the cluster.</p> <h2>Prerequisites</h2> <p>You need to have the following credentials for your Docker subscription, either a personal or a paid one:</p> <ul> <li>Docker Username</li> <li>Docker Password or Access Token</li> </ul> <p>The personal Docker subscription credentials allow you to log in to Docker as an authenticated user in which the image rate limit is set to 200 pulls per 6 hour period. The users with a paid Docker subscription have no limits in image downloads. It could make sense to have a paid Docker subscription at team or company level. However, you do not really need to upgrade your Docker account to a paid one. 200 pulls per 6-hour period as an authenticated user should be enough for an individual developer to work on application deployment to the Kubernetes cluster.</p> <p>Note that the Docker access token, instead of a Docker password, would be recommended to be used as the credentials. You can refer to <a href="https://docs.docker.com/docker-hub/access-tokens/">Docker’s Manage Access Tokens</a> to create such an access token of your Docker subscription. A Docker access token provides some advantages over the password. It can't be used for performing any admin activity on your Docker account. It also provides a way to check the last usage and can easily be disabled or deleted. In case you have two-factor authentication setup on your account, using the access token is the only way to authenticate to Docker.</p> <h2>Setup Details</h2> <h3>Create a Registry Secret</h3> <p>You can use the following command to create a registry secret using your Docker credentials:</p> <pre><code class="language-markdown">$ kubectl create secret docker-registry cfe-registry-key --docker-server=https://index.docker.io/v1/ --docker-username=&#x3C;username> --docker-password=&#x3C;password> --docker-email=&#x3C;email> </code></pre> <p>Using <code>--docker-email</code> is optional. The <code>cfe-registry-key</code> is the sample secret name used in the setup process.</p> <p>You can verify that the registry secret has been created:</p> <pre><code class="language-markdown">$ kubectl get secrets cfe-registry-key NAME TYPE DATA AGE cfe-registry-key kubernetes.io/dockerconfigjson 1 2m11s </code></pre> <h3>Specify <code>imagePullSecrets</code> on Pods</h3> <p>You can create pods or update existing ones to refer to the created registry secret by adding the following <code>imagePullSecrets</code> section to the Pod definition:</p> <pre><code class="language-json">imagePullSecrets: - name: cfe-registry-key </code></pre> <p>With those pod manifest files, the Kubernetes cluster downloads the image as an authenticated user using the credentials from the registry secret. The image download will not count against the 100 download limit shared across all anonymous cluster users in the Kubernetes cluster.</p> <p>Although the setup works for most of your application deployment, it requires you to modify manifest files to add <code>imagePullSecrets</code> section.</p> <p>You can use the following step to add image pull secret to <code>service accounts</code>. When you try to create new applications, the <code>imagePullSecrets</code> will be automatically injected and used in downloading the images. This setup makes sense, especially when you have the paid Docker subscription. Other developers who use the same Kubernetes cluster can benefit from the setup having unlimited image download for their application deployment.</p> <h3>Add <code>imagePullSecrets</code> to Service Accounts</h3> <p>You can run the following command to modify the default service account for the namespace to use <code>imagePullSecrets</code>:</p> <pre><code class="language-markdown">$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "cfe-registry-key"}]}' serviceaccount/default patched </code></pre> <p>You can verify that the <code>imagePullSecrets</code> section has been added to the service account:</p> <pre><code class="language-markdown">$ kubectl get serviceaccount default -o yaml kind: ServiceAccount metadata: name: default ... ... imagePullSecrets: - name: cfe-registry-key </code></pre> <p>When you try to create a pod in the current namespace, you can verify the pod has its <code>sepc.imagePullSecrets</code> field set automatically:</p> <pre><code class="language-markdown">$ kubectl run cfe-nginx --image=nginx pod/cfe-nginx created $ kubectl get pods NAME READY STATUS RESTARTS AGE cfe-nginx 1/1 Running 0 4m28s $ kubectl get pod cfe-nginx -o=jsonpath='{.spec.imagePullSecrets[0].name}{"\n"}' cfe-registry-key </code></pre> <p>If you deploy an application to a different namespace than the current one, you need run the command <code>kubectl patch serviceaccount </code> with the additional option <code>-n &#x3C;namespace></code> to add <code>imagePullSecrets</code> to the default service account in the new namespace.</p> <p>Similarly, if you deploy an application to a namespace using a different service account than the default one, you need to replace <code>default</code> in the command <code>kubectl patch serviceaccount </code> with the service account name to add <code>imagePullSecrets</code> to this service account. By default, deploying an application to a namespace uses the default service account. You can define the <code>serviceAccountName</code> in your manifest files to specify the service account you want to use.</p> <h2>Conclusion</h2> <p>Docker's image download rate limit has caused quite a bit of confusion for those using HPE GreenLake for Containers. This blog post describes you how to set up a secret using the credentials for your Docker subscription to pull images for your application deployment. Once follow up the procedure, your application deployment will be able to download images without hitting the rate limit error any more.</p>https://developer.hpe.com/understanding-hpe-greenlake-identity-access-management/https://developer.hpe.com/understanding-hpe-greenlake-identity-access-management/Tue, 10 May 2022 07:35:34 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h1>Introduction</h1> <p>When configuring an HPE GreenLake Central tenant for diverse customer Identity and Access Management (IAM) needs, it is important to understand how the various features work, so you can take appropriate actions. Once you understand the features and how they work, you'll be able to arrange resources in an optimal way. It will also prevent the need to reconfigure anything in the future.</p> <p>With this in mind, I'd like to begin by describing the various features of Identity and Access Management available in HPE GreenLake Central and how to perform common tasks associated with IAM configuration. I'll also include some simple steps you can follow when designing IAM configurations based on customer requirements. Finally, I'll present several fictitious customer setups and show you how each is configured.</p> <h1>IAM Definitions</h1> <h2>Users</h2> <p>All HPE GreenLake Central users start with getting an HPE GreenLake Central account and profile. Users are invited by a tenant administrator, or another user with appropriate permissions, to join the HPE GreenLake Central tenant.  HPE can assist with user management and can configure your HPE GreenLake environment to use the Customer's Single Sign-On (SSO) mechanism through a custom Services engagement. <strong>NOTE: All users in a tenant have the same email domain name.</strong></p> <h2>API Clients</h2> <p>API clients are nonhuman HPE GreenLake Central users for programmatic access to HPE GreenLake Central. They are assigned spaces and roles, similar to a regular user. </p> <h2>Tenants</h2> <p>A tenant is an isolated environment with unique users and workloads.  Within a tenant, users are invited to join the tenant by a tenant administrator or another user with the appropriate permissions.</p> <h2>User Groups</h2> <p>User groups are a named set of users that share a common job function or access requirements. </p> <h2>Spaces</h2> <p>Spaces enable you to grant access to a defined subset of resources within a tenant. Resources can be a part of multiple spaces. Access to resources in a space is managed through the assignment of users, user groups or API clients to roles in a space. There is a default space where users land automatically upon login if they are assigned appropriate roles for that space. The all resources space is a dynamic list of all resources in a tenant. </p> <p>Users can be granted access to multiple spaces.</p> <h2>Roles</h2> <p>Roles are a named set of permissions used to access resources. They are assigned to users, user groups or API clients. Roles grant permissions for a specific set of resources in a space.</p> <p>Roles are available for the services that are available within the tenant. The following table is incomplete but lists the most common roles and definitions.</p> <table> <thead> <tr> <th><strong>Role</strong></th> <th><strong>Responsibility</strong></th> </tr> </thead> <tbody> <tr> <td><strong>HPE Consumption Analytics</strong></td> <td></td> </tr> <tr> <td>Consumption Analytics Viewer</td> <td>View cost information in HPE GreenLake Central (Customer only)</td> </tr> <tr> <td>Consumption Analytics Contributor</td> <td>View cost information in HPE GreenLake Central and access HPE Consumption Analytics Platform (Customer only)</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake Capacity Planning</strong></td> <td></td> </tr> <tr> <td>Capacity Planning Viewer</td> <td>Read capacity planning information</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake Billing</strong></td> <td></td> </tr> <tr> <td>Billing Contributor</td> <td>View billing information and edit downstream rates</td> </tr> <tr> <td>Billing Viewer</td> <td>View billing information</td> </tr> <tr> <td>Billing Usage Viewer</td> <td>View monthly charges card and report (usage information only - no cost information)</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake for Private Cloud Enterprise: Virtual Machines</strong></td> <td></td> </tr> <tr> <td>Private Cloud Tenant Owner</td> <td>Administer HPE GreenLake for Private Cloud Enterprise: Virtual Machines dashboard<br>Manage scheduling and activity<br>Manage infrastructure<br>Manage provisioning</td> </tr> <tr> <td>Private Cloud Tenant Contributor</td> <td>Manage self-service VMs and app provisions</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake for Private Cloud Enterprise: Containers on Virtual Machines</strong></td> <td></td> </tr> <tr> <td>Container Platform Cluster Owner</td> <td>Manage predefined and custom cluster blueprints, along with machine blueprints</td> </tr> <tr> <td>Container Platform Cluster Resource Contributor</td> <td>Manage resources within a cluster namespace</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake for ML Ops</strong></td> <td></td> </tr> <tr> <td>MLOps Admin and IAM Owner</td> <td>Install services and manage projects, resources, and users, and view sites information</td> </tr> <tr> <td>MLOps Project Admin</td> <td>Build, train, and deploy models, launch the AI/ML services<br>Corresponds to Project Administrator role in HPE Ezmeral Runtime Enterprise (previously known as HPE Ezmeral Container Platform)</td> </tr> <tr> <td>MLOps Project Member</td> <td>To build, train, and deploy models, launch the AI/ML projects and consume the AI/ML services<br>Corresponds to Project Member role in HPE Ezmeral Runtime Enterprise (previously known as HPE Ezmeral Container Platform)</td> </tr> <tr> <td></td> <td></td> </tr> <tr> <td><strong>HPE GreenLake for HPC</strong></td> <td></td> </tr> <tr> <td>HPCaaS Admin</td> <td>Perform all actions</td> </tr> <tr> <td>HPCaaS Job Contributor</td> <td>Manage job and job-related information</td> </tr> <tr> <td>HPCaaS Viewer</td> <td>View only</td> </tr> </tbody> </table> <p>For more detail on roles, please refer to the HPE GreenLake Central User Guide which can be found here: <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=index.html">HPE GreenLake Central User Guide</a></p> <ul> <li>Owner roles apply to different aspects of Services. Some administrator Roles are only available to HPE Information Technology Operations Center (ITOC) and DevOps teams.</li> <li>Contributor roles also apply to services. These roles allow non-admin operations within a service.</li> <li>Viewer roles are 'read-only' roles which can be used to allow users to access services but not to modify them in any way.</li> <li>Custom roles can also be defined. These allow arbitrary collections of permissions to be combined into one or more roles and can be assigned as normal.</li> </ul> <h1>Tenant On-Boarding</h1> <p>New tenants are requested via an HPE representative. The request includes an initial contact email address. This new user account is associated with the new tenant. Once the tenant is created and the billing account is promoted into production, the contact email address receives an email invitation to activate their account. The new user can then log into HPE GreenLake Central and can start the process of setting up their spaces, user groups, role assignments, etc. They can also invite other users to join the tenant and perform other activities relevant to the specific user permissions.</p> <h2>Inviting Users</h2> <p>Once a tenant administrator is on-boarded, they can invite their users to join the tenant.</p> <ol> <li>Access the User Management Service within the <strong>wrench</strong> icon.</li> <li>Select the <strong>Users</strong> Tab</li> <li>Under the <strong>Actions</strong> pull-down, choose Invite User</li> </ol> <img src="/img/invite-user.png" width="480" height="538" alt="Invite User"> <p>The invited user receives an email inviting them to join the tenant. Once the user activates their account they can log into HPE GreenLake Central and switch to the tenant. The tenant administrator can add the new User to various User Groups, etc. The user will only receive an invitation email if they have not already been invited to join a tenant.</p> <h2>Creating and modifying User Groups</h2> <ol> <li> <p>Access the User Management Service within the <strong>wrench</strong> icon.</p> </li> <li> <p>Select the <strong>User Groups</strong> tab</p> </li> <li> <p>To modify an existing user group, click on it from the list of user groups</p> <ul> <li>Members can be added by selecting the <strong>Members</strong> tab and selecting <em>Add Members</em> under the <strong>Actions</strong> pull-down</li> <li>Members can be removed from the user group by clicking on the <strong>Trash</strong> icon beside the member name</li> </ul> </li> <li> <p>To create a new user group, from the <strong>User Groups</strong> tab, click on the <em>Create User Group</em> button</p> </li> <li> <p>Enter a user group name and description</p> </li> <li> <p>Once the user group has been created it may be modified using the instructions above</p> </li> </ol> <h2>Creating and Modifying Spaces</h2> <ol> <li> <p>Access the Management Service within the <strong>wrench</strong> icon.</p> </li> <li> <p>Select the <strong>Spaces</strong> tab</p> </li> <li> <p>To modify an existing space, click on it from the list of spaces</p> <ol> <li> <p>Users and user groups can be added by selecting the <strong>Assignments</strong> tab and selecting <em>Create Assignment</em> under the <strong>Actions</strong> pull-down</p> </li> <li> <p>Select appropriate role(s) for the space</p> <ul> <li>Users and user groups can be removed from the space by clicking on the <strong>Trash</strong> icon beside the subject name</li> <li>Resources can be added and removed from the space by selecting the <strong>Resources</strong> tab and selecting <em>Update Resources</em> under the <strong>Actions</strong> pull-down</li> </ul> </li> </ol> </li> <li> <p>To create a new space, click on the <strong>Create Space</strong> button</p> <ul> <li>Enter a space name and parent space</li> <li>Select <strong>Resources</strong> by expanding the <strong>All Resources</strong> list</li> </ul> </li> </ol> <h1>Let's Explore Some Fictitious Customer Examples</h1> <h2>Example 1: ACME Corp.</h2> <h3>Design</h3> <p><img src="/img/iam-document-example-1-tenant-1-1a.jpg" alt="" title="Example 1: ACME Corp."></p> <p>Customer ACME Corp. has users spread around the globe. The users' email addresses all take the form of <a href="mailto:xyz@acmecorp.com.">xyz@acmecorp.com.</a> ACME Corp. has several departments that are also distributed around the globe. ACME Corp. wishes to use HPE GreenLake for Private Cloud Enterprise to create and manage virtual machines and containers on behalf of the various departments. They also wish to use HPE GreenLake for ML Ops to examine their internal data and perform AI operations upon it and use HPE GreenLake for containers to run a sales application.</p> <p>The ACME Corp. departments include:</p> <ul> <li>Office of CEO</li> <li>IT department</li> <li>Sales department</li> <li>R&#x26;D department</li> <li>Data Science department</li> </ul> <p>Since all users in ACME Corp. have the same email address format, and since all departments wish to be able to use all resources, only one tenant is required. ACME Corp wishes to be billed centrally for all services so a single billing account is sufficient.</p> <p>Once this is done, they can create user groups for each department, e.g.</p> <ul> <li>Office of CEO users</li> <li>IT department users</li> <li>Sales department users</li> <li>R&#x26;D department users</li> <li>Data Science department users</li> </ul> <p>Each user group will consist of sets of users from each department:</p> <p><img src="/img/sales-department-users-members.png" alt="" title="Sales Department Users"></p> <p>Next, the tenant administrator can create spaces for each department, e.g.</p> <ul> <li>Office of CEO space</li> <li>IT department space</li> <li>Sales department space</li> <li>R&#x26;D department space</li> <li>Data Science department space</li> </ul> <p>When each space is created, appropriate resources are chosen and mapped to the space:</p> <img src="/img/resource-selection.png" width="600" height="359" alt="Resource Selection"> <h4>Office of CEO</h4> <p>The Office of CEO space needs access to billing across the entire company. They may also use HPE GreenLake for Private Cloud Enterprise to run some virtual machines. Finally, they occasionally run some AI on internal data.</p> <p><img src="/img/office-of-ceo-space-resources.png" alt=""></p> <p>The user group is assigned appropriate roles for the selected services:</p> <p><img src="/img/office-of-ceo-space-assignments.png" alt="" title="Office of CEO Space"></p> <p>The space looks like this to the users:</p> <p><img src="/img/office-of-ceo-space-view.png" alt="" title="Office of CEO Space View"></p> <h4>IT Department</h4> <p>The IT department space would need access to HPE GreenLake for Private Cloud Enterprise: Virtual Machines, HPE GreenLake for ML Ops and HPE GreenLake for Private Cloud Enterprise: Containers resources. This would allow users in this department to manage the resources running in both services. Users in this department require admin roles for the selected resources.</p> <p><img src="/img/it-department-space-resources.png" alt="" title="IT Department Space Resources"></p> <p>The user group is assigned appropriate roles for the selected services:</p> <p><img src="/img/it-department-space-assignments.png" alt="" title="IT Department Space Assignments"></p> <p>The space looks like this to the users:</p> <p><img src="/img/it-department-space-view.png" alt="" title="IT Department Space View"></p> <h4>Sales Department</h4> <p>The sales department space runs a series of containers that provide a service to their field sales users. Since they maintain their own code they have two clusters: Integration and Production. They do not use other services.</p> <p><img src="/img/sales-department-space-resources.png" alt="" title="Sales Department Space Resources"></p> <p>The user group is assigned appropriate roles for the selected services:</p> <p><img src="/img/sales-department-space-assignments.png" alt="" title="Sales Department Space Assignments"></p> <p>The space looks like this to the users:</p> <p><img src="/img/sales-department-space-view.png" alt=""></p> <h4>R&#x26;D Department</h4> <p>The R&#x26;D department accesses a series of virtual machines in the HPE GreenLake for Private Cloud Enterprise: Virtual Machines service. Since the service is managed by the IT department, these users do not need administrative rights.</p> <h4>Data Science Users</h4> <p>These users access several HPE GreenLake for ML Ops projects. The service is managed by the IT department.</p> <h2>Example 2: ABC Corp.</h2> <h3>Design</h3> <p><img src="/img/iam-document-example-2-tenant-1-1-.jpg" alt="" title="Example 2: ABC Corp"></p> <p>ABC Corp. is a small company with three departments, A, B and C. Since the company is small, some employees work across departments. ABC Corp. uses multiple HPE GreenLake services, each with separate billing accounts. A single tenant is sufficient for the small number of employees of ABC Corp.</p> <p>The tenant administrator for ABC Corp can create user groups for each department, e.g.</p> <ul> <li> <p>Department A users</p> <ul> <li>Users: John, Mary</li> </ul> </li> <li> <p>Department B users</p> <ul> <li>Users: Mary</li> </ul> </li> <li> <p>Department C users</p> <ul> <li>Users: Robert</li> </ul> </li> </ul> <p>NOTE: User Mary is a member of both Department A and Department B user groups.</p> <p>Next, the tenant administrator can create spaces for each department and select billing account resources and user groups as appropriate, e.g.</p> <p>Department A space</p> <ul> <li>Billing resource HP-AMS-DMO-USA-99918</li> <li>Billing resource HP-AMS-DMO-USA-99919</li> <li>Department A users</li> </ul> <p>Department B space</p> <ul> <li>Billing resource HP-AMS-DMO-USA-99920</li> <li>Department B users</li> </ul> <p>Department C space</p> <ul> <li>Billing resource HP-AMS-DMO-USA-99918</li> <li>Billing resource HP-AMS-DMO-USA-99920</li> <li>Department C users</li> </ul> <p>In this way, user John, who is a member of Department A users would only have access to Department A space and would be able to access billing resources HP-AMS-DMO-USA-99918 and HP-AMS-DMO-USA-99919.</p> <p>User Mary, who is a member of both Department A users and Department B users would have access to both Department A space and Department B space. User Mary would be able to see any of the billing resources by selecting the appropriate space.</p> <p>User Robert who is a member of the Department C users, would be able to access the Department C space with access to billing resources HP-AMS-DMO-USA-99918 and HP-AMS-DMO-USA-99920.</p> <h2>Example 3: Big and Small Corp</h2> <h3>Design</h3> <p><img src="/img/iam-document-example-3-tenant-1-1a.jpg" alt=""></p> <p><img src="/img/iam-document-example-3-tenant-2-1-.jpg" alt=""></p> <p><img src="/img/iam-document-example-3-tenant-3-1-.jpg" alt=""></p> <p>Big and Small Corp. is a large multinational company with several divisions. Each division is located in a different region and manages its services separately. Big and Small (USA) is headquartered in the US and is the holding company for all US-based business. This division runs a large marketing service which has a web presence, hosted on a public cloud. Big and Small (USA) would like to replace the public cloud with HPE GreenLake for Private Cloud Enterprise and manage the charges for this service within the division.</p> <p>Big and Small (Europe) is headquartered in Berlin, Germany. This division designs and manufactures a wide range of products for the European market. To support the division, they currently own a large HPC cluster, which they would like to replace with HPE GreenLake for HPC. This division also would like to manage their HPE GreenLake expenses in a separate account.</p> <p>Finally, Big and Small (Japan) is an acquisition, based in Tokyo, Japan. This division had a previous relationship with HPE and already has a billing account, which they would like to retain. This division is responsible for future product development. They would like to use HPE GreenLake for ML Ops to develop innovative new products.</p> <p>Since some divisions of Big and Small Corp. came via acquisition, they have different email addresses than the other divisions. Therefore, it is decided that each division should have a separate tenant. This will allow each division to manage their own HPE GreenLake Services and billing accounts.</p> <p><strong>NOTE: Unless the customer has multiple entities and/or multiple email domains – HPE recommends consolidating all resources under a single tenant. This example is to show that multiple tenants/billing Accounts are supported but is not generally recommended unless specifically required.</strong></p> <p>Each division has their own unique billing account:</p> <table> <thead> <tr> <th>Division</th> <th>Billing Account</th> </tr> </thead> <tbody> <tr> <td>Big and Small (USA)</td> <td>HP-AMS-DMO-USA-99918</td> </tr> <tr> <td>Big and Small (Europe)</td> <td>HP-EMEA-DMO-DEU-99919</td> </tr> <tr> <td>Big and Small (Japan)</td> <td>HP-APJ-DMO-JPN-99920</td> </tr> </tbody> </table> <p>A tenant for Big and Small (USA) is created. The main service in this tenant is HPE GreenLake for Private Cloud Enterprise. This is configured such that the metrics from this service are sent to the billing account 'HP-AMS-DMO-USA-99918'.</p> <p>Each department has their own separate user group, e.g. Department A users. Users from the various departments are added to the appropriate user groups.</p> <p>A space, '<strong>Main Space</strong>' is created within the IAM Service of this tenant and resources are added. In this case, the resource for the billing account HP-AMS-DMO-USA-99918 and also the resource for the HPE GreenLake for Private Cloud Enterprise are added. Finally, the user groups are assigned to the space with the appropriate roles.</p> <p>A tenant for Big and Small (Europe) is created. The main service in this tenant is HPE GreenLake for HPC. This is configured such that the metrics from this service are sent to the billing account 'HP-EMEA-DMO-DEU-99919'.</p> <p>Each department has their own separate user group, e.g. Department D users. Users from the various departments are added to the appropriate user groups.</p> <p>A space '<strong>Main Space</strong>' is created within the IAM Service of this tenant and resources are added. In this case, the resource for the billing account HP-EMEA-DMO-DEU-99919 and also the resource for the HPE GreenLake for HPC are added. Finally, the user groups are assigned to the space with the appropriate roles.</p> <p>A tenant for Big and Small (Japan) is created. The main service in this tenant is HPE GreenLake for ML Ops. This is configured such that the metrics from this service are sent to the billing account 'HP-APJ-DMO-JPN-99920'.</p> <p>Each department has their own separate user group, e.g. Department F users. Users from the various departments are added to the appropriate user groups.</p> <p>A space '<strong>Main Space</strong>' is created within the IAM service of this tenant and resources are added. In this case, the resource for the billing account HP-APJ-DMO-JPN-99920 and also the resource for the HPE GreenLake for ML Ops are added. Finally, the user groups are assigned to the space with the appropriate roles.</p> <h1>Conclusion</h1> <p>As you can see, the key to configuring Identity and Access Management in a tenant is to understand the building blocks and how to make informed decisions on how to design IAM resources to match customer requirements. I hope these examples will help you to see how the flexibility of HPE GreenLake Central Identity and Access Management allows an almost infinite number of possible configurations, giving the flexibility to match diverse customer requirements.</p> <p>Please feel free to reach out directly to me on the <a href="https://slack.hpedev.io/">HPE Developer Slack Workspace</a> in the <a href="https://hpedev.slack.com/archives/C02EG5XFK8Q">#hpe-greenlake</a> channel.</p>https://developer.hpe.com/infrastructure-as-code-on-hpe-greenlake-using-terraform-–-part-2/https://developer.hpe.com/infrastructure-as-code-on-hpe-greenlake-using-terraform-–-part-2/Wed, 04 May 2022 13:13:17 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <p>The process of managing and provisioning computer data centers through machine-readable definition files, otherwise known as Infrastructure-as-Code (IaC), offers many significant benefits. It helps to increase operational agility, simplify management, reduce errors, and save cost. In this second post, I’ll explore how to extract more of the benefits of using IaC on HPE GreenLake through the use of Terraform.</p> <h2>Let’s recap</h2> <p>In <a href="https://developer.hpe.com/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/">my first blog</a>, I covered HPE GreenLake with its Private Cloud Service and showed how to get started with Terraform and the Terraform provider for HPE GreenLake. In this post, I will start with the same VM configuration created in Part 1 and show you how to tap into more advanced functionality that’s provided by Terraform and the HPE GreenLake provider. If you’re coming in to this just now, you might want to follow the steps shown in my Part 1 blog post, up to the point (Terraform ready to apply) where the VM DidierTest1 is created in HPE GreenLake, as illustrated below:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture1-1.png" alt="DidierTest1 VM as created in Part 1" title="DidierTest1 VM as created in Part 1"></p> <p>The corresponding Terraform configuration file showed the following:</p> <pre><code class="language-markdown"># Load HPE GreenLake terraform provider terraform {       required_providers {          hpegl = {             source  = "hpe/hpegl"             version = "0.3.17"          }       }    } # Setup provider environment (location and space) provider "hpegl" {       vmaas {          location   = "HPE"          space_name = "TerraForm Space"       }    } # Start retrieving resources # Retrieve a cloud data "hpegl_vmaas_cloud" "cloud" {      name = "HPE GreenLake VMaaS Cloud-Trial4 "    } # And a few networks data "hpegl_vmaas_network" "blue_net" {      name = "Blue-Network"    } data "hpegl_vmaas_network" "green_net" {      name = "Green-network"    } data "hpegl_vmaas_cloud_folder" "compute_folder" {    cloud_id = data.hpegl_vmaas_cloud.cloud.id    name     = "ComputeFolder"    } # Locate a resource pool data "hpegl_vmaas_resource_pool" "cl_resource_pool" {      cloud_id = data.hpegl_vmaas_cloud.cloud.id      name = "ComputeResourcePool"    } # And a group data "hpegl_vmaas_group" "default_group" {   name = "HPEDEV-HackShackTenant-Group" } # Locate a plan data "hpegl_vmaas_plan" "g1_small" {      name = "G1-Small"    } # A layout data "hpegl_vmaas_layout" "vmware" {   name               = "VMware VM with vanilla CentOS"   instance_type_code = "glhc-vanilla-centos" } # And a template data "hpegl_vmaas_template" "vanilla" {      name = "vanilla-centos7-x86_64-09072020"    } resource "hpegl_vmaas_instance" "DidierTest1" {      name               = "DidierTest1"      cloud_id           = data.hpegl_vmaas_cloud.cloud.id      group_id           = data.hpegl_vmaas_group.default_group.id      layout_id          = data.hpegl_vmaas_layout.vmware.id      plan_id            = data.hpegl_vmaas_plan.g1_small.id      instance_type_code = data.hpegl_vmaas_layout.vmware.instance_type_code      network {          id = data.hpegl_vmaas_network.green_net.id      }       volume {          name         = "root_vol"          size         = 15         datastore_id = "auto"      }      config {          resource_pool_id = data.hpegl_vmaas_resource_pool.cl_resource_pool.id          template_id      = data.hpegl_vmaas_template.vanilla.id          no_agent         = true          asset_tag        = "vm_terraform"          folder_code      = data.hpegl_vmaas_cloud_folder.compute_folder.code      }      power = "poweron"    } </code></pre> <h2>Keeping things in-sync</h2> <p>To understand what the current state of our configuration file is, let’s run another <strong>terraform plan</strong> after the VM has materialized on HPE GreenLake.</p> <pre><code class="language-markdown">terraform plan hpegl_vmaas_instance.DidierTest1: Refreshing state... [id=149] Note: Objects have changed outside of Terraform Terraform detected the following changes made outside of Terraform since the last "terraform apply": # hpegl_vmaas_instance.DidierTest1 has changed ~ resource "hpegl_vmaas_instance" "DidierTest1" { ~ containers = [ ~ { ~ server = [ - { - compute_server_type = [ - { - external_delete = true - managed = true - name = "VMware Linux VM" }, ] - date_created = "2022-03-16T16:28:08Z" - id = 155 - last_updated = "2022-03-16T16:29:11Z" - owner = [ - { - username = "hpedev-hackshack-terraform" }, ] - platform = "" - platform_version = "" - server_os = [] - ssh_host = "172.17.70.27" - ssh_port = 22 - visibility = "private" }, + { + compute_server_type = [ + { + external_delete = true + managed = true + name = "VMware Linux VM" }, ] + date_created = "2022-03-16T16:28:08Z" + id = 155 + last_updated = "2022-03-30T11:37:40Z" + owner = [ + { + username = "hpedev-hackshack-terraform" }, ] + platform = "" + platform_version = "" + server_os = [ + { + name = "centOS 7 64-bit" }, ] + ssh_host = "172.17.70.27" + ssh_port = 22 + visibility = "private" }, ] # (9 unchanged elements hidden) }, ] id = "149" name = "DidierTest1" # (9 unchanged attributes hidden) + config { + asset_tag = "vm_terraform" + create_user = false + folder_code = "group-v41" + no_agent = true + resource_pool_id = 2 + template_id = 573 } - config { - asset_tag = "vm_terraform" -> null - folder_code = "group-v41" -> null - no_agent = true -> null - resource_pool_id = 2 -> null - template_id = 573 -> null } # (2 unchanged blocks hidden) } Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── No changes. Your infrastructure matches the configuration. Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan: terraform apply -refresh-only </code></pre> <p>Terraform has now detected that there are now additional configuration details (such as the IP address of the VM) that can be expressed concerning the infrastructure. It’s even proposing to synchronize it using a <strong>terraform apply -refresh-only</strong> command. We can follow the advice and bring our configuration state in sync with the backend environment.</p> <pre><code class="language-markdown">This is a refresh-only plan, so Terraform will not take any actions to undo these. If you were expecting these changes then you can apply this plan to record the updated values in the Terraform state without changing any remote objects. Would you like to update the Terraform state to reflect these detected changes?   Terraform will write these changes to the state without modifying any real infrastructure.   There is no undo. Only 'yes' will be accepted to confirm.   Enter a value: yes Apply complete! Resources: 0 added, 0 changed, 0 destroyed. </code></pre> <h2>Infrastructure lifecycle management</h2> <p>Once your infrastructure is created, it will need to evolve over time in order to cope with any change of workload or to perform maintenance. Let’s look at a few scenarios.</p> <h3>Use case 1: Stop this VM</h3> <p>To start, let’s keep things simple. You might want to just turn off the VMs that are part of an infrastructure when you don’t need them to save cost or limit your carbon impact on the planet. If you paid attention to the current configuration file, you’ll see that we inserted a power statement when originally creating the VM. While <em>poweron</em> is the only valid option when creating a new resource of type <strong>hpegl_vmaas_instance</strong>, other values are available for lifecycle management, such as <em>poweroff</em> and <em>suspend</em>.</p> <p>Locate the following section in your configuration file:</p> <pre><code class="language-markdown"> } power = "poweron" } </code></pre> <p>And change it so that the power desired state is set to <em>poweroff</em> as shown below:</p> <pre><code class="language-markdown"> } power = "poweroff" } </code></pre> <p>Save and run a <strong>terraform apply</strong> command, which will prompt you to accept the following change:</p> <pre><code class="language-markdown">Terraform will perform the following actions: # hpegl_vmaas_instance.DidierTest1 will be updated in-place   ~ resource "hpegl_vmaas_instance" "DidierTest1" {         id                 = "149"         name               = "DidierTest1"       ~ power              = "poweron" -> "poweroff" </code></pre> <p>Pretty soon afterwards, you can check out the HPE GreenLake console and see that the VM status was changed to stopped.</p> <p><img src="/img/terraform-greenlake-part2-blog-picture2-2.png" alt="VM is now stopped" title="VM is now stopped"></p> <h3>Use case 2: Setting up tags and labels</h3> <p>That was straight forward, right? Now, restart the VM and try something else. You might want to test out tabs and labels. As organizations scale their cloud environments, they often need to define methodologies for organizing resources. For this, they can leverage tags and labels. Tags consist of key/value pairs that make it easier to search for, or filter, your cloud resources based on categories relevant to the organization. Another option is to attach labels, which are simple values, to your VMs in order to keep track of what it’s used for or who it belongs to.</p> <p>Why don’t you try adding metadata to the VM using tags and labels. According to the <a href="https://github.com/HPE/terraform-provider-hpegl/blob/main/docs/resources/vmaas_instance.md">documentation</a>, you can add labels using the following syntax in our configuration file:</p> <pre><code class="language-markdown"># Using labels labels = ["hackshack", "hpedev"] </code></pre> <p>And you can add tags by inserting the following code snippet in your configuration file:</p> <pre><code class="language-markdown"># Using tags tags = {         team  = "HPE Developer"         support = "gold"   } </code></pre> <p>Save and apply those changes with <strong>terraform apply</strong>, wait a little and look at the VM details. You can see the labels and the tags in the capture below:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture3.png" alt="tags and labels applied to VM" title="tags and labels applied to VM"></p> <h3>Use case 3: Get me more disk space</h3> <p>Another typical use case would be to add another disk to a VM, say a data volume for application usage. The syntax for this is the same as you used to create the VM with its <em>root_vol</em>, already visible in the Storage details of the VM:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture4.png" alt="VM was created with one disk: root_vol" title="VM was created with one disk: root_vol"></p> <p>Go ahead and add the following code snippet to your configuration file:</p> <pre><code class="language-markdown">  # Add another volume volume {          name         = "data_vol"          size         = 25          datastore_id = "auto"      } </code></pre> <p>Save the file, apply those changes, wait a little and check the VM storage configuration again:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture5.png" alt="VM with two disks" title="VM with two disks"></p> <h3>Use case 4: Please snap this VM</h3> <p>Here’s one last use case you can try, which consists of snapshotting the VM. You can do tis by adding the following Terraform code snippet in your configuration file:</p> <pre><code class="language-markdown">   snapshot {     name        = "Snap1     description = "Snap this VM so we can restart from this state"   } </code></pre> <p>Save the file, apply those changes, wait a little and check the details of the VM once again, in the Backups section:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture6.png" alt="Snap of VM ready" title="Snap of VM ready"></p> <h2>Debugging when things go wrong</h2> <p>In this post, I’ve showed you how to make sure the Terraform configuration file is valid before applying changes using the <strong>terraform validate</strong> command. To see more details during an apply command, you can also enable Terraform debug by simply setting up the TF_LOG environment variable. I suggest setting it up to DEBUG but other supported values are: TRACE, INFO, WARN and ERROR.</p> <pre><code class="language-markdown">export TF_LOG=DEBUG </code></pre> <h2>What’s next?</h2> <p>In <a href="https://developer.hpe.com/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform/">my first blog post</a>, I covered how to get started with the Terraform provider for HPE GreenLake, I explained how to collect data from the platform and request the creation of a Virtual Machine instance. In this article, I showed you how to manage the lifecycle of a Virtual Machine using Terraform. I applied several changes to the Terraform infrastructure configuration file and observe how the desired state is automatically tracked by Terraform and applied to HPE GreenLake.</p> <p>-      <a href="https://www.terraform.io/"> Learn more about Terraform</a></p> <p>-      <a href="https://www.hpe.com/us/en/greenlake.html"> Learn more about HPE GreenLake</a></p> <p>-      <a href="https://registry.terraform.io/providers/HPE/hpegl"> Learn more about the HPE GreenLake Terraform provider</a></p> <p>Don’t forget, you can always find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog">HPE Developer blog</a>.</p>https://developer.hpe.com/2022-May-04/https://developer.hpe.com/2022-May-04/Wed, 04 May 2022 13:00:00 GMThttps://developer.hpe.com/deep-learning-model-training-–-a-first-time-user’s-experience-with-determined-–-part-2/https://developer.hpe.com/deep-learning-model-training-–-a-first-time-user’s-experience-with-determined-–-part-2/Tue, 03 May 2022 13:51:27 GMT<p>Determined is an open-source training platform that aims to simplify deep learning (DL) model development and experimentation for data science teams by providing tools like distributing training, automatic model tuning, GPU resource management, and automatic experiment tracking.</p> <p>In <a href="https://developer.hpe.com/blog/deep-learning-model-training-%E2%80%93-a-first-time-user%E2%80%99s-experience-with-determined-part-1/">my previous blog post</a>, I put on my IT operations manager’s hat and discussed how I deployed Determined on a Kubernetes cluster in an on-premises HPE Ezmeral Runtime Enterprise environment. I also showed how easy it is to get started with the Determined CLI, REST API, and Web User Interface to interact with Determined.</p> <p>In this second part of the series, using the Determined setup from Part I, I’ll assume the role of a data scientist/ML Engineer who wants to:</p> <ul> <li>Explore fundamental Determined concepts and features to train a TensorFlow model</li> <li>Track and visualize the progress and results of the training process using a single GPU</li> <li>Use distributed training across multiple GPUs and fine-tune the model with state-of-the-art hyperparameter search <br/></li> </ul> <p>I will also use the Determined Python API in a Jupyter Notebook to load and test the trained model and see how well it performs. I’ll evaluate this by making inferences, which uses a trained model and new, unlabelled data to make predictions.</p> <h2>Overview of the Determined training model process</h2> <p>In short, Determined permits data science teams to launch deep learning model training tasks, called <em><strong>trials</strong></em>, for their ported model. These tasks are distributed across one or more GPUs and grouped as a Determined <em><strong>experiment</strong></em> using a particular set of configuration parameters specified in an <em><strong>experiment configuration file</strong></em>. This configuration file tells Determined how to run the model training process on Determined in terms of many different parameters, including, but not limited to, the following:</p> <ul> <li>The hyperparameters</li> <li>The number of GPUs to use for the training task</li> <li>The amount of data on which to train a model</li> <li>How often the trial task must report the training metrics and the validation metrics to the Determined Master</li> <li>How often the trial task must save the model file</li> </ul> <p>The experiment configuration file and the Python code for the model used to load the datasets, build, optimize, and compile the model are collected in a <em><strong>model definition directory</strong></em>. The directory can optionally contain a startup-hook.sh script to install additional Python dependencies and libraries before the training process starts.</p> <blockquote> <p>Note: The Experiment configuration file has required and optional fields. I’ll explore the most common fields in this post. To learn more about Experiment configuration settings, check out the online documentation <a href="https://docs.determined.ai/latest/training-apis/experiment-config.html">here</a>.</p> </blockquote> <h2>The use case and the model</h2> <p>To get started with Determined, I need a use case and a model to train in Determined. As a Data Scientist running an experiment on Iris species, I pick the simple and well-known <em>“Iris classification”</em> model for predicting the likelihood that the flowers are an Iris species (Iris setosa, Iris versicolor or Iris virginica) based on their sepal and petal length and width measurements.</p> <p>To take advantage of Determined's functionalities, I need to port the model to Determined framework APIs such as PyTorch, Tensorflow, and Keras, the most commonly used deep learning frameworks. You can check out the <em>Iris deep learning model</em> code — a TensorFlow Keras based model — in the <a href="https://github.com/determined-ai/determined/tree/master/examples/computer_vision/iris_tf_keras">Determined GitHub repository</a> and download the complete code for this use case <a href="https://docs.determined.ai/latest/_downloads/b8b05d77875d7d5a43ea2bd4b35fb0f4/iris_tf_keras.tgz">here</a>. I’ll train the model on the publicly available Iris <a href="https://storage.googleapis.com/download.tensorflow.org/data/iris_training.csv">training dataset</a> and <a href="https://storage.googleapis.com/download.tensorflow.org/data/iris_test.csv">validation dataset</a>, which consist of 120 samples and 30 samples, respectively. Each sample consists of the following Iris flower properties:</p> <ul> <li>Features: sepal length, sepal width, petal length, petal width</li> <li>Label: the species of Iris to predict</li> </ul> <p>I’ve also stored a copy of the datasets in the shared storage volume for my Determined deployment described in the <a href="https://developer.hpe.com/blog/deep-learning-model-training-%E2%80%93-a-first-time-user%E2%80%99s-experience-with-determined-part-1/">first post of this series</a>. I simply changed the model code to ensure the data loader function loads and reads training and validation datasets from the shared storage volume.</p> <blockquote> <p>Note: Porting deep learning model code to Determined is beyond the scope of this blog series. The easiest way to learn how to port an existing deep learning model code to Determined is to start with the <a href="https://docs.determined.ai/latest/tutorials/pytorch-porting-tutorial.html">PyTorch Porting tutorial</a>.</p> </blockquote> <p><strong>Time to see Determined in action!</strong></p> <h2>Launching my first experiment to train our model on a single GPU</h2> <p>Let’s start by simply launching an experiment with a single training task for the Iris deep learning model on a single GPU by defining the hyperparameters as fixed values in the experiment configuration file.</p> <h3>The Experiment configuration file</h3> <p>Here’s a quick look at the experiment configuration file (<em>const.yaml</em>):</p> <pre><code class="language-yaml">name: iris_const_testuser1 hyperparameters: # the hyperparameters to use for the training task learning_rate: 1.0e-4 learning_rate_decay: 1.0e-6 layer1_dense_size: 16 global_batch_size: 30 # Number of data records within a batch resources: slots_per_trial: 1 # Default. Use 1 GPU to train the model. searcher: name: single # Single searcher method disables hyperparameter search (HPO) metric: val_categorical_accuracy # The validation metric to evaluate the performance smaller_is_better: false # The higher the metric the better the performance max_length: # Amount of data on which to train the model batches: 5000 # Set in the unit of batches (can be expressed as epochs too) entrypoint: model_def:IrisTrial # Starting point of the model code min_validation_period: batches: 1000 # Report validation metrics to Master every 1000 batches scheduling_unit: 100 # Report training metrics to Master every 100 batches (default) bind_mounts: # Training and validation datasets location in shared volume. Ensure the datasets on the shared volume are accessible to the training tasks. - host_path: /opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/repo/data container_path: /opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/repo/data read_only: true </code></pre> <p>At a glance, this configuration YAML file is full of concepts that data scientists and ML engineers are familiar with, such as hyperparameters, batches, batch size, metrics, etc. In other words, using the experiment configuration file allows me, as a data scientist, to describe experiments with the terms I’m familiar with. From this file, Determined can take care of the model training for me.</p> <h3>Creating the experiment</h3> <p>With the <em>const.yaml</em> experiment configuration file and the model code in the model definition directory, after being authenticated as a Determined user (<code>det user login &#x3C;username></code>), I’m going to start an experiment using the Determined CLI command line:</p> <pre><code class="language-bash">det experiment create const.yaml &#x3C;model-definition-directory> </code></pre> <p>Determined returns the experiment_Id and schedules the training task as a Kubernetes POD in my Kubernetes cluster. The POD container has all the libraries and dependencies required for training typical deep learning models with common deep learning frameworks like PyTorch, TensorFlow, and Keras.</p> <p>I can then use a couple more Det CLI commands to track the execution progress of my experiment:</p> <pre><code class="language-bash">det experiment list | tail -1 det experiment describe &#x3C;my-experiment-Id> </code></pre> <p>Once the experiment completes, I can use the CLI command below to discover the performance metric for the best version of the validated model for my experiment:</p> <pre><code class="language-bash">det experiment list-checkpoints --best 1 &#x3C;my-experiment-Id> </code></pre> <h3>Visualizing and inspecting the learning curve of the trained model</h3> <p>I can also access information on both training and validation performance for my experiment using the Determined WebUI. From the dashboard I see my experiment status, as shown in the following screenshot:</p> <p><img src="/img/webui-myexp-const-status.png" alt="Determined WebUi Dashboard" title="Determined WebUi Dashboard"></p> <p>Selecting the experiment, I visualize the learning curve, which shows the model validation and training accuracy metric over the number of completed batches:</p> <p><img src="/img/webui-myexp-const-graph.png" alt="Experiment performance visualization" title="Experiment performance visualization"></p> <p>Here, I see the graph changing in real-time as the experiment runs. Determined plots training metrics every <em><strong>100 batches</strong></em> of training data (the purple line) by default. Validation metrics (the blue line) are plotted every <em><strong>1000 batches</strong></em> over the amount of data (5000 batches) based on the parameters I specified in the experiment configuration file.</p> <p>I can also see the training and the validation metrics along with the checkpoints, which are the saved versions of the best-validated model. With the default checkpoint collection policy, Determined will checkpoint and save to a file the most recently validated model and the best model per training task (trial). If the most recent checkpoint is also the best checkpoint for a given trial, only one checkpoint will be saved for that trial, as in my example above.</p> <p>For data scientists familiar with TensorBoard, Determined also provides learning curve visualization through TensorBoard. I can launch a TensorBoard task from the WebUI or by using the Det CLI command below to start an instance of TensorBoard server in Determined.</p> <pre><code class="language-bash">det tensorboard start &#x3C;Experiment-Id> </code></pre> <p>The TensorBoard server instance will be launched as a container POD in the Kubernetes cluster. To stop the instance, I just use the command:</p> <pre><code class="language-bash">det tensorboard kill &#x3C;tensorboard-Id> </code></pre> <h3>Evaluating the model by making inferences using a local Jupyter Notebook</h3> <p>With the model trained and the best model files saved on the shared storage volume for my Determined deployment, I can now download the checkpoint files for the best version of the model, test it and see how well it performs using the <a href="https://docs.determined.ai/latest/interact/api-experimental-client.html">Determined Python API</a>. Using the command below, I simply start a <strong>CPU-only</strong> JupyterLab server instance with a bind-mounting configured to ensure that the validated model file and checkpoint files are accessible by the JupyterLab instance. Like any other task launched in Determined, the JupyterLab instance is launched as a container POD in the Kubernetes cluster.</p> <pre><code class="language-bash">det notebook start --config-file Notebook-config.yaml </code></pre> <p>The <em>Notebook-config</em> YAML configuration file below is used to control a JupyterLab instance deployment. When downloading checkpoints from a shared file system, Determined assumes the checkpoints location is mounted to the mount point <em><strong>determined_shared_fs</strong></em> inside the JupyterLab POD container. To learn more about Jupyter Notebook in Determined, check out the <a href="https://docs.determined.ai/latest/features/notebooks.html">Notebook documentation</a> and <a href="https://www.determined.ai/blog/maximize-juptyter-notebook-experience-determined">a recent Determined’s blog post on this topic</a>.</p> <pre><code class="language-yaml">description: My-notebook resources: slots: 0 # Launch a Notebook that does not use any GPU bind_mounts: # Validated model checkpoints location in the shared volume - host_path: /opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/checkpoints container_path: /determined_shared_fs # Mount point the host_path is mounted to inside the JupyterLab POD container idle_timeout: 30m </code></pre> <p>With the JupyterLab instance deployed, I use the <code>det notebook list</code> command to get the Notebook-Id, then connect to a Jupyterlab instance using my browser at the following URL: http://&#x3C;DET_MASTER_URL>/proxy/&#x3C;Notebook-Id>.</p> <p>Finally, I use the Determined <a href="https://docs.determined.ai/latest/interact/api-experimental-client.html">Python Client API</a> and its TensorFlow Keras <a href="https://docs.determined.ai/latest/post-training/use-trained-models.html">Checkpoint API</a> in the following Python code example to download the best model file, load it into memory as a Python process for TensorFlow Keras based model, and make inferences.</p> <p>Based on the flower's measurements, the model will predict, for each unlabelled example, the likelihood that the flower is the given Iris species (Iris setosa, Iris versicolor or Iris virginica) and print out the actual numeric value for each unlabelled example that has the highest confidence value.</p> <p>The Python code below:</p> <ul> <li>Hides the GPU device to run the code on CPU-only Jupyter Notebook</li> <li>Disables all logging output from TensorFlow Keras</li> <li>Imports the Determined Python libraries</li> <li>Authenticates me as Determined user to interact with my Determined experiment</li> <li>Downloads and loads the best model checkpoint for my experiment from the shared checkpoint storage volume</li> <li>Tests the model by making predictions using the loaded model and unlabelled examples data</li> </ul> <pre><code class="language-python">import os os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" os.environ["CUDA_VISIBLE_DEVICES"]="-1" os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # import numpy as np from determined.experimental import client from determined import keras # myexpId=”&#x3C;experiment-Id>” client.login(master=&#x3C;DET_MASTER_URL>, user="MyUsername", password="MyPassword") best_checkpoint = client.get_experiment(myexpId).top_checkpoint() checkpoint_path = best_checkpoint.download() model = keras.load_model_from_checkpoint_path(checkpoint_path) # # Inferences X_new = np.array([[5, 3.9, 2, 0.5], [5, 2.5, 3, 1], [6.9, 3.1, 5.4, 2.1]]) prediction = model(X_new) print("Let's predict the likelihood that the flower is the given Iris species 0: Iris setosa, 1: Iris versicolor, 2: Iris virginica {}".format(prediction)) print("") print("the prediction of species for the 3 unlabelled examples is:") print(np.argmax(prediction,axis=1)) </code></pre> <h3>Cleaning up computational resources</h3> <p>Once I’ve finished with the test of my model, I can stop the JupyterLab instance and release computational resources on my Kubernetes cluster with the command:</p> <pre><code class="language-bash">det notebook kill &#x3C;notebook-Id> </code></pre> <h2>Distributed training with multiple GPUs</h2> <p>I’ll now launch another experiment that trains a single instance of my deep learning model using multiple GPUs, known as <a href="https://docs.determined.ai/latest/training-distributed/index.html">distributed training</a>. Similar to my first experiment, this experiment features a single trial with a set of constant hyperparameters.</p> <p>Determined can coordinate multiple GPUs to train a deep learning model more quickly by leveraging multiple GPUs on a single machine or over multiple machines. Typically, data science teams use distributed training to train models on larger datasets to improve model performance and accuracy, leveraging additional compute resources.</p> <p>Determined automatically executes <a href="https://www.oreilly.com/content/distributed-tensorflow/">data parallelization</a> training, where a data set is divided into multiple pieces and distributed across the GPUs, <strong>requiring minimal changes to model code</strong>. Each GPU has the full model code but trains the model on its portion of the data. Determined ensures training coordination across multiple GPUs on a single machine or multiple machines to keep the overall training task in sync.</p> <p>To launch a multi-GPU experiment, all I need to do is specify the desired number of GPUs I want to use in the experiment configuration file without requiring any model code changes, and Determined takes care of the rest. For example, in the <em>distributed.yaml</em> experiment configuration file, I specify two GPUs per trial in the <strong>resources</strong> section:</p> <pre><code class="language-Yaml">resources: slots_per_trial: 2 </code></pre> <p>I launch the experiment using the command:</p> <pre><code class="language-bash">det experiment create distributed.yaml &#x3C;model-definition-directory> </code></pre> <p>With this configuration, Determined runs a single trial for my experiment. The trial uses two GPUs to train my model, whether leveraging two GPUs on a single machine or two GPUs across multiple machines in the Kubernetes cluster.</p> <p>As with the other experiments, I navigate to the WebUI to monitor the progress of the training task for my experiment and visualize information on both training and validation performance over the number of completed batches. I can use the same <em><strong>Det</strong></em> CLI commands that I used for my first experiment to discover the performance metric for my model and launch auxiliary tasks, such as a TensorBoard server or a JupyterLab Notebook server. And, of course, I can use the same Determined Python API code I used for my first experiment to load and test the trained model to see how well it performs to make predictions.</p> <h2>Automatic model tuning with Determined</h2> <p>Previously, I showed you how to easily distribute a training task across multiple GPUs without changing your model code. Here, I’ll look at another way that an experiment can benefit from multiple GPUs. Determined makes it easy for data scientists and ML engineers to apply advanced functionality such as automatic model tuning with hyperparameter search, known as <strong>Hyperparameter Optimization</strong> (HPO), to accelerate the hyperparameter search for their model with minimal effort. <a href="https://docs.determined.ai/latest/training-hyperparameter/index.html#hyperparameter-tuning">Determined HPO</a> uses a Searcher algorithm like Random, Grid, Adaptive, or PBT, and ranges of hyperparameters, which are specified in the experiment configuration file.</p> <p>In general, data scientists experiment with several learning algorithms using a variety of hyperparameters on the same dataset by launching several training processes. They do so to find the model that works best for the business problem they are trying to solve. Determined HPO automates this process to find the best-performing model by running many training tasks, or trials, on the same dataset and code. Determined launches the trials simultaneously on different GPUs. Each trial uses a different configuration of hyperparameters <strong>randomly</strong> chosen by the Searcher from the range of values specified in the experiment configuration file. Determined then chooses the set of hyperparameter values that result in a model that performs the best, as measured by the validation metric defined in the experiment configuration file.</p> <p>I’ll use the following experiment configuration file (<em>adaptive.yaml</em>) to launch the HPO experiment in Determined. As you can see, the set of user-defined hyperparameters range and the state-of-the-art searcher method <em>Adaptive_ASHA</em> are now defined. In the <em><strong>searcher</strong></em> section, the <em><strong>max_trials</strong></em> parameter indicates the number of trials that the experiment will create. Determined will explore <em>max_trials</em> model configurations for you. The validation metric and the amount of data on which to train the model are the same as in my previous experiments. Each trial runs on one GPU because the resource parameter slot_per_trial is not specified. Therefore the default setting of a single GPU per trial is used.</p> <blockquote> <p>Note: The Adaptive_ASHA searcher method works best with many hundreds of trials. For the purpose of my experimental use case, I set the maximum number of trials to six. Determined will explore six model configurations.</p> </blockquote> <pre><code class="language-yaml">name: iris_adaptivesearch hyperparameters: # Hyperparameters specified as ranges learning_rate: type: log minval: -5.0 maxval: 1.0 base: 10.0 learning_rate_decay: 1.0e-6 layer1_dense_size: type: int minval: 4 maxval: 32 global_batch_size: type: int minval: 5 maxval: 30 searcher: name: adaptive_asha # The HPO Searcher algorithm metric: val_categorical_accuracy smaller_is_better: false max_length: batches: 5000 max_trials: 6 # Number of trials to launch for the HPO experiment entrypoint: model_def:IrisTrial bind_mounts: - host_path: /opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/repo/data container_path: /opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/repo/data read_only: true </code></pre> <p>I then launch the experiment using the command:</p> <pre><code class="language-bash">det experiment create adaptive.yaml &#x3C;model-definition-directory> </code></pre> <p>As in the previous experiments, I navigate to the WebUI to monitor the training task progress and to access information on both training and validation performance for the experiment trials. As shown in the following figure, Determined hyperparameter search functionality gives me with several <a href="https://www.determined.ai/blog/hyperparameter-visualizations-determined">visualization options</a> for analyzing results: Learning Curve, Parallel Plot, Scatter Plot, Heat Map.</p> <p>As the experiment runs, I select the training metric <em>categorical_accuracy</em> in the <em><strong>Learning Curve</strong></em> tab to visualize the model accuracy on training data for each trial over the number of completed batches. I can see that the Searcher Adaptive ASHA's <em><strong>early stopping</strong></em> capability has stopped poor-performing trials that do not require extra training. Determined releases valuable GPU resources on trials that are evaluated to never produce the best model.</p> <p><img src="/img/webui-myexp-adaptive-graphs-v2.png" alt="HPO Adaptive experiment trials visualization" title="HPO Adaptive experiment trials visualization"></p> <p>When the Determined experiment is complete, I’ll navigate to the WebUI <strong>Trials</strong> tab and compare the results of different trials and discover the hyperparameters that yield the best model that will enable me to perform better future experiments by further tuning the hyperparameters. One could also use the CLI commands below to get the best trial and discover the hyperparameters values for the best model:</p> <pre><code class="language-bash">det experiment list-checkpoints --best 1 &#x3C;MyExperiment-Id> det trial describe &#x3C;Trial-Id> </code></pre> <p>And, of course, I can use the same Python API code I used earlier to load and test the best model and make inferences.</p> <h2>Summary</h2> <p>During both parts of this blog series, I wore a couple of hats: an IT operations manager’s hat and a data scientist/ML engineer’s hat.</p> <p>With my IT operations manager’s hat, I deployed Determined on a Kubernetes cluster running on HPE Ezmeral Runtime Enterprise that provides all the components needed to run Determined: a workload scheduler, such as Kubernetes, a namespace, multi-tenancy, an ingress gateway, persistent storage for experiments tracking, and a shared file system for storing model artifacts and datasets.</p> <p>With my data scientist/ML engineer’s hat, I used Determined and its interfaces (CLI and the Web User Interface) to get started with Determined fundamental concepts, training a simple Iris classification neural network model using multiple GPUs with distributed training and advanced functionality such as state-of-the-art hyperparameter search. I also used the Determined Python API to load and test the trained model and to make inferences.</p> <p>The Iris classification example used in this post is relatively simple. In reality, you would use Determined to build and train more complex deep learning models with much larger datasets, probably using a larger compute infrastructure with plenty of GPUs available to parallelize training models across data science teams.</p> <p>I hope you found this information interesting and useful in helping you get started with Determined. I was able to write this two-part blog series by joining and receiving help from the Determined Community Slack, which you can do by <a href="https://join.slack.com/t/determined-community/shared_invite/zt-cnj7802v-KcVbaUrIzQOwmkmY7gP0Ew">following this link</a>. You can begin training models with Determined today by visiting the <a href="https://github.com/determined-ai/determined">Determined project on GitHub</a>.</p>https://developer.hpe.com/becoming-a-linux-kernel-contributor-following-the-journey-of-souptick-joarder/https://developer.hpe.com/becoming-a-linux-kernel-contributor-following-the-journey-of-souptick-joarder/Tue, 26 Apr 2022 19:38:25 GMT<p><img src="/img/linux_source_code_primary_site.png" alt="The Linux Kernel Archive" title="The Linux Kernel Archive"></p> <p>As a leading global, edge-to-cloud company, Hewlett Packard Enterprise (HPE) prides itself in employing team members who share one common purpose: to advance the way people live and work. Because of this, HPE boasts some of the finest <a href="https://www.hpe.com/us/en/open-source.html">Open Source</a> engineering talent. In this blog series, you’ll get to meet a number of them as I interview some of the Open Source experts who make up our team.</p> <p>In this blog interview, Souptick Joarder, who has been contributing to the Linux Kernel for the last four years, describes his journey on becoming a trusted patch reviewer and contributor to Linux. Souptick first encountered Linux while studying embedded systems and became interested in Linux Kernel programming due to how Linux lent itself to exploration and modification according to one’s needs. He appreciated being able to give back to the community. Souptick received his Master of Technology degree in software systems at the Birla Institute of Technology and Science and works for HPE as a storage systems engineer.</p> <h3>Contributing to such a large Open Source project must be a daunting task. What advice would you give to would-be contributors?</h3> <p>To be sure, in the last four years that I’ve been involved, I’ve had to learn the ins-and-outs of making contributions and how to overcome the challenges involved. First and foremost, it’s important to understand that contributions to Linux are done through a trust-based development model. And it takes contributors a while to build that trust. Another thing that’s important to understand is the inherent hierarchy of the contribution process, since you’ll need to know who to connect to in order to build that trust. So, if you’re interested in being a kernel contributor, understand that you’ll need a lot of patience.</p> <h3>What would you recommend as the best way to get started?</h3> <p>When getting started, I’d recommend that you choose a particular area that interests you to focus on. I spend most of my time acting as a reviewer for patches posted in the Memory Management mailing list for the Virtual Memory Management subsystem. It contains the implementation of demand paging and virtual memory. Also, it contains memory allocation for user space programs and kernel internal structures.</p> <p><img src="/img/mm_mailing_list.png" alt="Memory Management Mailist List" title="Memory Management Mailist List"></p> <p>In parallel, I fix warning/errors reported by different kernel test bots on the memory management mailing list. Here is <a href="https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/?qt=grep&#x26;q=jrdr.linux@gmail.com">the link for a list of patches</a> for which I remain an Author/ Reviewer/ Part of Discussion.</p> <p>There are many different subsystems in Linux and each has a named maintainer called the Lieutenant. For example, Andrew Morton is the Memory Management (MM) maintainer. He’ll screen patches posted primarily for memory management and check to see that the patches have followed the process and whether they actually do what they set out to do. Post review, he will merge it into his MM Git branch. He usually accepts small patches from new developers as well but for large scale changes he takes only from trusted developers. Over time, you’ll learn how to submit to each subsystem according to how the lieutenants like to receive contributions.</p> <h3>How does one make an actual contribution?</h3> <p>After the source code change is made and the pull request opened in the repository, fixes/patches/enhancements are all submitted through mailing lists. Linux development is set up using a number of different mailing lists that align with different subsystems of the kernel. When you make your contribution, you’ll send it off to a specific mailing list and then you also want to CC the more general associated mailing list.</p> <p>For instance, when I make my contributions for the virtual memory management subsystem, I send it to the memory management mailing list. Then I also CC the higher-level, more general Linux Kernel Mailing List (LKML). There is a <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS">MAINTAINERS file</a> (text file) inside the kernel source code that will tell you exactly who should receive a review request for your patches. Once your patch is ready, you need to refer to this MAINTAINERS file and, based on the files you have modified, send patches to only those maintainers and mailing lists.</p> <h3>How do you determine what to add or fix?</h3> <p>Reviewing patches is considered one of the most important things to do in terms of the Linux Kernel development process. You can really make substantial contributions here. An additional benefit of reviewing patches from other developers is that you get the opportunity to be involved in their contribution work and learn from it. Kernel test bots work from the Git repositories and report out on warnings and errors. Any developer can analyze those issues and provide a fix by following the correct protocols. It’s a continuous process that developers can spend a good amount of time on.</p> <h3>Could you give an example of one of your contributions?</h3> <p>There had been multiple VM_FAULT_ERR error codes noted, usually reported by drivers upon failures. These VM_FAULT_ERR error codes had been defined as macro. When the drivers return a VM_FAULT_ERR error code, the data type was not shown as unique across the kernel. Each driver chose their own datatype to report the error, which turned out to be a problem.</p> <p>In many cases, the drivers failed to return the VM_FAULT_ERR error code despite seeing a failure and wound up returning SUCCESS instead. It also turned out that there was this silly inefficiency in the drivers (due to the lack of an appropriate API), resulting in ERRNO error codes sometimes being converted to a VM_FAULT_ERR error code before returning to memory management.</p> <p>The plan was to introduce a new datatype (initially named vm_fault_t type) and then to enable all the drivers/filesystems in the entire kernel to use this new type. This would ensure that when any new driver used a datatype other than vm_fault_t to report the VM_FAULT_ERR error code, the compiler would catch it. This way, we would restrict all future callers to use only the vm_fault_t type to report a VM_FAULT_ERR error code. This allows the catching of errors much earlier in the build process (at compile time) instead of waiting for higher in the chain for execution.</p> <p>As part of this change, we also identified all the buggy drivers that were not returning the correct VM_FAULT_ERR error code upon failure and fixed them. We also identified all the other drivers that were converting ERRNO to VM_FAULT_ERR before reporting a failure to memory management. A few new wrapper APIs were added to make the conversions easy.</p> <p>Just to give you an idea of the patience that’s required to do this, this work started with Linux V4.17 and will finish with V5.1 – a whole year’s worth of time.</p> <h3>Any final advice to developers who might like to start contributing to Linux?</h3> <p>It’s important to remember that Linux Kernel development doesn’t work with a deadline-based approach. Patches are reviewed and tested by developers and maintainers all around the world and this takes time. So you need to have a lot of patience. It could take months, or even years, to get your changes merged into the mainline code. Because the maintainers, in the end, are responsible for maintaining any new pieces of code, they believe in the approach of <strong>breaking the code first before accepting it</strong>. This can take time.</p> <p>Something else to keep in mind is that others will often criticize your changes. Sometimes it turns ugly. Learn to take criticism in a positive way. This is probably one of the most important pieces of advice I can give to fellow developers interested in working with the kernel community.</p> <p>Finally, sometimes you’ll find more senior developers who will help you bring your patches forward. You’ll find this a great opportunity to learn new things and it’s also a great way to build your reputation within the community. Remember, since the entire development process is done through email communications, the only way the community will judge you will be by your work – so make your work shine!</p>https://developer.hpe.com/kubecon-cloudnativecon-europe-16-20-may-2022/https://developer.hpe.com/kubecon-cloudnativecon-europe-16-20-may-2022/Fri, 22 Apr 2022 19:03:59 GMT<p><img src="/img/kubeconeu-2022-event-card-image.png" alt="KubeCon EU 2022" title="KubeCon EU 2022"></p> <p>Come join the <a href="https://developer.hpe.com/">HPE Developer Community</a> team at Cloud Native Computing Foundation’s flagship conference, KubeCon + CloudNativeCon Europe 2022. Thousands of cloud-native leaders will come together from May 16th through 20th to this conference to learn about emerging trends in container architecture, advancing the exciting world of cloud-native computing. Hewlett Packard Enterprise (HPE) will have a booth onsite (booth number G11) at the Feria Valencia where the show floor will open on Wednesday, May 18th. We will also provide a virtual presence, including the <a href="https://developer.hpe.com/hackshack/">HPE Developer Hack Shack</a>.</p> <p>In the HPE booth (G11), you’ll get to talk with HPE experts and HPE Developer Community team members who can give you the rundown on key Open Source projects we’re involved in, like SPIFFE/SPIRE, CSI driver, Apache Spark™, etc.</p> <h2>Hear Agustín Martínez Fayó and Marcos Yacob Speak on SPIRE</h2> <p>On Friday, May 20th, contributors and maintainers of the SPIRE project, Agustín Martínez Fayó and Marcos Yacob, will lead a session where you’ll be introduced to the SPIRE project and dive deep into its new Microsoft Windows support. SPIRE, the SPIFFE Runtime Environment, is an extensible system that implements the principles embodied in the SPIFFE (Secure Production Identity Framework for Everyone) standards. It manages platform and workload attestation, provides an API for controlling attestation policies, and coordinates certificate issuance and rotation.</p> <p>The session will provide a high-level overview of the basic concepts behind SPIRE and explain why you should consider using it if you find issuing workload identities at scale challenging. This talk will also offer a deep dive into the Windows support that is being introduced in SPIRE, providing detailed information about the implementation, explaining the differences between running SPIRE on Windows and Linux platforms, and comparing the experience from both a user and developer perspective. <a href="https://kccnceu2022.sched.com/event/yttL">Don’t miss this session</a>, which runs from 14:55 to 15:30. Make sure you catch it before you head out.</p> <h2>In-Booth Presentations</h2> <p>There will be many other opportunities to learn about HPE’s software development work and open source efforts at the HPE booth. Our magician will dazzle you with his magic and quiz you on how much you know about HPE’s involvement in open source. If you’ve paid attention, you may even win a prize! And don’t miss the raffle: Three lucky winners will get a chance to walk away with one Raspberry Pi 4 starter kit each day.</p> <p>In-booth presentations will be given by many of the HPE Developer Community experts every couple of hours.</p> <p>Each day, you’ll have opportunities to learn about the following topics:</p> <ul> <li>What? Not yet an HPE DEV Member? There's Treasure to be Found!</li> <li>Open Source @ HPE – a 25+ Years Love Story</li> <li>Overview of HPE Storage for Kubernetes</li> <li>Introduction to HPE Point Next</li> <li>Introduction to SPIFFE &#x26; SPIRE</li> </ul> <h2>You Can Still Attend Virtually</h2> <p>As we’ve done the last few years, we’ll be offering live office hours where you can connect with HPE subject matter experts and collaborate, learning more about Open Source and HPE technologies. The featured topics will mirror those in the booth.</p> <p>As part of its virtual presence, HPE will once again feature <a href="https://developer.hpe.com/hackshack/">the HPE Developer Hack Shack</a>, a place to learn and have fun. It’s a unique place designed to give virtual events a more personal touch and extend the experience beyond the event. Here’s what the HPE Developer Community will provide to you through the Hack Shack and web portal:</p> <ul> <li><a href="https://developer.hpe.com/hackshack/workshops">WORKSHOPS-ON-DEMAND</a>: Access over two dozen free, on-demand hands-on training courses. Using a Jupyter Notebook environment, they provide you with hands-on experience across the latest HPE and Open Source technologies. Check out our newest workshops, including: <ul> <li><a href="https://developer.hpe.com/hackshack/workshop/33">Docker 101 - Introduction to Docker Concepts</a></li> <li><a href="https://developer.hpe.com/hackshack/workshop/34">Spark 101 - Introduction to Apache Spark Concepts</a></li> <li><a href="https://developer.hpe.com/hackshack/workshop/32">Creating a Zero Trust Model for Microservices Architectures with SPIRE and Envoy</a></li> <li><a href="https://developer.hpe.com/hackshack/workshop/24">Kubernetes 101 - Introduction to the Kubernetes Concepts</a></li> <li><a href="https://developer.hpe.com/hackshack/workshop/18">Building a dynamic Machine Learning pipeline with KubeDirector</a></li> <li><a href="https://developer.hpe.com/hackshack/workshop/2">Using Kubernetes CSI with HPE Ezmeral Container Platform</a></li> </ul> </li> <li><a href="https://bit.ly/kubecon-eu-2022-hpedev-treasure-hunt">TREASURE HUNT</a>: Our scavenger-hunt style challenge encourages you to check out all the resources that are available on the <a href="https://developer.hpe.com/">HPE Developer website</a> and the Hack Shack. Be one of the first 15 people to answer all the questions correctly and win an HPE Developer branded swag!</li> <li><a href="https://developer.hpe.com/hackshack/replays/">REPLAYS</a>: You can find replays of many of the technical workshops we’ve offered live in the past in the Hack Shack as well. View them to learn more about the <a href="https://developer.hpe.com/hackshack/replays/0">HPE Ezmeral Container Platform</a> (now known as HPE Ezmeral Runtime Enterprise), <a href="https://developer.hpe.com/hackshack/replays/27">SPIFFE and SPIRE authentication</a>, and the <a href="https://developer.hpe.com/hackshack/replays/2">HPE Container Storage Interface</a> for Kubernetes.</li> <li><a href="https://developer.hpe.com/hackshack/arcade/">ARCADE</a>: In our arcade, you’ll find Hack Shack Attack! Give this popular retro-style video game a try and compete with your friends for the highest score. There’s also a place where you can download stickers, Zoom backgrounds, and cool artwork to use on your social channels.</li> <li><a href="https://developer.hpe.com/community/">COMMUNITY</a>: We invite you to join and contribute your expertise on our blog or deliver an on-demand workshop. Connect with others in the community via the <a href="https://www.hpe.com/forum/ezmeral">HPE Ezmeral Software Forum</a> or our <a href="https://slack.hpedev.io/">Slack</a> and <a href="https://twitter.com/HPE_Developer">Twitter</a> channels to start conversations and get answers to questions. Sign-up for our <a href="https://developer.hpe.com/newsletter-signup">HPE Developer Newsletter</a> to stay up-to-date on the newest blog posts and tutorials.</li> </ul> <p>As always, HPE is excited to have an opportunity to connect with other technologists at an event such as this. <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/register/">Register now</a> and meet us in Valencia to learn how HPE envisions Open Source running across edge to cloud, working to improve management and analytics, cloud-native security, and data insights. It’s been a long time since we’ve had the opportunity to see each other in person!</p>https://developer.hpe.com/it’s-all-fun-and-games-at-the-hack-shack/https://developer.hpe.com/it’s-all-fun-and-games-at-the-hack-shack/Fri, 22 Apr 2022 12:36:51 GMT<p><img src="/img/lets-hack-shack-512.png" alt="Let&#x27;s Hack Shack at HPE Discover 2022" title="Let&#x27;s Hack Shack at HPE Discover 2022"></p> <p>Can you believe it? Hewlett Packard Enterprise (HPE) will be holding its first major in-person conference since 2019 in just over a month! HPE Discover 2022, the Edge-To-Cloud conference, will be held in Las Vegas at the Venetian/Palazzo Resort from June 28-30th. The <a href="https://developer.hpe.com/">HPE Developer Community</a> team is really excited to be able to engage with attendees in person once again. Make sure you <a href="https://attend.hpe.com/discover2022/index.cfm?iLangID=1">register</a> soon so you don’t miss out!</p> <h2>A place designed especially for you</h2> <p>HPE Discover 2022 attendees will have plenty of opportunity to sit and hear about all the newest HPE solutions and technologies throughout the event. But once you enter the Hack Shack, things get personal! This is where developers, data scientists, and IT technologists have the opportunity to sit down with experts and discuss topics that are important to them. Experts on hot topics like zero-trust security, data lakes, machine learning and analytics workloads, and open source projects will be hanging around just so you can get your questions answered.</p> <p>Within the Hack Shack, you’ll engage with members of the HPE Developer Community in topical meetups. These 30-minute sessions are informal gatherings where you can listen to various subject matter experts (SMEs) and then engage in further discussion on the topic as a group. SMEs will also hang out for a bit after each session in case you want to delve into further detail regarding your own specific needs. (Stay tuned for our next blog post where we’ll get into details on these meetup sessions.)</p> <p>There will also an opportunity for you to sit down with the HPE UX design team to help them improve the user experience across HPE products and services. Share your opinions and make your voice heard by participating in a variety of user research activities that will inform the design and direction of HPE products and services.</p> <h2>Compete in games and win prizes!</h2> <p>While many come to the Hack Shack to connect with subject matter experts and get their questions answered, many also come to enjoy a little competitive fun with their colleagues. We’ll have a fussball table out on the front lawn, a Jenga and cornhole game in the backyard, chairs to relax in, and a Shield console to play some cool video games.</p> <p>In addition to these popular games, the HPE Developer Community team has developed a couple of other diversions for you. The first one is our Treasure Hunt. Designed to familiarize players with the HPE Developer web portal, this scavenger-hunt style challenge encourages you to check out all the resources that are available on the site, including the <a href="https://developer.hpe.com/hackshack/">virtual Hack Shack</a>. To be sure, there is treasure to be had, so be sure to check out the** <a href="https://developer.hpe.com/hackshack/hpediscover2022-treasurehunt-terms-conditions/">Terms &#x26; Conditions</a>** for details.</p> <p>We also designed five role-based hands-on challenges for you to take. Each onsite participant has the opportunity to go home with some nifty swag. Six of those participants will walk away with a CanaKit Raspberry Pi 4 Extreme kit after having successfully completed one of the challenges and correctly answered the associated quiz. Grand prize winners will be announced at our Hack Shack celebration on Wednesday night. <strong>Note: You must be present to win. Please see the <a href="https://developer.hpe.com/hackshack/hpediscover2022-swchallenges-terms-conditions/">Terms &#x26; Conditions</a> for details.</strong></p> <p>Are you ready to take these challenges?</p> <ul> <li><strong>Cloud Architect Challenge - VM Desired State Management in HPE GreenLake</strong> – Your mission is to deploy a VM in HPE GreenLake using the open source configuration management tool, Terraform. For this challenge, you’ll describe the desired state of your environment and use Terraform to analyze and build the necessary infrastructure artifacts. You’ll be provided with everything you need in a nice and friendly Jupyter Notebook environment with little to no code to write. Take an hour to experience Infrastructure-as-Code on HPE GreenLake and earn a chance to win a prize.</li> <li><strong>Open Source Advocate Challenge - Play with Python or Discover Ansible - Your Choice!</strong> – Feeling competitive? Expand your Open Source skills with a chance to win cool prizes. In this challenge, you will be tasked with completing one of two popular HPE DEV Workshops-on-Demand. Choose from <em>Python 101 - A simple introduction to Python programming language</em> or <em>Ansible 101 - Introduction to Ansible concepts</em> and respond correctly to the quiz for a chance to win a prize. An hour should be enough for you to complete this challenge!</li> <li><strong>Developer Challenge - Building Modern Software with Zero Trust Security</strong> – In today’s highly distributed modern software environments, security is a major concern. Who do you trust? In this challenge, choose one of two workshops, <em>SPIFFE – SPIRE 101 – An introduction to SPIFFE server and SPIRE agent security concepts</em> or <em>Creating a Zero Trust Model for Microservices Architectures with SPIRE and Envoy</em>, to understand, in less than an hour, how open source projects SPIFFE and SPIRE enable zero trust security at the heart of your solution and compete for a prize.</li> <li><strong>ML Engineer Challenge - Deep Learning Model Training at Scale with Determined</strong> – Deep learning at scale is difficult, right? Explore the fundamentals of Determined, the open-source deep learning training platform, to learn how it can help. Take this challenge and respond correctly to a quiz to try and win a cool prize. In this challenge, you will train a TensorFlow model in Determined using one GPU, and scale up your training across multiple GPUs using distributed training, while finding accurate models faster using state-of-the-art hyperparameter search methods.</li> <li><strong>Data Scientist Challenge - Finding the Data You Didn't Know You Needed</strong> – In this challenge, you’ll get to see how Project Data Map can help you discover new and meaningful datasets that enhance your model building experience, all whilst keeping track of the datasets you know and love, so next time you don’t have to go digging through old notebooks to find them! You’ll even learn how you can share them with your classmates or trade them for valuable tokens! This challenge should take out about an hour and gives you a chance to win an awesome prize.</li> </ul> <h2>Come party with us</h2> <p>One of the most anticipated events at HPE Discover is the Hack Shack celebration. Planned for Wednesday evening, we’ll be serving refreshments and hosting a very special speaker who will be presenting the CanaKit Raspberry Pi sets to our lucky winners of the role-based challenges.</p> <p>Stay tuned! We’ll be publishing more details on the meetup sessions and the schedule in an upcoming <a href="https://developer.hpe.com/blog">HPE Developer Blog post</a>. You can also refer to the <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1049&#x26;locale=en_US">HPE Discover 2022 Edge-To-Cloud conference catalog</a> for details on HPE Developer sessions.</p>https://developer.hpe.com/deep-learning-model-training-–-a-first-time-user’s-experience-with-determined-part-1/https://developer.hpe.com/deep-learning-model-training-–-a-first-time-user’s-experience-with-determined-part-1/Thu, 14 Apr 2022 17:21:31 GMT<p><a href="https://github.com/determined-ai/determined">Determined</a> is an open-source deep learning training platform that helps data science teams train models more quickly, easily share GPU resources, and collaborate more effectively. The open-source version of Determined can be deployed on-premises in your data center, on any hardware, on Kubernetes, or in public clouds – wherever GPU resources are available to obtain the full benefit of Determined.</p> <p>In this two-part blog series, I’ll share my experience as a first-time user of Determined. This blog series aims to provide a high-level overview of the basic concepts behind Determined and why you should consider it if you find doing deep learning at scale a bit challenging.</p> <p>In this first part, I’ll put on my IT Operations manager’s hat and explain how I deploy Determined on a Kubernetes cluster in an on-premises HPE Ezmeral Runtime Enterprise deployment. In this instance, it will enable my organization’s data science team to quickly try out Determined and assess its capabilities for their data science work.</p> <center><img src="/img/detai-high-levl-architecture-thumbnail-v2.png" width="543" height="708" alt="High Level architecture diagram" title="High Level architecture diagram"></center> <p><a href="https://developer.hpe.com/blog/deep-learning-model-training-%E2%80%93-a-first-time-user%E2%80%99s-experience-with-determined-%E2%80%93-part-2/">In the second part of this series</a>, I'll wear my data scientist/ML engineer hat as a member of a larger data science team that wants to get started with Determined and explore some of its fundamental concepts and features. I’ll review how to train neural network models using one or more GPUs with distributed training, and advanced functionality such as state-of-the-art hyperparameter search to improve model accuracy and find the best version of a model.</p> <h2>Determined AI</h2> <p>Determined is an open-source platform built to accelerate deep learning (DL) model development and experimentation for data science teams at scale, handling the load easily as teams, clusters and data sets all increase in size. Teams can use Determined to build, train, and optimize their deep learning models while easily sharing GPU compute resources. Determined AI was acquired by HPE in June 2021 and now operates as a part of the High Performance Computing (HPC) and Artificial Intelligence (AI) business unit.</p> <p>Determined provides the APIs, a command line interface (CLI), a web user interface, and tools for accelerating model experiments with integrated capabilities such as distributing training and automatic model tuning with hyperparameter search, also known as hyperparameter optimization (HPO).</p> <h2>HPE Ezmeral Runtime Enterprise</h2> <p>Built from the ground up to be open and run in hybrid environment, <a href="https://developer.hpe.com/platform/hpe-ezmeral-runtime/home/">HPE Ezmeral Runtime Enterprise</a> provides a secure, enterprise-grade platform designed to run both cloud-native and non-cloud-native applications at scale. It provides an integrated data fabric, multi-cluster Kubernetes management, enterprise-grade security and multi-tenancy capabilities.</p> <p>HPE Ezmeral Runtime Enterprise with the pre-integrated HPE Ezmeral Data Fabric provides all the networking, compute, and storage resources needed to run the Determined open-source platform on premises on Kubernetes.</p> <h2>Components of my Determined deployment</h2> <center><img src="/img/detai-lab-environment-architecture-v2.png" width="1332" height="725" alt="Figure1 Determined High Level Architecture on Kubernetes" title="Figure1 Determined High Level Architecture on Kubernetes"></center> <p>As the figure above indicates, my experimental deployment of Determined consists of:</p> <ul> <li> <p>A Kubernetes cluster, managed by HPE Ezmeral Runtime Enterprise, with a set of worker nodes with <a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">NVIDIA GPUs support enabled</a> (1 GPU device per worker node in my Kubernetes cluster).</p> </li> <li> <p>A Determined <strong>Master</strong>, which is attached to a <strong>PostgreSQL</strong> database. The Determined Master and Database run as containers, each within a Kubernetes POD, in the worker nodes of the Kubernetes cluster.</p> <ul> <li> <p>The Master hosts the interface service endpoint that clients use to communicate with Determined through a CLI, WebUI, and APIs.</p> </li> <li> <p>The Master schedules tasks and brings up PODs on Kubernetes worker nodes to run tasks on demand. For example, the model training tasks and auxiliary tasks (TensorBoard, Notebook).</p> </li> <li> <p>As training tasks execute, the Master maintains communication with training task PODs and saves training model metadata, like the training and validation metrics received from the training tasks, as well as the state of the tasks, in the PostgreSQL database, for model experiment tracking and analysis.</p> </li> </ul> </li> <li> <p>An ingress gateway makes the Master's interface service endpoint reachable from outside the Kubernetes cluster.</p> </li> <li> <p>A persistent storage volume for experiment tracking by logging the model’s metadata information, such as hyperparameters, the training and validation metrics, logs, date/time, on the PostgreSQL database.</p> </li> <li> <p>A volume shared across the Kubernetes worker nodes. The shared file system is needed to store the <strong>model artifacts</strong>, such as model code and model <strong>checkpoint</strong> files. The model checkpoint files are saved versions of the validated models that data science teams can access later for testing and analysis. This makes them available to a deployment or serving solution such as Seldon core. The shared file system can also be used by Determined to store the model datasets on which the model is trained by the training tasks.</p> </li> </ul> <h2>Installing Determined on a Kubernetes cluster managed by HPE Ezmeral Runtime Enterprise</h2> <p>The open-source version of Determined is available as a Helm chart and can be installed on a Kubernetes cluster running on HPE Ezmeral Runtime Enterprise. As such, I download the chart and modify the chart <em>values.yaml</em> file (which I’ll explain in this section) before installation of the Helm chart in my Kubernetes cluster.</p> <p>Before deploying the Helm chart, an important aspect to understand is how to connect Determined to a shared storage volume. For this, I need to create a new tenant named <strong>determinedai</strong> on HPE Ezmeral Runtime Enterprise for my Kubernetes cluster. This tenant serves as a Kubernetes cluster "namespace". Each tenant created in HPE Ezmeral Runtime Enterprise is automatically provisioned with a tenant’s shared storage volume on the pre-integrated HPE Ezmeral Data Fabric cluster located at <strong>/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount</strong>. The tenant’s shared storage volume is then automatically mounted on each Kubernetes cluster’s host on the path <strong>/opt/bluedata/mapr/mnt</strong>. This enables Determined to connect to the shared storage <em>/opt/bluedata/mapr/mnt/&#x3C;DataFabric-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/</em> for accessing the training and validation datasets, and for storing model artifacts.</p> <p>Furthermore, some aspects of Helm chart deployment must be configured before installing Determined on Kubernetes. Although most of the default Helm chart configuration settings are suitable for getting started with Determined on Kubernetes, some parameters must be configured in the chart <em>values.yaml</em> file to match the designated Kubernetes cluster deployment and available compute, storage and network resources such as:</p> <ul> <li> <p>The use of the Kubernetes <em>NodePort</em> service type to expose the Determined Master service endpoint outside the Kubernetes cluster,</p> </li> <li> <p>The shared storage volume path to use to save validated model files and checkpoints for fault tolerance,</p> </li> <li> <p>The amount of GPU resources (known as <em><strong>slot</strong></em>) available on the Kubernetes worker hosts. In my Determined deployment I have 1 GPU per worker host,</p> </li> <li> <p>The <a href="https://docs.determined.ai/latest/concepts/scheduling.html">advanced scheduler</a> to use for large Kubernetes clusters with multiple GPUs per worker host. For my experimental Determined deployment, as I only have 1 GPU per worker host, it is recommended to let Determined use the <strong>default Kubernetes scheduler</strong> to schedule training tasks,</p> </li> <li> <p>The Determined <em>Admin</em> and <em>Determined</em> default user account passwords,</p> </li> <li> <p>The friendly name for Determined deployment.</p> </li> </ul> <p>For more information about the configuration options for the Helm Chart deployment, see the <a href="https://docs.determined.ai/latest/sysadmin-deploy-on-k8s/install-on-kubernetes.html">installation guide documentation</a>.</p> <p>In my Determined deployment on Kubernetes, the following aspects of the Determined Helm chart deployment configuration is set in the chart <em>values.yaml</em> file as shown below. Other configuration settings are set to their default values.</p> <pre><code class="language-Markdown">useNodePortForMaster: true checkpointStorage: type: shared_fs hostPath: /opt/bluedata/mapr/mnt/&#x3C;DF-clusterName>/exthcp/tenant-&#x3C;ID>/fsmount/checkpoints maxSlotsPerPod: 1 clusterName: stagingdetai defaultPassword: &#x3C;myPassword> </code></pre> <p>With the namespace created, the kubeconfig file for the Kubernetes cluster sourced in my Linux workstation, and the Helm chart deployment configuration files in hand, I can deploy Determined software on the Kubernetes namespace <em>determinedai</em> using the following command:</p> <pre><code class="language-bash">helm install stagingdetai &#x3C;relative path to determined-helm-chart repository> –n determinedai [--dry-run] </code></pre> <blockquote> <p>Note: I recommend first using the <code>--dry-run</code> flag to validate and verify the chart manifest before actual Helm chart deployment.</p> </blockquote> <p>Upon completion, I can use the following commands to check the status of the Helm chart deployment for my Determined instance:</p> <pre><code class="language-bash">helm list -n determinedai helm status stagingdetai -n determinedai </code></pre> <p><img src="/img/determined-helm-install-status.png" alt="Helm Chart deployment status" title="Helm Chart deployment status"></p> <p>At the time of the Determined installation on the Kubernetes cluster, an instance of the <strong>Determined Master</strong> and a <strong>PostgreSQL database</strong> are deployed in the Kubernetes cluster. Using the kubectl command below allows me to check the resources that are deployed on the Kubernetes cluster:</p> <pre><code class="language-bash">kubectl get pod,services –n determinedai </code></pre> <p><img src="/img/determined-pod-svc.png" alt=""></p> <p>As shown in the above image, these components run as a container within a Kubernetes POD. Service endpoints for the Determined’s Master and the Database services are also deployed. The Determined Master service endpoint is a NodePort service that enables HPE Ezmeral Runtime Enterprise to expose that service outside the Kubernetes cluster through its <strong>ingress gateway.</strong></p> <h2>Installing the Determined Command Line Interface</h2> <p>As mentioned earlier, Determined provides a web user interface (WebUI), APIs (REST API and Python API), and a command line interface (CLI) tool to interact with the system. The CLI is the most common tool used by data scientists and ML engineers to interact with Determined, especially for launching deep learning model training tasks on Determined. The WebUI is mainly used to monitor the progress of model experiments and training tasks, and visualize the model training performance in graphs.</p> <p>The Determined CLI is distributed as a Python package. I need Python 3.6 or later installed on my Linux workstation along with the latest version of <code>pip</code>. I can use the following command to install the CLI tool on my workstation:</p> <pre><code class="language-bash">#install latest version of pip if needed python3 -m pip install --upgrade pip #install the Determined CLI pip install determined </code></pre> <h2>Using the Determined Command Line Interface</h2> <p>I am now ready to enter Determined CLI commands. All commands begin with <strong>det</strong> and any CLI command has the form:</p> <p><em><strong>det [-m &#x3C;det_master_URL_or_IP:port>] &#x3C;command_argument> &#x3C;action_verb> [-h]</strong></em></p> <p>The Master service endpoint is referenced using the -m flag to specify the URL of the Determined Master that the CLI connects to. Instead of specifying the <em><strong>-m</strong></em> flag in every command, I can define an environmental variable, <em><strong>DET_MASTER</strong></em>, that points to the Determined Master service endpoint URL.</p> <blockquote> <p>Note: The help flag [-h] can be used to learn more about CLI options.</p> </blockquote> <p>To use and interact with Determined using the CLI, I need to tell the CLI where the Determined Master service is running. To do so, I first use the <code>kubectl describe service</code> command below and look at the <strong>Annotations</strong> section to get the <strong>ingress gateway URL</strong> and <strong>network port</strong> provided by HPE Ezmeral Runtime Enterprise for the Master service of the Determined deployment:</p> <pre><code class="language-bash">kubectl describe service determined-master-service-stagingdetai -n determinedai </code></pre> <p><img src="/img/determined-master-endpoint-ofuscated.png" alt="Ingress Gateway URL for the Determined Master endpoint service" title="Ingress Gateway URL for the Determined Master endpoint service"></p> <p>In this example, the network port is 13047.</p> <p>I now need to export on my workstation the <strong>DET_MASTER</strong> environmental variable, which points to that URL and port:</p> <pre><code class="language-bash">export DET_MASTER=http://gateway2.&#x3C;mydomain.name>:13047 </code></pre> <p>Finally, I need to authenticate as a Determined user. By default, at the time of the Determined installation, two user accounts are created: <em><strong>Admin</strong></em> an administrator account, and <em><strong>Determined</strong></em> a non-privileged user account with the password specified in the Helm chart <em>values.yaml</em> configuration file. Using the following command allows me to authenticate as an admin user. I will be prompted by the CLI for the password.</p> <pre><code class="language-bash">#format: det user login &#x3C;username> det user login admin </code></pre> <h2>Creating User accounts for the data science team</h2> <p>Determined is designed for data science teams. As such, I’d recommend creating a user account for each member of the team who wants to use Determined. This provides the organizational benefits of associating each Determined entity, such as model experiments and associated training tasks, with the user who created it.</p> <p>I have experienced user account creation using both the CLI and the REST API. The <em><strong>Admin</strong></em> privileged user account must be used to create a user account and set the newly created user account password.</p> <h3>Using the Det CLI</h3> <p>Once logged in as Admin user on Determined, I can use the following command to create a test user account. First, I create the user account. The newly created user account has a blank password by default. Then, I set the password for the user account using the second command, which prompts me for the password and password confirmation.</p> <pre><code class="language-bash"># Create the user account det user create &#x3C;username> # Set the password for the user account det user change-password &#x3C;target-username> </code></pre> <h3>Using the REST API for a programmatic approach</h3> <p>Unlike the DET CLI, which requires keyboard input for the password, a programmatic approach to create user accounts might be more appropriate depending on the organization’s use case. Determined is also REST API enabled. The Determined REST API documentation is available <a href="https://docs.determined.ai/latest/rest-api/index.html">here</a>.</p> <p>Below is the sequence of REST API calls I can use to create a new user account (testuser1) in Determined and to set the password, all using code. You can see how I use <em><strong>cURL</strong></em> as an HTTP client to interact with Determined through its REST API.</p> <p>I first need to authenticate as Admin user to Determined and save the authentication token (bearer token) for subsequent REST API calls:</p> <pre><code class="language-bash">detMaster=http://gateway2.&#x3C;mydomain.name>:13047 # Authenticate as admin user and get the authentication token for subsequent calls: token=$(curl -i -s -X 'POST' \ "${detMaster}/api/v1/auth/login" \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "username": "admin", "password": "&#x3C;MyPassword>" }' | grep token | awk '{print $1}' | tr -d '\r') # Extract token value and remove trailing quotes MyToken=$(echo $token | cut -d':' -f 2 | cut -d',' -f 1 | tr -d '"') </code></pre> <p>I then create a non-admin user account using the Admin access token as the bearer token authentication:</p> <pre><code class="language-bash"># Create a new user account “testuser1” curl -X 'POST' \ "${detMaster}/api/v1/users" \ -H 'accept: application/json' \ -H "Authorization: Bearer $MyToken" \ -d '{ "user": { "username": "testuser1", "admin": false, "active": true }, "password": "&#x3C;UserPassword>" }' </code></pre> <p>Changing the password for an existing user account is a two-step process. You must first obtain the userID of the user before changing the password:</p> <pre><code class="language-bash"># Fetch the user ID for user "testuser1" detUserId=$(curl -s -X 'GET' \ "${detMaster}/api/v1/users" \ -H 'accept: application/json' \ -H "Authorization: Bearer $MyToken"| jq '.users[] | select(.username=="testuser1") | .id' | tr -d '"') echo " the determined AI user ID for user testuser1 is : $detUserId" # Set password for the user account “testuser1” curl -X 'POST' \ "${detMaster}/api/v1/users/${detUserId}/password" \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $MyToken" \ -d '"&#x3C;userPassword>"' </code></pre> <blockquote> <p>Note: The open-source version of Determined does not provide user access control features in case you have multiple data science teams (i.e.: multiple tenants). Open-source Determined uses a local user directory as a convenient method to show the entity created by the logged in users. However, the open-source version makes any entity (experiments, tasks) visible to all users, regardless of who created it. This can be a challenge for enterprises that need to keep strong model governance for audit purposes. The enterprise version of the open-source Determined product, <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> addresses this limitation.</p> </blockquote> <h2>Checking connectivity to the WebUI using the newly created user account</h2> <p>A good method to verify that a member of the data science team can interact with Determined is to test the connectivity to the WebUI. The WebUI is available on the same service endpoint URL as the CLI. Using my browser, I connect to the Master service URL and verify that I am prompted to login to the WebUI as shown in the following figure:</p> <p><img src="/img/determined-webui-login.png" alt="Determined WebUI login page" title="Determined WebUI login page"></p> <p>Upon successful login, I land on the <em><strong>dashboard</strong></em> below. You’ll learn more about the WebUI in my second blog post in this series.</p> <blockquote> <p>Note: At the bottom left of the menu bar, you can see that having access to a running Determined instance allows me to navigate a Swagger UI version of the REST API in an interactive fashion.</p> </blockquote> <p><img src="/img/determined-webui-dashboard.png" alt="Determined WebUi Dashboard" title="Determined WebUi Dashboard"></p> <p>That’s it! Everything is set. I am now ready to put on my data scientist hat, go and use Determined to train and tune a deep learning model in Determined using the CLI, visualize training results using the WebUI, and load and test models by making inferences.</p> <h2>Summary</h2> <p>As you can see, using my IT operations manager’s hat, I deployed Determined on a Kubernetes cluster running on HPE Ezmeral Runtime Enterprise, which provides all the components needed to run Determined: a task scheduler such as Kubernetes, a namespace, multi-tenancy, an ingress gateway, persistent storage for experiment tracking, and a shared file system for storing model artifacts and datasets.</p> <p><a href="https://developer.hpe.com/blog/deep-learning-model-training-%E2%80%93-a-first-time-user%E2%80%99s-experience-with-determined-%E2%80%93-part-2/">In the second post in this series</a>, I will walk through training a TensorFlow Keras model in Determined using features such as distributed training and automatic model tuning with hyperparameter search.</p> <p>You can subscribe for updates from the HPE Dev Community by subscribing to our <a href="https://developer.hpe.com/community">newsletter</a>. I was able to write this post by joining and receiving help from the <a href="https://join.slack.com/t/determined-community/shared_invite/zt-cnj7802v-KcVbaUrIzQOwmkmY7gP0Ew">Determined Community Slack</a>, which you can also do .You can begin training models with Determined today by visiting the <a href="https://github.com/determined-ai/determined">Determined project on GitHub</a>.</p>https://developer.hpe.com/2022-April-05/https://developer.hpe.com/2022-April-05/Tue, 05 Apr 2022 13:00:00 GMThttps://developer.hpe.com/highlighting-key-features-of-hpe-ezmeral-runtime-enterprise-release-5-4/https://developer.hpe.com/highlighting-key-features-of-hpe-ezmeral-runtime-enterprise-release-5-4/Thu, 31 Mar 2022 18:18:53 GMT<p>It’s a cliché to say that data is an important asset to organizations. Organizations of all sizes are looking at data as a strategic asset to help them create a digital advantage. From delivering frictionless customer experiences and fraud detection, to accelerating breakthrough innovations in healthcare and personal medicine, enterprises are moving into the digital economy very rapidly. Data-first artificial intelligence (AI) and machine learning (ML) initiatives are seen as a strategic enabler and a top investment priority for these organizations. Yet, 80-85% of Enterprises* find it difficult to move their AI/ML initiatives beyond the experimentation stage and thus fail to deliver any meaningful business outcomes.</p> <h6>* Gartner: Don't Stumble at the Last Mile: Leveraging MLOps and DataOps to Operationalize ML and AI</h6> <h2>Common operational challenges</h2> <p>I speak to a lot of customers across different industries and verticals. It is fascinating to see that they all have many challenges in common. Some that I hear repeatedly are:</p> <h4>Fragmented data and governance controls</h4> <p>It is true that enterprises have large amounts of data at their disposal. However, the data is often silo’ed, with inconsistent governance controls, which makes it difficult for the consumers of data, namely, Data Engineers, Data Analysts and Data Scientists, to access the data in a timely, safe and self-service fashion.</p> <h4>Enabling modern analytics at scale</h4> <p>Enterprises are looking beyond Hadoop to modernize their analytics stacks, considering applications such as Apache Spark™ to run on Kubernetes. While the new approach yields several benefits, including self-service access to Spark clusters, elastic and independent scaling of compute and storage, rolling upgrades to newer Spark versions, etc., the transition from the legacy world is not that simple. There is a learning curve associated with operationalizing Kubernetes, putting the right security and access controls in place, establishing data connectivity to many different data sources, etc., just to name a few considerations.</p> <p>In addition, every persona has a certain preference for using their tool of choice to interact with Spark. Then, there is the data consistency problem. While data lakes are cost efficient, ensuring the data consistency and data reliability that is required for AI/ML applications is a huge challenge.</p> <h4>Runtime environment consistency</h4> <p>Deploying Kubernetes has become a lot simpler today, which leads to different development teams easily spinning up new clusters. However, these clusters may not be in compliance or meet enterprise security standards. The question is, how do you make it easier for application teams to self-provision Kubernetes clusters and, at the same time, enforce immutable security controls that are centrally governed?</p> <h2>HPE Ezmeral Runtime Enterprise – Modern approach to delivering end-to-end analytics solutions at scale</h2> <p><a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a> is a turnkey platform designed to support clients’ AI, analytics and data needs, as well as help them innovate faster. Built on the foundation of Kubernetes, the Ezmeral Runtime Enterprise platform brings simplicity to operationalizing the ML lifecycle through <a href="https://www.hpe.com/us/en/solutions/ezmeral-machine-learning-operations.html">HPE Ezmeral ML Ops </a>and <a href="https://docs.containerplatform.hpe.com/54/reference/HPE_Ezmeral_Runtime_Analytics_for_Spark.html">HPE Ezmeral Runtime Enterprise Analytics for Apache Spark</a> at the edge, on-premises or on public clouds with an infrastructure agnostic approach to deploying and managing the analytics stack.</p> <p>HPE Ezmeral Runtime Enterprise natively includes <em><strong><a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">Ezmeral Data Fabric</a></strong></em> – a modern data platform with global namespace and built-in capabilities to store files, objects, event stream store, databases – binary and JSON db – and provide access to data via various protocols including NFS, HDFS, S3 and POSIX. HPE Ezmeral Runtime Enterprise also provides the flexibility to connect to existing data lakes and data sources such as HDFS, S3 Object stores and third-party storage systems, to enable analytical applications running on K8s clusters remote access to data, thus enabling compute-storage separation.</p> <p>The latest 5.4 release of HPE Ezmeral Runtime Enterprise includes quite a few new capabilities, so without further delay, let me highlight these new features.</p> <h4>HPE Ezmeral Unified Analytics</h4> <p><a href="https://www.hpe.com/us/en/software/ezmeral-unified-analytics.html">HPE Ezmeral Unified Analytics</a> is a modern approach to delivering elastic open-source based Spark clusters combined with the Delta Lake for a data lakehouse architecture solution. More on this later. But before that, let me walk you through some common challenges that I hear repeatedly from both Spark administrators and users (think Data Engineers, Data Analysts, and Data Scientists):</p> <p>Challenges of a Spark administrator:</p> <ul> <li>Provision multiple versions of Spark to cater to different application teams’ needs</li> <li>Consistent user authentication and authorization from application to downstream data layer</li> <li>Efficient use of resources, including expensive GPU accelerators</li> <li>Ability to connect to diverse sets of data sources</li> <li>Enterprise support with SLAs</li> </ul> <p>Spark users, on the other hand, have their own concerns:</p> <ul> <li>Ability to use REST, CLI, or scripted workflows for Spark job submissions</li> <li>Desire to use CPU and/or GPU for Spark jobs, including for dynamic scaling</li> <li>Multiple storage options – S3, MapR FS, Hadoop data lakes access</li> <li>Support for real-time and batch workloads at scale</li> </ul> <h2>How does HPE Ezmeral Unified Analytics solve these challenges?</h2> <p>Let me summarize the features and benefits here.</p> <h4>Multi-version Spark support</h4> <p>One-click deployment of a <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes-applications/spark/Spark_Overview.html">Spark Operator</a> on Kubernetes cluster. This single Spark Operator manages Spark version 2.4.7 and Spark version 3.1.2 and enables users to run multi-version Spark applications. This also includes Spark images with support for multiple languages including Python, R, Java, Scala, etc.</p> <h4>Interactive Spark experience through Livy</h4> <p>This is for personas that are adept at using REST APIs to interact with Spark. HPE has included <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes-applications/spark/submit_spark_application_using_livy.html">Livy support</a> for both Spark 2 and 3 versions. This allows personas, like Data Scientists, to interact with Spark from their Jupyter notebooks. You simply launch the Livy server with one click, connect to the server endpoint, and start submitting Spark jobs.</p> <h4>Interaction with BI tools</h4> <p>The <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes-applications/spark/installing-and-configuring-spark-thrift-server.html">Thrift Server</a> component provides a JDBC/ODBC interface for enabling Data Analysts to interact with Spark from BI tools such as Tableau, PowerBI, Qlik, etc.</p> <h4>Shared Hive metastore service</h4> <p>The <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes-applications/spark/install_configure_hive_metastore.html">Hive metastore service</a> enables multiple Spark applications running across different tenants or K8s clusters to share common metadata/schema.</p> <h4>Intuitive Job submission interface</h4> <p>This provides a <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes-applications/spark/managing_spark_applications_using_gui.html">wizard-driven UI</a> for tenant users to submit their Spark applications. HPE Ezmeral Runtime Enterprise also allows users to bring their YAML files and upload them to UI to run their Spark job immediately, or schedule it to run at a specified time interval. Users can monitor the status of running jobs, as well as look at the events and log files all from an intuitive user interface.</p> <h4>Airflow DAGs</h4> <p>Data engineers responsible for building data pipelines often resort to automation provided by workflow orchestration tools, like <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes/kubernetes-administrator/airflow/airflow_installation.html">Airflow</a>, to sequence a set of tasks using Directed Acyclic Graph (DAG). HPE Ezmeral Runtime Enterprise includes an Airflow operator - enabled with one click during Kubernetes cluster creation. Once enabled, users create and submit Spark jobs using DAGs. Users can store their DAGs in Git, version control it and configure HPE Ezmeral Runtime Enterprise to run those from the Git repository.</p> <h4>Delta Lake support</h4> <p><a href="https://docs.containerplatform.hpe.com/54/reference/Whats_New_in_Version.html?hl=delta%2Clake">Delta Lake</a> is a storage architecture that enables a lakehouse architecture on Spark. It combines the best of data warehouse characteristics, such as ACID properties, schema enforcement, and versioning features, to unstructured and semi-structured data stored in a massively scalable and cost-optimized data lake. With this, you get the data consistency and data reliability that are key requirements in environments with multiple concurrent data pipelines acting on data. The HPE Ezmeral Unified Analytics stack includes open delta libraries by default, enabling applications to read and write data using the delta format in parquet files on HPE Ezmeral Data Fabric or on any S3-compatible object stores.</p> <h4>Read and write data from different data sources</h4> <p>HPE Ezmeral Runtime Enterprise provides <a href="https://docs.containerplatform.hpe.com/54/reference/universal-concepts/About_DataTaps.html">DataTap</a> to connect Spark applications to remote HDFS data sources, thereby bringing compute closer to data. You can also connect Spark apps to any S3-compatible object store to access and store data.</p> <h4>Enterprise-grade Security</h4> <p>Enterprise-grade security secures access to Spark clusters. Users are authenticated against AD/LDAP, and authenticated users are allowed to perform operations on Spark clusters based on the RBACs. The identity of authenticated/authorized user is preserved during data access.</p> <p>Hopefully, this gives you enough information to whet your appetite about the HPE Ezmeral Unified Analytics solution.</p> <h2>Let’s move on to HPE Ezmeral ML Ops</h2> <p><a href="https://docs.containerplatform.hpe.com/54/reference/universal-concepts/About_HPE_Ezmeral_ML_Ops.html">HPE Ezmeral MLOps</a> is an end-to-end Machine Learning lifecycle management platform for building, training and operationalizing ML models. Here are some of the key features of HPE Ezmeral ML Ops in release 5.4.</p> <h4>Introducing support for KubeFlow (KF) 1.3</h4> <p>Single click deployment of <a href="https://docs.containerplatform.hpe.com/54/reference/kubernetes/kubernetes-administrator/kubeflow/kubeflow.html">KF 1.3</a>. This includes integration with MLFlow for model management, experiment tracking and improved collaboration among data scientists operating on projects to share ML models and model metadata.</p> <h4>Enhancing data scientist productivity</h4> <p><strong>Kale extensions:</strong> Kale stands for KubeFlow Automated Pipeline Engine. This is an add-on to Jupyter notebooks that provides a simple way for data scientists to annotate notebook cells, as opposed to writing complex code to craft KF pipelines.</p> <p><strong>Prebuilt library of functions:</strong> The release adds “ezmllib”, which includes several functions that makes it easy for code-first data scientists to interact with Spark, KubeFlow, ML Model registry, etc. from their notebooks and simplifies the coding experience.</p> <h4>Accelerating Model training and inferencing</h4> <p>HPE Ezmeral ML Ops is adding support for <a href="https://docs.containerplatform.hpe.com/54/reference/nvidia-gpu-support/nvidia-gpus.html">NVIDIA MIG-enabled GPUs</a> to be leveraged by data scientists in their notebooks for model building and training. ML/DL frameworks such as Tensorflow and Pytorch have been updated with the right CUDA (Compute Unified Device Architecture) libraries to take advantage of fractionalized GPUs providing cost-efficiency and secure isolation of workloads.</p> <h2>HPE Ezmeral Runtime Control plane – The wizard behind the curtain</h2> <p>The HPE Ezmeral Runtime Enterprise control plane is responsible for lifecycle management of on-premises K8s clusters; orchestrating cloud provider-managed K8s clusters – EKS, AKS and GKE; managing user identity and access controls; and securing ingress and access to service endpoints, to name a few of its capabilities.</p> <h4>New features in this release include:</h4> <p><strong>Fractional GPU support for NVIDIA MIG-enabled devices:</strong> Multi-instance GPU (MIG) provides a way to partition GPU resources into slices to improve GPU utilization, as well as offer workload security through fault isolation. HPE Ezmeral Runtime Enterprise has been enhanced to set up MIG profiles (single and mixed strategy) and expose those to workloads (Tensorflow, Pytorch, etc.) via CUDA libraries. Different classes of GPU devices (V, P, T, A classes of NVIDIA GPUs) can coexist on the Kubernetes hosts and HPE Ezmeral Runtime Enterprise can orchestrate workload placement on the right GPU device. In addition, HPE Ezmeral Runtime Enterprise automatically persists these MIG profiles to survive host reboots.</p> <p><strong>Day 2 enhancements for Centralized Policy management:</strong> The previous 5.3 release included a GitOps-based centralized policy management feature to declaratively construct K8s clusters that conformed to organizations policies. The policy management feature also brought in drift management and reconciliation capabilities to create immutable clusters. The 5.4 release offers a simpler way for admins to visualize the effects of cluster policies on a dashboard. Admins can quickly see which of the existing Kubernetes objects and workloads are in non-compliance with the policy set or the new objects that are being blocked from creation due to the enforced policies. This way, admins can make better decisions about the effects of policies and fine tune them as needed.</p> <h2>Where to go from here</h2> <p>As you can tell, there are quite a few new features and capabilities included in the HPE Ezmeral Runtime Enterprise 5.4 release and we are excited to share them with you.</p> <p>To learn more about the HPE Ezmeral products, please contact the HPE Sales team, or visit <a href="https://www.hpe.com/us/en/software.html">www.hpe.com/ezmeral</a> to explore how HPE Ezmeral Runtime Enterprise can accelerate your AI and Analytics journey. To learn more about this release, please refer to the <a href="https://docs.containerplatform.hpe.com/54/">documentation and release notes</a>. More information on <a href="https://developer.hpe.com/platform/hpe-ezmeral/home/">HPE Ezmeral</a> can be found on the HPE Developer portal. You can view other articles on HPE Ezmeral here on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a>.</p> <h6>Apache® and Apache Spark™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks.</h6>https://developer.hpe.com/file-objects-databases-and-streams-–-oh-my/https://developer.hpe.com/file-objects-databases-and-streams-–-oh-my/Thu, 31 Mar 2022 15:49:36 GMT<p>Analytics and machine learning (ML) have become key elements of every data-driven enterprise. The good news is that organizations have a lot of data at their disposal. The not-so-good news is that this data is distributed across data lakes, warehouses, edge, core, and data centers, making it more difficult for data engineers and scientists to do their jobs. HPE Ezmeral Data Fabric solves this problem by ingesting and cleansing hybrid data distributed across edge to cloud into a single logical infrastructure providing data teams with a unified data source that increases data integrity and trust in analytic insights.</p> <p><img src="/img/architecture-edf-7-1140x542.png" alt="Figure 1 Solution stack for HPE Ezmeral Data Fabric" title="Figure 1 Solution stack for HPE Ezmeral Data Fabric"></p> <h5><em>Figure 1 Solution stack for HPE Ezmeral Data Fabric</em></h5> <h2>Introducing HPE Ezmeral Data Fabric 7.0 with support for S3 compatible object store and FIPS compliance</h2> <p>The<a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-News-GA-for-the-Enterprise-Data-and-Unified/ba-p/7163244"> latest release of HPE Ezmeral Data Fabric</a> adds support for the fastest growing data type today, object. This release makes HPE Ezmeral Data Fabric the industry’s first data fabric to centralize files, objects, NoSQL databases and streams into a single data store, simplifying data lifecycle and operational management. Key features of this release include:</p> <ul> <li>A high-performance object store</li> <li>Federal Information Processing Standard (FIPS) compliance</li> <li>Security enhancements</li> <li>Dynamic data masking for JSON document databases</li> <li>Performance improvements using Remote Direct Memory Access (RDMA)</li> </ul> <p>Let’s examine these features in detail.</p> <h2>High-Performance Object Store</h2> <p>High-performance object store capabilities optimize all object sizes for both performance and storage efficiency. With HPE Ezmeral Data Fabric, multi-protocol access, using S3 API or standard interfaces, increases object availability to a wider set of traditional and cloud-native applications. As a part of this, HPE Ezmeral Data Fabric now allows you to:</p> <ul> <li>Be Write Once Read Many (WORM) compliant, helping organizations that must satisfy record keeping rules that ensure data is not tampered with in any possible way. This governance and compliance for data retention periods is very useful in financial and healthcare vertical markets.</li> <li>Integrate with MCS: With the new object store UI, administrators can manage accounts, buckets, objects, access policies, and users through a simple, intuitive interface.</li> <li>Archive data and build on-premises applications or migrate to cloud-native applications.</li> <li>Store media for operational use with fast retrieval, reduce costs of storing globally distributed media, such as music, video, and images.</li> <li>Run analytics on object data with tools like Apache Spark, Apache Drill, Presto, and <a href="https://docs-datafabric.mip.storage.hpecorp.net/70/MapROverview/query-s3-select.html">S3 Select</a> to gain valuable insights into customers, operations, or markets.</li> <li>Store ML model data and share the ML models in real-time with downstream applications.</li> <li>Publish S3 events to HPE Ezmeral Data Fabric streams for monitoring activity.</li> <li>Be compatible with opensource AWS S3 SDK and Minio SDK.</li> </ul> <h2>Federal Information Processing Standard Support</h2> <p>The Federal Information Processing Standard (FIPS) is a US government standard used to approve cryptographic modules. FIPS-validated products give users the assurance that data within the product is protected using cryptographic algorithms meeting the stringent guidelines and testing procedures established by the FIPS standard. FIPS was established by the National Institute of Standards and Technology (NIST) and defines critical security parameters that vendors must use for encryption. Products sold to the US government must meet FIPS validation criteria. In addition, there is a growing need by organizations processing sensitive data, such as banks, financial institutions, legal and medical institutions, to have the products that are FIPS 140-2/3 validated.</p> <p>The release of HPE Ezmeral Data Fabric includes <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/819">FIPS 140-2</a> Level 1 compliance to leverage operating systems that include FIPS 140-2 Level 1 certified cryptographic libraries provided by the user. It also includes support for Bouncy Castle Java FIPS API bundled with HPE Ezmeral Data Fabric, which runs on a compatible user-supplied JDK.</p> <p>In this release, HPE Ezmeral Data Fabric uses the open SSL cryptographic model distributed in operating systems supported by the core platform, which has obtained FIPS 140-2 Level 1 certification. It includes enhancements to the data fabric core platform that invokes FIPS 140-2 Level validated cryptography when FIPS mode is enabled and ensures that no sensitive data is stored in plain text. The following operating systems are supported: Red Hat 8.x, Ubuntu 18.04 and 20.04 and SLES 15 SP 2.</p> <p>Note: For all supported operating systems listed above, HPE Ezmeral Data Fabric uses the Java FIPS API from <a href="https://www.bouncycastle.org/">Bouncy Castle</a>, which has FIPS 140-2 Level 1 approval.</p> <h2>Security enhancements</h2> <p>This release includes a number of features that harden the HPE Ezmeral Data Fabric platform. The HPE Ezmeral Data Fabric platform now:</p> <ul> <li>Eliminates all clear-text passwords.</li> <li>Supports for the separation of passwords for key and trust stores</li> <li>Enhances data-fabric SASL to enable applications that are not cluster aware (such as data-fabric ecosystem components) to gain access to services in another cluster for which they have a ticket.</li> <li>Enhances the <em><strong>mrhsm</strong></em> utility. <em><strong>mrhsm</strong></em> is used to configure KMIP support and includes support for file-based key stores.</li> <li>Includes a new property <em><strong>(isFips)</strong></em> in the output of the <em><strong>maprcli node list</strong></em> command to indicate whether a particular node is FIPS-enabled.</li> <li>Offers a new ticket type: A new <em><strong>servicewithimpersonationandticket</strong></em> ticket type is introduced that allows some ticket holders to generate tickets subject to their impersonation authority.</li> <li>Has cross-cluster security enhancements: <em><strong>configure-crosscluster.sh</strong></em> script to automate establishing security between two clusters. New options are provided to specify trust store passwords. The file <em><strong>/etc/hadoop/ssl-server.xml</strong></em> no longer includes the trust store passwords.</li> </ul> <h2>Other enhancements</h2> <h4>Dynamic Data Masking for JSON document database</h4> <p>Dynamic data masking is the ability to apply variety of data masks in real-time on customer-designated fields in database queries, to hide sensitive data. This feature is perfect for Personal Identifiable Information (PII) or General Data Protection Regulation (GDPR) use cases.</p> <h4>Performance Improvements using Remote Direct Memory Access</h4> <p>Remote Direct Memory Access (RDMA) transfers data directly between user space process buffers on separate servers to bypass the Linux kernel and server CPU for increased performance and lower CPU utilization.</p> <h4>Ezmeral Ecosystem Pack (EEP)</h4> <p>Ezmeral Ecosystem Pack (EEP) provides certified open-source tools and engines that can be directly layered onto the data fabric reducing time spent integrating and configuring open-source tools for analytics. New to this release is Apache Airflow 2.2.1 along with significant updates to Apache Spark™ 3.2.0. Enhancements are also made to many other Hadoop components as well as Apache Drill 1.16.1.</p> <h2>Get to know HPE Ezmeral Data Fabric better</h2> <p>Developers, data engineers and scientists need high quality data to deliver the trusted insights businesses run on. Trusted insights only happen when distributed data is centralized into a single data layer that is optimized for analytics use cases. HPE Ezmeral Data Fabric delivers that high-performance, unified data layer without compromising security and governance that allows organizations to go to the next level by layering open-source engines and tools to deliver insights faster.</p> <p>For more information, please refer to the below listed resources. You can find other articles on HPE Ezmeral Data Fabric on the<a href="https://developer.hpe.com/blog"> HPE DEV blog</a>.</p> <ul> <li><a href="http://package.mapr.hpe.com/releases/v7.0.0">Core</a></li> <li><a href="http://package.mapr.hpe.com/releases/MEP/MEP-8.1.0/">EEP 8.1.0</a></li> <li><a href="https://docs.datafabric.hpe.com/70/home.html">Documentation and release notes</a></li> </ul>https://developer.hpe.com/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/https://developer.hpe.com/get-started-building-dscc-api-client-libraries-for-python-using-openapi-generator/Mon, 28 Mar 2022 16:57:29 GMT<p>In this tutorial, you will be introduced to the process that is required to convert the HPE Data Services Cloud Console public API specification in the OpenAPI 3.X definition to any client libraries from several popular programming languages. The goal of this conversion process is to achieve the agility of cloud-like operations where updates to console API client libraries are automatic and painless.</p> <p>Data Services Cloud Console public REST API is for customers looking to enhance their data-ops using the programmatic extensions from Data Services Cloud Console. Please see the <a href="https://developer.hpe.com/platform/data-services-cloud-console/home/">Data Services Cloud Console Platform page</a> for detailed information about the console benefits to customer's Data-Ops operation. Please see the <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/">Getting Started with the HPE Data Services Cloud Console Public REST API</a> for the authentication mechanism used to access the console API.</p> <p>The Data Services Cloud Console API definition is available for download in either YAML or JSON format from the console API website (<a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">US region</a>) as shown below:</p> <p><img src="/img/dscc-api-spec.png" alt="Data Services Cloud Console API download" title="Data Services Cloud Console API specification download"></p> <p>You can also download the console API YAML file using the following Unix command line:</p> <pre><code class="language-shell">$ wget https://console-us1.data.cloud.hpe.com/doc/api/v1/storage-api.yaml </code></pre> <p>The definition file contains the following information:</p> <ul> <li>A brief description of the API definition along with the version of the API in this file.</li> <li>The available regions with the base-URL that must be concatenated to every API path. For more information about each region, please see the <a href="https://developer.hpe.com/blog/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/">Getting Started with Data Services Cloud Console API</a> blog post.</li> <li>Summary tags for the content of this API definition.</li> </ul> <p><img src="/img/the-introduction-to-the-api-definition.png" alt="" title="Data Services Cloud Console Open API specification (YAML)"></p> <p>In addition, the definition file also contains:</p> <ul> <li>All available endpoints of the console resources, along with their HTTP headers, parameters, and the responses for each endpoint.</li> <li>The syntax of the HTTP methods (GET, POST, UPDATE, DELETE) and path (relative path).</li> <li>A more detailed description of the content of each response.</li> </ul> <p><img src="/img/examples-of-the-api-definition-end-points.png" alt="" title="Detail of a resource - host-initiator group"></p> <p>With this definition file (YAML or JSON), anyone can generate client libraries into a selected programming language or scripts. With the client libraries, a user can use them to programmatically consume the capabilities of Data Services Cloud Console. Currently, there are many tools on the market that are capable to perform the conversion. The list of some of the well-known open-API converter tools are:</p> <ul> <li><a href="https://openapi-generator.tech/">OpenAPI generator</a></li> <li><a href="https://swagger.io/tools/swagger-codegen/">Swagger Codegen</a></li> <li><a href="https://github.com/Azure/autorest">Azure AutoRest</a></li> <li>REST API Client Code generator (Found within <a href="https://marketplace.visualstudio.com/items?itemName=ChristianResmaHelle.ApiClientCodeGenerator">Visual Studio MarketPlace</a>)</li> </ul> <p>In this tutorial, I am going to cover the popular and simple OpenAPI generator from the <a href="https://openapi-generator.tech/">OpenAPI generator</a> from soup to nuts.</p> <h3><em>Let's get on with it!</em></h3> <h3>Generating Client Libraries using OpenAPI Generator painlessly</h3> <p>The OpenAPI Generator tool allows the automatic generation of API client libraries (SDKs), server stubs, documentation, and configuration with a given input of <a href="https://github.com/OAI/OpenAPI-Specification">OpenAPI spec</a> (supporting both 2.0 and 3.0 OpenAPI formats). This tool can generate more than 50 programming languages that can be consumed by various DevOps tools.</p> <p>The OpenAPI Generator tool is available for the variety of applications that meet with the user's familiarity. The OpenAPI Generator website provides 4 different sets of applications:</p> <ul> <li>A downloadable and executable <strong>JAR</strong> file that can be executed using <strong>Java Run Time tool</strong>.</li> <li>A <strong>Docker</strong> image that can be executed using the docker engine.</li> <li>Some dependencies in <strong>Maven</strong> and <strong>Gradle</strong> projects that can be used for building the automation tool.</li> <li>A node package manager (<strong>npm</strong>) package wrapper.</li> </ul> <p>Those applications and the information about using the OpenAPI generator are available in the Readme section found on this <a href="https://github.com/OpenAPITools/openapi-generator">GitHub page</a> <a href="https://github.com/OpenAPITools/openapi-generator"></a>as shown below:</p> <p><img src="/img/openapi-generator.png" alt="" title="OpenAPI Generator GitHub Page"></p> <p>The key information found in this GitHub website that will be useful and important is the latest stable version number that can be used for the conversion. This version is available at the right column of this webpage as shown below:</p> <p><img src="/img/openapi-generator-version-locatoin.png" alt="" title="Stable version for the openAPI generator project"></p> <p>In this tutorial, let's take a look at the painless and simplest application of the OpenAPI Generator, which is using the JAR file. Actually, using the JAR file doesn't require any application installation at all. The JAR file can be downloaded and executed directly from the command line window in your workstation. The minimum requirement for executing the JAR file is that your workstation must be deployed with JAVA Run Time Environment (JRE) version 8.</p> <p><strong>Note:</strong> For more information about the deployment of JAVA Run Time executeables based on the operating system of your workstation, please take a look at the installation page from <a href="https://www.java.com/en/download/help/download_options.html">the JAVA website</a>.</p> <p>The JAR file for this OpenAPI generator is available at Maven.org. You can download it from the following <a href="https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/5.4.0/openapi-generator-cli-5.4.0.jar">location</a>. Below, you will find the syntax required to download the OpenAPI generator JAR files from each corresponding operating systems:</p> <p><strong>For Mac/Linux users:</strong></p> <pre><code class="language-shell">~$ wget https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/5.4.0/openapi-generator-cli-5.4.0.jar -O openapi-generator-cli.jar </code></pre> <p><strong>For Microsoft Windows users:</strong></p> <pre><code class="language-powershell">> Invoke-WebRequest -OutFile openapi-generator-cli.jar https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/5.4.0/openapi-generator-cli-5.4.0.jar </code></pre> <p><strong>Note:</strong> The OpenAPI Generator version shown above is 5.4.0 (current version as of March 2022). Please keep in mind that, in the future, the location of the JAR file will change once the new version is released. Please take a look at the figure above to obtain the latest version number, and modify the path to download the latest <em>openapi-generator-cli</em> JAR file.</p> <p>Once the JAR file is downloaded, you can execute the following CLI at the folder where the JAR file is downloaded, to display the brief information on how to use this JAR file.</p> <pre><code class="language-shell">~$ java -jar openapi-generator-cli.jar help </code></pre> <p>The output will be something like this:</p> <pre><code class="language-typescript">usage: openapi-generator-cli &#x3C;command> [&#x3C;args>] The most commonly used openapi-generator-cli commands are: author Utilities for authoring generators or customizing templates. batch Generate code in batch via external configs. config-help Config help for chosen lang generate Generate code with the specified generator. help Display help information about openapi-generator list Lists the available generators meta MetaGenerator. Generator for creating a new template set and configuration for Codegen. The output will be based on the language you specify, and includes default templates to include. validate Validate specification version Show version information used in tooling See 'openapi-generator-cli help &#x3C;command>' for more information on a specific command. </code></pre> <p>Now that the JAR file is downloaded and ready to be used, let's create a Python SDK using the OpenAPI generator JAR file. The following command line is used for generating a Python client library using the <em>openapi-generator-cli.jar</em> file.</p> <pre><code class="language-shell">~$ java -jar openapi-generator-cli.jar generate -i storage-api.yaml -g python -o sdks/dscc-python-sdk </code></pre> <ul> <li>'-i' represents the input file, that is the downloaded Data Services Cloud Console OpenAPI spec which can be in the form of JSON or YAML.</li> <li>'generate' represents generating the code based on the specified generator</li> <li>'g' represents the generator/language name like Java, Go.</li> <li>'o' represents output directory where the client library will be generated. In the above example, the generated files are going to be in the ~/sdks/dscc-python-sdk</li> </ul> <p>This Python DSCC client library can be generated in a few minutes. Below, you can see the screen output generated from the Python client library using the <em>openapi-generator.jar</em> file.</p> <p><img src="/img/client-generation.jpg" alt="" title="Generating Python SDK using OpenAPI generator"></p> <p>The results from the conversion are available under the following folder:</p> <pre><code class="language-shell">~/sdks/dscc-python-sdk$ ls -al total 232 drwxrwxr-x 6 ronald ronald 4096 Mar 25 16:44 . drwxrwxr-x 3 ronald ronald 4096 Mar 25 16:44 .. drwxrwxr-x 2 ronald ronald 32768 Mar 25 16:44 docs -rw-rw-r-- 1 ronald ronald 807 Mar 25 16:44 .gitignore -rw-rw-r-- 1 ronald ronald 433 Mar 25 16:44 .gitlab-ci.yml -rw-rw-r-- 1 ronald ronald 1830 Mar 25 16:44 git_push.sh drwxrwxr-x 6 ronald ronald 4096 Mar 25 16:44 openapi_client drwxrwxr-x 2 ronald ronald 4096 Mar 25 16:44 .openapi-generator -rw-rw-r-- 1 ronald ronald 1040 Mar 25 16:44 .openapi-generator-ignore -rw-rw-r-- 1 ronald ronald 98525 Mar 25 16:44 README.md -rw-rw-r-- 1 ronald ronald 64 Mar 25 16:44 requirements.txt -rw-rw-r-- 1 ronald ronald 28 Mar 25 16:44 setup.cfg -rw-rw-r-- 1 ronald ronald 1002 Mar 25 16:44 setup.py drwxrwxr-x 2 ronald ronald 36864 Mar 25 16:44 test -rw-rw-r-- 1 ronald ronald 18 Mar 25 16:44 test-requirements.txt -rw-rw-r-- 1 ronald ronald 150 Mar 25 16:44 tox.ini -rw-rw-r-- 1 ronald ronald 304 Mar 25 16:44 .travis.yml ~/sdks/dscc-python-sdk$ </code></pre> <p>The generated client library can be made available on your workstation, or it can also be uploaded to a GitHub Repo so that it can be made available for others to use. An example of the GitHub repository of a sample Python client library looks like this:</p> <p><img src="/img/python-open-api-sdk-repo.jpg" alt="" title="GitHub repository of Python Client library generated using OpenAPI generator"></p> <p>For information on uploading a project and the associated files into GitHub, please see the following <a href="https://docs.github.com/en/get-started/importing-your-projects-to-github/importing-source-code-to-github/adding-locally-hosted-code-to-github">website</a>.</p> <p>Now, the generated client library comes with the following files:</p> <ul> <li>The code for assisting the console API calls.</li> <li>The documentation for this console client library derived from the API spec.</li> <li>Some test codes that can be used to validate the operation of this client library.</li> <li>Examples of every endpoint available in the README.md</li> <li>The required Python dependencies (requirements.txt and test-requirements.txt) that are required for using this SDK</li> </ul> <p>Now, you have all the components that are required for invoking the console API using Python scripts. To use this Data Services Cloud Console API client library, let's go through the steps that are described in the README.md:</p> <ol> <li>Installation instructions</li> </ol> <p><img src="/img/readme.png" alt="" title="Python SDK installation instructions"></p> <ol start="2"> <li>A sample code to get started with</li> </ol> <p><img src="/img/sample.png" alt="" title="Sample code"></p> <ol start="3"> <li>Documentation list for all endpoints</li> </ol> <p><img src="/img/api-endpoints.png" alt="" title="List of endpoints"></p> <ol start="4"> <li>Documentation list for all models</li> </ol> <p><img src="/img/models.png" alt="" title="List of models"></p> <ol start="5"> <li>Documentation about authorization</li> </ol> <p><img src="/img/auth.png" alt="" title="Authorization of API calls"></p> <p>Let's run a sample code that displays the access types in the console. The required parameters and the returned results for each endpoint are described under each of the endpoints in the sample code. To execute the an API call on the console, you will need:</p> <ol> <li>The authorized access token which is generated from the HPE GreenLake API Gateway as mentioned in the blog <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">post</a>.</li> <li>To install the required Python dependencies using the following command:</li> </ol> <pre><code class="language-shell">~$ pip install requirements.txt </code></pre> <p>Below is the example code for obtaining the information about the associated user's RBAC association. This code sample will provide list of the capabilities (port.read, volume.create) that are authorized for the user to exercise. To execute this code, please substitute the <strong>YOUR_BEARER_TOKEN</strong> with the access token generated in your example.</p> <pre><code class="language-python">import time import openapi_client from openapi_client.api import authz_api from openapi_client.model.error_response import ErrorResponse from openapi_client.model.access_controls_response import AccessControlsResponse from pprint import pprint # Defining the host is optional and defaults to https://eu1.data.cloud.hpe.com # See configuration.py for a list of all supported configuration parameters. configuration = openapi_client.Configuration( host = "https://eu1.data.cloud.hpe.com" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure Bearer authorization (JWT): JWTAuth configuration = openapi_client.Configuration( access_token = 'YOUR_BEARER_TOKEN' ) # Enter a context with an instance of the API client with openapi_client.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = authz_api.AuthzApi(api_client) permission = ["volume.create","port.read","audit.read"] # [str] | List of permissions, each of which, has the form \"resource type.permission\" (ex. volume.read,volume.write). The word \"ANY\" can be used as a wild card for the resource type (ex. ANY.read). Omitting the permission parameters is equivalent to asking for all user permissions. (optional) # example passing only required values which don't have defaults set # and optional values try: # Get User Accessible Resources api_response = api_instance.get_access_controls(permission=permission) pprint(api_response) except openapi_client.ApiException as e: print("Exception when calling AuthzApi->get_access_controls: %s\n" % e) </code></pre> <p>The output from the execution of the above code is shown below:</p> <pre><code class="language-Shell">$ python .\GetAudits.py {'items': ['port.read', 'volume.create']} </code></pre> <p>Using the client generator tool such as described above, the client libraries for DSCC can be generated in agile manner. As the result, the Data Services Cloud Console API client library in Python can be pushed to a GitHub repository automatically so that it will be ready to be used by any projects. Further advantage of this method is the automation using Continuous Integration/Continuous Deployment (CI/CD) pipeline which require no manual intervention in updating any projects to use the latest released version of the DSCC API.</p> <p>I hope this blog post on generating Python client libraries for Data Services Cloud Console is helpful.</p> <p>More blog posts will be coming to help you taking further advantage of Data Services Cloud Console REST API capabilities. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV</a> blog for more blog posts about Data Services Cloud Console REST API.</p>https://developer.hpe.com/3-ways-a-data-fabric-enables-a-data-first-approach/https://developer.hpe.com/3-ways-a-data-fabric-enables-a-data-first-approach/Tue, 15 Mar 2022 10:07:10 GMT<p><strong>Editor’s note: This article was originally posted on HPE Enterprise.nxt on March 15, 2022</strong></p> <hr> <p>A well-engineered modern data fabric allows DevOps and other teams to access data in the way they prefer.</p> <p>A data-first enterprise is a big advantage, but this strategy also puts a lot of demands on your data technology. That's all right unless your data technology puts a lot of demands on you.</p> <p>Take modern cars. Vehicles can now easily contain over 100 computers to manage functions like adaptive cruise control, stability control, and anti-lock braking. These systems make cars much more complicated internally than decades ago, but they are much easier and safer to drive because of them.</p> <p>Similarly, modern data technology needs to make it easier for users and system administrators of large-scale systems to work with data in more sophisticated and varied ways and at more locations, as is the case in a data-first enterprise. What does your data infrastructure need to do to help rather than be a hindrance?</p> <h3>How does data fabric meet the demands of a data-first approach?</h3> <p>A data fabric is a highly scalable data infrastructure designed to store, manage, and move data as a unifying layer across an enterprise from edge to data center, on-premises or in the cloud.</p> <p>In a data-first environment, data is treated as a foundational resource, one that is not used up when it is accessed. The capabilities of your data infrastructure should support the reuse of data and use by multiple applications. For example, the HPE Ezmeral Data Fabric File and Object Store software supports data reuse and data sharing by not requiring specialized data access methods. Off-the-shelf and custom applications can directly access data stored in the data fabric.</p> <p>Here's a sample of some of the many ways a modern data fabric has a positive impact in a data-first approach.</p> <h3>To move or not to move: Data where it needs to be</h3> <p>Data motion is a key issue in large-scale data systems. Data motion can include motion within a cluster and between clusters. Making wrong assumptions about data motion is one of the most common ways businesses inadvertently give up their ability to extract value from data.</p> <p>At one extreme, people may have an ingrained assumption that data motion is not a viable option, based on legacy systems that lack any provision for moving data. Without motion, data that could have value if put into a more global context may be discarded instead.</p> <p><img src="/img/3waysadatafabric-enablesadatafirstapproach-quote.png" alt="Block text" title="Block text"></p> <p>At other companies, the pendulum has swung radically to the opposite extreme, with a policy that all data to be analyzed must be moved to a central data center, either on premises or in the cloud. Unfortunately, the costs of data motion mount up, and where large amounts of data are at issue, only a tiny fraction of all possible data will be moved. Once again, data you could analyze is simply discarded.</p> <p>Fortunately, a new middle ground is emerging. In telecommunications, finance, media, manufacturing, and other business sectors, far more data is collected than could possibly be moved back to headquarters. Data is partially processed in situ before extracts and summaries are moved to a central location or a regional sub-center for further analysis on aggregates from many edge sources. This edge processing strategy commonly uses pattern recognition to pull out the interesting bits or anomalies for transfer.</p> <p>There are many reasons to move data, including communication between data centers or to a secondary cluster as part of a disaster recovery plan. The key to success is choice: You should be able to efficiently move data as needed or store and analyze it in place, all within the same system. Data motion should not have to be coded into each application.</p> <p>Taking the example of HPE Ezmeral Data Fabric, selective data motion can be configured rather than coded, and data fabric moves the data invisibly. The data fabric even builds in remote access via a global namespace in case you need it.</p> <h3>Decouple the query engine from data storage</h3> <p>The term database conjures up images of big iron running a system like Postgres or Oracle or a data warehouse like Teradata. All of the classical databases had this in common: The software that handles the storage of data is tightly integrated with the software that optimizes and executes queries.</p> <p>Another common element of such database systems is that when it comes to processing the data in the database, it is their way or the highway. You could submit queries from practically any computer language around, but you can't do what SQL won't do. For applications such as machine learning, a SQL database just isn't a good fit except for data extraction. Even then, severe scale limitations are common with high-end databases.</p> <p>The situation is changing. The trend now is to separate the query and storage engines into independent parts. The functional independence of query and storage isn't entirely new, but the idea that a SQL query engine should mostly query data stored in ordinary files is a big change.</p> <p>The practical impact of this separation is that you can reuse data for a completely different purpose than originally intended. If your original purpose was to clear payments and produce statements and bills for tens of millions of credit card accounts, then a SQL query engine like Presto might be just the ticket.</p> <p>However, in a data-driven enterprise, the real value from data doesn't usually come from collecting entirely new data. Instead, it comes from reusing or combining existing data in new ways, often with new tools. Mixing and matching query engines on the same data strikes gold. Locking data up in a monolithic database is just the opposite.</p> <p>For example, while recently working on an open source project, one of the authors (Dunning) built some geospatial processing that quickly outgrew the relational database in use. Python and Parquet files worked great for initial extraction and cleaning, but indexing and sorting the historical data involved billions of geohash (a public domain geocode system) operations.</p> <p>Storing that data in a data fabric allowed a seamless transition to distributed processing steps in <a href="https://julialang.org/">the Julia programming language</a> that ran 100 times faster and could scale more easily. Keeping simple tasks simple is a big win for data fabric in these kinds of systems.</p> <h3>Object storage vs. files</h3> <p>One change that characterizes a data-first enterprise is that architectural control moves much closer to the developer and data scientists in the line of business. Previously, much of that control was in the technology group in IT. This change is, in fact, the core driving force behind the DevOps movement. A major consequence of this shift has been the commoditization of IT services, an approach taken to extremes by the public cloud vendors.</p> <p>An unforeseen (but obvious in hindsight) side effect of this shift is a divergence between how DevOps teams view data infrastructure and how IT teams view it. The DevOps point of view is all about simplicity and flexibility, while the IT view has always been about optimization of provisioning combined with centralized control.</p> <p>Pushing for simplicity and flexibility drives a preference for data access with as little participation by the operating system as possible and certainly no special OS privileges. These constraints may put something as simple as mounting a file system out of bounds. These limits make object storage systems very attractive, since objects are accessed directly using simple protocols like HTTP instead of asking the OS to get file data from some file store. All you need is network access. On the other hand, performance isn't usually very high, and objects don't work like files, so compatibility suffers.</p> <p>Prioritizing optimization, in contrast, leads to highly manageable systems like storage appliances to provide block storage to blade servers. In this model, just enough storage is allocated to applications that primarily use non-distributed file systems for data. The operating system kernel mounts these file systems and controls all access. That's fine for some things, but it hurts scalability and makes DevOps harder.</p> <p>The fact is, recent technology makes both goals achievable. If a modern data fabric is engineered to allow access to data as either files or as objects, this flexibility frees DevOps teams to access data in the way they prefer. In addition, a data fabric can have built-in capabilities for data management and data motion. These capabilities make it much easier for IT teams to manage the overall system at scale.</p> <h3>Data fabric for data-first</h3> <p>Having the right data infrastructure lets you focus on the decisions that will make your organization a data-first enterprise. Whether it is choosing the right level of data motion, using multiple tools to analyze the same data, or storing data as objects or files, your data infrastructure needs to provide a host of advanced capabilities but still be easy enough to drive. A modern data fabric does just that.</p> <h3>Lessons for leaders</h3> <ul> <li>A data fabric is a highly scalable data infrastructure designed to store, manage, and move data as a unifying layer across an enterprise from edge to data center, on premises or in the cloud.</li> <li>A well-engineered modern data fabric allows DevOps and other teams to access data in the way they prefer.</li> <li>Making wrong assumptions about data motion is one of the most common ways businesses inadvertently give up their ability to extract value from data.</li> </ul> <br /> <blockquote> <blockquote> <p><span style="color:grey; font-family:Arial; font-size:1em"> This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.</span></p> </blockquote> </blockquote> <br /> <p><u><strong>About the authors:</strong></u></p> <p>Ted Dunning is chief technologist for data fabric at HPE. He has a PhD in computer science and authored more than 10 books focused on data sciences. He has more than 25 patents in advanced computing and plays the mandolin and guitar, both poorly.</p> <p>Ellen Friedman is a principal technologist at Hewlett Packard Enterprise focused on large-scale data analytics and machine learning. Ellen worked at MapR Technologies prior to her current role at HPE. She was a committer for the Apache Drill and Apache Mahout open source projects and is a co-author of multiple books published by O’Reilly Media, including AI &#x26; Analytics in Production, Machine Learning Logistics, and the Practical Machine Learning series.</p>https://developer.hpe.com/infrastructure-as-code-on-hpe-greenlake-using-terraform/https://developer.hpe.com/infrastructure-as-code-on-hpe-greenlake-using-terraform/Tue, 08 Mar 2022 15:17:41 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <p>The process of managing and provisioning computer data centers through machine-readable definition files, otherwise known as Infrastructure-as-Code (IaC), offers many significant benefits. It helps to increase operational agility, simplify management, reduce errors, and save cost. In this post, I’ll explore some of the benefits of using IaC on HPE GreenLake through the use of Terraform.</p> <h2>Let’s harness some of the benefits of Infrastructure as Code</h2> <p>One of the superpowers of IaC is its repeatability, the fact that you can set something up once and then use the same information in multiple ways. Implementing IaC allows organizations to store configuration files describing the desired infrastructure as a single source of truth. It also allows you to apply the DevOps methodology that’s already in place for application code directly to the infrastructure. For example, configuration files can be stored and managed through GitHub using the same way your DevOps team manages the application code. This concept is often called “Shifting Left”, as you are describing the infrastructure to host an application earlier (left) in the delivery pipeline of the application. This allows for easier and consistent deployments of infrastructure across the complete infrastructure landscape of an organization.</p> <h2>HPE GreenLake</h2> <p>HPE GreenLake is HPE’s edge-to-cloud platform. The HPE GreenLake platform provides a unified experience wherever your applications and its data are located on the edge, in colocations or in your own datacenter. This cloud experience everywhere includes the following capabilities:</p> <ul> <li>Self-service</li> <li>Infinite scalability</li> <li>Pay-as-you-go</li> <li>Managed for you</li> </ul> <h2>HPE GreenLake Cloud Services</h2> <p>The HPE GreenLake ecosystem provides solutions for several top workloads such as containers, Machine Learning, private cloud, virtual machines, SAP HANA, HPC, VDI and many more. This page on <a href="https://www.hpe.com/us/en/greenlake/services.html">HPE GreenLake cloud services and ecosystem</a> provides a complete list. The ecosystem also leverages many technologies from HPE partners such as Microsoft, VMware, SAP, Nutanix, Veeam and others. </p> <h2>HPE GreenLake for private cloud</h2> <p>One of the options provided by HPE GreenLake is to make it easy for customers to order and operate a private cloud with a mix of virtual machines, containers, and physical servers. This is exactly what the private cloud Service is all about. This service allows customers to point and click to create resources such as virtual machines. It also provides access via a public API, allowing developers to use  an Infrastructure-as-Code type of tool to automate provisioning, for example using Terraform.</p> <h2>Terraform</h2> <p><a href="https://terraform.io">Terraform</a> is an open source Infrastructure-as-Code framework originally created by HashiCorp that is written in Go. It uses a declarative language (HashiCorp Configuration Language HCL or JSON more recently) to describe the desired state of the infrastructure in terms of cloud, virtual machines, networks, storage, and many other components. Terraform uses the concept of “providers” to integrate with all major public clouds. Terraform is a so-called idempotent system in the sense that it doesn’t generate any side effects if applied multiple times on an infrastructure already in its desired state. Terraform has gained quite the momentum in the last few years. Its main competition is Ansible, Amazon Cloud Formation, Puppet and Chef.</p> <h2>Readying for your Infrastructure-as-Code implementation</h2> <h3>Terraform installation</h3> <p>Your first step is to get your system ready to run Terraform. In case this has not been done yet, this will include:</p> <ol> <li>Installing Terraform: follow <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">these steps</a></li> <li>Verifying installation: <strong>terraform --help</strong></li> </ol> <p>At this point, you are ready to start building your infrastructure description file.  </p> <h3>Building a Terraform configuration file from scratch</h3> <p>Let’s start building this TF file using your favorite editor.</p> <h4>Selecting a Terraform provider</h4> <p>The first section of the file will enumerate the “providers” you rely upon for building your infrastructure, and they could be multiple providers in a single TF file. In this case here, you will only have the HPE GreenLake provider referenced as hpe/hpegl in the official <a href="https://registry.terraform.io/">Terraform registry</a>.</p> <p>The first lines of your Terraform configuration file should look like this:</p> <pre><code class="language-json"># Load HPE GreenLake terraform provider terraform { required_providers { hpegl = { source = "hpe/hpegl" version = "0.3.17" } } } </code></pre> <p>You can find out more about the HPE GreenLake Terraform provider from its <a href="https://registry.terraform.io/providers/HPE/hpegl/latest">Terraform Registry page</a>.</p> <p>This page also provides a link to the GitHub repository corresponding to this provider. The <a href="https://github.com/hpe/terraform-provider-hpegl/tree/main/docs">docs</a> folder is your best source of information for using the different data sources and resources provided by the provider. If you navigate to the resources section, you will see that one resource you can manipulate with this provider is a <a href="https://github.com/hpe/terraform-provider-hpegl/blob/main/docs/resources/vmaas_instance.md">VM instance</a>. Let’s focus on this resource in this article.</p> <blockquote> <p>Note: Because this is open source, don’t hesitate to open issues, or even a pull request, if you identify an issue.</p> </blockquote> <h4>Setting up the Terraform provider</h4> <p>Now that you have expressed the fact that the hpegl provider will be used, you need to setup some parameters for it. As explained on this <a href="https://github.com/hpe/terraform-provider-hpegl/blob/main/docs/index.md">page</a>, you can either explicitly set those parameters in your TF file, or have them set in a series of environment variables, or a mix of both. I suggest the following two parameters be added in your TF file:</p> <pre><code class="language-json"># Setup provider environment (location and space) provider "hpegl" { vmaas { location = "HPE" space_name = "TerraForm Space" } } </code></pre> <p>And the rest (such as tenant id, user id and user secret key) can be placed in a RC file, which you can source before running your Terraform command.</p> <p>You can find your location and your space name from the HPE GreenLake user interface. In our example shown below, HPE is our location:</p> <p><img src="/img/greenlakeforprivatecloud.png" alt="GreenLake for private cloud" title="GreenLake for private cloud"></p> <p>And in the capture below, <strong>Terraform Space</strong> is the space we have created for our work with Terraform. You can check your available Spaces from the HPE GreenLake console under your profile icon, <strong>Change space</strong>.</p> <p><img src="/img/greenlakeselectingspace.png" alt="GreenLake select new space" title="GreenLake select new space"></p> <h4>Setting up a API Client access</h4> <p>Next, you need to create a new API Client access dedicated to Terraform. You can do this from the HPE GreenLake console under your settings icon, <strong>Identity &#x26; Access</strong> and then the <strong>API Clients</strong> tab.</p> <p><img src="/img/greenlakeapiclients.png" alt="GreenLake API Clients" title="GreenLake API Clients"></p> <p>Create a new API Client (hpedev-hackshack-terraform in the screen capture above), and make sure the Tenant Contributor role is assigned on your space, better yet, create a new role from the default Tenant Contributor role for this API client access. Also take note of the client id and the Issuer URL as shown in capture below.</p> <blockquote> <p>Notes: More details on GreenLake user roles can be found in the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=GUID-328019F3-4305-4153-BD1A-B5E43D66FB1B.html">GreenLake documentation</a>.</p> </blockquote> <p><img src="/img/greenlaketerraformapiclient.png" alt="GreenLake Terraform API Client" title="GreenLake Terraform API Client"></p> <blockquote> <p>Note: You need to remember the API Client secret key, as it’s not displayed anymore after creation.</p> </blockquote> <p>Finally, you’ll need your Tenant ID as shown from the HPE GreenLake console under your profile icon, <strong>API Access</strong>.</p> <p><img src="/img/greenlaketenantid.png" alt="GreenLake Tenant ID" title="GreenLake Tenant ID"></p> <p>With this you can now build a resource file that defines the following environment variables:</p> <pre><code class="language-markdown">export HPEGL_TENANT_ID=&#x3C;Your Tenant ID> export HPEGL_USER_ID=&#x3C;Client ID of the API Client> export HPEGL_USER_SECRET=&#x3C;Secret Key displayed when you created the API Client> export HPEGL_IAM_SERVICE_URL=&#x3C;Issuer URL> </code></pre> <p>And execute it on your machine to set these environment variables.</p> <h4>Querying for infrastructure components</h4> <p>Your next step with the TF file is to query the HPE GreenLake provider to collect information needed to create your first VM instance. From the <a href="https://github.com/hpe/terraform-provider-hpegl/blob/main/docs/resources/vmaas_instance.md">documentation</a>, you can see that you need to gather the following information:</p> <ul> <li>Cloud ID</li> <li>Group ID</li> <li>Layout ID</li> <li>Plan ID</li> <li>Instance type code</li> <li>Network ID</li> <li>Resource Pool ID</li> <li>Template ID</li> <li>Folder Code</li> </ul> <p>For this, you will use the Terraform <strong>data</strong> statements. For example, the following statement retrieves the Cloud ID and stores it (in variable called <strong>cloud</strong>), which we can later retrieve using: <strong>data.hpegl_vmaas_cloud.cloud.id</strong></p> <pre><code class="language-json"># Retrieve cloud id data "hpegl_vmaas_cloud" "cloud" { name = "HPE GreenLake VMaaS Cloud-Trial4 " } </code></pre> <p>Using a similar technique, you can retrieve the rest of the data you need:</p> <pre><code class="language-json"># And a few networks data "hpegl_vmaas_network" "blue_net" { name = "Blue-Network" } data "hpegl_vmaas_network" "green_net" { name = "Green-network" } data "hpegl_vmaas_cloud_folder" "compute_folder" { cloud_id = data.hpegl_vmaas_cloud.cloud.id name = "ComputeFolder" } # Locate a resource pool data "hpegl_vmaas_resource_pool" "cl_resource_pool" { cloud_id = data.hpegl_vmaas_cloud.cloud.id name = "ComputeResourcePool" } # And a group data "hpegl_vmaas_group" "default_group" { name = "HPEDEV-HackShackTenant-Group" } # Locate a plan data "hpegl_vmaas_plan" "g1_small" { name = "G1-Small" } # A layout data "hpegl_vmaas_layout" "vmware" { name = "VMware VM with vanilla CentOS" instance_type_code = "glhc-vanilla-centos" } # And a template data "hpegl_vmaas_template" "vanilla" { name = "vanilla-centos7-x86_64-09072020" } </code></pre> <blockquote> <p>You can get information about each of the data statements supported by the hpegl provider from <a href="https://github.com/hpe/terraform-provider-hpegl/tree/main/docs/data-sources">GitHub</a>.</p> </blockquote> <h4>Creating a VM resource</h4> <p>The last step is to use a Terraform <strong>resource</strong> statement to request the creation of a new VM instance:</p> <pre><code class="language-json">resource "hpegl_vmaas_instance" "DidierTest1" { name = "DidierTest1" cloud_id = data.hpegl_vmaas_cloud.cloud.id group_id = data.hpegl_vmaas_group.default_group.id layout_id = data.hpegl_vmaas_layout.vmware.id plan_id = data.hpegl_vmaas_plan.g1_small.id instance_type_code = data.hpegl_vmaas_layout.vmware.instance_type_code network { id = data.hpegl_vmaas_network.green_net.id } volume { name = "root_vol" size = 15 datastore_id = "auto" } config { resource_pool_id = data.hpegl_vmaas_resource_pool.cl_resource_pool.id template_id = data.hpegl_vmaas_template.vanilla.id no_agent = true asset_tag = "vm_terraform" folder_code = data.hpegl_vmaas_cloud_folder.compute_folder.code } power = "poweron" } </code></pre> <blockquote> <p>Note: You can get information about each of the resource statements supported by the hpegl provider from <a href="https://github.com/hpe/terraform-provider-hpegl/tree/main/docs/resources">GitHub</a>.</p> </blockquote> <h4>Terraform init</h4> <p>Before you can use Terraform, you will have to initialize it from the configuration file we have created. This is done with the following step: <strong>terraform init</strong></p> <pre><code class="language-markdown">$ terraform init Initializing the backend... Initializing provider plugins... - Finding hpe/hpegl versions matching "0.1.7"... - Installing hpe/hpegl v0.3.17... - Installed hpe/hpegl v0.3.17 (signed by a HashiCorp partner, key ID D1F277A1AC66CE3D) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. </code></pre> <h4>Terraform ready to plan</h4> <p>To validate your configuration file, I recommend running the <strong>terraform validate</strong> command as you add sections to your file to track syntax errors. Once ready, the <strong>terraform plan</strong> command will provide information about what will be created when the <strong>terraform apply</strong> method is finally used.</p> <pre><code class="language-markdown">$ terraform plan Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # hpegl_vmaas_instance.DidierTest1 will be created + resource "hpegl_vmaas_instance" "DidierTest1" { + cloud_id = 1 + containers = (known after apply) + group_id = 3 + history = (known after apply) + hostname = (known after apply) + id = (known after apply) + instance_type_code = "glhc-vanilla-centos" + layout_id = 1159 + name = "DidierTest1" + plan_id = 402 + power = "poweron" + server_id = (known after apply) + status = (known after apply) + config { + asset_tag = "vm_terraform" + folder_code = "1" + no_agent = true + resource_pool_id = 1 + template_id = 573 } + network { + id = 6 + internal_id = (known after apply) + is_primary = (known after apply) + name = (known after apply) } + volume { + datastore_id = "auto" + id = (known after apply) + name = "root_vol" + root = true + size = 10 } } Plan: 1 to add, 0 to change, 0 to destroy. </code></pre> <p>If you agree with the plan, and what is going to be created, you can move to the last step, i.e. applying the configuration.</p> <h4>Terraform ready to apply</h4> <p>The command you need to use is now: <strong>terraform apply</strong>. This will rerun the plan command, then prompt you to confirm before it starts building what’s in the plan:</p> <pre><code class="language-markdown">Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes hpegl_vmaas_instance.DidierTest1: Creating... hpegl_vmaas_instance.DidierTest1: Still creating... [10s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [20s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [30s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [40s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [50s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m0s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m10s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m20s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m30s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m40s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [1m50s elapsed] hpegl_vmaas_instance.DidierTest1: Still creating... [2m0s elapsed] hpegl_vmaas_instance.DidierTest1: Creation complete after 2m9s [id=145] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. </code></pre> <p>If you open your HPE GreenLake console to monitor the VM resources, you will see the effect of the <strong>terraform apply</strong> command:</p> <p><img src="/img/terraform-greenlake-part2-blog-picture1-1.png" alt="GreenLake instance created" title="GreenLake instance created"></p> <h4>Cleaning it all up</h4> <p>In Terraform, clean-up can be done using the <strong>destroy</strong> command. This will automatically use the HPE GreenLake provider to clean the infrastructure in HPE GreenLake.</p> <pre><code class="language-markdown">$ terraform destroy hpegl_vmaas_instance.DidierTest1: Refreshing state... [id=145] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # hpegl_vmaas_instance.DidierTest1 will be destroyed - resource "hpegl_vmaas_instance" "DidierTest1" { - cloud_id = 1 -> null - containers = [ - { - container_type = [ - { - name = "vanilla-centos7-node" }, ] - external_fqdn = "didiertest1.localdomain" - hostname = "didiertest1" - id = 145 - ip = "172.17.70.29" - max_cores = 1 - max_memory = 4294967296 - max_storage = 16106127360 - name = "DidierTest1_145" - server = [ - { - compute_server_type = [ - { - external_delete = true - managed = true - name = "VMware Linux VM" }, ] - date_created = "2022-02-23T14:42:10Z" - id = 151 - last_updated = "2022-02-23T22:05:33Z" - owner = [ - { - username = "hpedev-hackshack-terraform" }, ] - platform = "" - platform_version = "" - server_os = [ - { - name = "centOS 7 64-bit" }, ] - ssh_host = "172.17.70.29" - ssh_port = 22 - visibility = "private" }, ] }, ] -> null - group_id = 3 -> null - history = [ - { - account_id = 2 - created_by = [ - { - display_name = "hpedev-hackshack-terraform" - username = "hpedev-hackshack-terraform" }, ] - date_created = "2022-02-23T14:42:11Z" - display_name = "DidierTest1" - duration = 54873 - end_date = "2022-02-23T14:43:06Z" - id = 1191 - instance_id = 145 - last_updated = "2022-02-23T14:43:06Z" - percent = 100 - process_type = [ - { - code = "provision" - name = "provision" }, ] - reason = "" - start_date = "2022-02-23T14:42:11Z" - status = "complete" - status_eta = 0 - unique_id = "dc48d7f7-f564-46b7-b60f-67eaf5193f38" - updated_by = [ - { - display_name = "hpedev-hackshack-terraform" - username = "hpedev-hackshack-terraform" }, ] }, ] -> null - id = "145" -> null - instance_type_code = "glhc-vanilla-centos" -> null - layout_id = 1159 -> null - name = "DidierTest1" -> null - plan_id = 402 -> null - power = "poweron" -> null - server_id = 151 -> null - status = "running" -> null - config { - asset_tag = "vm_terraform" -> null - create_user = false -> null - folder_code = "group-v41" -> null - no_agent = true -> null - resource_pool_id = 2 -> null - template_id = 573 -> null } - network { - id = 7 -> null - interface_id = 0 -> null - internal_id = 197 -> null - is_primary = true -> null - name = "eth0" -> null } - volume { - datastore_id = "auto" -> null - id = 282 -> null - name = "root_vol" -> null - root = false -> null - size = 15 -> null } } Plan: 0 to add, 0 to change, 1 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: Enter a value: yes hpegl_vmaas_instance.DidierTest1: Destroying... [id=145] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 10s elapsed] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 20s elapsed] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 30s elapsed] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 40s elapsed] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 50s elapsed] hpegl_vmaas_instance.DidierTest1: Still destroying... [id=145, 1m0s elapsed] hpegl_vmaas_instance.DidierTest1: Destruction complete after 1m7s Destroy complete! Resources: 1 destroyed. </code></pre> <h3>What’s next?</h3> <p>In this blog post, I covered how to get started with the Terraform provider for HPE GreenLake, explained how to collect data from the platform, and showed how to request the creation of a VM instance. <a href="https://developer.hpe.com/blog/infrastructure-as-code-on-hpe-greenlake-using-terraform-%E2%80%93-part-2/">In my next article</a>, I will apply changes to infrastructure configuration file and demonstrate how the desired state is automatically tracked by Terraform and applied to HPE GreenLake.</p> <ul> <li><a href="https://www.terraform.io/">Learn more about Terraform</a></li> <li><a href="https://www.hpe.com/us/en/greenlake.html">Learn more about HPE GreenLake</a></li> <li><a href="https://registry.terraform.io/providers/hpe/hpegl/latest">Learn more about the HPE GreenLake Terraform provider</a></li> </ul> <p>Find other tutorials and articles on HPE GreenLake on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a>.</p>https://developer.hpe.com/how-to-provision-hpe-greenlake-private-cloud-resources-from-servicenow-service-catalog/https://developer.hpe.com/how-to-provision-hpe-greenlake-private-cloud-resources-from-servicenow-service-catalog/Tue, 08 Mar 2022 06:13:00 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>A key strength of HPE GreenLake for private cloud is its self-service orchestration and automation, which is done from the HPE GreenLake for private cloud dashboard and can be consumed using API and Infrastructure-as-Code (IaC). However, some ServiceNow users prefer to provision resources from the ServiceNow Service Catalog. For these users, HPE GreenLake for private cloud offers a free Morpheus plugin for ServiceNow, which can be installed from the ServiceNow Store. Once the plugin is installed and configured, HPE GreenLake for private cloud <a href="https://developer.hpe.com/blog/curate-and-expose-service-catalog-items-using-hpe-greenlake-for-private-cloud">catalog items</a> can be presented in the ServiceNow Service Catalog for ordering.</p> <p>This article walks through the process of integrating ServiceNow with HPE GreenLake for private cloud and exposing Service Catalog items to ServiceNow.</p> <h2>Prerequisites</h2> <!--StartFragment--> <p>The process of integrating HPE GreenLake for private cloud on ServiceNow requires different things depending on your environment. In any case, you will need the Morpheus plugin on ServiceNow, HPE GreenLake for private cloud service user account and Private cloud user in ServiceNow.</p> <h4>Install the Morpheus plugin</h4> <p>To obtain and install the Morpheus plugin, you must have your HI credentials. Using your HI account, simply search for Morpheus and install the Morpheus plugin.</p> <h4>Private cloud service user account</h4> <p>Before you begin the integration process, raise a ticket on HPE GreenLake support to create a service user in your HPE GreenLake for private cloud environment. For details, see <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=request-for-subtenant-service-user-for-servicenow-integration.html">Request for sub-tenant service user for ServiceNow integration</a>.</p> <h4>Private cloud user in ServiceNow</h4> <p>Using ServiceNow admin user, create a HPE GreenLake for private cloud user in ServiceNow with the below roles:</p> <ul> <li>catalog_admin</li> <li>import_transformer</li> <li>itil</li> <li>rest_service</li> <li>xmodamorpheus_ca.integration <em>(NOTE: You can only assign this role after installing the Morpheus plugin. If you are creating this user before that, you must go back and add this role after installing the plugin.)</em></li> </ul> <!--EndFragment--> <h2>Integrating private cloud on ServiceNow</h2> <p>Configure Morpheus properties in ServiceNow with the following parameters:</p> <ul> <li>The Morpheus Appliance endpoint: Enter the full URL to your HPE GreenLake for private cloud appliance</li> <li>Username: Enter the name of the HPE GreenLake for private cloud Service user (Received from HPE Support) that the Morpheus plugin used to connect to the HPE GreenLake Private Cloud API</li> <li>Password: Enter the HPE GreenLake for private cloud Service user password (Received from HPE Support)</li> </ul> <p><img src="/img/figure1.png" alt=""></p> <h2>Integrating ServiceNow with HPE GreenLake for private cloud</h2> <!--StartFragment--> <p>Navigate to <em>Administration > Integrations > +New Integration > ITSM> ServiceNow</em> in the HPE GreenLake for private cloud portal.</p> <p>Configure ServiceNow integration with the below parameters:</p> <ul> <li>Name - Enter the integration name</li> <li>ENABLED - Select to enable consumption of this ServiceNow integration in HPE GreenLake for private cloud. The integration is enabled by default</li> <li>SERVICE NOW HOST-Enter the ServiceNow instance host URL (example: <a href="https://your.instance.service-now.com/">https://your.instance.service-now.com</a>).</li> <li>USER - Enter a ServiceNow user created as part of pre-requistes</li> <li>PASSWORD - Password of the above-mentioned user</li> <li>Optional variables (CMDB CUSTOM MAPPING, CMDB CLASS MAPPING DEFAULT CMDB BUSINESS CLASS) not required for this use case</li> <li>Click SAVE CHANGES</li> </ul> <p><img src="/img/figure10.png" alt=""></p> <p>The ServiceNow integration is now displayed in the list of integrations.</p> <p>Sample of the ServiceNow integration summary:</p> <!--EndFragment--> <p><img src="/img/figure2.png" alt=""></p> <h2>Expose private cloud resources to the ServiceNow Service Catalog</h2> <p>After creating catalog items in the HPE GreenLake for private cloud as discussed in the previous <a href="https://developer.hpe.com/blog/curate-and-expose-service-catalog-items-using-hpe-greenlake-for-private-cloud/">blog</a>, the catalog items can be made available to ServiceNow Service Catalog by following procedure:</p> <ul> <li>Navigate to Administration > Integrations</li> <li>Select the ServiceNow integration</li> <li>From the Catalog Items tab, click + ADD CATALOG ITEM</li> <li>Select the catalog item using the drop-down and click SAVE CHANGES</li> </ul> <p><img src="/img/figure3.png" alt=""></p> <p>Below is the sample list of catalog items exposed to the ServiceNow integration:</p> <p><img src="/img/figure4.png" alt=""></p> <p>Now, from ServiceNow, verify the availability of the resources as follows:</p> <ul> <li>Log in to ServiceNow to access the Service Catalog</li> <li>From the Service Catalog, access the Morpheus plugin</li> </ul> <p><img src="/img/figure5.png" alt=""></p> <p>Exposed HPE GreenLake private cloud resources are now available in the Self-Service Service Catalog.</p> <p><img src="/img/figure6.png" alt=""></p> <h2>Provision private cloud resources from ServiceNow</h2> <p>Click on the service catalog and select the catalog item to order. For this example, RDS_MariaDB is chosen.</p> <p><img src="/img/figure7.png" alt=""></p> <p>Click Order Now, after filling in the necessary details. Below is the sample order status.</p> <p><img src="/img/figure8.png" alt=""></p> <p>Order is deployed in HPE GreenLake for private cloud.</p> <p><img src="/img/figure9.png" alt=""></p> <h2>Summary</h2> <p>With Morpheus plugin integration, Self-Service catalog items from HPE GreenLake for private cloud can be presented as provisioning options in the ServiceNow Service Catalog and the user can order and manage private cloud resources directly from ServiceNow. Hopefully, you found this tutorial helpful. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more posts on topics like this.</p>https://developer.hpe.com/2022-March-02/https://developer.hpe.com/2022-March-02/Wed, 02 Mar 2022 13:00:00 GMThttps://developer.hpe.com/come-and-engage-with-the-hpe-dev-team-at-hpe-technology-and-solution-summit-2022/https://developer.hpe.com/come-and-engage-with-the-hpe-dev-team-at-hpe-technology-and-solution-summit-2022/Wed, 23 Feb 2022 15:16:18 GMT<p><img src="/img/tss-fred-blog-1.png" alt=""></p> <p>Today, more and more purchasing decisions related to software, including cloud-based initiatives, are being influenced by developers and data scientists. The hands-on engagement opportunities HPE DEV offers at events like the Hewlett Packard Enterprise (HPE) <a href="https://h41382.www4.hpe.com/tss/">Technology Solutions Summit 2022</a> (TSS 2022) helps build a collaborative, open discussion where we can each learn from each other to address today's challenges with innovative (and often rather fun) methods.</p> <p>If you're one of the lucky ones in the HPE and HPE partner presales community who can take advantage of HPE's largest and most comprehensive technical and solutions knowledge transfer event, we encourage you to seek out the many different sessions that will be offered by the HPE DEV team. From March 28-31, you'll be able to participate virtually and attend numerous sessions on new open source and HPE technologies – from breakouts to Speaker-led (pre-registration required) and Walk-in Hands-on sessions.</p> <p><strong>Engage directly with the HPE DEV team</strong></p> <p>At HPE TSS 2022, you will have a couple of unique opportunities to speak directly with the HPE DEV team to hear about exciting work that's going on and to make your own voice heard. There will be two breakout sessions that cover HPE DEV activities and open source technologies (<a href="https://h41382.www4.hpe.com/tss/application/assets/pdf/TSS22-More_About_Our_Sessions.pdf">refer to the agenda for specifics</a>). These tech talks are about 75 minutes long and include 20 minutes of Q&#x26;A's at the end.</p> <p>In addition to the breakouts, the HPE DEV team is running a focus group.</p> <p>This highly-interactive session will occur on the last day of the event, acting as a wrap up where you will get opportunities to:</p> <ul> <li>Be recognized for your achievements</li> <li>Discover our new HPE DEV Evangelists program</li> <li>View the HPE DEV Workshops-on-Demand roadmaps</li> <li>Provide feedback on sessions and labs as well as on the overall HPE DEV program</li> </ul> <p><strong>HPE DEV Hands-on Sessions</strong></p> <p><img src="/img/tss-fred-blog-2.png" alt=""></p> <p>HPE DEV will be offering two types of Hands-on Sessions at HPE TSS 2022. The Speaker-led sessions are labs where we leverage our HPE DEV Workshops-on-Demand infrastructure to introduce you to some of our newest workshops. These labs require you to pre-register in order to attend. Make sure you reserve your seat today in order to learn more about:</p> <ul> <li>Securing Microservice Communication with Envoy using X.509 SPIFFE IDs</li> <li>Ansible 101</li> <li>Machine Learning 101</li> <li>Spark 101</li> </ul> <p>The Walk-in sessions are also labs but they don't require pre-registration. They are handled on a first-come/first-served basis. There, you will have access to all of our existing workshops:</p> <p><img src="/img/tss-fred-blog-3.png" alt=""></p> <p>Several workshops will be offered during a given time slot, with each track designed to address the needs of a specific role or "persona". The tracks are divided out into:</p> <ul> <li> <p><strong>Open Source Enthusiasts</strong></p> <ul> <li>API 101</li> <li>Python 101</li> <li>GIT 101</li> <li>Jupyter Notebook 101</li> <li>Kubernetes 101</li> <li>Docker 101</li> </ul> </li> <li> <p><strong>Becoming Data Scientists</strong></p> <ul> <li>Intro to the HPE Ezmeral Container Platform REST API</li> <li>HPE Ezmeral Data Fabric 101</li> <li>Using Kubernetes CSI with HPE Ezmeral Container Platform</li> <li>Deploying end-to-end machine learning workflows​ with HPE Ezmeral MLOPS​</li> <li>AI 101 - Convolutional neural network (CNN) for MNIST​ with HPE Ezmeral Container Platform</li> </ul> </li> <li> <p><strong>Cloud and Data Centers Operators</strong></p> <ul> <li>Redfish API 101</li> <li>HPE iLOrest - Overview of the HPE RESTful Interface Tool</li> <li>Using the iLO Redfish API with Ansible and HPE OneView</li> <li>Introduction to the HPE OneView REST API</li> <li>Stackstorm 101</li> </ul> </li> </ul> <p>If you are going to TSS 2022, make sure to sign up for the HPE DEV sessions. As you know, we value the importance of this technical event. We expect you do as well and appreciate that you have regularly rated our sessions highly. We look forward to reconnecting with you there.</p>https://developer.hpe.com/curate-and-expose-service-catalog-items-using-hpe-greenlake-for-private-cloud/https://developer.hpe.com/curate-and-expose-service-catalog-items-using-hpe-greenlake-for-private-cloud/Wed, 16 Feb 2022 08:49:59 GMT<center><img src="/img/intro-greenlake-512-pixels-rgb-image.jpg" width="512" height="384" alt="Curate and expose catalog items using the HPE GreenLake Service Catalog persona view"></center> <br /> <hr> <p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <p>The HPE GreenLake Edge-to-Cloud platform has a very useful interface for managing the HPE GreenLake for private cloud service called HPE GreenLake Central. It provides a cloud experience to manage VMs in your on-premises, pay-per-use datacenter. Personas are alternate views in the HPE GreenLake for private cloud user interface (UI). A user’s access to the various personas is controlled by role permissions.</p> <p>At present, there are two persona types: Standard and Service Catalog. The Standard persona is the typical default view. The Service Catalog persona is a simplified view where users are presented with different pre-configured instance types, blueprints, and workflows to choose from based upon their role. This improves the deployment experience with just a few clicks and without presenting an overwhelming list of options.</p> <p>The goal of this article is to discuss the Service Catalog persona in greater detail, including how administrators can curate the catalog and how users can use the Service Catalog to deploy their services. For readers needing a good introduction to HPE GreenLake for private cloud concepts, check out <a href="https://www.hpe.com/psnow/doc/a50003040enw?jumpid=in_lit-psnow-red">this technical paper</a> that will help you better understand the different concepts leveraged in this blog.</p> <h2>Using the Service Catalog persona view</h2> <p>Access to a persona view is controlled by a user’s role. By default, new roles and roles that existed prior to the creation of the personas will only have access to the Standard persona.</p> <p>With the Tenant Admin user, connect to HPE GreenLake Central, locate the HPE GreenLake for private cloud dashboard widget and click the Launch icon to open the HPE GreenLake for private cloud dashboard.</p> <ul> <li>Navigate to <strong>Administration > Roles</strong></li> <li>Click the name of the role to modify</li> <li>Click the <strong>Personas</strong> tab</li> <li>From the default persona drop-down list, select the default setting. The <strong>Standard</strong> type is typically used as the default.</li> <li>From the list of personas, locate the Service Catalog persona</li> <li>From the <strong>ACCESS</strong> column of the role, select <strong>FULL</strong></li> </ul> <p><img src="/img/catalog-image1.png" alt=""></p> <h2>Configuring Service Catalog item access</h2> <p>By default, user roles have no access to any catalog items. When enabling the Service Catalog persona access for user roles, you will also need to provide access to some or all catalog items.</p> <p>Configuring global access:</p> <ul> <li><strong>Full:</strong> Gives access to all catalog items</li> <li><strong>Custom:</strong> Gives access to individually-selected items from the list below</li> <li><strong>None:</strong> No access is given to any catalog items</li> </ul> <p><img src="/img/catalog-image2.png" alt=""></p> <h2>Build Service Catalog items</h2> <p>A Tenant Admin user can add catalog items (instance types, blueprints, and workflows) and allow some configurable options using the <strong>Option Types</strong>.</p> <blockquote> <p>Note: A Tenant Admin user should have full permission for <strong>Tools: Self Service</strong>.</p> </blockquote> <h2>Adding an instance catalog item</h2> <p>This example shows how to create an Apache instance catalog item with the assumption that the Apache instance type exists in the platform.</p> <ul> <li>From your user name drop-down list, ensure that the Standard persona is selected (For more details, refer to the <em><strong>Accessing Service Catalog persona</strong></em> section of this post)</li> <li>Navigate to <strong>Tools > Self Service</strong></li> <li>From the ADD drop-down list, select <strong>Instance</strong></li> </ul> <p>The ADD CATALOG ITEM dialog box opens</p> <p><img src="/img/catalog-image3.png" alt=""></p> <ul> <li> <p>Configure the catalog items as follows:</p> <ul> <li>NAME - Enter the catalog item name</li> <li>DESCRIPTION - (Optional) Enter the catalog item description</li> <li>ENABLED - Select to enable the catalog item, making it available for provisioning (default). Clear to disable.</li> <li>FEATURED - Select to enable special visibility of this catalog item in the Service Catalog persona view. Clear to disable. Special visibility means that an item can be featured (a tag is added in the item) and given priority in the Service Catalog and Dashboard views.</li> <li>LOGO - From the drop-down list, do one of the following: <ul> <li>Select an existing logo</li> <li>Select custom and click Browse to locate and upload a logo</li> </ul> </li> <li>CONFIG - Enter, view, or edit the instance configuration <ul> <li>To build this catalog item using the CREATE INSTANCE wizard, click CONFIGURATION WIZARD. For more details, see <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00092451en_us&#x26;page=GUID-3C344C62-EA07-4263-A540-29B5B92E3CE2.html">Instance creation configuration parameters</a></li> </ul> </li> <li>CONTENT - Enter a more detailed description about the instance, which will display in the order screen</li> <li>Option Types - (Optional) Enter the Option Types to present users with mandatory or optional selections during provisioning. Option Types can then be used in the CONFIG section. Below is the sample usage of Options Types in the CONFIG section.</li> </ul> </li> </ul> <p><img src="/img/catalog-image5.png" alt=""></p> <ul> <li>Click SAVE CHANGES</li> <li>Optionally, a Tenant Admin can provide access to the catalog items to a specific user or role (see <em><strong>Configuring Service Catalog item access</strong></em> earlier in this article)</li> </ul> <h2>Adding a blueprint catalog item</h2> <p>Blueprints enable full multi-tier application deployment. In the self-service catalog, you can create catalog items based on existing app blueprints. You can preconfigure blueprints and expose them to the Service Catalog persona for a click-to-deploy use case.</p> <ul> <li>Click on your name in the upper-right corner and ensure that the STANDARD persona is selected (For more details, refer to the <em><strong>Accessing the Service Catalog persona</strong></em> section later in this tutorial)</li> <li>Navigate to <strong>Tools > Self Service</strong></li> <li>From the ADD drop-down list, select <strong>Blueprint</strong></li> </ul> <p>The ADD CATALOG ITEM dialog box opens</p> <p><img src="/img/catalog-image6.png" alt=""></p> <p>This example shows the steps to create blueprint catalog item, which can be used to deploy Nodejs with MariaDB with the assumption that the blueprint exists in the platform.</p> <ul> <li> <p>Configure the catalog items as follows:</p> <ul> <li>NAME - Enter the catalog item name</li> <li>DESCRIPTION - (Optional) Enter the catalog item description</li> <li>ENABLED - Select to enable the catalog item, making it available for provisioning (default). Clear to disable.</li> <li>FEATURED - Select to enable special visibility of this catalog item in the Service Catalog persona view. Clear to disable. Special visibility means that an item can be featured (a tag is added in the item) and given priority in the Service Catalog and Dashboard views.</li> <li>LOGO - From the drop-down list, do one of the following: <ul> <li>Select an existing logo</li> <li>Select custom and click <strong>Browse</strong> to locate and upload a logo</li> </ul> </li> <li>CONFIGURE - Click CONFIGURE to build this catalog item. The NEW APP wizard opens. For information about using the wizard, refer to Creating an app from a blueprint.</li> <li>APP SPEC - (Optional) Inject an override blueprint spec in yaml format</li> <li>CONTENT - Enter a more detailed description about the app, which will display in the order screen</li> <li>Option Types - (Optional) Enter the option types to present to users with mandatory or optional selections prior to provisioning</li> </ul> </li> </ul> <p><img src="/img/catalog-image7.png" alt=""></p> <ul> <li>Click SAVE CHANGES.</li> <li>Optionally, a Tenant Admin can provide access to the catalog items to a specific user or role (see Configuring Service Catalog item access earlier in this article)</li> </ul> <h2>Adding a workflow catalog item</h2> <p>Workflows are groups of Tasks and can be run on-demand against an existing instance. You can preconfigure operational workflows and expose them to the Service Catalog persona for a click-to-deploy use case.</p> <ul> <li>Click on your name in the upper-right corner and ensure that the Standard persona is selected (For more details, refer to the Accessing the Service Catalog persona section of this post)</li> <li>Navigate to Tools > Self Service</li> <li>From the ADD drop-down list, select WORKFLOW</li> </ul> <p>The ADD CATALOG ITEM dialog box opens</p> <p><img src="/img/catalog-image8.png" alt=""></p> <ul> <li> <p>Configure the catalog items as follows:</p> <ul> <li>NAME - Enter the catalog item name</li> <li>DESCRIPTION - (Optional) Enter the catalog item description</li> <li>ENABLED - Select to enable the catalog item, making it available for provisioning (default). Clear to disable.</li> <li>FEATURED - Select to enable special visibility of this catalog item in the Service Catalog persona view. Clear to disable. Special visibility means that an item can be featured (a tag is added in the item) and given priority in the Service Catalog and Dashboard views.</li> <li>LOGO - From the drop-down list, do one of the following: <ul> <li>Select an existing logo</li> <li>Select custom and click Browse to locate and upload a logo</li> </ul> </li> <li>WORKFLOW - From the drop-down list, select the desired workflow</li> <li>CONTEXT TYPE - (Optional) From the drop-down list, select the context type: none,server or instance.</li> <li>CONTENT - Enter a more detailed description about the instance, which will display in the order screen</li> </ul> </li> </ul> <p><img src="/img/catalog-image9.png" alt=""></p> <h2>Accessing the Service Catalog persona</h2> <p>If your role’s default persona is set as “Service Catalog”, the link launching HPE GreenLake for private cloud dashboard card will open the Service Catalog persona dashboard.</p> <p>Otherwise, switch personas by clicking on your name in the upper-right corner of the application window. If your role gives you access to any additional personas, they will be listed here.</p> <p>The catalog shows the complete list of pre-defined catalog items available to the user for provisioning.</p> <p><img src="/img/catalog-image10.png" alt=""></p> <h2>Ordering a Service catalog item</h2> <p>From the Service Catalog page, select the tile for your chosen item to see any custom options that need to be set prior to provisioning.</p> <p>This example shows the ordering for the “Apache” catalog item created in the previous steps.</p> <p><img src="/img/catalog-image11.png" alt=""></p> <p>Based on the option types specified in the catalog item definition, custom options are displayed on the ordering page.</p> <p>Click <strong>Order Now</strong> to place the order immediately or click <strong>Add to Order</strong> and proceed to the cart.</p> <p>Click the cart to review and place the order.</p> <p><img src="/img/catalog-image12.png" alt=""></p> <p><img src="/img/catalog-image13.png" alt=""></p> <p>Click <strong>PLACE ORDER</strong> after reviewing the order.</p> <p><img src="/img/catalog-image14.png" alt=""></p> <p>The state of the order can be seen from the <strong>INVENTORY</strong> page.</p> <p><img src="/img/catalog-image15.png" alt=""></p> <p>Upon clicking the inventory item <strong>Demo_VM</strong>, the instance details page is opened.</p> <p><img src="/img/catalog-image16.png" alt=""></p> <p>The order is now complete. In this case, the instance has been deployed in just a few steps as compared to how it is done with the Standard persona.</p> <p><img src="/img/catalog-image17.png" alt=""></p> <h2>Ordering a Service catalog item for multiple resources</h2> <p>The example below shows the ordering process for a catalog item created for a workflow that will deploy Docker on multiple resources. On the <strong>Resource</strong> box, specify the list of instances to deploy the sample Docker install workflow and click on <strong>ORDER NOW</strong>.</p> <p><img src="/img/catalog-image18.png" alt=""></p> <p>The order is now complete.</p> <p><img src="/img/catalog-image19.png" alt=""></p> <p>Clicking on the order in the inventory list shows the detail of the execution.</p> <p><img src="/img/catalog-image20.png" alt=""></p> <h2>Summary</h2> <p>Use of the HPE GreenLake for private cloud Service Catalog persona improves the user experience with a simplified catalog where users can select and deploy instances or blueprints with a pre-defined configuration using just a few clicks and without presenting an overwhelming list of options. Hopefully, you found this tutorial helpful. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more posts on topics like this.</p>https://developer.hpe.com/mapr-for-predictive-maintenance/https://developer.hpe.com/mapr-for-predictive-maintenance/Tue, 15 Feb 2022 05:31:46 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">https://www.hpe.com/us/en/software/ezmeral-data-fabric.html</a></p> <p>This project is intended to show how to build predictive maintenance applications on MapR. Predictive maintenance applications place high demands on data streaming, time-series data storage, and machine learning. Therefore, this project focuses on data ingest with MapR Streams, time-series data storage with MapR-DB and OpenTSDB, and feature engineering with MapR-DB and Apache Spark.</p> <h2>Overview:</h2> <p>Predictive maintenance requires a cutting-edge data platform that handles fast streams of Internet of Things (IoT) data with the processing required for on-the-fly feature engineering and the flexibility required for data science and machine learning.</p> <h2>Ingesting Factory IoT Data</h2> <p>Predictive maintenance applications rely heavily on ingesting multiple data sources, each with its own format and throughput. MapR Streams can ingest data, regardless of format or speed, with standard Kafka and RESTful APIs.</p> <h2>Machine Learning on Factory IoT Data</h2> <p>The "predictive" aspects of predictive maintenance applications are usually realized through machine learning. Feature engineering is often considered the most important aspect of machine learning (as opposed to neural network design, for example). Feature engineering places high demands on the data layer because of the amount of data that IoT data streams generate. The tendency for failures to occur infrequently and without warning means vast amounts of raw time-series data must be stored. Not only must it be stored, but it must also be possible to retroactively update the lagging features necessary in order to label failures for the purposes of supervised machine learning. MapR-DB and Spark can work together to provide the capabilities required to put machine learning into practice for predictive maintenance.</p> <p>In summary:</p> <ul> <li>MapR Streams provide a convenient way to ingest IoT data because it is scalable and provides convenient interfaces.</li> <li>The integration of MapR DB with Spark provides a convenient way to label lagging features needed for predicting failures via supervised machine learning.</li> <li>Drill provides a convenient way to load ML data sets into Tensorflow for unsupervised and supervised machine learning</li> </ul> <h2>Implementation Summary</h2> <p>There are two objectives relating to predictive maintenance implemented in this project. The first objective is to visualize time-series data in an interactive real-time dashboard in Grafana. The second objective is to make raw data streams and derived features available to machine learning frameworks, such as Tensorflow, in order to develop algorithms for anomaly detection and predictive maintenance. These two objects are realized using two seperate data flows:</p> <ol> <li>The first flow, located on the top half of the image below, is intended to persist IoT data and label training data for sequence prediction and anomaly detection of time-series data in Tensorflow.</li> </ol> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/bi_pipeline.png" width="100%"> <ol start="2"> <li>The second flow, located on the bottom half, is intended to persist time-series IoT data in OpenTSDB for visualization in a Grafana dashboard.</li> </ol> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/ml_pipeline.png" width="100%"> <p>Put together, these data pipelines look like this. The APIs used for reading and writing data are shown in red.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/dataflow.png" width="100%"> <h2>Preliminary Steps</h2> <p>These steps explain how to set up this tutorial using the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">MapR Container for Developers</a> on macOS.</p> <h2>Allocate 12GB to Docker</h2> <p>This project requires a lot of memory. We recommend allocating 12GB RAM, 4GB swap, and 2 CPUs to the Docker Community Edition for macOS.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/docker_config.png" width="100%"> <h2>Start the MapR sandbox (Now known as the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/RunMapRContainerDevelopers.html">Development Environment for HPE Ezmeral Data Fabric</a>)</h2> <p>Download and run the <code>./mapr_devsandbox_container_setup.sh</code> script.</p> <pre><code class="language-console">git clone https://github.com/mapr-demos/mapr-db-60-getting-started cd mapr-db-60-getting-started ./mapr_devsandbox_container_setup.sh </code></pre> <h2>Run the <code>init.sh</code> script</h2> <p>Run the <code>init.sh</code> script to install Spark, OpenTSDB, Grafana, and some other things necessary to use the sample applications in this tutorial. SSH to the sandbox container, with password "mapr" and run the following commands. This should take about 20 minutes.</p> <pre><code class="language-console">ssh -p 2222 root@localhost wget https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/init.sh chmod 700 ./init.sh sudo ./init.sh </code></pre> <h2>Import the Grafana dashboard</h2> <pre><code class="language-console">sudo /opt/mapr/server/configure.sh -R -OT `hostname -f` sudo /opt/mapr/opentsdb/opentsdb-2.4.0/etc/init.d/opentsdb start </code></pre> <p>Open Grafana data sources, with a URL like <a href="http://maprdemo:3000/datasources/edit/1">http://maprdemo:3000/datasources/edit/1</a>, and add OpenTSDB as a new data source.</p> <p>Load the <code>Grafana/IoT_dashboard.json</code> file using Grafana's dashboard import functionality, and specify "MaprMonitoringOpenTSDB" as the data source, as shown below:</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_import.png" width="100%"> <hr> <h2>Predictive Maintenance Demo Procedure</h2> <p>For learning or debugging purposes you should run each of the following steps manually, but if you just want to see data show up in Grafana then just run <code>./run.sh</code>.</p> <h2>Step 1 - Simulate HVAC data stream:</h2> <p>We have provided a dataset captured from a real-world heating, ventilation, and air conditioning (HVAC) system in the <code>sample_dataset</code> folder. Run the following command to replay that HVAC stream to <code>/apps/factory:mqtt</code>.</p> <pre><code class="language-console">cat ~/predictive-maintenance/sample_dataset/mqtt.json | while read line; do echo $line | sed 's/{/{"timestamp":"'$(date +%s)'",/g' | /opt/mapr/kafka/kafka-*/bin/kafka-console-producer.sh --topic /apps/factory:mqtt --broker-list this.will.be.ignored:9092; sleep 1; done </code></pre> <h2>Step 2 - Save IoT data stream to MapR-DB:</h2> <p>In the next step, we'll save the IoT data stream to OpenTSDB so we can visualize it in Grafana, but in this step, we save that stream to MapR-DB so we can apply labels necessary for supervised machine learning, as discussed in Step 4.</p> <p>Run the following command to persist messages from stream <code>/apps/factory:mqtt</code> to MapR-DB table <code>/apps/mqtt_records</code>.</p> <pre><code class="language-console">/opt/mapr/spark/spark-*/bin/spark-submit --class com.mapr.examples.MqttConsumer ~/predictive-maintenance/target/predictive-maintenance-1.0-jar-with-dependencies.jar /apps/factory:mqtt /apps/mqtt_records </code></pre> <p>Run this command to see how the row count increases:</p> <pre><code class="language-console">/opt/mapr/drill/drill-*/bin/sqlline -u jdbc:drill: -n mapr select count(*) from dfs.`/apps/mqtt_records`; </code></pre> <h2>Step 3 - Save IoT data stream to OpenTSDB:</h2> <p>In this step, we save the IoT data stream to OpenTSDB so we can visualize it in Grafana.</p> <p>Update <code>localhost:4242</code> with the hostname and port of your OpenTSDB server before running the following command:</p> <pre><code class="language-console">/opt/mapr/kafka/kafka-*/bin/kafka-console-consumer.sh --new-consumer --topic /apps/factory:mqtt --bootstrap-server not.applicable:0000 | while read line; do echo $line | jq -r "to_entries | map(\"\(.key) \(.value | tostring)\") | {t: .[0], x: .[]} | .[]" | paste -d ' ' - - | awk '{system("curl -X POST --data \x27{\"metric\": \""$3"\", \"timestamp\": "$2", \"value\": "$4", \"tags\": {\"host\": \"localhost\"}}\x27 http://localhost:4242/api/put")}'; echo -n "."; done </code></pre> <p>After you have run that command you should be able to visualize the streaming IoT data in Grafana. Depending on where you have installed Grafana, this can be opened with a URL like <a href="http://maprdemo:3000">http://maprdemo:3000</a>:</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_screenshot_2.png" width="100%" align="center"> <h2>Step 4 - Update lagging features in MapR-DB for each failure event:</h2> <p>This process will listen for failure events on a MapR Streams topic and retroactively label lagging features in MapR-DB when failures occur, as well as render the failure event in Grafana. Update "<a href="http://localhost:3000">http://localhost:3000</a>" with the hostname and port for your Grafana instance.</p> <pre><code class="language-console">/opt/mapr/spark/spark-*/bin/spark-submit --class com.mapr.examples.UpdateLaggingFeatures ~/predictive-maintenance/target/predictive-maintenance-1.0-jar-with-dependencies.jar /apps/factory:failures /apps/mqtt_records http://localhost:3000 </code></pre> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/lagging_features_explanation.png" width="100%" align="center"> <p>This particular step probably says the most about the value of MapR. Consider this scenario: You have a factory instrumented by IoT devices reporting hundres of metrics per machine per second and you're tasked with the challenge of saving all that data until one day, often months into the future, you finally experience a machine failure. At that point, you have to retroactively go back and update all those records as being "about to fail" or "x days to failure" so that you can use that data for training models to predict those lagging features. That's one heck of a DB update, right? The only way to store all that data is with a distributed database. This is what makes Spark and MapR-DB such a great fit. Spark - the distributed processing engine for big data, and MapR-DB - the distributed data store for big data, working together to process and store lots of data with speed and scalability.</p> <h2>Step 5 - Simulate a failure event:</h2> <p>To simulate a device failure, run this command:</p> <pre><code class="language-java">echo "{\"timestamp\":"$(date +%s -d '60 sec ago')",\"deviceName\":\"Chiller1\"}" | /opt/mapr/kafka/kafka-*/bin/kafka-console-producer.sh --topic /apps/factory:failures --broker-list this.will.be.ignored:9092 </code></pre> <p>This will trigger the Spark process you ran in the previous step to update lagging features in the MapR-DB table "/apps/mqtt_records". Once it sees the event you simulated, you should see it output information about the lagging features it labeled, like this:</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/UpdateLagging_screenshot.png?raw=true" width="100%" align="center" alt="Update Lagging Features screenshot"> <h2>Step 6 - Validate that lagging features have been updated:</h2> <pre><code class="language-java">$ mapr dbshell find /apps/mqtt_records --where '{ "$eq" : {"_Chiller1AboutToFail":"true"} }' --f _id,_Chiller1AboutToFail,timestamp </code></pre> <p>Here are a few example commands to look at that table with <code>mapr dbshell</code>:</p> <pre><code class="language-java">$ mapr dbshell find /apps/mqtt_records --f timestamp find --table /apps/mqtt_records --orderby _id --fields _Chiller2RemainingUsefulLife,_id find --table /apps/mqtt_records --where '{"$gt" : {"_id" : "1523079964"}}' --orderby _id --fields _Chiller2RemainingUsefulLife,_id find --table /apps/mqtt_records --where '{"$gt" : {"timestamp" : "1523079964"}}' --fields _Chiller2RemainingUsefulLife,timestamp </code></pre> <p>Here's an example of querying IoT data records table with Drill:</p> <pre><code class="language-java">/opt/mapr/drill/drill-*/bin/sqlline -u jdbc:drill: -n mapr select * from dfs.`/apps/mqtt_records` limit 2; </code></pre> <h2>Step 7 - Synthesize a high-speed data stream:</h2> <p>This command simulates a high-speed data stream from a vibration sensor sampling once per 10ms.</p> <pre><code class="language-java">java -cp ~/predictive-maintenance/target/predictive-maintenance-1.0-jar-with-dependencies.jar com.mapr.examples.HighSpeedProducer /apps/fastdata:vibrations 10 </code></pre> <h2>Why are we simulating a vibration sensor?</h2> <p>Degradation in machines often manifests itself as a low rumble or a small shake. These unusual vibrations give you the first clue that a machine is nearing the end of its useful life, so it's very important to detect those anomalies. Vibration sensors measure the displacement or velocity of motion thousands of times per second. Analyzing those signals is typically done in the frequency domain. An algorithm called "fast Fourier transform" (FFT) can sample time-series vibration data and identify its component frequencies. In the next step, you will run a command the converts the simulated vibration data to the frequency domain with an FFT and raises alarms when vibration frequencies vary more than a predefined threshold.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/vibration_analysis.png?raw=true" width="100%" align="center" alt="vibration analysis"> <p>This demonstrates the capacity of MapR to ingest and process high-speed streaming data. Depending on the hardware, you will probably see MapR Streams processing more than 40,000 messages per second in this step.</p> <h2>Step 8 - Process high-speed data stream:</h2> <p>This will calculate FFTs on the fly for the high-speed streaming data, and render an event in Grafana when FFTs changed more than 25% over a rolling window. This simulates anomaly detection for a vibration signal. Update "<a href="http://localhost:3000">http://localhost:3000</a>" with the hostname and port for your Grafana instance.</p> <pre><code class="language-bash">/opt/mapr/spark/spark-*/bin/spark-submit --class com.mapr.examples.StreamingFourierTransform ~/predictive-maintenance/target/predictive-maintenance-1.0-jar-with-dependencies.jar /apps/fastdata:vibrations 25.0 http://localhost:3000 </code></pre> <h2>Step 9 - Visualize data in Grafana</h2> <p>By now, you should be able to see streaming IoT data, vibration faults, and device failures in the Grafana dashboard.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_screenshot.png" width="100%" align="center"> <h2>Step 10 - Explore IoT data with Apache Drill</h2> <p><a href="https://drill.apache.org/">Apache Drill</a> is a Unix service that unifies access to data across a variety of data formats and sources. MapR is the primary contributor to Apache Drill, so naturally, it is included in the MapR platform. Here's how you can use it to explore some of the IoT data we've gathered so far in this tutorial.</p> <p>Open the Drill web interface. If you're running MapR on your laptop then that's probably at <a href="http://localhost:8047">http://localhost:8047</a>. Here are a couple of useful queries:</p> <h2>Show how many messages have arrived cumulatively over days of the week:</h2> <pre><code class="language-sql">SELECT _day_of_week_long, count(_day_of_week_long) FROM dfs.`/apps/mqtt_records` group by _day_of_week_long; </code></pre> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/drill_query_1.png" width="100%"> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/drill_result_1.png" width="100%"> <h2>Count how many faults have been detected:</h2> <pre><code class="language-java">WITH x AS ( SELECT _id, _day_of_week_long, _Chiller1AboutToFail, ROW_NUMBER() OVER (PARTITION BY _Chiller1AboutToFail ORDER BY _id) as fault FROM dfs.`/apps/mqtt_records` ) SELECT * from x WHERE _Chiller1AboutToFail = 'true' and fault = 1; </code></pre> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/drill_query_2.png" width="100%"> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/drill_result_2.png" width="100%"> <p>Drill can also be used to load data from MapR-DB into data science notebooks. Examples of this are shown in the following section.</p> <h2>References for Machine Learning techniques for Predictive Maintenance</h2> <p>This tutorial focuses on data engineering - i.e. getting data in the right format and in the right place in order to take advantage of machine learning (ML) for predictive maintenance applications. The details of ML are beyond the scope of this tutorial but to better understand ML techniques commonly used for predictive maintenance, check out the provided Jupyter notebook for <a href="https://github.com/mapr-demos/predictive-maintenance/blob/master/notebooks/jupyter/LSTM%20For%20Predictive%20Maintenance-ian01.ipynb">LSTM predictions for About To Fail</a>. This notebook not only talks about how to use Long Short Term Memory (LSTM) but also how to generate a sample dataset with <a href="https://github.com/tdunning/log-synth">logsynth</a> that resembles what you might see in a real factory. This notebook is great because it explains how to experiment with LSTMs entirely on your laptop.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/lstm-about_to_fail-50.png?raw=true" width="100%" alt="LSTM About To Fail prediction"> <p>Here are some of the other notebooks included in this repo that demonstrate ML concepts for predictive maintenance:</p> <ul> <li><a href="https://github.com/mapr-demos/predictive-maintenance/blob/master/notebooks/jupyter/LSTM%20predictions%20for%20Remaining%20Useful%20Life.ipynb">LSTM predictions for Remaining Useful Life</a> shows how to train a point regression model using an LSTM neural network in Keras to predict the point in time when an airplane engine will fail.</li> <li><a href="https://github.com/mapr-demos/predictive-maintenance/blob/master/notebooks/jupyter/LSTM%20time%20series%20prediction%20from%20OpenTSDB.ipynb">LSTM time-series predictions from OpenTSDB</a> shows how to train a model to predict the next value in a sequence of numbers, which in this case, is a time-series sequence of numbers stored in OpenTSDB. This notebook also shows how to load training data into Tensorflow using the REST API for OpenTSDB.</li> <li><a href="https://github.com/mapr-demos/predictive-maintenance/blob/master/notebooks/jupyter/RNN%20time%20series%20prediction%20from%20OpenTSDB.ipynb">RNN time-series predictions from OpenTSDB</a> also shows how to train a model to predict the next value in a sequence of numbers, except it uses an RNN model instead LSTM.</li> <li><a href="https://github.com/mapr-demos/predictive-maintenance/blob/master/notebooks/jupyter/RNN%20predictions%20on%20MapR-DB%20data%20via%20Drill.ipynb">RNN predictions for MapR-DB data via Drill</a> also shows how to train a model to predict the next value in a sequence of numbers, except it reads time-series data from MapR-DB using Drill as a SQL engine.</li> </ul> <h2>StreamSets Demonstration</h2> <p>To give you an idea of what dataflow management tools do, I’ve prepared a simple StreamSets project that you can run on a laptop with Docker. This project demonstrates a pipeline that streams time-series data recorded from an industrial HVAC system into OpenTSDB for visualization in Grafana.</p> <p>Create a docker network to bridge containers:</p> <pre><code class="language-console">docker network create mynetwork </code></pre> <p>Start StreamSets, OpenTSDB, and Grafana:</p> <pre><code class="language-console">docker run -it -p 18630:18630 -d --name sdc --network mynetwork \ streamsets/datacollector docker run -dp 4242:4242 --name hbase --network mynetwork \ petergrace/opentsdb-docker docker run -d -p 3000:3000 --name grafana --network mynetwork \ grafana/grafana </code></pre> <p>Open Grafana at <a href="http://localhost:3000">http://localhost:3000</a> and login with admin / admin</p> <p>Add <a href="http://hbase:4242">http://hbase:4242</a> as an OpenTSDB datasource to Grafana. If you don’t know how to add a data source, refer to Grafana docs. Your datasource definition should look like this:</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_opentsdb_config.png" width="100%" align="center"> <p>Download the following Grafana dashboard file:</p> <pre><code class="language-console">wget https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/Grafana/IoT_dashboard.json </code></pre> <p>Import that file into Grafana. If you don’t know how to import a dashboard, see Grafana docs. The Grafana import dialog should look like this:</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_dashboard_config.png" width="100%" align="center"> <p>Download, unzip, and copy the MQTT dataset to the StreamSets container:</p> <pre><code class="language-console">wget https://github.com/mapr-demos/predictive-maintenance/raw/master/sample_dataset/mqtt.json.gz unzip mqtt.json.gz docker cp mqtt.json sdc:/tmp/mqtt.json </code></pre> <p>Open StreamSets at <a href="http://localhost:18630">http://localhost:18630</a> and login with admin / admin</p> <p>Download and import the following pipeline into StreamSets. If you don’t know how to import a pipeline, refer to StreamSets docs.</p> <pre><code class="language-console">wget https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/StreamSets/MQTT%20File%20Tail.json </code></pre> <p>You will see a warning about a missing library in the “Parse MQTT JSON” stage. Click that stage and follow the instructions to install the Jython library.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/streamSets_warning.png" width="100%" align="center"> <p>Finally, run the StreamSets pipeline.</p> <img src="https://raw.githubusercontent.com/mapr-demos/predictive-maintenance/master/images/grafana_streamset_animation.gif" width="100%" align="center"> <p>Hopefully, by setting up this pipeline and exploring StreamSets you’ll get the gist of what dataflow management tools can do.</p>https://developer.hpe.com/writing-deep-learning-tools-for-all-data-scientists-not-just-unicorns/https://developer.hpe.com/writing-deep-learning-tools-for-all-data-scientists-not-just-unicorns/Fri, 11 Feb 2022 15:45:26 GMT<p>Machine learning (ML) is exploding in popularity, and, as it does, ML tooling is frantically trying to keep up. Tools for everything you can imagine are popping up: data versioning, experiment tracking, model serving, and, yes, even tools to help ML run on Kubernetes. Although some projects are more popular than others, given the newness of this space, there are no clear winners yet and no one has yet established the perfect set of software to enable and accelerate ML.</p> <h2>Here’s the problem – Kubernetes</h2> <p>Kubernetes is one of the most important pieces of software produced in the last decade and one of the most influential open source projects ever. It was <em><strong>built by software engineers specifically for software engineers</strong></em> to orchestrate containers in the new cloud-native/DevOps paradigm and has completely revolutionized how applications are developed and how infrastructure is deployed and managed. Given the ease of being able to spin up and down a service, developers now have easier access to more GPU time, resulting in an increased popularity of computationally-demanding applications like deep learning (DL).</p> <p>As a result, however, the tools that have been developed to train DL models require data scientists to use and understand Kubernetes. If you look at the knowledge base of a data scientist versus a software engineer, you’ll understand why there are two different titles. The few whose skills overlap are considered unicorns… those that can be found are few and far between.</p> <center><img src="/img/updated-unicorns-picture-smaller.png"></center> <p>If you happen to be a Unicorn, congratulations! You can use your magic to wield Kubernetes and Deep Learning together and build something beautiful. The rest of us, though, get pretty annoyed when we need to dive deep into computer systems engineering before we can make progress developing new ML models. The crazy thing is, this hassle is completely avoidable! We just need to start developing <strong>ML tools built for data scientists, not software engineers</strong>.</p> <h2>Diving Deeper</h2> <p>Let’s take a quick look at <a href="https://www.kubeflow.org/">Kubeflow</a> to understand where ML tooling has gone wrong. Kubeflow started as an adaptation of how Google was running TensorFlow internally, as a tool that allowed <em>TensorFlow to run on Kubernetes</em>. This technology was <em>very</em> impactful, creating a much simpler way to use hardware managed by Kubernetes to do deep learning.</p> <p>That initial version of Kubeflow is now the Kubeflow component called <a href="https://www.kubeflow.org/docs/components/training/tftraining/">TFJob</a>. Without TFJob, running TensorFlow on Kubernetes would be miserable — you would need to specify a complex topology of containers, networking, and storage before you could even start writing your ML code. With TFJob, this is simplified, but, crucially, <em>it is not nearly simple enough</em>. To use TFJob, you need to:</p> <p><em><strong>Wrap your ML code up neatly in a container.</strong></em> This will be a clunky experience that will require you to package your code and upload it if you want to make changes. Docker is great, but this will slow down your development cycle significantly.</p> <p><em><strong>Write a Kubernetes TFJob manifest.</strong></em> This might not sound that intimidating, but for a data scientist not fluent in Kubernetes it can be a daunting task. To do this well, you’ll need to learn a lot about Kubernetes — a far cry from the Python that these scientists are used to. Let’s look at the <em>most simple version of this, from the Kubeflow docs</em>:</p> <pre><code class="language-yaml">apiVersion: kubeflow.org/v1 kind: TFJob metadata: generateName: tfjob namespace: your-user-namespace spec: tfReplicaSpecs: PS: replicas: 1 restartPolicy: OnFailure template: metadata: annotations: sidecar.istio.io/inject: "false" spec: containers: - name: tensorflow image: gcr.io/your-project/your-image command: - python - -m - trainer.task - --batch_size=32 - --training_steps=1000 Worker: replicas: 3 restartPolicy: OnFailure template: metadata: annotations: sidecar.istio.io/inject: "false" spec: containers: - name: tensorflow image: gcr.io/your-project/your-image command: - python - -m - trainer.task - --batch_size=32 - --training_steps=1000 </code></pre> <p>This configuration file is <strong>full</strong> of concepts that are foreign to most data scientists. Pods, replicas, sidecars, restart policies, Kubernetes APIs — all of this is confusing, complex, and detracts from our ability to focus on data science.</p> <p><em><strong>Learn the Kubernetes CLI.</strong></em> This is minor, but again navigating Kubernetes is not a trivial thing to figure out. Submitting jobs may be relatively straightforward, but seeing results, artifacts, and logs of experiments is unintuitive and clunky.</p> <p>If you think about where this all came from, it makes sense that it evolved as it did. As this is an adaptation of the technology that Google uses to train DL models, and Google (unlike the rest of the world) tends to have a lot of unicorns, running Deep Learning on Kubernetes for them wasn’t a big deal.</p> <p>But not everyone is a Google unicorn – many other highly-skilled data scientists simply bounce off of Kubeflow because Kubernetes is too far outside of their comfort zone.</p> <p>If you look at some of the other components of Kubeflow, you’ll find similar philosophies: <strong>MPIJob</strong>, <strong>PyTorchJob</strong>, and <strong>Katib</strong> all expect data scientists to work with Kubernetes concepts and APIs. All of them suffer from the exact same usability issues — most data scientists don’t want to dive into the weeds of how Kubernetes is orchestrating the hardware; they just want an easier way to train their models. They want tools that abstract away foreign concepts and let them communicate ML concepts succinctly.</p> <p>One fascinating thing about Kubeflow is that some of the components of Kubeflow have clearly figured this out and realized that we can do better! The best example is <strong>Kubeflow Pipelines</strong> (KFP). The core underlying technology of Kubeflow Pipelines is <a href="https://github.com/argoproj/argo">Argo Workflows</a>, which are very similar to TFJob, providing a way to declare workflows in Kubernetes. <strong>Kubeflow Pipelines</strong> goes the crucial extra step of providing a Domain Specific Language that allows data scientists to write pipelines <em>in Python</em>! The builders of KFP realized that building containers and writing Kubernetes manifests wasn’t how data scientists wanted to interact with their work, so they <em><strong>abstracted away the k8s</strong></em> and made a tool that data scientists love.</p> <h2>Where do we go from here?</h2> <p>The key to providing ML practitioners with tools they’ll actually use is to truly understand them – what they like and dislike about their workflows – and then enhancing what they like and slicing out what they don’t. ML tools should allow data scientists to accomplish more with less work. When developing tools for data scientists, it’s important to avoid thinking about them as software engineers, and instead build tools that allow ML people to build, train, and deploy ML models without needing to become DevOps experts.</p> <p>Abstractions here can help. Abstractions make it possible to perform high-performance data science on complex, modern infrastructure without needing to be a systems expert. Designing the right abstractions is not easy but it is crucially important to making modern ML more accessible, more convenient, and more cost-effective.</p> <h2>Consider Determined AI – a tool that speaks your language</h2> <p><a href="https://developer.hpe.com/platform/determined-ai/home/">Determined AI</a> accomplishes many of the same goals as a tool like Kubeflow — allowing scientists to build and train deep learning models on Kubernetes (or any hardware, really), but without expecting data scientists to master countless new technologies along the way.</p> <p>Compare a Determined configuration file to the TFJob configuration above:</p> <pre><code class="language-yaml">description: cifar10_pytorch hyperparameters: learning_rate: 1e-4 learning_rate_decay: 1e-6 layer1_dropout: 0.25 layer2_dropout: 0.25 layer3_dropout: 0.5 global_batch_size: 32 records_per_epoch: 50000 searcher: name: single metric: validation_error max_length: epochs: 32 entrypoint: model_def:CIFARTrial </code></pre> <p>This configuration accomplishes essentially the same goal — describing an ML training workflow. The big difference is that this configuration is <em>written in the language of data scientists</em>, with complicated infrastructure concepts abstracted away using terms they are comfortable with, like hyperparameters, epochs, metrics, etc.</p> <p>This means that users can do more with less. Instead of having to learn Kubernetes or configure a cluster of machines to work with Horovod, they simply need to install Determined and describe experiments with their own terms. Determined unlocks incredibly powerful tools like <a href="https://www.determined.ai/blog/faster-nlp-with-deep-learning-distributed-training/">distributed training</a>, <a href="https://www.determined.ai/blog/why-does-no-one-use-advanced-hp-tuning/">hyperparameter search</a>, and experiment tracking, without placing extra burden on the user to understand what is happening behind the scenes. Determined has carefully built <a href="https://www.determined.ai/blog/standardized-models-with-determined/">powerful abstractions that allow data scientists to focus on science</a>, and not engineering, systems, and infrastructure.</p> <p>To see it in action, <a href="https://docs.determined.ai/latest/tutorials/quick-start.html">start with our quick start guide</a>! Determined is open source, so you can see how we do it in our <a href="https://github.com/determined-ai/determined">GitHub repository</a>. To learn more about the issues surrounding today’s DL infrastructure tools and how Determined AI helps resolve them, view this replay of the HPE DEV Munch &#x26; Learn seminar on <a href="https://www.youtube.com/watch?v=ktZFLD-9qgw&#x26;list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">Golden Age of AI, Dark Ages of AI Infrastructure</a>.</p> <p>If you have any questions along the way, <a href="https://join.slack.com/t/determined-community/shared_invite/zt-cnj7802v-KcVbaUrIzQOwmkmY7gP0Ew">hop on our community Slack</a>; we’re happy to help!</p>https://developer.hpe.com/2022-February-01/https://developer.hpe.com/2022-February-01/Tue, 01 Feb 2022 13:00:00 GMThttps://developer.hpe.com/new-powershell-toolkit-available-for-managing-hpe-data-services-cloud-console/https://developer.hpe.com/new-powershell-toolkit-available-for-managing-hpe-data-services-cloud-console/Fri, 28 Jan 2022 10:27:53 GMT<p>Hewlett Packard Enterprise (HPE) recently introduced a new way to manage IT data storage resources: <a href="https://www.hpe.com/us/en/storage/data-services-cloud-console.html">Data Services Cloud Console</a> (DSCC) which takes a cloud-based approach to running on-prem data storage infrastructure and offers <a href="https://www.storagereview.com/news/hpe-alletra-cloud-native-high-performance-storage">an innovative way </a>to meet the management challenges of modern IT workloads. While DSCC initially added a Storage-as-a-Service layer on top of the new <a href="https://www.hpe.com/us/en/storage/alletra.html">HPE Alletra storage arrays,</a> the vision of DSCC goes well beyond this, encompassing a full range of data storage services, monitoring, planning and data protection.</p> <p>The DSCC architecture is built on an API-driven abstraction layer. The REST API was a key aspect of the service, enabling the linking of distributed components - of course across HPE, but also as a means to enable future customer and partner developers to create, consume, maintain and monitor various HPE data infrastructure services. And it enabled the development of this new toolkit you can download and use today!</p> <h2>The cloud advantage</h2> <p>HPE GreenLake and DSCC allows you to manage across multiple storage platforms and across multiple sites, providing the advantage of using a single UI to span across servers and storage. It also offers optimizations not possible when managing devices as single management endpoints. A fully functional REST API helps accomplish all standard management tasks, like deploying volumes to servers or configuring snapshots - all designed to enable Management-at-scale.</p> <p>But what about the administrator who simply wants to either use a CLI or write very simple scripts to deploy and configure resources? This is where the DSCC PowerShell Toolkit comes in, helping you manage your cloud storage like a pro with PowerShell.</p> <h2>How to implement the use of the REST API</h2> <p>In this post, I've outlined the steps required. To learn more about the DSCC REST API, check out the post <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">here</a>. First, you'll need to log into your <a href="https://common.cloud.hpe.com/">HPE GreenLake Portal</a>. Under the <em>Manage Account</em> option, you will see an option to configure API access.</p> <p><img src="/img/powershell-manage-account-img.png" alt=""></p> <p>You can then create your new API account which allows you to granularly control your scripting environment. You are allowed up to 5 API Client credentials per GreenLake user account, and each API Client credential can be separately managed. A Client credential is made up of the combination of a Client_ID and a Client_Secret, which can be used to communicate with the Authentication server to obtain a valid session token. To generate these credentials, click on the appropriate button:</p> <p><img src="/img/powershell-api-client-credentials-img.png" alt=""></p> <p>You will need to identify which Cloud you want to generate these credentials for, as well as a common name for the Credential for later identification.</p> <p><img src="/img/powershell-create-credentials-img.png" alt=""></p> <blockquote> <p>Note: In the above example image, the application selected is FLEETSCALE(US West). Please take a look at the article <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console">here</a> for the supported DSCC Application instance and the end-points for each DSCC regions.</p> </blockquote> <p>Once this has been created you will see a new screen that will show the Client_Id and Client_Secret. These will need to be recorded as once they are created and displayed to the user, there is no other way to retrieve them from the Cloud Console.</p> <p><img src="/img/powershell-credentials-created-img.png" alt=""></p> <p>Next, you need to download and install the DSCC PowerShell Toolkit. From any Microsoft Windows machine, open a PowerShell window, and type in the following commands:</p> <pre><code class="language-powershell">PS:> $PSTK = ‘[https://codeload.github.com/HPEDSCC-PowerShell-Toolkit-main.zip](https://codeload.github.com/HPEDSCC-PowerShell-Toolkit-main.zip%E2%80%99)’ PS:> $FOLDER = 'C:\Windows\System32\WindowsPowerShell\v1.0\Modules' PS:> invoke-webrequest -uri $PSTK -outfile “MyFile.zip" PS:> expand-archive -path “MyFile.zip" -DestinationPath $FOLDER </code></pre> <p>As you can see, this little set of lines downloads the PowerShell Toolkit, which can also be found <a href="https://github.com/HewlettPackard/HPEDSCC-PowerShell-Toolkit">here</a>. It also unzips that folder into the Windows Modules folder so that it can later be found.</p> <p>Once this download and install is complete, you can Import the Module into your current session, and then connect to your HPE GreenLake DSCC service using the following commands:</p> <pre><code class="language-powershell">PS:> Import-Module HPEDSCC PS:> Connect-DSCC –Client_id ‘IdHere’ –Client_secret ‘Sercrethere’ –GreenLakeType Dev –AutoRenew </code></pre> <p><img src="/img/powershell-codeblock-dscc-img1.png" alt=""></p> <p>The Connection command above is fairly straightforward, however there are a few items you may not recognize, such as the GreenLake Type. The GreenLake type can be <em>Dev</em>, <em>US</em>, <em>JP</em>, and <em>EU</em> depending on your cloud location, and using the <em><TAB></em> key will cycle through the valid values. Additionally the <em>AutoRenew</em> is a switch that is used to enable the PowerShell Toolkit to refresh the Token every 1 hour and 45 minutes as the Token will naturally expire every 2 hours. Without the –<em>AutoRenew</em> option, your PowerShell Toolkit session will simply time out at the 2 hour mark and you will need to run the connection command again. Once you close the PowerShell session, the Client credentials are lost, as they are not persistently stored.</p> <p>Once you have connected, you will notice that the returned data from each PowerShell Call is a true PowerShell Object, which can be explored. To get a list of all of the available commands, and then to get detailed help from any command, use the following commands.</p> <pre><code class="language-powershell">PS:> get-command –module HPEDSCC PS:> get-help New-DSCCInitiator –Detailed </code></pre> <p><img src="/img/powershell-codeblock-dscc-img2.png" alt=""></p> <p>As you can see, each command has complete <em>Help</em> defined for it that includes the parameters, as well as the valid values, for those parameters required, as well as multiple examples of the usage of each command with sample output from those commands.</p> <h2>Objects are deeper than they may appear</h2> <p>Let's talk about Objects. When a PowerShell command returns data, you are presented with a table of the many values that are most likely to be useful at a glance. However, most understand that PowerShell Objects are far deeper than they appear to be on the surface. To explore this, first you can cast the value of any returned object to a variable. You can initially only return one item instead of all items. But you can then dig into the value of any part of that object using a ‘dot’ notation. In many cases you will find that as you go deeper and deeper into the object, even more details emerge. Using the PowerShell Count feature, you can determine, as in the following example, that 8 records exist in the object, and you can walk through each object using square brackets as shown below:</p> <pre><code class="language-powershell">PS:> $DATA = Get-DSCCStorageSystem –DeviceType device-type1 PS:> $DATA.Count PS:> $DATA </code></pre> <p><img src="/img/powershell-codeblock-dscc-img3.png" alt=""></p> <p>The value here is, that to extract the value of any of these fields, you need not write any parser or process any data, you simply need to know the object layout. If you want to see the entire object in long form, you can use the following command to list out all of the fields instead of just the most common. Additionally, if you would rather read the details of an object in the same format as it comes from the REST API call, you can use the PowerShell built-in function to convert it to raw JSON format.</p> <pre><code class="language-powershell">PS:> $DATA | format-list </code></pre> <p><img src="/img/powershell-codeblock-dscc-img4.png" alt=""></p> <p>Here, the objects are rather deep and you can get significantly more information than a GUI would present to you. You can dig into each object deeper and deeper using standard ‘DOT’ notation as shown below</p> <pre><code class="language-powershell">PS:> $DATA\[3].SystemWWN PS:> ( $DATA\[3].SoftwareVersions ).fullVersion PS:> ( $DATA\[3].SoftwareVersions ).Components </code></pre> <p><img src="/img/powershell-codeblock-dscc-img5.png" alt=""></p> <p>In addition to this object manipulation, you can also filter your results to see only the details you are interested directly from the command line using the pipe operator. There is a special symbol in PowerShell that allows you to access the piped data that looks like this ‘$_’</p> <p>You can also work directly from the original command instead of casting the data to a variable. In the following example, you can limit the returned data to a single record using a search term such as an InitiatorID, or an IP Address, or you can limit the results to a sub-collection of results that that match a specific criteria such as iSCSI vs FC initiators. In the below example, you can see how using the ‘.count’ feature that the Get-DSCCInitiator command returns 202 results, however by filtering the results to only return those with Protocol type ‘ISCSI’ it decreases this count to 22. Running the same command without the ‘.count’ option returns the actual object collection.</p> <pre><code class="language-powershell">PS:> ( Get-DSCCInitiator ).count PS:> ( Get-DSCCInitiator ) | where { $_.protocol –like ‘ISCSI’ } ).count PS:> ( Get-DSCCInitiator ) | where { $_.protocol –like ‘ISCSI’ } ) </code></pre> <p><img src="/img/powershell-codeblock-dscc-img6.png" alt=""></p> <p>You can also limit the results to only those with values such as all Volumes with a Free Space greater than a specific value. You may also chain these pipelines to add additional filters as well. In this case, you may want to filter a Volume list by size, but then filter that list again by additional criteria. In the example below, you can see it's been filtered again to limit the results to a single Storage System ID.</p> <pre><code class="language-powershell">PS:> ( Get-DSCCStorageSystem –DeviceType device-type1 | Get-DSCCVolume ).count PS:> ( Get-DSCCStorageSystem –DeviceType device-type1 | Get-DSCCVolume | where { $_.sizeMiB –gt 524000 } ).count PS:> Get-DSCCStorageSystem –DeviceType device-type1 | Get-DSCCVolume | where { $_.sizeMiB –gt 524000 } </code></pre> <p><img src="/img/powershell-codeblock-dscc-img7.png" alt=""></p> <p>There may also exist the case where you may wish to see the raw JSON return that each call returns. The good news is that PowerShell Objects and JSON Objects are the same thing, and with slight formatting changes, you can use the following pipeline operation to convert any PowerShell Object to native JSON formatting.</p> <pre><code class="language-powershell">PS:> ( Get-DSCCStorageSystem –DeviceType device-type1 )[1] PS:> ( Get-DSCCStorageSystem –DeviceType device-type1 )[1] | ConvertTo-JSON </code></pre> <p><img src="/img/powershell-codeblock-dscc-img8.png" alt=""></p> <p>You may have noticed in the example that, when I make calls to Storage Systems I had to specify a DeviceType. This is because the RestAPI needs to treat the Primera/Alletra9K systems (Device-Type1) differently than Alletra6K/NimbleStorage systems (Device-Type2).</p> <h2>More tips and tricks</h2> <p>Another interesting trick of the toolkit is that many of the commands support discovery mode. For example, if you want to get the details of a specific storage system, you would run the command shown below. But how do you find the required Storage System ID? That’s easy – you run the command without the System ID specified, in which case it will discover all of the Storage Systems and return that data.</p> <pre><code class="language-powershell">PS:> Get-DSCCStorageSystem –DeviceType device-type1 PS:> Get-DSCCStorageSystem –DeviceType device-type1 –SystemId ‘2M2042059T’ </code></pre> <p><img src="/img/powershell-codeblock-dscc-img9.png" alt=""></p> <p>Another very useful tip is to accept pipeline input for some variables; i.e. the command to obtain a Storage Systems Controller Nodes require the System ID to query against; but If I want ALL of the Controller Nodes, I can discover them by using the first command to feed the system ID to the second command. Optionally you could request the Controller if you know the Storage System ID.</p> <pre><code class="language-powershell">PS:> Get-DSCCStorageSystem –DeviceType device-type1 –SystemId ‘2M2042059T’ PS:> Get-DSCCStorageSystem –DeviceType device-type1 –SystemId ‘2M2042059T’ | Get-DSCCController PS:> Get-DSCCStorageSystem –DeviceType device-type1 | Get-DSCCController </code></pre> <p><img src="/img/powershell-codeblock-dscc-img10.png" alt=""></p> <h2>Understanding the REST API more deeply</h2> <p>Now that you are an expert at using the Toolkit, let’s see how you can use it to help you better understand the RestAPI model, in addition to the embedded help, Examples, and JSON conversion.</p> <p>This would include a feature of all commands implemented as a switch called ‘-WHATIF’. By adding a ‘-whatif’ option to any command, you can see all the details of what is going to be sent to the RestAPI endpoint, including the URI, the Header, the method, connection type, and Body if one exists. Below is the example of this type of data.</p> <pre><code class="language-powershell">PS:> New-DSCCInitiator –address ‘deadbeefdeadbeeddeadbeef’ –hbaModel ‘LPe12002 –hostSpeed 10000 –ipAddress ’10.10.10.1’ –name ‘MyInit’ –protocol iSCSI –vendor ‘Emulex’ -whatif </code></pre> <p><img src="/img/powershell-codeblock-dscc-img11.png" alt=""></p> <p>With this information, you can replicate any call in either Python, CURL, or other language of your choice. The primary reference should always be the RestAPI documentation, as well as any SDK that is produced for the DSCC, but the PowerShell Toolkit can be used to provide a functional solution, which you could then turn into a true programming project.</p> <h2>Get the free toolkit today to manage your storage like a pro</h2> <p>The new HPE Data Services Cloud Console PowerShell Toolkit is a valuable innovation for managing the HPE Data Services Cloud Console. It leverages your familiarity with PowerShell to speed the management of HPE storage resources programmatically and at scale. You can download the toolkit for free <a href="https://github.com/HewlettPackard/HPEDSCC-PowerShell-Toolkit">here</a> on GitHub.</p>https://developer.hpe.com/hpe-dev-launches-its-munch-learn-technical-talks/https://developer.hpe.com/hpe-dev-launches-its-munch-learn-technical-talks/Thu, 27 Jan 2022 16:44:13 GMT<p>Here’s your opportunity to learn more about today’s most popular technologies and connect with some of the industry’s leading technical specialists. Held monthly by the HPE DEV community, the Munch &#x26; Learn technology talks are free, 60-minute sessions where subject matter experts (SMEs) offer valuable insights and take your questions. Each month we will present a different topic, ranging from using containers to develop code to how to deploy data fabrics in AI and machine learning applications. We’ll cover a variety of technologies, including open source, Kubernetes, and <a href="https://www.hpe.com/us/en/ezmeral.html" target="_blank">HPE Ezmeral software</a>. To view each month’s topic and catch replays of previous sessions, check out our <a href="/blog/munch-and-learn" target="_blank">Munch &#x26; Learn calendar</a>.</p> <p>At each monthly gathering, a technology expert is introduced by an industry luminary to the online audience. These are live sessions where you can bring questions and participate in the discussion. In keeping with the HPE DEV community’s fun and engaging atmosphere, during each session we share recipes and pictures of our favorite munchies on the <a href="https://hpedev.slack.com/archives/C01GVQUPM3P" target="_blank">#munch-and-learn channel</a> of our <a href="https://slack.hpedev.io" target="_blank">Slack workspace</a>. Attendees are able to not only get a better taste of new technologies, but also benefit by being able to try out some new recipes.</p> <h2>Harkening back to our very first sessions</h2> <p>For our first session, held on January 27, 2021, Ellen Friedman, principal technologist at HPE, hosted Ted Dunning, HPE Ezmeral Data Fabric CTO, to discuss the value of data fabric and how it can run, manage, control, and secure the apps, data, and IT that run your business – from edge to cloud. Starting at a very high-level, but quickly getting into details, they explored three examples of how data fabric has been used in production at scale. Details included how these and similar systems are constructed using data fabric primitives. Ted also touched on some of the details of how the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a> works. And we had the Slack channel open for folks to pose their questions. For details on this session, read <a href="/blog/exploring-data-fabric-and-containers-in-hpe-devs-new-munch-learn-monthly" target="_blank">Exploring Data Fabric and Containers in HPE DEVs new Munch &#x26; Learn monthly gatherings</a>. You can also catch a replay of the session <a href="https://youtu.be/qi6sTvu8osk?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY" target="_blank">here</a>.</p> <p>During our February meeting, Nigel Poulton, Kubernetes technology author, introduced Tom Phelan, HPE Fellow and Ezmeral Software Platform CTO at HPE. They shared their thoughts on container architectures and how to leverage the Kubernetes Container Orchestrator to deploy and manage stateful, as well as microservice-based, applications. Nigel was kind enough to donate several eBooks to reward five attendees for posing excellent questions and sharing pictures of their favorite munchy. You can view the replay of this session <a href="https://youtu.be/9PvKpe7yMpI?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY" target="_blank">here</a>.</p> <p>On March 24, 2021, Ellen Friedman hosted Doug Cackett, who led us all in a whiteboard discussion about how data is transformed to create meaningful business value. He covered why data science was important, automating decisions, how it works in the real world using some simple examples, and some of the reasons why a data scientist might want to finesse some of the data. You can catch the <a href="https://youtu.be/Inh6eXM0EbA?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">replay of this session</a> and also review the <a href="https://developer.hpe.com/uploads/media/2021/3/munchandlearn-3-chat-1617017930299.pdf">side chat conversations</a>.</p> <h2>Things start heating up</h2> <p>Daniel Feldman of HPE and Frederick Kautz of Sharecare joined us on April 21st for a discussion on how to build a foundation for zero trust using <a href="https://spiffe.io/">SPIFFE</a>, the secure production identity framework for everyone. As was pointed out, enterprise applications and other software services are increasingly running across multiple platforms spanning diverse data centers and public clouds, all in various domains. To truly secure them, many are turning to a Zero Trust model where nothing is taken for granted and every request is verified. Because Service identity and authentication is perhaps the most fundamental piece of building a Zero Trust environment, it’s important to get it right so as to not undermine later efforts to enable Zero Trust in one’s organization. Catch the replay <a href="https://youtu.be/G1ceKr16nn8">here</a>.</p> <p>On May 19th, Doug Cackett, an expert on the HPE Ezmeral Container Platform, returned to talk with us more about Data Science 101. Ellen Friedman joined as well as the session moderator, adding in interesting insights on different operational models used in the industry. If you missed the session, be sure to listen to the replay <a href="https://youtu.be/Va4tSr__Yok?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">here</a>.</p> <p>We were really excited to have Owen Garrett of Deepfence.io join us on June 30th to discuss the fundamentals of production microservices. This session was one of our most highly attended sessions so far, and understandably so. Owen ran us through a very interesting presentation on why microservices are important for architecting and deploying modern applications; pointing out their unique characteristics, what opportunities they offer, and the challenges they introduce. He provided a high-level introduction to some of the technologies encountered when deploying a microservices application (i.e. Containers, Kuberetes, CI/CD), and then proceeded to get into a bit more detail on Ingress Controllers and Service Meshes as they pertain to microservices apps. He finished off by discussing some of the technologies you might want to keep in mind to secure your microservices application. Listen to the replay of this fascinating session <a href="https://youtu.be/qyyxQU37ZyQ?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">here</a>.</p> <p>For the July 28th Munch &#x26; Learn session, Ellen Friedman and Ted Dunning once again joined us to explore more about how to make data consumable for real world data science. With as much data being gathered today, businesses are eager to make the most of it, but it can be challenging when you work with diverse and mismatched data sets and teams. Ted shared some interesting issues he's come across as well as how those situations were handled. They pointed out how data might often be initially collected for one purpose but hoped to be applied by a different team and how a little advance planning can make this possible. If you missed the talk, you can still take advantage of the replay <a href="https://youtu.be/4WKjRqflF7M">here.</a></p> <h2>Schooling on Kubernetes and Data Analytics</h2> <p>Heading into the fall, the HPE DEV team invited Nigel Poulton, reknowned author and K8s expert, to come back on August 25th to reprise his Kubernetes 101 session that he gave at the HPE Discover 2021 Edge-to-Cloud Conference. Nigel walked us through the basics of Kubernetes in one of our most highly attended sessions yet, which you can catch in this <a href="https://www.youtube.com/watch?v=PWVJKK1obKQ">replay</a>.</p> <p>For our September 22nd session, we focused on how organizations that have progressed along their digital transformation journey are now looking at the next set of workloads and processes to modernize. For many, this includes Data and Analytics. HPE Ezmeral data science experts, Matt Maccaux and Randy Thomasson, joined us to give their insights, discussing strategic best practices and offering helpful tips for real-world applications.</p> <h2>And they just keep coming<a href="https://hpe.zoom.us/meeting/register/tJcrduuuqzgsHNEC-u8l_Y86YeZLMxEDF5fP"></a></h2> <p>The year 2021 closed out with two additional Munch &#x26; Learn sessions; one on using HPE Ezmeral Unified Analytics and another on Redfish. Replays of those sessions are available through our <a href="https://developer.hpe.com/campaign/munch-and-learn">calendar page</a>. The Munch &#x26; Learn sessions proved quite popular, bring in over 1200 attendees over the course of the year. This success sparked the launch of a new set of technology talks which the HPE Developer team also offers on a monthly basis called Meetups. Whereas the Munch &#x26; Learn sessions bring in industry luminaries to cover emerging industry trends, the Meetups are opportunities to dig down deep into new technologies, inviting product experts to come in and talk on detail about a given subject. You can learn more about these meetups in this <a href="https://developer.hpe.com/blog/new-for-2022-hpe-dev-meetups/">blog post</a> and check the <a href="https://developer.hpe.com/campaign/meetups">schedule</a> for upcoming talks and video replays.</p> <p>Keep checking back to see what new topics will be covered for HPE DEV Munch &#x26; Learns on <a href="https://developer.hpe.com/campaign/munch-and-learn">our schedule</a> so you don’t miss any of these fun and engaging discussions. Better yet, subscribe to our <a href="https://developer.hpe.com/newsletter-signup">HPE DEV Newsletter</a>. In it, you’ll find information on our upcoming Munch &#x26; Learn sessions as well as learn about everything else that’s new.</p>https://developer.hpe.com/real-time-smart-city-traffic-monitoring-using-microservices-based-streaming-architecture-part-2/https://developer.hpe.com/real-time-smart-city-traffic-monitoring-using-microservices-based-streaming-architecture-part-2/Thu, 27 Jan 2022 09:56:33 GMT<pre><code class="language-json">"authorDisplayName": "Mathieu Dumoulin", "category": "machine-learning", "publish": "2017-01-10T06:00:00.000Z" </code></pre> <p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">https://www.hpe.com/us/en/software/ezmeral-data-fabric.html</a></p> <p><em><strong>Modern Open Source Complex Event Processing For IoT</strong></em></p> <p>This series of blog posts details my findings as I bring to production a fully modern take on Complex Event Processing, or CEP for short. In many applications, ranging from financials to retail and IoT applications, there is tremendous value in automating tasks that require one to take action in real time. Putting aside the IT system and frameworks that would support this capability, this is clearly a useful capability.</p> <p>In the <a href="https://developer.hpe.com/blog/better-complex-event-processing-at-scale-using-a-microservices-based-str/">first post of the series</a>, I explain how CEP has evolved to meet this requirement and how the requirements for CEP can be met in a modern big data context. In short, I present an approach based on the best practices of modern architecture, microservices, and Kafka-style stream messaging, with an up-to-date open source business rule engine.</p> <p>In this second part, I’ll get more concrete and work through a working example using the system I propose. Let’s get started.</p> <h2>Smart City Traffic Monitoring</h2> <p>We made a working demo for which the code will be released on the MapR GitHub. It is made to work on either the <a href="https://developer.hpe.com/blog/getting-started-with-spark-on-mapr-sandbox/">MapR Sandbox</a> or using a real MapR cluster.</p> <p>For this example, we’ll use a very simple “smart city” use case for traffic monitoring. In this case, we’ll model a single sensor that can measure the speed of cars that pass on it. Using such sensor data, it would be fairly easy to detect traffic jams in real time, and thus notify the police to take action much more quickly than they otherwise could.</p> <img src="/img/picture1.png" width="600"> <p>Some other types of use cases are easy to envision. We can add data from a public, real-time weather feed and connect to information panels along the roads and show an advisory to drivers without any human intervention. By combining road condition sensors with the weather data, we can provide advisory feedback to drivers about road conditions and warn the public very quickly. Furthermore, by adding historical accident data and using predictive analytics, we can imagine road safety measures that can be deployed in real time to areas with a higher probability of accidents.</p> <p>To be honest, I’m only scratching the surface of what this kind of smart road could do to make a commuter’s life both easier and safer while saving money for the city. But how would we build a prototype of such a system without it being hugely complicated and expensive?</p> <h2>How to Build a Rule-Engine Based CEP System</h2> <p>So we now have a proper, concrete target in mind. It turns out that if we decide to base our system around reading our data from a Kafka-style stream (i.e., persistent, scalable, and high performance), then we will naturally end up with a pretty cool, modern CEP microservice.</p> <p>The important point here is not to show how to build a super complicated enterprise architecture, but rather that, by making some simple technology choices and building our demo in a reasonable way, we <strong>naturally end up with</strong> this elegant, modern, and <strong>simple architecture</strong>.</p> <p>For simplicity’s sake, I have decided to implement my prototype on a MapR Sandbox<sup>*</sup>. This is because it will include the stream messaging system, MapR Event Store, which I can use through the Kafka 0.9 API with very little configuration and know it will work the same on a production MapR 5.1+ cluster.</p> <p><em><sup>*</sup>Use <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">The Development Environment for HPE Ezmeral Data Fabric</a>.</em></p> <p>Finally, it should be noted that an Apache Kafka cluster could be used as well with the same design and code, just with some additional work to get it up and running.</p> <p><em>High-Level Architecture View</em></p> <p><img src="/img/picture2.png" alt=""></p> <p>As shown in the diagram above, the flow is to have sensor data get aggregated to a producer gateway, which will forward the data to the stream with a topic named “data.” The data will be in JSON format so it is easy to manipulate, human readable, and easily sent to Elasticsearch as-is for monitoring with a Kibana dashboard.</p> <p>The consumer side will have two tasks: to read the data from the stream and host an instance of a KieSession where the rule engine can apply rules on the facts as they are added to it.</p> <p>Rules are edited in the Workbench GUI, a Java webapp that can be run on a Java application server such as <a target='\_blank' href='http://wildfly.org/'>WildFly</a> or <a target='\_blank' href='http://tomcat.apache.org/'>Tomcat</a>.</p> <p>Rules are fetched from the workbench by the consumer application using the appropriate methods provided by the Drools framework, which are entirely based on Maven Repository.</p> <p>We can now look at the proposed system in more detail in the following sections.</p> <p><strong>List of Technologies Used</strong></p> <p>The technology we’re going to use is as follows:</p> <ul> <li>MapR Sandbox 5.2</li> <li>The Java programming language (any JVM language is fine)</li> <li>The Jackson 2 library to convert to and from JSON</li> <li>MapR Event Store or Apache Kafka stream messaging system</li> <li>Wildfly 10 application server to host the Workbench</li> <li><a target='\_blank' href='https://www.drools.org/'>JBoss Drools</a> as our choice of OSS business rule engine</li> <li><a target='\_blank' href='https://github.com/tdunning/log-synth'>Log Synth</a> to generate some synthetic data for our prototype</li> <li><a target='\_blank' href='https://streamsets.com/'>Streamsets</a> 1.6 to connect MapR Event Store and Elasticsearch</li> <li>Elasticsearch 2.4 and Kibana 4 for monitoring</li> </ul> <p><em>Traffic Monitoring Prototype Architecture</em></p> <p><img src="/img/picture3.png" alt=""></p> <p>As I built this demo to run on the MapR Sandbox, I’m using instructions for CentOS 6.X, an open-source version of RHEL 6.X. Instructions for CentOS 7 are almost identical, and finding similar instructions for Ubuntu would be pretty straightforward and left up to the reader.</p> <p>To build the core of the traffic monitoring system, we’re going to need two basic parts:</p> <ul> <li>A program to feed sensor data into MapR Event Store/Kafka. This part will be using fake data modeled by a vehicle simulation coded with Log Synth. We’ll use the MapR <a target='\_blank' href='http://docs.confluent.io/2.0.0/kafka-rest/docs/index.html'>Kafka-rest</a> proxy implementation (just introduced with MEP 2.0) to add the data with Python.</li> <li>A JVM-language application that will read data from the stream and pass it to a KieSession. The minimal code to get this working is surprisingly small.</li> </ul> <p>To edit the rules, we deploy the Workbench on Wildfly 10, which is a fairly straightforward process. <a target='\_blank' href='http://betzelblog.blogspot.jp/2015/02/setting-up-drools-workbench-and.html'>Check this blog post</a> for instructions, or read the Drools Documentation. Installing Wildfly is pretty simple; <a target='\_blank' href='https://docs.wildfly.org/22/Installation_Guide.html'>see this article</a> for great instructions on how to install it as a service on Centos/RHEL (it’s for Wildfly, but the same instructions work for 9 and 10).</p> <p>We made a single configuration change to Wildfly. We changed the port to 28080 instead of 8080, as it is already used by the sandbox. Wildfly runs in standalone mode, so the configuration file is in WILDFLY_HOME/standalone/configuration/standalone.xml.</p> <p>For monitoring, we let the streaming architecture work for us. We use the <a target='\_blank' href='https://streamsets.com/products/dataops-platform/data-collector-engine/'>Streamset Data collector</a> to easily redirect the sensor data to Elasticsearch so that we can actually monitor the system with a nice dashboard with Kibana. To set up Streamsets with MapR Event Store requires some work with version 1.6, or from the <a target='\_blank' href='https://streamsets.com/documentation/datacollector/latest/help/#Install_Config/MapR-Prerequisites.html'>official Streamsets documentation</a>).</p> <p>Finally, installation and setup of Elasticsearch and Kibana is <a target='\_blank' href='https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7'>well documented on Centos/RHEL</a>.</p> <p>For production, all those parts can be readily separated to run on separate servers. They can be run on either cluster nodes or edge nodes. If it’s a MapR cluster, installing the MapR Client and pointing it to the cluster CLDB nodes will be all the configuration needed for full access to the streams. For an Apache Kafka cluster, refer to the <a target='\_blank' href='http://kafka.apache.org/documentation'>official Kafka documentation</a>.</p> <h2>Traffic Monitoring Prototype - How To</h2> <p><strong>Creating the Streams with maprcli</strong></p> <p>The first task is to create streams and topics for our application. To do this, it’s a best practice to create a volume for streams first. Acting as the user: <code>mapr</code>, type from the command line:</p> <pre><code class="language-python">maprcli volume create -name streams -path /streams maprcli stream create -path /streams/traffic -produceperm p -consumerperm p maprcli stream topic create -path /streams/traffic -topic data maprcli stream topic create -path /streams/traffic -topic agenda maprcli stream topic create -path /streams/traffic -topic rule-runtime </code></pre> <p>Note: MapR Event Store is more of a superset of Kafka than simply a clone. In addition to being faster, MapR Event Store can use all the advantages of the MapR Distributed File and Object Store, such as volumes (with permissions and quotas and so on) and replication. A cluster is not limited to just defining topics, but can define several streams that each can have several topics. Therefore, instead of a topic name, a MapR Stream has a <code>path:topic</code> notation. Here, our data stream’s full name is <code>“/streams/traffic:data”</code>.</p> <p><strong>Generating Fake Data</strong></p> <p>I used the <a target='\_blank' href='https://github.com/tdunning/log-synth'>Log-Synth</a> tool to generate data for this prototype. Log-Synth uses a schema combined with a Sampler class to generate data in a very flexible and simple manner.</p> <p>My Schema:</p> <pre><code class="language-json">[ {"name":"traffic", "class":"cars", "speed": "70 kph", "variance": "10 kph", "arrival": "25/min", "sensors": {"locations":[1, 2, 3, 4, 5, 6, 7,8,9,10], "unit":"km"}, "slowdown":[{"speed":"11 kph", "location":"2.9 km - 5.1 km", "time": "5min - 60min"}]} ] </code></pre> <p>The command to generate the data is:</p> <pre><code class="language-python">synth -count 10K -schema my-schema.json >> output.json </code></pre> <p>Data is generated one car at a time, and each data point is a reading at a sensor. The data will model a flow of cars driving at 70 km/h, arriving at a rate of 25 cars per minute. A slowdown will happen between km 2.9 and 5.1 where speed will be reduced to 11km/h 5 minutes to 60 minutes after the start of the simulation. This will be the traffic jam we wish to detect using our CEP system.</p> <p>The generated data is a file where each line is the resulting list of sensor measurements for a single car:</p> <pre><code class="language-bash">[{"id":"s01-b648b87c-848d131","time":52.565782936267404,"speed":19.62484385513174},{"id":"s02-4ec04b36-2dc4a6c0","time":103.5216023752337,"speed":19.62484385513174},{"id":"s03-e06eb821-cda86389","time":154.4774218142,"speed":19.62484385513174},{"id":"s04-c44b23f0-3f3e0b9e","time":205.43324125316627,"speed":19.62484385513174},{"id":"s05-f57b9004-9f884721","time":256.38906069213255,"speed":19.62484385513174},{"id":"s06-567ebda7-f3d1013b","time":307.3448801310988,"speed":19.62484385513174},{"id":"s07-3dd6ca94-81ca8132","time":358.3006995700651,"speed":19.62484385513174},{"id":"s08-2d1ca66f-65696817","time":409.25651900903136,"speed":19.62484385513174},{"id":"s09-d3eded13-cf6294d6","time":460.21233844799764,"speed":19.62484385513174},{"id":"s0a-1cbe97e8-3fc279c0","time":511.1681578869639,"speed":19.62484385513174}] </code></pre> <p>A reading has a sensor ID, a speed in meters per second, and a time delta from time 0 (the moment the simulation starts) in seconds.</p> <p>My producer code simply translates the readings into a list of sensor readings ordered by time, and I transform the speed into km/s and the time into a timestamp as milliseconds from epoch.</p> <p><img src="/img/picture4.png" alt=""></p> <p>Sending to the code onto the stream can be done one line at a time using standard producer code. The <a href="https://docs.datafabric.hpe.com/61/MapR_Streams/code_for_the_sample_java_producer.html">code in the sample Java producer</a> works just fine.</p> <p>Another exciting new possibility is to use the brand new <a target='\_blank' href='http://docs.confluent.io/2.0.0/kafka-rest/docs/index.html'>Kafka Rest Proxy</a>, which is also available on MapR from MEP 2.0 (MapR Ecosystem Pack). This means sensors can connect directly to Kafka from any language since HTTP-based REST API is a global standard.</p> <p><strong>Using the Workbench</strong></p> <p>We can login to the Workbench and login with an admin user (a user with the role “admin”) created with the <code>add-user.sh</code> script from Wildfly.</p> <p>Using the Workbench is beyond the scope of the article, but the general idea is to create an Organizational Unit and a Repository, and then create a project.</p> <p><img src="/img/picture5.png" alt=""></p> <p><strong>Data Objects</strong></p> <p>We will need to create facts for the rule engine to work with. The best practice for Drools is to use a separate Maven project for your data model. For the sake of simplicity, I created them right from the workbench.</p> <p><img src="/img/picture6.png" alt=""></p> <p>The Measurement is a super generic bean that models the sensor speed measurements. For the prototype, I kept it as simple as the raw data with a timestamp, a sensor ID, and a speed.</p> <p><img src="/img/picture7.png" alt=""></p> <p>The Sensor bean models the sensor itself and will have an ID and the average speed of all cars that it measures over a time window defined by our rules. This average speed will be used to trigger an alert for heavy traffic. The traffic String is to mark the current traffic level which can be “NONE”, “LIGHT” or “HEAVY.”</p> <p><img src="/img/picture8.png" alt=""></p> <p><strong>The Traffic Monitoring Rules</strong></p> <ul> <li>create sensors for new id</li> <li>detect heavy traffic</li> <li>detect light traffic</li> <li>detect normal traffic</li> <li>get average speed at the sensor</li> </ul> <p>The create sensors rule ensures that there are sensor objects available in memory. This is the fact we use to know the average speed at a certain sensor.</p> <p>The detect heavy traffic is the key rule we want to use to send an alarm to the police if heavy traffic is detected at a certain sensor.</p> <p><img src="/img/picture9.png" alt=""></p> <p>So when the average speed reaches 20 km/h or less, and the sensor isn’t already in the HEAVY traffic level, set the level to HEAVY and send an alert.</p> <p>This means we need to know the average speed. Here is the rule to compute it using the Drools rule DSL (domain-specific language):</p> <p><img src="/img/picture10.png" alt=""></p> <p>This is not rocket science! These rules illustrate rather clearly how making up simple but useful rules can be realistically left up to business analysts and developed separately from the whole stream and big data platform.</p> <p><strong>The Consumer Side</strong></p> <p>The consumer reads the data from the stream. The tutorial code in Java from the documentation is perfectly adequate. Jackson is used to convert the JSON to Measurement objects.</p> <p>The consumer has an instance of the KieSession, and each measurement is added to the session using <code>kieSession.insert(fact)</code> and followed by a call to <code>kieSession.fireAllRules()</code>, which triggers the algorithm to check if any rules match with the new state of the session given the new data.</p> <p>A channel, which is just a callback, is used to allow a rule to call a function “outside” of the <code>KieSession</code>. My Prototype uses this method to log the alert. In a production system, we could easily change the code to send an email, an SMS, or take some other action.</p> <p><img src="/img/picture11.png" alt=""></p> <p><strong>Importing the Rules into the Consumer Application</strong></p> <p>The way we get the rules into the running application is by fetching it from the Maven repository integrated into the Workbench.</p> <pre><code class="language-python">KieServices kieServices = KieServices.Factory.get(); ReleaseId releaseId = kieServices.newReleaseId( "org.mapr.demo", "smart-traffic-kjar", "1.0.0" ); KieContainer kContainer = kieServices.newKieContainer( releaseId ); KieSession kieSession = kContainer.newKieSession(); </code></pre> <p>So the question becomes, how does the call to <code>newReleaseId</code> know to fetch the artifact with our rules from the Maven repository in the Workbench?</p> <p>The answer is with the <code>~/.m2/settings.xml</code> file, where you add this information. We recommend to use the user mapr for everything in the sandbox, so the full path is: <code>/home/mapr/.m2/settings.xml</code></p> <pre><code class="language-xml">[mapr@maprdemo .m2]$ cat settings.xml &#x3C;?xml version="1.0" encoding="UTF-8"?> &#x3C;settings> &#x3C;servers> &#x3C;server> &#x3C;id>guvnor-m2-repo&#x3C;/id> &#x3C;username>admin&#x3C;/username> &#x3C;password>admin&#x3C;/password> &#x3C;/server> &#x3C;/servers> &#x3C;profiles> &#x3C;profile> &#x3C;id>cep&#x3C;/id> &#x3C;repositories> &#x3C;repository> &#x3C;id>guvnor-m2-repo&#x3C;/id> &#x3C;name>Guvnor M2 Repo&#x3C;/name> &#x3C;url>**http://127.0.0.1:28080/drools-wb-webapp/maven2/**&#x3C;/url> &#x3C;releases> &#x3C;enabled>true&#x3C;/enabled> &#x3C;updatePolicy>interval:1&#x3C;/updatePolicy> &#x3C;/releases> &#x3C;snapshots> &#x3C;enabled>true&#x3C;/enabled> &#x3C;updatePolicy>interval:1&#x3C;/updatePolicy> &#x3C;/snapshots> &#x3C;/repository> &#x3C;/repositories> &#x3C;/profile> &#x3C;/profiles> &#x3C;activeProfiles> &#x3C;activeProfile>cep&#x3C;/activeProfile> &#x3C;/activeProfiles> &#x3C;/settings> </code></pre> <p>The key information is in bold, which corresponds to the URL of the maven2 repository of the Drools workbench. This information can be copied and pasted from the pom.xml, which can be seen by using the repository view:</p> <p><img src="/img/picture12.png" alt=""></p> <p><img src="/img/picture13.png" alt=""></p> <p>So I just copy-pasted that and now everything works like magic.</p> <h2>Monitoring the Smart Traffic Prototype</h2> <p>We have one stream with data, and two other streams to monitor the rule engine internals. This is very easy with Drools because it uses Java Listeners to report on its internal state. We simply provide a custom implementation of the listeners to produce the data to a stream, and then use Streamsets to redirect everybody to Elasticsearch.</p> <p><strong>Elasticsearch Mappings</strong></p> <p>The mappings are defined in a small script I created:</p> <p><a target='\_blank' href='http://pastebin.com/kRCbvAkU'><a href="http://pastebin.com/kRCbvAkU">http://pastebin.com/kRCbvAkU</a></a></p> <p><strong>Streamsets for No-Code Stream To Elasticsearch Pipeline</strong></p> <p><img src="/img/picture14.png" alt=""></p> <p>Each stream has its own pipeline, where each looks like this:</p> <p><img src="/img/picture15.png" alt=""></p> <p>The Jython Evaluator adds timestamp information if necessary.</p> <h2>Running the Prototype</h2> <p>Start the consumer:</p> <p><img src="/img/picture16.png" alt=""></p> <p>Then start the producer:</p> <p><img src="/img/picture17.png" alt=""></p> <p>In my prototype, I added code to control the rate at which the data is sent to the stream to make the rules firing easier to see. 10,000 events are quite small for Drools and for MapR Event Store/Kafka, and so the whole demo would be over in less than a second. This is the meaning of the “-r 25” for 25 events per second rate.</p> <p>The dashboard looks like this:</p> <p><img src="/img/picture18.png" alt=""></p> <p>Once data starts to stream in:</p> <p><img src="/img/picture19.png" alt=""></p> <p>The traffic jam is quite visible now:</p> <p><img src="/img/picture20.png" alt=""></p> <p>As soon as a sensor average speed drops below 20 km/h, an alarm is fired:</p> <p><img src="/img/picture21.png" alt=""></p> <p>And the dashboard will display a count of “1”</p> <p><img src="/img/picture22.png" alt=""></p> <p>The simulation will continue and the two other sensors will go below 20 in turn for a total of 3 alarms fired.</p> <h2>In Conclusion</h2> <p>This project turned into a great illustration of the power and ease of streaming architectures using microservices. A common misconception of microservices is that they are limited to synchronous REST services. This is not the case at all. Our system is a backend system and so the microservices (the producer, the consumer, streamsets, and ES/Kibana) are all run asynchronously, dealing with the data as it comes off the stream.</p> <p>The project was fairly easy to build technically because every part is totally independent of the others, and can therefore be tested in isolation. Once the producer can send data to the stream properly, it doesn’t need to be tested ever again for issues that concern other parts of the system. Also, each part was added one at a time, making it easy to identify issues and fix them.</p> <p>In total, with very little original code, we could implement a working, useful system that would need very little change to be put into production. Only the producer needs to be customized for the particular sensor or data source.</p> <p>Rules are the most time-consuming and error-prone part. This is no different than if the project had been done using custom-coded Spark code. The win for an approach based on a rule engine such as Drools and the Drools Workbench is that the rules can be edited, tested, and improved independently of how the code runs on the cluster. The work in Workbench has no dependency on the system at all, as it is pulled in by the consumer application automatically.</p> <p>From a business value point of view, all the value is in the rules, assuming a stable production system. There is no reason for organizations not to take advantage of this quick edition capability to become ever more agile and responsive to evolving conditions for the benefit of their customers…and the bottom line.</p>https://developer.hpe.com/get-started-with-prometheus-and-grafana-on-docker-with-hpe-storage-array-exporter/https://developer.hpe.com/get-started-with-prometheus-and-grafana-on-docker-with-hpe-storage-array-exporter/Wed, 26 Jan 2022 16:00:00 GMT<p>With the recently released <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-enhancements-with-monitoring-and/ba-p/7158137">HPE Storage Array Exporter for Prometheus and HPE CSI Info Metrics Provider for Prometheus</a>, it's a good time to familiarize ourselves with the cloud native technologies involved and get some first-hand experience.</p> <p>Prometheus is a time-series database that also provides monitoring and alerting. It's a <a href="https://www.cncf.io/projects/">CNCF graduated project</a>. Grafana is a web-based visualization tool that uses time-series data to create beautiful graphs and elements to present data in views referred to as dashboards. Prometheus scrapes an HTTP endpoint of a target periodically to consume an assortment of metric types and and metadata. The target is usually referred to as an exporter and the data being scraped is the current state of one or many exporter data points.</p> <p><img src="/img/prometheus-1.0.0-reva.png" alt="Prometheus Overview"></p> <p>In this tutorial we'll learn how to deploy Prometheus, Grafana and the HPE Storage Array Exporter for Prometheus using nothing but Docker. A supported storage backend, such as a HPE Alletra, Nimble Storage, Primera or 3PAR is needed to gather metrics for visualization. These prerequisites are assumed along with basic Docker and container knowledge. While this is a highly opinionated tutorial, the different components may be run standalone on Windows or Linux and may also be deployed directly on Kubernetes with Helm.</p> <p><strong>Note:</strong> This tutorial is purely for educational purposes and not intended for production deployments.</p> <h1>Get started</h1> <p>This tutorial assumes Docker is installed and functional. Credentials are also needed for a backend system. HPE Nimble Storage is being used in the examples and metrics may be named differently when using something else. Still, the principles remain the same.</p> <p>To allow the containers to communicate with each other by name, a separate bridge network is created.</p> <pre><code class="language-markdown">docker network create prometheus </code></pre> <p>Note that all containers created below do not utilize any persistent storage at all. Any dashboards that have been created or metrics that have been gathered will not be accessible after a container restart.</p> <h1>Run HPE Storage Array Exporter for Prometheus</h1> <p>Let's begin by starting the HPE Storage Array Exporter for Prometheus container. It's a passive background process that responds to HTTP requests made by Prometheus. It also requires a backend configuration file. In the case of having multiple backends, one container and configuration file is needed for each one.</p> <p>Create a file named <code>my-array-1.yaml</code>:</p> <pre><code class="language-yaml">address: my-array-1 username: admin password: admin </code></pre> <p>The <code>address</code> value can be an IP address or hostname resolvable in DNS. Then, run <code>docker</code>:</p> <pre><code class="language-markdown">docker run -d --rm --name my-array-1 \ -v $(pwd)/my-array-1.yaml:/etc/config/storage-system.yaml \ --network prometheus \ quay.io/hpestorage/array-exporter:v1.0.0 \ --log.path /var/log/hpe-array-exporter.log \ --accept-eula \ /etc/config/storage-system.yaml </code></pre> <p>By running the above command with the <code>--accept-eula</code> parameter, the end-user accepts the HPE End User License Agreement at <a href="https://www.hpe.com/us/en/software/licensing.html">https://www.hpe.com/us/en/software/licensing.html</a>.</p> <p>The exporter is now up and running listening on port 8080. To ensure we can reach the metrics endpoint, run the following <code>docker</code> command:</p> <pre><code class="language-markdown">docker run --rm --network prometheus \ curlimages/curl -s http://my-array-1:8080/metrics </code></pre> <p>It should produce an output that enumerates the metric keys, value, type and brief help. An example:</p> <pre><code class="language-markdown"># HELP hpenimble_pool_snapshot_used_bytes Total logical size of all snapshots in a storage pool # TYPE hpenimble_pool_snapshot_used_bytes gauge hpenimble_pool_snapshot_used_bytes{pool="default"} 2.659803136e+09 </code></pre> <p>Next, let's start Prometheus and scrape some metrics.</p> <h1>Run and Inspect Prometheus</h1> <p>When Prometheus is deployed on Kubernetes, discovery of scrape targets may be configured to be performed dynamically. In this example we need to instruct Prometheus where to find our target. Create a configuration file named <code>prometheus.yml</code>:</p> <pre><code class="language-yaml">global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: "my-array-1" static_configs: - targets: ["my-array-1:8080"] </code></pre> <p>Create the Prometheus container:</p> <pre><code class="language-markdown">docker run -d --rm --name prometheus-server \ --network prometheus \ -p 80:9090 \ -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus:v2.30.3 </code></pre> <p>In a few moments, the Prometheus server should be up and listening on the Docker host and accessible from <a href="http://your.docker.host.example.com/">http://your.docker.host.example.com/</a> or <a href="http://localhost">http://localhost</a>.</p> <p>To inspect if metrics are being collected properly, access the web interface and start typing <em>hpe</em> in the query bar.</p> <p><img src="/img/prometheus-auto.png" alt="Prometheus Query"></p> <p>Select a <code>volume_reads_per_second</code> metric, hit <em>Execute</em> and click <em>Graph</em>. This should produce a filled line graph with all the volumes on the array and their read IOPS. Wait a few minutes if there's no data being presented.</p> <p><img src="/img/prometheus-graph.png" alt="Prometheus Graph"></p> <p>Next, we'll bolt on Grafana and connect to the Prometheus data source.</p> <h1>Run Grafana and Build Your First Dashboard</h1> <p>In this tutorial, we'll just use the built-in dashboards and plugins that come with Grafana. Accessing Grafana is mainly done through the intuitive web UI found on port 3000. Let's start it up:</p> <pre><code class="language-markdown">docker run --rm -d --name grafana \ --network prometheus \ -p 3000:3000 \ grafana/grafana:8.2.2 </code></pre> <p>Next, navigate to the web UI (<a href="http://your.docker.host.example.com:3000/">http://your.docker.host.example.com:3000/</a> or <a href="http://localhost:3000">http://localhost:3000</a>) and log in with user "admin" and password "admin". You'll be prompted to create a new password.</p> <p>Now, a data source needs to be added. A panel to do so is available on the main screen.</p> <p><img src="/img/grafana-welcome.png" alt="Add new data source"></p> <p>Next, select "Prometheus".</p> <p><img src="/img/grafana-select.png" alt="Select Prometheus"></p> <p>In the HTTP section of the configuration, enter <code>http://prometheus-server:9090</code>. Click <em>Save &#x26; test</em> at the bottom of the page. A green checkmark should be displayed and indicate that the data source is working.</p> <p><img src="/img/grafana-server.png" alt="Configure Prometheus data source"></p> <p>At this point, Grafana is able to retrieve data series from Prometheus and dashboards may now be created.</p> <p>Hit the big <strong>+</strong> sign on the right, click <em>Dashboards</em> and <em>Add an empty panel</em>.</p> <p>Creating a graph involves adding a query from the data source. The field uses auto-complete, so start typing <em>hpe</em> and the field will populate. In the example below, a simple <em>sum</em> operation is performed on the <code>hpenimble_volume_reads_per_second_avg5m</code> metric which produces a combined read IOPS performance graph. The axis itself has been customized to show <em>io/s</em>.</p> <p><img src="/img/grafana-panel.png" alt="A simple Grafana dashboard"></p> <p>This concludes this introductory tutorial. Next, the possibilities are endless. Use cases include full stack monitoring of applications and the underlying infrastructure. Setting up thresholds for alarms with different outputs, such as email or Slack. Project growth trends for both capacity and performance for your storage estate. Please find additional resources below on how to learn more about Prometheus, Grafana and the HPE Storage Array Exporter for Prometheus.</p> <h1>Where to go next?</h1> <p>I've only touched the tip of the iceberg in this tutorial to inspire deeper learning. Both Prometheus and Grafana are rich ecosystems that require their own exploration to better understand the fundamentals.</p> <p>It's also important to highlight again that this does not reflect a production deployment. It's more common to deploy and manage Prometheus on Kubernetes. The same patterns also apply for Grafana. The HPE Storage Array Exporter for Kubernetes may also be deployed on Kubernetes along with the HPE CSI Info Metrics Provider for Prometheus (Container Storage Interface) to give a holistic view of stateful workloads utilizing persistent storage from any of the supported HPE storage backends.</p> <ul> <li>Introduction to <a href="https://prometheus.io/docs/introduction/overview/">Prometheus</a></li> <li>Get started with <a href="https://grafana.com/docs/grafana/latest/getting-started/getting-started/">Grafana</a></li> <li>Learn more about <a href="https://hpe-storage.github.io/array-exporter/">HPE Storage Array Exporter for Prometheus</a></li> <li>Deploy <a href="https://scod.hpedev.io/csi_driver/metrics.html">HPE CSI Info Metrics Provider for Prometheus</a> with the HPE CSI Driver for Kubernetes</li> <li>Explore the preconfigured HPE Storage dashboards on <a href="https://grafana.com/orgs/hpestorage/dashboards%5D">grafana.com</a>.</li> </ul> <p>Watch this space for future updates on monitoring and alerting for HPE Storage using the aforementioned technologies. Don't forget that you can reach the team on the HPE DEV Slack. Sign up <a href="https://slack.hpedev.io">here</a> and log in at <a href="hpedev.slack.com">https://hpedev.slack.com</a>.</p>https://developer.hpe.com/2022-January-01/https://developer.hpe.com/2022-January-01/Fri, 07 Jan 2022 13:00:00 GMThttps://developer.hpe.com/new-for-2022-hpe-dev-meetups/https://developer.hpe.com/new-for-2022-hpe-dev-meetups/Mon, 03 Jan 2022 17:24:41 GMT<p><img src="/img/gettyimages-1213470242_v2_1600_0_72_rgb.jpg" alt=""></p> <p>The HPE Developer team is happy to announce another opportunity to come and learn with us in 2022 called Meetups. These sessions will complement the continuing HPE DEV <a href="https://developer.hpe.com/campaign/munch-and-learn">Munch &#x26; Learn Technology Talks</a> that started in 2021.</p> <p>The Munch &#x26; Learn Technology Talks were designed to provide industry thought-leadership insights on today's newest technologies. They proved quite successful with a grand total of 1,415 participants attending the 11 sessions that were held this past year. Participants of these sessions indicated that, while they enjoyed and wished to continue attending Munch &#x26; Learn sessions, they also needed an opportunity to dive in deep and more fully explore specific projects or products. The new Meetups are designed to fill that gap, covering topics that sit at the intersection of development and open source. The cadence for these meetups will be monthly, occurring the last week of every month.</p> <p>The first meetup was held on January 26th, 5PM CET (8AM PST). Entitled <strong>Quarkus - Supersonic Subatomic Java</strong>, it was delivered by <strong>Dimitris Andreadis</strong>, Engineering Director at Red Hat. The next session was held on February 23rd and covered <strong>Streamlit - the fastest way to build and share data science apps.</strong> The session was delivered by <strong>Arnaud Miribel</strong>, a machine learning engineer at Streamlit.io. You can catch replays of these sessions and see what's coming next in our <a href="https://developer.hpe.com/campaign/meetups">Meetups calendar</a>. As you can see, we are already off to a good start and will continue to look for the best possible speakers inside and outside of HPE to cover the most relevant topics for developers and open source enthusiasts.</p> <p>Please check our <a href="https://developer.hpe.com/campaign/meetups/">Meetup calendar page</a> for the complete list of registration links. We will continue to post links to replays once they are available. We hope you will join us and continue to expand your skills and expertise.</p>https://developer.hpe.com/massive-parallel-management-with-ilorest-on-linux/https://developer.hpe.com/massive-parallel-management-with-ilorest-on-linux/Mon, 03 Jan 2022 14:57:18 GMT<p>Thanks to HPE having provided the <a href="http://hpe.com/info/restfulapi">iLOrest Python API</a>, an entirely new avenue has been opened to developers for managing their servers. It allows them to add in their favorite deployment tools (i.e. <a href="https://hackshack.hpedev.io/workshop/23">Ansible</a>) or other management tools, like <a href="http://hpe.com/info/oneview">HPE OneView</a> (as described in several articles found here, in the <a href="https://developer.hpe.com/blog/">HPE DEV Blog</a>).</p> <p>But what if you are not a Python geek? What if you prefer managing your servers the old way with <code>bash</code> / <code>grep</code> / <code>awk</code> / <code>jq</code> / <code>curl</code> / <code>wget</code>? In that case, you can loop over the list of your iLO IP addresses and use <code>curl</code> / <code>wget</code> for getting and setting Redfish parameters.</p> <p>However, this loop approach brings out two fundamental problems: 1) Using <code>curl</code> and <code>wget</code> in management scripts implies the creation of smart crawling functions (as explained in this <a href="https://developer.hpe.com/blog/getting-started-with-ilo-restful-api-redfish-api-conformance">article</a>), 2) a sequential management with loops is fine for taking care of five or ten servers, but not more. A parallel approach should be considered for an important number of managed servers.</p> <p>This blog post explains how to manage many servers in parallel using bash scripts without the need of implementing schema crawlers, thus allowing a long-term stability to the scripts.</p> <p>To achieve this goal, you will combine a parallel task launcher (namely the <a href="https://clustershell.readthedocs.io/en/latest/index.html">ClusterShell</a> and/or the Parallel Distributed Shell) with <a href="https://www.hpe.com/info/resttool">iLOrest</a> (former <code>hprest</code>) the HPE RESTful Interface Tool in a more detailed and slightly different manner than what you can read already in the <a href="https://hewlettpackard.github.io/python-redfish-utility/#overview">user guide</a>.</p> <h2>Clush and pdsh: Quick (and dirty) overview</h2> <p>The Parallel Distributed Shell (<code>pdsh</code>) and the ClusterShell (<code>clush</code>) are both part of the <a href="https://fedoraproject.org/wiki/EPEL">Extra Package for Enterprise Linux</a> (EPEL) repository and they give the ability to perform local or remote commands in parallel. <code>Pdsh</code> came first as a replacement of an IBM utility (DSH) with Google being the main contributor. However, Google pulled out from this project in 2013.</p> <p>The ClusterShell is an interesting alternative to <code>pdsh</code> for many reasons. First the basic syntax and most of the switches, options and arguments are identical to <code>pdsh</code>. Hence, moving from pdsh to clush is easy. Additionally, <code>clush</code> provides more flexibility for selecting the remote nodes you want to execute commands against or on. It is associated with the <a href="https://clustershell.readthedocs.io/en/latest/tools/nodeset.html"><code>nodeset(1)</code></a> command for computing advanced nodeset operations like <code>expand</code>, <code>fold</code>, <code>group</code>, etc. Another advantage of <code>clush</code> over <code>pdsh</code> is that it is written in Python and can be used in Python scripts whenever you are ready to use this language.</p> <p>For the purposes of this article, I will use ClusterShell only.</p> <p>For those systems running Red Hat or a similar distribution, you need to mention the two packages containing the code and the eco-system:</p> <pre><code class="language-bash">sudo dnf install python3-clustershell clustershell </code></pre> <p>On Ubuntu systems, just mention the <code>clustershell</code> parameter on the command line to install both packages:</p> <pre><code class="language-bash">sudo apt install clustershell </code></pre> <p>Here are basic examples to illustrate the launch of several commands in parallel on the local node and then one command on several remote nodes.</p> <p>If you want to perform a parallel <code>ping</code> toward <code>node1</code>, <code>node2</code> and <code>node3</code> type:</p> <pre><code class="language-bash">clush -R exec -w node[1-3] ping %h </code></pre> <p><code>-R exec</code> specifies a local execution of the <code>ping</code> command toward the list of hosts: <code>node[1-3]</code>. <code>%h</code> is substituted with <code>node1</code>, <code>node2</code> and <code>node3</code> for each <code>ping</code> command. When you hit the return key, your system forks in parallel the following 3 tasks:</p> <pre><code class="language-bash">ping node1 ping node2 ping node3 </code></pre> <p>Imagine now that you want the root user of <code>node1</code>, <code>node2</code> and <code>node4</code> to <code>ping</code> a remote gateway called <code>aremotegateway</code>. With the above syntax you can issue:</p> <pre><code class="language-bash">clush -R exec -w node[1-2,4] ssh -l root %h "ping aremotegateway"` </code></pre> <p>However, EPEL provides <code>clush</code> compiled with an <code>ssh</code> (default) module, allowing the following simplified syntax:</p> <pre><code class="language-bash">clush -w node[1-2,4] -l root "ping aremotegateway"` </code></pre> <p>Since the <code>-R</code> switch is not present, the <code>ssh</code> default module is used toward <code>node[1-2,4]</code> and the <code>ping</code> command is executed on this list or remote nodes.</p> <h2>Nodeset management</h2> <p>The list of remote nodes following the <code>-w</code> switch can be the output of a <code>nodeset</code> command taking its input from a node set configuration file. The possible locations of the configuration file depends on the <code>groups.conf(5)</code> file. For this article, I populated the default file located in my home directory (<code>~/.local/etc/clustershell/groups.d/local.cfg</code>).</p> <p>Here is one example showing the power of this <code>clush</code> companion command.</p> <p>Imagine you define two groups of iLOs in the nodeset configuration file:</p> <pre><code class="language-bash">ilo-sales: ilo-sales[1-10] ilo-marketing: ilo-foo,ilo-bar,ilo-test </code></pre> <p>The <code>ilo-sales</code> group contains ten iLO names and ilo-marketing group contains three. If you want to select both groups and exclude node <code>ilo-test</code>, you can do it with:</p> <pre><code class="language-bash">nodeset --fold @ilo-sales,@ilo-marketing --exclude ilo-test </code></pre> <p><img src="/img/1-nodeset-exclude.png" alt="nodeset exclude example" title="nodeset exclude example"></p> <p>You can now combine the <code>nodeset</code> command with <code>clush</code> to issue a parallel <code>ping</code> toward this list with:</p> <pre><code class="language-bash">clush -R exec -w $(nodeset -f @ilo-sales,@ilo-marketing -x ilo-test) ping -c 2 %h </code></pre> <h2>Out-of-band or in-band: that is the question</h2> <p>As discussed in this <a href="https://developer.hpe.com/blog/chif-driver-not-found/">blog</a> post, iLOrest offers both out-of-band and in-band management. In the first form, you need to log into a remote iLO with the credentials of an iLO user. The following sequence of commands retrieves the Bios version of a remote server using an out-of-band management:</p> <pre><code class="language-bash">ilorest login ilo-IP -u ilo-user -p ilo-password ilorest select computersystem. ilorest ls oem/hpe/bios/current # On Gen9, replace with bios/current ilorest logout </code></pre> <blockquote> <p>Note: You can always use out-of-band management even when the server is OFF or when there is no operating system installed. The iLO constantly listens on the network as long as a power cable is plugged in the server.</p> </blockquote> <p>Performing in-band management with iLOrest means that you need to first access the operating system before launching the local <code>ilorest</code>. If you login as a privileged user (root or Administrator), you don't need to supply any username/password credentials to access the underlying iLO:</p> <pre><code class="language-bash">ssh root@server ilorest login ssh root@server ilorest select computersystem. ssh root@server ilorest ls oem/hpe/bios/current # On Gen9 replace with bios/current ssh root@server ilorest logout </code></pre> <h2>Who kicked me out?</h2> <p>All of this is awesome except that in both examples (out-of-band and in-band), a cache directory location is not explicitly specified. Hence, you will be kicked out if a concurrent iLOrest session is opened using the default cache directory as shown in the following picture.</p> <p>The picture below shows the following sequence of events: (1) you first log into ilo-lab1. (2) Then you select the <code>Bios.</code> type (select <code>HpBios.</code> on Gen9 servers) and (3) verify that it has been selected.</p> <p>(4) In the other terminal session and from the same server, log into ilo-lab2 without selecting any type. This second session clears off the iLOrest cache including the selection we performed in the first terminal. As a result, (5) asking for the resource type selection returns an error.</p> <p><img src="/img/2-kicked-out.png" alt="Demonstrating iLOrest local cache" title="Demonstrating iLOrest local cache"></p> <p>As a consequence (and to be safe), you should always specify a specific cache directory location for your iLOrest sessions which will avoid such contention. One possible solution is to generate a random string of ASCII characters and use it as a location for the iLOrest cache.</p> <p>With a random string of 16 characters (lowercase, uppercase, digits) for a cache directory name, the out-of-band example becomes:</p> <pre><code class="language-bash">UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 16) iLOrest="ilorest --cache-dir=/tmp/$UUID" $iLOrest login ilo-IP -u ilo-user -p ilo-password $iLOrest select computersystem. $iLOrest ls oem/hpe/bios/current # On Gen9, replace with bios/current $iLOrest logout </code></pre> <p>And the in-band example:</p> <pre><code class="language-bash">UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 16) iLOrest="ilorest --cache-dir=/tmp/$UUID" ssh root@server $iLOrest login ssh root@server $iLOrest select computersystem. ssh root@server $iLOrest ls oem/hpe/bios/current # On Gen9, replace with bios/current ssh root@server $iLOrest logout </code></pre> <h2>Parallel out-of-band management</h2> <p>You are now ready to send iLOrest commands in parallel toward a set of iLOs sharing common administrator credentials.</p> <p>This example applies a Bios configuration file to a list of servers. The Bios configuration has been created and customized in an ASCII/json file called <code>BiosConfig.json</code> using the <code>ilorest save</code> command. The list of target servers to configure is represented by a group of iLOs defined in the nodeset configuration file as explained earlier. Note that, for a performance reason, I grouped the type selection (<code>Bios.</code>) with the login process:</p> <pre><code class="language-bash">CLUSH="clush -R exec" iLOrest="ilorest --nologo --cache-dir=/tmp/cache-%h" ILO_LIST=$(nodeset @myilos) $CLUSH -w $ILO_LIST $iLOrest login %h -u ilo-admin -p password --selector=Bios. # On gen9, selector is HpBios $CLUSH -w $ILO_LIST $iLOrest load -f BiosConfig.json $CLUSH -w $ILO_LIST $iLOrest logout </code></pre> <p>I personally successfully tested this example against 24 servers. I am pretty optimistic that it works as well for a larger number of servers, although a risk of bumping into iLOrest timeouts exists.</p> <h2>Parallel in-band management</h2> <p>Offloading iLOrest processing onto remote servers may be required in secure environments where iLOs are physically disconnected from the network.</p> <p>In this case, in-band management requires that an operating environment is up and running with <code>sshd</code> and <code>ilorest</code> installed and configured on all the managed nodes. This operating environment could be a customized Live-CD booted via the iLO virtual Drive facility as explained in a <a href="https://developer.hpe.com/blog/in-band-management-with-ilorest-and-a-livecd/">previous article</a>.</p> <p>The in-band version of our Bios configuration requires one more step compared to the out-of-band method: the copy of the JSON configuration file onto all the managed nodes. This operation is simple to perform with the <code>--copy</code> and <code>--dest</code> clush options dedicated to that purpose. Moreover, this copy will not impact the overall process time because the JSON configuration file to transfer is a small ASCII file:</p> <pre><code class="language-bash">UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 16) CLUSH="clush -l root " iLOrest="ilorest --nologo --cache-dir=/tmp/cache-${UUID}" SRV_LIST=$(nodeset -f "@lab") $CLUSH -w SRV_LIST --copy BiosConfig.json --dest /tmp/. $CLUSH -w $SRV_LIST "$iLOrest login --selector=Bios." # selector is HpBios. on Gen9 servers $CLUSH -w $SRV_LIST "$iLOrest load -f /tmp/BiosConfig.json " $CLUSH -w $SRV_LIST $iLOrest logout </code></pre> <h2>Conclusion</h2> <p>In Linux infrastructures, iLOrest combined to the ClusterShell offers easy out-of-band and in-band management scripting possibilities for system administrators who prefer using Bash to pure Python. Of course, there are pros and cons to both methods. Out-of-band iLOrest management does not require a customized operating environment running on the managed servers. In-band management offers the possibility to physically disconnect iLOs to decrease the network footprint.</p> <p>In Windows environments, system managers should be able to perform similar efficient parallel configuration of their servers using <a href="https://devblogs.microsoft.com/scripting/parallel-processing-with-jobs-in-powershell/">PowerShell</a>.</p>https://developer.hpe.com/here’s-to-2021-and-the-hpe-dev-community-take-a-bow-–-you-deserve-it/https://developer.hpe.com/here’s-to-2021-and-the-hpe-dev-community-take-a-bow-–-you-deserve-it/Thu, 16 Dec 2021 15:55:54 GMT<p>The rolling in of a new year often brings with it a sense of nostalgia, offering a time for reflection to take stock of what we’ve accomplished. It’s also a time when we resolve to do even better and try new things in the time ahead.</p> <p>When I look back on 2021, I can’t help but to feel a certain amount of pride in being a part of HPE DEV and all that our Community has accomplished. I specifically use the term Community here, because, without you, these things would not have happened. Without you lending your expertise and sharing your skill sets and talents, we could not have delivered all of this. So, let’s take a look back on our achievements and take a moment to appreciate all the hard work that it took to get here.</p> <p><strong>The HPE DEV Portal</strong></p> <p>One of the biggest undertakings was the refurbishment of the HPE DEV portal early in the year. With more and more platform pages being added, including, SmartSim, Determined AI, HPE Alletra , and Data Services Cloud Console, as well as new learning opportunities, it was evident that the site needed to be revamped to facilitate navigation for portal users.</p> <p>The new portal now offers great ways to <a href="https://developer.hpe.com/community">engage</a>, new <a href="https://developer.hpe.com/skillup">Skill Up</a> opportunities, and drop down menus to quickly find what you need.</p> <center><img src="/img/hpedev-website.png" width="800" height=455" alt="portal benefits"></center> <p>One of the newest additions to the site is the link to our new <a href="https://www.youtube.com/playlist?list=PLtS6YX0YOX4fWMwKbp9blyI1GLdXlbWjY">YouTube video channel</a> where you can now find over 50 videos, including many in-depth interviews with industry experts explaining new technologies and offering tips and tricks on how to work with them.</p> <p>Another reason for the site revamp was to implement a better back-end content management system. From the outside, you might not notice this much, but if you’ve written blog posts for us this year, you’ve undoubtedly noticed the improved method for submitting your posts and the ability to manage them through GitHub.</p> <p><strong>The HPE Blog Site</strong></p> <p>To all of you who submitted articles and tutorials this past year, thank you! In 2021, we had about 100 blog posts published. The articles ranged in content from high-level interviews with developers and open source experts to exceptional, deep-dive tutorials helping readers learn how to take advantage of newer technologies to make their lives easier. Data analytics, data/ML/AI pipelines, data fabric, and containers appeared to be the hottest topics, although systems management with Redfish proved to be highly popular as well. We’re hoping to add even more open source and edge-to-cloud articles as we move into 2022.</p> <p><strong>HPE DEV Workshops-on-Demand</strong></p> <p>The <a href="https://hackshack.hpedev.io/workshops">HPE DEV Workshops-on-Demand </a>were introduced in November of 2020, starting off with just three Jupyter Notebooks. In 2021, Community members contributed and added to that so we now have 23 workshops with more coming soon. While adding 20 workshops over the course of the year is notable, it’s not nearly as impressive as the fact that we were able to deliver these workshops to over 2,000 registrants this year!</p> <p>The hottest workshops for 2021 were the two focused on Kubernetes, quickly followed by the sessions on Redfish. HPE Ezmeral workshops also made a good showing, as did workshops on Python, SPIFFE/SPIRE, and HPE OneView. The workshops garnered fantastic ratings in the post-workshop surveys registrants are requested to fill out, earning an average of 4.6 stars out of 5. A great addition to the program is our new set of recognition badges awarded to those who take the workshops. If you’re not familiar with this program, it’s outlined in this <a href="https://developer.hpe.com/blog/become-a-legend/">blog post</a>. Basically, you receive a badge for each workshop you complete and additional badges based on how many workshops you complete. These badges can be shared via social media to brag to your friends on how technically advanced you are!</p> <p><strong>HPE DEV Munch &#x26; Learn Technology Talks</strong></p> <p>Another place the Community shined this year is in the area of collaborating with one another in the <a href="https://developer.hpe.com/campaign/munch-and-learn">HPE DEV Munch &#x26; Learn</a> sessions. These free, one-hour monthly webinars provide an opportunity for industry experts to share their knowledge with and answer questions from the Community. Many of these talks introduce topics that eventually make their way into blog posts or Jupyter Notebooks for new HPE DEV Workshops-on-Demand.</p> <p>This year we delivered 11 Munch &#x26; Learn sessions on some really great topics. The most recent session had a record-breaking number of registrants – over 350 – who all came to hear about Redfish: Past, Present and Future.</p> <p>For a quick look at the topics delivered and to access their replays, check out the list below:</p> <table> <thead> <tr> <th><strong>Date</strong></th> <th><strong>Title</strong></th> <th><strong>Link</strong></th> </tr> </thead> <tbody> <tr> <td>January</td> <td>What’s a data fabric and how does it work?</td> <td><a href="https://youtu.be/qi6sTvu8osk">Replay</a></td> </tr> <tr> <td>February</td> <td>Explore Containerization and MLOps</td> <td><a href="https://youtu.be/9PvKpe7yMpI">Replay</a></td> </tr> <tr> <td>March</td> <td>Data Science Unplugged: Part 1</td> <td><a href="https://youtu.be/Inh6eXM0EbA">Replay</a></td> </tr> <tr> <td>April</td> <td>Building a foundation for zero trust with SPIFFE</td> <td><a href="https://youtu.be/G1ceKr16nn8">Replay</a></td> </tr> <tr> <td>May</td> <td>Data Science Unplugged: Part 2</td> <td><a href="https://www.youtube.com/watch?v=Va4tSr__Yok">Replay</a></td> </tr> <tr> <td>June</td> <td>Microservices Architecture 101</td> <td><a href="https://youtu.be/qyyxQU37ZyQ">Replay</a></td> </tr> <tr> <td>July</td> <td>How to make data consumable for real-world data science</td> <td><a href="https://youtu.be/4WKjRqflF7M">Replay</a></td> </tr> <tr> <td>August</td> <td>Kubernetes 101</td> <td><a href="https://youtu.be/PWVJKK1obKQ">Replay</a></td> </tr> <tr> <td>September</td> <td>Digital Transformation.Next: Data &#x26; Analytics Workloads</td> <td><a href="https://youtu.be/Q4kJKCS7rbo">Replay</a></td> </tr> <tr> <td>November</td> <td>The Great Unification: Building Analytic pipelines with Apache Spark Workloads</td> <td><a href="https://youtu.be/TxZP_T9CC5Y">Replay</a></td> </tr> <tr> <td>December</td> <td>Redfish: Past, Present and Future</td> <td><a href="https://youtu.be/Q1Qeb24lpKg">Replay</a></td> </tr> </tbody> </table> <p>In addition to our monthly Munch &#x26; Learn Technology Talks, HPE DEV facilitated 19 different online gatherings to share technology information and collaborate between HPE engineering groups. The effects of these meetings are expected to result in some interesting new meetup opportunities for the Community at large in 2022, so keep an eye out for announcements soon in this area.</p> <p><strong>Events</strong></p> <p>As we continued to struggle worldwide with the pandemic, in-person event opportunities were limited, however we did have HPE DEV Community representation at KubeCon 2021 NA in Los Angeles where Sir Hackington once again stood watch over HPE DEV swag.</p> <center><img src="/img/sir-hackington-2021-small.png"></center> <p>We were still able to come together for a number of virtual events, including KubeCon Europe, 3D Data World 2021 (a really cool event where we participated as avatars!), and Discover 2021 where we achieved the highest technology session participation with 672 attendees!</p> <p><strong>Looking Forward to a Bright New Year!</strong></p> <p>2021 gave us all a great opportunity to explore and learn more about HPE Ezmeral, data fabric, Kubernetes, and open source projects like SPIFFE/SPIRE. We’ve added more content around data analytics, and ML and AI technologies. We also on-boarded six ISV (independent software vendor) partners to the HPE DEV program who delivered content related to HPE Ezmeral via blog posts and technology talks helping to grow developer awareness around this solution. Because of this, we’ve welcomed more and more data scientists and related personas into our Community.</p> <p>We look forward to our continued collaboration, diving deeper into exciting new ways of dealing with all this data around us, making it useful, and expanding work in open source projects.</p>https://developer.hpe.com/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/https://developer.hpe.com/getting-started-with-the-hpe-data-services-cloud-console-public-rest-api/Wed, 15 Dec 2021 16:01:03 GMT<center><img src="/img/dscc-idp-core-architect.png" width="500" height="501" alt="Unified DataOps"></center> <p>Customers across industries are struggling with the complexity of managing data and infrastructure, because it creates a roadblock to innovation and agility. Today, every organization is required to unleash the power of data to drive digital transformation, but fragmented data management tools, manual processes, and infrastructure silos - spanning edge to cloud - are getting in the way. This complexity also amplifies business risk, and it's only getting harder as data continues to grow, apps evolve, and infrastructure continues its spread from edge to cloud.</p> <p>Data Services Cloud Console public REST API provides a resource for customers who are looking to enhance their infrastructure management and data-ops using the programmatic extensions from Data Services Cloud Console.</p> <h3>A Public REST API which is based on the OpenAPI 3.X Specification</h3> <p><img src="/img/universal-public-api.png" alt="API diagram" title="API "></p> <p>Hewlett Packard Enterprise (HPE) offers the Data Services Cloud Console unified REST API to provide the agility previously mentioned. It is specified based on the OpenAPI format version 3 (OpenAPI 3.0 information). The specification defines a standard, language-agnostic interface to the RESTful API allowing clients (both human and computer) to consume capabilities of the console's services efficiently. The API definition is available for download in either OpenAPI 3 YAML or JSON format at the link mentioned in the next chapter.</p> <p>Some of the advantages of distributing the API in OpenAPI 3.0 format:</p> <ol> <li>Updates to the API can be generated in a more agile manner where documentation is embedded, describing any endpoints, parameters, and more; such as contact information, license, terms of use.</li> <li>Consumers of this API also gain the benefits of agility by using the converter from openAPI yaml, or json to any programming language that is used as part of their automation or CI/CD workflow. (Please check <a href="https://openapi.tools">https://openapi.tools</a> for more information for the API tools to generate client code)</li> </ol> <h3>Data Services Cloud Console REST API Details</h3> <p>Anyone can download this OpenAPI Specification (OAS) v3 definition of the Data Services Cloud Console from the following: <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">Link to the API repository</a></p> <p><img src="/img/api-documentation-display.png" alt="HPE GreenLake API documentation" title="API Doc"></p> <p>The website also provides additional information:</p> <ol> <li> <p>The list of the REST API resources that are supported as of the release.</p> </li> <li> <p>The information about the HTTP method, parameters and the responses that are expected from each resource.</p> </li> <li> <p>The syntax for the HTTP method and path to this resource. Note that this path is a relative path. For the complete path, please add the base-URL documented below.</p> </li> <li> <p>The body of response is returned in JSON format according to the response status of the REST API.</p> <p>The website also provides the links to download the cloud console OpenAPI definitions in either json or yaml format. Below is an example of the downloaded yaml definition file from the Data Services Cloud Console REST API documentation website.</p> </li> </ol> <p><img src="/img/open-api-yaml.png" alt="API definition yaml" title="yaml"></p> <p>Users can download the API definition from the API documentation website, and the API definition is available in both YAML and JSON version. It can be downloaded by clicking on the download button on the top left of the documentation website.</p> <h3>Documented Attributes</h3> <ul> <li> <p>API Name &#x26; Description</p> <ul> <li>Provides a short description of the objective for this API with the supported HTTP request method (POST, GET, DELETE, PATCH, PUT etc).</li> </ul> </li> <li> <p>API Path</p> <ul> <li>Provides the detailed URL path as the end-point to issue the API call. Note that the user must add the base path URL to extend this path to the correct resource.</li> </ul> </li> <li> <p>API Parameter</p> <ul> <li>Allows the client to input information, such as the object for manipulation, select a filter to limit the returned objects, and other purposes.</li> </ul> </li> <li> <p>API Data/Body/Payload</p> <ul> <li>This is the data passed along in a different part of the REST API request, usually associated with HTTP method such as POST/PATCH/PUT.</li> </ul> </li> <li> <p>API Response</p> <ul> <li>Provides detailed response information on the result of the particular API and may include more data in JSON format.</li> </ul> </li> <li> <p>API Error Codes</p> <ul> <li>Provides the result of the execution of the API, returning either good or error, along with the error message due to an incorrect or unauthorized API call.</li> </ul> </li> </ul> <h3>Supported API Categories (Services)</h3> <p>The API categories for Data Services Cloud Console will grow in accordance to the expansion of future services. As recorded today, the current services that are available include:</p> <h4><strong>Common (Alletra-6K, Alletra-9K, Primera, Nimble)</strong></h4> <ol> <li>authentication</li> <li>tasks</li> <li>event Audit</li> <li>authZ (User RBAC permissions)</li> <li>issues</li> <li>controllers</li> <li>host-initiator-groups</li> <li>host-initiators</li> <li>ports</li> <li>shelves</li> <li>storage-pools</li> <li>storage-systems</li> <li>system-settings</li> <li>volume-sets</li> <li>volumes</li> </ol> <h4><strong>Alletra-6K or Nimble</strong></h4> <ol> <li>protection-templates</li> <li>disks</li> </ol> <h3>Versioning</h3> <p>The major version number will be provided in the resource path as "v1" in this example:</p> <pre><code class="language-jsoniq">/api/v1/&#x3C;resource group>/... </code></pre> <p>Here are some examples of these resource paths that contain several resource groups under the same root:</p> <pre><code class="language-jsoniq">/api/v1/storage-systems/... /api/v1/controllers/... /api/v1/volumes/... </code></pre> <p>Existing clients will be able to maintain the backward compatibility from the higher major version incremental and adopt any newly introduced API. However, both the new and old version of the API will be supported until the announcement of the deprecation for the old version of the API. Nonetheless, the older major version will always be frozen, with the exception of bug fixes. There will also be an announcement of deprecation in the header and sunset header.</p> <pre><code class="language-jsoniq">/api/v1/&#x3C;resource group>/... /api/v2/&#x3C;resource group>/... </code></pre> <h3>HTTP Request Methods</h3> <table> <thead> <tr> <th>HTTP Verbs</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>GET</td> <td>Retrieves target resource</td> </tr> <tr> <td>POST</td> <td>Creates an entity or changes state</td> </tr> <tr> <td>PUT</td> <td>Replaces target resource with data part of the HTTP request payload</td> </tr> <tr> <td>DELETE</td> <td>Removes the resource</td> </tr> </tbody> </table> <h3>Authorization through OAuth2 Client Credential Workflow</h3> <p>Glossary of the terms:</p> <p><strong><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Resources</a></strong>: Components inside the cloud console, such as storage array, volumes, and many other objects that are consumable and related to each other, and provides methods to operate on it. Usually is represented by path that is appended to Endpoint e.g. /api/v1/storage-array.</p> <p><strong><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Resource Owner</a>:</strong> The user that is registered inside the HPE GreenLake console that has the capability to authorize the client application access to the cloud console resources.</p> <p><strong><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Client Application</a>:</strong> The stand-alone application that runs on the client machine, and usually represent the customer's business application for automation, ticketing, monitoring and many other business processes.</p> <p><strong><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Access Token:</a></strong> This is the object that describes the permission and the security context of which the client application was delegated. This token contains the identity and privileges of the cloud console user which create this token.</p> <p><strong><a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">API Gateway</a></strong>: The API Gateway is the menu in the HPE GreenLake console that is used to register a client application and obtain the API client credentials (client-id and client-secret) for that client application. These credentials are required to generate a short-lived access token that is used to make secure REST API calls to the Data Services Cloud Console application instance.</p> <p><strong><a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">Endpoint</a></strong>: Location where service can be accessed, usually is represented by URL (Uniform Resource Locator) e.g. <a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></p> <p><img src="/img/dscc-public-api-introduction-updated_111122.jpg" alt="OAuth 2.0 flow" title="authentication and authorization flow"></p> <p>The client's application can issue a REST API request using the access token as the bearer of the token. The client can obtain this access token from the authorization API end point, after the client successfully authenticates through an associated customer's application credential (client-id and client-secret). This application credential is created by the console's user who has the permission to access resources (such as controllers, volumes etc.) under the console instances. This access token expiration time, by default, is set for 7200 seconds (2 hours). When the resource server sees this expired access token, it returns a 0x401 response (not authorized). The client must then authenticate using the associated client-id and client-secret to obtain the next access-token to use for the next REST API request.</p> <h3>Authorization Policies</h3> <p>The client can only receive properties from the authorized API resources based on the Role Base Access Control for the user who created the client-credential pair (client-id and client-secret). This authorization derives from the organization, capability, and scope (roles) that the associated user is assigned. As a result, the authorization for the client application will inherit the user's permission who created the client-application registration under the API Gateway. Note that subsequent changes to the user's permission after the client application registered will impact the response returned based on current authority.</p> <h3>The API Endpoints (base-URL) for each Data Services Cloud Console's Region</h3> <p>The REST API for Data Services Cloud Console requires the client application to issue the REST API request to the URL that is associated with the console's instance deployed at the associated region of the storage array. As of November 2021, here are the Domain URLs which the client application must use as the base-URL to the resource path of REST API.</p> <table> <thead> <tr> <th>Data Services Cloud Console Region</th> <th>Base-URL</th> </tr> </thead> <tbody> <tr> <td>EU Central (Europe)</td> <td><a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></td> </tr> <tr> <td>AP Northeast (Asia Pacific)</td> <td><a href="https://jp1.data.cloud.hpe.com">https://jp1.data.cloud.hpe.com</a></td> </tr> <tr> <td>US West (United States)</td> <td><a href="https://us1.data.cloud.hpe.com">https://us1.data.cloud.hpe.com</a></td> </tr> </tbody> </table> <h3>Asynchronous Response</h3> <p>All of the REST API operations are stateless in nature. One example of such is POST. In that scenario, the task resource will return a response with HTTP code 0x202 "Accepted" and the reference to the task as follows:</p> <pre><code class="language-jsoniq">Response: 202 (Accepted) { "taskURI":"/api/v1/tasks/{task id} } </code></pre> <p>In order to ensure the completion of this remote procedural call through POST, the user will use the task resource to query the status of this asynchronous task.</p> <pre><code class="language-jsoniq">/api/v1/tasks/{task id} GET responses { state: {state ENUM} } state ENUM: - UNSPECIFIED - INITIALIZED - RUNNING - FAILED - SUCCEEDED - TIMEDOUT - PAUSED </code></pre> <p>For more in depth discussion on the topics about API Gateway and OAuth 2.0 (Open Authorization), please take a look at these blogs in HPE DEV, tektalk on point at On24 website, and also the demo in Youtube.</p> <p><a href="https://www.youtube.com/watch?v=g3UO0S-4r6I"><img src="https://img.youtube.com/vi/g3UO0S-4r6I/hqdefault.jpg" alt="Introduction to DSCC"></a></p> <ul> <li>Blog: <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">Using HPE GreenLake's API Gateway for Data Services Cloud Console</a></li> <li>Blog: <a href="https://developer.hpe.com/blog/oauth2-for-hpe-greenlake-data-services-cloud-console/">Implementing OAuth 2 Flow for Data Services Cloud Console's Client Application</a></li> <li>TEKTALK ON POINT: <a href="https://vshow.on24.com/vshow/HPETekTalks/content/3571890/">Introduction to Data Services Cloud Console public API</a></li> </ul> <p>More blog posts will be coming to help you take further advantage of its capabilities. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more blog posts about HPE Data Services Cloud Console REST API.</p>https://developer.hpe.com/getting-started-with-spark-on-mapr-sandbox/https://developer.hpe.com/getting-started-with-spark-on-mapr-sandbox/Tue, 14 Dec 2021 09:35:31 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">https://www.hpe.com/us/en/software/ezmeral-data-fabric.html</a></p> <p>At MapR, we distribute and support Apache Spark as part of the MapR Converged Data Platform. This tutorial will help you get started with running Spark application on the MapR Sandbox (now known as the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/RunMapRContainerDevelopers.html">Development Environment for HPE Ezmeral Data Fabric</a>).</p> <h2>Prerequisites</h2> <p><strong>HARDWARE REQUIREMENTS</strong></p> <ul> <li>8GB RAM, multi-core CPU</li> <li>20GB minimum HDD space</li> <li>Internet access</li> </ul> <p><strong>SOFTWARE REQUIREMENTS</strong></p> <ul> <li>A hypervisor. This example uses VMware Fusion 6.0.2 on OSX; however, other VMware products could be used instead. Additionally, <a href="https://www.virtualbox.org/">VirtualBox</a> can be used</li> <li>A virtual machine image for the MapR Sandbox. Spark comes pre-loaded with version 5.0 and later of the MapR Sandbox.</li> </ul> <p><strong>Starting up and Logging into the Sandbox</strong><br /> Install and fire up the Sandbox using the instructions <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/RunMapRContainerDevelopers.html">here</a>. Once you are able to log in to the web interface for the Sandbox, you are ready to start setting up Spark.</p> <p><strong>Logging in to the Command Line</strong></p> <ul> <li> <p>Before you get started, you'll want to have the IP address handy for your Sandbox VM. See the screenshot below for an example of where to find that.</p> <p><img src="/img/tutorial_spark1.png" alt=""></p> </li> <li> <p>Next, use an SSH client such as Putty (Windows) or Terminal (Mac) to login. See below for an example:<br /> use userid: user01 and password: mapr.</p> <p><img src="/img/tutorial_spark2.png" alt=""></p> <p>For VMWare use: <code>$ ssh user01@ipaddress</code></p> <p>For Virtualbox use: <code>$ ssh user01@127.0.0.1 -p 2222</code></p> </li> </ul> <h2>“How to” for a Spark Application</h2> <p>Next, we will look at how to write, compile, and run a Spark word count application on the MapR Sandbox. First, we will walk step by step through the following word count application in Java.</p> <p><strong>Example Word Count App in Java</strong></p> <p><img src="/img/tutorial_spark3.png" alt=""></p> <p>You can download the complete Maven project Java and Scala code <a href="https://github.com/caroljmcdonald/sparkwordcountapp">here</a>.</p> <p><strong>Get a text-based dataset</strong></p> <p>First, let's grab a text-based dataset that will be used for counting the words. Today, we'll be using the freely available Alice In Wonderland text. Create a new folder:<br /> <code>mkdir -p /mapr/demo.mapr.com/input</code></p> <p>Pull down the text file:</p> <pre><code class="language-Java">wget -O /mapr/demo.mapr.com/input/alice.txt http://www.gutenberg.org/cache/epub/11/pg11.txt </code></pre> <p><strong>Initializing a SparkContext</strong></p> <p>The first thing a Spark program has to do is create a SparkContext object, which represents a connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster.</p> <p><img src="/img/tutorial_spark4.png" alt=""></p> <p>To create a SparkContext, you first need to create a SparkConf object to configure your application, as shown below:</p> <pre><code class="language-Java"> // Create a Java Spark Context. SparkConf conf = new SparkConf().setAppName("JavaWordCount"); JavaSparkContext sc = new JavaSparkContext(conf); </code></pre> <p><strong>Create an RDD from a file</strong></p> <p>Spark’s primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from Hadoop InputFormats (such as HDFS files) or by transforming other RDDs. The following code uses the SparkContext to define a base RDD from the file inputFile:</p> <pre><code class="language-Java"> String inputFile = args[0]; JavaRDD &#x3C;string>input = sc.textFile(inputFile);&#x3C;/string> </code></pre> <p><strong>Transform input RDD with flatMap</strong></p> <p>To split the input text into separate words, we use the <a href="https://spark.apache.org/docs/1.3.0/programming-guide.html">flatMap(func)</a> RDD transformation, which returns a new RDD formed by passing each element of the source through a function. The String split function is applied to each line of text, returning an RDD of the words in the input RDD:</p> <p><img src="/img/tutorial_spark5.png" alt=""></p> <pre><code class="language-Java"> // map/split each line to multiple words **JavaRDD&#x3C;String> words = input.flatMap(** new FlatMapFunction&#x3C;String, String>() { public Iterable&#x3C;String> call(String x) { return Arrays.asList(x.split(" ")); } } ); </code></pre> <p><strong>Transform words RDD with map</strong></p> <p>We use the <a href="https://spark.apache.org/docs/1.3.0/programming-guide.html">map(func)</a> to transform the words RDD into an RDD of (word, 1) key-value pairs:</p> <p><img src="/img/tutorial_spark6.png" alt=""></p> <pre><code class="language-Java"> JavaPairRDD&#x3C;String, Integer> wordOnePairs = words.mapToPair( new PairFunction&#x3C;String, String, Integer>() { public Tuple2&#x3C;String, Integer> call(String x) { return new Tuple2(x, 1); } } ); </code></pre> <p><strong>Transform wordOnePairs RDD with reduceByKey</strong></p> <p>To count the number of times each word occurs, we combine the values (1) in the wordOnePairs with the same key (word) using <a href="https://spark.apache.org/docs/1.3.0/programming-guide.html">reduceByKey(func)</a>, This transformation will return an RDD of (word, count) pairs where the values for each word are aggregated using the given reduce function func x+y:</p> <p><img src="/img/tutorial_spark7.png" alt=""></p> <pre><code class="language-Java"> // reduce add the pairs by key to produce counts JavaPairRDD&#x3C;String, Integer> counts = wordOnePairs.reduceByKey( new Function2&#x3C;Integer, Integer, Integer>() { public Integer call(Integer x, Integer y) { return x + y; } } ); </code></pre> <p><strong>Output with RDD action saveAsTextFile</strong></p> <p>Finally, the RDD action saveAsTextFile(path) writes the elements of the dataset as a text file (or set of text files) in the outputFile directory.</p> <pre><code class="language-Java"> String outputFile = args[1]; // Save the word count back out to a text file, causing evaluation. counts.saveAsTextFile(outputFile); </code></pre> <p><strong>Example Word Count App in Scala</strong></p> <p>Here is the same example in Scala:</p> <p><img src="/img/tutorial_spark8.png" alt=""></p> <p><strong>Building a Simple Application</strong></p> <p>Spark can be linked into applications in either Java, Scala, or Python. Maven is a popular package management tool for Java-based languages that lets you link to libraries in public repositories. In Java and Scala, you give your application a Maven dependency on the spark-core artifact. The current Spark version is 1.6.1 and the Maven coordinates are:</p> <pre><code class="language-xml"> &#x3C;dependency> &#x3C;groupId>org.apache.spark&#x3C;/groupId> &#x3C;artifactId>spark-core_2.10&#x3C;/artifactId> &#x3C;version>1.6.1&#x3C;/version> &#x3C;scope>provided&#x3C;/scope> &#x3C;/dependency> </code></pre> <p>You can build with Maven using IDEs like Eclipse or NetBeans, and then copy the JAR file to your MapR Sandbox, or you can install Maven on your sandbox and build from the Linux command line.</p> <p><img src="/img/tutorial_spark9.png" alt=""></p> <p>You can use scp to copy your JAR file to the MapR Sandbox:</p> <p><img src="/img/tutorial_spark10.png" alt=""></p> <p>See below for an example of using scp from the command line:<br /> <code>use userid: user01 and password: mapr.</code></p> <p>For VMWare use: <code>$ scp nameoffile.jar user01@ipaddress:/user/user01/.</code></p> <p>For Virtualbox use: <code>$ scp -P 2222 nameoffile.jar user01@127.0.0.1:/user/user01/.</code></p> <p><strong>Running Your Application</strong></p> <p>First, find the version of Spark on the sandbox with <code>ls /opt/mapr/spark/</code>. Then you can use the spark commands in the /opt/mapr/spark/spark-version/bin directory.</p> <p>You use the <code>bin/spark-submit</code> script to launch your application. This script takes care of setting up the classpath with Spark and its dependencies. Here is the spark-submit format:</p> <pre><code class="language-xml">./bin/spark-submit \ --class &#x3C;main-class> --master &#x3C;master-url> \ &#x3C;application-jar> \ [application-arguments] </code></pre> <p>Here is the spark-submit command for our example, passing the input file and output directory as arguments to the main method of the <code>JavaWordCount</code> class:</p> <pre><code class="language-scala">/opt/mapr/spark/spark-1.6.1/bin/spark-submit --class example.wordcount.JavaWordCount --master yarn \ sparkwordcount-1.0.jar /user/user01/input/alice.txt /user/user01/output </code></pre> <p>Here is the spark-submit command to run the <code>scala SparkWordCount</code>:</p> <pre><code class="language-scala">/opt/mapr/spark/spark-1.6.1/bin/spark-submit --class SparkWordCount --master yarn \ sparkwordcount-1.0.jar /user/user01/input/alice.txt /user/user01/output </code></pre> <p>This concludes the Getting Started with Spark on the MapR Sandbox tutorial. You can download the example Maven project code <a href="https://github.com/caroljmcdonald/sparkwordcountapp">here</a>.</p> <p>For more information:</p> <ul> <li><a href="http://spark.apache.org/docs/latest/">http://spark.apache.org/docs/latest/</a></li> <li><a href="https://spark.apache.org/examples.html">https://spark.apache.org/examples.html</a></li> <li><a href="https://spark.apache.org/docs/latest/quick-start.html">https://spark.apache.org/docs/latest/quick-start.html</a></li> </ul>https://developer.hpe.com/using-apache-spark-dataframes-for-processing-of-tabular-data/https://developer.hpe.com/using-apache-spark-dataframes-for-processing-of-tabular-data/Mon, 13 Dec 2021 10:27:46 GMT<pre><code>"authorDisplayName": "Carol McDonald", "category": "apache-spark", "publish": "2015-06-24T07:00:00.000Z" </code></pre> <p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">https://www.hpe.com/us/en/software/ezmeral-data-fabric.html</a></p> <p>This post will help you get started using Apache Spark DataFrames with Scala on the MapR Sandbox (now known as the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">Development Environment for HPE Ezmeral Data Fabric</a>). The <a target='\_blank' href='http://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes'>new Spark DataFrames API</a> is designed to make big data processing on tabular data easier.</p> <h2>What is a Spark DataFrame?</h2> <p>A Spark DataFrame is a distributed collection of data organized into named columns that provides operations to filter, group, or compute aggregates, and can be used with Spark SQL. DataFrames can be constructed from structured data files, existing RDDs, tables in Hive, or external databases.</p> <p>In this post, you’ll learn how to:</p> <ul> <li>Load data into Spark DataFrames</li> <li>Explore data with Spark SQL</li> </ul> <p>This post assumes a basic understanding of Spark concepts. If you have not already read the tutorial on <a href="https://developer.hpe.com/blog/getting-started-with-spark-on-mapr-sandbox/">Getting Started with Spark on MapR Sandbox</a>, it would be good to read that first.</p> <h2>Software</h2> <p><strong>The sample data sets</strong></p> <p>We will use two example datasets - one from <a target='\_blank' href='http://www.modelingonlineauctions.com/datasets'>eBay online auctions</a> and one from the <a target='\_blank' href='https://data.sfgov.org/Public-Safety/SFPD-Incidents-from-1-January-2003/tmnf-yvry'>SFPD Crime Incident Reporting system</a>.</p> <p>The eBay online auction dataset has the following data fields:</p> <p><em><strong>auctionid</strong></em> - unique identifier of an auction <br /> <em><strong>bid</strong></em> - the proxy bid placed by a bidder <br /> <em><strong>bidtime</strong></em> - the time (in days) that the bid was placed, from the start of the auction <br /> <em><strong>bidder</strong></em> - eBay username of the bidder <br /> <em><strong>bidderrate</strong></em> - eBay feedback rating of the bidder <br /> <em><strong>openbid</strong></em> - the opening bid set by the seller <br /> <em><strong>price</strong></em> - the closing price that the item sold for (equivalent to the second highest bid + an increment) <br /></p> <p>The table below shows the data fields with some sample data:</p> <p><img src="/img/blog_sparkdataframes_table1.png" alt=""></p> <p>Using Spark DataFrames, we will explore the data with questions like:</p> <ul> <li>How many auctions were held?</li> <li>How many bids were made per item?</li> <li>What's the minimum, maximum, and average number of bids per item?</li> <li>Show the bids with price > 100</li> </ul> <p>The table below shows the SFPD data fields with some sample data:</p> <p><img src="/img/blog_sparkdataframes_table2.png" alt=""></p> <p>Using Spark DataFrames, we will explore the SFPD data with questions like:</p> <ul> <li>What are the top 10 Resolutions?</li> <li>How many Categories are there?</li> <li>What are the top 10 incident Categories?</li> </ul> <h2>Loading data into Spark DataFrames</h2> <p>Log into the MapR Sandbox, as explained in <a href="https://developer.hpe.com/blog/getting-started-with-spark-on-mapr-sandbox/">Getting Started with Spark on MapR Sandbox</a>, using userid user01, password mapr. Copy the sample data files to your sandbox home directory /user/user01 using scp. Start the spark shell with: <br /> <code>$ spark-shell</code></p> <p>First, we will import some packages and instantiate a sqlContext, which is the entry point for working with structured data (rows and columns) in Spark and allows the creation of DataFrame objects. <br /> (In the code boxes, <font color="green">comments are in Green</font> and <font color="#005CB9">output is in Blue</font>)</p> <pre> <font color="green">// SQLContext entry point for working with structured data</font> val sqlContext = new org.apache.spark.sql.SQLContext(sc) <font color="green">// this is used to implicitly convert an RDD to a DataFrame.</font> import sqlContext.implicits._ <font color="green">// Import Spark SQL data types and Row.</font> import org.apache.spark.sql._ </pre> <p>Below, we load the data from the ebay.csv file into a Resilient Distributed Dataset (RDD). RDDs can have transformations and actions; the first() action returns the first element in the RDD, which is the String <code>“8213034705,95,2.927373,jake7870,0,95,117.5,xbox,3”</code></p> <pre> <font color="green">// load the data into a new RDD</font> val ebayText = sc.textFile("ebay.csv") <font color="green">// Return the first element in this RDD</font> ebayText.first() <font color="#005CB9">// res6: String = 8213034705,95,2.927373,jake7870,0,95,117.5,xbox,3</font> </pre> <p>Below, we use a Scala case class to define the Auction schema corresponding to the ebay.csv file. Then, map() transformations are applied to each element of ebayText to create the ebay RDD of Auction objects.</p> <pre> <font color="green">//define the schema using a case class</font> case class Auction(auctionid: String, bid: Float, bidtime: Float, bidder: String, bidderrate: Integer, openbid: Float, price: Float, item: String, daystolive: Integer) <font color="green">// create an RDD of Auction objects</font> val ebay = ebayText.map(_.split(",")).map(p => Auction(p(0),p(1).toFloat,p(2).toFloat,p(3),p(4).toInt,p(5).toFloat,p(6).toFloat,p(7),p(8).toInt )) </pre> <p>The ebay RDD first() action returns the first element in the RDD, Auction = Auction( 8213034705, 95.0, 2.927373, jake7870, 0, 95.0, 117.5, xbox,3).</p> <pre> <font color="green">// Return the first element in this RDD</font> ebay.first() <font color="#005CB9">//res7: Auction = Auction(8213034705,95.0,2.927373,jake7870,0,95.0,117.5,xbox,3)</font> <font color="green">// Return the number of elements in the RDD</font> ebay.count() res8: Long = 10654 </pre> <p>A DataFrame is <a target='\_blank' href='https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html'>a distributed collection of data organized into named columns.</a> Spark SQL supports automatically converting an RDD containing case classes to a DataFrame with the method toDF():</p> <pre> <font color="green">// change ebay RDD of Auction objects to a DataFrame</font> val auction = ebay.toDF() </pre> <p>The previous RDD transformations can also be written on one line like this:</p> <pre> val auction = sc.textFile("ebay.csv").map(_.split(",")).map(p => Auction(p(0),p(1).toFloat,p(2).toFloat,p(3),p(4).toInt,p(5).toFloat,p(6).toFloat,p(7),p(8).toInt )).toDF() </pre> <h2>Explore and query the eBay auction data with Spark DataFrames</h2> <p>DataFrames provide a domain-specific language for structured data manipulation in Scala, Java, and Python; below are some examples with the auction DataFrame. The DataFrame show() action displays the top 20 rows in a tabular form.</p> <pre> <font color="green">// Display the top 20 rows of DataFrame</font> auction.show() <font color="#005CB9">// auctionid bid bidtime bidder bidderrate openbid price item daystolive // 8213034705 95.0 2.927373 jake7870 0 95.0 117.5 xbox 3 // 8213034705 115.0 2.943484 davidbresler2 1 95.0 117.5 xbox 3 …</font> </pre> <p>DataFrame printSchema() Prints the schema to the console in a tree format</p> <pre> <font color="green">// Return the schema of this DataFrame</font> auction.printSchema() <font color="#005CB9">root |-- auctionid: string (nullable = true) |-- bid: float (nullable = false) |-- bidtime: float (nullable = false) |-- bidder: string (nullable = true) |-- bidderrate: integer (nullable = true) |-- openbid: float (nullable = false) |-- price: float (nullable = false) |-- item: string (nullable = true) |-- daystolive: integer (nullable = true)</font> </pre> <p>After a dataframe is instantiated, you can query it using SQL queries. Here are some example queries using the Scala DataFrame API:</p> <pre> <font color="green">// How many auctions were held?</font> auction.select("auctionid").distinct.count <font color="#005CB9">// Long = 627</font> <font color="green">// How many bids per item?</font> auction.groupBy("auctionid", "item").count.show auctionid item count 3016429446 palm 10 8211851222 xbox 28 3014480955 palm 12 8214279576 xbox 4 3014844871 palm 18 3014012355 palm 35 1641457876 cartier 2 . . . <font color="green">// What's the min number of bids per item? what's the average? what's the max?</font> auction.groupBy("item", "auctionid").count.agg(min("count"), avg("count"),max("count")).show <font color="#005CB9">// MIN(count) AVG(count) MAX(count) // 1 16.992025518341308 75</font> <font color="green">// Get the auctions with closing price > 100</font> val highprice= auction.filter("price > 100") <font color="#005CB9">// highprice: org.apache.spark.sql.DataFrame = [auctionid: string, bid: float, bidtime: float, bidder: // string, bidderrate: int, openbid: float, price: float, item: string, daystolive: int]</font> <font color="green">// display dataframe in a tabular format</font> highprice.show() <font color="#005CB9">// auctionid bid bidtime bidder bidderrate openbid price item daystolive // 8213034705 95.0 2.927373 jake7870 0 95.0 117.5 xbox 3 // 8213034705 115.0 2.943484 davidbresler2 1 95.0 117.5 xbox 3</font> </pre> <p>You can register a DataFrame as a temporary table using a given name, and then run SQL statements using the sql methods provided by sqlContext. Here are some example queries using <a target='\_blank' href='https://spark.apache.org/docs/1.3.0/sql-programming-guide.html'>sqlContext</a>:</p> <pre> <font color="green">// register the DataFrame as a temp table</font> auction.registerTempTable("auction") <font color="green">// SQL statements can be run // How many bids per auction?</font> val results =sqlContext.sql("SELECT auctionid, item, count(bid) FROM auction GROUP BY auctionid, item") <font color="green">// display dataframe in a tabular format</font> results.show() <font color="#005CB9">// auctionid item count // 3016429446 palm 10 // 8211851222 xbox 28\. . .</font> val results =sqlContext.sql("SELECT auctionid, MAX(price) FROM auction GROUP BY item,auctionid") results.show() <font color="#005CB9">// auctionid c1 // 3019326300 207.5 // 8213060420 120.0 . . .</font> </pre> <h2>Loading the SFPD data into Spark dataframes using a csv parsing library</h2> <p>Now we will load the SFPD dataset into a Spark dataframe using the <a target='\_blank' href='https://github.com/databricks/spark-csv'>spark-csv parsing library</a> from Databricks. You can use this library at the Spark shell by specifying <em>--packages com.databricks:spark-csv_2.10:1.0.3</em> when starting the shell as shown below:</p> <p><code>$ spark-shell --packages com.databricks:spark-csv_2.10:1.0.3</code></p> <p>The load operation will parse the sfpd.csv file and return a dataframe using the first header line of the file for column names.</p> <pre> import sqlContext.implicits._ import org.apache.spark.sql._ <font color="green">// Return the dataset specified by data source as a DataFrame, use the header for column names</font> val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "sfpd.csv", "header" -> "true")) </pre> <p>The take operation returns the specified number of rows in the DataFame.</p> <pre> <font color="green">// Return the first n rows in the DataFrame</font> df.take(1) <font color="#005CB9">// res4: Array[org.apache.spark.sql.Row] = Array([150467944,LARCENY/THEFT,GRAND THEFT FROM LOCKED AUTO,Thursday,05/28/2015,23:59,TENDERLOIN,NONE,TAYLOR ST / OFARRELL ST,-122.411328369311,37.7859963050476,(37.7859963050476, -122.411328369311),15046794406244])</font> <font color="green">// Print the schema to the console in a tree format</font> df.printSchema() <font color="#005CB9">root |-- IncidntNum: string (nullable = true) |-- Category: string (nullable = true) |-- Descript: string (nullable = true) |-- DayOfWeek: string (nullable = true) |-- Date: string (nullable = true) |-- Time: string (nullable = true) |-- PdDistrict: string (nullable = true) |-- Resolution: string (nullable = true) |-- Address: string (nullable = true) |-- X: string (nullable = true) |-- Y: string (nullable = true) |-- Location: string (nullable = true) |-- PdId: string (nullable = true)</font> <font color="green">// display dataframe in a tabular format</font> df.show() <font color="#005CB9">IncidntNum Category Descript DayOfWeek Date Time PdDistrict Resolution Address X Y Location PdId 150467944 LARCENY/THEFT GRAND THEFT FROM ... Thursday 05/28/2015 23:59 TENDERLOIN NONE TAYLOR ST / OFARR... -122.411328369311 37.7859963050476 (37.7859963050476... 15046794406244</font> </pre> <h2>Here are some example queries using sqlContext:</h2> <pre> <font color="green">// how many categories are there?</font> df.select("Category").distinct.count <font color="#005CB9">// res5: Long = 39</font> <font color="green">// register as a temp table inorder to use sql</font> df.registerTempTable("sfpd") <font color="green">// How many categories are there</font> sqlContext.sql("SELECT distinct Category FROM sfpd").collect().foreach(println) <font color="#005CB9">// [ASSAULT] // [MISSING PERSON] // [TREA] . . .</font> </pre> <pre> <font color="green">// What are the top 10 Resolutions ?</font> sqlContext.sql("SELECT Resolution , count(Resolution) as rescount FROM sfpd group by Resolution order by rescount desc limit 10").collect().foreach(println) <font color="#005CB9">// [NONE,1063775] // [ARREST, BOOKED,414219] // [ARREST, CITED,154033] . . .</font> </pre> <pre> <font color="green">// What are the top 10 most incident Categories?</font> val t = sqlContext.sql("SELECT Category , count(Category) as catcount FROM sfpd group by Category order by catcount desc limit 10") t.show() <font color="#005CB9">// Category catcount // LARCENY/THEFT 353793 // OTHER OFFENSES 253627 // NON-CRIMINAL 186272\. . .</font> <font color="green">// The results of SQL queries are DataFrames and support RDD operations. // The columns of a row in the result can be accessed by ordinal</font> t.map(t => "column 0: " + t(0)).collect().foreach(println) <font color="#005CB9">// column 0: LARCENY/THEFT // column 0: OTHER OFFENSES // column 0: NON-CRIMINAL // column 0: ASSAULT …</font> </pre> <h2>The physical plan for DataFrames</h2> <p>The <a target='\_blank' href='https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html'>Catalyst query optimizer creates the physical Execution Plan</a> for DataFrames as shown in the diagram below:</p> <p><img src="/img/blog_sparkdataframes_image3.png" alt=""></p> <p><strong>Print the physical plan to the console</strong></p> <p>DataFrames are designed to take the SQL queries constructed against them and optimize the execution as sequences of Spark Jobs as required. You can print the physical plan for a DataFrame using the explain operation as shown below:</p> <pre> <font color="green">// Prints the physical plan to the console for debugging purpose</font> auction.select("auctionid").distinct.**explain**() <font color="#005CB9">// == Physical Plan == // Distinct false // Exchange (HashPartitioning [auctionid#0], 200) // Distinct true // Project [auctionid#0] // PhysicalRDD //[auctionid#0,bid#1,bidtime#2,bidder#3,bidderrate#4,openbid#5,price#6,item#7,daystolive#8], MapPartitionsRDD[11] at mapPartitions at ExistingRDD.scala:37</font> </pre> <h2>Summary</h2> <p>In this blog post, you’ve learned how to load data into Spark DataFrames, and explore data with Spark SQL.</p> <p><strong>Want to learn more?</strong></p> <ul> <li><a href="https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#dataframes">Spark SQL and DataFrame Guide</a></li> <li><a href="https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/">Learning Spark - O'Reilly Book</a></li> </ul>https://developer.hpe.com/mapping-kubernetes-services-to-hpe-ezmeral-container-platform-gateway/https://developer.hpe.com/mapping-kubernetes-services-to-hpe-ezmeral-container-platform-gateway/Mon, 06 Dec 2021 10:44:42 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise.</strong> For more information on why the name was changed, please click <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">here</a>.</p> <p>Imagine you have different Kubernetes services and different services that come with different IP addresses. Are there tools that unify different services into single domain name? A gateway can answer that question. There are several benefits of using a gateway. First, the gateway can act as a load-balancer for different services. Second, only a gateway host IP address is exposed to the public while the rest remains behind the firewall. Follow this blog post to learn more about how to map Kubernetes Services to HPE Ezmeral Runtime Enterprise Gateway.</p> <p>HPE Ezmeral Runtime Enterprise (formerly known as HPE Ezmeral Container Platform) comes with one or more gateway hosts. A gateway host acts as a proxy server that carries client requests like HPE Ezmeral Runtime Enterprise UI, REST API calls, Kubernetes API, and containerized application service endpoints. The Gateway host maps both the IP address of the Controller host and the private IP endpoints of services running on the Kubernetes nodes inside the Kubernetes clusters to publicly-accessible IP addresses/ports.</p> <blockquote> <p>Note: To learn more about the the Gateway Host, check out the online documentation <a href="https://docs.containerplatform.hpe.com/53/reference/universal-concepts/Gateway_Hosts.html#v52_gateway-hosts__logical">here</a>.</p> </blockquote> <p>To set up this architecture properly, you'll want to first set up a new Kubernetes (K8s) tenant, as shown in the image below. Just check the box next to "Map Services To Gateway":</p> <p><img src="https://user-images.githubusercontent.com/72959956/138654527-77f3bf2c-f001-4fc7-88f3-d17436368dc3.png" alt="image"></p> <h1>Create a Hello World Kubernetes Application</h1> <p>Let's create a hello-world application for Kubernetes. This is a simple webpage showing which pod are you using. To begin with, create a deployment called <code>k8s-helloworld</code> with the hello-world image. After that, run <code>get deployment</code> and <code>describe deployment</code> to view the detail of the deployment. If you see <code>1/1</code> under the <code>READY</code> column, it is good to go.</p> <pre><code class="language-bash"># Create deployment of the application k8s-helloworld using the specific image kubectl create deployment k8s-helloworld --image=gcr.io/google-samples/kubernetes-bootcamp:v1 # Get the information of the deployment with label k8s-helloworld kubectl get deployment -l app=k8s-helloworld # Describe the detail information of the deployment named as k8s-helloworld kubectl describe deployment k8s-helloworld </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/138656214-73c9418f-e291-4678-b3a2-c318a318d325.png" alt="image"></p> <p>The deployment will spin up some pods. To view which pods are running, you can run the <code>get pods</code> command. It will return a list of pods. Copy the pod name starting with k8s-helloworld and run an <code>exec</code> command to check if the website is up or not. If you see the terminal reply with <code>Hello Kubernetes bootcamp!</code>, you have successfully deployed a website on Kubernetes.</p> <pre><code class="language-bash"># Get the information of pods labeled with k8s-helloworld kubectl get pods -l app=k8s-helloworld # copy your pod id kubectl describe pods k8s-helloworld-5f84bb5d68-l9vch # exec curl command inside the pod kubectl exec k8s-helloworld-5f84bb5d68-l9vch -- curl -s http://localhost:8080 </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/138670950-75f96e40-3bc6-4ef6-aff6-578f45b90c04.png" alt="image"></p> <h1>Using a Service to Expose an Application in a Cluster</h1> <p>So, now you have a website running on Kubernetes, and you want to share this website to your friends. It turns out that your friends cannot open the website. The reason for this is that, in order to get a deployment exposed to the public, a service object is needed to tell Kubernetes which services port is mapped to the deployment.</p> <p>To expose your deployment, just run the command <code>expose deployment</code> with the name of the deployment and the port number the pod used to create a service object. In this case, it will be port 8080. Run <code>get services</code> to view the services and the mapped services port. Now you can access the website externally using the Kubernetes nodes IP address together with the services port.</p> <pre><code class="language-markdown"># Expose the deployment with Port number 8080 kubectl expose deployment/k8s-helloworld --type="NodePort" --port 8080 # Get the information on Services labeled with k8s-helloworld kubectl get svc -l app=k8s-helloworld # Check if the application can be accessed from Kubernetes Cluster curl ez-vm01.hpeilab.com:31856 </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/138665803-dea57cb9-1209-4b55-810a-5d564ea2b7e5.png" alt="image"></p> <p>At this point, you are half way done. Now, move to the Kubernetes tenant management GUI (shown in the image below). In the <code>Service endpoints</code> tab, you can see that the access point is not yet mapped to HPE Ezmeral Runtime Enterprise Gateway.</p> <p><img src="https://user-images.githubusercontent.com/72959956/138668470-ce8b6846-5fb4-4494-9a90-24aa2be73456.png" alt="image"></p> <h1>Making the application available to users</h1> <p>One more step is needed to expose your containerized application to users outside your HPE Ezmeral Runtime Enterprise infrastructure. You can expose the Kubernetes NodePort service of your application via the HPE Ezmeral Runtime Enterprise Gateway by setting up a port mapping. You have to apply a label <code>hpecp.hpe.com/hpecp-internal-gateway: "true"</code> on your NodePort Service object. You can do that by adding one line in your YAML files or run <code>label</code> command. The label generates a service port on the gateway host.</p> <blockquote> <p>Note:</p> <p>This behavior will be done automatically within any namespace associated with an HPE Ezmeral Runtime Enterprise tenant and if that tenant has the "Map Services To Gateway" enabled. However, you can control this behavior by labelling the NodePort service, either to force or to prevent the port mapping on the gateway host.</p> </blockquote> <pre><code class="language-bash"># Label the service named k8s-helloworld with hpecp.hpe.com/hpecp-internal-gateway=true kubectl label service k8s-helloworld hpecp.hpe.com/hpecp-internal-gateway=true </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/138669273-fa2969b3-61f3-4bae-a2f6-66425daf0a7b.png" alt="image"></p> <p>Go back to the Kubernetes tenant management GUI. Now, in the <code>Service endpoints</code> tab, you can see the access point is mapped to HPE Ezmeral Runtime Enterprise Gateway.</p> <p><img src="https://user-images.githubusercontent.com/72959956/138668836-0313c1c5-e720-4575-a759-842c85d5502c.png" alt="image"></p> <p>You can also find the port by running the command <code>kubectl describe services</code>. The access point will be shown under the key annotations.</p> <p><img src="https://user-images.githubusercontent.com/72959956/138810536-f1255048-2d91-44eb-ba33-ccc4bc52ca1e.png" alt="image"></p> <p>Now, the services has mapped to the gateway and you can now access it though the gateway. A hello world message will appear when you <code>curl</code> the URL.</p> <pre><code class="language-bash"># check your port number on the GUI curl http://ez-gateway.hpeilab.com:10022/ </code></pre> <h1>Deleting the services and deployment</h1> <p>After playing around with Kubernetes, if you would like to clean up the application, remember to delete both services and the deployment.</p> <pre><code class="language-bash"># delete everything kubectl delete services/k8s-helloworld kubectl delete deployment/k8s-helloworld </code></pre> <h1>Take away</h1> <p>Mapping Kubernetes services to HPE Ezmeral Runtime Enterprise Gateway provides a single point of secure access to the platform, which also allows for load-balancing. As you can see from what we just went through, it really isn't that hard. Feel free to navigate to HPE DEV Hack Shack to register for a <a href="/hackshack/workshop/24">Workshop-on-Demand</a> for Kubernetes 101. Here, you can try it for yourself. After playing around with this, if you would like to clean up the application, remember to delete both the services and the deployment. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> to learn more on other <a href="https://developer.hpe.com/platform/hpe-ezmeral/home/">HPE Ezmeral Runtime Enterprise</a> related topics.</p>https://developer.hpe.com/2021-December-2/https://developer.hpe.com/2021-December-2/Fri, 03 Dec 2021 05:00:00 GMThttps://developer.hpe.com/oauth2-for-hpe-greenlake-data-services-cloud-console/https://developer.hpe.com/oauth2-for-hpe-greenlake-data-services-cloud-console/Tue, 30 Nov 2021 16:07:28 GMT<h2>HPE GreenLake API Security</h2> <p>In my <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">blog</a> post, <strong>Using HPE GreenLake Console's API Gateway for Data Services Cloud Console</strong>, I explained that the HPE GreenLake console supports the Client Credential authentication grant type (This concept is known as OAuth 2 client credential authorization workflow). This particular grant type allows the client application (scripts, applications, programs that leverage the console features via the API) to authenticate using separate credentials (client id and client secret) that are authorized inside the API Gateway menu using the HPE GreenLake user account (resource owner).</p> <p><img src="/img/dscc-public-api-introduction-updated_111122.jpg" alt="Rehash the flow of the GreenLake access token acquisition" title="Client Credentials process"></p> <p><strong>Some of the benefits of the Data Services Cloud Console Client Credential OAuth authentication grant:</strong></p> <ol> <li>The authorization for the client does not involve the transmission of the HPE GreenLake user credentials.</li> <li>Changing the client secret or deleting the client credentials will not impact HPE GreenLake user credentials.</li> <li>According to OAuth 2.0 Authorization Framework, <a href="https://datatracker.ietf.org/doc/html/rfc6749#section-4.4">the Client Credentials Grant type</a> allows the client application to authenticate itself independent of the user (no user intervention), which makes this grant type appropriate for machine-to-machine (M2M) applications that can safely protect the registered client credentials (confidential clients), such as scripts, daemon, or services contained in a host. Please refer to the <a href="https://tools.ietf.org/html/rfc6749#section-2.1">OAuth 2.0 Authorization Framework</a> for more information.</li> <li>Each client application uses a different set of client ids and client secrets to ensure the secrecy and the independence of each client application.</li> </ol> <h3>How do I implement Data Services Cloud Console API in my client application or my script?</h3> <p>This blog post will go through an example of setting up the client application using the client id, client secret, console API definition in yaml, and the Postman tool. The flow to get the client id and client secret from this menu is detailed in my <a href="https://developer.hpe.com/blog/api-console-for-data-services-cloud-console/">blog</a> entitled <strong>Using HPE GreenLake's API Gateway to Data Services Cloud Console.</strong> Note that the client id and client secret are shown only once during the API credential creation; hence they need to be securely recorded.</p> <p><img src="/img/credentials-created-client.png" alt="image of the client id and client secret" title="Client Credentials"></p> <p>The user who generates this client id and client secret pair must store them and transfer them securely to the designated client application. Using the client id and the client secret, the client application can generate the access token in order to issue the REST API request to resources in the console. The client application access to the permitted console's resources depends on the role-based access control (RBAC) of the user. If the user does not have the correct authority for a resource, such as the volumes of an array, then the REST API request will receive an "unauthorized request" response.</p> <p>For the client application to perform the REST API request, the application must obtain the access token from HPE GreenLake as described in below diagram. This special end-point <code>https://sso.common.cloud.hpe.com/as/token.oauth2</code> provides the access token in the response of the authorization request from any client application.</p> <p><img src="/img/client-credential-access-token.png" alt="Diagram for client credential " title="Client Credential"></p> <p>The method required to obtain the access token is described in the following HTTPs request as shown in this snippet of HTTP strings.</p> <pre><code class="language-markdown">HTTP method: POST URL: https://sso.common.cloud.hpe.com/as/token.oauth2 Headers: Content-Type: application/x-www-form-urlencoded Body: grant_type=client_credentials &#x26;client_id=xxxxxxxxxx &#x26;client_secret=xxxxxxxxxx </code></pre> <p>The following code snippet shows an example using cURL to obtain the access token. The variables of the $YOUR_CLIENT_ID and $YOUR_CLIENT_SECRET will be substituted with the client id and client secret from the above menu.</p> <pre><code class="language-shell">curl -X POST https://sso.common.cloud.hpe.com/as/token.oauth2 -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&#x26;client_id=$YOUR_CLIENT_ID&#x26;client_secret=$YOUR_CLIENT_SECRET" </code></pre> <p>The next code snippet shows an example using Python to obtain the access token. As in the previous code snippet, YOUR_CLIENT_ID and YOUR_CLIENT_SECRET variables must be substituted accordingly.</p> <pre><code class="language-python">from oauthlib.oauth2 import BackendApplicationClient from requests.auth import HTTPBasicAuth from requests_oauthlib import OAuth2Session client = BackendApplicationClient(YOUR_CLIENT_ID) oauth = OAuth2Session(client=client) auth = HTTPBasicAuth(YOUR_CLIENT_ID, YOUR_CLIENT_SECRET) token = oauth.fetch_token(token_url='https://sso.common.cloud.hpe.com/as/token.oauth2', auth=auth) print(token["access_token"]) </code></pre> <p>Another code snippet below shows an example using the PowerShell to obtain the access token. As in the previous code snippet, the client id and the client secret variables must be substituted accordingly.</p> <pre><code class="language-powershell">$AuthUri = "https://sso.common.cloud.hpe.com/as/token.oauth2" [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 $AuthHeaders = @{ 'Content-Type' = 'application/x-www-form-urlencoded' } $AuthBody = [ordered]@{ 'grant_type' = 'client_credentials', 'client_id' = {Insert Client ID Here}, 'client_secret' = {Insert Client Secret Here } } (invoke-restmethod -uri "$AuthURI" -Method Post -headers $AuthHeaders -body $MyBody).access_token </code></pre> <p>The access token contains the information in JWT format that is self-contained, securely encoded, and signed using RS256. The access token is designed to enable secure transmission between the client application and the REST API server with a limited lifetime (2 hours).</p> <pre><code class="language-http">eyJhbGciOiJSUzI1NiIsImtpZCI6IjFvVEFmay1UOTZ1ZDd5cDBZTGlYM1ROSWdDWSIsInBpLmF0bSI6ImRlejAifQ.eyJjbGllbnRfaWQiOiIwMGNmZmY3MC04NmFiLTRmNjYtODI0NS0xZWIwNTQ2MzljMzgiLCJpc3MiOiJodHRwczovL3Nzby5jb21tb24uY2xvdWQuaHBlLmNvbSIsImF1ZCI6ImV4dGVybmFsX2FwaSIsInN1YiI6InJvbmFsZC5kaGFybWFAaHBlLmNvbSIsInVzZXJfY3R4IjoiZThhNGRhMmVlZmMzMTFlYmEwMmNiNjAzNDIyYmMwYTAiLCJhdXRoX3NvdXJjZSI6ImNjc190b2tlbl9tYW5hZ2VtZW50IiwicGxhdGZvcm1fY3VzdG9tZXJfaWQiOiIyMzRkNzZjNmU5ZDAxMWViYjczMDgyYjIxMmFkNmZlYSIsImlhdCI6MTYzNDc3OTIwNiwiYXBwbGljYXRpb25faW5zdGFuY2VfaWQiOiIzYzE4YmQwMy04MzA2LTRjN2MtOTQyZS1jNzA0YTRiODc0NGMiLCJleHAiOjE2MzQ3ODY0MDZ9.sz7GHvCdO_NjPgVt5rz7JHRSegZWD0pimNqiw7s_SC9vB2XsQnSEP71Kh1y3SqQxkKF8AgbJ02iEZYsk-GO-JmufGfeIUbl2idrFlfXPiKsKftn35dHO-uHW8s4KwL7mUF_HiPxUPIsixQ1zS_88-qdUGzAWDjcR0JO2gKnkaWeQ_AUGzdDw09ZSYZG3sxIoqU_HNjLF1c8hJmVV9Q6IN1ItKAspECc_UYTnjUBrZz5mpupDxuLIMJytTFUFwCriphi9cXQCTyQ3TXW_EALtRq_KdLEe311WFMX9mAL87zXP2JNc8bf8CTiiAty5eCjM2wxrPK9-ep0i5J5v6kJW_Q </code></pre> <p>Some of the information inside the JWT details the client id, source of authentication, and other details, including time of expiration (in Unix epoch time).</p> <pre><code class="language-json">{ "client_id": "00cfff70-86ab-4f66-8245-1eb054639c38", "iss": "https://sso.common.cloud.hpe.com", "aud": "external_api", "sub": "xxxxxxxxxxxxxx@hpe.com", "user_ctx": "e8a4da2eefc311eba02cb603422bc0a0", "auth_source": "ccs_token_management", "platform_customer_id": "234d76c6e9d011ebb73082b212ad6fea", "iat": 1634779206, "application_instance_id": "3c18bd03-8306-4c7c-942e-c704a4b8744c", "exp": 1634786406 } </code></pre> <h4>I don't know any programming language. How can I explore this console's REST API?</h4> <p>Postman is the well-known tool used to explore a REST API that provides enough flexibility to import the API definition, automate the access token retrieval and experiment with the parameters. Postman provides this capability with a lot less of typing and knowledge of any programming language. To experiment with Postman, I recommend you download the application-based (rather than web-based) version. This is the <strong><a href="https://www.postman.com/downloads/">download link</a></strong> for the Postman app, which is available in either a Microsoft Windows or Apple MacOS version. Install the application on a workstation that has access to the internet via website browser (HTTPS) and is capable of connecting to <em><strong>cloud.hpe.com</strong></em> such as <a href="https://common.cloud.hpe.com">https://common.cloud.hpe.com</a> or <a href="https://us-west.data.cloud.hpe.com">https://us-west.data.cloud.hpe.com</a>.</p> <p>Start by downloading the storage-api.yaml OpenAPI 3.0 definition into the workstation. Anyone can go to the <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">DSCC API documentation website</a> and click on the download button for either the Yaml or JSON definition file.</p> <p>Postman provides the ability to create a workspace where one can experiment with the DSCC OpenAPI by importing the DSCC OpenAPI definition (storage-api.yaml) into the API library. Start the process by selecting the Workspaces menu, then click on the <strong>Create Workspace</strong> button and type in the desired workspace name. In this example, it's called HPE DSCC API.</p> <p><img src="/img/postman-create-workspace.png" alt="Create Workspace" title="Create the workspace in the Postman"></p> <p>Inside this new workspace, you will need to create an environment variable called {baseUrl} that represents the baseURL toward the console's API path for that specified region. This is the current list of the URLs based on the region where the console are allowed to be deployed as of November 2021:</p> <table> <thead> <tr> <th>Data Services Cloud Console Region</th> <th>base-URL</th> </tr> </thead> <tbody> <tr> <td>EU Central (Europe)</td> <td><a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></td> </tr> <tr> <td>AP Northeast (Asia Pacific)</td> <td><a href="https://jp1.data.cloud.hpe.com">https://jp1.data.cloud.hpe.com</a></td> </tr> <tr> <td>US West (United States)</td> <td><a href="https://us1.data.cloud.hpe.com">https://us1.data.cloud.hpe.com</a></td> </tr> </tbody> </table> <p>In this example, the baseURL points to a testing instance of console (<a href="https://scint-app.qa.cds.hpe.com">https://scint-app.qa.cds.hpe.com</a>). For your exercise, you must replace this value with any of the baseUrl values that match the region where the console is deployed based on the above table. For any activities issuing the console's API request, configure the environment context to "DSCC testing" under the "HPE DSCC API" workspace, and ensure that this environment contains the current value of the variable of {baseUrl}.</p> <p><img src="/img/postman-create-environment-variable.png" alt="Set Enviroment" title="Set baseUrl under this environment under the workspace"></p> <p>Next, import the Data Services Cloud Console API definition into this workspace. Note that you will be importing the storage-api.yaml rather than JSON; nevertheless, Postman can recognize the console API in either format. To save the time required for importing, you can uncheck the <strong>Create Documentation</strong> button. If need be, the documentation can be created after the importing.</p> <p><img src="/img/postman-import-api.png" alt="Select upload files to import API" title="Upload API definition"></p> <p>Select the Data Services Cloud Console API definition in yaml format that was downloaded from the <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">HPE Data Services Cloud Console API documentation</a> by clicking on the <strong>Download YAML</strong> button as shown below.</p> <p><img src="/img/download-dscc-api-definition.png" alt="Download the API definition" title="Save the API definition"></p> <p>Once the API definition completes the import, Postman recognizes the fact that it's an OpenAPI 3.0 definition and builds the API library automatically.</p> <p><img src="/img/postman-import-load.png" alt="Confirmation of the API Definition Import" title="OpenAPI 3.0 Confirmation"></p> <p>As a result of the upload of the console API definition file, Postman will show the API definition tree under the Postman API menu as shown in the picture below. Note that Postman validates that the console API definition doesn't contain any issues, as shown by the message at the very bottom of the following picture.</p> <p><img src="/img/postman-api-loaded.png" alt="DSCC API loaded" title="DSCC API definition loaded"></p> <p>After the Data Services Cloud Console API definition is loaded, you can then use the automation for obtaining that access token that is facilitated by Postman. To start the OAuth2 automation, select the <strong>Collections</strong> menu and display the rest of the API listing under the tree. At the top of the tree, you will initialize the authorization with correct parameters, such as the client id, client secret, Data Services Cloud Console OAuth2 end point, and other required parameters. With this setup, any API request that inherits the authorization from the top of tree will be able to populate their header for REST API request with the access token as the token bearer. Below, you can see a display of the configuration that is populated with the required parameters under the authorization menu.</p> <ol> <li><strong>Type</strong> = OAuth 2.0</li> <li><strong>Add auth data to</strong> = Request Headers</li> <li><strong>Token Name</strong> = <Strings></li> <li><strong>Grant Type</strong> = Client Credentials</li> <li><strong>Access Token URL</strong> = <a href="https://sso.common.cloud.hpe.com/as/token.oauth2">https://sso.common.cloud.hpe.com/as/token.oauth2</a></li> <li><strong>Client ID</strong> = obtained from the client credential creation in API Gateway</li> <li><strong>Client Secret</strong> = obtained from the client credential creation in API Gateway (The pair to the Client ID)</li> <li><strong>Client Authentication</strong> = Send client credentials in body</li> </ol> <p>Click on the <strong>Get New Access Token</strong> button to obtain the valid access token</p> <p><img src="/img/postman-setup-access-token-at-top-folder.png" alt="Setting Up Authorization" title="Automation for OAuth 2.0"></p> <p>As a result of getting the new access token, a new menu will be shown that shows the content of the access token. Click the <strong>Use Token</strong> button to select this new access token. If there is an existing access token, it will be invalidated and the name of that invalid access token will have a strike-through, as shown in the image below.</p> <p><img src="/img/postman-use-access-token.png" alt="Obtain New Access Token" title="Valid Access Token"></p> <p>Once selection to use the valid access token is done, this token can be made available for all of the API Requests under the same tree. To make this happen, click on the <strong>Sync Access Token</strong> button until it's synced-up (the icon color changes to green). To enable this sync-up, please select <strong>Delete Expired Tokens</strong> and select <strong>Available Tokens</strong>. Lastly, ensure that the header prefix is set to <strong>Bearer</strong>.</p> <p><img src="/img/postman-setup-sync-token-at-top-folder.png" alt="Access Token is sync-ed" title="Sync access token"></p> <p>After the access token is synced-up, you can then issue any REST API request by selecting the appropriate REST API request. For this example, you are going to issue the REST API to show all of the storage systems connected to the Data Services Cloud Console(<strong>Get all Storage systems</strong>). Select the <strong>Headers (7)</strong> to display the parameters of the REST API header and note that the Authorization parameter contains the valid access token.</p> <ul> <li>Note that the environment selection (the menu at top right) must be set to the above mentioned environment (console testing for the current exercise) using the <strong>V</strong> button.</li> </ul> <p><img src="/img/postman-set-correct-environment.png" alt="Use the correct environment" title="Correct Environment Variable"></p> <ul> <li>Select the <strong>Authorization</strong> tab and set the <strong>Type</strong> to <strong>inherit auth from the parent's</strong> authorization to allow this REST API request to use the access token acquired at the top of the console API tree. Note that this inheritance requires the sync of the valid access token obtained at the top of the tree.</li> </ul> <p><img src="/img/postman-set-inherit-auth.png" alt="Inherit Authorization" title="Parent&#x27;s Authorization"></p> <p>For the first REST API request in this example, you will issue the <strong>Get all storage systems</strong> API request. This request can return a huge amount of data depending on the arrays that were registered in this instance of the console. To enable concise information to be returned from this REST API request, we can check the following parameters and uncheck all others:</p> <ul> <li><strong>sort</strong> = id asc, name dscc</li> <li><strong>select</strong> = id</li> </ul> <p>Click on the <strong>Send</strong> button to issue the GET all storage system API request. Within seconds, the body section on the bottom of this menu is filled with the list of the available arrays. It's very easy to use, simple, with no programming required and minimal typing.</p> <p>Isn't it awesome?</p> <p><img src="/img/postman-get-storage-system-sort-select-id-only.png" alt=""></p> <p>I hope you find this blog post is helpful in using the Data Services Cloud Console public REST API. More blog posts will be coming to help you take further advantage of its capabilities. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more blog posts about HPE Data Services Cloud Console REST API.</p>https://developer.hpe.com/api-console-for-data-services-cloud-console/https://developer.hpe.com/api-console-for-data-services-cloud-console/Tue, 30 Nov 2021 15:28:17 GMT<h2>Secured, Yet Agile</h2> <p>A major guiding principle in the creation of the Application Programming Interface (API) for Data Services Cloud Console from Hewlett Packard Enterprise (HPE) is security. However, to be able to be used by applications or tools that rely on the API to extend their features using the console, the API must also be flexible. To provide both security and flexibility, the console's REST API uses the 0Auth 2.0 authentication flow based on the client credential, which generates a limited lifetime access token. This access token is then embedded in the header of each REST API request as the authorization bearer.</p> <p>This blog will walk through the essential steps required to exercise or experiment with the Data Services Cloud Console REST API.</p> <h3>Authentication Process to Obtain the Access Token</h3> <p>The Data Services Cloud Console public API relies on an OAuth 2.0 third party authorization framework on behalf of the resource owner (HPE GreenLake's user) for security. The user starts by logging and authenticating into HPE GreenLake console, which is authenticated by the Identity Provider (validated through username, password, or Multi Factor Authentication). Using the API gateway menu in the HPE GreenLake, a customer registers their client application (REST API client) to obtain the OAuth 2.0 API client credentials (client id and client secret). This association allows the user to obtain the access token from the menu, and the user can then use the access token inside the token bearer field (header) with any REST API request. This action allows any client application or script to perform any API request to the correct instance of Data Services Cloud Console.</p> <p><img src="/img/greenlake-api-access-flow.png" alt="client-credential application flow" title="obtain client-id and client-secret"></p> <p>The access token has a limited lifetime (about 7200 seconds or 2 hours). Once it expires, the client application must use the obtained client id and client secret to generate a new access token. One indication of the expiration of the access token, the request to console's API will return a response error: '401 Unauthorized HTTP.' If the client application generates a new access token before the current one has expired, then the current access token will be invalidated or treated as not authorized.</p> <p>Additionally, a user can also change the client secret to update the authorization when the authorized client application has lost it's client secret, or when the client secret has been compromised.</p> <p>And lastly, when access to the console's REST API must be disabled, a user can delete the API client credential associated with client id and client secret in the API Gateway menu.</p> <p>The following flow chart describes steps required to perform the console's REST API request. The flow starts from the GreenLake authorized user creating the client id and client secret to be used to obtain the access token. The access token will be used in the authorization bearer to ensure the secure REST API request.</p> <p><img src="/img/user-guide-for-authorization.png" alt="Access API process" title="Process to authenticate and to obtain secure access "></p> <h2>Accessing the API Gateway Menu</h2> <p>To access the API gateway menu, the user must log into the <a href="https://common.cloud.hpe.com">HPE GreenLake,</a> deployed Data Services Cloud Console to the intended region, and onboarded a storage array (HPE Alletra, HPE Nimble, or HPE Primera) into the organization that is associated with the user account. The user must have the role that is required to perform the intended operation at the instance of console where the storage has been deployed. For instance, the user must have volume management capability in the Data Ops Manager to create a storage volume in US region. For more information about the role based access control, please take a look at the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=ccs-help_en_us">HPE GreenLake User Guide</a></p> <p>The API Gateway menu is available inside the HPE GreenLake's Manage menu. From HPE Greenlake Console click on Menu to access this Manage Menu.</p> <p><img src="/img/accesing-manage-menu-from-console.png" alt="Access Menu " title="Menu in Cloud Console"></p> <p><img src="/img/accesing-api-gateway-from-manage-menu.png" alt="CCS Menu" title="GreenLake Common Cloud Menu"></p> <p>The API Gateway menu provides the following operations:</p> <ol> <li> <p>Creates and manages API client credential association in order to obtain:</p> <ol> <li>Instance ID of the Data Services Cloud Console at the particular region (Hexadecimals).</li> <li>Client ID (Hexadecimals).</li> <li>Client Secret (Hexadecimals).</li> </ol> </li> <li> <p>Generates access token, changes client secret, and deletes client credential.</p> </li> </ol> <p><img src="/img/api-gateway-block.png" alt="API Gateway Menu" title="DSCC API Gateway Menu"></p> <h3>Manages API client application</h3> <p>Each instance of API client credential represents the authorization relationship between the client application and the Data Services Cloud Console REST API resources. Please click on the Create Credentials button to generate a client credential. Afterwards, the user can obtain the client secret and client secret, and use them to generate the access token.</p> <p><img src="/img/create-credentials-button.png" alt="API client credentials" title="Create API Client Credentials"></p> <p>Inside the Create Credentials menu, click on the V button to show the pull down list and use the mouse to click on the desired application. For the purpose of using the Data Services Cloud Console REST API, please select the console instance in the region where the array has been deployed. The list shows all instances with the regions where the applications are deployed.</p> <p><img src="/img/select-the-desired-application.png" alt="select the application" title="Choose the application from Data Services Cloud Console"></p> <p>After selecting the correct application, enter the Credential Name (Please see <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=ccs-help_en_us">HPE GreenLake Cloud Console User Guide</a> for supported characters). Click the Create Credentials button to proceed with Client Credentials creation.</p> <p><img src="/img/create-credentials-menu.png" alt="Create Credential no input yet" title="Generate Client Credentials 1st time"></p> <p>Once the Create Credential button is clicked, the following information about the OAuth (Open Authorization) will be shown. These are the Client ID and Client Secret information menus that are going to be shown only once. Please copy both client id and client secret to a safely recorded location. In the case that the user missed copying this information, then the user can only regenerate the client secret at a later time using the Reset Client Secret menu.</p> <p><img src="/img/api-client-credential-created.png" alt="" title="Credentials Created Close"></p> <p>After closing the credential creation menu, the user can observe the prior created API client credential by identifying the credential name on the menu.</p> <p><img src="/img/application-credential-created-prior-shown.png" alt="" title="API Client Credentials are created"></p> <p>After clicking on the down arrow button, the user can see the Generate Access Token button in order to generate the access token required to be used for the Data Services Cloud Console REST API request. Please click on the Generate Access Token to generate an access token.</p> <p><img src="/img/api-client-credential-get-access-token.png" alt="" title="Time to obtain the Access Token"></p> <p>After clicking on the Generate Access Token button, the Generate Access Token menu will appear. The menu requires the user to enter the client secret obtained from the associated API client credential. The user must copy and paste the recorded client secret from Credential Created menu (in the previous image in this blog) so that the user can obtain the access token. Click on the Create Access Token button to generate the access token.</p> <p><img src="/img/generate-access-token-with-secret.png" alt="obtained from client credential." title="Use the client secret to generate Access Token"></p> <p>The Access Token Created menu will appear and shows the generated access token. Copy the access token using using the "sheets" icon and store this access token to a safely recorded location. Click Close button to continue.</p> <p><img src="/img/access-token-created-and-close.png" alt="" title="Access Token Generated and Consumed"></p> <p>Afterward, the user can embed the access token to the REST API request header in order to perform the HTTP method against the desired resource in order to obtain the response. Note that the user must use the correct base-URL according to the region where the DSCC is deployed. Currently these are the base-URL and the corresponding region where the DSCC is deployed (November 2021).</p> <table> <thead> <tr> <th>DSCC Region</th> <th>base-URL</th> </tr> </thead> <tbody> <tr> <td>EU Central</td> <td><a href="https://eu1.data.cloud.hpe.com">https://eu1.data.cloud.hpe.com</a></td> </tr> <tr> <td>AP Northeast</td> <td><a href="https://jp1.data.cloud.hpe.com">https://jp1.data.cloud.hpe.com</a></td> </tr> <tr> <td>US West</td> <td><a href="https://us1.data.cloud.hpe.com">https://us1.data.cloud.hpe.com</a></td> </tr> </tbody> </table> <h4>Oops! What if I missed to copy the client secret for this instance of client credential?</h4> <p>The client secret can be recreated inside the create credentials menu by clicking on the three dots at the bottom menu. Once the three dot menu opened, the user must click on the Reset Client Secret button to display the newly created client secret. This prior action will open the Credentials Created menu to show the new client secret for one time only. Note that user must copy this newly created secret to a secured location so that it can be used to generate new access token.</p> <p><img src="/img/reset-the-client-secret-in-client-credential-menu.png" alt="Reset the client secret" title="resetting the client secret"></p> <h4>Nice! Can you give me an example of using the access token?</h4> <p>The access token is a long string of JSON Web Token that is signed using RS256 algorithm. Note that the access token must be added into the header of with keyword "Authorization: Bearer <access token in JWT>" for any Data Services Cloud Console REST API request. This following example is based on ubiquitous cURL tool, and it uses "<a href="https://scalpha-app.qa.cds.hpe.com">https://scalpha-app.qa.cds.hpe.com</a>" as the base URL. Note that this base URL is the DSCC testing-site only. For your exercise, please use one of the base-URL noted in the above table. The following example of the DSCC REST API request uses GET method for this resource /api/v1/audit-events to obtain a list of the available audit-events. Note the additional parameter with keyword "Authorization: Bearer" is added into the header of this REST API request.</p> <pre><code class="language-shell">>curl -X GET https://scalpha-app.qa.cds.hpe.com/api/v1/audit-events -H "Accept: application/json" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IllUMU9MZWRYeDFCbHZ2and6OU1FNm8ya1BQayIsInBpLmF0bSI6ImRlejAifQ.eyJjbGllbnRfaWQiOiIwMGNmZmY3MC04NmFiLTRmNjYtODI0NS0xZWIwNTQ2MzljMzgiLCJpc3MiOiJodHRwczovL3Nzby5jb21tb24uY2xvdWQuaHBlLmNvbSIsImF1ZCI6ImV4dGVybmFsX2FwaSIsInN1YiI6InJvbmFsZC5kaGFybWFAaHBlLmNvbSIsInVzZXJfY3R4IjoiZThhNGRhMmVlZmMzMTFlYmEwMmNiNjAzNDIyYmMwYTAiLCJhdXRoX3NvdXJjZSI6ImNjc190b2tlbl9tYW5hZ2VtZW50IiwicGxhdGZvcm1fY3VzdG9tZXJfaWQiOiIyMzRkNzZjNmU5ZDAxMWViYjczMDgyYjIxMmFkNmZlYSIsImlhdCI6MTYzNzAwNjk0NSwiYXBwbGljYXRpb25faW5zdGFuY2VfaWQiOiIzYzE4YmQwMy04MzA2LTRjN2MtOTQyZS1jNzA0YTRiODc0NGMiLCJleHAiOjE2MzcwMTQxNDV9.gHcBzl0n2wwrMRR2tSbT6jHN68d1TSNT743GED3LuF2B08ABYh9ePKQjhqYW6mjY-oSfEW2BTfG7TfTzZj9MtQ2kJGmq3DvLBl6fAaN6MEkSIz54hu0PdmDW8His6oET2txq_0kp5XJ7T6n_QJzZY0xvSoquE-48gCxwGFPWIRwefIpdw_1URFXYgfdKCxCIDTdPfYKs8kD8hzwyF9uvgLgVPWZJD6b1UHJK5OpNnBOpAxrs1xfFBz688b0vheZdARCJsl5E3Qxjyg68hw2cjavZZOX-_RWpd6JWPrQnqxyxQeYQ5yYy7giVCViM5SUZkv6j0Ts3TVguapE2kvahkQ" </code></pre> <p>The response is returned in the form of JSON string, as shown in the below example. Note that, the user can use additional parameter of the REST API Get audit-events to filter particular events. Please take a look at the <a href="https://console-us1.data.cloud.hpe.com/doc/api/v1/">Data Services Cloud Console API documentation</a> for more information on additional parameters that are available for /api/v1/audit-events resource.</p> <pre><code class="language-javascript">{ "items": [ { "associatedResource": { "id": "/api/v1/host-initiators?filter=editStatus%20in%20(Update_In_Progress,Update_Success,Update_Failed,Not_Applicable,Delete_Failed)", "name": "", "type": "" }, "code": "Unauthorized privilege", "contextId": "", "customerId": "e8a4da2eefc311eba02cb603422bc0a0", "id": "9bafe7ae-84bf-42a2-9b82-2592ce62715e", "loggedAt": "2021-11-10T04:09:07Z", "message": "Unauthorized user access", "occurredAt": "2021-11-10T04:09:07Z", "permission": "data-services.host-initiator.read", "scope": "", "source": "/api/v1/host-initiators?filter=editStatus%20in%20(Update_In_Progress,Update_Success,Update_Failed,Not_Applicable,Delete_Failed)", "sourceIpAddress": "fleet-gql-data-graph:4000", "state": "PermissionDenied", "taskId": "", "uniqueId": "audit.events+0+3936", "userEmail": "mandy.shen@hpe.com", "version": 1 }, { "associatedResource": { "id": "/api/v1/host-initiators?filter=editStatus%20in%20(Update_In_Progress,Update_Success,Update_Failed,Not_Applicable,Delete_Failed)", "name": "", "type": "" }, "code": "Unauthorized privilege", "contextId": "", "customerId": "e8a4da2eefc311eba02cb603422bc0a0", "id": "f0d2c4c6-d859-42f3-ae4b-60f8d3d2d89d", "loggedAt": "2021-11-10T04:09:03Z", "message": "Unauthorized user access", "occurredAt": "2021-11-10T04:09:03Z", "permission": "data-services.host-initiator.read", "scope": "", "source": "/api/v1/host-initiators?filter=editStatus%20in%20(Update_In_Progress,Update_Success,Update_Failed,Not_Applicable,Delete_Failed)", "sourceIpAddress": "fleet-gql-data-graph:4000", "state": "PermissionDenied", "taskId": "", "uniqueId": "audit.events+2+3975", "userEmail": "mandy.shen@hpe.com", "version": 1 }, .... snippet .... { "associatedResource": { "id": "/api/v1/storage-systems/volumes?filter=isSystemVolume%20eq%20false\u0026limit=1\u0026offset=0", "name": "", "type": "" }, "code": "Unauthorized privilege", "contextId": "", "customerId": "e8a4da2eefc311eba02cb603422bc0a0", "id": "86988487-f7f4-403e-b33e-e15abfcd568a", "loggedAt": "2021-08-19T15:38:49Z", "message": "Unauthorized user access", "occurredAt": "2021-08-19T15:38:49Z", "permission": "data-services.volume.read", "scope": "", "source": "/api/v1/storage-systems/volumes?filter=isSystemVolume%20eq%20false\u0026limit=1\u0026offset=0", "sourceIpAddress": "scalpha-app.qa.cds.hpe.com", "state": "PermissionDenied", "taskId": "", "uniqueId": "audit.events+6+3057", "userEmail": "matt.haron@hpe.com", "version": "" } ], "total": 123, "numItems": 123, "pageLimit": 500, "pageOffset": 0 } </code></pre> <p>The recommended tool at this moment of time to experiment with the REST API for the console is the Postman which is downloadable from the <a href="https://www.postman.com/downloads/">Postman website</a>. The postman is a versatile tool, that anyone can copy the access token (or better to use the client id and client secret) from the API Gateway menu and issue a REST API request without using programming language. Furthermore, user can also test the parameters and format the responses of each REST API request using the Postman tool.</p> <p>In conclusion, this blog gives you a great example on how to obtain the access token and experiment with the Data Services Cloud Console REST API. Please take a look at the next blog on getting the access token programmatically to enable any client application using any familiar tool like Postman, use a programming, or a scripting language.</p>https://developer.hpe.com/autopilot-kubernetes-deployments-on-hpe-ezmeral-runtime-enterprise/https://developer.hpe.com/autopilot-kubernetes-deployments-on-hpe-ezmeral-runtime-enterprise/Tue, 23 Nov 2021 18:20:34 GMT<p>‘Autopilot’ is a concept that is familiar to many in the aviation and automobile industries. As it pertains to airplanes, autopilot systems aid pilots in focusing on crucial tasks by automating and controlling every aspect of the flight, from take-off to landing. Modern-day autopilot systems can be customized and pilots can decide on which features are to be manually controlled. Drawing parallels, in the context of Kubernetes, an autopilot system can manage the end-to-end automation of deployments as well as the maintenance of applications, relieving developers from the pain of manual intervention. A system like this incorporates the necessary checks and balances to ensure the health of the applications and the CI/CD (continuous integration/continuous delivery) pipelines.</p> <p>It’s important to note that the success of such an autopilot system relies on the extent to which developers can fine-tune and intervene in the autopilot capabilities. Just like a pilot warming up the aircraft during take-off, developers need to prepare their applications for an autopilot system to kick in. This largely depends on how the applications are structured/architected.</p> <p>In this post, I will cover autopilot systems for Kubernetes and how the combination of gopaddle and HPE Ezmeral Runtime Enterprise enables enterprises to speed up their modernization journey. Before I do that, let me first explain the need for such systems. It all starts with a demand for efficiency of the business operations.</p> <h2>Addressing the need to optimize operations</h2> <p>Application development has changed significantly in recent years, with the introduction of microservices, DevOps processes, and Kubernetes. Maintaining a traditional, monolithic application is arduous. It requires coordinated releases, as the entire team contributes to a single binary, which is then configured and re-configured to suit the hybrid environments it may encounter. Depending upon where the application is deployed, the application has to be configured to fit the underlying infrastructure. How you scale the application or how you react to an application’s downtime would greatly depend on where the application is being deployed and used. These sequential and tedious software releases result in sub-optimal business operations.</p> <p>When businesses start adopting a microservices architecture and Kubernetes, they gain multiple benefits. A microservices architecture is nimble and gives developers the independence required to build and deploy their services as long as they conform to their API contracts. Using a microservices architecture, developers can more easily incorporate new features and ship them quickly. Kubernetes aids these agile practices by standardizing the infrastructure. Using these cloud-native technologies is a great win for businesses, as they help them respond to the market quickly and deliver more in a short period of time.</p> <p><img src="/img/fig1-k8s-complex-gopaddle.png" alt="Kubernetes is complex to manage" title="Kubernetes management requirements"></p> <p>But, even though the value of Kubernetes is apparent, implementing Kubernetes requires a huge cultural shift amongst developers. Developers are now responsible for the operational demands of how to configure and deploy their microservices and maintain those services at scale. This introduces a huge learning curve. In addition, developers end up in a cycle of maintenance and automation that, in turn, pushes them away from innovation. In other words, implementing Kubernetes can become stressful and counter-productive.</p> <h2>Autopilot System for Kubernetes</h2> <p>The answer to this double-edged requirement of balancing business efficiency without compromising developer productivity and morale is to leverage an autopilot system for Kubernetes. Such a system should abstract the complexity across the three stages of a Kubernetes implementation.</p> <ol> <li><strong>Infrastructure</strong> — Provisioning and maintaining Kubernetes environments</li> <li><strong>Application</strong> — Deploying and maintaining applications on Kubernetes</li> <li><strong>Automation</strong> — DevOps automation, i.e., CI/CD, monitoring, logging, and alerting</li> </ol> <p><img src="/img/fig2-hpe-gopaddle-solution.png" alt="Autopilot system for Kubernetes" title="Autopilot system for Kubernetes"></p> <p>HPE Ezmeral Runtime Enterprise and gopaddle offer a combined solution that addresses these end-to-end requirements for developers.</p> <p><a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a> provides you with an enterprise-grade container orchestration platform that is designed to run modern applications (both cloud-native and non-cloud-native monolithic applications) with persistent data.</p> <p><a href="https://gopaddle.io/">gopaddle</a> is a no-code platform used to build, deploy and maintain Kubernetes workloads across hybrid environments. It offers with three main capabilities:</p> <ol> <li><strong>Accelerate modernization</strong> — With its intelligent scaffolding, gopaddle can transform code into Kubernetes deployments in minutes.</li> <li><strong>Increase productivity</strong> — With its out-of-the-box tools integration and ready-to-use templates, gopaddle reduces the overhead of tedious maintenance and increases productivity.</li> <li><strong>Increase operational efficiency</strong> — gopaddle’s centralized control plane helps provision and manage Kubernetes clusters across clouds and on-premise/edge environments.</li> </ol> <h2>How does the solution work?</h2> <h3>1. Installing the gopaddle Enterprise edition on HPE Ezmeral Runtime Enterprise platform</h3> <p>gopaddle is available in two variations. gopaddle SaaS is a subscribe and pay-as-you-go model where the Kubernetes cluster managed by HPE Ezmeral Runtime Enterprise can be registered with gopaddle, securely via a bastion host. An external cluster in the context of gopaddle is a cluster that is not provisioned by gopaddle. The second variation described in detail in this post, is to install the gopaddle Enterprise edition on the Kubernetes cluster. gopaddle Enterprise edition is available as a Helm chart and can be installed on a Kubernetes cluster running on HPE Ezmeral Runtime Enterprise platform within the corporate firewall. To achieve a self-contained CI/CD environment within the corporate firewall, the GitLab repository and container registry can be installed on the Kubernetes cluster and registered with gopaddle.</p> <h3>2. Registering and configuring HPE Ezmeral Runtime Environment with gopaddle</h3> <p>Once gopaddle Enterprise Edition is installed on the Kubernetes cluster managed by HPE Ezmeral Runtime Enterprise, you can register that Kubernetes cluster securely with gopaddle for further application build and/or deployment processes.</p> <p><img src="/img/fig3-hpe-cluster-dashboard-gopaddle.png" alt="gopaddle dashboard" title="gopaddle dashboard"></p> <p>At the time of Kubernetes cluster registration, gopaddle also installs a set of add-ons, like Prometheus, Event exporters and Grafana, on the Kubernetes cluster for monitoring and alerting.</p> <p><img src="/img/fig4-hpe-addons-builds-apps-gopaddle.png" alt="Add-ons, container builds and applications" title="Add-ons, container builds and applications"></p> <p>As soon as the Kubernetes cluster is registered, gopaddle discovers the nodes and gives a visual representation of their configurations. Configurations like node labels and taints can be updated from the gopaddle UI.</p> <p><img src="/img/fig5-hpe-node-detail-gopaddle.png" alt="Kubernetes nodes details" title="Kubernetes nodes details"></p> <p>The node logs section can be used for troubleshooting HPE Ezmeral Runtime Enterprise managed Kubernetes node-related issues.</p> <p><img src="/img/fig6-hpe-node-logs-gopaddle.png" alt="Kubernetes nodes logs" title="Kubernetes nodes logs"></p> <p>The Kubernetes cluster can now be used for building and deploying applications.</p> <h3>3. Registering source control repositories and Docker registries</h3> <p>Before modernizing the applications, the source control repository and a container registry need to be registered with gopaddle. Below is a sample configuration for registering the on-premise GitLab repository installed on the HPE Ezmeral Runtime Enterprise managed Kubernetes cluster registered with gopaddle. First, a custom GitLab authenticator is registered and then the GitLab repository is registered using the custom authenticator. Once registered, the GitLab repository can now be used to create containers within gopaddle. Similarly, an on-premise GitLab container registry can be registered with gopaddle to push Docker images.</p> <p><img src="/img/fig7-hpe-code-rep-reg-updated-gopaddle.png" alt="GitLab source control repository and Container registry" title="GitLab source control repository and Container registry"></p> <h3>4. Building and deploying applications</h3> <p>Source code projects in the GitLab repository can be on-boarded either through the gopaddle’s intuitive wizard-based UI or through its command-line utility — <em><strong>gpctl</strong></em>.</p> <p>For instance, <em><strong>gpctl</strong></em> can be installed locally on the developer’s desktop and can be used to perform an intelligent scaffolding — a process that automatically converts a source code project to a Kubernetes deployment or a Helm chart. Here is an example of how <em>gpctl init</em> can be used to import the ‘Contentful’ application from the GitLab repository to the registered Kubernetes cluster running on HPE Ezmeral Runtime Enterprise. The outcome of this utility is the access URL of the application deployed on the HPE Ezmeral Runtime Enterprise. This transformation process takes less than 10 minutes to onboard the ‘Contentful’ application on the HPE Ezmeral Runtime Enterprise.</p> <p>As you can see, the HPE Ezmeral Runtime Enterprise is used seamlessly for both building the container image and for deploying the application. gpctl supports any type of Linux workloads like NodeJS, Java, Python, GoLang, Ruby, .NET Core. More information on <em><strong>gpctl</strong></em> can be found <a href="https://help.gopaddle.io/en/articles/5056807-initializing-a-microservice-from-scratch">here</a>.</p> <p><img src="/img/fig8-hpe-gpctl-init-1-gopaddle.png" alt="gopaddle gpctl command line interface" title="gopaddle gpctl command line interface"></p> <h3>5. CI/CD automation</h3> <p>To enable continuous integration for the containers, the CI toggle must be enabled for the container from the gopaddle UI. When enabled, any changes to the source code will be automatically detected and a new container build will be triggered. Using gopaddle’s alerts and notification channels, the Jenkins pipeline can be triggered to run a custom workflow — say, run a regression suite or perform a rolling update on an application.</p> <p><img src="/img/fig9-enable-ci-gopaddle.png" alt="gopaddle enables Continuous Integration" title="gopaddle enables Continuous Integration"></p> <h3>6. HPE Ezmeral Runtime Enterprise for Stateful Applications</h3> <p>gopaddle integrates with HPE Ezmeral Runtime Enterprise seamlessly to manage stateful workloads. HPE Ezmeral Runtime Enterprise provides a pre-integrated Data Fabric (formerly known as MapR Data Platform) for persistent volumes for stateful applications that require persistence of data The below screenshot gives an overview of the Data Fabric volumes and the applications/services using those volumes.</p> <p><img src="/img/fig10-hpe-volumes-gopaddle.png" alt="gopaddle dashboard for volumes" title="gopaddle dashboard for volumes"></p> <p>The HPE Ezmeral Runtime Enterprise and gopaddle joint solution is a great autopilot solution for developers. While HPE Ezmeral Runtime Enterprise offers a robust enterprise-grade Kubernetes environment, gopaddle automates the deployment and application lifecycle on the HPE Ezmeral platform. Developers can now focus on creativity, innovation, and collaboration rather than spending their time on resolving configuration issues. At the same time, businesses can achieve operational efficiency without compromising on developer productivity.</p> <p>More information on the HPE Ezmeral Runtime Enterprise and gopaddle joint solution can be found <a href="https://www.hpe.com/psnow/doc/a50005229enw">here</a>. Check back on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more articles on how to automate and improve your DevOps environment. You can follow Vinothini (Founder &#x26; CEO @gopaddle.io) and her team on Twitter <a href="https://twitter.com/gopaddleio?lang=en">@gopaddle.io</a>. You can also reach Vinothini directly on <a href="https://www.linkedin.com/in/vinothini-raju-9817ab5/">LinkedIn</a>.</p>https://developer.hpe.com/stackstorm-simple-elegant-automation-for-everyone/https://developer.hpe.com/stackstorm-simple-elegant-automation-for-everyone/Thu, 18 Nov 2021 15:16:44 GMT<p>There is so much information speeding through a data center from one resource to another - vCenter, storage arrays, network controllers, switches, routers, power systems, cooling systems, etc. Yet, oddly enough, these resources don't all speak the same language, hindering communications and their ability to react to one another. But, what if routers could could gain some insightful knowledge from a storage array? Or, maybe HPE OneView could automatically exchange vlan information with an Aruba Cx switch?</p> <p>Since just about every piece of hardware today has a Restful API (application programming interface), there is actually a way to make this happen. Using a high-level programming language, like Python, one could develop some code to leverage those APIs to get traffic flowing between two disparate types of systems.</p> <p><img src="/img/actors.png" alt=""></p> <p>Sounds like a lot of code to write. For every system I want to talk to, I will have to write a separate module. For example, in vCenter, I want to build a new port group. Wouldn't it be awesome if the act of creating the port group would automatically trigger something to add that port group information to a new VLAN (virtual LAN) Aruba central? I could write a python script to get that information from vCenter and then refactor it, and use a different API to send the information to Aruba Central. I still wouldn't have the automation I am looking for unless I scheduled a <strong>cron job</strong> to run my code. The task becomes exponentially harder when I want to add another device like HPE OneView into the conversation.</p> <p>I like writing code, but prefer to work smarter, not harder. After all, there is only so much time in the day. That is why I love using Python. Python has a lot of pre-written modules that make it extremely extensible. By using the <strong>import</strong> command in my script I can take advantage of modules I have no idea how to even start writing. I like the <strong>plug and play</strong> of how easy it is to consume those extra modules. I like using <strong>requests</strong>, because I can just code <strong>import requests</strong> and I have access to a well-developed chunk of code that I did not have to write.</p> <p>What if we applied the same concept to our problem at the start of this blog? What if each of the resources in the data center had a <strong>module</strong> of some sorts, that I could plug into some sort of framework? Something like StackStorm!</p> <p><img src="/img/stackstorm.png" alt=""></p> <p>Before I go too far, let me give you a quick recap of what <a href="https://developer.hpe.com/blog/master-the-automation-universe-the-easy-way-part-1-introduction-to-stack/">StackStorm</a> is. StackStorm uses <strong>sensors</strong>, <strong>rules</strong>, <strong>triggers</strong>, <strong>actions</strong>, and <strong>workflows</strong>. You can think of it as <strong>If-this-then-that</strong> automation. Based on a event on some device, a StackStorm sensor can detect it and this could trigger a rule. The rule would have some logic in it and if the trigger launched the rule, the rule would run an action, or a series of actions called a workflow. It might sound a little involved but all of the rules, actions, sensors and workflows are packaged together in a StackStorm pack to make it easy to deploy these solutions.</p> <p><img src="/img/process.png" alt=""></p> <p>What if I told you there was a place called the StackStorm Exchange that already had 170 of these modules you can plug into your projects? <a href="https://exchange.stackstorm.org/">Look at this link</a>, It is quite an impressive stockpile of automation just waiting for you to take advantage of it. I recently finished developing a StackStorm integration pack for HPE Primera Storage. Before that, I developed a pack to integrate HPE Nimble Storage. What communication can we develop?</p> <p>Using these packs, I went about trying to determine what sort of communication could be developed between the two storage systems, I settled on volumes. I wanted to be able to create a volume on the HPE Primera and have it <strong>automatically</strong> appear on a HPE Nimble storage array. Maybe this is something cool, maybe not, but it was just going to demonstrate the power of HPE devices talking to one another and have the capability to talk to the StackStorm integration packs out on the exchange. The process would look something like this:</p> <p><img src="/img/flow.png" alt=""></p> <p>The most important thing to remember is that when I run a StackStorm workflow, one step in the process can be running a StackStorm action from one integration pack, and the next task can be a StackStorm action from any other pack! Now, I have StackStorm integration packs for HPE OneView, HPE Nimble, and HPE Primera. Using these StackStorm integration packs allows HPE OneView, HPE Nimble, and HPE Primera communicate with one another. Because they are part of StackStorm, they now have access to all the pre-written automation packs available on the StackStorm Exchange!</p> <p>I have written StackStorm integration packs for <strong><a href="https://github.com/xod442/Stackstorm-qumulo">Qumulo</a></strong>, HPE <strong><a href="https://github.com/HewlettPackard/stackstorm-hpe-oneview">OneView</a></strong>, <strong><a href="https://github.com/HewlettPackard/stackstorm-aruba-fc">Aruba Fabric Composer</a></strong>, <strong><a href="https://github.com/xod442/stackstorm-hpe-iloamplifier">iLo-amplifier</a></strong>, <a href="https://github.com/HewlettPackard/stackstorm-hpe-nimble">HPE Nimble</a>. and even one for <strong><a href="https://github.com/xod442/stackstorm-arista-dev">Arista Cloud Vision Portal</a></strong>. The list has grown quite a bit since I started this endeavour. What is the net result of this activity? The communication between servers, and storage, and networking in the data center just got a little better. I challenge myself, as well as you, to think about the solutions that could be developed by applying this automation. It is said that any process you can document, you should automate. If you can automate the process you no longer have to do it yourself and that can give you back precious minutes of your busy schedule.</p> <p>If you have something you would like to have a StackStorm integration pack developed for, you can take action. You can start by reading my other blog posts <a href="https://developer.hpe.com/search/?term=stackstorm">here:</a> You could check out the <a href="/hackshack/workshop/21">Workshop-on-Demand</a> or take my free <a href="https://github.com/xod442/stackstorm-tutorial">StackStorm introductory training </a>and start your journey on your way to simple, elegant automation.</p>https://developer.hpe.com/2021-November-4/https://developer.hpe.com/2021-November-4/Fri, 05 Nov 2021 05:00:00 GMThttps://developer.hpe.com/build-transformative-ai-applications-at-scale-with-hpe-cray-ai-development-environment/https://developer.hpe.com/build-transformative-ai-applications-at-scale-with-hpe-cray-ai-development-environment/Thu, 04 Nov 2021 09:34:21 GMT<p>Building and training optimized machine learning (ML) models at scale is considered the most demanding and critical stage of ML development. Doing it well requires researchers and data scientists to overcome many challenges typically encountered in High Performance Computing (HPC) environments.</p> <p>These challenges often include properly setting up and managing a fast-moving ML software ecosystem and infrastructure spanning specialized compute, storage, network fabric, and tensor processors (e.g., GPUs). Additionally, users need to program, schedule, and train their models to maximize the use of the highly specialized infrastructure they have set up, which can create complexity and impede productivity.</p> <p>To meet these challenges, ML Engineers and data scientists are on a never-ending search for novel and innovative solutions that help them focus on building better models and accelerate their time-to-production. <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">HPE Machine Learning Development Environment</a> is designed to help them specifically achieve this.</p> <p>Built upon the widely popular open-source <a href="https://www.determined.ai/">Determined Training Platform</a>, HPE Machine Learning Development Environment reduces the complexity and cost associated with machine learning model development by removing the need to write infrastructure code and makes it easy for IT administrators to set up, manage, secure, and share AI compute clusters.</p> <p>By adopting HPE Machine Learning Development Environment, model developers can:</p> <ul> <li> <p><strong>Train models faster</strong> using state-of-the-art distributed training without changing the model code.</p> </li> <li> <p><strong>Automatically find high-quality models</strong> with advanced hyperparameter tuning from the creators of Hyperband.</p> </li> <li> <p><strong>Maximize GPU performance</strong> with smart scheduling and cut cloud GPU costs by seamlessly using spot/preemptible instances.</p> </li> <li> <p><strong>Track and reproduce work</strong> with experiment tracking that runs out-of-the-box, covering code versions, metrics, checkpoints, and hyperparameters.</p> </li> </ul> <p>HPE Machine Learning Development Environment integrates these features into an easy-to-use, high-performance machine learning environment — which means you can spend your time building models instead of managing infrastructure.</p> <p>To learn more about HPE Machine Learning Development Environment <a href="https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html">visit our landing page</a> and get in touch with our team of ML and distributed system experts.</p> <p>To learn more about the open-source project that powers HPE Machine Learning Development Environment, we invite you to check out the Determined Training Platform on <a href="https://github.com/determined-ai/determined">GitHub</a>, read the <a href="https://docs.determined.ai/latest/">Documentation</a>, and <a href="https://join.slack.com/t/determined-community/shared_invite/zt-cnj7802v-KcVbaUrIzQOwmkmY7gP0Ew">join the Determined Community Slack</a> to get started. Stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> for more informative articles on this subject.</p>https://developer.hpe.com/how-fine-grained-data-placement-helps-optimize-application-performance/https://developer.hpe.com/how-fine-grained-data-placement-helps-optimize-application-performance/Fri, 22 Oct 2021 17:49:57 GMT<p>Does data locality matter? In an ideal world, after all the work you put into developing an analytics or AI application, you would have unlimited access to resources to run the application to get top performance when it’s deployed in production. But the world is not perfect.</p> <p>Access may be hampered through latency caused by distance, limitations on compute power, transmission mediums, or poorly optimized databases. What can you do about these issues and how does fine-grained control of data locality help?</p> <h2>Getting the resources your applications need</h2> <p>Even though there are limitations on the availability of resources in any shared system, you don’t need resources at the same level at all points in the lifecycle of your data. For instance, with <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Budgeting-time-for-AI-ML-projects/ba-p/7090807#.YTmA_-lKhE4">AI and machine learning projects</a>, data latency and computational requirements change at various stages in the lifetime of models. The learning process, when models are trained, tends to be compute-intensive as compared to requirements when models run in production. Model training also requires high throughput, low-latency data access. It might seem ideal (if there were no limitations on resources) to run your entire machine learning project on high performance computing (HPC) machines with specialized numerical accelerators such as graphical processing units (<a href="https://www.hpe.com/us/en/solutions/hpc-high-performance-computing/nvidia-collaboration.html">GPUs</a>).</p> <p>But while these are ideal for the model training phase, they may be less useful for the high-bulk, low-compute steps of raw data ingestion, data exploration and initial processing for feature extraction. Instead, you may get the best net performance by carrying out these operations with data on traditional spinning media, especially if you live on a fixed budget (like everyone I know).</p> <p>The point is that it’s not just real world limitations on resources that drive the need to place data on different types of storage media. For top performance, you’d want to consider how the application processes the data. For certain, you’d want flexibility in any event.</p> <center><img src="/img/text-block-data-locality.jpg" width="867" height="133"></center> <p>To make this work, the system on which your applications are deployed must allocate resources efficiently. The good news is that with a data infrastructure engineered to support <a href="https://www.hpe.com/us/en/resources/software/ai-and-analytics-systems.html">scale-efficiency</a> through granular data placement, it’s easy to optimize resource use and, in turn, to maximize application performance. Here’s how.</p> <h2>Match storage type to data requirements to maximize performance</h2> <p>The key to optimizing application performance and resource usage is to be able to match data throughput, latency and total size requirements with the appropriate type of storage media. Keep in mind that to get the full benefit of high performance storage, it’s important to <a href="https://www.youtube.com/watch?v=4E2beYyhux8">support GPUs</a> and other accelerators from a data point of view.</p> <p>In large systems, this optimization is accomplished by giving dedicated HPC machines high performance storage, such as solid-state disks (SSDs) or nVME drives and provisioning regular machines with slower, spinning media (HDDs), capable of handling large amounts of data storage at lower cost. This type of large-scale cluster is depicted in Figure 1.</p> <center><img src="/img/datastorage-fig1.png" width="1200" height="459"></center> _Figure 1. Large cluster containing a combination of dedicated, fast-compute/fast storage nodes (orange) and regular nodes/slower storage devices (green)_ <p>In the figure above, the orange squares represent SSDs and orange lines represent machines with computational accelerators (such as GPUs). Green cylinders stand for slower spinning storage media (HDDs) and servers with green lines indicate traditional CPUs. In a typical machine learning/AI scenario, raw data is ingested on the non-HPC machines, where data exploration and feature extraction would take place on very large amounts of raw data. In a scale-efficient system, bulk analytic workloads, such as monthly billing, would also take place on the non-HPC (green) machines.</p> <p>Once feature extraction is complete, training data is written to fast storage machines (orange) with SSDs and GPUs, ready to support the model training process. Other compute-intensive applications, such as simulations, would also run on the fast machines.</p> <p>Smaller systems (clusters with less than 20 machines) often cannot afford dedicated HPC machines with high performance storage. Instead, the need for high performance computing is met by employing some heterogeneous mix -- nodes with fast-compute capabilities but with a mix of different kinds of data storage devices rather than just SSDs. This arrangement is shown in Figure 2.</p> <center><img src="/img/datstorage-fig2.png" width="600" height="224"></center> _Figure 2. Small cluster containing fast-compute nodes (orange) having a mixture of SSDs (orange squares) plus slower HDDs (green cylinders) and regular nodes with HDDs only._ <p>Similar to the earlier example, you need a way to assign what data will be placed on which machines. Fortunately, <a href="https://www.hpe.com/us/en/software/ezmeral-data-fabric.html">HPE Ezmeral Data Fabric</a> lets you use <a href="https://docs.datafabric.hpe.com/62/AdministratorGuide/LBS.html">storage labels</a> to do just that.</p> <h2>Fine-grained data locality with HPE Ezmeral Data Fabric</h2> <p><a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a> is a highly scalable, unifying data infrastructure engineered for data storage, management, and motion. Data fabric is software-defined and hardware agnostic. It lets you <a href="https://docs.datafabric.hpe.com/62/AdministratorGuide/SettingUpTopology-Volume-MCS.html?hl=data%2Cplacement">conveniently position data at the level of different racks, machines,</a> or even <a href="https://docs.datafabric.hpe.com/62/AdministratorGuide/LBS.html">different storage types</a> within machines.</p> <p>Figure 3 below shows how easy it is to create a data fabric volume, assign topology and apply data placement policies via storage labels. (A data fabric volume is a data management unit holding files, directories, NoSQL tables, and event streams all together that act like directories with <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/What-s-your-superpower-for-data-management/ba-p/7100920#.YThGGOlKhE4">superpowers for data management</a>. Many policies, including data placement, are assigned to volumes.)</p> <center><img src="/img/datafabric-volume-fig3.png" width="1382" height="749"></center> _Figure 3. Screenshot of the control plane for HPE Ezmeral Data Fabric._ <p>What happens if you need cross-cutting requirements for data placement? Data fabric lets you define data locality to address multiple goals, such as placement across multiple racks within topologies designated for different failure domains plus additional requirements for particular storage media imposed by assigning storage labels. Locality of the data fabric volume would have to meet both requirements.</p> <p>Figure 4 illustrates an example of fine-grained data placement accomplished using storage labels in a cluster with heterogeneous machines.</p> <center><img src="/img/storagelabels-fig4.jpg" width="515" height="372"></center> _Figure 4. Using the storage labels feature of HPE Ezmeral Data Fabric for differential data placement on particular types of storage devices at the sub-machine level._ <h2>Benefits of high performance metadata with HPE Ezmeral Data Fabric</h2> <p>Performance in distributed systems running many different applications is further enhanced by fine-grained data placement using HPE Ezmeral Data Fabric storage labels. This capability lets you easily assign data locality down to the level of <a href="https://docs.datafabric.hpe.com/62/glossary/gloss_storage_pool.html?hl=storage%2Cpool">storage pools</a>, a unit of storage within a machine made up of multiple disks. To understand how this additional performance boost works, you’ll need a little background information about the data fabric and to understand how metadata is handled.</p> <p>HPE Ezmeral Data Fabric uses a large unit of data storage, known as a <a href="https://docs.datafabric.hpe.com/62/MapR-DB/Architecture-MapRDBandMAPRFS.html?hl=data%2Ccontainer">data fabric container</a> (not to be confused with a Kubernetes container, despite the similarity in the name) as the unit of replication. Data replication is an automatic feature of the data fabric – the basis for data fabric’s self-healing capabilities – with data replicas spread across multiple machines by default. But you can also specify particular data placement policies, and data fabric containers and their replicas will automatically be placed according to the policies you apply.</p> <p>Data fabric also has a special container, known as a name container, which holds metadata for the files, directories, tables, and event streams associated with a data fabric volume. The name container is a strength of the HPE Ezmeral Data Fabric design because it provides a way for metadata to be distributed across a cluster, resulting in extreme reliability and high performance.</p> <p>With the fine granularity for data placement afforded by the storage labels feature, <em>data fabric containers and their replicas can have one placement policy while the name container can have a different policy</em>. As Figure 4 shows, you can apply a label “Warm” to position data for bulk workloads on storage pools with slower devices while maintaining the metadata for that volume on fast solid-state devices by applying the label “Hot” to the name container.</p> <p>This situation can result in significant throughput improvements in processes such as massive disk-based sorts where a very large number of spill files must be created quickly (requiring super-fast meta-data updates on SSDs) and then these spill files must be written very quickly (requiring fast sequential I/O that hordes of hard drives can provide). The combination can work better than either option in isolation by providing the right resources for the right micro-workloads.</p> <h2>Making the most of fine-grained data placement</h2> <p>Turns out you don’t need unlimited resources to get excellent performance for your applications when you take advantage of the fine granularity of data placement afforded by HPE Ezmeral Data Fabric. You can easily assign data topologies when you create a data volume, and you can use convenient storage labels for differential data placement on particular types of storage devices even down to different storage pools within machines. And with the added capability of placing metadata independently of data containers, you can further optimize performance for both bulk applications and in situations using many small files.</p> <p>To find out more about the capabilities provided by HPE Ezmeral Data Fabric visit the <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home/">data fabric platform page</a> in the HPE Developer Community.</p> <p>For a hands-on workshop highlighting data fabric volumes, go to <a href="/hackshack/workshop/26">HPE Ezmeral Data Fabric 101 – Get to know the basics around the data fabric</a>.</p> <p>To learn about data access management using HPE Ezmeral Data Fabric, read the New Stack article <a href="https://thenewstack.io/data-access-management-with-aces-vs-acls-the-power-of-and-and-not/">Data Access Control via ACEs vs ACLs: The power of “AND” and “NOT”</a>.</p>https://developer.hpe.com/java-coding-style-or-bad-practice/https://developer.hpe.com/java-coding-style-or-bad-practice/Fri, 22 Oct 2021 11:32:06 GMT<!--StartFragment--> <p>We all have different styles and preferences in everything in life, including how we write code.</p> <p>Imprinting your personality in the code brings originality and a sense of <a href="https://books.google.co.uk/books/about/Patterns_of_Software.html?id=0-i3QgAACAAJ&#x26;redir_esc=y">ownership and responsibility</a> It’s essential to keep us motivated, and it makes us feel good (at least I do). However, is one's coding style always just a harmless style? Or does it impact readability and hence maintenance?</p> <p>This has been on my mind a lot lately. For instance, during a code review, I often question whether I should bring specific ways of <a href="https://medium.com/javarevisited/5-coding-interview-books-to-prepare-for-programming-job-interviews-d8f63348afaf">coding </a>into the discussion or not. How does it affect the application; Is it readable, is it easy to maintain?</p> <p>Or perhaps I should leave it alone, thinking to myself — <em>Don’t be picky, it’s just their preference, it’s not a matter of right or wrong.</em></p> <h1>Identifying a programmer's fingerprint</h1> <p>We could say a developer has a coding identity or ‘fingerprint’, similar to what happens with regular writing. When writing, there is often a pattern with which someone writes — the terms, vocabulary, structure. A linguistic expert, for instance, can identify the author of some anonymous material simply by analyzing these patterns.</p> <p>Analizing these patterns can even tell things such as the age and place of birth of the author. This technique is called <em>Stylometry.</em> It’s even used in criminal investigations. Machine learning algorithms are used for <a href="http://www.scielo.org.mx/scielo.php?script=sci_arttext&#x26;pid=S1405-55462018000100047">Stylometry </a>as well — as they can process many texts/books and identify patterns.</p> <p>We probably can’t tell who committed a crime based on the coding style (<em>can we?</em>). But, let’s say in a team of ten developers, if there are no strict standards to follow, I believe it’s possible to identify who wrote a code block without looking at the author information.</p> <p>In this post, I’ll list a number of different ways of writing code I’ve encountered throughout my career as a Software Engineer. I’ll focus mostly on Java, but some things are applicable in general.</p> <p>I’ll also offer my perspective on whether it is just a coding preference that we shouldn’t care about, or if perhaps there is a right (and wrong) way of doing it.</p> <h1>Multiple or single “returns”</h1> <p><img src="https://miro.medium.com/max/700/1*NNP98veaLxsykgqKP876Ig.png" alt=""></p> <p>One coding practice that tends to reflect a developer's preference is the use of a single or multiple 'returns'.</p> <p>I used to prefer a single ‘return’ at the end of the method, and I still do this sometimes. But more recently, I find that I tend to return where the condition satisfies — I think it’s easier to maintain (it looks uglier, though). You’re more sure of when the method returns a particular value, and you can be certain that any code after the return won’t be executed.</p> <p>Otherwise, you need to read every if-else or break inside a loop. Often the logic is not as simple as the one presented above.</p> <p>If I see some complex logic with multiple if-else conditions chained together, mixed with ‘break’ inside loops, etc., and one single return at the end, when a particular value could’ve been returned before — I’d explain my perspective and see if the person agrees with doing the change. However, I wouldn’t push it too much and be picky about it. It’s a subtle benefit that may be hard to convey.</p> <h1>To Else or not?</h1> <p><img src="https://miro.medium.com/max/700/1*-_0Gs6GdptRNMA1Efh5KAg.png" alt=""></p> <p>Another variation I tend to see is whether the coder uses the Else statement. Is it really necessary? I commonly do the version on the left — “Default value with no else” when it’s a simple variable assignment case. It just feels cleaner to me.</p> <p>A counter-argument could be that the first example uses fewer resources, because you start with null, and only one value (A or B) is assigned at max. On the other hand, a maximum of two variable assignments could happen (if booleanFlag is true). I’d agree with that, but not for all cases. Setting a default first would be fine. It depends on what is being executed as the ‘default’.</p> <p>This example was one ‘challenge’ that my Bachelor course coordinator threw at us newbies in the first semester during a programming class — “How could you rewrite the first version in fewer lines?!”</p> <p>No one in the class could answer it. Everyone was still coming to terms with the fact that the course wasn’t really about learning Microsoft Office.</p> <p>Although I prefer the second version (for a simple variable assignment), I’d probably not bring it up to discuss or ask to change in a code review.</p> <h1>Curly braces or not</h1> <p><img src="https://miro.medium.com/max/700/1*rHOOuQTYZtI66rephlNgZg.png" alt=""></p> <p>Curly braces are used to delimit the start and end of a block of code. Curly braces become ‘optional’ when only one statement is inside an IF condition, a While or For loop.</p> <p>Both code snippets do the same thing; there is no difference functionality wise. Which one do you prefer?</p> <p>For me, I’m totally in favour of using curly braces, always. It shouldn’t be optional. I think that mainly because, in languages like Java, the indentation doesn’t drive what will be executed as part of the if condition or loop (for Python, it does, for example). Indentation only, without curly braces, cannot be relied on — it may trick you into thinking that something will be executed (or won’t be executed) when it won’t (or when it will).</p> <p>So NOT using curly braces may lead to hidden bugs and bad readability in general. In contrast, using it leaves no room for doubt on which line will run or not. It becomes easier to maintain, in my view.</p> <p>Here are some examples to help illustrate what I mean:</p> <pre><code>if (count > 10) System.out.println(1); System.out.println(2); System.out.println(3); </code></pre> <p>When you read the code above, you think all three lines will be executed if the condition satisfies. But it’s not true. There are no curly braces; hence only one will be printed if, let’s say, the count is equal to eleven. Two and three will be printed in any case, even if, for example, the count is five.</p> <p>Another example:</p> <pre><code>int count=15; if (count > 10) if (count > 20) return 1; else return 2; return 3; </code></pre> <p>The else is aligned with the first if condition, but it’s instead part of the if condition just before. The program returns two as the count is less than twenty.</p> <p>In a code review, I would probably ask to change it (very politely and diplomatically, of course). The other team member may prefer without curly braces and depending on my position, that’s fine — I wouldn’t push it too much.</p> <h1>Checked or unchecked exception</h1> <p>Exceptions are events that happen outside of the normal flow. It allows programmers to separate the code that deals with the success path from those that deal with errors.</p> <p><a href="https://medium.com/javarevisited/10-free-courses-to-learn-java-in-2019-22d1f33a3915?source=collection_home---4------8-----------------------">Java </a>has its Exception classes, or the developer can create its own by extending Exception or RuntimeException.</p> <p>Let’s say there is some particular error validation related to your business. You could create a class, for example, ProductNotFoundException, that extends the Exception class.</p> <p>Another characteristic of how exceptions in Java work is that there are two types of exceptions: Checked and Unchecked.</p> <p><a href="https://javarevisited.blogspot.com/2011/12/checked-vs-unchecked-exception-in-java.html">Checked exceptions</a> are exceptions that extend the Exception class. Their behaviour is: If a code inside method A throws a checked exception, any method that calls method A must handle the checked exception by either catching or throwing (or perhaps both). The code will not compile otherwise. Extending a <a href="https://docs.oracle.com/javase/tutorial/essential/exceptions/definition.html">Checked exception</a> is a way to force programmers to handle a specific error.</p> <p>Unchecked exceptions are used for unrecoverable errors. Such errors are not to be handled. Instead, programmers should tackle the root cause that triggers them. Example: <a href="https://javarevisited.blogspot.com/2012/06/common-cause-of-javalangnullpointerexce.html">NullpointerException</a>.</p> <p>These exceptions extend RuntimeException, and are different from Checked ones. The caller method is not enforced to handle it by catching or throwing it.</p> <p>Despite being used for unrecoverable errors, one could create an Unchecked exception. It’s just a matter of extending RuntimeException.</p> <p>Any method can handle such exceptions, but the compiler doesn’t complain if they don’t. That means that you can have the exception handling code only where it is needed.</p> <p>I learned and used to code by always using <a href="http://www.java67.com/2012/12/difference-between-runtimeexception-and-checked-exception.html">Checked exceptions</a>. You probably learned that way too. If you implement a method that calls another method A that generates a checked error, the compiler will tell you that you need to do something about it. And the intent of who created method A was exactly that, to alert and force others to handle the error.</p> <p>Oracle does recommend always using a Checked exception if you expect to recover from the error. <a href="https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html">https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html</a></p> <blockquote> <p><em>Here’s the bottom line guideline: If a client can reasonably be expected to recover from an exception, make it a checked exception. If a client cannot do anything to recover from the exception, make it an unchecked exception.</em></p> </blockquote> <p>I admit that despite Oracle’s recommendation and being a good practice to use Checked exceptions, I have extended RuntimeException before. I understand that throwing exceptions should be considered just as essential as the method’s parameters and return value; <a href="https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html">it’s part of the method programming interface</a>.</p> <p>But, no one creates a method that receives a parameter and does nothing with it. I find that throwing it, and re-throwing it upstream without doing any meaningful handling (log, return message to the user), creates a bit of clutter. It’s unnecessary.</p> <p>With unchecked exceptions, only the method that generates the error and the one that handles it needs to deal with it. It’s a calculated risk I choose to take sometimes. It’s a calculated risk in the sense that an error that is supposed to be handled may not be — another developer that calls your method won’t be alerted by the compiler to handle the exception that your method raises. It’s a drawback from making it Unchecked, that’s why is considered a bad practice in general.</p> <p>If I see that one of the team members chose to create and use an Unchecked exception, I would probably want to know the thought process and make sure they know the pros and cons.</p> <h1>Using an If then versus an Else exception</h1> <p><img src="https://miro.medium.com/max/700/1*i9wP8lt_G0auF6VFncfKYg.png" alt=""></p> <p>Another thing programmers tend to differ on is the use of an If then exception when an Else exception would also work.</p> <p>I’ve experimented with both ways throughout my career as a developer. Today I prefer the version on the right — “If then exception”. I see it as clearer — easier to read where errors are generated. And I usually have it aside from the main logic in a private ‘validate’ method.</p> <p>I would probably try to change in a code review if one of my peers uses the second version. Unless the code is as simple as the example in this section — then I’d leave it (maybe).</p> <h1>Positioning the curly braces</h1> <p><img src="https://miro.medium.com/max/700/1*w0PHfP8XQZejoWbIJRKRuw.png" alt=""></p> <p>This one is just cosmetics. It’s silly. It’s like preferring a toast cut diagonally versus horizontally or vertically (You’re probably wondering — How can someone** <em>not</em>** choose diagonally?! Anyway…).</p> <p>I find it funny that, even in things like this, people have preferences.</p> <p>I prefer the first one, with curly braces in the same line as if condition or loop. I don’t see any considerable benefit of one or the other. I would not ask another programmer to change it.</p> <h1>Final thoughts</h1> <p>Certain coding style choices are personal, with no benefit or cons over others. It’s like preferring blue to red, orange to apple.</p> <p>However, some other preferences are more arguable — does it make the code less readable or error-prone? Out of the stylistic differences I covered, the no use of curly braces and checked versus unchecked exception examples stand out. These are the ones with the most impact.</p> <p>Even if there are coding standards set in place, it's probably best if they aren't too rigid. One still needs to allow the developer a certain amount of leeway to make their own personal mark. If it were up to me, I would set a rule to always use curly braces and, possibly, to use checked exceptions (because it tends to be safer), but that's about it. In the end, it should be discussed and agreed upon as a team.</p> <!--EndFragment-->https://developer.hpe.com/determined-ai-is-joining-hewlett-packard-enterprise/https://developer.hpe.com/determined-ai-is-joining-hewlett-packard-enterprise/Tue, 12 Oct 2021 17:24:39 GMT<p><strong>Editor’s Note:</strong> Determined AI products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2021 may have older product names that differ from current solutions.</p> <p>Today is a big day for Determined AI, our customers, and our open source community. We are thrilled to announce that <a href="https://www.hpe.com/us/en/newsroom/press-release/2021/06/hewlett-packard-enterprise-acquires-determined-ai-to-accelerate-artificial-intelligence-innovation-with-fast-and-simple-machine-learning-modeling.html">Determined AI is joining one of Silicon Valley’s most iconic companies: Hewlett Packard Enterprise (HPE)</a>, working under the umbrella of the High Performance Computing (HPC) and Mission Critical Solutions (MCS) business unit.</p> <center><img src="/img/hpe-logo.png" width="500" height="333"></center> <p>This is a <em><strong>massive</strong></em> accelerant for our mission to empower users to efficiently build cutting-edge AI applications. Over the last several years, building AI applications has become <em><strong>extremely</strong></em> compute, data, and communication intensive. Ten years ago you could do cutting-edge computer vision research on a laptop. Today, you need a massive farm of GPUs or other specialized chips to remain competitive. These problems aren’t unique to vision or academic research – they’re affecting organizations large and small, as we are hearing from our thriving open source community. To put it a different way: AI is rapidly becoming a High Performance Computing problem. HPE is already a global leader in designing and delivering High Performance Computing systems and, via their acquisitions of Cray and SGI, they have decades of experience in the space working with some of the most sophisticated users on the planet. We are thrilled about the opportunity to partner with HPE to deliver co-designed software and hardware and tackle some of society’s most pressing challenges.</p> <p>HPE shares our vision that driving an open standard for AI software infrastructure is the fastest way for the industry to realize the potential of AI. Consequently, <strong>HPE is committed to investing in and rapidly growing the Determined Training Platform as an open source project.</strong> Our customers and open source community members will continue to receive the same high level of service and support that they always have, from a team of experts who are intimately familiar with the challenges they’re facing. Moving forward, we’ll be in a position to serve a much more global network of users, both large and small – we’re excited to continue building new features to both our customers and our open-source community.</p> <p>Relatedly, <a href="https://www.determined.ai/contact">we are actively hiring!</a> We are truly humbled by the amazing team we have had the good fortune to work with over the past 4 years. Building infrastructure for AI is fundamentally an interdisciplinary endeavor, and from the day we started the company, we have been committed to fostering an environment that encourages diversity of thought. At HPE we’ll be rapidly expanding the team’s headcount and continuing to build on our culture of collaboration.</p> <p>Today is a big day for Determined AI, and we’re immensely grateful to the customers and community members we have had the privilege of working with for the past 4 years. The next phase in our journey promises to be even more exciting, and we’re thrilled to continue to work with the ML community to build the future of AI infrastructure.</p> <p>For more information on using Determined AI to build and train deep learning (DL) models at scale, please check out the <a href="https://developer.hpe.com/platform/determined-ai/home/">Determined AI platform page</a> on HPE DEV.</p>https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-3/https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-3/Mon, 11 Oct 2021 21:12:59 GMT<pre><code class="language-bash">"authorDisplayName": "Carol McDonald", "tags": "use-cases", "publish": "2018-02-08T12:00:00.000Z" </code></pre> <p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <p><strong>Editor's Note:</strong> This is a 3-part Series, see the previously published posts below:</p> <ul> <li><a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/">Part 1 - Spark Machine Learning</a></li> <li><a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-2/">Part 2 - Kafka and Spark Streaming</a></li> </ul> <p>According to Bob Renner, previous CEO of Liaison Technologies, the possibility to blend machine learning with real-time transactional data flowing through a single platform is opening a world of new possibilities, such as enabling organizations to take advantage of opportunities as they arise. According to <a href="https://www.gartner.com/newsroom/id/3812063">Gartner</a> over the next few years, virtually every app, application and service will incorporate some level of machine learning. Leveraging these opportunities requires fast and scalable data processing pipelines.</p> <center><img alt="MapR Platform" src="/img/mapr-platform.png" width="700"></center> <p>This is the third in a series of blog posts, that discuss the architecture of a data pipeline that combines streaming data with machine learning and fast storage. <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/">The first post</a> discussed creating a machine learning model to predict flight delays. The <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-2/">second post</a> discussed using the saved model with streaming data to do a real-time analysis of flight delays. This third post will discuss fast storage and analysis with MapR Database, Apache Spark, Apache Drill and OJAI.</p> <p><strong>Machine Learning Logistics and Data Pipelines</strong></p> <p>Machine Learning usually refers to the model training piece of an ML workflow. But, as data fabric expert Ted Dunning says, 90% of the effort around Machine Learning is data logistics which includes all of the aspects that occur before and after this training. When you combine event streams with microservices, you can greatly enhance the agility with which you build, deploy, and maintain complex data pipelines. Pipelines are constructed by chaining together microservices, each of which listens for the arrival of some data, performs its designated task, and optionally publishes its own messages to another topic. Combining event-driven data pipelines with machine learning can handle the logistics of machine learning in a flexible way by:</p> <ul> <li>Making input and output data available to independent consumers</li> <li>Managing and evaluating multiple models and easily deploying new models</li> <li>Monitoring and analyzing models, with historical and real-time data.</li> </ul> <center><img alt="MapR Architectures" src="/img/mapr-architectures.png" width="700"></center> <p>Architectures for these types of applications are discussed in more detail in the eBooks: <a href="https://www.oreilly.com/library/view/machine-learning-logistics/9781491997628/">Machine Learning Logistics</a>, <a href="https://www.oreilly.com/library/view/streaming-architecture/9781491953914/">Streaming Architecture</a>, and <a href="https://www.academia.edu/41522528/A_Practical_Guide_to_Microservices_and_Containers_Mastering_the_Cloud_Data_and_Digital_Transformation">Microservices and Containers</a>.</p> <p>The following figure depicts the (simplified) data pipeline for this tutorial:</p> <ul> <li>Flight trip data is published to a MapR Event Streams (ES) topic using the Kafka API. (Note this data contains the actual delay label. In the real-world architecture, the actual delay label would come later in a different topic, but to keep the tutorial code simple it is combined with the input data).</li> <li>A Spark Streaming application subscribed to the first topic enriches the event with the flight predictions and publishes the results in JSON format to another topic. ( In the real-world architecture, there would be multiple consumers publishing model predictions, but to keep the tutorial code simple there is only one here).</li> <li>A Spark Streaming application subscribed to the second topic stores the flight trip data and predictions in MapR Database using the Spark MapR Database Connector.</li> <li>Apache Spark SQL, Apache Drill SQL, and Open JSON applications query MapR Database to analyze flight data and prediction performance.</li> </ul> <center><img alt="MapR Database" src="/img/mapr-db.png" width="700"></center> <p><strong>How to Store the Data</strong></p> <p>One of the challenges, when you are processing lots of streaming data, is where do you want to store it? With a relational database and a normalized schema, related data is stored in different tables. Queries joining this data together can cause bottlenecks with lots of data. For this application, <a href="https://docs.datafabric.hpe.com/62/MapR-DB/developing_client_applications_for_mapr_db.html">MapR Database JSON</a>, a high-performance NoSQL database, was chosen for its scalability and flexible ease of use with JSON. MapR Database and a denormalized schema scale, because data that is read together is stored together.</p> <center><img alt="Storage Model" src="/img/storage-model.png" width="700"></center> <p>With MapR Database (HBase API or JSON API), a table is automatically partitioned into tablets across a cluster by key range, providing for scalable and fast reads and writes by row key.</p> <center><img alt="MapR Database Connector" src="/img/mapr-db-connector.png" width="700"></center> <p>The Spark MapR Database Connector leverages the Spark <a href="https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html">DataSource API</a>. The connector architecture has a connection object in every Spark Executor, allowing for distributed parallel writes, reads, or scans with MapR Database tablets.</p> <center><img alt="Spark Executor" src="/img/spark-executor.png" width="700"></center> <p><strong>JSON Schema Flexibility</strong></p> <p>MapR Database supports JSON documents as a native data store. MapR Database makes it easy to store, query and build applications with JSON documents. The Spark connector makes it easy to build real-time or batch pipelines between your JSON data and MapR Database and leverage Spark within the pipeline.</p> <center><img alt="JSON Schema Flexibility" src="/img/json-schema-flexibility.png" width="700"></center> <p>JSON facilitates the natural evolution of your data schema during the life of your application. For example, suppose at first we have the following schema, where each JSON message has the predicted flight delay using a decision tree:</p> <pre><code class="language-json">{ "_id": "UA_2017-03-28_DEN_SFO_721", "dofW": 2, "carrier": "UA", "origin": "DEN", "dest": "SFO", "crsdephour": 11, "crsdeptime": 1120.0, "crsarrtime": 1308.0, "crselapsedtime": 168.0, "pred_dtree": 1.0 } </code></pre> <p>Later, you can easily capture more prediction data values quickly without changing the architecture of your application and without updating a database schema, by adding attributes. In the example below, we have added predictions for other machine learning models. These can be added dynamically to the same document instance in MapR Database without any database schema changes:</p> <pre><code class="language-json">{ "_id": "UA_2017-03-28_DEN_SFO_721", "dofW": 2, "carrier": "UA", "origin": "DEN", "dest": "SFO", "crsdephour": 11, "crsdeptime": 1120.0, "crsarrtime": 1308.0, "crselapsedtime": 168.0, "pred_dtree": 1.0, "pred_randforest": 1.0, "pred_svm": 1.0, "actual_delay": 1.0 } </code></pre> <p>MapR Event Store allows processing of the same messages by different consumers. This makes it easy to add different consumers for the same message. With this type of architecture and flexible schema, you can easily add and deploy new microservices with new machine learning models.</p> <p><strong>Spark Streaming writing to MapR Database</strong></p> <p>The MapR Database OJAI Connector for Apache Spark enables you to use <a href="https://docs.datafabric.hpe.com/62/Spark/SavingDStreamMapRDB.html">MapR Database as a sink for Apache Spark Data Streams</a>.</p> <center><img alt="Spark Streaming writing to MapR Database" src="/img/spark-streaming-writing-to-mapr-db.png" width="700"></center> <p>You can read about the MapR Event Streams Spark Streaming code in <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-2/">part 2 of this series</a>; here, we will focus on Spark streaming writing to MapR Database. The messages from the MapR Database topic are in JSON format and contain the following for each flight: the flight id, day of the week, carrier, origin, destination, scheduled departure hour, scheduled departure time, scheduled arrival time, scheduled travel time, delay prediction, and actual delay label (Note in the real-world architecture, the actual delay label would come later in a different topic, but to keep the tutorial code simple, it is combined here). An example is shown below:</p> <pre><code class="language-json">{ "_id": "UA_2017-03-28_DEN_SFO_721", "dofW": 2, "carrier": "UA", "origin": "DEN", "dest": "SFO", "crsdephour": 11, "crsdeptime": 1120.0, "crsarrtime": 1308.0, "crselapsedtime": 168.0, "label": 0.0, "pred_dtree": 1.0 } </code></pre> <p>Below, we use a Scala case class and <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema">Structype</a> to define the schema, corresponding to the input data.</p> <center><img alt="Scala case class 1" src="/img/scala-case-class.png" width="500"></center> <center><img alt="Scala case class 2" src="/img/scala-case-class-2.png" width="500"></center> <p>We use the KafkaUtils createDirectStream method with Kafka configuration parameters to create an input stream from a MapR Event Store topic. This creates a DStream that represents the stream of incoming data, where each message is a key value pair. We use the DStream map transformation to create a DStream with the message values.</p> <center><img alt="KafkaUtils createDirectStream method" src="/img/kafkautils-createdirectstream-method.png" width="700"></center> <center><img alt="DStream" src="/img/dstream.png" width="700"></center> <p>In the code below, each RDD in the valuesDStream is transformed into a Spark Dataset. Then, the MapR Database Spark Connector DStream saveToMapRDB method performs a parallel partitioned bulk insert of JSON FlightwPred objects into MapR Database.</p> <center><img alt="RDD in the valuesDStream" src="/img/rdd-in-valuesdstream.png" width="700"></center> <center><img alt="Save DStream to MapR Database JSON" src="/img/save-dsstream-mapr-db-json.png" width="700"></center> <p><strong>Querying MapR Database JSON with Spark SQL</strong></p> <p>The Spark MapR Database Connector enables users to perform complex SQL queries and updates on top of MapR Database using a Spark Dataset, while applying critical techniques such as Projection and filter pushdown, custom partitioning, and data locality.</p> <center><img alt="Application" src="/img/application.png" width="700"></center> <p>A Spark Dataset is a distributed collection of data. Dataset is a newer interface, which provides the benefits of strong typing, the ability to use powerful lambda functions, efficient object serialization/deserialization, combined with the benefits of Spark SQL's optimized execution engine.</p> <center><img alt="Spark Dataset" src="/img/spark-dataset.png" width="700"></center> <p>A DataFrame is a Dataset organized into named columns Dataset[Row]. (In Spark 2.0, the DataFrame APIs merged with Datasets APIs.)</p> <center><img alt="Unified Apache Spark 2.0 API" src="/img/unified-apache-spark.png" width="500"></center> <p><strong>Loading data from MapR Database into a Spark Dataset</strong></p> <p>To <a href="https://docs.datafabric.hpe.com/62/Spark/LoadDataFromMapRDBasDataset.html">load data from a MapR Database JSON</a> table into an Apache Spark Dataset, we invoke the loadFromMapRDB method on a SparkSession object, providing the tableName, schema and case class. This will return a Dataset of FlightwPred objects:</p> <center><img alt="load data from a MapR Database JSON" src="/img/load-data-from-mapr-db-json.png" width="700"></center> <p><strong>Explore and query the Flight data with Spark SQL</strong></p> <p>Datasets provide a domain-specific language for structured data manipulation in Scala, Java, and Python. Below are some examples in scala. The Dataset show() action displays the top 20 rows in a tabular form.</p> <center><img alt="top 20 rows in a tabular form" src="/img/top-20-rows-tabular-form.png" width="700"></center> <p>In the code below, a filter is used to count the predicted delays, actual delays and total. This is then used to calculate the ratio wrong, correct, false positive. These type of calculations would be useful for continued analysis of models in production.</p> <center><img alt="delays, actual delays and total" src="/img/filter-count-predicted-delays.png" width="700"></center> <p>The output is shown below.</p> <center><img alt="Output of predicted delays, actual delays and total" src="/img/delays-actual-delays-total-output.png" width="500"></center> <p><strong>What is the count of predicted delay/notdelay for this dstream dataset?</strong></p> <center><img alt="What is the count of predicted delay/notdelay for this dstream dataset?" src="/img/count-predicted-dstream-dataset.png" width="700"></center> <p>You can register a Dataset as a temporary table using a given name, and then run Spark SQL. Here are some example Spark SQL queries on the Dataset of FlightwPred objects:</p> <p><strong>What is the count of predicted delay/notdelay by day of the week?</strong></p> <pre><code class="language-scala">scala> spark.sql("select dofW, pred_dtree, count(pred_dtree) from flight group by dofW, pred_dtree order by dofW").show </code></pre> <center><img alt="Spark SQL queries on the Dataset of FlightwPred objects" src="/img/queries-dataset-flightwpred-objects.png" width="500"></center> <p><strong>What is the count of predicted delay/notdelay by destination?</strong></p> <pre><code class="language-scala">scala> spark.sql("select dest, pred_dtree, count(pred_dtree) from flight group by dest, pred_dtree order by dest").show </code></pre> <center><img alt="Delay/NotDelay destination" src="/img/delay-notdelay-destination.png" width="500"></center> <p>(The complete code, instructions and more example queries are in the github code link at the end.)</p> <p><strong>Querying the Data with Apache Drill</strong></p> <p>Apache Drill is an open source, low-latency query engine for big data that delivers interactive SQL analytics at petabyte scale. Drill provides a massively parallel processing execution engine, built to perform distributed query processing across the various nodes in a cluster.</p> <center><img alt="Querying the Data with Apache Drill" src="/img/querying-data-with-apache-drill.png" width="700"></center> <p>With Drill, you can use SQL to interactively query and join data from files in JSON, Parquet, or CSV format, Hive, and NoSQL stores, including HBase, MapR Database, and Mongo, without defining schemas. MapR provides a <a href="https://package.mapr.com/tools/MapR-JDBC/MapR_Drill/">Drill JDBC</a> driver that you can use to connect Java applications, BI tools, such as SquirreL and Spotfire, to Drill. Below is a snippit of Java code for querying MapR Database using Drill and JDBC:</p> <center><img alt="querying MapR Database using Drill and JDBC" src="/img/querying-mapr-db-using-drill-jdbc.png" width="700"></center> <p>The output for this query <strong>"What is the count of predicted delay/notdelay by origin?"</strong> is shown below:</p> <center><img alt="output of predicted delay/notdelay by origin" src="/img/output-delay-notdelay-origin.png" width="500"></center> <p>Below are some example SQL queries using the Drill shell.</p> <p><strong>What is the count of predicted delay/notdelay by origin</strong></p> <center><img alt="count of predicted delay/notdelay by origin" src="/img/count-predicted-delay-notdelay-origin.png" width="700"></center> <p><strong>What is the count of predicted delay/notdelay by origin and dest?</strong></p> <center><img alt="What is the count of predicted delay/notdelay by origin and dest?" src="/img/count-predicted-delay-notdelay-origin-dest.png" width="700"></center> <p>Follow the instructions in the github code README to <a href="https://docs.datafabric.hpe.com/62/Drill/optimizing_queries_with_indexes.html">add a secondary index to MapR Database</a> and try more queries using the index.</p> <p><strong>Querying with the Open JSON API (OJAI)</strong></p> <p>Below is a Java example of using the OJAI Query interface to query documents in a MapR Database JSON table:</p> <center><img alt="OJAI Query interface to query documents in a MapR Database JSON" src="/img/ojai-query-interface-mapr-db.png" width="700"></center> <p>Partial output for this query to "find predicted late flights for AA" is shown below:</p> <center><img alt="find predicted late flights for AA" src="/img/find-predicted-late-flights-aa.png" width="700"></center> <p>Below are some example OJAI queries using the MapR Database shell.</p> <p><strong>What are the SFO to DEN flights that were predicted late ?</strong></p> <pre><code class="language-json">maprdb> find /apps/flights -where '{"$and":[{"$eq":{"pred_dtree":1.0}},{ "$like" : {"_id":"%SFO_DEN%"} }]}' --f _id,pred_dtree </code></pre> <center><img alt="MapR Database find /apps/flights" src="/img/mapr-db-find.png" width="500"></center> <p><strong>Summary</strong></p> <p>In this blog post, you've learned how to consume streaming JSON events, store in a document database, and explore with SQL using Apache Spark, Apache Kafka API, Apache Drill, MapR Event Store, MapR Database, and OJAI.</p> <p><strong>Code</strong></p> <p>You can download the code and data to run these examples from here (refer to the README for complete instructions to run): <a href="https://github.com/mapr-demos/mapr-es-db-60-spark-flight">https://github.com/mapr-demos/mapr-es-db-60-spark-flight</a></p> <p><strong>Running the Code</strong></p> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform. The MapR Data Platform integrates global event streaming, real-time database capabilities, and scalable enterprise storage with a collection of data processing and analytical engines to power data processing pipelines and intelligent applications.</p> <center><img alt="MapR Data Platformm" src="/img/mapr-cdp.png" width="700"></center> <p>This example was developed using the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">MapR 6.0 container for developers</a>, a docker container that enables you to create a single node MapR cluster. The container is lightweight and designed to run on your laptop. (refer to the code README for instructions on running the code).</p> <p>You can also look at the following examples:</p> <ul> <li><a href="https://github.com/mapr-demos/mapr-db-60-getting-started">MapR Database 60-getting-started</a> to discover how to use DB Shell, Drill and OJAI to query and update documents, but also how to use indexes.</li> <li><a href="https://github.com/mapr-demos/mapr-streams-60-getting-started">MapR Event Store getting started on MapR 6.0 developer container</a></li> <li><a href="https://github.com/mapr-demos/ojai-2-examples">Ojai 2.0 Examples</a> to learn more about OJAI 2.0 features</li> <li><a href="https://github.com/mapr-demos/mapr-db-cdc-sample">MapR Database Change Data Capture</a> to capture database events such as insert, update, delete and react to these events.</li> </ul> <p><strong>WANT TO LEARN MORE?</strong></p> <ul> <li><a href="https://learn.ezmeral.software.hpe.com/">Free On-Demand Training</a></li> <li><a href="https://docs.datafabric.hpe.com/62/MapR-DB/designing_row_keys.html?hl=designing%2Crow%2Ckeys%2Cmapr-db">MapR Database Rowkey Design</a></li> <li><a href="https://docs.datafabric.hpe.com/62/Spark/NativeSparkConnectorJSON.html">MapR Database OJAI Connector for Apache Spark</a></li> <li><a href="https://docs.datafabric.hpe.com/62/ReferenceGuide/mapr_dbshell.html">MapR Database shell</a></li> <li><a href="https://docs.datafabric.hpe.com/62/Spark/Spark_IntegrateMapRStreams.html">Integrate Spark with MapR Event Store Documentation</a></li> <li><a href="https://docs.datafabric.hpe.com/62/home/r_home_intro.html?origin=/display/MapR/Drill+Tutorial">Apache Drill Tutorial</a></li> <li><a href="https://docs.datafabric.hpe.com/62/MapR-DB/Indexes/Indexes.html">MapR Database secondary indexes</a></li> <li><a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">MapR Container for Developers</a></li> <li><a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html">Apache Spark Streaming programming guide</a></li> <li><a href="http://spark.apache.org/docs/latest/sql-programming-guide.html">Spark SQL programming Guide</a></li> </ul>https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-2/https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-apis-kafka-spark-streaming-and-machine-learning-part-2/Mon, 11 Oct 2021 19:54:43 GMT<pre><code class="language-json">"authorDisplayName": "Carol McDonald", "tags": "use-cases", "publish": "2018-01-10T10:00:00.000Z" </code></pre> <p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <p><em><strong>Editor's Note: You can find Part 1 of this series <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/">here</a></strong></em></p> <p>According to Bob Renner, former CEO of Liaison Technologies, the possibility to blend machine learning with real-time transactional data flowing through a single platform is opening a world of new possibilities, such as enabling organizations to take advantage of opportunities as they arise. Leveraging these opportunities requires fast, scalable data processing pipelines that process, analyze, and store events as they arrive.</p> <p>This is the second in a series of blog posts that discusses the architecture of a data pipeline that combines streaming data with machine learning and fast storage. <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/">The first post</a> discussed creating a machine learning model to predict flight delays. This second post will discuss using the saved model with streaming data to do a real-time analysis of flight delays. The third post will discuss fast storage with MapR Database.</p> <center><img alt="Saved model with streaming data" src="/img/saved-model-with-streaming-data.png" width="700"></center> <h2>Microservices, Data Pipelines, and Machine Learning Logistics</h2> <p>The <a href="https://martinfowler.com/articles/microservices.html">microservice architectural style</a> is an approach to developing an application as a suite of small independently deployable services. A common architecture pattern combined with microservices is event sourcing using an append-only publish-subscribe event stream such as MapR Event Streams (which provides a Kafka API).</p> <center><img alt="Immediate access to operational and analytical data in MapR" src="/img/immediate-access-to-operational-analytical-data.png" width="700"></center> <h2>Publish Subscribe Event Streams with MapR Event Store</h2> <p>A central principle of publish/subscribe systems is decoupled communications, wherein producers don’t know who subscribes, and consumers don’t know who publishes; this system makes it easy to add new listeners or new publishers without disrupting existing processes. MapR Event Store allows any number of information producers (potentially millions of them) to publish information to a specified topic. MapR Event Store will reliably persist those messages and make them accessible to any number of subscribers (again, potentially millions). Topics are partitioned for throughput and scalability, producers are load balanced, and consumers can be grouped to read in parallel. MapR Event Store can scale to very high throughput levels, easily delivering millions of messages per second using very modest hardware.</p> <center><img alt="Kafka API" src="/img/kafka-api.png" width="700"></center> <p>You can think of a partition like a queue; new messages are appended to the end, and messages are delivered in the order they are received.</p> <center><img src="/img/mapr-cluster.png" width="700"></center> <center><img alt="Messages" src="/img/messages.png" width="700"></center> <p>Unlike a queue, messages are not deleted when read; they remain on the partition, available to other consumers. Messages, once published, are immutable, and can be retained forever.</p> <center><img alt="Not deleting messages" src="/img/not-deleting-messages.png" width="700"></center> <p>Not deleting messages when they are read allows for high performance at scale and also for processing of the same messages by different consumers for different purposes such as multiple views with polyglot persistence.</p> <center><img alt="New subscribers of information" src="/img/new-subscribers-information.png" width="700"></center> <p>New subscribers of information can replay the data stream, specifying a starting point as far back as the data retention policy enables.</p> <center><img alt="Create new view, index, cache" src="/img/create-new.png" width="700"></center> <center><img alt="Read From new View" src="/img/read-from-new-view.png" width="700"></center> <h2>Data Pipelines</h2> <p>When you combine these messaging capabilities with the simple concept of microservices, you can greatly enhance the agility with which you build, deploy, and maintain complex data pipelines. Pipelines are constructed by simply chaining together multiple microservices, each of which listens for the arrival of some data, performs its designated task, and optionally publishes its own messages to a topic. Development teams can deploy new services or service upgrades more frequently and with less risk, because the production version does not need to be taken offline. Both versions of the service simply run in parallel, consuming new data as it arrives and producing multiple versions of output. Both output streams can be monitored over time; the older version can be decommissioned when it ceases to be useful.</p> <h2>Machine Learning Logistics and Data Pipelines</h2> <p>Combining data pipelines with machine learning can handle the logistics of machine learning in a flexible way by:</p> <ul> <li>Making input and output data available to independent consumers</li> <li>Managing and evaluating multiple models and easily deploying new models</li> </ul> <center><img alt="Machine Learning Logistics and Data Pipelines" src="/img/machine-learning-logistics.png" width="700"></center> <p>Architectures for these types of applications are discussed in more detail in the ebooks <a href="https://www.oreilly.com/library/view/machine-learning-logistics/9781491997628/">Machine Learning logistics</a>, <a href="https://www.oreilly.com/library/view/streaming-architecture/9781491953914/">Streaming Architecture</a>, and <a href="https://www.academia.edu/41522528/A_Practical_Guide_to_Microservices_and_Containers_Mastering_the_Cloud_Data_and_Digital_Transformation">Microservices and Containers</a>.</p> <p>Below is the data processing pipeline for this use case of predicting flight delays. This pipeline could be augmented to be part of the rendezvous architecture discussed in the Oreilly <a href="https://www.oreilly.com/library/view/machine-learning-logistics/9781491997628/">Machine Learning Logistics ebook</a>.</p> <center><img alt="Data Processing Pipeline" src="/img/data-processing-pipeline.png" width="700"></center> <h2>Spark Streaming Use Case Example Code</h2> <p>The following figure depicts the architecture for the part of the use case data pipeline discussed in this post:</p> <center><img alt="architecture data pipeline" src="/img/architechture-data-pipeline.png" width="700"></center> <ol> <li>Flight data is published to a MapR Event Store topic using the Kafka API.</li> <li>A Spark streaming application, subscribed to the first topic: Ingests a stream of flight data</li> <li>Uses a deployed machine learning model to enrich the flight data with a delayed/not delayed prediction</li> <li>publishes the results in JSON format to another topic.</li> <li>(In the 3rd blog) A Spark streaming application subscribed to the second topic: Stores the input data and predictions in MapR Database</li> </ol> <h2>Example Use Case Data</h2> <p>You can read more about the data set <a href="https://developer.hpe.com/blog/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/">in part 1 of this series</a>. The incoming and outgoing data is in JSON format, an example is shown below:</p> <pre><code class="language-json">{"_id":"AA_2017-02-16_EWR_ORD_1124”, "dofW”:4, "carrier":"AA", "origin":"EWR", "dest":"ORD”, "crsdeptime":705, "crsarrtime”:851, "crselapsedtime":166.0,"dist":719.0} </code></pre> <center><img alt="Example Use Case Data" src="/img/example-use-case-data.png" width="700"></center> <h2>Spark Kafka Consumer Producer Code</h2> <p><strong>Parsing the Data Set Records</strong></p> <p>We use a Scala case class and <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema">Structype</a> to define the schema, corresponding to the input data.</p> <center><img alt="Structype" src="/img/structype.png" width="700"></center> <p><strong>Loading the Model</strong></p> <p>The Spark <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.tuning.CrossValidatorModel">CrossValidatorModel</a> class is used to load the saved model fitted on the historical flight data.</p> <center><img alt="Loading the Model" src="/img/loading-model.png" width="700"></center> <p><strong>Spark Streaming Code</strong></p> <p>These are the basic steps for the Spark Streaming Consumer Producer code:</p> <ol> <li>Configure Kafka Consumer and Producer properties.</li> <li>Initialize a Spark StreamingContext object. Using this context, create a DStream that reads a message from a Topic.</li> <li>Apply transformations (which create new DStreams).</li> <li>Write messages from the transformed DStream to a Topic.</li> <li>Start receiving data and processing. Wait for the processing to be stopped.</li> </ol> <p><strong>We will go through each of these steps with the example application code.</strong></p> <h2><strong>1) Configure Kafka Consumer Producer properties</strong></h2> <p>The first step is to set the KafkaConsumer and KafkaProducer configuration properties, which will be used later to create a DStream for receiving/sending messages to topics. You need to set the following parameters:</p> <ul> <li>Key and value deserializers: for deserializing the message.</li> <li>Auto offset reset: to start reading from the earliest or latest message.</li> <li>Bootstrap servers: this can be set to a dummy host:port since the broker address is not actually used by MapR Event Store.</li> </ul> <p>For more information on the configuration parameters, <a href="https://docs.datafabric.hpe.com/62/MapR_Streams/differences_in_configuration_parameters_for_producers_and_consumers.html">see the MapR Event Store documentation.</a></p> <center><img alt="Configure Kafka" src="/img/configure-kafka-cosumer.png" width="700"></center> <h2><strong>2) Initialize a Spark StreamingContext object.</strong></h2> <p>ConsumerStrategies.Subscribe, as shown below, is used to set the topics and Kafka configuration parameters. We use the <a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">KafkaUtils createDirectStream</a> method with a StreamingContext, the consumer and location strategies, to create an input stream from a MapR Event Store topic. This creates a DStream that represents the stream of incoming data, where each message is a key value pair. We use the DStream map transformation to create a DStream with the message values.</p> <center><img alt="Initialize a Spark StreamingContext Object 1" src="/img/initialize-spark-streamingcontext.png" width="700"></center> <center><img alt="Initialize a Spark StreamingContext Object 2" src="/img/initialize-spark-streamingcontext-2.png" width="700"></center> <h2><strong>3) Apply transformations (which create new DStreams)</strong></h2> <p>We use the DStream foreachRDD method to apply processing to each RDD in this DStream. We read the RDD of JSON strings into a Flight Dataset. Then we display 20 rows with the Dataset show method. We also create a temporary view of the Dataset in order to execute SQL queries.</p> <center><img alt="Apply transformations" src="/img/apply-transformation.png" width="700"></center> <p><strong>Here is example output from the df.show :</strong></p> <center><img alt="Example output from the df.show" src="/img/example-df-show.png" width="700"></center> <p>We transform the Dataset with the model pipeline, which will transform the features according to the pipeline, estimate and then return the predictions in a column of a new Dateset. We also create a temporary view of the new Dataset in order to execute SQL queries.</p> <center><img alt="Temporary view" src="/img/temporary-view.png" width="700"></center> <h2><strong>4) Write messages from the transformed DStream to a Topic</strong></h2> <p>The Dataset result of the query is converted to JSON RDD Strings, then the RDD sendToKafka method is used to send the JSON key-value messages to a topic (the key is null in this case).</p> <center><img alt="Write messages from the transformed DStream" src="/img/write-messages-from-dstream.png" width="700"></center> <center><img alt="Write messages from the transformed DStream 2" src="/img/write-messages-from-dstream-2.png" width="700"></center> <p>Example message values (the output for <code>temp.take(2)</code> ) are shown below:</p> <pre><code class="language-json">{"_id":"DL_2017-01-01_MIA_LGA_1489","dofW":7,"carrier":"DL","origin":"MIA","dest":"LGA","crsdephour":13,"crsdeptime":1315.0,"crsarrtime":1618.0,"crselapsedtime":183.0,"label":0.0,"pred_dtree":0.0} {"_id":"DL_2017-01-01_LGA_MIA_1622","dofW":7,"carrier":"DL","origin":"LGA","dest":"MIA","crsdephour":8,"crsdeptime":800.0,"crsarrtime":1115.0,"crselapsedtime":195.0,"label":0.0,"pred_dtree":0.0} </code></pre> <h2><strong>5) Start receiving data and processing it. Wait for the processing to be stopped.</strong></h2> <p>To start receiving data, we must explicitly call start() on the StreamingContext, then call <code>awaitTermination</code> to wait for the streaming computation to finish. We use <code>ssc.remember</code> to cache data for queries.</p> <center><img alt="Receive and process data" src="/img/receive-process-data.png" width="700"></center> <h2>Streaming Data Exploration</h2> <p>Now we can query the cached streaming data in the input temporary view flights, and the predictions temporary view <code>flightsp</code>. Below we display a few rows from the flights view:</p> <center><img alt="Flights view" src="/img/flights-view.png" width="700"></center> <p>Below we display the count of predicted delayed/not delayed departures by Origin:</p> <center><img alt="Predicted Departures" src="/img/predicted-departures.png" width="700"></center> <p>Below we display the count of predicted delayed/not delayed departures by Destination:</p> <center><img alt="Predicted Delayed" src="/img/predicted-delayed.png" width="700"></center> <p>Below we display the count of predicted delayed/not delayed departures by Origin,Destination:</p> <center><img alt="Count predicted delayed" src="/img/count-predicted-delayed.png" width="700"></center> <h2>Summary</h2> <p>In this blog post, you learned how to use a Spark machine learning model in a Spark Streaming application, and how to integrate Spark Streaming with MapR Event Streams to consume and produce messages using the Kafka API.</p> <h2>Code</h2> <ul> <li>You can download the code and data to run these examples from <a href="https://github.com/caroljmcdonald/spark-ml-flightdelay">here</a></li> <li><a href="https://github.com/caroljmcdonald/spark-ml-flightdelay/blob/master/notebooks/sparkmlpipelineflightdelays.json">Zeppelin Notebook for the code</a></li> </ul> <h2>Running the Code</h2> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform.</p> <center><img alt="MapR Data Platform" src="/img/mapr-cdp.png" width="700"></center>https://developer.hpe.com/celebrate-hacktoberfest-2021-with-grommet/https://developer.hpe.com/celebrate-hacktoberfest-2021-with-grommet/Thu, 07 Oct 2021 18:02:43 GMT<p>It’s that time of the year again… the leaves are falling, the pumpkins are ripe, and developers once again get to participate in Hacktoberfest – a month-long celebration run by Digital Ocean that encourages contributions to open source projects. Coders are rewarded for their participation. This year, if you make 4 contributions during the month of October to open source projects you can earn a limited edition T-shirt or opt to help save the planet by having a tree planted in your name.</p> <p>Grommet contributors have traditionally participated heavily in this event and it looks like it’s going to be another great event. In just the first week of Hacktoberfest, the Grommet team saw 22 Pull Requests come in, including a number of pull requests coming from first time contributors. Congratulations, and welcome to the Grommet community!</p> <p><strong>What are Grommet contributors focusing on this year?</strong></p> <p>This year, coders have been heavily focused on Typescript, working on an issue that has recently gotten a lot of attention. A lot of progress has already been made in converting our unit tests to Typescript, which improves our type definitions. There’s a really good chance we’re going to come out of this Hacktoberfest with a much more Typescript-friendly experience for Grommet users.</p> <p>We’ve also seen some great collaboration going on in the repository. Community contributors are helping review each other’s pull requests, submitting issues, making suggestions for improvement, and helping answer questions. While these types of contributions don’t necessarily count towards the 4 pull requests needed to win a prize, this sort of assistance is greatly appreciated by everyone in the community and helps make it a more friendly, collaborative space.</p> <p><strong>There’s still time to participate!</strong></p> <p>Haven’t had a chance to check it out yet? There’s still a few weeks left before the end of the month. All you need to do is register on the <a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest</a> site to link your Github account and track your contributions. To contribute to Grommet, simply take a look at issues labeled <a href="https://github.com/grommet/grommet/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest">‘hacktoberfest’</a> on Grommet’s repository.</p> <p>We welcome contributions from any skill level. There are a variety of issues to pick from. You can even contribute to issues that don’t have the <a href="https://github.com/grommet/grommet/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest">‘hacktoberfest’</a> label, as they will still count. If you are a first time contributor check out our <a href="https://github.com/grommet/grommet/blob/master/CONTRIBUTING.md">contribution guide</a> and keep an eye out for issues labeled <a href="https://github.com/grommet/grommet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22">‘good first issue’</a>.</p> <p>Happy Hacking!</p>https://developer.hpe.com/2021-October-1/https://developer.hpe.com/2021-October-1/Sat, 02 Oct 2021 05:00:00 GMThttps://developer.hpe.com/secure-containerized-and-traditional-apps-concurrently/https://developer.hpe.com/secure-containerized-and-traditional-apps-concurrently/Thu, 30 Sep 2021 08:48:31 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>IT Ops teams have run and managed Windows and Linux deployments for years. They are greatly experienced at dealing with the storage, networking, virtual machines and firewalls required for these environments. But container use, and the modern apps with which they are built, continues to gain ground as businesses move from small, proof-of-concept implementations to broader deployments. These modern applications require a whole new set of skills, which many administrative teams are just learning as new technologies, like Kubernetes, come into play.</p> <p>The best practices used to monitor and guide these implementations continue to evolve, taking these newer technologies into account. One area that continues to crop up as an area of concern is security. Just how does one ensure the security of containerized workloads, especially during a time when they most probably need to co-exist with traditional applications?</p> <h2><strong>Centralize security and compliance management with Runecast</strong></h2> <p>Hewlett Packard Enterprise (HPE) offers the <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Ezmeral Container Platform</a>, which comes integrated with Falco, an open source runtime threat detection engine.  Falco uses community-sourced detection of malicious activity and Common Vulnerabilities and Exposures (CVE) exploits to generate alerts.  In addition, HPE Ezmeral Container Platform also provides a core set of monitoring and alerting capabilities using a combination of Metricbeat data collector, Elasticsearch for search and analytics, and Kibana for dashboard displays. These tools provide IT Ops teams with the metrics they need to monitor traditional applications.</p> <p>To further enhance the security capabilities on the HPE Ezmeral Container Platform, HPE partnered with <a href="https://www.runecast.com/">Runecast</a> to offer an analyzer that provides insights to container security compliance and improves the stability of mission-critical IT applications as they migrate to a modern cloud architecture. Organizations can leverage Runecast Analyzer as a central security and compliance management console for both container and non-container environments.</p> <h2><strong>Automate audits</strong></h2> <p><a href="https://www.runecast.com/how-does-runecast-analyzer-work">Runecast Analyzer</a> complements the base HPE Ezmeral Container Platform monitoring features with an analysis of best practices and security compliance checks. It does this for container-based workloads as well as workloads running on more traditional platforms; like VMware’s vSphere, vSAN, NSX, Horizon, and VMware Cloud Director. These capabilities are also available for AWS and Microsoft Azure – all from a single interface.</p> <p>Runecast Analyzer provides continuous analysis of the workload infrastructure. It includes best practices (as detailed by the Cloud Native Computing Foundation, the maintainer of the Kubernetes open-source project) and security compliance checks against the latest benchmark from the Center for Internet Security (CIS). Runecast Analyzer provides full coverage for the entire CIS Benchmark for Kubernetes with 71 individual cross-referenced checks against entire Kubernetes environments, highlighting areas that may need improvement.</p> <p><img src="/img/runecast.png" alt="How Runecast works with the HPE Ezmeral Container Platform" title="How Runecast works with the HPE Ezmeral Container Platform"></p> <p>With Runecast Analyzer, IT organizations gain a common tool for monitoring VMs, cloud, and HPE Ezmeral Container Platform applications. This shortens the learning curve for IT Ops teams in adopting Kubernetes. It also helps IT administrators deploy and manage VMs, containers, and cloud environments at scale with confidence.</p> <p>To learn how Runecast helps you monitor and improve the security of your containerized and traditional apps, visit <a href="https://urldefense.com/v3/__http:/www.runecast.com__;!!NpxR!20PeQhlxWuRFNki74flD2O5Cb4wduoVPQd30Aso29B0LbmGLbcLmPg9JZ2O3D_ao$">www.runecast.com</a>. To learn more about Runecast and how it works on HPE Ezmeral, check out the <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Ezmeral Container Platform website</a> and <a href="https://psnow.ext.hpe.com/doc/a50003809enw">technical paper</a>. To get Runecast Analyzer and run it natively on your HPE Ezmeral Container Platform (now HPE Ezmeral Runtime), go to the <a href="https://www.hpe.com/us/en/software/marketplace.html">HPE Ezmeral Marketplace</a>. For more articles on HPE Ezmeral Container Platform, check out the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> and <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/bg-p/software/label-name/containers%20and%20devops#.YVNc4LhKg2w">HPE Ezmeral UnCut blog</a>.</p>https://developer.hpe.com/hpe-oneview-and-servicenow-integration-with-stackstorm/https://developer.hpe.com/hpe-oneview-and-servicenow-integration-with-stackstorm/Wed, 29 Sep 2021 17:53:58 GMT<p>HPE OneView is a powerful infrastructure automation/management platform from Hewlett Packard Enterprise (HPE) used to manage and monitor HPE DL servers and HPE Synergy products. Recently, I wanted to get all the alarms from HPE OneView and automatically save them as records in a ServiceNow table.</p> <p>ServiceNow is a software as a service (SaS) used by many large corporations for automating critical business workflows and information. I wanted to make an event-based automation that would leverage HPE OneView and ServiceNow's Restful APIs. There is another way to do this but I found it was a bit too 'involved' and I wanted something easy. Who doesn't like easy?</p> <p>Having developed solutions to complete an identical task in StackStorm (HPE Nimble Storage to ServiceNow), it was super easy to dust off the StackStorm integration pack I created for HPE Nimble and refactor it for HPE OneView. Creating such an integration pack would give users a way to transfer these alarms into a ServiceNow table, with very little human intervention. Naturally, my second thought was how can I use the Twitter platform to 'tweet' some VLAN (or any other) information into HPE OneView! Can it be done?</p> <p>HPE OneView has a powerful RESTful API that can be used to get information in and out of HPE OneView. ServiceNow has a powerful RESTful API as well. All that I need to write is some middleware and leverage a couple of Python bindings (Python code that abstracts the API). Turns out the python bindings are already written for both systems and available on GitHub! To solve this problem I can write a handful of Python scripts and I should be good to go (or GTG if you're hip and cool).</p> <p>I quickly realized that, in order to do what I wanted to do, it would involve writing the code for both systems. But what if I were to leverage <a href="https://stackstorm.com/">StackStorm</a>? StackStorm is an event based automation platform with over one hundred and seventy 3rd party integrations just waiting to be consumed! A quick check of the <a href="https://exchange.stackstorm.org/">StackStorm Exchange</a> indicates that there's a StackStorm integration pack available for ServiceNow. Using StackStorm, I'd only have to write half the code, as I would only have to write the code for an HPE OneView StackStorm integration pack. The other benefit of using StackStorm is I can take advantage of the programmable rules and triggers. Something I like to call 'Real automation'.</p> <p>Note: I have written a couple of other blog posts on StackStorm. If you are interested in trying this approach, I suggest you go to the HPE DEV blog and <a href="https://developer.hpe.com/search/?term=stackstorm">read my other posts</a>.</p> <p><img src="/img/flowchart.png" alt=""></p> <p>Developing the stackstorm-hpe-oneview integration pack (which is available <a href="https://github.com/HewlettPackard/stackstorm-hpe-oneview">here</a>) is fairly straightforward. For this interaction to function, I wrote five very short actions and a couple of simple rules. You can see in the chart at the top of this blog that two of the actions will be used with the first workflow and three will be need for the second workflow. Actions are the workhorse of StackStorm. Actions have a 'runner-type' and there are <a href="https://docs.stackstorm.com/actions.html">12 different ones</a> to choose from. They can be shell scripts, Python scripts, or Orquesta for creating workflows. I could use a single action to connect to HPE OneView and request all of the current alarms and another to format and store the alarms in a MongoDB database for further processing.</p> <p>In the code example below I am using the alerts.get_all() function to retrieve the alarms from HPE OneView. A quick check to see if the object is a list and return it.</p> <pre><code class="language-python">from lib.actions import HpeOVBaseAction class networks(HpeOVBaseAction): def run(self): ov_alerts = self.client.alerts.get_all() if isinstance(ov_alerts, list): return (True, ov_alerts) return (False) </code></pre> <p>The second action in workflow "A" will format the information into a MongoDB record, add a process field and save the MongoDb BSON document. Again, this is very simple to code and test. The class is passed the alarms and iterates through each one, a query to check if the document exists and if not, formats a Python dictionary and writes the MongoDb BSON document via pymongo. This is all it takes to collect the alarms and save them in the database.</p> <pre><code class="language-python">import pymongo from lib.actions import MongoBaseAction class loadDb(MongoBaseAction): def run(self, alarms): mydb = self.dbclient["app_db"] known = mydb["dwralarms"] new_alarm={} for alarm in alarms: myquery = { "_id" : alarm['created'] } records = known.find(myquery).count() if records == 0: new_alarm['u_vendor']='hpe-oneview' new_alarm['u_sev']=alarm['severity'] new_alarm['u_desc']=alarm['description'] new_alarm['u_uuid']=alarm['resourceUri'] new_alarm['_id']=alarm['created'] new_alarm['u_created']=alarm['created'] new_alarm['u_process']='no' write_record = known.insert_one(new_alarm) return (records) </code></pre> <p>Here is an example of the StackStorm workflow that calls two actions every five minutes.</p> <pre><code class="language-yaml">version: 1.0 description: A workflow to copy HPE OneView alarms into a mongo database. tasks: getalarms: action: hpeoneview.get_alerts next: - when: &#x3C;% succeeded() %> publish: - alarms: &#x3C;% result().result %> do: sendmongo sendmongo: action: hpeoneview.load-hpeov-alarms alarms=&#x3C;% ctx().alarms %> </code></pre> <p>The second workflow, workflow "B" will call another action every five minutes that reads the documents from the MongoDB database, looks for the processed flag set to no, collects the results into a Python list and returns it.</p> <pre><code class="language-python">import pymongo from lib.actions import MongoBaseAction class loadDb(MongoBaseAction): def run(self): mydb = self.dbclient["app_db"] known = mydb["dwralarms"] list_to_process = [] myquery = { "u_process" : 'no' } records = known.find(myquery) for r in records: list_to_process.append(r) return (list_to_process) </code></pre> <p>The next task is to send the list of alarms that have not been processed to ServiceNow. Here is where the power of integration packs comes into view. All I need to do is issue a command on my StackStorm server <strong>"st2 pack install servicenow"</strong>. By issuing this command, I gain access to the automation scripts (actions) that are pre-written for ServiceNow. Now that I am using StackStorm and have access to all the automation on the StackStorm exchange. I can communicate with many other systems without writing any vendor specific code to do so. The following example is the ServiceNow action that creates records in a ServiceNow table.</p> <pre><code class="language-python">from lib.actions import BaseAction class CreateRecordAction(BaseAction): def run(self, table, payload): s = self.client path = '/table/{0}'.format(table) response = s.resource(api_path=path).create(payload=payload) return response </code></pre> <p>Here is what the workflow looks like for the collection of alarms and sending them to ServiceNow. The first action pulls the unprocessed alarms from the MongoDB database and publishes the array to the 'context', a place to stash variables that can be accessed by other actions. Next it iterates through the array by using the 'with' statement and sends the contents to ServiceNow.</p> <pre><code class="language-yaml">version: 1.0 description: A workflow to copy HPE OneView alarms from mongo and into snow. tasks: getalerts: action: hpeoneview.get_mongo_alarms next: - when: &#x3C;% succeeded() %> publish: - alarms: &#x3C;% result().result %> do: snowalerts snowalerts: with: &#x3C;% ctx().alarms %> action: servicenow.create_record table="u_hpeov_alarms" payload='&#x3C;% item() %>' next: - when: &#x3C;% succeeded() %> do: processalarms processalarms: action: hpeoneview.process_alarms alarms=&#x3C;% ctx().alarms %> </code></pre> <p>To finish this up, set the process flag to "Yes" so you do not duplicate records into ServiceNow. It looks like the example below:</p> <pre><code class="language-python">import pymongo from lib.actions import MongoBaseAction class loadDb(MongoBaseAction): def run(self, alarms): mydb = self.dbclient["app_db"] known = mydb["dwralarms"] for a in alarms: known.update_one({"_id":a['_id']},{"$set":{"u_process":"yes"}}) return () </code></pre> <p>That's it! Once both integration packs are installed on a StackStorm server and authorized, the rules will 'fire' every five minutes and the workflows will do the heavy lifting so you don't have to.</p> <p>Finally, a diagram that shows all the moving parts of workflow "A". The rule that runs on the interval timer calls an action that, in turn, calls a workflow that calls a couple other actions. Notice that actions can be Python scripts or YAML files. It just depends on their function.</p> <p><img src="/img/full-workflow.png" alt="" title="Workflow "></p> <p>To make this a truly automated process, the ServiceNow account needs to exist and the tables need to be created in advance.</p> <p><strong>Let's break down the steps to get this going.</strong></p> <ol> <li>Create the tables in ServiceNow, you will need your instance ID from ServiceNow to authorize the StackStorm server.</li> <li>Install the HPE OneView StackStorm integration pack and authorize it.</li> <li>Install the ServiceNow StackStorm integration pack and authorize it.</li> <li>Wait five minutes.</li> </ol> <p>In the beginning of this blog I mentioned tweeting information into HPE OneView. I don't know why you would want to do this but yes, with StackStorm it is possible. StackStorm exchange has a Twitter integration pack and it can be installed on the StackStorm server by issuing the command 'st2 pack install twitter'. I can use the twitter StackStorm sensor to 'watch' the twittersphere for any tweets containing a certain word or phrase contained in the tweet-body. If the sensor reacts, it will cause a StackStorm trigger to fire and I can call an action to pull the tweet and collect the information from the tweet-body and send that information to the HPE OneView Stackstorm actions. This can be done with ANYTHING on the StackStorm exchange.</p> <p>In conclusion, this may seem complicated at first. In reality, its a group of small simple scripts that are linked together inside the StackStorm framework. It also provides for the adoption of many different integration packs and allows for the event-based automation of many different systems. To learn more about StackStorm, you can take my <a href="https://github.com/xod442/stackstorm-tutorial">tutorial</a>, attend the StackStorm Workshop-on-Demand available <a href="/hackshack/workshop/21">here</a> <a href="/hackshack/workshop/21"></a>and join the automation revolution!</p>https://developer.hpe.com/sir-hackington-rides-again/https://developer.hpe.com/sir-hackington-rides-again/Wed, 22 Sep 2021 17:20:11 GMT<p>If you’re like me, you’re probably itching to get out and about after months and months of virtual meetings and events. When I heard that <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/">KubeCon|CloudNativeCon NA</a> was going to be a hybrid live/virtual event, I could hardly contain myself! From October 13th to 15th, this Cloud Native Computing Foundation’s flagship conference will take place both in Los Angeles, California and online where vendors will be showcasing a full range of technologies that support the cloud native ecosystem. And I’m going to proudly be there myself in the Platinum Sponsor Hewlett Packard Enterprise (HPE) booth!</p> <p>Who am I, you ask? Could it be? Has it been so long that you’ve been to an event that you’ve forgotten your dear, little friend, Sir Hackington? Or perhaps you have only recently joined our HPE Developer Community and we haven’t yet been properly introduced. My name is Sir Hackington Appbuilder III. I’m the cute little canine that travels with the HPE DEV team to different events as Guardian of the Swag. Here’s a picture of me in my glory days:</p> <p><img src="/img/vmworld_collage-1538516807872-2-.jpg" alt="Sir Hackington at Discover" title="Sir Hackington at Discover"></p> <p>A lot has happened since then. HPE acquired several companies, along with important technologies, that boosted its position in the market by offering <a href="https://siliconangle.com/2021/03/17/hpe-ezmeral-proving-enterprise-ready-container-platform-hybrid-world-hpeezmeral/">the enterprise-ready container platform for a hybrid world</a>. Recognizing the critical need of companies today to manage data pipelines across hybrid computing environments from edge to cloud, HPE invested heavily in a 100% open source Kubernetes-based container environment from BlueData. It blended it with groundbreaking data fabric technology from MapR and added hooks for important open source projects; SPIFFE and SPIRE for security, and Apache Spark for data analytics. The final result is the HPE Ezmeral software platform, which ensures those who require data have easy and secure access to what they need regardless where they are. And customers can even consume this as a service through HPE GreenLake.</p> <h2>KubeCon|CloudNativeCon NA 2021</h2> <h3>What’s happening onsite?</h3> <p>Starting at 10:30am PT, you’ll be able to find the HPE team at booth P4 in the Los Angeles Convention Center. The theme of our booth for this event is <strong>BUILD FAST. SECURE. TOGETHER. ANYWHERE</strong>. And that’s really what it’s all about – HPE working <em><strong>with</strong></em> developers and providing the tools to innovate and deliver solutions quickly and securely, from edge to cloud. Folks still working from home? Great! Need to share apps and information from one site to another and collaborate seamlessly? Great! The cloud <em><strong>is</strong></em> hybrid – just like this conference is – and HPE really knows how to enable this environment.</p> <p>In the booth, there’s a presentation theater and three dedicated demo stations where you can physically talk with HPE subject matter experts. Technologies being presented include:</p> <p><strong>Apache Spark on HPE Ezmeral</strong> – where you’ll discover how to accelerate analytics with HPE Ezmeral by leveraging Apache Spark’s latest features</p> <ul> <li>Discover how HPE Ezmeral helps data scientists and engineers leverage Apache Spark 3 with Delta Lake integration</li> <li>Learn how Apache Spark applications easily access S3 storage</li> <li>Dig into how Delta Lake enables ACID transactions for enterprise-class storage access in Spark applications</li> <li>Watch a demonstration of the tools HPE Ezmeral provides for DevOps and Sys Admins to assist data scientists and engineers with elastic compute and storage resources in CNCF Kubernetes-based clusters</li> </ul> <p><strong>HPE CSI Driver for Kubernetes</strong> – where you’ll learn to speed up DevOps and boost operations with advanced data services</p> <ul> <li>Hear about multitenancy involving HPE storage for Kubernetes clusters with private cloud</li> <li>See how the HPE CSI Driver’s advanced security features encrypt volumes regardless of the storage backend, both in-flight and at-rest</li> <li>Learn about opportunities with Prometheus monitoring and alerting for cloud-native apps and infrastructure</li> <li>Become familiar with HPE partnerships in data protection and data management, including Veeam, Commvault, Cohesity, and Zerto, HPE’s latest acquisition</li> <li>Explore enhanced developer efficiency with file and block for K8apps and data management for stateful apps</li> <li>Get to know more about operational agility and data protection</li> </ul> <p><strong>Zero Trust</strong> – where you’ll observe how SPIFFE/SPIRE delivers strongly attested identities from edge to cloud to establish a zero trust security model</p> <ul> <li>Learn what HPE, the leading contributor to CNCF’s SPIFFE and SPIRE projects, is developing around zero-trust security for application infrastructures in Kubernetes and beyond</li> <li>Celebrate the milestones reached: The release of SPIRE V1.0 and the successful completion of a detailed CNCF security audit</li> <li>Familiarize yourself with new feature development, like support for serverless functions, hardware attestation using TPM devices, and integration with Istio, that are in progress</li> </ul> <p>Check out these links to find out more information about what will be shown regarding <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Don-t-Miss-HPE-at-KubeCon-CloudNativeCon-North-America-2021/ba-p/7149736">HPE Ezmeral </a>and the <a href="https://community.hpe.com/t5/Around-the-Storage-Block/Visit-HPE-Storage-at-KubeCon-North-America-Virtual-and-in-person/ba-p/7149757">HPE CSI Driver for Kubernetes</a>. You might also want to check out this blog post on how to <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Ready-to-become-a-superhero-Build-an-ML-model-with-Spark-on-HPE/ba-p/7149454#.YVH9BLhKg2y">Build an ML model with Spark on HPE</a> to get some cool insight on one of the demos being shown there.</p> <p>In the booth’s HPE DEV Community Center, take a tour of the <a href="https://developer.hpe.com/">HPE DEV portal</a> – a door into the online dimension where developers and data scientists alike can find the resources they need to build innovative solutions on HPE products and solutions. This is where you’ll find me and all the HPE DEV swag.</p> <p>I also heard that there was going to be a magician over in my area! As long as he doesn’t make me disappear, I’m good with that.</p> <p>You’re going to want to pace yourself, because there are a lot of breakout sessions in addition to the presentations on the event floor. On Monday, October 11th, HPE will be hosting and sponsoring the <a href="https://events.linuxfoundation.org/production-identity-day-spiffe-spire-north-america/">CNCF’s Production Identity Day</a> event. This provides a forum for SPIFFE and SPIRE users to work together. Speakers will include engineers from HPE and other vendors, as well as SPIRE users. If you can’t make it to this event, stop by the HPE booth to see a demo of new SPIRE features.</p> <p>On Friday, October 15th, from 3:25-4:00pm, catch HPE Chief Technologist Tom Golway and HPE Fellow Thomas Phelan in their talk on <a href="https://kccncna2021.sched.com/event/lV5v?iframe=no">Using Kubernetes with Data Processing Units to Offload Infrastructure</a>. They plan on sharing some novel work related to offloading core Kubernetes software infrastructure components from the main CPU onto the processing units of DPUs that looks pretty interesting.</p> <h3>What’s happening online?</h3> <p>As I mentioned earlier, this is a hybrid event, with people being able to attend the conference virtually as well as in person. While I admit to missing the physical aspect of conferences like this, some of the best activities are found <a href="https://kubecon-cloudnativecon-na.com/virtual-exhibitor/?v0326b739525aaf6a5900c153ea6485e67109462e8db159b156161fc07c7e3d8016769932b4c0398e64b5ea52edb3d1c5=98D89AED6140001531DE1D5095DD75E0A2A33735543DD0787B548CBAEAA423B2F93B8EABF8C08225934CAB9C3C342DEF&#x26;fromHall">online</a>.</p> <p><img src="/img/virtual-booth-update-3.jpg" alt="HPE Virtual Booth" title="HPE Virtual Booth"></p> <p>As part of our online presence, the HPE DEV team will be delivering several live office hour sessions. Speak with HPE DEV team members to learn more about the <a href="https://developer.hpe.com/community">HPE Developer Community </a>and how you can benefit from joining it. Whether it’s information found on the web portal, technical blogs and tutorials, GitHub repos, free hands-on technical workshops, interactive technology talks, or online collaboration <em>there’s treasure to be found</em> when you’re an HPE DEV member! And it’s all free!</p> <p>Just like what was done for KubeCon|CloudNativeCon Europe, the HPE DEV team is running a <a href="https://bit.ly/kubecon-na-2021-hpedev-treasure-hunt">Treasure Hunt</a>. This giveaway contest will run from October 11-17, 2021 – the week of the KubeCon event. Be one of the first to answer all the online questions correctly during this time period and you could win an HPE DEV hat!</p> <p>One of my fondest memories of pre-pandemic events was when folks used to stop by the HPE DEV Hack Shack. There, they would engage in coding challenges, workshops, and play a fun, little retro arcade game called <a href="/hackshack/hackshackattack">Hack Shack Attack!</a> When the pandemic hit, the Hack Shack moved to an online presence. It’s now open 24/7. This means you can take workshops any time in the form of our <a href="/hackshack/workshops">HPE DEV Workshops-on-Demand</a>. You can even play Hack Shack Attack! by visiting the Hack Shack <a href="/hackshack/arcade">Arcade</a>. We even offer the ability to <a href="/hackshack/stickerwall">download stickers</a> (virtual swag) from the site. Make sure you stop by and visit!</p> <h3>So, are you coming?</h3> <p>I’ve only scratched the surface of everything that will be going on at the event. I hear there’s going to be a couple of really cool surprises in the booth as well (did someone mention hoodies?), so come on by and check things out.</p> <p>It feels like such a long time since we’ve had the chance to meet in person. I’m really excited about this opportunity. For those of you attending in person, remember to wear your mask. If you need an extra, we’ll have some over in the HPE booth. And for those of you attending online, don’t forget to visit the <a href="/hackshack/">HPE DEV Hack Shack</a> and check out what’s being offered there as well.</p>https://developer.hpe.com/analyzing-the-success-of-the-hpe-dev-workshops-on-demand/https://developer.hpe.com/analyzing-the-success-of-the-hpe-dev-workshops-on-demand/Thu, 16 Sep 2021 07:14:18 GMT<p><img src="/img/wodanalysisblog1.png" alt="HPE DEV Hack Shack" title="HPE DEV Hackshack"></p> <p>In October of 2020, the HPE DEV team introduced a <strong>very easy</strong> way for users to learn about a variety of different software topics by interacting with code virtually – in the form of <strong>free</strong> HPE DEV Workshops-on-Demand. Starting with just three workshops, the program has continued to grow and now offers over 20 courses, ranging from basic coding 101 to sophisticated infrastructure automation.</p> <p><img src="/img/wodanalysisblog2.png" alt="HPE DEV Hack Shack Workshops-on-demand catalog" title="HPE DEV Hack Shack Workshops-on-demand catalog"></p> <p>These hands-on workshops cover a variety of <a href="/hackshack/workshops">topics</a>, including open source technologies (i.e. Kubernetes, SPIFFE/SPIRE, Grommet, StackStorm), popular programming languages (i.e. Python, RUST), coding tools (i.e. APIs, Git), and market-leading HPE technologies (i.e. HPE Ezmeral Container Platform and Data Fabric, HPE OneView, HPE iLO). Whether it’s due to the popularity of these topics, the workshop’s unique, easy-to-follow methodology, the associated badge recognition program, or the fact that they’re free – these workshops have certainly gained quite a lot of interest!</p> <p><strong>Measuring success</strong></p> <p>The goal of the HPE DEV program is to foster a community where members can innovate through shared experiences, collaboration, and learning. As just one of the services HPE DEV offers to the developer community, the Workshops-on-Demand program is an important one that fits these goals very nicely. Every so often we take stock of our programs to determine how much value each one offers so we can make any necessary adjustments. And, as the program manager for the HPE DEV Workshops-on-Demand, it really falls on me to make that determination.</p> <p>From my standpoint, rocketing from offering 3 to now over 20 workshops during the course of a year could easily be seen as success. But from a participant‘s side, it might be different. So what data should be considered a good measure of success for this program?</p> <p>I decided to analyze a few data points from different categories that appeared to provide some important insights:</p> <ul> <li> <p>Participants:</p> <ul> <li>Number of attendees</li> <li>Type of attendees (including HPE employees, partners, customers)</li> <li>Completed workshop surveys</li> </ul> </li> <li> <p>Content:</p> <ul> <li>Number of workshops produced</li> <li>Topics covered</li> </ul> </li> <li> <p>Infrastructure:</p> <ul> <li>Resiliency of the platform</li> <li>Accessibility</li> </ul> </li> </ul> <p><img src="/img/wodanalysisblog3.png" alt="" title="Total Number of participants per month"></p> <p>As you can see on the chart above, the increase in the number of people using the workshops has been growing steadily month over month. The times where you see noticeable jumps in attendance are due to events where the team would make use of the HPE DEV Workshops-on-Demand platform to deliver technical sessions; i.e. HPE Technology and Solutions Summit in March or HPE Discover in June. These events help promote the HPE DEV Workshops-on-Demand, as many event participants come back later to register for additional workshops. In my mind, having a participant show up to a workshop is good, but having one come back for more sessions, knowing that they could take up to 4 hours, is impressive. I consider this as a good mark of success.</p> <p>Analyzing the types of attendees helps us gauge whether the content being provided is appropriate. It helps us answer questions like “Is the technical level too deep?”, “Is more time required?”, and “Are we hitting the appropriate target audience?”</p> <p>Another important measurement is the number of surveys filled out at the end of each workshop. These surveys are very important to us for several reasons:</p> <p><strong>Reason #1</strong></p> <p>They allow us to see if our content is relevant to the participant. In the completed surveys, participants gave the workshops an average rating of</p> <ul> <li><strong>4.64 stars out of 5 for technical level</strong></li> <li><strong>4.63 stars out of 5 for the overall value of the workshop</strong></li> </ul> <p><strong>Reason #2</strong></p> <p>They allow us to see if the delivery method is correct. The surveys indicated that the Jupyter Notebooks format has been very well accepted, with an average rating of</p> <ul> <li><strong>4.60 stars out of 5</strong></li> </ul> <p><strong>Reason #3</strong></p> <p>They help us define new topics and give us insight into the user’s experience. Comments such as “<em>Great initiative, the more technical format is the way to go for us engineers and developers, just try to go deeper (using more time if required)</em>.”and “<em>Great idea and a very suitable form of delivery.</em>” provide feedback that we can use to modify content and add additional, appropriate courses.</p> <p><strong>Content</strong></p> <p>Aligning with the HPE DEV charter, the workshops span topics that interest our audience, which tend to be related to:</p> <ul> <li>IT OPS and DevOPs in terms of API interaction, Automation</li> <li>Open source</li> <li>Programming languages</li> <li>Data analytics</li> <li>AI / MLOPS</li> </ul> <p>The workshops range from simple 101-level sessions to more advanced labs. We try to keep a balance of courses available for participants who are just starting, to courses for those community members who are looking for more technical depth.</p> <p>As mentioned earlier, we started small and have continued to build out a wide selection of workshops over this past year. We plan to continue to expand this program, always taking your feedback on which new topics should be offered.</p> <p><img src="/img/wodanalysisblog4.png" alt=""></p> <p>Some topics tend to be more popular than others. You can see that the HPE Ezmeral workshops are very well attended. AI / MLOPS and HPE iLO/Redfish workshops are also quite popular. Open Source and Programming workshops follow close behind.</p> <p><strong>Infrastructure</strong></p> <p>From a backend perspective, we started the program with a single JupyterHub server to support the very first workshops. Each of the workshops are using different types of API endpoints. When reaching a public weather forecasting API, nothing much is required on the backend side apart from the JupyterHub server itself.</p> <p>When targeting an HPE Ezmeral Container Platform API, however, the requirements are completely different. To face an ever growing number of workshops and participants, we decided to implement additional JupyterHub servers. In addition to the first JupyterHub server sitting in an HPE site in France, we created a second JupyterHub server in another HPE location. We also leverage HPE GreenLake cloud services to offload some workshops. We now have three production sites, which allow us to improve our workshop’s resiliency. We can now easily redirect an existing workshop from one site to another.</p> <p>Although this is all transparent to the participant, this is key to the success of the program, as it allows our workshops to run 24/7, all year long, and still for free!</p> <p>Our Ansible-based automation allows us to replicate content easily when combined with the right set of Git commands. We can build or rebuild a complete working and production-ready JupyterHub server in just a few hours and monitor the stability of the environment to ensure our workshops are always ready to go.</p> <p>From a frontend perspective, we also improved several aspects. Our developer did a great job in redesigning the tiles layout making it easier to navigate through the Workshops-on-Demand catalog. We added a dedicated registration page per workshop too. Finally, we also performed a REST API integration of our frontend with the <a href="https://hpedemoportal.ext.hpe.com/">HPE Demonstration Portal</a>, which allows HPE Demonstration Portal users to book workshops while remaining in the context of HPE Demonstration Portal.</p> <p><strong>Looking forward</strong></p> <p>I plan to come back next year and summarize the achievements of the team regarding Phase 2 of the program. Planned upgrades include an even nicer portal for our workshops, a broader range of 101 workshops covering artificial intelligence (AI), MLOPS, Spark, Concourse and more.</p> <p>The HPE DEV Workshops-on-Demand continue to gain momentum. The quality and relevance of the content has definitely played a part in that. Internal and external promotion also helps to build awareness. But another, more fun, reason why more and more users are taking them may have to do with our badge recognition program. For each workshop that you take, you earn a badge. Check out this <a href="https://developer.hpe.com/blog/become-a-legend/">blog post</a> for more information on this program.</p> <p>If you haven’t taken one of our workshops yet, please <a href="/hackshack/workshops">register for one of our Workshops-on-Demand</a>, and see why they have attracted so many users. It’s really important to fill out the survey at the end of the workshop to provide us with feedback on your experience, as well as inform us about new subjects you would like to see offered. As always, it’s your feedback that really helps to improve this program within the <a href="https://hpedev.io">HPE DEV Community</a>.</p>https://developer.hpe.com/accessing-hpe-data-fabric-s3-storage-from-spring-boot-s3-micro-service-deployed-in-k3s-cluster/https://developer.hpe.com/accessing-hpe-data-fabric-s3-storage-from-spring-boot-s3-micro-service-deployed-in-k3s-cluster/Mon, 13 Sep 2021 08:13:40 GMT<p>Containers and microservices are transforming Edge and IoT platform use cases that can be deployed in small footprint Kubernetes clusters on edge nodes and persisting data at a central location. This data pipeline can be easily accessed by downstream complex analytics applications for further processing.</p> <p>In this article, I will discuss how to access HPE Ezmeral Data Fabric Object Store (S3) using Spring Boot S3 Micro Service application deployed in a <a href="https://k3s.io/">K3s cluster</a> and perform basic S3 operations like upload, list, delete etc. The below diagram gives an overview of the architecture.</p> <p><img src="/img/hpe-ezmeral-data-fabric-s3-springboot-k3s.png" alt=""></p> <h5>Figure 1: Architecture overview of Spring Boot S3 Micro Service on K3s with HPE Data Fabric Object Storage as the back end.</h5> <p>A brief description of the technology stack used is described in the sections below:</p> <p><strong>Data Fabric Object Storage</strong></p> <p>The Object Store with an S3-compatible API stores data generated through multiple data protocols, such as NFS, POSIX, S3, and HDFS. Data in the Object Store is accessible through S3 API requests. The Object Store manages all inbound S3 API requests to store data in or retrieve data from an HPE Ezmeral Data Fabric cluster. More details can be found <a href="https://docs.datafabric.hpe.com/62/MapRObjectStore/MapRObjectStorewithS3-compatibleAPI.html">here.</a></p> <p><strong>K3s</strong></p> <p>Lightweight Kubernetes, aka K3s, is easy to install and consumes half the memory, all in a binary of less than 100MB. It's great for edge and IoT use cases. More information on K3s can be found at <a href="https://rancher.com/docs/k3s/latest/en/">Rancher</a> site<a href="https://rancher.com/docs/k3s/latest/en/"></a>. Follow the steps as mentioned in <a href="https://rancher.com/docs/k3s/latest/en/quick-start/">QuickStart Guide</a> <a href="https://rancher.com/docs/k3s/latest/en/quick-start/"></a> for installation of the K3s cluster. </p> <p><strong>Spring Boot</strong></p> <p>Spring Boot is an open source Java-based framework used to create a micro Service. Many real world applications are written in Spring Boot for faster development and better maintainability. More information can be found at <a href="https://spring.io/projects/spring-boot">spring.io</a></p> <p>Note: We can move the Spring Boot application to Quarkus with no change in code. Moving code to Quarkus will reduce the footprint of the Spring Boot application. More information can be found at <a href="https://quarkus.io/blog/quarkus-for-spring-developers/">Quarkus site</a></p> <p>In this blog post, you'll checkout the existing Spring Boot application from <a href="https://github.hpe.com/kiran-mavatoor/df-s3-springboot-k3s-demo">GitHub</a>, customise it and execute. </p> <p><strong>Application Prerequisites</strong></p> <p>1.      The K3s cluster must be accessible. Note down the control plan node details. This information is required to deploy the Spring Boot application.</p> <p>2.      Access the HPE Data Fabric Object Store service UI running on port 9000. For example URL  - <a href="https://FQDN:9000/.%C2%A0%C2%A0Note">https://FQDN:9000/.  Note</a> down the access key and secret key. It is advised to change the default values.</p> <p>3.      Java 11, Apache Maven 3.8+, Docker Client.</p> <p><strong>Build and Install Steps</strong></p> <p>1.      Check out an existing Spring Boot application from <a href="https://github.hpe.com/kiran-mavatoor/df-s3-springboot-k3s-demo">GitHub</a>.</p> <p>2.      Copy the ssl_usertruststore.p12 from the HPE Data Fabric cluster into certs folder under project directory. The ssl_usertruststore.p12 file is located at /opt/mapr/conf directory in cluster node. The password for p12 can be copied from “ssl.client.truststore.password” property value in /opt/mapr/conf/ssl-client.xml .</p> <p>3.      From the project directory, open resources/application.properties. Change the key values as per your environment. </p> <p>4.      Execute “mvn clean install” .</p> <p>5.      The distributable is available in target/df-s3-springboot-k3s-demo-1.0-SNAPSHOT.jar .</p> <p>6.      Edit the DockerFile located in project directory. The value of  “-Djavax.net.ssl.trustStorePassword”  must be same as the value of  “ssl.client.truststore.password” obtained from Step 2.</p> <p>Note: This value can be configured using config-map.yaml in K3s cluster.</p> <p>7.      Execute below docker commands to build docker image and push it to docker hub. </p> <p>Note: Alternatively, we can use podman instead of docker to create images. More information on podman can be obtained from <a href="https://developers.redhat.com/blog/2020/11/19/transitioning-from-docker-to-podman#transition_to_the_podman_cli">here.</a> <a href="https://developers.redhat.com/blog/2020/11/19/transitioning-from-docker-to-podman#transition_to_the_podman_cli"></a></p> <pre><code class="language-bash">docker build -f Dockerfile -t &#x3C;Dockerhub user id>/df-s3-springboot-k3s-demo:latest . docker image ls docker login -u &#x3C;Dockerhub user id> > enter password: &#x3C;Dockerhub password> docker push &#x3C;Dockerhub user id>/df-s3-springboot-k3s-demo:latest </code></pre> <p>8.      Next, create below df-s3-springboot-k3s-demo.yaml file for deploying the executable in K3s cluster. A sample yaml file is given in the project directory. Please replace <dockerhub userid> with a valid id.</p> <pre><code class="language-Yaml">apiVersion: v1 kind: Service metadata: name: df-s3-springboot-k3s-demo-service spec: selector: app: df-s3-springboot-k3s-demo ports: - protocol: TCP name: df-s3-springboot-k3s-demo port: 8000 targetPort: 8000 type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment metadata: name: df-s3-springboot-k3s-demo spec: selector: matchLabels: app: df-s3-springboot-k3s-demo replicas: 1 template: metadata: labels: app: df-s3-springboot-k3s-demo spec: containers: - name: df-s3-springboot-k3s-demo image: &#x3C;Dockerhub userid>/df-s3-springboot-k3s-demo:latest imagePullPolicy: Always ports: - containerPort: 8000 </code></pre> <p>9.      Before deploying it in the Kubernetes cluster, validate the docker or podman image by running the image. </p> <pre><code class="language-bash">docker run -p 8000:8000 &#x3C;Dockerhub userid>/df-s3-springboot-k3s-demo </code></pre> <pre><code class="language-UI">http://localhost:8000/swagger-ui.hmtl </code></pre> <p><strong>Deploying in K3s cluster</strong></p> <p>1.      Log into the control plane node of the K3s cluster. Create/Copy the df-s3-springboot-k3s-demo.yaml to the node.</p> <p>2.      Execute “kubectl apply -f df-s3-springboot-k3s-demo.yaml”. If required, you can specify the namespace option.</p> <p>3.      Check the pod creation status by using the command “kubectl get pods -l app=df-s3-springboot-k3s-demo -o wide”.</p> <p>4.      Verify if the services are properly deployed by using the command “kubectl get service df-s3-springboot-k3s-demo-service”.</p> <p><strong>Accessing the Swagger UI from the pod</strong></p> <p>1.      Connect to the pod <a href="http://pod-ip:8000/swagger-ui.html">http://pod-ip:8000/swagger-ui.html</a>.</p> <p>2.      Verify the services exposed in the Swagger-UI.</p> <p><img src="/img/swagger-ui.png" alt="Swagger UI"></p> <p>In this blog post, you learned how data can be processed from edge to a persistence store by creating a data pipeline using a different technology stack. The aforementioned data pipeline is not limited to this current use case but can be used in many diversified microservice use cases.</p>https://developer.hpe.com/data-analytic-with-pyspark-using-hpe-ezmeral-container-platform/https://developer.hpe.com/data-analytic-with-pyspark-using-hpe-ezmeral-container-platform/Tue, 07 Sep 2021 02:12:15 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>PySpark is an interface for Apache Spark in Python. <a href="https://spark.apache.org/">Apache Spark</a> is a unified analytics engine for big data processing. It allows developers to perform data processing on files in a distributed filesystem, like the Hadoop distributed filesystem or HPE Ezmeral Data Fabric (formerly known as MapR-XD). Due to its complexity, setting up a Spark environment is always a pain for data scientists. Hopefully, HPE Ezmeral Container Platform can make this much easier.</p> <p>You can run Apache Spark jobs on Kubernetes-managed clusters on the HPE Ezmeral Container Platform. The HPE Ezmeral Container Platform provides you with access to a wealth of MLOps tools such as Apache Spark Operator, and things like a <a href="https://kubedirector.io/">Kubedirector</a> Jupyter Notebook where you will do your Data Science work and interact with the Apache Spark Operator to run your Apache Spark jobs.</p> <p>Once logged in as an MLOps tenant member, you can deploy an instance of Jupyter Notebook. From the Jupyter Notebook, you can either run Spark jobs with Apache Livy to make REST API calls to Spark Operator, or you can directly run a Spark job against the Spark Operator with the PySpark module.</p> <p>In this post, I will focus on running simple Spark jobs using the PySpark module on a Jupyter Notebook cluster instance deployed on HPE Ezmeral Container Platform. For those who want to squeeze the best performance out of Spark and run Spark Jobs with Apache Livy, visit this <a href="https://developer.hpe.com/blog/on-premise-adventures-how-to-build-an-apache-spark-lab-on-kubernetes/">post</a>.</p> <h2>Preparing the Jupyter Notebook Cluster</h2> <p>First, we have to prepare our favorite Jupyter Notebook environment. Inside a MLOps tenant, navigate to <strong>Notebooks</strong> tab. You will see a Jupyter Notebook KubeDirector app prepared for you. After clicking the <strong>Launch</strong> button, you will need to configure the compute resource needed.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120459929-39c63300-c3cb-11eb-9e7a-65189f4367d3.png" alt="image"></p> <p>As you can see below, you must specify the name of the Jupyter Notebook cluster. Click <strong>Enable DataTap</strong> to expand access to shared data by specifying a named path to a specified storage resource.</p> <p><img src="https://user-images.githubusercontent.com/72959956/132812471-d1ce5ce8-0d47-41ae-bd96-879262018f84.png" alt="image"></p> <p>Switching to the <strong>Notebook Endpoints</strong> tab, you can see that the access points are prepared for you. Just click the link and, after logging in with your LDAP/AD account, your favorite Jupyter environment is ready for you.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120460678-ea343700-c3cb-11eb-9aef-8afc9252d471.png" alt="image"></p> <p>Different kernels are already installed for you. No matter which languages you are using, there will be one that suits your ML project. There are two kernels which are Python related, i.e. Python3 and PySpark kernel. Python3 kernel is the kernel for you to run single node workloads while PySpark kernel, connected with Spark Operator and Livy, is the kernel for you to run distributed workloads. In this post, I am running a simple Spark job. So I will pick the Python3 kernel and import the PySpark module in the runtime.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120460537-cc66d200-c3cb-11eb-8410-3b7ec95051d5.png" alt="image"></p> <h2>Preparing the datasets</h2> <p>Imagine that you have a very large CSV file ready for analysis. You need to put the file to the distributed filesystem. Of course, you can do that with the graphic user interface provided through HPE Ezmeral Container Platform.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120461217-67f84280-c3cc-11eb-9126-e69cacef4432.png" alt="image"></p> <p>The other way to do this would be to drag the file to the left panel of the local Jupyter Notebook cluster and run the following HDFS commands to put the file to the "TenantStorage" through DataTap.</p> <pre><code class="language-bash"># Put a file from local filesystem to distributed filesystem using HDFS commands hdfs dfs -put enhanced_sur_covid_19_eng.csv dtap://TenantStorage/enhanced_sur_covid_19_eng.csv # List the files or directories hdfs dfs -ls dtap://TenantStorage/ # List the files or directories hdfs dfs -tail dtap://TenantStorage/enhanced_sur_covid_19_eng_.csv </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/129331881-dbe602e7-b3d9-4541-a9d0-4ea274aa7e51.png" alt="image"></p> <h1>Getting Started with PySpark</h1> <p>The PySpark module is already installed for you. No extra installation is needed. So convenient, isn't it? Some configurations for the PySpark runtime are needed in order to read files from DataTap.</p> <pre><code class="language-py"># python3 kernel from PySpark import SparkConf, SparkContext # Specify the path of the jars files conf = SparkConf().set("spark.jars", "/opt/bdfs/bluedata-dtap.jar") sc = SparkContext(conf=conf) # Specify the Hadoop configurations. sc._jsc.hadoopConfiguration().set('fs.dtap.impl', 'com.bluedata.hadoop.bdfs.Bdfs') sc._jsc.hadoopConfiguration().set('fs.AbstractFileSystem.dtap.impl', 'com.bluedata.hadoop.bdfs.BdAbstractFS') </code></pre> <h2>Reading datasets from HPE Ezmeral Data Fabric</h2> <p>After some configuration, your Spark engine is connected to the platform and you can now read files from HPE Ezmeral Data Fabric through Data Tap.</p> <pre><code class="language-py"># Commands for reading DataTap file. text = sc.textFile("dtap://TenantStorage/hello.txt") text.take(5) </code></pre> <p>For reading CSV files as a Spark dataframe, run the following commands:</p> <pre><code class="language-py"># Commands for importing PySpark SQL module. from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() # Commands for reading DataTap csv file. df = spark.read.csv('dtap://TenantStorage/enhanced_sur_covid_19_eng.csv', header=True, inferSchema=True) df.take(3) </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/122021373-333ab100-cdf8-11eb-9e58-edbccf43f0b2.png" alt="image"></p> <p><img src="https://user-images.githubusercontent.com/72959956/122021431-3e8ddc80-cdf8-11eb-9c61-d9bd400a4c9b.png" alt="image"></p> <h2>Data analytics with PySpark</h2> <p>With PySpark, you can easily read data from files, cleanse data and do analytics within a Jupyter notebook. To view the entire notebook, click <a href="https://github.com/helloezmeral/HPE-Ezmeral-HelloWorld/blob/main/pyspark/pyspark_covidhk.ipynb">here</a>.</p> <table> <thead> <tr> <th>Screenshot</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/122021467-45b4ea80-cdf8-11eb-8ca4-ffc11c03f1ad.png" alt="image"></td> <td>You can run <code>df.printSchema()</code> to view the schema of your dataframe.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/122021502-4baacb80-cdf8-11eb-87d3-b29ef643b373.png" alt="image"></td> <td>These are the commands for selecting columns of data and filtering according to the criteria.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/122021550-56fdf700-cdf8-11eb-9c31-e0d171c7406e.png" alt="image"></td> <td>Some common commands to interact with your datasets.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/122021576-5ebd9b80-cdf8-11eb-9810-36d744560327.png" alt="image"></td> <td>Commands for data aggregation.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/122021616-667d4000-cdf8-11eb-8400-2dc03f4290f3.png" alt="image"></td> <td>Example for visualizing your datasets.</td> </tr> </tbody> </table> <h2>Possible error</h2> <p>You may encounter the error, "permission denied", when running HDFS commands. To solve this error, you have to "exec" into the pod and change the access mode for the core-site.xml.</p> <p><img src="https://user-images.githubusercontent.com/72959956/124234611-d6086480-db46-11eb-849e-7d4f7a8c35e4.png" alt="image"></p> <p>You can "exec" into it through the Jupyter notebook or using the WebTerminal that comes along with HPE Ezmeral Container Platform. To grab the Kubectl credential from HPE Ezmeral Container Platform, run the following command:</p> <pre><code class="language-bash"># Bash Kernel # Grap the kubectl credential kubectl hpecp refresh ez-gateway.hpeilab.com --insecure --hpecp-user=hpecli --hpecp-pass=hpecli kubectl get pods --all-namespaces kubectl get pods --namespace=poc-tenant </code></pre> <p>Run the following command for accessing the bash of pod:</p> <pre><code class="language-bash"># 1: exec into the pod kubectl exec -it &#x3C;pod name> -- /bin/bash # example kubectl exec -it testnotebook-controller-6kq7r-0 --namespace=poc-tenant -- /bin/bash </code></pre> <p>Run the following command for changing the access mode:</p> <pre><code class="language-bash"># 2: changed the access mode for the core-site.xml chmod 666 /opt/bluedata/hadoop-2.8.5/etc/hadoop/core-site.xml </code></pre> <p>And now you can run the HDFS command without error.</p> <h1>Key takeaway</h1> <p>I hope this post offered you some tips on how to do big data analytics using the Apache Spark Python API with less time spent on setting up the environment and more time on digging out business insight from your data. Keep this notebook handy so you can refer back to it often. Also, keep an eye out on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> to make sure you catch future articles on this subject.</p>https://developer.hpe.com/2021-September-03/https://developer.hpe.com/2021-September-03/Fri, 03 Sep 2021 05:00:00 GMThttps://developer.hpe.com/discovering-hpe-dev-one-developer’s-journey/https://developer.hpe.com/discovering-hpe-dev-one-developer’s-journey/Thu, 19 Aug 2021 18:13:30 GMT<p>As a full-stack developer, I’m always looking for ways to more easily build out applications. In my exploration, I came across the <a href="https://developer.hpe.com/">HPE Developer web portal</a> (aka, the HPE DEV site). One of the first things that caught my eye was the variety of different platforms that HPE offers and supports, many of which are open source. One platform in particular, <a href="https://developer.hpe.com/platform/grommet/home/">Grommet</a>, really piqued my interest.</p> <p>As described on the site, the open source project <a href="https://v2.grommet.io/">Grommet</a> helps you create responsive and accessible mobile-first projects for the web with an <a href="https://v2.grommet.io/components">easy-to-use</a>, <a href="https://reactjs.org/">React</a>-based component library that is part design system and part development framework. While reading the Grommet page on the HPE DEV site, I discovered the HPE Design System, which helps web developers offer their website visitors with an improved user experience. The HPE Design System shows how Grommet can be themed and used to build user interfaces for your own brand. From what I understand, the HPE Design System is actually being used within HPE to guide the design of the user interfaces that HPE creates.</p> <h2>Moseying on over to the Hack Shack</h2> <p>Also, on the HPE DEV <a href="https://developer.hpe.com/platform/grommet/home/">Grommet platform page</a>, I discovered that there are a number of in-depth, hands-on technical workshops that I can take, which are found over in the <a href="/hackshack/">HPE DEV Hack Shack</a>. In the <a href="/hackshack/workshops">Workshops-on-Demand library</a>, I found an awesome course entitled <em><a href="/hackshack/workshop/14">Streamline app development with open source Grommet</a></em>. When I clicked on the Learn More button, I found a video replay of the course, which looked really interesting, so I decided to register for it. It didn’t cost me anything, and I actually found that there were real, live people at the other end of the Slack channel supporting it who were happy to help me with any questions I had. That, in and of itself, was really cool because, being developers like me, we had a lot of things to talk about! <a href="https://grommet.slack.com/">Grommet has a Slack channel</a> as well, which is highly active. Folks are making contributions there all the time.</p> <p>But I digress… back to the Hack Shack. In my meanderings about the Hack Shack, I found that there was a ton of other information there, too. The Workshops-on-Demand cover a broad range of topics of interest to all developers: API 101, Python, Kubernetes, Redfish, Rust, Git, etc. While taking my Grommet workshop, I noticed a couple of other workshops that I’m interested in taking. The ones that caught my eye are the courses on technology basics – the 101 level – especially the workshops on <a href="/hackshack/workshop/24">Kubernetes 101</a> and <a href="/hackshack/workshop/26">HPE Ezmeral Data Fabric 101</a>. I’ve heard a lot about HPE Ezmeral and I am curious to learn more.</p> <p>What really got me, though, was the <a href="/hackshack/arcade">Arcade</a> where they house the <a href="/hackshack/hackshackattack">Hack Shack Attack</a>! retro video game and offer stickers and wallpaper you can use on your own computer. It’s a really cool place, and I have to admit I spent some time trying to tackle the IT Monster.</p> <h2>Back to the HPE DEV portal</h2> <p>While my initial path took me over to all things Grommet, I wondered what else I might find on the <a href="https://developer.hpe.com/">HPE DEV portal</a>, so I headed back there. On the Grommet platform page, I was impressed to find so many blog articles on Grommet. I found <a href="https://developer.hpe.com/blog/using-grommet-with-gatsby/">Using Grommet with Gatsby</a> to be very helpful with setting up a situation where I wanted to convert a website from a create-react-app to Gatsby. After reading that one, I decided to go over to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a> and see what else they had. Like the Workshops-on-Demand, the blog site covers a variety of different topics. It even had a <a href="https://developer.hpe.com/blog/hpe-dev-launches-its-munch-learn-technical-talks/">blog post</a> on another HPE DEV offering, the <a href="https://developer.hpe.com/campaign/munch-and-learn">Munch &#x26; Learn technical talks</a>. From what I understand, the Munch &#x26; Learn sessions bring in industry experts to talk on different subjects. You can view some of the replays of these sessions on that page and register for upcoming sessions. I’m hoping they’ll have a session on Grommet soon.</p> <p>After spending some time exploring the HPE DEV portal, I decided to sign up for the <a href="https://developer.hpe.com/newsletter-signup">HPE DEV Monthly Newsletter</a>. Taking a look at their archive, I found that these newsletters highlighted some of the more interesting blogs and tutorials that were published in a given month. I figured this was the easiest way for me to stay up to date. Still, I do find that I come back to the portal often to see if I’m missing out on anything new. I also found that I could follow what’s happening on the site by following <a href="https://twitter.com/HPE_Developer">HPE DEV on Twitter</a>. I hear that sometimes HPE DEV will host coding challenges in the Hack Shack. I figured this was probably the best way to stay in touch and get any alerts on upcoming challenges. Those could be fun!</p> <p>Something else I found rather fun was actually getting this article published on their blog! I found this one post on the blog called <a href="https://developer.hpe.com/blog/be-an-hpe-dev-blogger/">Be an HPE DEV blogger!</a> I wasn’t sure if writing a post as simple as this would be something they’d find valuable, but when I checked in with the editor, I found that they were very welcoming and eager to help me get published. Who knows? I may write more posts in the future.</p> <p>I hope my journey has inspired you to see what you can find on the <a href="https://developer.hpe.com/">HPE Developer Portal</a>. There’s a ton more there that I haven’t even touched on because I was so focused on learning more about Grommet. Maybe your interests lie elsewhere. Maybe you need access to a platform-specific API or SDK. In any case, you’re sure to find something interesting. Check it out!</p>https://developer.hpe.com/meet-linux-distinguished-technologist-and-open-source-evangelist-bruno-cornec/https://developer.hpe.com/meet-linux-distinguished-technologist-and-open-source-evangelist-bruno-cornec/Wed, 18 Aug 2021 16:39:18 GMT<p>As a leading global, edge-to-cloud platform-as-a-service company, Hewlett Packard Enterprise (HPE) prides itself in employing team members who share one common purpose: to advance the way people live and work. In this blog series, you’ll get to meet a number of them as I interview some of the open source experts working with the HPE DEV team.</p> <p>Bruno Cornec has been administrating Unix systems since 1987, and Linux systems since 1993. He began his career as an engineer in Software Engineering focused on Software Configuration Management. He was responsible for nearly everything related to the product, including design, development, presales, consulting, and training.</p> <p>Bruno discovered the Free Libre Open Source Software (FLOSS) movement in 1993 thanks to Linux and has worked with it ever since, engaging in various aspects such as development, presales, technical writing, translation, consulting, training, and small technical team management. He is also involved in a number of open source projects, like <a href="http://www.mondorescue.org/">MondoRescue</a>, <a href="http://www.mageia.org/">Mageia</a>, <a href="http://www.project-builder.org/">project-builder.org</a>, <a href="https://opendev.org/x/python-redfish/">python-redfish</a>, <a href="http://www.fossology.org/">FOSSology</a>, and LinuxCOE. He is now a strong Linux and open source advocate and continues to evangelize its use in many conferences around the globe.</p> <h2>Bruno, can you tell me a little about these different open source projects that you are working on?</h2> <p>My work on open source projects all started with MondoRescue back in 2000 when I began working in what was then HP in Grenoble, France as a Linux Technologist. I was chartered with allowing our plants to preinstall Linux distributions on our servers, and I found this tool that was nearly doing everything I wanted for that purpose.</p> <p>Two or three patches later, it was ready for me to use to build the installation media for a remote team to use to preinstall Red Hat and SUSE distributions on our servers that even launched automatic tests at the end of the setup so the machines could be validated before being sent to customers.</p> <p>I worked with the upstream MondoRescue project lead to have my patches adopted in the project, and voilà, that’s how you become an upstream contributor! I then took over the maintenance of the project and created a package build environment for more than 150 different distributions. This became the Project-Builder.org project. I used my packaging knowledge to help the Mageia distribution, which I’m using daily on HPE desktops and servers, both at work and at home.</p> <p>Each time I looked at another technology, there were FLOSS projects related to it that could benefit from some contributions linked to an HPE usage. This has allowed me to work with the Fossology developer team while looking at FLOSS to tool open source governance, as well as create a Python-Redfish module to have a Python-based implementation of a Redfish (DMTF standard) library and CLI tool.</p> <h2>I understand that you’ve been collaborating with the HPE DEV team to automate the backend infrastructure that hosts our Workshops-on-Demand. Can you tell me a little more about your role in regards to this project?</h2> <p>My official role is to act as the liaison between the HPE DEV team and the World Wide Customer Innovation Center (CIC) team for which I work. We are in charge of the WW <a href="https://hpedemoportal.ext.hpe.com/">HPE demonstration portal</a>, and are always looking for content to share with our presales and partners communities. Integrating the technical and educational HPE DEV developed content into our portal was an obvious thing to do.</p> <p>My role quickly expanded, however. Before integrating the content, I helped the HPE DEV team with the automation steps I thought were required to ensure the best customer service. We worked on the set up and automation of the JupyterHub portals, using some scripts and lots of Ansible playbooks. We also automated the setup of training appliances from end-to-end to serve both as the documentation of these installations and a nice way to recover in case of any issues. As the back-end infrastructure is not directly accessible from the Internet, I introduced the team to some old-fashioned, but extremely useful, tools such as procmail to create a very functional non-REST API. Going through the firewall without security concerns using mail was, I think, one of the more interesting ideas I introduced to the HPE DEV team in parallel to their more “classical” HTTPS-based REST API development.</p> <p>Once all that was ready and working well, we could finally make the two portals interact so that, with single sign-on authentication, our presales and partner colleagues could easily consume the HPE DEV Workshops-on-Demand from our CIC portal.</p> <h2>I understand you collaborate with the HPE DEV team on other projects as well. Can you expand on that?</h2> <p>I’m also helping with the Greenlake integration. We’re designing it so the full JupyterHub and appliances stack can be instantiated at will as-a-service in Greenlake through one touch of a button. At least, that’s the goal, and we now have many pieces in place to make that successful.</p> <p>As I’m also interested in sharing knowledge, and have always been active around training either in HPE or Open Source events in the past, I’m converting <a href="https://github.com/bcornec/Labs/">my former Labs</a> into Jupyter Notebooks starting with the most popular, the Docker 101. I’m hoping this will attract more system admins and help them discover all the other Workshops-on-Demand the HPE DEV team offers. I’m a firm believer in continuing to educate oneself over the course of their career. The HPE DEV Workshops-on-Demand offer a great way to do this.</p> <h2>Speaking of training, you recently held a training class for the Campus Numérique in the Alps. I’d love to hear more about that.</h2> <p>The <a href="https://le-campus-numerique.fr/">Campus Numérique in the Alps’</a> role is to bring people back to IT jobs. They train a wide variety of students, from people without a diploma who become Webmasters all the way up to doctors in science to become future (big) data specialists (the Data-8 track). They manage 170 students across 4 sites, with the help of benevolent experts that share their knowledge, in addition to their own teachers - 100 people in total. They deliver 5 different diplomas that are recognized at the European level.</p> <p>Their teaching practice is based on experimentation first and introducing theory as they see fit. Thus, the possibility of using the HPE DEV Workshops-on-Demand in this context was particularly well-suited. I was chartered with explaining to Data-8 students the FLOSS ecosystem and the notion of contribution, licenses, etc.</p> <p>Before working on a concrete example of code contribution, I had the students refresh their Git knowledge of using Git as a base tool for project contribution using the <a href="/hackshack/workshop/17">Git 101 Workshop-on-Demand</a>. I also used the <a href="/hackshack/workshop/9">API 101 Workshop-on-Demand</a> to teach them what an API is and how they could interact with a tool providing one. As the <a href="/hackshack/">Hack Shack portal</a> is easily accessible to everybody, I used it to support my training sessions.</p> <h2>Is there anything else you’d like to share with our readers?</h2> <p>What is great about FLOSS is that, even 28 years after having started looking at it, you continue to learn. The ability to have such easy access to everything from code, to docs, to training materials, empowers you incredibly. My next step with the HPE DEV team is to make the contents we develop, and later on, the platform itself, available under a FLOSS license in order to give back for what I got out of it for all these years. Many thanks to HPE for supporting <a href="https://www.hpe.com/us/en/open-source.html">our FLOSS contributions</a> with our Open Source Program Office!</p> <p>And all that can only continue if you, dear readers, go on sharing your knowledge and code so we can all capitalize on it to always build even greater pieces of software. As said on the MondoRescue website: “All rights reversed”.</p> <p>To learn more about the open source projects that HPE is involved with, please visit our website. Interested in exploring what HPE offers for developers and data scientists? Check out our <a href="https://developer.hpe.com/">HPE DEV site</a> for a ton of articles, workshops, tutorials, and other resources.</p>https://developer.hpe.com/2021-August-02/https://developer.hpe.com/2021-August-02/Tue, 03 Aug 2021 05:00:00 GMThttps://developer.hpe.com/getting-started-with-ilo-restful-api-redfish-api-conformance/https://developer.hpe.com/getting-started-with-ilo-restful-api-redfish-api-conformance/Tue, 20 Jul 2021 18:09:08 GMT<h3>Updated July 25, 2023</h3> <h1><strong>A primer for coders</strong></h1> <h2>Introduction</h2> <p>With the introduction of iLO 4 2.00 on ProLiant Gen9 servers, we introduced our next generation programmatic interface for server management. HPE iLO has a rich history of remote management capabilities including IPMI, SNMP and the XML scripting language used with HPQONCFG. The need for a new API was so obvious to us that we also began an effort with the DMTF to create a standard around it, which eventually emerged in August 2015 as the “Redfish API”. The fundamental features of the API were quickly agreed upon by the participants, but as always happens in standards bodies, what emerged had some details changed. The iLO 2.30 release in September 2015 was the beginning of convergence with the Redfish standard. The release included Redfish properties (including the newly introduced <code>@odata</code> meta-properties) as well as the compatible pre-Redfish data model. As we move forward with our HPE Gen10 Servers, iLO 4 and iLO 5 will follow Redfish mode and we encourage our customers to code to the Redfish standard and move away from any pre-Redfish implementation. If you are just now beginning to look at leveraging the Redfish API, you should make sure your client code is interacting with iLO using Redfish API standards:</p> <ul> <li> <p>Read the <a href="http://www.dmtf.org/standards/redfish">Redfish specification</a>. Make sure your assumptions about service URIs, including the starting URI, do not exceed the specification’s guarantees.</p> </li> <li> <p>Include the HTTP header `"OData-Version": "4.0"' in all HTTP requests. This causes iLO 4 2.30 to hide pre-Redfish properties, decreasing your chance of inadvertently creating a dependency on something that will be removed in the future.</p> </li> <li> <p>For iLO 5, we updated the HPE branding on the OEM extension properties, so some updates might be needed from your iLO 4 scripts if you have already integrated with the API. For detailed information of the differences between iLO 4 and iLO 5 visit our <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo5/ilo5_adaptation/">documentation</a>.</p> </li> </ul> <p>Differences between HPE iLO 5 and iLO 6 are also listed in the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_adaptation/">documentation</a></p> <h2>Redfish Versioning</h2> <p>Redfish and its release process is designed to allow the data model to develop quickly as needed by the industry while limiting protocol and specification changes to very infrequent updates.  Data model and protocol are both versioned independently and move at their own pace. The goal of the design is to ensure the data model can be adapted easily to future implementations, as new technologies need to be represented, without having to revise the specification repeatedly. This is one reason you do not find many specified URIs in the specification.</p> <h2>Redfish URIs – Writing Durable Client Code</h2> <p>Keeping with the vision of infrequent specification and protocol updates, the API is designed with a self-describing data model with each resource containing its own type and version information and links to other resources. Redfish is a "hypermedia API" by design which means that the navigation of the data model is built into the data itself rather than defined by specification. Instead of a published list of resource URIs, a client discovers the data model dynamically by following links between resources. This is to avoid building in restrictive assumptions to the data model that will make it difficult to adapt to future hardware implementations.</p> <p>A URI should be treated by the client as opaque. A client should not attempt to deconstruct URIs into a template pattern. It is entirely legitimate that a resource at <code>/redfish/v1/systems/1</code> could contain a link to <code>/arbitrary/stuff</code>. Only specific top level URIs documented in the specification may be assumed, and even these may be absent based upon the implementation. The other URIs must be discovered dynamically by following <code>@odata.id</code> links contained in the data model, which point to other resources.</p> <p>For example, iLO on an HPE ProLiant DL360 server has one compute node, and we happen to give the resource that describes that compute node the URI <code>/redfish/v1/systems/1/</code> but there are implementations in the industry, and even the demonstration mockups, that use alternate paths including things like system serial number as the “leaf” of the URI (e.g. <code>/redfish/v1/systems/</code>). If you assume too much about the URIs you will discover that your client code is not portable across various implementations of Redfish.</p> <h2>Traversing the Resource Model</h2> <p>Because objects link together, there are some best practices you should be aware of as you create new client code. If you create a generic “crawler” app that simply GETs every resource and follows its links, your crawl will never terminate because the various object interlinks mean that the data model is not strictly a tree, but a graph.  A generic crawler must keep track of visited URIs and not re-crawl them.  Additionally, as best practice you should treat the visited resource URIs set as case insensitive (iLO does). Most use cases are not generic crawls so this won’t be an issue. Typically you know what you want to find in the data model and you should make sure you correctly find and iterate the collections needed to get to the specific resource you are interested in. To explore a data model demo visit our <a href="https://ilorestfulapiexplorer.ext.hpe.com/">iLO RESTful API Demo</a>.</p> <h2>Interoperability</h2> <p>The good news is that our internal tools at HPE required very little tweaking to work correctly on another industry implementation of Redfish. If you read the specification and understand the principles behind the design choices you have a very good chance of writing durable client code that is widely interoperable.</p> <p>HPE Developers have additional resources that can help you understand and integrate more effectively with the iLO RESTful API. To get the latest libraries and sample code available visit the <a href="https://developer.hpe.com/platform/ilo-restful-api/home/">iLO RESTful API</a> platform page.</p>https://developer.hpe.com/getting-started-with-the-redfish-api-part-2/https://developer.hpe.com/getting-started-with-the-redfish-api-part-2/Tue, 20 Jul 2021 17:55:56 GMT<h3>Updated July 25, 2023</h3> <h1><strong>A primer for coders</strong></h1> <p>In my last blog <a href="/blog/getting-started-with-ilo-restful-api-redfish-api-conformance">post</a> I began a discussion about best practices for writing Redfish API client code. Last time we talked about resource versioning and resource inter-linking. I explained that client code should discover the data model and avoid making incorrect assumptions. In this post I will continue discussing some issues you should be aware of in order to create durable clients that interoperate across different implementations of the Redfish API.</p> <h2>HTTP Status Codes and Redirect</h2> <p>HTTP requests to any REST API return an HTTP status code as part of the response. When writing Redfish client code you should encounter and handle any status codes defined in the Redfish specification. In addition, iLO uses <code>308 Redirect</code> on some URIs to redirect the client from an older version of the API to the newer version. For instance, iLO 4 responded to GET <code>/rest/v1</code>. If you try this with iLO 5, you will receive an HTTP <code>308 Redirect</code> to <code>/redfish/v1/</code>.</p> <h2>Unimplemented Properties</h2> <p>Redfish is designed to be extremely flexible in implementation which is why so much emphasis is placed upon a self-describing data model (<code>@odata</code> meta-data and links to related resources). Part of this flexibility is that properties within a resource may be omitted if not supported by an implementation. For instance, a server that does not implement an indicator LED may completely omit the <code>IndicatorLED</code> property in the data model. Likewise, entire resources and the links to them may vary between implementations. Good client code should handle missing properties and links in whatever way is appropriate for your application.</p> <h2>Null values for Properties</h2> <p>If you examine the <a href="http://redfish.dmtf.org/schemas/v1/">Redfish schema</a> documents, you may notice that many properties are allowed to return a JSON null value. As an implementer of the API, returning null is not ideal but because of various relationships between open-architecture server components, the value of a property may not be available at all times. The iLO RESTful API uses null as a return value when, due to system state, the value is either not yet available or too stale to be of use. Again, good client code should test the value.</p> <h2>What Version of Redfish does iLO support?</h2> <p>We are often asked the question: “What version of Redfish does iLO support?” Due to the flexibility of the API, the question is not straightforward to answer. Recall from <a href="/blog/getting-started-with-ilo-restful-api-redfish-api-conformance">part 1</a> that the “Protocol” is versioned separately from the “Data Model”. iLO 5 implements the latest protocol version 1.3 at the time of writing. However, the individual resources in the data model each implement a specific schema and version. Some resources may report the latest version while some may report older versions. Since the versions are cumulative, if we need to report a newly defined property we will update the version to the schema that defines the new property. However, if we do not need to update a resource, we may choose to leave the versions information unchanged as well. Since schema updates are likely to occur on each iLO firmware update, the best answer is to inspect the API dynamically.</p> <h2>What is next?</h2> <p>We hope we have given you enough information to get started with the iLO RESTful API and see the benefits of server automation from a single API. As we continue to improve and update our Redfish interface, please refer to our <a href="https://servermanagementportal.ext.hpe.com/">documentation</a>.</p>https://developer.hpe.com/using-docker-wrong-my-journey-to-a-better-container/https://developer.hpe.com/using-docker-wrong-my-journey-to-a-better-container/Tue, 13 Jul 2021 06:03:11 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code class="language-markdown">"authorDisplayName": "John Omernik", "publish": "2018-06-07T13:00:00.000", "category": "use-case", </code></pre> <hr> <p>Since starting at MapR, I’ve been focusing on a number of things, including learning more about MapR, meeting coworkers and customers, and a topic near and dear to my heart: immersing myself in containers and their integration with the MapR Data Platform.  In past jobs, I’ve used MapR with containers but have not dug into the Persistent Application Client Container (PACC) nearly as much as I should.  This cautionary tale is one to remind everyone how amazing containers are AND how to keep the benefits of containers at the top of your mind when working to create them, so you don’t design your containers in such a way that you give up those benefits and end up writing a mea culpa blog post like this one.  </p> <p><img src="/img/container-wide.jpg" alt=""></p> <p>I was tasked to come up with a way to have a syslog server receive log messages and produce those messages to MapR Event Store – a powerful setup for Information Security and IT Operations professions.  I had a few requirements:</p> <ol> <li>Must run in Docker.  This is now a self-imposed philosophy for all things.  I want everything to be portable, sharable, and usable by everyone on the MapR Data Platform.</li> <li>Simple in and out.  It has to listen on a syslog port (TCP and UDP) and produce to MapR Event Store.  I wanted to show how it works in the Dockerfile, but I didn’t want to chain multiple containers together.</li> </ol> <p>After some research I decided to utilize a great MapR partner, <a href="https://streamsets.com/">StreamSets</a>, and their <a href="https://streamsets.com/products/sdc">StreamSets Data Collector</a> (SDC) as the core.  The SDC’s interface, capabilities, and integrations with MapR would serve me well here.  I had worked with StreamSets before in Docker with MapR; however, I had NOT used the MapR PACC, so it would be a learning experience.  I pulled out my old Dockerfiles, modified some components, ensured I was handling my newly secured cluster with tickets, and it worked great!  I even produced a script to create the MapR Event Store and generate test messages, all contained in one handy <a href="https://github.com/johnomernik/maprssss_wrong">GitHub repository</a>.</p> <p>As you can see by the title of this post, I did it wrong. Now, most folks would just update their code once they realized their mistake; however, I created a second repository because I wanted to share what I did wrong as an example to others.  What did I do wrong?  I did not create a deterministic container (<a href="https://en.wikipedia.org/wiki/Deterministic_system">https://en.wikipedia.org/wiki/Deterministic_system</a>).  </p> <p>One of the primary benefits of Docker should be that, if I create a Docker container, I will be able run it on all my systems, and if I hand that container to someone else, it should run the same way, thereby being a deterministic system. Now, let me say there are always caveats, but this should be a primary goal for Docker users.</p> <p>Where did I go wrong? Well, my main issue was using my old code from before the MapR PACC. In this code, I took a number of directories for StreamSets, including logs, conf (config), and others, and placed it on the local system.  When I ran this on a MapR node in my cluster, it worked great, because all the users that I would use inside the container were also on my MapR nodes running Docker. Thus, I could set up permissions on those local directories, and when the container ran, the user inside the container had proper filesystem permissions to the directories required by StreamSets.</p> <p><img src="/img/docker-host.png" alt=""></p> <p>However, when I shared my repository with a colleague at MapR, they reported it didn’t work on their system. I was baffled: it should work; it’s Docker! That’s the whole point of using containers. Well, my colleague was not running the container on MapR nodes; instead, it was running on a client machine. This client machine did NOT have the same users as the MapR cluster. Thus there were a number of issues that my scripts ran into with creating the configuration directories, settings permissions, and even running StreamSets.  I made assumptions in my original Docker, and by doing so, made a container that ran differently depending on the execution system.</p> <p>This was an eye-opener for me.  From one perspective, I had tried to get StreamSets running in the PACC by using old knowledge and approaches in an effort to expedite the process. By doing so, I hampered the usability, portability, and shareability of my container. Because of this, I’ve created another <a href="https://github.com/johnomernik/maprssss">GitHub repository</a>.</p> <p>Here is my first attempt to “get it right.”  Instead of volume mounting the configuration directories for StreamSets to the local filesystem, I use the MapR POSIX Client built into the PACC to put the configuration directly on MapR XD.  This ensures that the user in the container always has access and that the config files are available to StreamSets anywhere they run.  </p> <p><img src="/img/docker-host-2.png" alt=""></p> <p>This allowed me to simplify my startup scripts and ensure a much cleaner runtime environment, independent of the system I was running the container on. This is a much better approach and really shows the power of using MapR with Docker containers.  I am still learning more about the PACC, but as I learn, I wanted to take the time to share my follies, so others can learn as well.  If you take the time to review my repository <a href="https://github.com/johnomernik/maprssss">here</a> and find ways to improve them even more, I am all ears! Leave a comment, or post a GitHub issue.</p> <h2>Related Links:</h2> <ul> <li><a href="https://docs.datafabric.hpe.com/62/AdvancedInstallation/RunningtheMapRPACC.html">Running the MapR PACC Using Docker</a></li> </ul>https://developer.hpe.com/comparing-to-kill-a-mockingbird-to-its-sequel-with-apache-spark/https://developer.hpe.com/comparing-to-kill-a-mockingbird-to-its-sequel-with-apache-spark/Tue, 13 Jul 2021 05:38:30 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code class="language-markdown">"authorDisplayName": "Joseph Blue", "publish": "2015-08-05T07:00:00.000Z", "category": "apache-spark", </code></pre> <hr> <h2>Courting Controversy</h2> <p>Did Harper Lee write <em>To Kill a Mockingbird</em>? For many years, conspiracy buffs supported the urban legend that Truman Capote, Lee’s close friend with considerably more literary creds, might have ghost-authored the novel. The author’s reticence on that subject (as well as every other subject) fueled the rumors and it became another urban legend.</p> <p>However, the recent ‘discovery’ and subsequent publication of her earlier novel <em>Go Set a Watchmen</em> has generated renewed scrutiny of the chain of events. Is the newly published book a discarded rough draft that was to become the universally beloved classic, or was it a truly forgotten separate work that deserves to be cast in the literary limelight for analysis? A concise summary of the publishing controversy can be found in this NYT <a target='\_blank' href='http://www.nytimes.com/2015/07/25/opinion/joe-nocera-the-watchman-fraud.html'>op-ed-column</a>.</p> <p>But the new book offers curious readers an opportunity to analyze the two works together with machine learning tools that are ideal for classifying text among a corpus of documents. Apache Spark has a mature set of libraries for text-based analysis that can be leveraged with very few lines of code.</p> <p>The publisher of <em>Go Set a Watchman</em> is unlikely to make available their best seller even for lofty academic purposes. Luckily, the Wall Street Journal printed the <a target='\_blank' href='http://www.wsj.com/articles/harper-lees-go-set-a-watchman-read-the-first-chapter-1436500861'>first chapter</a> on July 10th for anyone to analyze. In this blog, we extract features from the first chapter of each book, and then build a model to tell the difference between them. Comparing passages from each may provide clues as to the authorship.</p> <p>All of the data and code to train the models and make your own conclusions using Apache Spark is located in this <a target='\_blank' href='https://github.com/joebluems/Mockingbird'>Github repository</a>.</p> <h2>Dissecting a Classic by the Numbers</h2> <p>The theory behind document classification is that text from the same source will contain similar combinations of words with comparable frequency. Any conclusions based from this type of analysis are only as strong as that assumption.</p> <p>To build a model to classify documents, text must be translated into numbers. This involves standardizing the text, converting to numbers (via hashing) then adjusting the word importance based on its relative frequency.</p> <p>Text standardization was done with Apache Lucene. An example below shows how to perform this with the Spark shell:</p> <pre><code class="language-scala">> ./bin/spark-shell --packages "org.apache.lucene:lucene-analyzers-common:5.1.0" val line="Flick. A tiny, almost invisible movement, and the house was still." val tokens=Stemmer.tokenize(line) tokens: Seq[String] = ArrayBuffer(flick, tini, almost, invis, movement, hous, still) </code></pre> <p>The Stemmer object that invokes the Lucene analyzer comes from <a target='\_blank' href='https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/'>this Chimpler example</a>. Notice how the line describing the tranquility of the Radley house is affected. The punctuation and capitalization is removed, and words like “house” are stemmed, so tokens with the same root (“housing”, “housed”, etc.) will be considered equal. Next, we translate those tokens into numbers and count how often they appear in each line. Spark’s HashingTF library performs both operations simultaneously.</p> <pre><code class="language-scala">import org.apache.spark.mllib.feature.HashingTF val tf = new HashingTF(10) tf: org.apache.spark.mllib.feature.HashingTF = org.apache.spark.mllib.feature.HashingTF@d32d034 val hashed = tf.transform(tokens) hashed: org.apache.spark.mllib.linalg.Vector = (10,[0,1,2,3,6,7],[1.0,1.0,1.0,2.0,1.0,1.0]) </code></pre> <p>A “hash” is a one-way translation from text to an integer (i.e. once it’s translated, there’s no way to go back). Initializing the hash with HashingTF(10) notifies Spark we want every string mapped to the integers 0-9. The transform method performs the hash on each word, and then provides the frequency count for each. This is an impractical illustration and would result in a huge number of “collisions” (different strings assigned the same number).</p> <p>The default size of the resulting Vector of token frequencies is 1,000,000. The size and number of collisions are inversely related. But a large hash also requires more memory. If your corpus contains millions of documents, this is an important factor to consider. For this analysis, a hash size of 10,000 was used.</p> <p>The last step in the text-preparation process is to account for the rareness of words- we want to reward uncommon words such as “chifferobe” with more importance than frequent words such as “house” or “brother”. This is referred to as TF-IDF transformation and is available as an (almost) one-liner in Spark.</p> <pre><code class="language-scala">import org.apache.spark.mllib.feature.IDF val idfModel = new IDF(minDocFreq = 3).fit(trainDocs) val idfs = idfModel.transform(hashed) idfs: org.apache.spark.mllib.linalg.Vector = (10,[0,1,2,3,6,7],[0.413734499590671,0.4244680552337798,0.4761400657781007, 1.4004620708967006,0.37876590175292424,0.48374466516332]) </code></pre> <p>The “fit” method of the IDF library examines the entire corpus to tabulate the document count for each word. On the second pass, Spark creates the TF-IDF for each non-zero element (tokeni) as the following:</p> <p><img src="/img/mockingbird-blog-figa.png" alt=""></p> <p>A corpus of many documents is needed to create an IDF dictionary, so in the example above, excerpts from both novels were fed into the fit method. The transform method was then used to convert individual passages to TF-IDF vectors.</p> <p>Having been transformed into TF-IDF vectors, passages from both books are now ready to be classified.</p> <h2>Building the Classifier</h2> <p>The secret to getting value from business problems is not the classification; it is primarily about ranking objects based on the confidence of our decision and then leveraging the value of a good decision minus the cost of a misidentification. Spark has several machine learning algorithms that are appropriate for this task.</p> <p>During examination of the text it was noted that a few modifications should be made to the novels to make the comparison more “fair”. <em>To Kill a Mockingbird</em> was written in the first person and includes many pronouns that would be giveaways (e.g. “I”,”our”,”my”,”we”, etc.). These were removed from both books. Due to the inevitability of variable sentence length in novels, passages were created as a series of ten consecutive words.</p> <p>The parsed passages were combined, split into training and testing sets, and then transformed with the idfModel built on the training data using the code below:</p> <pre><code class="language-scala">val data = mockData.union(watchData) val splits = data.randomSplit(Array(0.7, 0.3)) val trainDocs = splits(0).map{ x=>x.features} val idfModel = new IDF(minDocFreq = 3).fit(trainDocs) val train = splits(0).map{ point=> LabeledPoint(point.label,idfModel.transform(point.features)) } val test = splits(1).map{ point=> LabeledPoint(point.label,idfModel.transform(point.features)) } train.cache() </code></pre> <p>Using randomly split data files for training and testing a model is standard procedure for insuring that performance is not a result of over-training (i.e. memorizing the specific examples instead of abstracting the true patterns). It is critical that the idfModel is built only on the training data. Failure to do so may result in over-stating your performance on the test data.</p> <p>The data is prepared for machine learning algorithms in Spark. Naïve Bayes is a reasonable first choice for document classification. The code below shows the training and evaluation of a Naïve Bayes model on the passages.</p> <pre><code class="language-scala">import org.apache.spark.mllib.classification.{NaiveBayes, NaiveBayesModel} val nbmodel = NaiveBayes.train(train, lambda = 1.0) val bayesTrain = train.map(p => (nbmodel.predict(p.features), p.label)) val bayesTest = test.map(p => (nbmodel.predict(p.features), p.label)) println("Mean Naive Bayes performance") (bayesTrain.filter(x => x._1 == x._2).count() / bayesTrain.count().toDouble, bayesTest.filter(x => x._1 == x._2).count() / bayesTest.count().toDouble) Mean Naive Bayes performance res25: (Double, Double) = (0.9053398058252428,0.7068965517241379) </code></pre> <p>Applying the Naïve Bayes algorithm in Spark gives a classification from which accuracy and a confusion matrix can be derived. The method makes the correct classification on 90.5% of the train records and 70.7% of the test records (performance on the training is almost always better than the test). The confusion matrix on the test data appears below:</p> <p><img src="/img/mockingbird-blog-fig1.png" alt=""></p> <p>The diagonal elements of the confusion matrix represent correct classifications and the off-diagonal counts are classification errors. It is informative to look at a confusion matrix (especially when there are more than two classes) – the better the classification rate on the test set, the more separable the populations. However, when data scientists are looking to apply classification to a business problem, they prefer to examine how well the algorithm rank-orders the results.</p> <p>Currently, Spark does not support a user-supplied threshold for Naïve Bayes. Only the best classification rate in the training data is reported. But in real business problems, there is an overhead associated with a misclassification so that the <em>“best” rate may not be the optimal rate</em>. It is of keen interest to the business to find the point at which maximum value of correct classifications is realized when accounting for incorrect answers. To do this via Spark, we need to use methods that allow for analysis of the threshold.</p> <p>Given the number of features (a TF-IDF vector of size 10,000) and the nature of the data, Spark’s tree-based ensemble methods are appropriate. Both Random Forest and Gradient Boosted Trees are available.</p> <pre><code class="language-scala">import org.apache.spark.mllib.tree.RandomForest import org.apache.spark.mllib.tree.model.RandomForestModel import org.apache.spark.mllib.tree.GradientBoostedTrees import org.apache.spark.mllib.tree.configuration.BoostingStrategy import org.apache.spark.mllib.tree.model.GradientBoostedTreesModel // RANDOM FOREST REGRESSION val categoricalFeaturesInfo = Map[Int, Int]() val numClasses = 2 val featureSubsetStrategy = "auto" val impurity = "variance" val maxDepth = 10 val maxBins = 32 val numTrees = 50 val modelRF = RandomForest.trainRegressor(train, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins) // GRADIENT BOOSTED TREES REGRESSION val boostingStrategy = BoostingStrategy.defaultParams("Regression") boostingStrategy.numIterations = 50 boostingStrategy.treeStrategy.maxDepth = 5 boostingStrategy.treeStrategy.categoricalFeaturesInfo = Map[Int, Int]() val modelGB = GradientBoostedTrees.train(train, boostingStrategy) </code></pre> <p>The regression model options (estimating vs. classifying) will produce continuous outputs that can be used to find the right threshold. Both of these methods can be configured with tree depth and number of trees– read the Spark <a target='\_blank' href='http://spark.apache.org/docs/1.3.0/mllib-ensembles.html'>documentation</a> for details, but general rules of thumb are the following:</p> <ul> <li>Random Forest – trees are built in parallel and overtraining decreases with more trees, so setting this number to be large is a great way to leverage a Hadoop environment. The max depth should be larger than Gradient Boosted Trees (GBT).</li> <li>Gradient Boosted Trees – the number of trees is directly related to overtraining and the trees are not built in parallel. This method can produce some extremely high classification rates on the training data, but set the max depth of trees to be smaller than random forest.</li> </ul> <p>The table below shows the commands to calculate the Receiver Operating Characteristic (ROC) for the Random Forest model – the ROC will tell the real story on the model performance.</p> <pre><code class="language-scala">//// Random forest model metrics on training data val trainScores = train.map { point => val prediction = modelRF.predict(point.features) (prediction, point.label) //// Random forest model metrics on training data val trainScores = train.map { point => val prediction = modelRF.predict(point.features) (prediction, point.label) } val metricsTrain = new BinaryClassificationMetrics(trainScores,100) val trainroc= metricsTrain.roc() trainroc.saveAsTextFile("/ROC/rftrain") metricsTrain.areaUnderROC() res11: Double = 0.9811325444963708 //// Random forest model metrics on test data val testScores = test.map { point => val prediction = modelRF.predict(point.features) (prediction, point.label) } val metricsTest = new BinaryClassificationMetrics(testScores,100) val testroc= metricsTest.roc() testroc.saveAsTextFile("/ROC/rftest") metricsTest.areaUnderROC() res12: Double = 0.8844304733727815 </code></pre> <p>To calculate an ROC, the following steps are performed:</p> <ol> <li>Results are binned according to score (highest to lowest).</li> <li>In each bin, the number of each class is tabulated (Mockingbird vs Watchman passages).</li> <li>Starting with the highest bin, generate a data point containing the cumulative percent of the total Mockingbird and Watchman passages that have occurred.</li> </ol> <p>Graphing those points for the Random Forest and Gradient Boosted Trees yields the following curves:</p> <p><img src="/img/mockingbird-blog2.png" alt=""></p> <p>The diagonal “baseline” is the performance one could expect from random guessing (i.e. selecting 50% of the passages, you would expect to find half of each book’s examples). Any performance better than that is considered the “lift” delivered by the model. It should be intuitive from examining the graph that steeper, higher curves provide greater lift. The table below quantifies the area under the ROC, which is a standard metric used by data scientists to evaluate the performance of many models simultaneously.</p> <p><img src="/img/mockingbird-blog-fig3.png" alt=""></p> <p>The Gradient Boosted Tree model achieved an essentially perfect 1.0 area under the curve. This implies that the model scored all Mockingbird passages higher than all Watchman passages. However, the Random Forest model has higher performance on the test set (0.884 vs 0.867) so it is assumed to generalize better.</p> <p>In the setting of a business problem, the underlying data of the ROC is used to estimate how many items of interest can be identified when the real cost of an error is considered. Focusing on the highest scoring items from the model and working down the list is where real value comes from.</p> <p>The results cannot be interpreted as conclusive, but there is significant lift displayed on these curves, and that doesn’t look good for Harper Lee.</p> <h2>The Verdict</h2> <p>There are plenty of great tools to build classification models. Apache Spark provides an excellent framework for building solutions to business problems that can extract value from massive, distributed files.</p> <p>Machine learning algorithms cannot answer the great mysteries of life. But they do provide evidence for humans to consider when interpreting results, assuming we ask the right question in the first place.</p> <p>Readers are encouraged to check out the books themselves and reach their own conclusions. If the controversy surrounding the publication of Harper Lee’s books causes more people to read them, that’s probably a good thing.</p>https://developer.hpe.com/2021-July-6/https://developer.hpe.com/2021-July-6/Wed, 07 Jul 2021 06:00:00 GMThttps://developer.hpe.com/accessing-dtap-in-pods/https://developer.hpe.com/accessing-dtap-in-pods/Tue, 06 Jul 2021 06:44:24 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <h2>What is DataTap?</h2> <p><img src="https://user-images.githubusercontent.com/72959956/126251305-e100faf1-aac5-410b-8c67-cb7cdd01a50b.png" alt="image"></p> <p>Handling different protocols of file systems is always a pain for a data analyst. DataTap is a file system connector that aims to alleviate this pain. DataTap provides HDFS protocol abstraction that allows big data applications like Spark to run unmodified with fast access to data sources other than HDFS, i.e. HPE Ezmeral Data Fabric XD (formerly named MapR-FS/XD) and GCS (Google Cloud Storage). Using DataTap, you can unify your code while the underlying data sources can be swapped from HDFS, MapR-FS. This flexibility allows developers like you to focus more on coding rather than the underlying infrastructure. More information on DataTap can be found <a href="https://docs.containerplatform.hpe.com/53/reference/universal-concepts/About_DataTaps.html">here</a>.</p> <p>In this blog, I will introduce two ways to access DataTaps in Kubernetes clusters managed by HPE Ezmeral Container Platform deployed with a pre-integrated HPE Ezmeral Data Fabric. The first method covers how to access the DataTaps using HDFS Commands and the second focuses on directly reading data from Apache Spark (using pyspark). Here we go!</p> <h2>Enable DataTap when creating KubeDirector App</h2> <p>First and foremost, you have to enable DataTaps while creating a KubeDirector app. This can be done by ticking the "Enable DataTap" box.</p> <p><img src="https://user-images.githubusercontent.com/72959956/119443704-9cc92180-bd5c-11eb-8fce-b6b53823336c.png" alt="image"></p> <p>This will result in mounting a lot of files to <code>/opt/bdfs/</code> of your pod. If you can see the files in your pod (as shown in the image below), it means that your pod is DataTap enabled and you are now ready to access the files in DataTap.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120776952-58593500-c557-11eb-9dcd-4146d581a761.png" alt="image"></p> <p>The generic approach can be summarized into these two steps:</p> <ol> <li>Add <code>/opt/bdfs/bluedata-dtap.jar</code> to the classpath.</li> <li>Configure Hadoop with the following values.</li> </ol> <table> <thead> <tr> <th>name</th> <th>value</th> </tr> </thead> <tbody> <tr> <td>fs.dtap.impl</td> <td>com.bluedata.hadoop.bdfs.Bdfs</td> </tr> <tr> <td>fs.AbstractFileSystem.dtap.impl</td> <td>com.bluedata.hadoop.bdfs.BdAbstractFS</td> </tr> <tr> <td>fs.dtap.impl.disable.cache</td> <td>false</td> </tr> </tbody> </table> <p>Note: fs.dtap.impl.disable.cache can be designated as an option.</p> <blockquote> <p>Reference: <a href="https://docs.containerplatform.hpe.com/53/reference/kubernetes/tenant-project-administration/datataps/Accessing_DataTaps_in_Kubernetes_Pods.html">Accessing DataTaps in Kubernetes Pods</a></p> </blockquote> <h2>Uniform Resource Identifier</h2> <p>In HPE Ezmeral Container Platform, you can see different types of file systems used by the shared storage resources. You can manage different data sources through a GUI while representing files with the same URI. The URI will be in the format of</p> <pre><code>dtap://datatap_name/some_subdirectory/another_subdirectory/some_file </code></pre> <table> <thead> <tr> <th>Screenshot</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/121467168-35150680-c9eb-11eb-901c-77e83097cdf9.png" alt="image"></td> <td>You can manage different data source whether they are in MapR filesystem or HDFS.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/126249359-0a192c2e-6dbf-4c22-b923-94b230cc1215.png" alt="image"></td> <td>You can add new DataTap with this screen.</td> </tr> <tr> <td><img src="https://user-images.githubusercontent.com/72959956/121467262-5f66c400-c9eb-11eb-958d-911f18281a27.png" alt="image"></td> <td>You can upload, delete or rename files using GUI.</td> </tr> </tbody> </table> <h1>Access DataTaps using HDFS commands</h1> <h2>Introduction</h2> <p>The Hadoop distributed file system (HDFS) is the key component of the Hadoop ecosystem. HDFS commands, of course, are the commands that are responsible for manipulating files for HDFS.</p> <p>To use the HDFS commands, first you need to start the Hadoop services using the following steps:</p> <h2>Prepare Hadoop</h2> <p>Some of the KubeDirector App provided by HPE is pre-installed a well-configured Hadoop for you. Hence, the following installation steps can be skipped.</p> <h3>Install OpenJDK and the dependency</h3> <pre><code class="language-bash">apt update &#x26;&#x26; apt upgrade -y apt install wget -y # install openjdk DEBIAN_FRONTEND=noninteractive apt-get install openjdk-11-jdk-headless -y </code></pre> <h3>Download Hadoop and untar Hadoop</h3> <p>You can always find the latest version of Hadoop on <a href="https://hadoop.apache.org/releases.html">Apache Hadoop Releases</a>.</p> <pre><code class="language-bash">wget https://apache.website-solution.net/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz # Download Hadoop binary tar zxf hadoop-*.tar.gz # Untar Hadoop binary mv hadoop-3.3.0 $HOME/hadoop # Rename and move Hadoop folder to $HOME cd $HOME/hadoop # Move directory to hadoop </code></pre> <h3>Configure the required environment</h3> <p>In <code>$HADOOP_HOME/etc/hadoop/hadoop-env.sh</code> file, assign the following environment variables (<code>$JAVA_HOME</code>, <code>$HADOOP_HOME</code>, <code>$HADOOP_CLASSPATH</code>):</p> <pre><code class="language-bash"># These two variables is needed for HDFS command. Located at line 54, 58. export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/ export HADOOP_HOME=$HOME/hadoop # This variable is DataTap specific. Located at line 126. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/lib/:/opt/bdfs/bluedata-dtap.jar </code></pre> <p>In <code>$HADOOP_HOME/etc/hadoop/core-site.xml</code> file, configure Hadoop with the following values:</p> <pre><code class="language-xml">&#x3C;configuration> &#x3C;property> &#x3C;name>fs.dtap.impl&#x3C;/name> &#x3C;value>com.bluedata.hadoop.bdfs.Bdfs&#x3C;/value> &#x3C;/property> &#x3C;property> &#x3C;name>fs.AbstractFileSystem.dtap.impl&#x3C;/name> &#x3C;value>com.bluedata.hadoop.bdfs.BdAbstractFS&#x3C;/value> &#x3C;/property> &#x3C;property> &#x3C;name>fs.dtap.impl.disable.cache&#x3C;/name> &#x3C;value>false&#x3C;/value> &#x3C;/property> &#x3C;/configuration> </code></pre> <h3>Alternative</h3> <p>I have prepared an example configuration file on <a href="https://github.com/helloezmeral/hpe-binary/tree/main/hadoop-dtap-config">Github</a>. If your Hadoop does not have a special configuration, you can simply download and replace your existing configuration file.</p> <h2>Test your HDFS command</h2> <p>Here are some common commands used to interact with DataTap.</p> <pre><code class="language-bash"># bash: Current working directory -> $HADOOP_HOME # Check the version of the Hadoop. bin/hadoop version # List the files from the default TenantStorage Data Source. bin/hdfs dfs -ls dtap://TenantStorage/ # Make new directory user in dtap://TenantStorage/. bin/hdfs dfs -mkdir dtap://TenantStorage/user # Move the text files helloworld.txt to "cenz" folder. bin/hdfs dfs -put helloworld.txt dtap://TenantStorage/cenz bin/hdfs dfs -put -f helloworld.txt dtap://TenantStorage/cenz # force replacement # Concatenate a file in dtap. bin/hdfs dfs -cat dtap://TenantStorage/cenz/helloworld.txt # Remove a file in dtap. bin/hdfs dfs -rm dtap://TenantStorage/cenz/helloworld.txt </code></pre> <blockquote> <p>Tip:</p> <p>To get rid of the file path <code>bin/</code>, we can add the Hadoop's <code>bin</code> and <code>sbin</code> file to <code>$PATH</code></p> <pre><code>export HADOOP_HOME=$HOME/hadoop export PATH=$PATH:$HADOOP_HOME:$HADOOP_HOME/sbin:$HADOOP_HOME/bin </code></pre> <p>Reference: <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html">Hadoop File System Shell Document</a></p> </blockquote> <h1>Access DataTaps using pyspark</h1> <h2>Introduction</h2> <p>PySpark is an interface for Apache Spark in Python. Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing. Apache Spark can access data from HDFS and, with the extension, file systems managed by DataTap.</p> <h2>Install pyspark</h2> <p>There are lots of ways to install Spark. The simplest way is to install the pyspark package directly using <code>pip install pyspark</code>. Run the following to install the prerequisite packages and pyspark.</p> <pre><code class="language-bash"># install pyspark &#x26; Java apt-get install python3 -y apt-get install python3-pip -y DEBIAN_FRONTEND=noninteractive apt-get install openjdk-11-jdk-headless -y pip install pyspark </code></pre> <p>There are two ways to interact with pyspark. The first one is to execute the <code>pyspark</code> command in bash to initiate the pyspark session. The second way is that to treat pyspark as a module that the python kernel can import to. (<code>import pyspark</code>)</p> <h3>Method one</h3> <h4>Initiate <em>pyspark</em> session with jars</h4> <p>In order to use datatap with pyspark, you have to add an external jar file as an argument to pyspark. Initiate Spark's interactive shell in python using the following command.</p> <pre><code class="language-bash"># bash # Specify the path of the jars files pyspark --jars /opt/bdfs/bluedata-dtap.jar </code></pre> <p>After starting the interactive shell, <code>Spark Context</code> and <code>Spark Session</code> are automatically initiated for you.</p> <p><img src="https://user-images.githubusercontent.com/72959956/120170783-e8d00680-c233-11eb-9fe8-136da9996fdc.png" alt="image"></p> <p>After specifying the Hadoop configurations, you can read files from DataTap just like you normally did with HDFS.</p> <pre><code class="language-py"># pyspark # Specify the Hadoop configurations. sc._jsc.hadoopConfiguration().set('fs.dtap.impl', 'com.bluedata.hadoop.bdfs.Bdfs') sc._jsc.hadoopConfiguration().set('fs.AbstractFileSystem.dtap.impl', 'com.bluedata.hadoop.bdfs.BdAbstractFS') # Commands for reading DataTap file text = sc.textFile("dtap://TenantStorage/HPE.txt") text.take(5) </code></pre> <p><img src="https://user-images.githubusercontent.com/72959956/120171213-61cf5e00-c234-11eb-8928-2514e8b867a8.png" alt="image"></p> <h3>Method two</h3> <h4>Initiate <em>python</em> and initiate <em>pyspark</em> with <em>jars</em> at runtime</h4> <p>Run the Python Shell first:</p> <pre><code class="language-bash"># bash python3 </code></pre> <p>At the Python runtime, add the path of the jar file using the Spark configuration command:</p> <pre><code class="language-py"># python from pyspark import SparkConf, SparkContext # Specify the path of the jars files. conf = SparkConf().set("spark.jars", "/opt/bdfs/bluedata-dtap.jar") sc = SparkContext( conf=conf) # Specify the Hadoop configurations. sc._jsc.hadoopConfiguration().set('fs.dtap.impl', 'com.bluedata.hadoop.bdfs.Bdfs') sc._jsc.hadoopConfiguration().set('fs.AbstractFileSystem.dtap.impl', 'com.bluedata.hadoop.bdfs.BdAbstractFS') # Commands for reading DataTap file text = sc.textFile("dtap://TenantStorage/HPE.txt") text.take(5) </code></pre> <blockquote> <p>References:</p> <ul> <li><a href="https://spark.apache.org/docs/latest/configuration.html#runtime-environment">Spark Document: Runtime Environment</a></li> <li><a href="https://github.com/delta-io/delta/issues/346">Related GitHub Issues</a></li> </ul> </blockquote> <h1>Conclusion</h1> <p>A distributed file system is fundamental for handling large amounts of data. Managing those file systems tends to always be a pain for developers. DataTaps unify different storage resources into a path that different clusters can use. This helps you to get rid of time-consuming copies or transfers of data. More time spent on extracting business insight from your data and less time handling tedious stuff - that's what DataTap can give you. And that's what you get with HPE Ezmeral.</p>https://developer.hpe.com/become-a-legend/https://developer.hpe.com/become-a-legend/Tue, 22 Jun 2021 19:32:31 GMT<center><img src="/img/15_workshops_legend_no_bg-img1.png" width="400" height="439"></center> <p>The <a href="/hackshack/workshops">HPE DEV Workshops-on-Demand</a> are a great way to get hands-on experience with the newest and most popular technologies. These free workshops are easy to take and really give you a good feel for how to interact with things like containers, data fabric, Kubernetes, etc. We understand that you put in time and effort whenever you take one of our HPE DEV Workshops-on-Demand and we now offer badges in recognition of your achievement.</p> <p>Now, every time you finish one of our Workshops-on-Demand, you will receive a badge commemorating having completed that specific workshop. You’ll receive the badge in your final, congratulatory email. Share your badge with friends and colleagues on Twitter or LinkedIn directly from the email to show them what new skills you’ve recently acquired.</p> <center><img src="/img/data_fabric_no_bg-img2.png" width="400" height="407"></center> <h3>More opportunities to collect badges!</h3> <p>But that’s not all! Once you have finished 3 different workshops, you’ll receive an additional badge celebrating your achievement. More badges will be provided once you’ve completed 5, 7, 10, and 15 workshops. Climb the ranks, from <strong>Apprentice</strong> to <strong>Expert</strong> on to <strong>Hero</strong> and <strong>Super Her</strong>o and, ultimately, to <strong>Legend!</strong> Share these on your social media channels as well, competing with friends and colleagues to see who can make it to the highest level!</p> <center><img src="/img/complete_set_of_option_2-img3.png" width="1000" height="222"></center> <p>In addition to any other recognition we grant them, special badges will be awarded to those who create a Jupyter Notebook-based workshop that we can offer to the rest of the HPE Developer Community. Put your thinking caps on to try and collect one of these special badges.</p> <p>The HPE DEV Community is a great place where we can all enjoy some fun now and again. Try out some of our newest workshops found here on the <a href="/hackshack/workshops">HPE DEV Workshops-on-Demand</a> page. You can also navigate there from our HPE DEV portal <a href="https://developer.hpe.com/skillup">Skill Up</a> page. The workshops are great for getting acquainted with these newer technologies and boosting your skill set. And now they’re even more fun! Don’t forget to check out the <a href="https://developer.hpe.com/">HPE DEV portal</a> on a regular basis. We’re always coming up with something new.</p>https://developer.hpe.com/chif-driver-not-found/https://developer.hpe.com/chif-driver-not-found/Tue, 22 Jun 2021 12:06:33 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/chif/chif-driver-not-found">Server Management Portal</a>.</big></p> <br>https://developer.hpe.com/on-premise-adventures-how-to-build-an-apache-spark-lab-on-kubernetes/https://developer.hpe.com/on-premise-adventures-how-to-build-an-apache-spark-lab-on-kubernetes/Tue, 15 Jun 2021 16:51:59 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Apache Spark™ is an awesomely powerful developer tool for finding the value in your data. I highly recommend you check out this Hewlett Packard Enterprise (HPE) white paper for some background – <a href="https://www.hpe.com/psnow/doc/a50004177enw">Apache Spark 3 on HPE Ezmeral</a>. In this post, I’m going to explain how I deployed Apache Spark in my own on-premises HPE Ezmeral Container Platform-managed lab so that I could try Apache Spark out for myself.</p> <h3>We need a story first, right?</h3> <p>Suppose your story is similar to mine: I am a data scientist and I work for ACME Windmills Incorporated – or just ACME, for short. ACME owns and operates windmills that generate power. They have multiple sites, such as one in Australia and another one in Southern California. I’ve been asked to predict how much power will be generated by these various sites.</p> <h3>What tools do I have at my disposal?</h3> <p>Our infrastructure team runs all of the hardware and software for the Data Science team. They provide me access to things like a Jupyter Notebook where I will do my Data Science work. The infrastructure team of DevOps engineers and system administrators have the ultimate toolbox to work with, a toolbox called the <a href="https://www.hpe.com/us/en/ezmeral.html">HPE Ezmeral Software Platform</a>. HPE Ezmeral software enables an “all of the above” approach to deployment and management of ACME's data, apps, and the compute &#x26; storage resources that run it all - from anywhere. The infrastructure team can use this toolbox to:</p> <ul> <li> <p>modernize legacy apps, and manage those apps alongside cloud-native apps</p> </li> <li> <p>use existing data lakes alongside HPE Ezmeral Data Fabric</p> </li> </ul> <p>and all of the hardware and software can be consumed as-a-service from HPE!</p> <p>For this particular job, the infrastructure team pulled out the <a href="http://www.hpe.com/containerplatform">HPE Ezmeral Container Platform</a> from the toolbox for application and cluster management. This will provide the data science teams with a wealth of MLOps tools such as an Apache Spark cluster configured to run a customized Jupyter Notebook with connections to ACME’s <a href="https://assets.ext.hpe.com/is/content/hpedam/a00110846enw">HPE Ezmeral Data Fabric</a>. I can’t wait to get access to my environment and get to coding!</p> <h3>What is the infrastructure team building for me?</h3> <p>As mentioned above, ACME needs to predict power output from their windmills, so they can make intelligent decisions about where to add more windmills and how to optimize the windmills they already have. This is a job for Apache Spark on Kubernetes on HPE Ezmeral! The infrastructure team amazingly built the following workspace for me in the blink of an eye! This used to take them weeks before they got their hands on the HPE Ezmeral software.</p> <p><img src="/img/spark-on-ezmeral-architecture.jpg" alt="" title="Spark on HPE Ezmeral Architecture"></p> <p>As the figure above indicates, the infrastructure team built a Kubernetes Cluster and deployed the Apache Spark Operator onto it. The Apache Spark Operator is the glue that allows you to run Apache Spark within a containerized environment, such as a Kubernetes Cluster. The infrastructure team then carved out a Kubernetes Namespace and deployed the MLOps applications I will need like that custom Jupyter Notebook I mentioned previously. I will use that Jupyter Notebook to run my Apache Spark jobs.</p> <p>The code running inside the Jupyter Notebook will make an API call to the Apache Spark Operator using an API server called “Livy”. Livy will ask Apache Spark to create an Apache Spark session so I can perform analytics on the data. Apache Spark will be able to ingest data from the Australia core deployment of HPE Ezmeral Data Fabric using an HPE Ezmeral Container Platform feature called a DataTap (more on that below).</p> <h2>What the infrastructure team built for me</h2> <p>If you are a data scientist or data engineer and you really don’t want to know how all these back-end system administrator tasks are done, then I highly recommend you skip to the <strong><a href="#runourapachesparkjob">Now we get to run our Apache Spark jobs!</a></strong> section. I’d definitely put this section firmly in the infrastructure person category.</p> <p>First, I log into my HPE Ezmeral Container Platform WebUI. My organization has already setup Active Directory integration to use with the HPE Ezmeral Container Platform WebUI.</p> <p><img src="/img/spark-on-ezmeral-image2.png" alt=""></p> <p>Below, on the Kubernetes Dashboard Web UI page, you can see a list of options in the left hand pane. One of those options is <strong>Clusters</strong> found under the <strong>Kubernetes</strong> section in the upper left hand portion of the panel on the left side of the screen. From my dashboard, I click on my <strong>Cluster’s</strong> option on the left under <strong>Kubernetes</strong>.</p> <p><img src="/img/spark-on-ezmeral-image3.png" alt=""></p> <p>Next, in the <strong>Kubernetes Clusters</strong> window, I see I have a single cluster called DataFabricOnK8s. This cluster is running a version of HPE Ezmeral Data Fabric that runs within a Kubernetes Cluster. This is handy since now all of my Kubernetes namespaces and other clusters will get the same enterprise-grade storage features available in a Bare Metal deployment of HPE Ezmeral Data Fabric. I will create a new Kubernetes namespace from this cluster. To do that, I click on <strong>Tenants</strong> on the left. Tenants, as the name implies, are how we organize multiple Kubernetes namespaces in the HPE Ezmeral Container Platform. Or, in other words, how we help you manage <em><strong>multi-tenancy</strong></em>.</p> <p><img src="/img/spark-on-ezmeral-image4.png" alt=""></p> <p>In the <strong>Tenants</strong> window below, you can see that I created a new tenant on the DataFabricOnK8s cluster. Now I have a place to work and it is called “SparkLivyDemo”. I can use the WebUI to create a Jupyter Notebook in that namespace/tenant by clicking on the hyperlinked name.</p> <p><img src="/img/spark-on-ezmeral-image5.png" alt=""></p> <p>Here, I am in my own private “lab”, as I like to call it. This is my tenant work area. When I created my tenant, I chose to include my MLOps licensed content. This provides me with a Kubernetes namespace pre-loaded with all the goodies I need for machine learning, including a Jupyter Notebook tailored to work with my HPE Ezmeral Container Platform DataTaps. Speaking of which, I should add DataTaps to my HPE Ezmeral Data Fabric deployments in Australia and Southern California. I do that by clicking <strong>DataTaps</strong> on the left.</p> <p><img src="/img/spark-on-ezmeral-image6.png" alt=""></p> <p>On the screen below, you can see that I’ve tapped into two HPE Ezmeral Data Fabric deployments. I simply clicked on <strong>DataTaps</strong> in the left hand column of the WebUI and provided the details you see summarized in the <strong>Details</strong> column to the right of each DataTap <strong>Name</strong>. You can see that I needed to provide FQDNs to my secure-by-default HPE Ezmeral Data Fabric Core &#x26; Edge deployments. I also provided a user authentication <strong>Ticket</strong> that was generated by the Data Fabric administrator, so access is tightly controlled and easily connected using this WebUI (or REST API if you prefer).</p> <p>In this <strong>DataTaps</strong> window, you will also see a 3rd DataTap with a name of <strong>TenantStorage</strong>. That is a DataTap that is automatically created for me when I created this Tenant, and the storage is persistent and protected and managed by my Data Fabric Kubernetes Cluster.</p> <p><img src="/img/spark-on-ezmeral-image7.png" alt=""></p> <p>Finally, I launched a custom Jupyter Notebook using the Notebooks section of this MLOps tenant. This Notebook application comes with a full toolkit pre-integrated, so all I need to do is send the access endpoint to my Data Science team.</p> <p><img src="/img/spark-on-ezmeral-image8.png" alt=""></p> <p>To put my Data Scientists to work, I can just click on the <strong>Notebook Endpoints</strong> text, copy the Access Point URL, and then send that securely to my Data Science team in whatever manner I wish. Active Directory is passed on to the Jupyter Hub itself, so only an authorized user can log into that Access Point. From the Data Scientist’s perspective, they just pasted a hyperlink into their browser.</p> <p>You may now remove your <em>infrastructure person</em> hat and proceed to doing some really cool Apache Spark Analytics!</p> <p><img src="/img/spark-on-ezmeral-image9.png" alt=""></p> <p><a id="runourapachesparkjob"></a></p> <h2>Now we get to run our Apache Spark jobs!</h2> <p>For those of you who put on your infrastructure hat to read the previous section, thank you for sticking with me. Now you get to put on your Data Scientist hat! Or, if you skipped to here, that’s cool, too.</p> <h3>What happened behind the scenes</h3> <p>In the previous section, an infrastructure nerd built a lab out of some servers running a Kubernetes Cluster. Then, that person created a Kubernetes namespace from that cluster and applied the HPE Ezmeral MLOps template to that namespace. The infrastructure nerd also tapped into ACME’s global HPE Ezmeral Data Fabric and connected those <em>DataTaps</em> to a Jupyter Notebook. Just now you, the Data Scientist, received a link or <em>Access Point</em> to your Jupyter Notebook, put that link into your web browser, and logged into Jupyter Hub with your Active Directory credentials.</p> <h3>Jupyter Notebook + Spark (PySpark) + Livy</h3> <p>Here, inside the Jupyter Notebook, is where all the Apache Spark analytics is done. At the top of the file, I set up my Livy URL. I was able to copy this URL directly from my HPE Ezmeral Container Platform’s webUI, within the tenant view. A helpful HPE Ezmeral feature is the addition of MLOps <strong>magic</strong> commands that get added to this Jupyter Notebook automatically. For more information regarding this, see <a href="https://docs.containerplatform.hpe.com/53/reference/kubernetes/using-kubernetes/ai-ml-functionality/notebooks/Kubernetes_Notebook_Magic_Functions.html?hl=kubernetes%2Cnotebook%2Cmagic%2Cfunctions">Kubernetes Notebook Magic Functions</a>. In the screen below, you can see that the first Jupyter Notebook cell includes the%setLivy magic command. This command allows a Jupyter Notebook developer to specify the URL of the Livy server.</p> <p><img src="/img/spark-on-ezmeral-image10.png" alt=""></p> <p>The URL that is provided to the <em>%setLivy</em> magic command can be obtained from the HPE Ezmeral Container Platform’s WebUI under the “Applications” menu option under the <em>Service Endpoints</em> link. Here, in the <em>Service Endpoints</em> page you will find the <em>livy-http</em> URL or <em>Access Point</em> and the corresponding port.</p> <p><img src="/img/spark-on-ezmeral-image11.png" alt=""></p> <p>Next, within this Jupyter Notebook I use the <em>%%configure</em> command to override the default Apache Spark configuration and customize this Spark environment. This allows me to specify exactly what I need for this particular Apache Spark job.</p> <p><img src="/img/spark-on-ezmeral-image12.png" alt=""></p> <p>Moving down in my Jupyter Notebook, I next import some libraries needed to analyze and transform my data.</p> <p><img src="/img/spark-on-ezmeral-image13.png" alt=""></p> <p>Here, I define a few simple variables and connect the Apache Spark code to the global HPE Ezmeral Data Fabric – all thanks to the infrastructure person who set up those DataTaps for me. I could also have taken advantage of some of the other pre-configured local sandbox storage options automatically generated for me when my MLOps tenant was created. Like, for example, an NFS shared repository called an FS Mount and the <strong>Tenant Storage</strong> sandbox managed by our HPE Ezmeral Data Fabric on Kubernetes Cluster.</p> <p><img src="/img/spark-on-ezmeral-image14.png" alt=""></p> <p>From this point on, it is pure Apache Spark running in this standard Jupyter Notebook framework, using the PySpark kernel of the Jupyter Notebook.</p> <p><img src="/img/spark-on-ezmeral-image15.png" alt=""></p> <p>The final result from all this hard work is a Wind Turbine Power Production Prediction graphic plot – created within this same Jupyter Notebook. You can see from the graphic that my predictive model is lining up with actual, real power data. I can now ship this model off to be used by other members of the team.</p> <p><img src="/img/spark-on-ezmeral-image16.png" alt=""></p> <p>In the final lines of my Jupyter Notebook, I again use the DataTaps and standard PySpark commands (and also the NFS FS Mount feature) to save this model back out to the Australia and Southern California Ezmeral Data Fabric deployments and to the local sandbox storage.</p> <p><img src="/img/spark-on-ezmeral-image17.png" alt=""></p> <h2>Well, that was fun!</h2> <p>As you can see, going through this whole process I wore a couple of hats: <em>Infrastructure</em> or <em>DevOps</em> and <em>Data Scientist</em> or <em>Data Engineer</em>. With my <em>Infrastucture</em> hat on, I used the HPE Ezmeral Container Platform to create a workspace for the <em>Data Science</em> teams. That workspace is, in fact, a Kubernetes Cluster and one or more namespaces managed in a multi-tenant, secure environment.</p> <p>Everything discussed here was illustrated using the HPE Ezmeral Container Platform WebUI, but that is for illustration purposes only. HPE Ezmeral software is totally REST API enabled. This one example of building a single MLOps model is relatively simple. In reality, you would run a full MLOps pipeline and use other tools such as KubeFlow or AirFlow to build a pipeline of much more complex and iterative models – and that is built into the HPE Ezmeral Container Platform, as well. In addition, you can integrate directly with ML Flow in the same tenant workspace that I demonstrated here for much more powerful model management. Further, you can use the MLOps training, model management, and deployment features to fully control your machine learning pipelines.</p> <p>I hope you found this information interesting and look forward to helping you to extract the value from your data more efficiently and securely than ever before in subsequent <a href="https://developer.hpe.com/blog">HPE DEV blog</a> posts.</p>https://developer.hpe.com/host-based-volume-encryption-with-hpe-csi-driver-for-kubernetes/https://developer.hpe.com/host-based-volume-encryption-with-hpe-csi-driver-for-kubernetes/Tue, 15 Jun 2021 15:00:55 GMT<p>Security is on everyone's mind today, and storage should be considered of the utmost importance. In highly dynamic and agile environments where many hosts and applications share the same wire for consolidation purposes, it has become important to not only secure communication between endpoints, but also store data fully encrypted where only the designated reader and writer is capable of unlocking the data through the use of a private key.</p> <p>In this blog post, we'll discuss how to use the host encryption feature of the HPE Container Storage Interface ("CSI") Driver for Kubernetes that was introduced in version 2.0. This functionality is available to use with all supported backend Container Storage Providers ("CSP").</p> <h1>In-flight encryption VS data-at-rest encryption</h1> <p>Many of the supported HPE CSI Driver backends support encryption in one way or another. Let's examine the different modes to understand a little bit better where host encryption comes in.</p> <h2>Full disk encryption</h2> <p>FDE (Full Disk encryption) and SED (Self-Encrypting Drives) are two technologies available to users on certain storage devices that encrypts the entire drive, whether it's a SSD or HDD. Software may be used to manage the keys to read and write data. Access is established during power on of the drive, and once the drive is powered off, the decryption key, or passphrase, is needed to read and write content to the drive. This method will protect data if a drive is stolen or lost. The downside is that FDE drives are usually more expensive and key/passphrase management can be impractical due to the key is needed close to the data it protects, unless an external key manager is being used.</p> <h2>Storage appliance software encryption</h2> <p>A more sensible approach for a storage appliance is to have a proprietary software component to allow administrators to selectively choose logical volumes to be encrypted. Keys can be stored on the appliance, either password protected or automatically put the keys in place at boot. Drives taken out of the appliance will have encrypted data on them and if concerns of the whole appliance might be stolen or tampered with, an optional passphrase could be used.</p> <p>Neither FDE or appliance-based encryption secures data coming off the data fabric serving client workloads, such as iSCSI, FC or NVMe-oF. If a host is compromised or spoofed on the fabric, full access to the volume content is granted. This is better known as data-at-rest encryption.</p> <h2>Host encryption</h2> <p>Data that travels across a wire or data fabric in an encrypted fashion is known as in-flight encryption. In most cases, data is encrypted and decrypted at each side with a shared secret that has been established from either a trusted entity or a simple password. Data may be stored either encrypted or bare depending on the use case and usually there are separate technologies doing in-flight encryption and data-at-rest encryption.</p> <p>Host based encryption works very similar to the storage appliance software encryption but the control of the encryption is at the disposal of the host administrator using platform independent standard on-disk format. If the host administrator lose the key, the data is lost. All data, in-flight and at-rest is done outside any controls the actual storage administrator has. If the storage appliance is compromised or a rogue node spoofs the fabric identity of a legit host, the data will not decrypt without the key.</p> <p>Storing the key securely is where Kubernetes comes in and is necessary to transition volumes from node to node within a cluster. Let's walk through a simple example on how to use the host encryption for persistent volumes and how end-users are in full control of their own destiny on how their data is being secured.</p> <h1>Cluster-wide or namespace local keys</h1> <p>Configuring host encryption for persistent volumes is controlled with <code>StorageClass</code> parameters. A <code>StorageClass</code> is a cluster object which restricted users that deploy apps normally don't have access to. The HPE CSI Driver for Kubernetes provide a construct that allows restricted users direct access to <code>StorageClass</code> parameters defined by an administrator. This gives the users and administrators immense flexibility in terms of what to encrypt and what to encrypt it with. Let's examine these in detail.</p> <h2>Cluster administrator controlled host encryption</h2> <p>If an organization policy is to encrypt everything by default and only allow Kubernetes cluster administrators access to the key, a default <code>StorageClass</code> could look like the one below. Pay attention to <code>allowOverrides</code>, that would allow users who create persistent volume claims to opt-out encryption of certain persistent volumes.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: hpe-standard provisioner: csi.hpe.com parameters: csi.storage.k8s.io/controller-expand-secret-name: hpe-backend csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage csi.storage.k8s.io/fstype: xfs allowOverrides: hostEncryption hostEncryption: "true" hostEncryptionSecretName: hpe-encrypt hostEncryptionSecretNamespace: hpe-storage reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>Any <code>PersistentVolumeClaim</code> created from this <code>StorageClass</code> would be encrypted with the below <code>Secret</code>.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: Secret metadata: name: hpe-encrypt namespace: hpe-storage stringData: hostEncryptionPassphrase: "This is a very secret passphrase" </code></pre> <p>If a user wants to opt-out of encryption for a certain <code>PersistentVolumeClaim</code>, the PVC needs to be annotated.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-unencrypted-pvc annotations: csi.hpe.com/hostEncryption: "false" spec: accessModes: - ReadWriteOnce resources: requests: storage: 64Gi </code></pre> <p>There are good reasons for allowing certain volumes to be unencrypted as not all data is confidential and encryption does come with both compute overhead and storage inefficiencies (more on this later).</p> <h2>Namespace controlled host encryption</h2> <p>In scenarios where the namespaced user need to be in control of the key, it's possible for the cluster administrator to delegate the key management. This is incredibly useful when data is being replicated or backed up to an off-site location and data is being reused for dev/test use cases, disaster recovery or running compute heavy analytics to offload production. To allow this behavior, the <code>StorageClass</code> needs to be tweaked. Pay attention to <code>allowOverrides</code>.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "false" name: hpe-standard-delegated provisioner: csi.hpe.com parameters: csi.storage.k8s.io/controller-expand-secret-name: hpe-backend csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage csi.storage.k8s.io/fstype: xfs allowOverrides: hostEncryption,hostEncryptionSecretName,hostEncryptionSecretNamespace reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>Now, the user first need to create their own <code>Secret</code>.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: Secret metadata: name: my-encryption-key namespace: my-namespace stringData: hostEncryptionPassphrase: "This is another secret, it's all mine!" </code></pre> <p>This <code>Secret</code> now needs to be referenced in a set of annotations of the <code>PersistentVolumeClaim</code>.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-encrypted-pvc annotations: csi.hpe.com/hostEncryption: "true" csi.hpe.com/hostEncryptionSecretName: my-encryption-key csi.hpe.com/hostEncryptionSecretNamespace: my-namespace spec: accessModes: - ReadWriteOnce resources: requests: storage: 64Gi storageClassName: hpe-standard-delegated </code></pre> <p>Let's use the above <code>StorageClass</code>, <code>Secret</code> and <code>PersistentVolumeClaim</code> and attach a workload.</p> <h1>Provisioning and writing data</h1> <p>In this example I'm just bringing up a sleeping <code>Pod</code> and attaching the <code>PersistentVolumeClaim</code> named "my-encrypted-pvc" created from the previous section.</p> <pre><code class="language-yaml">--- kind: Pod apiVersion: v1 metadata: name: my-pod spec: containers: - name: my-test-pod image: alpine command: ["/bin/sh"] args: ["-c", "while true; do echo Snoozing...; sleep 10; done"] volumeMounts: - name: my-encrypted-mount mountPath: /data volumes: - name: my-encrypted-mount persistentVolumeClaim: claimName: my-encrypted-pvc </code></pre> <p>Once the <code>Pod</code> has come up, we can inspect the mount point.</p> <pre><code class="language-markdown">$ kubectl exec pod/my-pod -- df -h /data Filesystem Size Used Available Use% Mounted on /dev/mapper/enc-mpatha 64.0G 32.2M 63.9G 0% /data </code></pre> <p>The important difference we can observe here versus an unencrypted <code>PersistentVolumeClaim</code> is the "enc" keyword in the device name. This means that the kernel module "dm-crypt" has been used to broker access to the actual volume via Linux Unified Key Setup (LUKS). The HPE CSI Driver uses the default cipher with "dm-crypt" which currently is "aes-xts-plain64" with a key size of 256 bits.</p> <h1>Inspecting the wire</h1> <p>Assume we would write "Hello World" into an unencrypted volume versus an encrypted. In the following experiment I captured <code>echo "Hello World" > /data/file.txt &#x26;&#x26; sync</code> with a network packet sniffer and loaded up the result in Wireshark.</p> <p><img src="/img/unecrypted.png" alt="Hello World unencrypted" title="Hello World unencrypted"></p> <p>Imagine if this would be Personal Identifiable Information (PII) traveling across a compromised network. A security breach could potentially result in an expensive lawsuit if the information end up in the wrong hands. Let's repeat the experiment with an encrypted device.</p> <p><img src="/img/encrypted.png" alt="Hello World encrypted" title="Hello World encrypted"></p> <p>Inspecting the same packet in the sequence, it's quite obvious the entire payload has been encrypted. This, of course, has impact on the actual storage. Data you know traditionally compress and deduplicate well, all of a sudden present a near 1:1 representation.</p> <p>I conducted an experiment where I wrote an 8GiB file, filled with zeroes. Under normal circumstances, such a file is nearly invisible to any enterprise array. Not when encrypted.</p> <pre><code class="language-markdown">HPE Alletra $ vol --info pvc-44159f94-0f2b-4109-ae90-e48d0df082b1 | grep ^Volume Volume mapped usage (MiB): 7859 Volume compression: 0.97X </code></pre> <p>It's advised to consult with the storage administrator to ensure data-at-rest encryption on the array is turned off along with deduplication and compression to spare CPU cycles.</p> <p>Naturally, there will be a performance impact on the host, and it's advised to study empirical data from benchmarks conducted with a production-like workload to understand the amount of CPU headroom that's needed to ensure the application meets its performance criteria.</p> <h1>Maliciously trying to retrieve data</h1> <p>In the event of a rogue host gaining access to a volume without having the key, assume the LUN gets connected and discovered.</p> <pre><code class="language-bash"># mount /dev/mapper/mpatha /mnt mount: unknown filesystem type 'crypto_LUKS' </code></pre> <p>The encrypted volume may, in this state, be brute forced or maliciously deleted. Therefore, it's advisable to use a strong, non-dictionary passphrase for encryption and decryption. The passphrase can be up to 512 characters long. The passphrase length does not affect performance or the cipher strength. It's only used to open the device to the host.</p> <p>As an extra layer of security when using the iSCSI protocol, you can use the Challenge-Handshake Authentication Protocol (CHAP) facility available to the HPE CSI Driver. This ensures a mutual (between initiator and target) shared secret is needed to perform a discovery in the first place.</p> <h1>Summary</h1> <p>Being able to confidently store sensitive data on storage systems out of your control over insecure networks is becoming more important in the era of data being the digital oil. With this solution, you can take your keys and walk away without any concern of your volumes (or replicas of them) being stolen, manipulated or sold to third parties.</p> <ul> <li> <p>Check out the release announcement of <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-now-available-for-HPE-Alletra/ba-p/7136280">HPE CSI Driver for Kubernetes 2.0.0</a></p> </li> <li> <p>Read the documentation on HPE Storage Container Orchestrator Documentation (SCOD) around the <a href="https://scod.hpedev.io/csi_driver/using.html#volume_encryption">host-based Volume Encryption</a> feature</p> </li> <li> <p>Learn about the <a href="https://scod.hpedev.io/container_storage_provider/hpe_alletra_6000/index.html#multitenant_deployment">new multitenancy feature on HPE Alletra 6000 and Nimble Storage</a> to further improve security of your storage infrastructure</p> </li> </ul> <p>The team hangs out on Slack and is eager to learn about your security challenges. Sign up at <a href="http://slack.hpedev.io">slack.hpedev.io</a> and login to the community at <a href="https://hpedev.slack.com">hpedev.slack.com</a>, to check out #kubernetes #nimblestorage and #3par-primera.</p>https://developer.hpe.com/multitenancy-for-kubernetes-clusters-using-hpe-alletra-6000-and-nimble-storage/https://developer.hpe.com/multitenancy-for-kubernetes-clusters-using-hpe-alletra-6000-and-nimble-storage/Tue, 15 Jun 2021 15:00:00 GMT<p>We live in a storage infrastructure economy, where IT is under constant pressure to deliver more with less, yet still provide a high standard in data services directly to end users without compromising system or data security.</p> <p>With the introduction of <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-now-available-for-HPE-Alletra/ba-p/7136280">HPE CSI Driver for Kubernetes 2.0</a> and the software powering HPE Alletra 6000 and Nimble Storage, Hewlett Packard Enterprise ("HPE") introduces multitenancy for Kubernetes clusters accessing persistent volumes on the aforementioned storage arrays.</p> <p>The term <em>tenant</em> is ambiguous within the industry. For HPE Alletra 6000 and Nimble Storage, a tenant is a storage appliance user account with confined privileges to volumes existing within one or many <em>folders</em> on the array defined by a storage administrator. In turn, a folder is a logical construct to loosely group volumes together, which allows capacity and performance accounting to be limited on per folder level. The tenant may manage any aspect of the volume within the folder and may not exceed the boundaries set on the folder.</p> <p>In this blog post, I'll step through some of the basic elements to enable storage administrators to safely hand over credentials to Kubernetes administrators.</p> <h1>The enabling primitives</h1> <p>HPE Alletra 6000 and NimbleOS 6.0 includes a new command-line interface (CLI) called <code>tenantadmin</code>. This new CLI enables storage administrators to confine a user account into specific folders. Folders need to exist on the array prior creating a new tenant.</p> <p>The synopsis of the <code>--help</code> help flag gives an overview of the supported workflows.</p> <pre><code class="language-markdown">HPE Alletra $ tenantadmin --help Usage: tenantadmin [options] Manage Tenants. Available options are: --help Program help. --list List Tenants. --info name Tenant info. --add tenant_name Add a tenant. --folders folders List of folder paths (comma separated pool_name:fqn) the tenant will be able to access (mandatory). --remove name Remove a tenant. --add_folder tenant_name Add a folder path for tenant access. --name folder_name Name of the folder path (pool_name:fqn) to be added (mandatory). --remove_folder tenant_name Remove a folder path from tenant access. --name folder_name Name of the folder path (pool_name:fqn) to be removed (mandatory). --passwd Change tenant's login password. --tenant name Change a specific tenant's login password (mandatory). </code></pre> <p>With these basic <em>create</em>, <em>read</em>, <em>update</em> and <em>delete</em> ("CRUD") elements, the storage administrator is now empowered to delegate and confine all the storage resource management to a folder for a Kubernetes administrator to use.</p> <h1>The storage administrator's workflow</h1> <p>Assume a new Kubernetes environment is being deployed within an Enterprise. Each cluster needs to be compartmentalized to not consume all the performance and capacity of the array.</p> <p>First step, create a new restricted folder in the pool of your choice.</p> <pre><code class="language-markdown">folder --create k8s-prod \ --iops_limit 75000 --usage_limit 2500000 \ --description="Kubernetes Production Cluster" </code></pre> <p>Next, create a new tenant.</p> <pre><code class="language-markdown">tenantadmin --add K8sAdminProd --folders default:/k8s-prod Enter new password: ******** Retype new password: ******** Created User K8sAdminProd </code></pre> <p><strong>Hint:</strong> Username must be alphanumeric, cannot start with a number and cannot exceed 32 characters in length. The password policy enforced is derived from the system policy.</p> <p>At this point, the storage administrator hands over the credentials to the Kubernetes administrator.</p> <p>It's important to understand that giving the folder name to the Kubernetes administrator is completely optional. This could be useful in situations where it has been determined that a tenant need different performance characteristics for the folders. Like a <em>gold</em>, <em>silver</em> and <em>bronze</em> scheme. The Container Storage Provider ("CSP") will, by default, pick the folder with the most available capacity for the tenant. The upside by omitting the folder information to the Kubernetes administrator is that the storage administrator has all the power and flexibility to grow storage to new pools for a tenant without the tenant knowing about it. A very popular cloud operational model.</p> <h1>Apply Kubernetes configuration</h1> <p>Applying the tenant configuration to the Kubernetes cluster is not more difficult than using a standard "administrator" or "poweruser" account on the array.</p> <p>YAML declarations below are created with:</p> <pre><code class="language-markdown">kubectl create -f- &#x3C; Paste the YAML content > Hit CTRL-D on a new line. </code></pre> <p>Let's create a <code>Secret</code> referencing the HPE Alletra 6000 with the tenant credentials.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: Secret metadata: name: hpe-backend namespace: hpe-storage stringData: serviceName: alletra6000-csp-svc servicePort: "8080" backend: 192.168.1.30 username: K8sAdminProd password: qweqwe123 </code></pre> <p>Next, create a new default <code>StorageClass</code>.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: hpe-standard provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: xfs csi.storage.k8s.io/controller-expand-secret-name: hpe-backend csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage description: "Volume created by the HPE CSI Driver for Kubernetes" reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>At this point the Kubernetes cluster is ready to accept <code>PersistentStorageClaims</code>. Let's create one.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-first-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi </code></pre> <p>Once the <code>PersistentVolume</code> has been bound we can inspect the array and determine that volume has been placed in the tenant's folder.</p> <p><img src="/img/multitenancy-hpedev-screenshot.png" alt="HPE Alletra 6000 Multitenancy for Kubernetes Clusters" title="HPE Alletra 6000 Multitenancy for Kubernetes Clusters"></p> <p>That's it. Multitenant storage is now properly configured!</p> <h1>Container Storage Provider (CSP) access only</h1> <p>Multitenant storage is only available today through the HPE Alletra 6000 and Nimble Storage Container Storage Provider for Kubernetes. The CSP uses an undisclosed REST API resource of the array to perform CRUD operations on objects that in turn are being tagged and grouped accordingly to create the notion of a full blown cloud experience both from the Kubernetes administrator perspective but most importantly for the storage administrator. The storage administrator does not need to worry about storage system credentials being compromised and wreaking havoc beyond the compartment they were assigned for.</p> <p>At the time of writing, no limitations on the CSP functionality is restricted by using a tenant instead of a system account with the "administrator" or "poweruser" role. HPE recommends switching over to the tenant model for Kubernetes clusters accessing HPE Alletra 6000 or Nimble Storage arrays running NimbleOS 6.0.0 or later.</p> <p>Visit <a href="https://scod.hpedev.io">HPE Storage Container Orchestrator Documentation</a> (SCOD) to learn more about what storage resources are being exposed to tenants.</p> <h1>Example use cases</h1> <p>There are plenty of different use cases that multitenancy enables for IT Ops looking to manage and secure storage resources for a diverse set of applications running on Kubernetes.</p> <h2>Ephemeral Inline Volumes</h2> <p>End users that deploy applications on Kubernetes that require ephemeral storage at a capacity beyond of what a worker node is capable of providing, may use Ephemeral Inline Volumes. Before multitenancy, Kubernetes administrators had to share the <code>Secret</code> with the end user to allow provisioning of the Ephemeral Inline Volume. That is not very practical from a security standpoint as application administrators would have privileges on the storage array. Now, the end user may request a separate tenant to allow management of Ephemeral Inline Volumes securely for their application.</p> <p>Example.</p> <pre><code class="language-yaml">kind: Pod apiVersion: v1 metadata: name: my-csi-app-inline-volume spec: containers: - name: my-frontend image: busybox command: [ "sleep", "100000" ] volumeMounts: - mountPath: "/data" name: my-csi-volume volumes: - name: my-csi-volume csi: driver: csi.hpe.com nodePublishSecretRef: name: my-tenant-secret fsType: ext3 volumeAttributes: csi.storage.k8s.io/ephemeral: "true" accessProtocol: "iscsi" size: "5Gi" </code></pre> <p>The <code>Secret</code> "my-tenant-secret" would have to exist in the same <code>Namespace</code> as the <code>Pod</code> and contain the necessary connectivity and credentials to the tenant.</p> <h2>Virtualization and Containers</h2> <p>Deploying Kubernetes on a virtualization platform such as VMware vSphere, OpenStack or Hyper-V is by far the most popular pattern for deploying on-premises Kubernetes. Many times customers want to leverage the same array to provide persistent storage both for the virtualization platform and the container platform.</p> <p>Allowing Kubernetes clusters "administrator" or "poweruser" access to the array served by the virtualization platform the cluster is running on might be feasible in a single tenant and single application type scenario. Once weaving in the Ephemeral Inline Volumes use case into the mix and we've basically given application administrators way too many privileges on the array.</p> <p>In many cases the virtualization and storage administrator is combined into the same role. Moving forward, this administrative function would be able to securely hand over credentials to Kubernetes administrators that need a first class persistent storage solution.</p> <h2>Kubernetes-as-a-Service</h2> <p>Cloud and Managed Service Providers (CSPs and MSPs) monetizing their infrastructure are in a constant battle to safely and securely share infrastructure resources between their tenants and at the same time provide a differentiating portfolio. In the case of dispensing Kubernetes clusters to their tenants, they would have to resort to either using the virtualization platform CSI driver (such as the vSphere CSI driver), which is incredibly limited in functionality, or running a Container Attached Storage ("CAS") solution on the Kubernetes cluster itself, which in turn would result in storage and performance inefficiencies.</p> <p>With multitenancy, CSPs and MSPs are now enabled to create new tenants on the array as part of their catalog workflows and provide an entirely new set of rich data services enabled by HPE Alletra 6000 and HPE Nimble Storage.</p> <p><img src="/img/kaas.png" alt="Kubernetes as a Service" title="Kubernetes as a Service"></p> <h1>Summary</h1> <p>Expect more content that elaborates on how multitenancy can be used with Kubernetes using HPE Alletra 6000 and Nimble Storage. Consider this blog post a teaser regarding the cornerstone capability of multitenancy.</p> <ul> <li>Visit SCOD to learn more about the <a href="https://scod.hpedev.io/container_storage_provider/hpe_nimble_storage/index.html">HPE Alletra 6000 CSP</a></li> <li>Explore the all-new <a href="https://hpe.com/storage/alletra">HPE Alletra</a> 6000</li> <li>Check out the release blog of <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-now-available-for-HPE-Alletra/ba-p/7136280">HPE CSI Driver for Kubernetes 2.0</a></li> </ul> <p>The team hangs out in #kubernetes and #nimblestorage (Alletra channels pending) on Slack. Join at <a href="https://slack.hpedev.io">slack.hpedev.io</a> and sign in at <a href="https://hpedev.slack.com">hpedev.slack.com</a>. We're eager to learn about how you'll put multitenancy to use!</p>https://developer.hpe.com/similar-document-search-using-apache-spark-with-tf-idf/https://developer.hpe.com/similar-document-search-using-apache-spark-with-tf-idf/Tue, 15 Jun 2021 06:09:32 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code class="language-markdown">"authorDisplayName": ["Prasad Singathi","Maikel Pereira"], "publish": "2019-06-18T07:00:00.000Z", "category": ["machine-learning"], </code></pre> <hr> <h2>Background</h2> <p>As a professional services group, we were tasked with providing a solution to automatically find messages in the archives that are similar to new messages and send them to the person asking the question.</p> <p>To accomplish that goal, we decided to apply machine learning to the process, so that there is an automated program able to find similarities between the current message and the historical data. The algorithm used was <em>term frequency—inverse document frequency</em> (TF-IDF). TF-IDF is used in a variety of applications. Typical use cases include document search, document tagging, and finding similar documents.  </p> <h2>Problem Description</h2> <p>The desired solution was built using two Apache Spark applications running in a MapR cluster: one of them uses the historical data to update data features and train the model on a regular basis, and the second one analyzes every new message and finds five similar ones.</p> <h2>Application 1 - Creates Features and Trains Model</h2> <p>This application was developed using Spark and Scala, and it can run on a schedule, depending on the needs. <strong>Here is what it does, step by step:</strong></p> <ol> <li>Loads all messages from MapR Database. For the sake of brevity, we omit preprocessing steps like tokenization, stop words removal, punctuation removal, and other types of cleanup.</li> </ol> <pre><code class="language-scala">val rawEmaiData=spark.loadFromMapRDB("/googlegroups/messages") val rawEmaiDataDF=rawEmaiData.select("_id","bodyWithHistory","threadId","emailDate") </code></pre> <ol start="2"> <li>Creates hashingTF, using HashingTF class available in Spark, and sets fixed-length feature vectors of 1000. It applies the hashing transformation to the document, resulting in the featurizedData.</li> </ol> <pre><code class="language-scala">val hashingTF = new HashingTF().setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(1000) val featurizedData = hashingTF.transform(wordsData) </code></pre> <ol start="3"> <li>Creates the IDF, and from the TF and the IDF, it creates the TF-IDF.</li> </ol> <pre><code class="language-scala">val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features") val idfModel = idf.fit(featurizedData) val rescaledData = idfModel.transform(featurizedData) </code></pre> <ol start="4"> <li>A UDF is necessary for pre-calculating sparse vector norm.</li> </ol> <pre><code class="language-scala">def calcNorm(vectorA: SparseVector): Double = { var norm = 0.0 for (i &#x3C;- vectorA.indices){ norm += vectorA(i)*vectorA(i) } (math.sqrt(norm)) } val calcNormDF = udf[Double,SparseVector](calcNorm) </code></pre> <ol start="5"> <li>Creates a TF-IDF corpus.</li> </ol> <pre><code class="language-scala">val normalized = rescaledData.withColumn("norm",calcNormDF(col("features"))) </code></pre> <ol start="6"> <li>Saves IDF model to MapR XD Distributed File and Object Store.</li> </ol> <pre><code class="language-scala">idfModel.write.overwrite().save("/googlegroups/save_model_idf") </code></pre> <ol start="7"> <li>To save features vector to the MapR Database table, we have to convert the features vector to JSON format. For this, we create and register a UDF.</li> </ol> <pre><code class="language-scala">def toJson(v: Vector): String = { v match { case SparseVector(size, indices, values) => val jValue = ("type" -> 0) ~ ("size" -> size) ~ ("indices" -> indices.toSeq) ~ ("values" -> values.toSeq) compact(render(jValue)) case DenseVector(values) => val jValue = ("type" -> 1) ~ ("values" -> values.toSeq) compact(render(jValue)) } } } val asJsonUDF = udf[String,Vector](toJson) </code></pre> <ol start="8"> <li>Finally, saves features vector to the MapR Database table.</li> </ol> <pre><code class="language-scala">val dfToSave = normalized.withColumn("rawFeaturesJson", asJsonUDF(col("rawFeatures"))).withColumn("featuresJson", asJsonUDF(col("features"))).drop("rawFeatures").drop("features") dfToSave.saveToMapRDB("/googlegroups/trained_model", createTable = false) </code></pre> <h2>Application 2 - New Messages</h2> <p>The second application is a <a href="https://developer.hpe.com/blog/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-/">Spark Stream Consumer application</a> that <strong>will execute the following steps:</strong></p> <ol> <li>Loads the previously saved <code>idfModel</code> and initializes a new <code>HashingTF</code> model.</li> </ol> <pre><code class="language-scala">val idfModel = IDFModel.load("path/to/serialized/model") val hashingTF = new HashingTF().setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(1000) </code></pre> <ol start="2"> <li>Loads in memory and caches the data with the features saved previously.</li> </ol> <pre><code class="language-scala">val all = contextFuntions.loadFromMapRDB(argsConfiguration.trained).toDF all.cache() </code></pre> <ol start="3"> <li>Creates a DataFrame with the current message.</li> </ol> <pre><code class="language-scala">val one = Seq((x._id,x.body)).toDF("_id", "contents") val newWords = prepareWords(one, "words") val newFeature = hashingTF.transform(newWords) val newRescale = idfModel.transform(newFeature) val normalized = newRescale.withColumn("norm2", UDF.calcNormUDF(col("features2"))) </code></pre> <ol start="4"> <li>Then, it finds the crossjoin DataFrame between the one element and all existing messages in the database and calculates the similarity.</li> </ol> <pre><code class="language-scala">val cross = normalized.crossJoin(all).drop(normalized.col("_id")) val cosine = cross.withColumn("similarity", UDF.calcCosineUDF(col("features"), col("features2"), col("norm"), col("norm2"))) </code></pre> <ol start="5"> <li>For this, it uses the cosine function implemented as follows and registered as a UDF.</li> </ol> <pre><code class="language-scala">def cosineSimilarity(vectorA: SparseVector, vectorB:SparseVector,normASqrt:Double,normBSqrt:Double) :(Double) = { var dotProduct = 0.0 for (i &#x3C;- vectorA.indices){ dotProduct += vectorA(i) * vectorB(i) } val div = (normASqrt * normBSqrt) if( div == 0 ) (0) else (dotProduct / div) } udf[Double,SparseVector,SparseVector,Double,Double](cosineSimilarity) </code></pre> <ol start="6"> <li>The result can then be ordered by similarity, in descending order, taking the top five elements.</li> </ol> <pre><code class="language-scala">val similarsDF = cosine.sort(desc("similarity")).select("similarity","_id").limit(5) </code></pre> <p><img src="/img/image2_.png" alt=""></p> <h2>Conclusions</h2> <p>MapR provides the ecosystem needed for Apache Spark applications to run and scale as needed. It integrates all database and streaming platforms and enables the ability to do distributed processing. It efficiently integrates Spark with the database and the file system by extending it. Both capabilities, which are particularly useful for this solution, will be implemented in production as a feature of a bigger product in an effort to organize the Google Groups forum and with the intention of extending it to other data sources and realms. Since it is tested in a MapR cluster, all that would be needed is to install it and dedicate more resources when the moment comes.</p>https://developer.hpe.com/build-your-own-ilo-redfish-simulator/https://developer.hpe.com/build-your-own-ilo-redfish-simulator/Fri, 11 Jun 2021 16:36:54 GMT<h2>Introduction</h2> <p>When I first began developing Redfish <a href="/hackshack/workshops">Workshops-on-Demand</a>, I quickly realized that I would not be able to provision more than one or two physical servers with an embedded Redfish service allowing students to perform concurrent write operations. This was a problem since the infrastructure had been designed to host many more students.</p> <p>So, I started to look for Redfish simulators and found the <a href="https://www.qemu.org/">qemu</a> based <a href="https://github.com/openbmc/openbmc">OpenBmc</a> simulator that I used for the <a href="/hackshack/workshops">Redfish API 101</a> workshop. This simulator is perfect for this introductory lab as its Redfish implementation is simple and without Original Equipment Manufacturer (OEM) <a href="https://redfish.dmtf.org/redfish/mockups/v1/1060">extensions</a>. Moreover, its light memory and CPU consumption allows the start of almost a hundred instances in a single virtual machine, leading to as many attendees able to perform concurrent modifications.</p> <p>For the other two <a href="/hackshack/workshops">workshops</a> (<a href="http://hpe.com/info/resttool">iLOrest</a> and Ansible/OneView), I had to look for a more fully featured Redfish implementation in order to propose a wider range of exercises.</p> <p>This article presents the <a href="https://redfish.dmtf.org/">Distributed Management Task Force (DMTF)</a> <a href="https://github.com/DMTF/Redfish-Mockup-Creator">Redfish Mockup Creator</a> and <a href="https://github.com/DMTF/Redfish-Mockup-Server">Redfish Mockup Server</a> and how they can be used to learn and test the Redfish API by several tens of students concurrently.</p> <h2>The Redfish Mockup Creator</h2> <h3>Basic presentation, installation and invocation</h3> <p>To create your own Redfish simulator, you need to have read mode access to a live Redfish service (i.e. iLO 5). Then, using the <a href="https://github.com/DMTF/Redfish-Mockup-Creator">DMTF Redfish Mockup Creator</a> deployed in a place with network connectivity to the live Redfish service, you will be able to retrieve the entire Redfish resources in <code>index.json</code> files under a specified directory.</p> <p>The Redfish Mockup Creator is a single, simple and easy to deploy <a href="https://www.python.org/downloads/">python 3</a> script with a very small number of parameters and options that makes it easy to use. The associated documentation is up to date and provides several deployment methods and invocation examples in its GitHub <a href="https://github.com/DMTF/Redfish-Mockup-Creator#readme"><code>README.md</code></a> file.</p> <p>You can download the latest sources from this <a href="https://github.com/DMTF/Redfish-Mockup-Creator/releases/tag/1.1.1">release location</a> in <code>.zip</code> or <code>.tar.gz</code> format. Once downloaded, extract the sources into a location reachable by Python 3 and companion modules.</p> <blockquote> <p><strong>IMPORTANT NOTE:</strong> As mentioned in the <a href="https://github.com/DMTF/Redfish-Mockup-Creator/blob/1.1.1/requirements.txt"><code>requirements.txt</code></a> file, the DMTF <code>redfish</code> Python module is required to run the Mockup Creator. However, this module is not compatible with the HPE <code>python-redfish-library</code> because both of them contain a class called <code>redfish</code> but with different content. Use <code>pip uninstall python-redfish-library</code> before installing the DMTF <code>redfish</code> Python module with <code>pip install redfish</code>.</p> </blockquote> <p>The <code>python3</code> command below, launches the <code>redfishMockupCreate.py</code> script against a remote Redfish service (<code>-r ilo5</code>) accessible with the <code>-u</code> and <code>-p</code> credentials.</p> <pre><code class="language-shell">python3 redfishMockupCreate.py -r ilo5 -u ilouser -p ilopassword \ --Secure --Auth Session --Headers \ --Dir ./ilo5 </code></pre> <p>The <code>--Secure</code> argument specifies the use of the <code>HTTPS</code> secure protocol. The <code>--Auth</code> parameter allows two modes of authentication in the remote Redfish service: <code>Basic</code> and <code>Session</code>. With the <code>Basic</code> authentication, the username/password credentials will be used for each GET request. You can use the <code>Session</code> authentication mechanism, if it is supported by the remote Redfish service. In this case, the Mockup Creator will create a Redfish session using the supplied credentials and retrieve a session token from the response headers. This token will be used for all the GET requests needed to create the mockup.</p> <p>The <code>None</code> authenticated mode displayed in the help message of the Mockup Creator is a synonym of <code>Basic</code>. See the <a href="https://github.com/DMTF/Redfish-Mockup-Creator/blob/1.1.1/redfishMockupCreate.py">Python code</a>.</p> <p>The <code>--Headers</code> options stores the response headers of each <code>GET</code> requests in a <code>headers.json</code>. More details are present in the in the next paragraph.</p> <p>Lastly, the <code>--Dir</code> option provides the folder entry point for the mockup.</p> <blockquote> <p>NOTE: The mockup target directory will be created if necessary. If it exists, it must be empty before the launch of the Mockup Creator.</p> </blockquote> <h3>Preparing the use of iLOrest against your Mockup Server</h3> <p>If you intend to use <a href="http://hpe.com/info/resttool">ilOrest</a> against the mockup you created with the above command, you should, right after the mockup creation, open an iLOrest session and capture its cache. This cache directory is created during the authentication process in a default location unless a specific location is specified on the command line.</p> <p>The easiest way to perform this action is to install <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest">iLOrest</a> on your favorite operating system and identify the default cache location with the <code>help</code> command.</p> <p><img src="/img/ilorestcachelocation.png" alt="iLOrest default cache directory location" title="iLOrest default cache directory location"></p> <p>Then, open an iLOrest session (<code>ilorest login &#x3C;ilo-ip> -u &#x3C;user> -p &#x3C;password></code>) and save the content of the cache directory in a <code>.zip</code> or <code>.tgz</code>file. Once the cache is saved, you can logout safely (<code>ilorest logout</code>). You will use it later when the mockup server is up and running.</p> <h3>Mockup structure</h3> <p>Once authenticated in the remote Redfish service, the Mockup Creator crawls recursively the service. For each endpoint, starting at <code>/redfish</code> it sends an HTTP(s) GET request and creates an <code>index.json</code> file containing the response body as well as a folder for each sub-endpoint present in the <code>index.json</code> file. The first three endpoint levels are shown in the next picture.</p> <p>If the <code>--Headers</code> is present on the command line, a 'headers.json` file is created with the content of the GET response headers. This file holds potentially interesting information like the HTTP requests allowed against the current endpoint.</p> <p>The following screenshot lists the content of the output directory of the of the above invocation of the Mockup Creator. The "root" folder (<code>ilo5</code>) contains a <code>README</code> file and a <code>redfish\</code> directory. The <code>README</code> file contains the command line invocation. The <code>redfish</code> sub-folder contains two files (<code>headers.json</code>, <code>index.json</code>) and a directory (<code>v1</code>). Lastly, the <code>v1</code> directory contains the same two files as well as a sub-folder for each endpoint contained in the <code>index.json</code> file (AccountService, Managers, ....).</p> <p><img src="/img/mockupdirstructure.png" alt="Mockup structure" title="Redfish mockup structure"></p> <p>The following picture displays the first lines of the <code>redfish/v1/index.json</code> file. Note that the endpoints location reflects the mockup directory structure.</p> <p><img src="/img/redfishv1indexjson.png" alt="index.json partial content" title="index.json partial content"></p> <p>The following image shows the content of the <code>/redfish/v1/Systems/1/headers.json</code> file with the list of possible requests: GET, HEAD, POST, PATCH.</p> <p><img src="/img/headersjson.png" alt="headers.json example" title="Headers.json example"></p> <h2>Redfish Mockup Server</h2> <h3>Basic presentation and invocation</h3> <p>Once the Redfish mockup is created, you can make it available to Redfish clients with the <a href="https://github.com/DMTF/Redfish-Mockup-Server/releases/latest">DMTF Mockup Server</a>. This Python 3 application is a web server taking the location of a Redfish mockup as input.</p> <p>Deployment and usage of this program is easy and well documented in its <a href="https://github.com/DMTF/Redfish-Mockup-Server">GitHub</a> repository. The following code block shows how I launch it in the <a href="/hackshack/workshops">Workshops-on-Demand</a> infrastructure.</p> <pre><code class="language-Shell">python3 redfishMockupServer.py \ --ssl \ --key /SecureLocation/FdzSelfSigned.pem \ --cert /SecureLocation/FdzSelfSigned.pem \ --host 10.31.86.81 \ --port 45675 \ -D /usr/kits/VMs/RedfishMockups/ilo5 </code></pre> <p>In the above command, the <code>--ssl</code> parameter specifies that the Redfish service simulator will be accessible via HTTPS/SSL. The required private and public keys are located in a single file (<code>FdzSelfSigned.pem</code>).</p> <p>The <code>--host</code> and <code>--port</code> parameters specify the IP address and TCP port to listen to. Of course, the IP/Port tuple must be accessible to Redfish clients willing to send requests to the simulator.</p> <p>The last option (<code>-D</code>) provides the location of the Redfish mockup created previously.</p> <h3>How does the simulator work ?</h3> <p>Upon startup, the simulator copies the Redfish mockup in a private location and use this copy for GET and set requests. The original mockup files are never modified. Hence, a restart of the simulator places the simulator back in a fresh and known state.</p> <p>As of the writing of this article the DMTF Redfish Mockup Server does not implement any authentication mechanism. Hence, you don't have to authenticate before sending your client requests.</p> <p>When a Redfish client sends a GET request to the Redfish Mockup Server, it responds with the <code>index.json</code> file located in the folder of the requested endpoint as well as <code>200 OK</code> response code. If the target endpoint is not valid, the usual response codes <code>40X</code> will be sent back.</p> <p>For POST, PUT and PATCH requests, the simulator performs limited verification of the query, modifies the requested endpoint and sends back a <code>204 No Content</code> status code with no associated body. For the same request, a real Redfish service performs additional verification and sends back a non-empty response body with a <code>20X</code> status code.</p> <p>The different behavior of the simulator, compared to a real iLO 5 Redfish service, can be illustrated with a PATCH request for modifying the <code>IndicatorLED</code> resource of a computer chassis. The authorized values for this parameter are defined by the DMTF in the Chassis Redfish schema. For an iLO 5 with firmware version 2.30, the <a href="http://redfish.dmtf.org/schemas/v1/Chassis.v1_10_2.json#/definitions/IndicatorLED">implemented schema</a> specifies the following possible values: <code>unknown</code>, <code>Lit</code>, <code>Blinking</code>, <code>Off</code>. A physical iLO 5 complains if you supply a value different from what the schema proposes. However, the DMTF Redfish Mockup Server accepts any string, as shown in the following screenshot.</p> <p>The command shown in the screen shot below show the sending of a <code>PATCH</code> request with an invalid value (<code>Foo</code>) toward an iLO 5 simulator. The simulator performs the patch action and responds with status code <code>204</code>. The last command of shows that the action has been successfully performed.</p> <p><img src="/img/fooindicatorled.png" alt="PATCH an invalid property in the DMTF Redfish Mockup Server" title="PATCH an invalid property in the DMTF Redfish Mockup Server"></p> <p>The same query against a physical iLO 5 returns a <code>400 Bad Request</code> status and a <code>@Message.ExtendedInfo</code> mentioning the faulty argument.</p> <p><img src="/img/fooindicatorledagainstilo5.png" alt="PATCH of an invalid property in a physical iLO 5" title="PATCH of an invalid property in a physical iLO 5"></p> <h3>Querying the Mockup Server with iLOrest</h3> <p>If you want to query your iLO simulator with <a href="http://hpe.com/info/resttool">iLOrest</a>, you have to extract the cache directory you saved during the mockup creation (see the "Preparing the use of iLOrest against your Mockup Server" paragraph above) and edit the <code>url</code> property of its two files <code>index</code> and, <code>&#x3C;longUniqIdenfier></code>, to make them point to the simulator.</p> <p>On a Linux system, this operation can be done with the following commands:</p> <pre><code class="language-Shell">cd $iloCacheDir/cache/ sed -i 's?\("url": "https://\)ilo-IP-physical"?\1ilo-IP-simulator"?' * </code></pre> <p>Once the iLOrest cache points to your mockup server, you can use this Redfish client tool to query the mockup.</p> <blockquote> <p>NOTE: If you use the <code>login</code> iLOrest commands, the cache will be overwritten. If you use the <code>logout</code> command, the cache will be erased.</p> </blockquote> <h2>Conclusion</h2> <p>To learn and test the Redfish API, the DMTF provides two very useful tools to create Redfish mockups and simulate a live service. They are easy to install and use, and their quality is good. In addition, from my personal experience, I found the maintainers of these active GitHub projects very responsive to address quality issues and proposed enhancements.</p> <p>I presented only iLOrest to query this Redfish simulator, but many other Redfish clients can be used, like the ones mentioned in this <a href="https://youtu.be/ur9UKRV_0S8">video</a>.</p>https://developer.hpe.com/2021-May-03/https://developer.hpe.com/2021-May-03/Tue, 01 Jun 2021 06:00:00 GMThttps://developer.hpe.com/ways-to-interact-with-kubernetes-clusters-managed-by-hpe-ezmeral-container-platform/https://developer.hpe.com/ways-to-interact-with-kubernetes-clusters-managed-by-hpe-ezmeral-container-platform/Fri, 28 May 2021 08:40:17 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>HPE Ezmeral Software Platform consists of a bundle of software that helps you run, manage, control, and secure the apps, data, and IT that run your business. A major component is the HPE Ezmeral Container Platform (HPE ECP),a unified cloud container software platform built on Kubernetes (K8s). You can use it to deploy a new Kubernetes cluster with a few clicks. But how exactly do you connect through HPE ECP to interact with K8s? Don't panic! This blog post will introduce to you most, if not all, of the ways to connect with a Kubernetes cluster using HPE ECP.</p> <h2>WebUI</h2> <p>The first method by which you can interact with a Kubernetes Clusters managed by HPE ECP is, of course, through the web user interface (WebUI).</p> <table> <thead> <tr> <th>Screenshot</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><img src="https://github.com/helloezmeral/cdn/raw/main/K8s-Cluster.png" alt=""></td> <td>In the Main Menu, if you are the administrator of the HPE Ezmeral CP, you can manage and monitor the status of the K8s clusters managed by HPE ECP.</td> </tr> <tr> <td><img src="https://github.com/helloezmeral/cdn/raw/main/K8s-Dashboard.png" alt=""></td> <td>You can access the Kubernetes Dashboard just as you would do with open source Kubernetes.</td> </tr> </tbody> </table> <h2>WebTerminal</h2> <p>Inside the Kubernetes Tenant, at the bottom, there is a web terminal for you to interact with Kubernetes using the <code>kubectl</code> command.</p> <table> <thead> <tr> <th>Screenshot</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><img src="https://github.com/helloezmeral/cdn/raw/main/K8s-Tenant.png" alt=""></td> <td>At the bottom, you can click "Initialize" to initiate the web terminal instance.</td> </tr> <tr> <td><img src="https://github.com/helloezmeral/cdn/raw/main/K8s-Tenant-02.png" alt=""></td> <td>Type your <code>kubectl</code> command to interact with Kubernetes.</td> </tr> </tbody> </table> <h2>HPE Kubectl Plugin</h2> <p>If you want to use kubectl remotely, you must first install the kubectl-plugin to get the session ID to access HPE Ezmeral. To install the kubectl-hpecp plugin, you can run the following command. The HPE kubectl plugin is required to establish the tenant’s authenticated kubectl requests. Therefore, if you want to use kubectl remotely as a tenant user, you must first install the HPE kubectl plugin (kubectl-hpecp) to get an HPE ECP authentication token (a session ID), fetch the kubconfig manifest file, and then interact with the managed K8s clusters.</p> <p>To install the kubectl-hpecp plugin, first fetch the plugin from the WebUI. Log in as a tenant user, and click on Download HPE Kubectl Plugin (as shown in the picture below) and download the plugin and installation instructions according to your target operating system. Note: the following code is all executed in a Linux environment.</p> <table> <thead> <tr> <th>Screenshot</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><img src="https://github.com/helloezmeral/cdn/raw/main/K8s-Tenant-03.png" alt=""></td> <td>You can download the required binary here as well.</td> </tr> </tbody> </table> <h3>Installation of <code>kubectl</code> and <code>kubectl-hpecp</code></h3> <h4>Step 1: Make sure you have <code>kubectl</code> installed.</h4> <pre><code class="language-bash"># Download the latest version of kubectl curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" # And place it anywhere in your PATH: sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl </code></pre> <blockquote> <p>References:</p> </blockquote> <blockquote> <ul> <li><a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/">How to install and set up kubectl on Linux</a></li> </ul> </blockquote> <h4>Step 2: Install hpe kubectl plugin</h4> <pre><code class="language-bash"># This link might subject to be changed, renew this link on HPE Ezmeral # Download kubectl-hpecp binary and untar the file wget https://bluedata-releases.s3.amazonaws.com/kubectl-epic/3.4/14/linux/kubectl-hpecp.star tar xf kubectl-hpecp.star # And place it anywhere in your PATH: sudo mv ./kubectl-hpecp /usr/local/bin </code></pre> <p>Check that <code>kubectl-hpecp</code> is installed correctly.</p> <table> <thead> <tr> <th>Command</th> <th>Screenshot</th> </tr> </thead> <tbody> <tr> <td><code>kubectl plugin list</code></td> <td><img src="https://github.com/helloezmeral/cdn/raw/main/kubectl-plugin-list.png" alt=""></td> </tr> <tr> <td><code>kubectl hpecp -h</code></td> <td><img src="https://github.com/helloezmeral/cdn/raw/main/kubectl-hpecp-h.png" alt=""></td> </tr> </tbody> </table> <blockquote> <p>References:</p> </blockquote> <blockquote> <ul> <li><a href="https://docs.containerplatform.hpe.com/53/reference/kubernetes/using-kubernetes/Using_the_HPE_Kubectl_Plugin.html">Using the HPE Kubectl Plugin</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/">Extend kubectl with plugins</a></li> </ul> </blockquote> <h3>Getting the <code>kubeconfig</code> file</h3> <h4>Using <code>kubectl hpecp refresh</code> command</h4> <p>The <code>kubectl hpecp refresh</code> command gets the user a new Kubeconfig. Using the Kubeconfig, you can interact with Kubernetes through the HPE Ezmeral Container Platform.</p> <pre><code class="language-bash">kubectl hpecp refresh &#x3C;ip_address, host alias, or hostname> --insecure --hpecp-user=&#x3C;new_username> --hpecp-pass=&#x3C;new_password> # Example kubectl hpecp refresh 172.16.10.41 --insecure --hpecp-user=your-username --hpecp-pass=your-pass kubectl hpecp refresh ez53-gateway.hpeilab.com --insecure --hpecp-user=your-username --hpecp-pass=your-pass kubectl hpecp refresh ez53-gateway.hpeilab.com --insecure </code></pre> <p>Download your <code>Kubeconfig</code> file, and define the path to the <code>Kubeconfig</code> file as a shell environment variable.</p> <p><img src="https://user-images.githubusercontent.com/72959956/117413580-bab71980-af48-11eb-808e-1f46f074451c.png" alt="image"></p> <pre><code class="language-bash"># Example export KUBECONFIG="/home/hpeadmin/.kube/.hpecp/ez53-gateway.hpeilab.com/config" </code></pre> <h4>Download the <code>Kubeconfig</code> file manually</h4> <p><img src="https://user-images.githubusercontent.com/72959956/119962105-4b799600-bfd9-11eb-985d-2c867162902e.png" alt="image"></p> <p>Download your <code>kubeconfig</code> file, and define the <code>Kubeconfig</code> file as a shell environment variable:</p> <pre><code class="language-bash"># Example export KUBECONFIG="/the/path/of/your/kubeconfig" </code></pre> <h4>Using REST API</h4> <p>HPE Ezmeral Container Platform provides a REST API for you to interact with. Here is the set of REST API calls that allows you to download the <code>Kubeconfig</code> file.</p> <pre><code class="language-bash"># Authenticate as a tenant user in the specified tenant, getting the session ID: curl -k -i -s --request POST "http://ez53-gateway.hpeilab.com:8080/api/v2/session" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "username", "password": "password", "tenant_name": "test-tenant" }' # output HTTP/1.1 201 Created Access-Control-Allow-Origin: * Content-Length: 13 Content-Type: text/plain Date: Fri, 30 Apr 2021 13:18:38 GMT Location: /api/v2/session/__thisisthesessionid__ Server: HPE Ezmeral Container Platform 5.3 201 Created </code></pre> <p>Get the Kubeconfig file for your tenant working context:</p> <pre><code class="language-bash">curl -k -s --request GET "http://ez53-gateway.hpeilab.com:8080/api/v2/k8skubeconfig" \ --header "X-BDS-SESSION: /api/v2/session/__thisisthesessionid__" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' > ./kubeconfig # Define the Kubeconfig file as a shell environment variable export KUBECONFIG=kubeconfig </code></pre> <p>The screenshot below shows you how you can combine two commands into a single command:</p> <pre><code class="language-bash">curl -k -s --request GET "http://&#x3C;you-ez-gateway>:8080/api/v2/k8skubeconfig" \ --header "X-BDS-SESSION: $(curl -k -i -s --request POST "http://&#x3C;you-ez-gateway>:8080/api/v2/session" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "&#x3C;change-your-user-name>", "password": "&#x3C;change-your-user-password>", "tenant_name": "&#x3C;change-the-tenant-you-want>" }' | grep Location | awk '{print $2}' | tr -d '\r')" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' > ./kubeconfig export KUBECONFIG="./kubeconfig" kubectl get pods </code></pre> <blockquote> <p>Resources:</p> </blockquote> <blockquote> <ul> <li><a href="https://docs.containerplatform.hpe.com/53/reference/accessing-the-applications/API_Access.html">API_Access</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://github.com/HewlettPackard/hpe-notebooks/tree/master/HPECPAPI">Jupyter Notebook: Introduction to the HPE Ezmeral Container Platform REST API</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://github.com/bluedatainc/solutions/tree/master/APIs">API documents</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://developer.hpe.com/blog/hpe-container-platform-rest-api-part-1-authenticating/">HPE Container Platform REST API</a></li> </ul> </blockquote> <h2>hpecp python library (pre-alpha)</h2> <p>If you are looking for a way to interact with HPE Ezmeral programmatically, you can keep an eye on the hpecp python library from HPE Container Platform Community. Note that it is still a prototype, it may be unstable and subject to change until this library reaches beta.</p> <blockquote> <p>References:</p> </blockquote> <blockquote> <ul> <li><a href="https://github.com/hpe-container-platform-community/hpecp-python-library">Github: HPECP Python Library</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://hpe-container-platform-community.github.io/hpecp-python-library/index.html">HPE Container Platform Python Library Documentation</a></li> </ul> </blockquote> <blockquote> <ul> <li><a href="https://pypi.org/project/hpecp/">HPECP Python Pypi</a></li> </ul> </blockquote> <h2>Conclusion</h2> <p>As you can see, HPE Ezmeral Container Platform provides a number of different ways for you to interact with Kubernetes clusters so you can take advantage of the benefits it provides. Just pick your favorite way for your favorite environment. Happy Kubectl!</p>https://developer.hpe.com/calling-all-developers-make-your-voices-heard/https://developer.hpe.com/calling-all-developers-make-your-voices-heard/Fri, 28 May 2021 06:23:25 GMT<p><img src="/img/02-developer-nation-1024.jpg" alt="survey"></p> <h1>HPE DEV partners with SlashData for its State of the Developer Nation 2021 survey</h1> <p>As technologists, your world is always changing. And you want to stay on top of what’s next. To understand these trends and what is important to engineers today, <a href="http://slashdata.co/">SlashData</a> runs its developer survey twice yearly. The survey is expected to reach more than 36,000 developers across 159 countries to better understand who they are, what tools they use, and what they need.</p> <p>HPE Dev will once again be joining forces with SlashData as a media partner for the Summer 2021 developer survey. Hewlett Packard Enterprise (HPE), a technology and everything-as-a-service company, understands the importance of developers and how working with them is key to improving the way people live and work. We know that data drives business decisions, and so we encourage all our HPE DEV community members to take this survey and make their voices heard.</p> <p>SlashData's global, independent research gathers data through a detailed survey which covers almost every aspect of a programmer's life. The survey reaches out to millions of developers to get diversified opinions. It is available in multiple languages and dives deeply into 12 important areas such as web, cloud, IoT, games, ML/AI, and data science. While this in-depth survey may take 20 minutes of your time, your effort will will be rewarded by helping to drive change that meets your future needs. You might also be rewarded with a prize, since they run prize drawings worth more than $15,000 USD during every wave.</p> <p>The survey covers programming languages and tools, skill sets, and the resources developers use. It also focuses on key technology areas, including cloud, AI, machine learning, and game development. Using the data acquired through this research, SlashData produces “State of the Developer Nation” reports twice a year. You can find these free reports <a href="https://www.slashdata.co/free-resources?section=subscribe">here</a>. The data derived from these reports makes its way back to technology companies to address the current needs of developers around the world. Use these reports yourself to understand how the industry continues to evolve and make the right choices for your business.</p> <p>The survey opens on June 9th and ends August 4, 2021. Although it’s promoted by several other media partners, we would grateful if you could use <a href="https://www.developereconomics.net/?member_id=hpe">our link</a> to participate in the survey because it will provide us with specific data on how we can best address your needs. Feel free to forward <a href="https://www.developereconomics.net/?member_id=hpe">this link</a> to customers, partners, colleagues and friends who might also be interested in participating.</p>https://developer.hpe.com/hpe-discover-2021-don’t-miss-these-hpe-dev-industry-expert-talks-and-hands-on-workshops/https://developer.hpe.com/hpe-discover-2021-don’t-miss-these-hpe-dev-industry-expert-talks-and-hands-on-workshops/Tue, 25 May 2021 17:06:00 GMT<p>HPE Discover 2021 is where the next wave of digital transformation begins, powered by the rise of the Intelligent Edge and the vital data it creates. Spanning three days packed with actionable live and on-demand sessions, HPE Discover is where you’ll learn what Hewlett Packard Enterprise (HPE) offers in a world where software reigns supreme and helps define so many of our experiences today.</p> <p><a href="https://attend.hpe.com/discover2021/email?l=15AC66757307&#x26;EID=78EF62707200">Register here for HPE Discover 2021.</a></p> <h2>Calling all Coders!</h2> <p>This year, HPE DEV, the team that supports HPE’s developer community, will be offering a virtual Hack Shack technology session and four hands-on coding workshops. <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24350&#x26;schid=9807&#x26;locale=en_US&#x26;from=virtualplatform.catalogue_session&#x26;sf=2967">In the technology session</a>, HPE customers, Sysdig and ORock will discuss how they take advantage of the HPE Ezmeral software platform to provide secure utility-based As-a-Service offerings. This 30-minute roundtable is be moderated by key HPE CTO Office executive, Robert Christiansen, who works with HPE global clients and partners to deepen relationships and align joint technology efforts that improve the way people live and work.</p> <p>The four hands-on workshops will be similar in style to those offered last year, but this time with a twist. In the first 30 minutes, you’ll be introduced to industry experts who’ll give an overview of a specific technology. For the remaining 60 minutes, you’ll be invited to stay to experience a unique, hands-on workshop where you’ll actually get to play with the technology by employing Jupyter Notebooks to facilitate your coding experience.</p> <p>Most of the HPE DEV Hack Shack sessions will take place on Day 3. You can find them in the session catalog in one of two ways:</p> <ul> <li>Access the <a href="https://content.attend.hpe.com/go/virtualplatform.catalogue_session/?l=1045&#x26;sf=2879&#x26;locale=en_US">HPE DEV courses here</a>.</li> <li>Go to the <a href="https://content.attend.hpe.com/go/virtualplatform.catalogue_session/?l=1045&#x26;locale=en_US">Discover 2021 session catalog here</a> and enter HPE DEV in the Keyword Search. The Hack Shack sessions and workshops will appear below.</li> </ul> <p>Make sure you put these recommended sessions on your calendar:</p> <h2>HPE DEV Sessions and Workshops:</h2> <ol> <li> <p><strong>How HPE Ezmeral Provides Secure Utility-Based As-a-Service Platforms <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24350&#x26;schid=0&#x26;locale=en_US&#x26;sf=546">HSW4350</a></strong></p> <p>Dates/Times: June 23, 11:30am-12:00pm PST / June 24, 11:00am-12:30pm CEST</p> <p>Industry luminary Robert Christiansen (<a href="https://twitter.com/rbchristiansen">@rbchristiansen</a>) talks with two industry insiders from Sysdig and ORock on how they provide secure container platforms. Alexander Lawrence (<a href="https://twitter.com/alaw_sd">@alaw_sd</a>), Principal Solutions Engineer with Sysdig, describes how they extend security and monitoring on HPE Ezmeral through their SaaS platform, while Matt Plummer, Chief Cloud Architect at ORock Technology (<a href="https://twitter.com/ORock_Tech">@ORock_Tech</a>), discusses their open-source IaaS and PaaS with FedRAMP security built on the HPE Ezmeral platform.</p> </li> <li> <p><strong>HPE Ezmeral Data Fabric 101 – Get to Know the Basics Around the Data Fabric <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24349&#x26;schid=0&#x26;locale=en_US&#x26;sf=547">HSW4349</a></strong></p> <p>Dates/Times: June 24, 9:00am-10:30am PST / June 25, 11:00am-12:30pm CEST</p> <p>Join industry experts—Ted Dunning (<a href="https://twitter.com/ted_dunning">@ted_dunning</a>) and Ellen Friedman (<a href="https://twitter.com/Ellen_Friedman">@ellen_friedman</a>)—for a 30-minute talk on how workflows and data management change when systems are built on a unified data layer with data fabric. A hands-on workshop will follow, where you’ll learn how to create/delete/update volumes, set up disaster recovery mechanisms, and apply security policies.</p> </li> <li> <p><strong>Kubernetes 101 – Intro to the Kubernetes Concepts Managed by HPE Ezmeral Container Platform <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24347&#x26;schid=0&#x26;locale=en_US&#x26;sf=548">HSW4347</a></strong></p> <p>Dates/Times: June 24, 11:00am-12:30pm PST / June 25, 1:00pm-2:30pm CEST</p> <p>Walk through the basics of the Kubernetes cluster orchestration system with industry expert, Nigel Poulton (<a href="https://twitter.com/nigelpoulton">@nigelpoulton</a>), Chief Technologist at Kubetrainer.com, and Thomas Phelan, HPE Ezmeral Container Platform CTO. During a following 60-min hands-on workshop, you’ll deploy a containerized application on a cluster, scale its deployment, update the application, and debug it, using info you learned on Kubernetes features and an interactive online tutorial.</p> </li> <li> <p><strong>SPIFFE and SPIRE Fundamentals <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24392&#x26;schid=0&#x26;locale=en_US&#x26;sf=549">HSW4392</a></strong></p> <p>Dates/Times: June 24, 12:00pm-1:30pm PST / June 25, 2:00pm-3:30pm CEST</p> <p>Join this workshop to better understand SPIFFE as a set of open-source standards for securely authenticating software services in dynamic and heterogeneous environments through the use of platform-agnostic, cryptographic identities. Explore SPIRE as an open-source system that implements the SPIFFE specification in a wide variety of environments. Featuring Sunil James of HPE (<a href="https://twitter.com/sunubunu">@sunubunu</a>) and Phil Vachon, Security leader at Bloomberg (<a href="https://twitter.com/pvachonnyc">@pvachonnyc</a>).</p> </li> <li> <p><strong>Redfish Programming Made Easy and Secure with Ansible and HPE OneView <a href="https://content.attend.hpe.com/go/virtualplatform.details/?l=1045&#x26;SID=24345&#x26;schid=0&#x26;locale=en_US&#x26;sf=553">HSW4345</a></strong></p> <p>Dates/Times: June 24, 10:00am-11:30am PST / June 25, 12:00pm-1:30pm CEST</p> <p>Learn to securely automate management, monitoring, and configuration of compute nodes. After a 30-min intro by Jeff Hilland, DMTF president, stay for a hands-on workshop using Redfish and Ansible, leveraging HPE OneView Single Sign On. You will learn multiple ways for writing Redfish Ansible playbooks using built-in modules, DMTF, and HPE examples.</p> <p>In addition to these HPE DEV Hack Shack sessions, there’s so much more to explore at Discover 2021 for those interested in software; from the HPE Ezmeral Container Platform and Data Fabric to open source projects sponsored by HPE. For details on AI, ML, and Data Analytics sessions you don’t want to miss, check out this <a href="https://community.hpe.com/t5/Advancing-Life-Work/HPE-Discover-2021-AI-ML-and-Data-Analytics-sessions-you-don-t/ba-p/7138437#.YMdnm5NKiMK">blog post</a>.</p> </li> </ol> <h2>Live Demos</h2> <p>We will also be featuring live, on-location, demos at Chase Center and the Mercedes Formula 1 Factory, scheduled to take place on June 22 or 23 (depending on which region you’re joining from). During the times specified below, HPE will have experts available to help answer your questions.</p> <ul> <li>AMS: 11:00 AM – 11:45 PM PDT and 11:45AM – 12:30PM PDT on Tuesday (Day 1)</li> <li>APJ: 2:30 PM – 3:15PM JST and 3:15PM – 4:30 PM JST on Wednesday (Day 1)</li> <li>EMEA: 12:30 PM – 1:15 PM CET and 1:15PM – 2:00PM on Wednesday (Day 1)</li> </ul> <p>Interested in other sessions? We invite you to explore the full line-up of sessions on <a href="https://content.attend.hpe.com/go/virtualplatform.catalogue/?l=1045&#x26;locale=en_US">our content catalog</a> and build your own agenda. You can also view the <a href="https://content.attend.hpe.com/go/virtualplatform.agenda/?l=1045&#x26;locale=en_US">agenda for each region here</a>.</p> <h2>Build Your Playlist</h2> <p>Want to build your own Playlist? Here’s how. Once registered for HPE Discover, <a href="https://content.attend.hpe.com/go/virtualplatform.landing/?l=1045&#x26;locale=en_US">log in</a> to the virtual platform to view each of the keynotes, sessions, demos, etc. You can filter based on content type, areas of interest, or keyword search, etc. Then simply click on the “+” icon to add the item to your My Playlist. You can also download your Playlist into your preferred personal calendar.</p> <p><em><a href="https://attend.hpe.com/discover2021/email?l=15AC66757307&#x26;EID=78EF62707200">Register now.</a></em></p> <p>We look forward to seeing you virtually at HPE Discover 2021!</p>https://developer.hpe.com/help-build-the-hospital-of-the-future-and-be-eligible-for-prizes/https://developer.hpe.com/help-build-the-hospital-of-the-future-and-be-eligible-for-prizes/Tue, 11 May 2021 14:58:32 GMT<p>Hewlett Packard Enterprise (HPE), the multinational enterprise information technology company based in Houston, Texas,  is proud to partner with <a href="https://www.texaschildrens.org/departments/us-news-world-report">Texas Children’s Hospital</a>, one of the top-ranked children’s hospitals in the US, in its <a href="https://www.hackerearth.com/challenges/hackathon/texas-childrens-hospital-healthcare-hackathon/">inaugural innovation-focused hackathon</a>. To celebrate the groundbreaking of its new hospital, Texas Children’s is hosting a two-week long <a href="https://www.hackerearth.com/challenges/hackathon/texas-childrens-hospital-healthcare-hackathon/">virtual Healthcare Hackathon</a> from May 14th through May 24th, 2021. This virtual event is designed to inspire and draw attention to the need for new ways of thinking and the use of technology in healthcare in treatments for women and children.</p> <p>Through this hackathon, the hospital is seeking innovative ideas that incorporate technology to address its biggest day-to-day challenges. Showcase your innovative, tech-based solutions to win awesome prizes! Coders are encouraged to work individually or in teams of 2-5 people to build solutions that align to one or more of the following <a href="https://www.hackerearth.com/challenges/hackathon/texas-childrens-hospital-healthcare-hackathon/">themes</a>:</p> <ul> <li><strong>Food Service and Delivery</strong> – How do we reimagine the food service and delivery experience to ensure patients, families, employees and guests get the nutritious meals they need?</li> <li><strong>Meeting People Where They Are</strong> – How do we transform traditional hospital experiences/processes so they are more convenient, personalized, and equitable?</li> <li><strong>Touchless Experience</strong> – How can we convert physical interactions into touchless, yet still personal and trustworthy, digital healthcare experiences?</li> <li><strong>AR/VR in Hospitals</strong> – How can we implement extended reality experiences (XR) to reimagine the doctor/patient experience, training, and patient/family/care team communication and coordination?</li> <li><strong>Evolution of the Child Life Experience</strong> – What innovative experiences can Child Life specialists use to help patients and their families cope, connect, socialize, and heal? </li> </ul> <h2><strong>For Kids By Kids</strong></h2> <p>As part of the hackathon, Texas Children’s is offering a special category exclusively for coders under 18 years of age called <a href="https://www.hackerearth.com/challenges/hackathon/texas-childrens-hospital-healthcare-hackathon/custom-tab/for-kids-by-kids/#For%20Kids%20By%20Kids">For Kids By Kids</a>.  Using Scratch, a free visual programming language that’s designed to teach kids how to program through drag and drop blocks of code, young coders are invited to animate patient journey’s at Texas Children’s Hospital. Examples of a patient’s journey include <a href="https://s3-ap-southeast-1.amazonaws.com/he-public-data/discharge-pathw_081420187d3b3ef.pdf">what to expect when being discharged from the hospital</a> or what to expect when going to see your pediatrician. Please note that coding classes and camps are welcome! Once done with the animation, a recording of the animation and/or any Scratch code the young developer would like to share should be submitted for the judges’ evaluation.</p> <h2><strong>Judging</strong></h2> <p>Submissions are due by May 24, 2021 11:55 PM CDT. Every submission will be evaluated by a panel of Texas Children’s Hospital experts and at least one of their sponsors, who include HPE, NVIDIA, Mark III Systems, the Butler Bros., Google Cloud, T-Mobile for Business, Unity, and HEB Digital. Winners will be announced on June 9th, 2021. There are up to $10,000 USD in prizes to be given away, including an NVIDIA Shield and leather duffle bags. Time is short! <a href="https://www.hackerearth.com/challenges/hackathon/texas-childrens-hospital-healthcare-hackathon/">Register on the bottom right of this page</a> for the hackathon.</p> <p>There is no better mission than the health of our children. That is why HPE has teamed up with Texas Children’s Hospital to identify innovative solutions to be a part of their new state-of-the-art hospital in Austin, TX. Sign up for the <a href="https://developer.hpe.com/newsletter-signup/">HPE DEV Newsletter</a> to stay abreast of news on hackathons like this in the future, as well as learn more on different topics through our <a href="https://developer.hpe.com/blog/">blog</a>.</p>https://developer.hpe.com/application-modernization-with-the-application-workbench/https://developer.hpe.com/application-modernization-with-the-application-workbench/Mon, 10 May 2021 08:22:34 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>One of the most significant issues facing enterprises in their journey towards digital transformation is the challenge of application modernization. Did you know that 70% of the Global IT budget is spent on legacy application maintenance? In fact, 7 in 10 companies today struggle with legacy application maintenance while they tackle their digital transformation. Consider how counterproductive it is to spend that much time, energy, and money on something that doesn’t really help them move forward.</p> <p>In this post, I’ll discuss the different approaches one can take to modernize an application and how the HPE Ezmeral Application Workbench can help.</p> <h2>What is Application Modernization?</h2> <p>Application modernization is the process of taking legacy applications and the platforms they run on and making them new again by replacing or updating each with modern features and capabilities. It includes changing the underlying application architecture from a monolithic to a distributed model by taking the advantage of the integration and automation implicit in DevOps practices. It also often means that the architecture and source code are updated through the use of modern programming languages in order to support containers and microservices and can take advantage of built-in security and storage features.</p> <p>One could equate application modernization to updating the design of a car. For instance, if you were to compare a vintage 1967 Ford Mustang to a modern 2021 version, the 1967 would not have disk brakes, ABS, fuel injection, air bags, shoulder belts, crush zones and all the other things that make it a modern, safe, reliable, and economical car.</p> <p>Truly, there is very little you can do to a 1967 Mustang to bring it up to the standards of a 2021 car. To transform it, you would need to change the 1967 car’s base architecture. Another option would be to completely replace the older car with a different brand that boasts a modern architecture, like Tesla. But that would probably change a lot about how you drive. If you happen to like Ford and want to stick with it because you’re familiar with how their cars work, that approach may not work well for you.</p> <p>This same idea applies to modernizing software applications. You cannot simply update the underlying compute and storage infrastructure and expect an application to automatically behave as if it had modern features. An application that was built for a mainframe environment is very different from applications that are built today, applications that are optimized for distributed environments and developed through the use of DevOps practices. To have an application work efficiently and securely across hybrid clouds, it needs to be modernized.</p> <h2>The Challenges of Modernizing Legacy Apps</h2> <p>Enterprises face a variety of challenges as they address application modernization, such as:</p> <ul> <li>Isolating applications from their infrastructure is often a very time consuming and manual process.</li> <li>Legacy, monolithic systems are not designed with distributed hardware in mind and are difficult to break apart.</li> <li>Legacy apps lack security integrations.</li> <li>They also lack tight DevOps integration.</li> <li>Lastly, budget and cost control are difficult to estimate, as well as difficult to test and consolidate amongst legacy applications.</li> </ul> <p>There are different approaches one can use to migrate legacy applications into modern architectures. Most of these solutions are offered in the <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Ezmeral Container Platform</a> along with the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>. Depending on the customer use case, application, cost, and time requirements, HPE can help to smooth the way in this journey.</p> <h2>What are the Options for App Modernization?</h2> <p>There are several different paths organizations take in their attempt to modernize their applications:</p> <img src="/img/1-gunna.png" width="600" height="359" class="center"> <ul> <li><strong>Rehost</strong> – This simply means that a legacy application is moved to a modern compute and storage platform, either on premise or in the cloud, without altering the original code. In this case, the application does not magically acquire modern features. Going back to our car analogy, it would be like giving a 1967 Mustang new tires. You can’t expect that it would now also have all the features found in a 2021 Mustang. Rehosting an application does not modernize it.</li> <li><strong>Refactor</strong>, <strong>Redesign</strong> and <strong>Rebuild</strong> – These options involve rewriting portions of the applications from scratch, while preserving the original scope and specifications. Using these methods will modernize an application by updating the source code to a newer programing language, the underlying data to modern formats, and the compute and storage infrastructure to support the newly modernized application.</li> <li><strong>Replace</strong> – Another option is to replace the application altogether. This option eliminates the original application completely and replaces it with a new application better suited to an organization's current business needs.</li> </ul> <p>HPE Ezmeral has a part to play in each of these scenarios to assist in your application modernization goals. For instance, if you decide to <strong>Rehost</strong>, the <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Ezmeral Container Platform</a> offers a central control plane for virtual or physical compute resources located on premise, in the cloud, or at the edge. It includes full support for standard Kubernetes (K8s) orchestration and the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>, which powers the data plane with a global namespace for data from core to edge – all wrapped in a secure, enterprise-class, highly available platform.</p> <p>For those who wish to <strong>Refactor</strong> their code, HPE Ezmeral smooths the path to do so. There are many different resource management types for deploying an application on Kubernetes, and all of these are supported by the HPE Ezmeral Container Platform. You can use different options for refactoring or rebuilding the app, depending on what best matches your organization needs. A key feature of the HPE Ezmeral Container Platform is its use of KubeDirector, an open source Kubernetes custom controller that addresses stateful scale out application deployment in standard K8s clusters. This approach enables transparent integration with K8s user/resource management and existing K8s clients and tools.</p> <p>HPE Ezmeral also offers the <a href="https://docs.containerplatform.hpe.com/53/app-workbench-5-1/getting-started/AWB51_App_Workbench.html">HPE Ezmeral Application Workbench</a> for those who wish to <strong>refactor</strong> their code. The Workbench helps you rebuild or redesign application architecture while preserving their scope and specifications. Once you have built your new application, it can be easily deployed on the HPE Ezmeral Container Platform and leverage the HPE Ezmeral Data Fabric.</p> <p>Finally, for those who prefer to <strong>Replace</strong> their application, HPE Ezmeral software is tested with a large partner ecosystem. Our partners offer modern applications through the <a href="https://www.hpe.com/us/en/software/marketplace.html">HPE Ezmeral Marketplace</a> that can be used to replace legacy applications.</p> <h2>What is HPE Ezmeral Application Workbench?</h2> <p>HPE Ezmeral Application Workbench is a free, stand-alone, Python-based software development kit (SDK) that allows for the quick development of applications for Kubernetes, AI/ML, and other use cases, either from scratch or from existing container images. The HPE Ezmeral Application Workbench takes advantage of KubeDirector, an open-source project from HPE, which acts as a custom controller that allows you to easily bring your app into a standard K8s cluster. Kubedirector makes it easy to run complex stateful scale-out application clusters on Kubernetes.</p> <p><img src="/img/2-gunna.png" alt=""></p> <p>HPE Ezmeral Application Workbench features a simple to use, workflow-based, web-based graphical user interface (WebUI) to help you build modern applications visually. Use it to update legacy source code to build a new, custom docker image, and convert your legacy application into a microservice-based, modern app. After installing the App Workbench on your workstation, you can <a href="https://docs.containerplatform.hpe.com/53/app-workbench-5-1/getting-started/AWB51_Overview.html">build your application</a> using just a few simple steps.</p> <p>Key features of the HPE Ezmeral Application Workbench include:</p> <ul> <li>KubeDirector Application support</li> <li>Public and private Docker registry support</li> <li>Source-to-Image capabilities, allowing users to transform monolithic application source code to executable Docker images</li> <li>A feature rich application development workspace, including the ability to:</li> <li>Edit Dockerfiles, HTML, JSON, Markdown, Python, SH, XML, YAML files in one place.</li> <li>Organize configuration scripts and application specific startup scripts in one place.</li> <li>Output a fully formatted JSON or YAML file to apply to your K8s cluster with <a href="https://kubedirector.io/">KubeDirector operator</a> installed.</li> </ul> <p>The HPE Ezmeral Application Workbench allows you to point to a repository, bring in a Docker image, modify config scripts, and create your own startup script. With the help of the Workspace editor in the App Workbench tool set, users can create Dockerfiles by splitting the monolithic application components into a microservice or distributed model. This feature also helps in containerization refactoring where monolithic apps are moved into containers with minimal modifications, enabling users to incorporate cloud-native features and improve portability. Once you’re done, all you need to do is to use a standard kubectl command: <strong>kubectl apply –f &#x3C;yourapp.json/yaml></strong> of the JSON or YAML file that’s been outputted. The HPE Ezmeral Application Workbench delivers your app right into your HPE Ezmeral Container Platform App catalog.</p> <h2>Advantages of using the HPE Ezmeral Application Workbench</h2> <p>The HPE Ezmeral Application Workbench offers support to build KubeDirector applications that can be deployed on a CNCF-certified K8s cluster with the KubeDirector operator add-on. KubeDirector is an open-source project that uses the Kubernetes Custom Resource Definition (CRD) framework to enable transparent integration with Kubernetes user/resource management, allowing you to deploy stateful applications on Kubernetes. It leverages native Kubernetes API extensions and acts as a scheduler and launcher of applications on top of the Kubernetes platform.</p> <p>KubeDirector empowers the application, doing a lot of stuff in the background for you. You don’t have to write “go code” or an operator for it; it will use what you have. And if you don’t have an operator, and you have a legacy application that needs statefulness and persistent directories, KubeDirector can assist. You can think of it as a custom operator. Using KubeDirector is a lot easier than using YAML, requiring less files and manual definition, and more powerful than using Helm charts, which don’t track the state of a node.</p> <p>Because of KubeDirector, applications built using the HPE Ezmeral Application Workbench can be easily deployed on the HPE Ezmeral Container Platform, which allows it to leverage the HPE Ezmeral Data Fabric, for a full end-to-end Kubernetes application experience. Using the HPE Ezmeral Application Workbench accelerates time to value for all your application modernization projects, including big data, artificial intelligence / machine learning (AI/ML), developer operations (DevOps), and continuous integration / continuous deployment (CI/CD).</p> <h2>Conclusion</h2> <p>Legacy applications can hold you back in your digital transformation because it is software developed on outdated hardware and design principles. The success of your digital transformation depends on your ability to innovate and accelerate developer productivity at scale through newer, cloud native technologies.</p> <p>Don’t let legacy applications hold you back in your digital transformation. Modernize your legacy applications in a cost-effective way, improve developer productivity, and accelerate time to value for your business with <a href="https://www.hpe.com/us/en/ezmeral.html">HPE Ezmeral</a>. For more information, access <a href="https://docs.containerplatform.hpe.com/53/app-workbench-5-1/getting-started/AWB51_App_Workbench.html">HPE Ezmeral Container Platform Documentation on App Workbench</a>.</p> <p>For more articles designed to help you in your digital transformation, stay tuned to the <a href="https://developer.hpe.com/blog">HPE DEV blog</a>.</p>https://developer.hpe.com/introducing-the-verified-hpe-oneview-terraform-provider/https://developer.hpe.com/introducing-the-verified-hpe-oneview-terraform-provider/Fri, 07 May 2021 05:00:00 GMT<!--StartFragment--> <p>Hewlett Packard Enterprise (HPE) and HashiCorp have worked together to verify the new HPE OneView Terraform Provider. The new provider is based on Terraform v0.13. This enables users to take full advantage of the improved infrastructure automation capabilities available in the latest versions of Terraform. The provider gives users the ability to automate infrastructure through <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a>, which uses software-defined intelligence via a template-driven approach to automate the deployment, provisioning, updating, and integration of resources, such as compute, storage, and networking infrastructure.</p> <p>HashiCorp verification of the HPE OneView Terraform Provider based on Terraform v0.13 permits HPE code to be made available in the <a href="https://registry.terraform.io/providers/HewlettPackard/oneview/latest">Terraform Registry</a>. The registry allows for the Terraform Provider for HPE OneView to be initiated directly from the registry maintained by HashiCorp by introducing the provider source attribute in Terraform. The verification process also ensures that provider code is from a reliable source, making automated installation a secure process. In the case of the HPE OneView Provider, GPG (GNU Privacy Guard) encryption is used to digitally sign the HPE code.</p> <p>Terraform v0.13 is a major update that includes dozens of improvements and features spanning the breadth and depth of Terraform’s functionality. One of the major changes in Terraform 0.13 is HCL2, the second generation of HashiCorp Configuration Language. HCL2 introduces Rich Data Types as a means to describe more complex structures with your Terraform Modules. </p> <p>The Terraform Provider for HPE OneView now uses Go Modules for dependency management and vending. The Terraform Provider for HPE OneView is also an upgrade to the Terraform Plugin SDK. More details about the Terraform Plugin SDK can be found <a href="https://www.terraform.io/docs/extend/guides/v1-upgrade-guide.html">here</a>.</p> <p>Terraform Provider for HPE OneView supports several installation paths. It can be installed from Source, a Docker container, or the Terraform Registry. HPE has produced an <a href="https://github.com/HewlettPackard/terraform-provider-oneview/tree/master/Migration%20Support">Installation and User Guide</a> to simplify migration from HPE OneView Providers based on previous versions of Terraform. The guide provides step-by-step instructions for each installation path.</p> <p>For more information:</p> <ul> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview">Code Repository and Examples</a> </li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">HPE OneView SDK Docker Image for Terraform</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/tree/master/Migration%20Support">Installation and User Guide</a></li> <li><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a></li> <li><a href="https://registry.terraform.io/providers/HewlettPackard/oneview/latest">Terraform Registry</a></li> </ul> <!--EndFragment-->https://developer.hpe.com/python-how-to-use-odbc-to-connect-hpe-nonstop-sql-mx/https://developer.hpe.com/python-how-to-use-odbc-to-connect-hpe-nonstop-sql-mx/Tue, 04 May 2021 06:27:49 GMT<p>Hello World! In this tutorial, I will show you how Python can execute queries on the HPE NonStop SQL/MX Database using pyodbc and the NonStop SQL/MX ODBC driver.</p> <p>This tutorial assumes that NonStop ODBC 3.x Unicode driver has already been installed. Check out the <a href="https://support.hpe.com/hpesc/public/docDisplay?docId=a00045523en_us&#x26;docLocale=en_US">NonStop ODBC/MX Client Drivers User Guide</a> for more information on the driver.</p> <p>This tutorial also assumes that on your host, NonStop SQL/MX has been installed, MXCS is running, and a MXCS data source has been added and started. Check with your administrator for the IP address, port number etc. (If you’re the administrator check out this <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=emr_na-a00090054en_us">manual</a>.)</p> <p><a href="https://github.com/shaniceabigail/python-odbc-nonstop-sqlmx">Link to source code.</a> <a href="https://github.com/shaniceabigail/python-odbc-nonstop-sqlmx"></a></p> <p>Let’s get started!</p> <h1>Getting Started</h1> <h2>Python and pip</h2> <p>First, <a href="https://www.python.org/downloads/">download Python,</a> if you have not already done so. You can check to see if your machine already has Python installed by running the command below in the Windows command prompt. It should return Python with its version number, if it has been installed properly.</p> <pre><code class="language-markdown">C:\> python --version Python 3.9.0 C:\> </code></pre> <p>Next, install pip (pip is the Python package manager that we will be using). Download <a href="https://bootstrap.pypa.io/get-pip.py">get-pip.py</a> on your laptop. Navigate to the folder containing the file and run the following command.</p> <pre><code class="language-markdown">C:\Downloads> python get-pip.py </code></pre> <p>Once it’s complete, double check to see if pip has been installed by running the following command in Windows.</p> <pre><code class="language-markdown">C:\> py -m pip --version pip 20.3.1 from C:\Miniconda3\lib\site-packages\pip (python 3.9) C:\> </code></pre> <h2>Installing pyodbc</h2> <p>pyodbc is an open source Python module that makes it simple to access ODBC databases. It implements the <a href="https://www.python.org/dev/peps/pep-0249">DB API 2.0</a> specification (the standard interfaces for Python to access a database), but is packed with even more Pythonic convenience. TLDR; it helps you to access the databases that use ODBC drivers.</p> <p>You can read more about pyodbc on its <a href="https://github.com/mkleehammer/pyodbc/wiki">Github wiki page</a>.</p> <p>Use pip to install pyodbc.</p> <pre><code>C:\> pip install pyodbc </code></pre> <h2>Configuring your ODBC</h2> <p>In order for pyodbc to recognize the ODBC driver and data source to use, it will check with the ODBC Data Source Administrator on your Windows machine. Here’s how you can configure a new data source that will use the HPE NonStop SQL/MX ODBC driver.</p> <ol> <li>Search and open the ODBC Data Source Administrator on your machine.</li> <li>Select the “Add” button to create a new data source.</li> </ol> <p><img src="https://miro.medium.com/max/594/1*PWpQ3yfwfB08ITElY9IHRQ.png" alt=""></p> <h3>ODBC Data Source Administrator</h3> <ol start="3"> <li>Select the HPE NonStop™ ODBCMX 3.x Unicode driver, and click “Finish”.</li> </ol> <h3>ODBC Data Source Administrator — Create New Data Source</h3> <ol start="4"> <li>A new window should pop up. Write a Data Source Name for this data source that you want to connect to. Note: The data source names <strong>must match</strong> between those defined to MXCS on the database server and the client PCs; otherwise the connection will <strong>FAIL.</strong></li> </ol> <p><img src="https://miro.medium.com/max/563/1*n48eArrYZ1moeC432v2gZg.png" alt=""></p> <h3>Data Source Name and Description</h3> <ol start="5"> <li>Insert the IP address of the NonStop SQL/MX database, as well as the port number that has been opened up for connections.</li> </ol> <p><img src="https://miro.medium.com/max/564/1*4FWFtcvDezDej8zjf90jhg.png" alt=""></p> <h3>IP address and Port number</h3> <ol start="6"> <li>Insert the catalog and schema that you want to connect to.</li> </ol> <p><img src="https://miro.medium.com/max/564/1*EPl5NDJsUHZJd6PI-U4eRA.png" alt=""></p> <h3>Catalog and Schema</h3> <ol start="7"> <li>Leave the “Translate DLL” portion (DLL Name and Option), and the Localization (Replacement Character) blank for now.</li> </ol> <p><img src="https://miro.medium.com/max/564/1*7BZPU6fI38qaTcXR6IrIag.png" alt=""></p> <h3>Leave blank</h3> <ol start="8"> <li>We will not be doing any tracing in this data source, so leave the settings as the default, and click finish.</li> </ol> <p><img src="https://miro.medium.com/max/564/1*DtYFoVsOh4fTpAHwG1n01w.png" alt=""></p> <p>Test the connection, and click “OK”. You should see that the data source has been added to the list.</p> <p>Alright, now it’s time to code!</p> <h1>The code:</h1> <h2>The setup</h2> <p>Create a new .py file in any of your favourite text editors — mine is VSCode.</p> <p>Import the Python package into the script.</p> <p>Add your Data Source Name, UID (User ID) and Password (PWD) in the fields.</p> <p>Finally, set the decoding and encoding parameters for the connection. These are database specific. NonStop SQL/MX supports “iso-8859–1”, but this varies with the database you’re using. (We set this up for good measure — so copy/paste the parameters from the code below for <strong>NonStop SQL/MX.</strong>)</p> <p>This is what you should have so far.</p> <pre><code class="language-python">import pyodbc conn = pyodbc.connect('DSN=[DATA SOURCE NAME];UID=[USER];PWD=[PASSWORD]') conn.setdecoding(pyodbc.SQL_CHAR, encoding='iso-8859-1') conn.setdecoding(pyodbc.SQL_WCHAR, encoding='iso-8859-1') conn.setencoding(encoding='iso-8859-1') </code></pre> <h2>Now onto executing an SQL query in the database.</h2> <p>Create a cursor variable and execute the SQL statement that you would like to have in your database.</p> <p>Note that <strong>YOU HAVE TO COMMIT THE TRANSACTION</strong> if you make an insert or update a table statement in the Python script. You can insert the commit at the end of the set of updates or inserts.</p> <p>The act of committing the transactions is how we make sure that the set of transactions / executions are properly executed, and data integrity is maintained.</p> <pre><code class="language-python">cursor = conn.cursor() cursor.execute('INSERT INTO CATALOG.SCHEMA.TABLE VALUES (VALUE1, VALUE2)') # makes sure that insert statement is a committed transaction in NonStop SQL/MX database conn.commit() </code></pre> <h2>Seeing the result / select statement</h2> <p>You can execute the select statement using the cursor and print the values in the cursor.</p> <pre><code class="language-python"># prints table to make sure that data was updated cursor.execute('SELECT * FROM CATALOG.SCHEMA.TABLE') for row in cursor: print(row) </code></pre> <p>And there you have it!</p> <p>The Python script should be able to insert, update, create, and select. Alright, that’s it for now. I hope you’ll have fun coding with the NonStop SQL/MX database using information you learned in this tutorial!</p> <p>For more interesting posts and tutorials, keep checking back on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a>.</p>https://developer.hpe.com/2021-June-01/https://developer.hpe.com/2021-June-01/Mon, 03 May 2021 06:00:00 GMThttps://developer.hpe.com/new-look-new-brains-all-the-tools/https://developer.hpe.com/new-look-new-brains-all-the-tools/Fri, 16 Apr 2021 11:24:13 GMT<center><img src="/img/full-reveal-top-of-page-image1.jpg" width="800" height="453"></center> <p>The HPE DEV team is excited to present its revamped <a href="https://developer.hpe.com/">HPE Developer Community web portal</a>. Featuring a fresh look and feel, with easier navigation, it provides the resources you need to design and build software experiences that harness the most value from your data. With a new backend system and new design, it’s easier to use and contribute to.</p> <h2>A rich set of tools</h2> <p>A key area of the portal is our <a href="https://developer.hpe.com/platforms">Platforms section</a>. Here, you can find APIs, GitHub repositories, and many of the other resources we make available for developers, designers, data scientists, and architects. We host numerous platforms here, including the HPE Ezmeral Container Platform, HPE Ezmeral Data Fabric, HPE GreenLake, SPIFFE and SPIRE projects, Chapel, Grommet, Aruba Developer Hub, HPE Nimble Storage, HPE OneView, iLO RESTful API, and many others.</p> <center><img src="/img/platforms-3-image2.jpg" width="800" height="225"></center> <p>Make sure you check out the new <a href="https://developer.hpe.com/skillup">Skill Up</a> section. It provides you with easy access to our popular <a href="https://developer.hpe.com/blog/munch-and-learn">Munch &#x26; Learn Technology Talks</a>, <a href="/hackshack/workshops">Workshops-on-Demand</a>, and other learning opportunities.</p> <center><img src="/img/skill-up-v2-image3.jpg" width="800" height="427"></center> <p>Blog posts continue to be a staple of the portal, with many of the more recent posts covering topics like AI, machine learning, how to stream data and capture it, and how to efficiently manage your hybrid cloud environment. The new <a href="https://developer.hpe.com/blog">blog site</a> makes it easier to find the articles you’re looking for. We believe that sharing expertise is a great way to move technology forward and invite you to browse through our extensive library of tutorials and articles to learn new ways to do things.</p> <center><img src="/img/blogs-v2-image4.jpg" width="800" height="518"></center> <h2>Easy access</h2> <p>Our new portal provides easy access to the <a href="/hackshack/">HPE DEV Hack Shack</a>, a place for fun and learning. During events, the Hack Shack is a place where you can connect with experts in HPE and open source technologies and learn more from the teams behind these products. It’s where attendees can come to participate in coding challenges, hands-on workshops, and entertaining games. Afterwards, it extends the event experience by allowing you to continue to revisit replays of workshops and explore the hands-on workshops at your own pace.</p> <center><img src="/img/home-page-2-bis.png" width="627" height="720"></center> <p>From the home page, you can explore the HPE Design System, a set of resources for designing great apps and websites that integrate seamlessly with HPE platforms. You can also jump from there to our <a href="/hackshack/workshops">HPE DEV Workshops-on-Demand</a> and our monthly community meetups, the <a href="https://developer.hpe.com/campaign/munch-and-learn/">Munch &#x26; Learn Technology Talks</a>, where you can interact with experts regarding popular new technologies and get your questions answered. You can also directly access the <a href="https://www.hpe.com/us/en/open-source.html">HPE Open Source portal</a> from our top page directory.</p> <p>Easily access past issues of our HPE DEV Community Newsletter in our <a href="https://developer.hpe.com/newsletter-signup">Newsletter Archive</a>. Make sure you’re signed up to get any subsequent issues we send out. From the<a href="https://developer.hpe.com/community"> Community</a> page you’ll be able to connect with us on our <a href="https://slack.hpedev.io/">HPE DEV</a> Slack channel and follow us on Twitter. You can start and participate in discussions on the <a href="https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/bd-p/ezmeral-software-platform">HPE Ezmeral Software Forum</a> as well.</p> <center><img src="/img/community-v2-image5.jpg" width="800" height="464"></center> <h2>We’re all developing something. Come join us in making the future.</h2> <p>We hope you’re as excited as we are about the new portal . Find us at hpedev.io.</p>https://developer.hpe.com/kubecon-cloudnativecon-europe-be-involved-via-this-virtual-opportunity/https://developer.hpe.com/kubecon-cloudnativecon-europe-be-involved-via-this-virtual-opportunity/Thu, 15 Apr 2021 14:50:34 GMT<p><img src="/img/kubecon-eu-2021-1.jpg" alt=""></p> <p>As stated in last year’s KubeCon keynote; “our lives have gone remote – our challenges have gone global.” The <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a> (CNCF) will address this challenge through its flagship conference, <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon | CloudNativeCon EU</a> delivered via a virtual platform May 4-7, 2021. And Hewlett Packard Enterprise (HPE) will once again be there, excited for the opportunity to show what it offers in the areas of containers and Kubernetes (k8s)!</p> <p>New this year, HPE Principal Software Engineer Daniel Feldman, will speak on how to securely bridge cloud-native and traditional workloads with SPIRE. Daniel is a major contributor to the SPIRE project, an open source zero trust security layer for cloud services. With a strong background in security, Daniel will cover the challenges organizations face when going cloud native, and how the CNCF’s SPIRE project provides the solution to help establish secure service identities across an organization. Make sure you check out his lightening talk on Tuesday, May 4th at 15:00 CEST. Find it on <a href="https://kccnceu2021.sched.com/event/igUc">the schedule here</a>.</p> <h2>Discover what’s new in our virtual Hack Shack!</h2> <p>HPE’s presence at the event will again include the <a href="https://developer.hpe.com/">HPE Developer Community</a> Hack Shack.<a href="https://developer.hpe.com/"></a> In the <a href="/hackshack/">HPE DEV Hack Shack,</a> developers, designers, and data scientists have the opportunity to connect with HPE subject matter experts and collaborate with them to accelerate innovation using open source and HPE technologies. It’s a unique place designed to give virtual events a more personal touch and extend the experience beyond the event. Here, you’ll find lots of opportunities to learn and have fun! Make sure you check out the HPE DEV Treasure hunt for an opportunity to win HPE DEV swag!</p> <p>To get a complete picture of what’s happening in the virtual Hack Shack, take a <a href="https://vimeo.com/444872340">tour now</a>. Here’s a snapshot of what you will find:</p> <ul> <li> <p><a href="/hackshack/workshops">WORKSHOPS-ON-DEMAND:</a> Access over a dozen free, on-demand hands-on training courses that include <em><strong>Building a dynamic Machine Learning pipeline with KubeDirector</strong></em> and <em><strong>Using Kubernetes CSI with HPE Ezmeral Container Platform</strong></em>. Using a Jupyter Notebook environment, you’ll have the opportunity to gain hands-on experience across different technologies, like KubeDirector and the HPE Ezmeral Container Platform REST API. You’ll also find numerous replays of talks we’ve given at different events and a complete selection of material on the HPE Ezmeral Platform, an enterprise-grade platform used to deploy Kubernetes at scale for a wide range of use cases on bare metal or VMs.</p> </li> <li> <p><a href="/hackshack/arcade">TREASURE HUNT:</a> New this year, our scavenger-hunt style challenge offers an opportunity for you to check out all the resources that are available on the HPE Developer Community website and the Hack Shack. Be one of the first people to answer all the questions correctly and win an HPE DEV hat!</p> </li> <li> <p><a href="/hackshack/replays">REPLAYS:</a> Augmenting our Workshops-on-Demand, we’ve posted replays of many of the technical workshops we’ve offered live in the past. View them to learn more about the <a href="/hackshack/replays/1">HPE Ezmeral Container Platform</a>, <a href="/hackshack/replays/5">SPIFFE and SPIRE authentication</a>, and the <a href="/hackshack/replays/2">HPE Container Storage Interface for Kubernetes</a>.</p> </li> <li> <p><a href="/hackshack/ezmeral">HPE EZMERAL:</a> We’ve made it easy for you to find detailed information on the <a href="https://developer.hpe.com/platform/hpe-ezmeral-container-platform/home">HPE Ezmeral Container Platform</a>, <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home">HPE Ezmeral Data Fabric</a>, <a href="https://www.hpe.com/us/en/solutions/machine-learning-operations.html">HPE Ezmeral ML Ops</a>, along with other products in this innovative set of software that can be deployed on any cloud, on any hardware, and is 100% open source Kubernetes.</p> </li> <li> <p><a href="/hackshack/arcade">ARCADE:</a> In our arcade, you’ll find Hack Shack Attack! Give our popular retro-style video game a try and compete with your friends for the highest score. Here, you can also download stickers, Zoom backgrounds, and cool artwork to use on your social channels.</p> </li> <li> <p><a href="/hackshack/community">COMMUNITY:</a> We invite you to join and contribute your expertise on our blog or deliver an on-demand workshop. Connect with others in the community via the <a href="https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/bd-p/ezmeral-software-platform#.YGzDjuhKg2w">HPE Ezmeral Software Forum</a>, or our <a href="https://slack.hpedev.io/">Slack </a>and <a href="https://twitter.com/HPE_Developer">Twitter</a> channels to start conversations and get answers to questions. Sign-up for our <a href="https://developer.hpe.com/newsletter-signup">HPE DEV Newsletter</a> to stay up-to-date on the newest blog posts and tutorials.</p> </li> <li> <p><a href="https://developer.hpe.com/skillup">SKILL UP:</a> There are even more opportunities to get together and learn when you’re part of the HPE DEV Community. From the Hack Shack, hop on over to the HPE DEV portal where you’ll find our monthly Munch &#x26; Learn gatherings. These free, one-hour meetups are designed to provide developers and data scientists the opportunity to meet and engage with experts in popular HPE and open source technologies.</p> </li> </ul> <p>As always, the<a href="https://developer.hpe.com/community"> HPE DEV community</a> is excited to have an opportunity to connect with other technologists at an event such as this. If you’re planning on attending, make sure you stop on by and say “Hi” in the Hack Shack by dropping us a note in our <a href="https://slack.hpedev.io/">Slack Channel</a>.</p>https://developer.hpe.com/hpe-developer-community-meets-a-changing-world-head-on/https://developer.hpe.com/hpe-developer-community-meets-a-changing-world-head-on/Wed, 14 Apr 2021 10:17:45 GMT<h2>The world is changing</h2> <p>When have those words never been true? The only real constant in life <strong>is</strong> change. <strong>How</strong> you deal with change is what defines how you will survive – and thrive. Consider the challenges posed by social distancing to any community of technologists. Take the HPE Developer Community, for example. We thrive on connecting with each other, discussing problems, and collaborating to build solutions. As much as we wish we could “get back to normal” and live in a post-COVID world where we can greet each other in the Hack Shack, there are things that will never go back… nor, perhaps, should they.</p> <p>As I said, <strong>how</strong> you deal with change defines your future success. By learning to adapt to our current circumstances, we have found ways to reach out to others as never before; more efficiently, economically, and engagingly. By confronting the challenges posed by social distancing head on, the HPE DEV team that leads the Developer Community has come up with some very innovative ways to connect and continue our mission – to Build, Communicate, and Collaborate. This includes developing a virtual <a href="/hackshack/">Hack Shack</a>, creating <a href="/hackshack/workshops">on-demand technical workshops</a>, offering <a href="https://developer.hpe.com/campaign/munch-and-learn">online meetups</a>, and redesigning our <a href="https://developer.hpe.com/">HPE Developer web portal</a>.</p> <h2>Morphing the Hack Shack</h2> <p>When COVID shut down the first tradeshows, it became obvious that going virtual was the only real option. Digital platforms were quickly made available. These helped companies show what they wanted to present, but from an event attendee’s perspective, they really weren’t all that engaging. Virtual platforms did make events easier to attend, as they could be streamed worldwide throughout the day. And they saved people a lot of money, since many events were free and no one had to travel. But the ability to connect and learn from each other was lacking.</p> <p>One of the HPE DEV team’s first challenges was to figure out a way to deliver the hands-on learning sessions HPE DEV had become known for through its physical presence in the Hack Shack. The Hack Shack was a friendly place where event attendees could relax, attend hands-on coding sessions, and mingle with like-minded technologists to pick each other’s brain. It was a place where you came to learn.</p> <p>As developers, designers, and owners of our own HPE DEV Portal, the most obvious answer to making the Hack Shack virtual was to actually create an on-line Hack Shack. Our resident ace designer, Chris Carlozzi, created one of the coolest sites you could imagine. One of the most popular attractions at the Hack Shack was our Hack Shack Attack game, which was already available online, so that was easily integrated. We made stickers available that people could download and offered prizes for high scores in the game.</p> <center><img src="/img/hpedev-world-is-changing-image1-small.png" height="542" width="500"></center> <p>But what about the coding sessions? These had been tremendously popular during our live events. Didier Lalli, HPE DEV Technology lead, and Frederic Passeron, our resident education evangelist, came up with a way to deliver the workshops in a way where students could interact with the code online through the use of Jupyter Notebooks. This gave birth to a whole set of <a href="/hackshack/workshops">Workshops-on-Demand</a>. The workshops became so popular that people began asking for them even when there wasn’t an event. Students returned surveys giving the workshops glowing reviews, explaining how happy they were to be able to work with the technologies through such a hands-on approach. (For more information on how we did this, read Frederic Passeron’s post <a href="https://developer.hpe.com/blog/from-jupyter-notebooks-as-a-service-to-hpe-dev-workshops-on-demand/">From Jupyter Notebooks-as-a-Service to HPE DEV Workshops-on-Demand</a>). Now, the workshops are not only available in the Hack Shack, but you can also find them through the <a href="https://hpedemoportal.ext.hpe.com/login">HPE Demo Portal</a> and through the <a href="https://developer.hpe.com/skillup/">Skill Up</a> section of the HPE DEV Portal.</p> <h2>Zooming in on Technology Talks</h2> <p>In addition to everything else the pandemic has done, it has certainly made Zoom a household name. And through its use, we’ve been able to find other ways to gather and collaborate. There’s a huge knowledge base in HPE. It’s the job of the HPE DEV team to connect subject matter experts with others in the community who can benefit from that knowledge, and Zoom has helped us to do that.</p> <p>In the past year, we’ve connected with more and more experts in the fields of data fabric, Kubernetes, containers, and open source technologies. Industry luminaries (e.g. Ted Dunning, Ellen Friedman, and Nigel Poulton), open source innovators (e.g. Umair Khan, Brad Chamberlain, Kartik Mathur, and Agustin Fayo), and brilliant technologists (i.e. Tom Phelan and Doug Cackett) have helped us inform others in the community about the nuances of these technologies by presenting in our forums and adding to our workshop library.</p> <p>Our newest offering, the <a href="https://developer.hpe.com/blog/munch-and-learn">Munch &#x26; Learn Technology Talks</a>, are monthly meetups where we invite everyone to gather and hear from these technologists. These are great opportunities for you to come and get your questions answered straight from the experts. And, because we’re a fun sort of group, we also encourage you to bring a munchie/snack, take a picture of it and share with the group via our Slack channel. You can even share a recipe, if you’d like.</p> <h2>Change Hits the Portal</h2> <p>As we added more and more to our online presence, we found that the existing HPE DEV Portal no longer served as a true representation of who the HPE DEV team is and what we do. We needed a better way for our community to find the things they were looking for, including the technology talks, the workshops, and other training resources we offer. We wanted to offer people easier ways to find us and connect through <a href="https://slack.hpedev.io/">Slack</a>, <a href="https://twitter.com/HPE_Developer">Twitter</a>, and the <a href="https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/bd-p/ezmeral-software-platform">HPE Ezmeral Software Forum</a>. Even finding older newsletters tended to be a chore, and that needed to be fixed.</p> <center><img src="/img/hpedev-world-is-changing-v2-portal-reveal-575-by-375.jpg" width="575" height="375" ></center> <p>So, we’ve updated the <a href="https://developer.hpe.com">HPE DEV Portal</a> and given it a new, fresh and clean look. It’s a lot easier to navigate. Here, you’ll find all the resources you need to design and build the best possible software experiences that harness the most value from your data. Blog posts continue to be a staple of the portal, with many of the more recent posts covering topics like AI, machine learning, how to stream data and capture it, and how to efficiently manage your hybrid cloud environment. You’ll also be able to access the brand new <a href="https://www.hpe.com/us/en/open-source.html">HPE Open Source project website</a> through our top navigation.</p> <h2>Join Us in Moving Forward</h2> <p>One of my favorite Comcast Business commercials extols the virtue of moving forward. It acknowledges how businesses that were previously thriving are now challenged with figuring out how to deal with a disastrous situation. The commercial asks “How do you bounce back? You don’t. You bounce forward.” And, at the end, it points out that we need to do this together. The HPE DEV team is excited by the changes we’ve made as a team and as a community – and how these changes are helping us to move forward.</p> <p>If you haven’t taken advantage of the HPE Developer Community yet, consider taking a closer look now and in the coming months. There’s a broad set of resources there that you can take advantage of. It costs you nothing to join. You just need to participate, which can be as simple as signing up for our <a href="https://developer.hpe.com/newsletter-signup">monthly newsletter</a>. Or, post a question on our <a href="https://slack.hpedev.io/">Slack Channel</a> and find out how quickly you can get an answer. You might want to start a discussion on the <a href="https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/bd-p/ezmeral-software-platform">HPE Ezmeral Software Forum</a>. Or, follow us on Twitter at our new <a href="https://twitter.com/HPE_Developer">Twitter</a> handle @HPE_DEV. I think you’ll find a lot of value in connecting and collaborating with us.</p>https://developer.hpe.com/how-to-integrate-chef-automation-with-hpe-greenlake-for-private-cloud/https://developer.hpe.com/how-to-integrate-chef-automation-with-hpe-greenlake-for-private-cloud/Tue, 13 Apr 2021 17:25:38 GMT<p><img src="/img/gettyimages-521980823_16x9_1600_0_72_rgb.jpg" alt=""></p> <br /> <hr> <p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>HPE GreenLake for private cloud is one of the cloud services powered by the HPE GreenLake Central platform. This service provides a cloud experience to manage virtual machines (VMs) in your on-premises, pay-per-use datacenter. It is an integrated solution comprised of HPE optimized hardware and software, fully managed by HPE. The solution provides rich application management capabilities for cloud-native and traditional applications along with a self-service portal and integrates with a lot of popular automation tools such as Ansible, Chef, Puppet, and others. This article explains how to integrate your existing Chef automation platform, Chef Infra, with HPE GreenLake for private cloud to help improve efficiency and minimize errors caused by manual configuration. And the tutorial walks you through the step-by-step process for two use case scenarios:</p> <ul> <li> <p>Scenario 1: Add Chef automation integration, and provision a new application instance (i.e. Nginx) bootstrapped to an integrated Chef Infra server using HPE GreenLake for private cloud self-service user interface (UI).</p> </li> <li> <p>Scenario 2: Bootstrap an existing VM instance to the integrated Chef infra server using the HPE GreenLake for private cloud automation feature Tasks.</p> </li> </ul> <p>Before we dive into the use cases, let me give you an overview of Chef and its prerequisites before any integration with HPE GreenLake for private cloud.</p> <h2>Chef overview</h2> <p>Chef is one of the most widely adopted open source automation solutions. Chef Infra is a powerful automation platform that transforms infrastructure into code. It provides for both infrastructure and application automation for physical or virtual machines, and helps in reducing manual and repetitive tasks. Chef Infra works on a three-tier client server model: Chef workstation, Chef Infra server, and Chef Infra client nodes.</p> <p><strong>Chef workstation:</strong> This is where a user develops configuration files, such as Chef recipes and cookbooks, and uploads them to the Chef Infra server.</p> <p><strong>Chef Infra server:</strong> This acts a hub for the configuration data. It stores cookbooks, policies that are applied to Chef Infra client nodes, and metadata that describes each registered Chef node.</p> <p><strong>Chef Infra client nodes:</strong> Chef nodes are registered and managed by Chef Infra server. Chef client is installed on each node, which helps in setting up the communication between the Chef Infra server and Chef node. Nodes use Chef client to ask the Chef server for configuration details, such as recipes, templates, and file distributions. Chef client then does the configuration work on the nodes.</p> <p>See the <a href="https://www.tutorialspoint.com/chef/index.htm">Chef Tutorial</a> for more information on Chef architecture.</p> <h2>Chef integration prerequisites with HPE GreenLake for private cloud</h2> <p>To integrate Chef with HPE GreenLake for private cloud, the following pre-requisites are assumed:</p> <ul> <li> <p>You have a Chef Infra server: It can be hosted on either a public or private network. For a Chef Infra server hosted in a private network, make sure it is reachable from the HPE GreenLake Central platform.</p> </li> <li> <p>You have configured the organization, organization validator key, users, and user key on a Chef Infra server.</p> </li> <li> <p>You have uploaded a sample cookbook <strong>mydocker</strong> to the Chef Infra server. Refer to this <a href="https://medium.com/@pierangelo1982/cooking-docker-nginx-with-chef-server-95f179aa17ca">blog post</a> to create the sample cookbook. The cookbook has a single recipe <strong>default.rb</strong> that will be referred as <strong>recipe[mydocker]</strong> in the Chef runlist. The cookbook installs a docker environment on the VM, and then installs a Nginx docker container. A Chef runlist defines all the information necessary for Chef to configure a node into desired state.</p> </li> </ul> <pre><code class="language-yaml"># # Cookbook:: mydocker # Recipe:: default # # Copyright:: 2021, The Authors, All Rights Reserved. docker_service 'default' do action [:create, :start] end # Create Docker Service directory directory '/etc/systemd/system/docker.service.d' do owner 'root' group 'root' mode '0755' action :create end # Creating Docker Proxy Configuration cookbook_file '/etc/systemd/system/docker.service.d/proxy.conf' do source 'proxy.conf' mode "0644" action :create end docker_service 'default' do action :restart end # Pull latest image docker_image 'nginx' do tag 'latest' action :pull end # Run container exposing ports docker_container 'my_nginx' do repo 'nginx' tag 'latest' port '85:80' volumes "/home/docker/default.conf:/etc/nginx/conf.d/default.conf:ro" volumes "/home/docker/html:/usr/share/nginx/html" end # create file default.conf for volumes docker template "/home/docker/default.conf" do source "default.conf.erb" #notifies :reload, "service[default]" end # create file index.html for volumes docker template '/home/docker/html/index.html' do source 'index.html.erb' variables( :ambiente => node.chef_environment ) action :create #notifies :restart, 'service[httpd]', :immediately end </code></pre> <h2>Scenario 1: Add Chef automation integration and provision a new application instance (Nginx) bootstrapped to the integrated Chef Infra server using HPE GreenLake for private cloud self-service UI</h2> <p>For this scenario, we need to integrate Chef automation with the HPE GreenLake for private cloud first, then we provision a new application instance (i.e. Nginx) bootstrapped to the integrated Chef Infra server.</p> <h3>Follow the below steps to add Chef automation integration in the HPE GreenLake for private cloud.</h3> <ol> <li>In the HPE GreenLake for private cloud main menu dashboard, navigate to <strong>Administration > Integrations</strong></li> </ol> <p><img src="/img/glpc-chef-integration-image1.png" alt="" title="Image1"></p> <ol start="2"> <li>Click the green <strong>+NEW INTEGRATION</strong> drop-down menu, and select integration type <strong>Chef</strong></li> </ol> <p><img src="/img/glpc-chef-integration-image2.png" alt="" title="Image2"></p> <ol start="3"> <li> <p>Populate the following fields and save changes</p> <ul> <li>Name: Name of the Chef integration; for example, <strong>Local-Chef-Server</strong></li> <li>Chef Endpoint: URL of Chef Infra server API endpoint in <a href="https://api.example.com">https://api.example.com</a> format. Do not add /organization/xxxx here, which is populated in the Chef Organization field. In this example, <strong><a href="https://chefserver.localdomain">https://chefserver.localdomain</a></strong> is used.</li> <li>Chef Version: Chef client version, which needs to be installed in the nodes. Use 16.1.X or greater. Version can be changed to use a different/more recent version of Chef.</li> <li>Chef Organization: Chef server organization</li> <li>Chef User: Chef Infra server user</li> <li>User Private Key: The private key of the user with access to this Chef Infra server</li> <li>Organization Validator: Validator key for the organization</li> <li>DataBags: Optional. Add it if it is configured in the Chef Infra server</li> </ul> </li> </ol> <p><img src="/img/glpc-chef-integration-image3-bis.png" alt="" title="image3"></p> <h3>Create a new infrastructure group for Chef integration.</h3> <ol> <li>From HPE GreenLake for private cloud main menu dashboard, navigate to <strong>Infrastructure > Groups</strong></li> </ol> <p><img src="/img/glpc-chef-integration-image4.png" alt="" title="Image4"></p> <ol start="2"> <li>Click the green <strong>+CREATE</strong> icon to add a new group with the name of <strong>ManagedChef</strong>. Expand the <strong>Advanced Options</strong> section in the <strong>Config Management</strong> field, and select the previously configured Chef Integration <strong>Local-Chef-Server</strong>.</li> </ol> <p><img src="/img/glpc-chef-integration-image5.png" alt="" title="Image5"></p> <p>Save changes. The added Chef integration is now available for use in HPE GreenLake for private cloud during instance provisioning. Please note that one infrastructure group can only associate with one Chef server.</p> <h3>Provision the new instance.</h3> <p>With Chef integration added to an infrastructure group, when users provision a new instance into the infrastructure group, a <strong>Chef</strong> section will appear in the <strong>Configure</strong> section of the provisioning wizard. By default, Chef is enabled, but it can be disabled by expanding the <strong>Chef</strong> section and unchecking <strong>Enable Chef</strong>. Follow the steps below to provision the new instance using <strong>CREATE INSTANCE</strong> wizard.</p> <ol> <li>In the wizard <strong>GROUP</strong> page, select the previous configured <strong>ManagedChef</strong> infrastructure group, enter the instance name <strong>Demo2</strong>, select environment, and click next.</li> </ol> <p><img src="/img/glpc-chef-integration-image6.png" alt="" title="Image 6"></p> <ol start="2"> <li>In the <strong>CONFIGURE</strong> menu, expand the <strong>Chef</strong> section and update the required fields as shown in the screen capture below. In the <strong>CHEF RUNLIST</strong> field, enter the Chef recipe <strong>recipe[mydocker]</strong> from the cookbook you previously uploaded to your Chef Infra server in the Prerequisite section. If the <strong>CHEF RUNLIST</strong> field is left empty, the instance will be just bootstrapped to the Chef Infra server.</li> </ol> <p><img src="/img/glpc-chef-integration-image7.png" alt=""></p> <ol start="3"> <li>Review the instance configuration and complete the installation.</li> </ol> <p><img src="/img/glpc-chef-integration-image8.png" alt="" title="Image 8"></p> <ol start="4"> <li>On successful completion of Instance provisioning, the instance <strong>Demo2</strong> is bootstrapped to the Chef server integrated to the selected Group. Expanding the instance <strong>History</strong> tab shows the bootstrap task status.</li> </ol> <p><img src="/img/glpc-chef-integration-image9.png" alt="" title="Image 9"></p> <ol start="5"> <li>You can also confirm the instance <strong>Demo2</strong> from Chef-Manage dashboard of the integrated Chef Infra server. Chef-Manage needs to be installed explicitly on the Chef Infra server. The screen below shows that instance <strong>Demo2</strong> is bootstrapped to the Chef server.</li> </ol> <p><img src="/img/glpc-chef-integration-image10.png" alt="" title="Image 10"></p> <p>Based on the recipe specified in the <strong>CHEF RUNLIST</strong> defined in the provisioning wizard, the Chef client on the newly created instance <strong>demo2</strong> pulled the recipe from the Chef server and configured an Nginx docker container on the instance.</p> <p><img src="/img/glpc-chef-integration-image11.png" alt="" title="Image 11"></p> <h2>Scenario 2: Bootstrap an existing VM instance to the integrated Chef Infra server using the HPE GreenLake for private cloud automation feature <strong>Tasks</strong></h2> <p>HPE GreenLake for private cloud provides an option to bootstrap existing VM instances to Chef Infra server. This section describes a step-by-step process you can use to achieve that.</p> <ol> <li>The first step is to create a Chef bootstrap task. On the HPE GreenLake for private cloud main menu screen, navigate to <strong>Provisioning > Automation > Tasks</strong>, and create a Chef bootstrap task. A task is an individual automation element; for example, a script or a Chef cookbook. Select <strong>TYPE</strong> as <strong>Chef bootstrap</strong> and choose the previously integrated Chef Infra server. In the <strong>RUN LIST</strong> field, enter <strong>recipe[mydocker]</strong> to refer the recipe from the cookbook you previously uploaded to your Chef Infra server.</li> </ol> <p><img src="/img/glpc-chef-integration-image12.png" alt="" title="Image 12"></p> <ol start="2"> <li>To bootstrap any VM instances with this task, on the HPE GreenLake for private cloud main menu, navigate to <strong>Provisioning > Instances</strong>, and select the VM instance that needs to be bootstrapped. Select <strong>Actions > Run</strong> Task. The screen below shows the VM instance <strong>Demo</strong> detail page.</li> </ol> <p><img src="/img/glpc-chef-integration-image13.png" alt="" title="Image 13"></p> <ol start="3"> <li>Select the <strong>chef-bootstrap</strong> task created previously and execute it by clicking <strong>EXECUTE</strong>.</li> </ol> <p><img src="/img/glpc-chef-integration-image14.png" alt="" title="Image 14"></p> <ul> <li>Task will start the execution on the instance.</li> </ul> <p><img src="/img/glpc-chef-integration-image15.png" alt="" title="Image 15"></p> <ol start="4"> <li>On completion, the instance status is green and the task status can be seen in <strong>History</strong> tab.</li> </ol> <p><img src="/img/glpc-chef-integration-image16.png" alt="" title="Image 16"></p> <p>The instance is now bootstrapped as a Chef client registered in the Chef Infra server. The recipe is run as specified in the <strong>Run List</strong> from Chef-Manage dashboard of the integrated Chef Infra server, shown in screenshot below.</p> <p><img src="/img/glpc-chef-integration-image17.png" alt="" title="Image 17"></p> <h2>Summary</h2> <p>Chef Infra is a powerful automation platform that transforms infrastructure into code. By integrating it with HPE GreenLake for private cloud, you can utilize your existing Chef Infra code to automate the VMs and applications provisioned in HPE GreenLake for private cloud. This will greatly reduce errors caused by manual configuration and improve efficiency and agility.</p> <p>To learn more about HPE GreenLake cloud services – the cloud that comes to wherever your apps and data live – visit the <a href="https://www.hpe.com/greenlake">HPE GreenLake homepage</a>.</p> <p>To learn more about Chef architecture, visit the <a href="https://www.tutorialspoint.com/chef/index.htm">Chef Tutorial</a>.</p> <p>More HPE GreenLake for private cloud white papers can be found on the <a href="https://developer.hpe.com/platform/hpe-greenlake/home">HPE Developer platform page</a>.</p>https://developer.hpe.com/from-jupyter-notebooks-as-a-service-to-hpe-dev-workshops-on-demand/https://developer.hpe.com/from-jupyter-notebooks-as-a-service-to-hpe-dev-workshops-on-demand/Fri, 09 Apr 2021 08:38:05 GMT<p><a href="/hackshack/workshops"><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/4/hackshack-1617960622993.png" alt="HackShack"></a></p> <p>At the end of a <a href="https://developer.hpe.com/blog/jupyter-saved-my-day">blog post</a> I wrote a year ago, I left off with an idea: Jupyter Notebooks-as-a-Service. Apparently, our HPE DEV team thought it was a great idea, because we went on to create the HPE DEV Workshops-on-Demand. While the name is different, the ultimate goal remained the same. Leveraging the JupyterHub technology, the HPE DEV team created a set of hands-on technology training workshops where anyone could connect to our service and benefit from notebook-based workshops at any time, and from anywhere.</p> <p>In our Workshops-on-Demand, users can actually interact and play with code to learn about a variety of different subjects. From infrastructure automation, coding 101 or API-related content, the Workshops-on-Demand catalog covers a broad range of subjects where you can learn about new technologies for free.</p> <p>Check out all the different topics we offer <a href="/hackshack/workshops">here</a>.</p> <p>Now, let me explain how we made this possible…</p> <h2>Early prototype</h2> <p>We built up our first JupyterHub server for the virtual HPE Technology &#x26; Solutions Summit (TSS) that took place in March 2020 and used it to deliver the workshops we would normally give in person in a virtual way. We hosted just a few Jupyter notebooks-based workshops and found that they scored really well in feedback we received from the students. All notebooks were delivered through a single instance of a JupyterHub server.</p> <p>During the event, we ran only a single workshop at a time. We first set up a range of users (students) to match a given workshop. It required a lot of manual setup, a few scripts and a single Ansible playbook to handle the copy of the workshop content assigned to the desired range of students, with a few variable substitutions like student IDs, passwords, and some API endpoints when relevant. The student assignment was done manually, at the beginning of every workshop. For instance, Fred was student1, Didier was student2, and so on… which was quite cumbersome. When you only have a range of 25 students, one or two people handling the back end is sufficient. But if 40 people show up, then it becomes tricky.</p> <h2>Standing up</h2> <p>We offered this virtual training first at TSS and then Aspire, (two internal HPE events.) Then HPE Discover, the largest HPE technology event, appeared on the radar. We were asked to renew the content of the workshops, add in new subjects like artificial intelligence (AI), machine learning operations (MLOPS), containers, etc. and that’s what we did. We also needed to provide a registration portal to allow automated customer registration.</p> <p>Both the workshop and challenge activities we planned on providing at HPE Discover were notebooks-based. The workshop was a follow-along, hands-on lab (aka an instructor-led workshop). The code challenges were different in that those who participated were asked to answer questions and then produce code snippets for a dedicated problem with the goal of winning a prize for the best submission. To assist in streamlining our processes, our developer, Pramod, stood up a nice app that allowed customers to register as well as automated the deployment of the notebooks. We implemented it only for the code challenges. The deployment for the workshops remained manual. These challenges allowed us to validate the beta version of the registration app.</p> <h2>Walking (or First steps)</h2> <p>By the end of the summer, we had many of the pieces in place that we needed to make the dream of Notebooks-as-a-Service a reality. We had the content and the automation to deliver it through an automation layer like Ansible. We also now had a registration app that simplified customer registration as well as workshop management (workshop capacity, workshop reset specificities, etc.). From this point, we really had something we could build upon.</p> <p>Where everything was manual or lightly scripted before, we really need it to be fully automated in order to make this a viable solution. We developed each new workshop on a staging environment. When ready, we moved it to the production environment. We also had a third environment located on a different site that could be used to recover a workshop. This proved very advantageous when, on the first day of HPE Discover 2020, we had to recover everything from the secondary site a few hours before the event started, since a backhoe cut the internet line on main site. Today, we are also leveraging our HPE GreenLake offering to build up one more environment. This continues to be a work in progress.</p> <p>As you can see, we have a few environments to deploy and manage. Without automation, it would be too much work and prone to error.</p> <p>Each server is associated to a logical location and role (Production, Staging, Sandbox, and GreenLake): Jupyter1 is the production server. Jupyter2 is the sandbox server. It is used to perform some early testing and is located in a different datacenter from the production and staging servers. Jupyter3 is the staging server. Workshops are developed and validated on this server before moving to production. Finally, Jupyter4 is deployed in a dedicated HPE GreenLake tenant. It will serve over time as a second production site.</p> <p>Each location is defined through a set of yaml files to specify the different parameters linked to the location (IP Addresses, Hostnames, etc…)</p> <pre><code class="language-yaml">PBKDIR: staging JPHOST: jupyter3.example.com JPIP: xx.xx.xx.xx JPHOSTEXT: nb3.example.com JPHUBAPISRV: http://{{ JPHOST }}:8000 JPHUBTOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx BASESTDID: 0 APIENDPOINT: https://example.com/api APIUSER: user APIPWD: password LDAPDMN: dc=example,dc=com # KIBANAPWD: "zzzzzzzzzzzzzzzzzzzzzzzzzzzzz" KIBANAPORT: "xxxxxx" # STACKSTORMSRVNAME: "sta-stackstorm" STACKSTORMSRIP: "xx.xx.xx.xx" STACKSTORMWEBUIPORT: "yyyy" # VCENTERAPP: "vcenter.example.com" VCENTERADMIN: "user" VCENTERPWD: "xxxxxxx" # </code></pre> <p>We standardized on Ubuntu 20.04 and Centos 7 for the operating systems and created a few Ansible playbooks to prepare the servers.</p> <p>The first playbook based on a location parameter would perform:</p> <ul> <li>the JupyterHub installation <ul> <li>System Update</li> <li>Repository Update</li> <li>Apps Installation</li> <li>System Performance Tuning</li> <li>Security Setup</li> <li>JupyterHub application installation and configuration</li> <li>kernels setup &#x26; configuration</li> <li>Linux users creation</li> <li>JupyterHub users creation</li> </ul> </li> </ul> <p>The second playbook would take care of deploying the reference notebooks on the newly created JupyterHub server.</p> <p>A third playbook is run on demand and nightly to ensure that the configuration is consistent and up to date.</p> <p>We created two Git repositories, one for the infrastructure management (and all the development we did to automate our deployments) and a second one for the reference notebooks’ content.</p> <h2>Running</h2> <p>While we worked on automating the deployment/redeployment of a JupyterHub at will, we also focused on improving the notebooks’ deployment automation that we implemented earlier during HPE Discover.</p> <p>Let me first show you how the overall process works for our Workshops-on-Demand:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/4/hackshack2-1617960613898.png" alt="Workshops-on-Demand"></p> <p>If you’re looking for a live explanation of this automation, you can review <a href="https://www.youtube.com/watch?v=D6Ss3T2p008&#x26;t=515s">the following session</a> Bruno Cornec and I delivered at Linuxconf in Australia in January 2021.</p> <p><strong>Step 1:</strong> The customer registers for a workshop online at our virtual <a href="/hackshack/workshops">Hack Shack</a>. When clicking the register button for the selected workshop and after agreeing to the terms and conditions, the front-end triggers the first REST API calls to:</p> <ul> <li>Register the student in the Customers Database.</li> <li>Send a welcome email to the student</li> <li>Assign a student ID to him/her according to the student range allocated to the given workshop</li> </ul> <p><strong>Step 2:</strong> The registration App then orders (through a mail API call managed by procmail) the backend to:</p> <ul> <li>Generate a random password for the selected student</li> <li>Deploy the selected workshop</li> </ul> <p><strong>Step 3:</strong> The backend Infrastructure calls back the registration application using its REST API to:</p> <ul> <li>Provide back the new student password</li> <li>Make the student’s database record active</li> <li>Decrement Workshop capacity</li> </ul> <p><strong>Step 4:</strong> The registration App sends:</p> <ul> <li>The credentials email to allow the student to connect to the workshop</li> </ul> <p>This infrastructure automation relies mainly on a few bash scripts and Ansible playbooks to:</p> <ul> <li>Generate random passwords</li> </ul> <pre><code> - Update LDAP passwords accordingly when required </code></pre> <ul> <li>Deploy the given workshop to the proper student home directory through an Ansible playbook</li> <li>Update the notebook content through Ansible variables substitutions <ul> <li>Student ID</li> <li>Student password</li> <li>API Endpoints definition</li> </ul> </li> <li>Update student permissions</li> <li>Perform all necessary create or reset actions linked to a given workshop outside of the notebook customization</li> </ul> <p>The registration app actually sends an email to the backend with a dedicated format (an API!) that is then parsed by the Procmail process to retrieve the necessary information to perform the different tasks. We use 3 verbs: CREATE (to setup to the student environment as described upper), DELETE (to remove the user from tables), and RESET (to clean up the student content and reset the back-end infrastructure when needed).</p> <p>At the end of this automated process, the backend makes a series of API calls to the registration app to send back the required information, like a new password, workshop status, etc.</p> <p>More info on Procmail can be found <a href="https://en.wikipedia.org/wiki/Procmail">here</a>.</p> <h2>Flying</h2> <p>In the fall, we started a pilot phase for a month. We opened the platform internally and gathered some feedback. We managed to discover a few bugs then that are now corrected. The platform went live in November 2020 and is used on a daily basis.</p> <p>Since then, we have added new workshops on a monthly basis. We lately returned to TSS using the Workshops-on-Demand to deliver our content; this time without any manual process in the scheme.</p> <p>Our next steps include the following:</p> <ul> <li>Implement complete deployment scenario <ul> <li>OS Bare metal deployment</li> </ul> </li> <li>Provide a new backend on HPE GreenLake <ul> <li> <pre><code>We have now a JupyterHub server running in HPE GreenLake </code></pre> </li> <li>We validated the relationship between this new backend and our registration application</li> <li>More testing to come</li> </ul> </li> <li>Automate the student ID range booking using a YAML configuration file</li> <li>Open source our Workshops-on-Demand Notebooks content:</li> </ul> <pre><code>- This is work in progress. The repository is ready. Licensing is ok. - Only missing some automation layer again to sync different repositories </code></pre> <p>Before leaving you with some final thoughts, let me show you a screenshot of our dashboard presenting:</p> <ul> <li>The number of Active Workshops</li> <li>The active workshops by type</li> <li>The total number of registrations from November 1st 2020 till today</li> <li>The total number of Customer (student) registrations</li> <li>The total workshops split</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/4/hackshack3-1617960606167.png" alt="Dashboard"></p> <p>Didier Lalli from our HPE DEV team created this informative dashboard and wrote a blog about it. If you are interested in learning more about ElasticSearch, read it out <a href="https://developer.hpe.com/blog/open-source-elasticsearch-helped-us-globally-support-virtual-labs">here</a>.</p> <p>As you can see, we have made significant progress over the past year, moving from a heavily manually oriented approach to now a fully automated one. If you are interested in seeing the result for yourself, <a href="/hackshack/workshops">please register for one of our Workshops-on-Demand</a>. Don’t forget to fill out the survey at the end of the workshop to provide us with feedback on your experience, as well as subjects you would like to see covered in the near future, as this really helps to improve the program.</p>https://developer.hpe.com/managing-multiple-instances-of-python-in-microsoft-windows/https://developer.hpe.com/managing-multiple-instances-of-python-in-microsoft-windows/Tue, 06 Apr 2021 09:08:53 GMT<p>I use many Microsoft Windows systems for development and testing; both real systems and VMs. Most of them have one or more versions of Python installed. Several have many versions. And, over the years, I've experienced a lot of problems:</p> <ul> <li> <p>Old programs, not compatible with Python 3, breaking on some systems.</p> </li> <li> <p>New programs, not compatible with Python 2, breaking on others. (Yes, I know that Python 2 is obsolete, but the reality is that I still have mission-critical tools that depend on it!)</p> </li> <li> <p>The py.exe tool not detecting some installed versions of Python.</p> </li> <li> <p>The python.exe command not always starting the most recent version of Python installed on a system.</p> </li> <li> <p>Etc.</p> </li> </ul> <p>Finally, I decided to get to the bottom of it. In this blog post, I'll share with you what I learned.</p> <h2>Where to get Python for Microsoft Windows</h2> <p>Recent versions of Microsoft Windows 10 have a python.exe stub pre-installed that gets you directly to a Microsoft Store page. From there you can download a free version of Python officially supported by Microsoft. Just run <code>python</code> at the command prompt and follow the instructions on the Microsoft Store page.</p> <p>An alternative is to go to <code>https://www.python.org/downloads/</code>, and get the official Python releases for Microsoft Windows. Advantage: Their version is slightly more recent than the one supported by Microsoft.</p> <p>Another possibility is to use a third party build for Microsoft Windows, like the <a href="https://www.activestate.com/products/python/downloads/">ActivePython</a> distribution maintained by <a href="https://www.activestate.com">ActiveState</a>. If you still need Python 2.7, their ActivePython 2.7 is an all-batteries included distribution, with all modules that you can think of built-in.</p> <blockquote> <p>Caution: These ActivePython distributions are free for development use, but not for production use.</p> </blockquote> <p>There are many other possible alternatives, the ones above being just some of the most common.</p> <p>This diversity is proof of the vitality of the Python ecosystem... But this is also the source of many problems, as different versions make different, sometimes conflicting, installation choices!</p> <h2>The py.exe launcher tool</h2> <p>The <em>py.exe</em> launcher tool is a critically important tool that comes with Python 3 distributions for Microsoft Windows. Unfortunately its installation is optional and usually <em>not</em> selected by default. If you ever install more that one Python version on your system (Python 2.7 and Python 3.x maybe), then it's a no-brainer: You <strong>must</strong> select it and install it.</p> <p>First and foremost, py.exe is a front end to python.exe. By default, it runs the latest installed version of python.exe. But it has options to select another version at will. Usage:<br> <code>py [py.exe options] [python.exe options and arguments]</code></p> <p>Example:</p> <pre><code class="language-markdown">C:\Temp>py --version Python 3.9.2 C:\Temp>py -2 --version Python 2.7.12 C:Temp> </code></pre> <p>Here, in both cases, the <code>--version</code> option was interpreted by python.exe, <em>not</em> by py.exe.<br> In the first case, it was python.exe version 3.9.2 that ran. In the second, it was python.exe version 2.7.12 that did.</p> <p>The <code>-2</code> option is a py.exe option that tells it to run the latest Python 2 version.<br> You can also specify the minor version, like <code>-3.5</code>, or even the target processor size, i.e. <code>-3-32</code> or <code>-3.8-64</code>.</p> <p>The <code>-0</code> option tells py.exe to just list the available instances:</p> <pre><code class="language-markdown">C:\Temp>py -0 Installed Pythons found by py Launcher for Windows -3.9-64 * -3.8-32 -3.7-64 -3.6-64 -3.5-64 -2.7-64 C:\Temp> </code></pre> <p>There's also a <code>-0p</code> option to list both the version and pathname of the python.exe instances it can use:</p> <pre><code class="language-markdown">C:\Temp>py -0p Installed Pythons found by py Launcher for Windows -3.9-64 C:\Program Files\Python39\python.exe * -3.8-32 C:\Users\Larvoire\AppData\Local\Programs\Python\Python38-32\python.exe -3.7-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe -3.6-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe -3.5-64 C:\Program Files\Python35\python.exe -2.7-64 C:\Program Files\Python27\python.exe C:\Temp> </code></pre> <h3>Python instances not seen by py.exe</h3> <p>Sometimes, py.exe does not see some installed versions of Python. This is often the case with Python 2.7 instances. The root cause is that the setup program for these instances did not create the necessary registry keys used by py.exe to enumerate the installed instances.</p> <p>To fix that, you need to manually create one of these registry keys and values:</p> <ul> <li> <p>For user-specific instances (Installed in C:\Users\YOURNAME\AppData\Local\Programs\Python), the base key is:<br> <code>HKEY_CURRENT_USER\Software\Python\PythonCore\MAJOR.MINOR\InstallPath\...</code></p> </li> <li> <p>For system-wide instances (Installed in C:\Program Files\python*, or even in C:\Python*), the base key is:<br> <code>HKEY_LOCAL_MACHINE\Software\Python\PythonCore\MAJOR.MINOR\InstallPath\...</code></p> </li> </ul> <p>In each of these keys, there are three values needed:</p> <table> <thead> <tr> <th>Value name</th> <th>Content</th> </tr> </thead> <tbody> <tr> <td>@ (In the key itself)</td> <td>The Python installation directory name</td> </tr> <tr> <td>ExecutablePath</td> <td>The full pathname of the python.exe instance</td> </tr> <tr> <td>WindowedExecutablePath</td> <td>The full pathname of the pythonw.exe instance</td> </tr> </tbody> </table> <p>For example, for my Python 2.7 installation, I had to add keys and values from this .reg file:</p> <pre><code class="language-markdown">Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\Software\Python\PythonCore\2.7] [HKEY_LOCAL_MACHINE\Software\Python\PythonCore\2.7\InstallPath] @="C:\\Program Files\\Python27\\" "ExecutablePath"="C:\\Program Files\\Python27\\python.exe" "WindowedExecutablePath"="C:\\Program Files\\Python27\\pythonw.exe" </code></pre> <h3>Automatically running Python 2 or Python 3 for scripts that need it</h3> <p>Another great feature of py.exe is that it interprets the Python scripts shebang. The shebang is the special #! comment on the first line of a script. This comment is intended to be used by Unix shells to select which interpreter to run.</p> <p>Problem: There is no such mechanism in Microsoft Windows shells (cmd or PowerShell). That's where py.exe steps in: Based on that shebang, it selects the right Python interpreter to use in Microsoft Windows.</p> <table> <thead> <tr> <th>Shebang</th> <th>py.exe action</th> </tr> </thead> <tbody> <tr> <td><code>#!/usr/bin/env python</code></td> <td>Select the most recent Python version available</td> </tr> <tr> <td><code>#!/usr/bin/env python2</code></td> <td>Select the most recent Python 2 version available</td> </tr> <tr> <td><code>#!/usr/bin/env python3</code></td> <td>Select the most recent Python 3 version available</td> </tr> </tbody> </table> <p>Use the first generic shebang <em>only</em> if your scripts is compatible with both Python 2 and Python 3. In the most common case where your script is compatible with just one of the two, use the right one for that version.</p> <p>Example:</p> <pre><code class="language-markdown">C:\Temp>type testpy2.py #!/usr/bin/env python2 import platform; print(platform.sys.version); C:\Temp>py testpy2.py 2.7.12 (default, Dec 19 2016, 15:56:45) [MSC v.1500 64 bit (AMD64)] C:\Temp>type testpy3.py #!/usr/bin/env python3 import platform; print(platform.sys.version); C:\Temp>py testpy3.py 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)] C:\Temp> </code></pre> <p>Note that this tip is good for Unix also. With the right shebang, the script will use the right Python version both in Unix and Microsoft Windows.</p> <p>One last important note about this: Unix shells choke on Python scripts created in Microsoft Windows with CRLF at the end of lines. If you want your Python script to be portable to Mac and Linux systems, use Unix LF line endings, even in Microsoft Windows.</p> <h2>Running Python scripts based on the .py extension</h2> <h3>The local machine configuration</h3> <p>Microsoft Windows can select interpreters based on the file extension. This should be setup correctly by recent versions of Python 3, but may not be so with old versions of Python 2! If you install an old Python 2.7 <em>after</em> a recent Python 3, this may bite you! To check if the configuration is correct, run <code>assoc.exe</code>, then <code>ftype.exe</code>, as shown here:</p> <pre><code class="language-markdown">C:\Temp>assoc .py .py=Python.File C:\Temp>ftype Python.File Python.File=C:\Windows\py.exe "%L" %* C:\Temp> </code></pre> <p>The <code>assoc</code> command must show that extension .py is associated with class <code>Python.File</code>. If it's not, run <code>assoc .py=Python.File</code> to correct it. And the <code>ftype</code> command must show that the <code>Python.File</code> class is associated with the py.exe command, or if py.exe is not available on your (very old) system, to the latest python.exe command available.</p> <p>The <code>"%L" %*</code> arguments tell the shell to append the script full pathname, and all its arguments if any.</p> <blockquote> <p>Note that I've seen cases where this command was corrupt, with a long string of garbage characters instead. If it's incorrect or corrupt, correct it by running:</p> </blockquote> <p><code>ftype Python.File=C:\Windows\py.exe "%L" %*</code></p> <h3>The current user configuration</h3> <p>However this is not always sufficient. If you've installed a Python instance for yourself only, not for all users, the system-wide extension-to-class and class-to-command associations are overridden by user-specific HKCU registry keys:</p> <pre><code class="language-markdown">C:\Temp>reg query "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.py\UserChoice" /v Progid HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.py\UserChoice Progid REG_SZ Python.File C:\Temp>reg query HKCU\Software\Classes\Python.File\shell\open\command /ve ERROR: The system was unable to find the specified registry key or value. C:\Temp> </code></pre> <p>In my case, the first key is defined with the same value as the global system-wide association, which is fine. If they're not defined, as with my second key, this is also fine, as the system-wide version will be used. But again, these two keys may be incorrect or corrupt. If they are, correct them with regedit.exe or delete them altogether.</p> <h3>Other failure causes</h3> <ul> <li>ftype.exe stores its data in the regitry in key <code>HKEY_LOCAL_MACHINE\Software\Classes\Python.File\shell\open\command</code></li> </ul> <p>If that key contains sub-keys (I've seen this), then Microsoft Windows shells get confused. If that happens, remove the sub-keys.</p> <ul> <li>There's also a python.exe association in <code>HKEY_LOCAL_MACHINE\Software\Classes\Applications\%EXE%\shell\open\command</code>.</li> </ul> <p>If defined, it should contain: <code>C:\Windows\py.exe "%1"</code> (The argument <code>"%L" %*</code> also works, not sure which is best.)</p> <ul> <li> <p>The above two have counterparts in <code>HKEY_CURRENT_USER\Software\Classes\...</code>. Same remarks.</p> </li> <li> <p>File type associations can also be overridden by global policies, including some set by your Domain Controller. I've never seen this, but be aware it's possible.</p> </li> </ul> <h2>Finding python.exe in the PATH</h2> <p>The Python setup optionally adds the Python and Python Scripts installation directories to the PATH. This allows starting interactive Python sessions by typing <code>python</code> at the command prompt. Likewise, this allows starting <code>pip</code>, or tools installed by pip into the Scripts directory just by typing their name.</p> <blockquote> <p>Note that this is optional, because as explained above, if you have the py.exe command available, you'll get the</p> </blockquote> <p>same result by typing <code>py</code>, or <code>py -n pip</code>, etc.</p> <p>Having the Python and Python Scripts directories in the PATH makes things a bit more intuitive and similar to how they work in Unix. The drawback is that this makes the ridiculously long Microsoft Windows PATH even longer, and so potentially slows down your system a little bit.</p> <p>But a worse problem may occur when you have several versions of Python installed. You may find situations where an old version was installed with the PATH updated and a newer version without. The result is that the <code>py</code> and <code>python</code> commands do not start the same version of Python, which can lead to problems. If this happens, you must correct your local, user, and system PATH to make sure the latest version of Python runs in all cases.</p> <h3>Tools for managing the Microsoft Windows PATH</h3> <h4>paths.bat</h4> <p>As the Microsoft Windows PATH is extremely long, it's often difficult to tell if a given directory is in the list and where. The open source <a href="https://github.com/JFLarvoire/SysToolsLib/blob/master/Batch/paths.bat">paths.bat</a> tool makes is easy to review, and optionally correct, your local or system PATH. By default, it displays all entries in your local PATH, one per line. This makes it much easier to review what's in there. This also allows filtering the output using command-line filtering tools. For example:</p> <pre><code class="language-markdown">C:\Temp>paths | findstr /i python C:\Program Files\Python39 C:\Program Files\Python39\scripts C:\Temp> </code></pre> <p>Run <code>paths -?</code> to display a help screen describing all available options. The following options will be particularly useful for fixing problems with your Python configuration:</p> <ul> <li> <p><code>-s</code> tells it to manage the system PATH, instead of the local shell PATH by default.</p> </li> <li> <p><code>-u</code> tells it to manage the user PATH, instead of the local shell PATH by default.</p> </li> <li> <p><code>-r DIRECTORY</code> tells it to remove that DIRECTORY from the managed PATH.</p> </li> <li> <p><code>-m DIR1 -b DIR2</code> tells it to move a DIRECTORY 1 just before DIRECTORY 2.</p> </li> </ul> <h4>Other System Tools Library tools</h4> <p>The <a href="https://github.com/JFLarvoire/SysToolsLib/releases/latest/download/SysTools.zip">System Tools Library</a> contains paths.bat and other tools for managing the PATH:</p> <ul> <li> <p><code>which.exe</code> includes the best of Unix which and Microsoft Windows where.exe and some more. For a detailed description of which.exe features, see <a href="https://www.dostips.com/forum/viewtopic.php?f=3&#x26;t=9058">this post</a>.</p> </li> <li> <p><a href="https://github.com/JFLarvoire/SysToolsLib/blob/master/Bash/paths">paths</a> is a simple Posix Shell script</p> </li> </ul> <p>for doing the same things in Unix shells as paths.bat does in Microsoft Windows shells.</p> <h3>Interference with the Microsoft Store link</h3> <p>In recent Microsoft Windows 10 versions, Microsoft installs a python.exe stub, that redirects you to the Microsoft Store, where you can download their distribution of Python. This stub is stored in <code>%LOCALAPPDATA%\Microsoft\WindowsApps</code> (cmd) or <code>$env:LOCALAPPDATA\Microsoft\WindowsApps</code> (PowerShell).</p> <p>Problem: If you install a non-Microsoft distribution and add its location in the Microsoft Windows PATH, it sometimes ends up in the PATH <em>behind</em> the Microsoft stub!</p> <p>Running <code>python</code> will open the Microsoft Store instead of the Python instance you just installed.</p> <p>Quick workaround: Use py.exe instead of python.exe. Py.exe does not use the PATH to locate instances, and will find your latest python.exe anyway.</p> <p>Long term solution: Move the Python directories in the system PATH <em>before</em> "%LOCALAPPDATA%\Microsoft\WindowsApps". Then restart the open shells to get the modified PATH.</p> <p>Below is an example of a system that has this problem: (Using the <code>paths</code> tool described in the previous section.)</p> <pre><code class="language-markdown">C:\Temp>where python C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps\python.exe C:\Program Files\Python39\python.exe C:\Temp>paths | findstr /i /c:python /c:WindowsApps C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps C:\Program Files\Python39 C:\Program Files\Python39\scripts C:\Temp> </code></pre> <p>In this case, to move the above two directories before "%LOCALAPPDATA%\Microsoft\WindowsApps", run:</p> <pre><code class="language-markdown">C:\Temp>paths -m "C:\Program Files\Python39" -b "%LOCALAPPDATA%\Microsoft\WindowsApps" [Outputs the updated PATH contents] C:\Temp>paths -m "C:\Program Files\Python39\scripts" -b "%LOCALAPPDATA%\Microsoft\WindowsApps" [Outputs the updated PATH contents] C:\Temp>paths | findstr /i /c:python /c:WindowsApps C:\Program Files\Python39 C:\Program Files\Python39\scripts C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps C:\Temp> </code></pre> <h2>Finding Python scripts in the PATH without specifying their extension</h2> <p>Microsoft Windows shells search for commands using a list of implicit extensions defined in the PATHEXT environment variable. This allows running a script by entering just its base name <em>without</em> the extension. For example on my system, reusing the testpy2.py and testpy3.py scripts described above:</p> <pre><code class="language-markdown">C:\Temp>set PATHEXT PATHEXT=.PY;.PY3;.PYC;.PYO;.PYW;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC C:\Temp>testpy3 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)] C:\Temp>testpy2 2.7.12 (default, Dec 19 2016, 15:56:45) [MSC v.1500 64 bit (AMD64)] C:\Temp> </code></pre> <p>If your PATHEXT does not contain .PY or .py (This is case-independant, so the two are equivalent), then add it into both the system PATHEXT and your local shell PATHEXT. Example for the cmd.exe shell:</p> <pre><code class="language-markdown">setx PATHEXT ".PY;%PATHEXT%" -m set PATHEXT=.PY;%PATHEXT% </code></pre> <p>and for PowerShell:</p> <pre><code class="language-markdown">$SystemPathExt = [Environment]::GetEnvironmentVariable('PATHEXT', 'Machine') [Environment]::SetEnvironmentVariable('PATH', ".PY;$SystemPathExt", 'Machine') $env:PATHEXT = ".PY;$env:PATHEXT" </code></pre> <h2>Automating all of this</h2> <p>If all the above gives you a headache (it definitely does for me!), there is a way to do everything automatically. The Open Source <a href="https://github.com/JFLarvoire/SysToolsLib/releases/latest/download/SysTools.zip">System Tools Library</a> contains a tool called <a href="https://github.com/JFLarvoire/SysToolsLib/blob/master/Python/PySetup.bat">PySetup.bat</a>. This tool checks all the information documented above and, optionally, fixes it.</p> <p>First run <code>pysetup</code> without any option to check the current configuration. Then, if anything is wrong, and you agree with the proposed changes, run <code>pysetup -s</code>.</p> <p>When everything is correct, pysetup.bat outputs all green <span style="color:green">[OK]</span> statuses:</p> <pre><code class="language-markdown">C:\Temp>pysetup Testing the "C:\Program Files\Python39\python.exe" configuration Using C:\Windows\py.exe The .py extension is globally associated with class: Python.File [OK] The .py extension is associated for user Larvoire with class: Python.File [OK] The open command for class Python.File is: C:\Windows\py.exe "%L" %* [OK] There's no additional command for class Python.File [OK] The open command for application python.exe is: C:\Windows\py.exe "%L" %* [OK] The Python InstallPath registration is: "C:\Program Files\Python39\" [OK] The Python PythonPath registration is: "C:\Program Files\Python39\Lib\;C:\Program Files\Python39\DLLs\" [OK] The PATH contains C:\Program Files\Python39 [OK] The PATH contains C:\Program Files\Python39\scripts [OK] Other Python directories in the PATH: [OK] The global system PATH contains C:\Program Files\Python39 [OK] The global system PATH contains C:\Program Files\Python39\scripts [OK] Other Python directories in the global system PATH: [OK] The global environment variable PATHEXT is: .PY;.PY3;.PYC;.PYO;.PYW;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC [OK] The local environment variable PATHEXT is: .PY;.PY3;.PYC;.PYO;.PYW;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC [OK] Verifying that 'python' starts "C:\Program Files\Python39\python.exe" [OK] Verifying that 'python.exe' starts "C:\Program Files\Python39\python.exe" [OK] The setup is good. C:\Temp> </code></pre> <h3>Fixing missing instances that py.exe does not see</h3> <p>Another feature of pysetup.bat is that it can scan known places on the disk for Python instances, and optionally register missing entries so that py.exe knows about them.</p> <ul> <li> <p>First run <code>pysetup -l</code> to list instances. (This may take some time if you have a slow disk.)</p> <pre><code> C:\Temp>pysetup -l #0 3.9.2 AMD64 C:\Windows\py.exe #1 2.7.12 AMD64 C:\Program Files\Python27\python.exe #2 3.5.2 AMD64 C:\Program Files\Python35\python.exe #3 3.7.4 AMD64 C:\Program Files\Python37\python.exe #4 3.9.2 AMD64 C:\Program Files\Python39\python.exe #5 3.6.6 AMD64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe #6 3.7.8 AMD64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe #7 3.8.7 x86 C:\Users\Larvoire\AppData\Local\Programs\Python\Python38-32\python.exe C:\Temp> </code></pre> </li> <li> <p>Compare that to the output of the <code>py -0p</code> command.</p> <pre><code> C:\Temp>py -0p Installed Pythons found by py Launcher for Windows -3.9-64 C:\Program Files\Python39\python.exe * -3.8-32 C:\Users\Larvoire\AppData\Local\Programs\Python\Python38-32\python.exe -3.7-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe -3.6-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe -3.5-64 C:\Program Files\Python35\python.exe C:\Temp> </code></pre> </li> <li> <p>If any one is missing (As is the case here for Python 2.7.12), then run <code>python -r VERSION</code> to fix the issue. Ex:</p> <pre><code> C:\Temp>pysetup -r 2.7.12 reg add "HKLM\Software\Python\PythonCore\2.7\InstallPath" /ve /d "C:\Program Files\Python27\\" /f The operation completed successfully. reg add "HKLM\Software\Python\PythonCore\2.7\InstallPath" /v ExecutablePath /d "C:\Program Files\Python27\python.exe" /f The operation completed successfully. reg add "HKLM\Software\Python\PythonCore\2.7\InstallPath" /v WindowedExecutablePath /d "C:\Program Files\Python27\pythonw.exe" /f The operation completed successfully. C:\Temp>py -0p Installed Pythons found by py Launcher for Windows -3.9-64 C:\Program Files\Python39\python.exe * -3.8-32 C:\Users\Larvoire\AppData\Local\Programs\Python\Python38-32\python.exe -3.7-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe -3.6-64 C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe -3.5-64 C:\Program Files\Python35\python.exe -2.7-64 C:\Program Files\Python27\python.exe C:\Temp> </code></pre> </li> </ul> <h2>Conclusion</h2> <p>I hope you found this post helpful when dealing with some of the issues that can occur when you are configuring multiple instances of Python in Microsoft Windows environments. For more informative tutorials, make sure you keep checking back on the <a href="https://developer.hpe.com/blog">HPE DEV blog</a>.</p>https://developer.hpe.com/2021-April-06/https://developer.hpe.com/2021-April-06/Tue, 06 Apr 2021 06:00:00 GMThttps://developer.hpe.com/streaming-ml-pipeline-for-sentiment-analysis-using-apache-apis-kafka-spark-and-drill-part-2/https://developer.hpe.com/streaming-ml-pipeline-for-sentiment-analysis-using-apache-apis-kafka-spark-and-drill-part-2/Wed, 31 Mar 2021 16:16:02 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Carol McDonald"], "publish": "2019-05-20T07:00:00.000Z", "tags": "mapr-platform" </code></pre> <hr> <p><em>Editor's Note: This tutorial is the second part in a series related to this topic. The first part in this series is found here: <a href="/blog/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-">Part 1.</a></em></p> <p>This is the second in a series of blogs that discuss the architecture of a data pipeline that combines streaming data with machine learning and fast storage. In the first part, we explored sentiment analysis using Spark Machine learning Data pipelines and saved a sentiment analysis machine learning model. This second post will discuss using the saved sentiment analysis model with streaming data to do real-time analysis of product sentiment, storing the results in MapR Database, and making them rapidly available for Spark and Drill SQL.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image1-1607498963348.png" alt=""></p> <p>In this post we will go over the following:</p> <ul> <li>Overview of Streaming concepts</li> <li>Ingesting Kafka Events with Spark Structured Streaming</li> <li>Enriching events with a machine learning model.</li> <li>Storing the events in MapR Database</li> <li>Querying the rapidly available enriched events in MapR Database with Apache Spark SQL and Apache Drill.</li> </ul> <h2>Streaming Concepts</h2> <h2>Publish-Subscribe Event Streams with MapR Event Store for Apache Kafka</h2> <p>MapR Event Store for Apache Kafka is a distributed publish-subscribe event streaming system that enables producers and consumers to exchange events in real time in a parallel and fault-tolerant manner via the Apache Kafka API.</p> <p>A stream represents a continuous sequence of events that goes from producers to consumers, where an event is defined as a key-value pair.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image2-1607498972008.png" alt=""></p> <p>Topics are a logical stream of events. Topics organize events into categories and decouple producers from consumers. Topics are partitioned for throughput and scalability. MapR Event Store can scale to very high throughput levels, easily delivering millions of messages per second using very modest hardware.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image13-1607498981360.png" alt=""></p> <p>You can think of a partition like an event log: new events are appended to the end and are assigned a sequential ID number called the <em>offset</em>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image22-1607498989167.png" alt=""></p> <p>Like a queue, events are delivered in the order they are received.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image20-1607498997119.png" alt=""></p> <p>Unlike a queue, however, messages are not deleted when read. They remain on the partition available to other consumers. Messages, once published, are immutable and can be retained forever.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image5-1607499005836.png" alt=""></p> <p>Not deleting messages when they are read allows for high performance at scale and also for processing of the same messages by different consumers for different purposes such as multiple views with polyglot persistence.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image15-1607499014273.png" alt=""></p> <h2>Spark Dataset, DataFrame, SQL</h2> <p>A Spark Dataset is a distributed collection of typed objects partitioned across multiple nodes in a cluster. A Dataset can be manipulated using functional transformations (map, flatMap, filter, etc.) and/or Spark SQL. A DataFrame is a Dataset of Row objects and represents a table of data with rows and columns.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image14-1607499022152.png" alt=""></p> <h2>Spark Structured Streaming</h2> <p>Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. Structured Streaming enables you to view data published to Kafka as an unbounded DataFrame and process this data with the same DataFrame, Dataset, and SQL APIs used for batch processing.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image6-1607499031160.png" alt=""></p> <p>As streaming data continues to arrive, the Spark SQL engine incrementally and continuously processes it and updates the final result.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image21-1607499040966.png" alt=""></p> <p>Stream processing of events is useful for real-time ETL, filtering, transforming, creating counters and aggregations, correlating values, enriching with other data sources or machine learning, persisting to files or Database, and publishing to a different topic for pipelines.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image4-1607499049861.png" alt=""></p> <h2>Spark Structured Streaming Use Case Example Code</h2> <p>Below is the data processing pipeline for this use case of sentiment analysis of Amazon product review data to detect positive and negative reviews.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image1-1607499057405.png" alt=""></p> <ol> <li> <p>Amazon product review JSON formatted events are published to a MapR Event Store topic using the Kafka API.</p> </li> <li> <p>A Spark Streaming application subscribed to the topic:</p> <ol> <li>Ingests a stream of product review events</li> <li>Uses a deployed machine learning model to enrich the review event with a positive or negative sentiment prediction</li> <li>Stores the transformed and enriched data in MapR Database in JSON format.</li> </ol> </li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image3-1607499065187.png" alt=""></p> <h2>Example Use Case Data</h2> <p>The example data set is Amazon product reviews data from <a href="https://developer.hpe.com/blog/wzvGV1qzj3c2YA8QQnMD/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-">the previous blog in this series</a>. The incoming data is in JSON format; an example is shown below:</p> <pre><code class="language-json">{"reviewerID": "A3V52OTJHKIJZX", "asin": "2094869245","reviewText": "Light just installed on bike, seems to be well built.", "overall": 5.0, "summary": "Be seen", "unixReviewTime": 1369612800} </code></pre> <p>We enrich this data with the sentiment prediction, drop some columns, then transform it into the following JSON object:</p> <pre><code class="language-json"> {"reviewerID": "A3V52OTJHKIJZX", "_id":"2094869245_1369612800", "reviewText": "Light just installed on bike, seems to be well built.", "overall": 5.0, "summary": "Be seen", "label":"1", "prediction":"1"} </code></pre> <h2>Loading the Spark <em>pipeline</em> Model</h2> <p>The Spark PipelineModel class is used to load the pipeline model, which was fitted on <a href="https://developer.hpe.com/blog/wzvGV1qzj3c2YA8QQnMD/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-">the historical product review data</a> and then saved to the MapR XD file system.</p> <pre><code class="language-python">// Directory to read the saved ML model from var modeldirectory ="/user/mapr/sentmodel/" // load the saved model from the distributed file system val model = PipelineModel.load(modeldirectory) </code></pre> <h2>Reading Data from Kafka Topics</h2> <p>In order to read from Kafka, we must first specify the stream format, topic, and offset options. For more information on the configuration parameters, <a href="https://docs.datafabric.hpe.com/62/MapR_Streams/differences_in_configuration_parameters_for_producers_and_consumers.html">see the MapR Streams documentation.</a></p> <pre><code class="language-python">var topic: String = "/user/mapr/stream:reviews" val df1 = spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "maprdemo:9092") .option("subscribe", topic) .option("group.id", "testgroup") .option("startingOffsets", "earliest") .option("failOnDataLoss", false) .option("maxOffsetsPerTrigger", 1000) .load() </code></pre> <p>This returns a DataFrame with the following schema:</p> <pre><code class="language-python">df1.printSchema() result: root |-- key: binary (nullable = true) |-- value: binary (nullable = true) |-- topic: string (nullable = true) |-- partition: integer (nullable = true) |-- offset: long (nullable = true) |-- timestamp: timestamp (nullable = true) |-- timestampType: integer (nullable = true) </code></pre> <h2>Parsing the Message Values into a DataFrame</h2> <p>The next step is to parse and transform the binary values column into a DataFrame with the product review schema. We will use Spark from_json to extract the JSON data from the Kafka DataFrame value field seen above. The Spark SQL from_json() function turns an input JSON string column into a Spark struct, with the specified input schema.</p> <p>First we use a Spark StructType to define the schema corresponding to the incoming JSON message value.</p> <pre><code class="language-json">val schema = StructType(Array( StructField("asin", StringType, true), StructField("helpful", ArrayType(StringType), true), StructField("overall", DoubleType, true), StructField("reviewText", StringType, true), StructField("reviewTime", StringType, true), StructField("reviewerID", StringType, true), StructField("reviewerName", StringType, true), StructField("summary", StringType, true), StructField("unixReviewTime", LongType, true) )) </code></pre> <p>In the code below, we use the <em>from_json()</em> Spark SQL function, in a <em>select expression</em> with a <em>string cast</em> of the df1 column <em>value</em>, which returns a DataFrame of thespecified schema.</p> <pre><code class="language-python">import spark.implicits._ val df2 = df1.select($"value" cast "string" as "json") .select(from_json($"json", schema) as "data") .select("data.*") </code></pre> <p>This returns a DataFrame with the following schema:</p> <pre><code class="language-python">df2.printSchema() result: root |-- asin: string (nullable = true) |-- helpful: array (nullable = true) | |-- element: string (containsNull = true) |-- overall: double (nullable = true) |-- reviewText: string (nullable = true) |-- reviewTime: string (nullable = true) |-- reviewerID: string (nullable = true) |-- reviewerName: string (nullable = true) |-- summary: string (nullable = true) |-- unixReviewTime: long (nullable = true) </code></pre> <p>In the code below:</p> <ul> <li>we use the withColumn method to add a column combining the review summary with the review text .</li> <li>we filter to remove neutral ratings (=3)</li> <li>a Spark <a href="https://spark.apache.org/docs/2.2.0/ml-features.html#bucketizer">Bucketizer</a> is used to add a label 0/1 column to the dataset for Positive (overall rating >=4) and not positive (overall rating &#x3C;4) reviews. (Note the label is for testing the predictions)</li> </ul> <pre><code class="language-python">// combine summary reviewText into one column val df3 = df2.withColumn("reviewTS", concat($"summary",lit(" "),$"reviewText" )) // remove neutral ratings val df4 = df3.filter("overall !=3") // add label column val bucketizer = new Bucketizer() .setInputCol("overall") .setOutputCol("label") .setSplits(Array(Double.NegativeInfinity,3.0,Double.PositiveInfinity)) val df5= bucketizer.transform(df4) </code></pre> <h2>Enriching the DataFrame of Reviews with Sentiment Predictions</h2> <p>Next we transform the DataFrame with the model pipeline, which will transform the features according to the pipeline, estimate and then return the predictions in a column of a new DateFrame.</p> <pre><code class="language-python"> // transform the DataFrame with the model pipeline val predictions = model.transform(df5) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image16-1607499077714.png" alt=""></p> <p>This returns a DataFrame with the following schema:</p> <pre><code class="language-python">predictions.printSchema() result: root |-- asin: string (nullable = true) |-- helpful: array (nullable = true) | |-- element: string (containsNull = true) |-- overall: double (nullable = true) |-- reviewText: string (nullable = true) |-- reviewTime: string (nullable = true) |-- reviewerID: string (nullable = true) |-- reviewerName: string (nullable = true) |-- summary: string (nullable = true) |-- unixReviewTime: long (nullable = true) |-- reviewTS: string (nullable = true) |-- label: double (nullable = true) |-- reviewTokensUf: array (nullable = true) | |-- element: string (containsNull = true) |-- reviewTokens: array (nullable = true) | |-- element: string (containsNull = true) |-- cv: vector (nullable = true) |-- features: vector (nullable = true) |-- rawPrediction: vector (nullable = true) |-- probability: vector (nullable = true) |-- prediction: double (nullable = false) </code></pre> <h2>Adding a Unique ID for MapR Database</h2> <p>In the code below we:</p> <ul> <li>drop the columns that we do not want to store</li> <li>create a unique id “_id” composed of the product id and review timestamp, to us as the row key for storing in MapR Database.</li> </ul> <pre><code class="language-python">// drop the columns that we do not want to store val df6 = predictions.drop("cv","probability", "features", "reviewTokens", "helpful", "reviewTokensUf", "rawPrediction") // create column with unique id for MapR Database val df7 = df6.withColumn("_id", concat($"asin",lit("_"), $"unixReviewTime")) </code></pre> <p>This returns a DataFrame with the following schema:</p> <pre><code class="language-python">predictions.printSchema() Result: root |-- asin: string (nullable = true) |-- overall: double (nullable = true) |-- reviewText: string (nullable = true) |-- reviewTime: string (nullable = true) |-- reviewerID: string (nullable = true) |-- reviewerName: string (nullable = true) |-- summary: string (nullable = true) |-- unixReviewTime: long (nullable = true) |-- label: double (nullable = true) |-- reviewTokens: array (nullable = true) | |-- element: string (containsNull = true) |-- prediction: double (nullable = false) |-- _id: string (nullable = true) </code></pre> <h2>Spark Streaming Writing to MapR Database</h2> <p>The MapR Database Connector for Apache Spark enables you to use MapR Database as a sink for Spark Structured Streaming or Spark Streaming.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image9-1607499085506.png" alt=""></p> <p>One of the challenges when you are processing lots of streaming data is: where do you want to store it? For this application, MapR Database a high performance NoSQL database, was chosen for its scalability and flexible ease of use with JSON.</p> <h2>JSON Schema Flexibility</h2> <p>MapR Database supports JSON documents as a native data store. MapR Database makes it easy to store, query, and build applications with JSON documents. The Spark connector makes it easy to build real-time or batch pipelines between your JSON data and MapR Database and leverage Spark within the pipeline.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image7-1607499093783.png" alt=""></p> <p>With MapR Database, a table is automatically partitioned into tablets across a cluster by key range, providing for scalable and fast reads and writes by row key. In this use case, the row key, the _id, consists of the cluster ID and reverse timestamp, so the table is automatically partitioned and sorted by cluster ID with the most recent first.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image11-1607499101776.png" alt=""></p> <p>The Spark MapR Database Connector architecture has a connection object in every Spark Executor, allowing for distributed parallel writes, reads, or scans with MapR Database tablets (partitions).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image25-1607499112026.png" alt=""></p> <h2>Writing to a MapR Database Sink</h2> <p>To write a Spark Stream to MapR Database, specify the <a href="https://docs.datafabric.hpe.com/62/Spark/StructuredSparkStreaming.html">format with the tablePath, idFieldPath, createTable, bulkMode, and sampleSize parameters</a>. The following example writes out the df7 DataFrame to MapR Database and starts the stream.</p> <pre><code class="language-python">import com.mapr.db.spark.impl._ import com.mapr.db.spark.streaming._ import com.mapr.db.spark.sql._ import com.mapr.db.spark.streaming.MapRDBSourceConfig var tableName: String = "/user/mapr/reviewtable" val writedb = df7.writeStream .format(MapRDBSourceConfig.Format) .option(MapRDBSourceConfig.TablePathOption, tableName) .option(MapRDBSourceConfig.IdFieldPathOption, "_id") .option(MapRDBSourceConfig.CreateTableOption, false) .option("checkpointLocation", "/tmp/reviewdb") .option(MapRDBSourceConfig.BulkModeOption, true) .option(MapRDBSourceConfig.SampleSizeOption, 1000) writedb.start() </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image8-1607499120962.png" alt=""></p> <h2>Querying MapR Database JSON with Spark SQL</h2> <p>The Spark MapR Database Connector enables users to perform complex SQL queries and updates on top of MapR Database using a Spark Dataset, while applying critical techniques such as projection and filter pushdown, custom partitioning, and data locality.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image19-1607499129679.png" alt=""></p> <h2>Loading Data from MapR Database into a Spark Dataset</h2> <p>To <a href="https://docs.datafabric.hpe.com/62/Spark/LoadDataFromMapRDBasDataset.html">load data from a MapR Database JSON</a> table into an Apache Spark Dataset, we invoke the <code>loadFromMapRDB</code> method on a SparkSession object, providing the tableName, schema, and case class. This returns a Dataset of with the product review schema:</p> <pre><code class="language-json">val schema = StructType(Array( StructField("_id", StringType, true), StructField("asin", StringType, true), StructField("overall", DoubleType, true), StructField("reviewText", StringType, true), StructField("reviewTime", StringType, true), StructField("reviewerID", StringType, true), StructField("reviewerName", StringType, true), StructField("summary", StringType, true), StructField("label", StringType, true), StructField("prediction", StringType, true), StructField("unixReviewTime", LongType, true) )) var tableName: String = "/user/mapr/reviewtable" val df = spark .loadFromMapRDB(tableName, schema) df.createOrReplaceTempView("reviews") </code></pre> <h2>Explore and Query the Product Review Data with Spark SQL</h2> <p>Now we can query the data that is continuously streaming into MapR Database to ask questions with the Spark DataFrames domain-specific language or with Spark SQL.</p> <p>Below, we use the DataFrames select and show methods to display the first 5 rows review summary , overall rating, label, and prediction in tabular format:</p> <pre><code class="language-python">df.select("summary","overall","label","prediction").show(5) result: +--------------------+-------+-----+----------+ | summary|overall|label|prediction| +--------------------+-------+-----+----------+ | Excellent Ammo Can| 5.0| 1.0| 1.0| | Glad I bought it| 5.0| 1.0| 1.0| |WILL BUY FROM AGA...| 5.0| 1.0| 1.0| |looked brand new ...| 5.0| 1.0| 1.0| | I LOVE THIS THING| 5.0| 1.0| 1.0| +--------------------+-------+-----+----------+ </code></pre> <p><strong>What are the products with the most high ratings?</strong></p> <pre><code class="language-python">df.filter($"overall" === 5.0) .groupBy("overall","asin") .count .orderBy(desc("count")).show(2) result: +-------+----------+-----+ |overall| asin|count| +-------+----------+-----+ | 5.0|B004TNWD40| 242| | 5.0|B004U8CP88| 201| +-------+----------+-----+ </code></pre> <p><strong>Or in SQL What are the products with the most high ratings?</strong></p> <pre><code class="language-sql">%sql SELECT asin,overall, count(overall) FROM reviews where overall=5.0 GROUP BY asin, overall order by count(overall) desc limit 2 </code></pre> <p><strong>Display the best rated product reviews text</strong></p> <pre><code class="language-python">df.select("summary","reviewText","overall","label","prediction").filter("asin='B004TNWD40'").show(5) result: +--------------------+--------------------+-------+-----+----------+ | summary| reviewText|overall|label|prediction| +--------------------+--------------------+-------+-----+----------+ | Awesome|This is the perfe...| 5.0| 1.0| 1.0| |for the price you...|Great first knife...| 5.0| 1.0| 1.0| |Great Mora qualit...|I have extensive ...| 4.0| 1.0| 1.0| | Amazing knife|All I can say is ...| 5.0| 1.0| 1.0| |Swedish Mil. Mora...|Overall a nice kn...| 4.0| 1.0| 1.0| +--------------------+--------------------+-------+-----+----------+ </code></pre> <p><strong>Or in SQL:</strong></p> <pre><code class="language-sql">%sql select summary, label, prediction, overall from reviews where asin='B004TNWD40' order by overall desc </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image17-1607499137170.png" alt=""></p> <p><strong>What are the products with the highest count of low ratings?</strong></p> <pre><code class="language-python">df.filter($"overall" === 1.0) .groupBy("overall","asin") .count.orderBy(desc("count")).show(2) result: +-------+----------+-----+ |overall| asin|count| +-------+----------+-----+ | 1.0|B00A17I99Q| 18| | 1.0|B00BGO0Q9O| 17| +-------+----------+-----+ </code></pre> <p><strong>Display the product reviews text for Product with highest count of low ratings</strong></p> <pre><code class="language-python">df.select("summary","reviewText","overall","label","prediction") .filter("asin='B00A17I99Q'") .orderBy("overall").show(8) result: result: +--------------------+--------------------+-------+-----+----------+ | summary| reviewText|overall|label|prediction| +--------------------+--------------------+-------+-----+----------+ | DO NOT BUY!|Do your research ...| 1.0| 0.0| 0.0| | Returned it|I could not get t...| 1.0| 0.0| 0.0| | didn't do it for me|didn't like it. ...| 1.0| 0.0| 0.0| |Fragile, just lik...|Update My second....| 1.0| 0.0| 0.0| |Almost perfect de...|I waited a while ...| 1.0| 0.0| 0.0| |Not all its crack...|I started with th...| 1.0| 0.0| 0.0| | Returned...|I gave it as a gi...| 1.0| 0.0| 0.0| |Defective product...|1st jawbone up 2n...| 1.0| 0.0| 0.0| +--------------------+--------------------+-------+-----+----------+ </code></pre> <p>Below, we calculate some prediction evaluation metrics for the streaming data continuously stored in MapR Database. The number of false/true positives:</p> <ul> <li>True positives are how often the model correctly positive sentiment.</li> <li>False positives are how often the model incorrectly positive sentiment..</li> <li>True negatives indicate how often the model correctly negative sentiment.</li> <li>False negatives indicate how often the model incorrectly negative sentiment.</li> </ul> <pre><code class="language-python">val lp = predictions.select("label", "prediction") val counttotal = predictions.count() val correct = lp.filter($"label" === $"prediction").count() val wrong = lp.filter(not($"label" === $"prediction")).count() val ratioWrong = wrong.toDouble / counttotal.toDouble val lp = predictions.select( "prediction","label") val counttotal = predictions.count().toDouble val correct = lp.filter($"label" === $"prediction") .count() val wrong = lp.filter("label != prediction") .count() val ratioWrong=wrong/counttotal val ratioCorrect=correct/counttotal val truen =( lp.filter($"label" === 0.0) .filter($"label" === $"prediction") .count()) /counttotal val truep = (lp.filter($"label" === 1.0) .filter($"label" === $"prediction") .count())/counttotal val falsen = (lp.filter($"label" === 0.0) .filter(not($"label" === $"prediction")) .count())/counttotal val falsep = (lp.filter($"label" === 1.0) .filter(not($"label" === $"prediction")) .count())/counttotal val precision= truep / (truep + falsep) val recall= truep / (truep + falsen) val fmeasure= 2 * precision * recall / (precision + recall) val accuracy=(truep + truen) / (truep + truen + falsep + falsen) results: counttotal: Double = 84160.0 correct: Double = 76925.0 wrong: Double = 7235.0 truep: Double = 0.8582461977186312 truen: Double = 0.05578659695817491 falsep: Double = 0.014543726235741445 falsen: Double = 0.07142347908745247 ratioWrong: Double = 0.08596720532319392 ratioCorrect: Double = 0.9140327946768061 </code></pre> <h2>Projection and Filter Push Down into MapR Database</h2> <p>You can see the physical plan for a DataFrame query by calling the explain method shown below. Here in red we see projection and filter push down, which means that the scanning of the <em>overall</em> and <em>summary</em> columns and the filter on the <em>overall</em> column are pushed down into MapR Database, which means that the scanning and filtering will take place in MapR Database before returning the data to Spark. Projection pushdown minimizes data transfer between MapR Database and the Spark engine by omitting unnecessary fields from table scans. It is especially beneficial when a table contains many columns. Filter pushdown improves performance by reducing the amount of data passed between MapR Database and the Spark engine when filtering data.</p> <pre><code class="language-python">// notice projection of selected fields [summary] // notice PushedFilters: overall df.filter("overall > 3").select("summary").explain result: == Physical Plan == \*(1) Project [summary#7] +- \*(1) Filter (isnotnull(overall#2) &#x26;&#x26; (overall#2 > 3.0)) +- \*(1) Scan MapRDBRelation MapRDBTableScanRDD [summary#7,overall#2] **PushedFilters: [IsNotNull(overall), GreaterThan(overall,3.0)],** ReadSchema: struct&#x3C;summary:string,overall:double> </code></pre> <h2>Querying the Data with Apache Drill</h2> <p>Apache Drillis an open source, low-latency query engine for big data that delivers interactive SQL analytics at petabyte scale. Drill provides a massively parallel processing execution engine, built to perform distributed query processing across the various nodes in a cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image12-1607499144193.png" alt=""></p> <p>With Drill, you can use SQL to interactively query and join data from files in JSON, Parquet, or CSV format, Hive, and NoSQL stores, including HBase, MapR Database, and Mongo, without defining schemas. MapR provides a <a href="https://package.mapr.com/tools/MapR-JDBC/MapR_Drill/">Drill JDBC</a> driver that you can use to connect Java applications, BI tools, such as SquirreL and Spotfire, to Drill.</p> <h2>Below are some example SQL queries using the Drill shell.</h2> <p>Start the Drill shell with:</p> <p><code>sqlline -u jdbc:drill:zk=localhost:5181 -n mapr -p mapr</code></p> <p><strong>How many streaming product reviews were stored in MapR Database?</strong></p> <pre><code class="language-python">select count(_id) as totalreviews from dfs.`/user/mapr/reviewtable`; result: +---------------+ | totalreviews | +---------------+ | 84160 | +---------------+ </code></pre> <p><strong>How many reviews are there for each rating?</strong></p> <pre><code class="language-python">select overall, count(overall) as countoverall from dfs.`/user/mapr/reviewtable` group by overall order by overall desc; result: +----------+---------------+ | overall | countoverall | +----------+---------------+ | 5.0 | 57827 | | 4.0 | 20414 | | 2.0 | 3166 | | 1.0 | 2753 | +----------+---------------+ </code></pre> <p><strong>What are the products with the most high review ratings?</strong></p> <pre><code class="language-python">select overall, asin, count(*) as ratingcount, sum(overall) as ratingsum from dfs.`/user/mapr/reviewtable` group by overall, asin order by sum(overall) desc limit 5; result: +----------+-------------+--------------+------------+ | overall | asin | ratingcount | ratingsum | +----------+-------------+--------------+------------+ | 5.0 | B004TNWD40 | 242 | 1210.0 | | 5.0 | B004U8CP88 | 201 | 1005.0 | | 5.0 | B006QF3TW4 | 186 | 930.0 | | 5.0 | B006X9DLQM | 183 | 915.0 | | 5.0 | B004RR0N8Q | 165 | 825.0 | +----------+-------------+--------------+------------+ </code></pre> <p><strong>What are the products with the most positive review predictions?</strong></p> <pre><code class="language-python">select prediction, asin, count(*) as predictioncount, sum(prediction) as predictionsum from dfs.`/user/mapr/reviewtable` group by prediction, asin order by sum(prediction) desc limit 5; result: +-------------+-------------+------------------+----------------+ | prediction | asin | predictioncount | predictionsum | +-------------+-------------+------------------+----------------+ | 1.0 | B004TNWD40 | 263 | 263.0 | | 1.0 | B004U8CP88 | 252 | 252.0 | | 1.0 | B006X9DLQM | 218 | 218.0 | | 1.0 | B006QF3TW4 | 217 | 217.0 | | 1.0 | B004RR0N8Q | 193 | 193.0 | +-------------+-------------+------------------+----------------+ </code></pre> <p><strong>Show the review summaries for the product with the most high review ratings</strong></p> <pre><code class="language-python">select summary, prediction from dfs.`/user/mapr/reviewtable` where asin='B004TNWD40' limit 5; result: +---------------------------------------------------+-------------+ | summary | prediction | +---------------------------------------------------+-------------+ | Awesome | 1.0 | | for the price you cant go wrong with this knife | 1.0 | | Great Mora quality and economy | 1.0 | | Amazing knife | 1.0 | | Swedish Mil. Mora Knife | 1.0 | +---------------------------------------------------+-------------+ </code></pre> <p><strong>Show the review tokens for the product with the most positive reviews</strong></p> <pre><code class="language-python">select reviewTokens from dfs.`/user/mapr/reviewtable` where asin='B004TNWD40' limit 1; [ "awesome", "perfect", "belt/pocket/neck", "knife", "carbon", "steel", "blade", "last", "life", "time!", "handle", "sheath", "plastic", "cheap", "kind", "plastic", "durable", "also", "last", "life", "time", "everyone", "loves", "doors", "this!", "yes", "ones", "bone", "handles", "leather", "sheaths", "$100+" ] </code></pre> <p><strong>What are the products with the most low review ratings?</strong></p> <pre><code class="language-python">SELECT asin,overall, count(overall) as rcount FROM dfs.`/user/mapr/reviewtable` where overall=1.0 GROUP BY asin, overall order by count(overall) desc limit 2 result: +-------------+----------+---------+ | asin | overall | rcount | +-------------+----------+---------+ | B00A17I99Q | 1.0 | 18 | | B008VS8M58 | 1.0 | 17 | +-------------+----------+---------+ </code></pre> <p><strong>What are the products with the most negative review predictions?</strong></p> <pre><code class="language-python">select prediction, asin, count(*) as predictioncount, sum(prediction) as predictionsum from dfs.`/user/mapr/reviewtable` group by prediction, asin order by sum(prediction) limit 2; result: +-------------+-------------+------------------+----------------+ | prediction | asin | predictioncount | predictionsum | +-------------+-------------+------------------+----------------+ | 0.0 | B007QEUWSI | 4 | 0.0 | | 0.0 | B007QTHPX8 | 4 | 0.0 | +-------------+-------------+------------------+---------------+ </code></pre> <p><strong>Show the review summaries for the product with the most low review ratings</strong></p> <pre><code class="language-python">select summary from dfs.`/user/mapr/reviewtable` where asin='B00A17I99Q' and prediction=0.0 limit 5; result: +---------------------------------------------------------+ | summary | +---------------------------------------------------------+ | A comparison to Fitbit One -- The Holistic Wrist | | Fragile, just like the first Jawbone UP! Overpriced | | Great concept, STILL horrible for daily use | | Excellent idea, bad ergonomics, worse manufacturing... | | get size larger | +---------------------------------------------------------+ </code></pre> <h2>Querying the Data with the MapR Database Shell</h2> <p>The mapr dbshell is a tool that enables you to create and perform basic manipulation of JSON tables and documents. You run dbshell by typing mapr dbshell on the command line after logging into a node in a MapR cluster.</p> <h2>Below are some example queries using the MapR dbshell</h2> <p><strong>Show the review summary, id, prediction for the product with the most high review ratings (<code>_id starts with B004TNWD40</code>)</strong></p> <pre><code class="language-python">find /user/mapr/reviewtable --where '{"$and":[{"$eq":{"overall":5.0}}, { "$like" : {"_id":"%B004TNWD40%"} }]}' --f _id,prediction,summary --limit 5 result: {"_id":"B004TNWD40_1256083200","prediction":1,"summary":"Awesome"} {"_id":"B004TNWD40_1257120000","prediction":1,"summary":"for the price you cant go wrong with this knife"} {"_id":"B004TNWD40_1279065600","prediction":1,"summary":"Amazing knife"} {"_id":"B004TNWD40_1302393600","prediction":1,"summary":"Great little knife"} {"_id":"B004TNWD40_1303257600","prediction":1,"summary":"AWESOME KNIFE"} </code></pre> <p><strong>Show the review summary, id, for 10 products with negative sentiment prediction and label</strong></p> <pre><code class="language-python">find /user/mapr/reviewtable --where '{"$and":[{"$eq":{"prediction":0.0}},{"$eq":{"label":0.0}} ]}' --f _id,summary --limit 10 result: {"_id":"B003Y64RBA_1312243200","summary":"A $3.55 rubber band!"} {"_id":"B003Y64RBA_1399334400","summary":"cheap not worthy"} {"_id":"B003Y71V2C_1359244800","summary":"Couple of Problems"} {"_id":"B003Y73EPY_1349740800","summary":"Short Term Pedals - Eggbeaters 1"} {"_id":"B003Y9CMGY_1306886400","summary":"Expensive batteries."} {"_id":"B003YCWFRM_1336089600","summary":"Poor design"} {"_id":"B003YCWFRM_1377043200","summary":"Great while it lasted"} {"_id":"B003YD0KZU_1321920000","summary":"No belt clip!!! Just like the other reviewer..."} {"_id":"B003YD0KZU_1338768000","summary":"Useless"} {"_id":"B003YD1M5M_1354665600","summary":"Can't recomend this knife."} </code></pre> <h2>Summary</h2> <p>In this post, you learned how to use the following:</p> <ul> <li>A Spark machine learning model in a Spark Structured Streaming application</li> <li>Spark Structured Streaming with MapR Event Store to ingest messages using the Kafka API</li> <li>Spark Structured Streaming to persist to MapR Database for continuously rapidly available SQL analysis</li> </ul> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform. The MapR Data Platform integrates global event streaming, real-time database capabilities, and scalable enterprise storage with Spark, Drill, and machine learning libraries to power the development of next-generation intelligent applications, which take advantage of modern computational paradigms powered by modern computational infrastructure.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image23-1607499151723.jpg" alt=""></p> <h2>Code</h2> <p>All of the data and code to train the models and make your own conclusions, using Apache Spark, are located in GitHub, Refer to github readme for more information about running the code.</p> <ul> <li><a href="https://github.com/caroljmcdonald/mapr-sparkml-sentiment-classification">https://github.com/caroljmcdonald/mapr-sparkml-sentiment-classification</a></li> </ul> <p><em>Editor's Note: This tutorial is the second part in a series related to this topic. The first part in this series is found here: <a href="/blog/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-">Part 1.</a></em></p>https://developer.hpe.com/kubernetes-tutorial-part-2-of-3-how-to-install-and-deploy-applications-at-scale-on-k8s/https://developer.hpe.com/kubernetes-tutorial-part-2-of-3-how-to-install-and-deploy-applications-at-scale-on-k8s/Wed, 31 Mar 2021 15:14:49 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Martijn Kieboom", "publish": "2018-04-26T10:46:00.000", "tags": "open-source-software" </code></pre> <hr> <p><em><strong>Editor’s Note:</strong></em> <a href="/blog/kubernetes-tutorial-how-to-install-and-deploy-applications-at-scale-on-k">Part 1 in this series</a></p> <h2>Introduction</h2> <p>In my previous blog, I described the reasoning behind containers and how to manage them in large-scale production environments. This blog describes two ways to get started with Kubernetes:</p> <p>For a small Kubernetes deployment, we will be using Minikube, a simple solution to run a single node environment locally on your environment using virtualization technology like VirtualBox or VMware.</p> <p>After having some hands-on experience with Kubernetes using the Minikube environment, it is time to deploy an actual multi-node Kubernetes cluster. For that, we will be using a three node environment, running Red Hat/CentOS 7.x.</p> <p>Time to roll up your sleeves and get started!</p> <h2>Kubernetes Installation Using Minikube</h2> <p>If you don't have experience with Kubernetes, Minikube is a perfect way to take your first steps into the container management world. Minikube is an easy way to deploy a single node Kubernetes environment.</p> <p><strong>Prerequisites</strong></p> <p>Minikube leverages virtualization technology, like VirtualBox, VMware, Hyper-V, and others. So prior to running Minikube, make sure you have one of these virtualization technologies installed on your client. For a complete list of supported virtualization layers, visit the Minikube GitHub page at: <a href="https://github.com/kubernetes/minikube">https://github.com/kubernetes/minikube</a></p> <p>With a virtualization technology installed on your machine, let's go ahead and install Minikube.</p> <p><strong>Download Minikube</strong></p> <p>Minikube comes in a precompiled, single executable file that you can simply download to your client:</p> <pre><code class="language-bash"># Install minikube on MacOS curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 &#x26;&#x26; chmod +x minikube &#x26;&#x26; sudo mv minikube /usr/local/bin/ # Install minikube on Linux curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 &#x26;&#x26; chmod +x minikube &#x26;&#x26; sudo mv minikube /usr/local/bin/ # Install minikube on Windows # Download the executable file and save it as minikube.exe: https://storage.googleapis.com/minikube/releases/latest/minikube-windows-amd64.exe </code></pre> <p><strong>Launch a Kubernetes Virtual Cluster</strong></p> <p>With Minikube installed, it is time to launch our virtual single node Kubernetes cluster.</p> <pre><code class="language-bash"># Launch Kubernetes cluster minikube start # Alternatively, launch Kubernetes cluster with a specific Memory allocation # default memory is 2048 MB minikube start --memory 6144 </code></pre> <p>That's all! No, really, it is.</p> <p>Minikube will automatically download, import,and launch an image matching your virtualization technology. For that reason it does take a few minutes to complete. Notice that Minikube automatically imported a virtual machine into your virtualization environment.</p> <p>There are many parameters you can configure as part of the 'start' command ('minikube start --help'), but as Minikube is aimed at ease of usage, the default values are a very solid standard to get started. One recommended parameter is to set the virtual machine's memory as configuration. It defaults to 2GB which might be on the small size if you want to try out a few containers on the Minikube cluster.</p> <p>With the Kubernetes cluster running, we can use Minikube to connect to the VM and launch the Kubernetes Web Dashboard:</p> <pre><code class="language-bash"># Launch Kubernetes Dashboard minikube dashboard </code></pre> <p>This will automatically launch your default browser and opens the Kubernetes dashboard. Whereas Kubernetes clusters will normally ask for a security token to login, Minikube is aimed at ease of use and therefore no login is required. Have a look around at the various Kubernetes menu options.</p> <p>From here, you can now start deploying the MapR Volume Driver Plugin for Kubernetes.</p> <p>If at any point in time you want to stop the Minikube cluster, it is as easy as you can imagine:</p> <pre><code class="language-bash"># Stop Kubernetes minikube stop </code></pre> <p>The 'stop' command will gently shutdown your virtualized Kubernetes cluster, including all containers that you might have deployed on it. A simple 'minikube start' will relaunch the existing VM again, after which you can use 'minikube dashboard' to open the Kubernetes Dashboard.</p> <p>Once you're done with the Minikube Kubernetes cluster, simply delete the VM:</p> <pre><code class="language-bash"># Delete the Kubernetes VM minikube delete </code></pre> <h2>Kubernetes Installation on Red Hat/CentOS 7.x</h2> <p>Once you have experience with Kubernetes by using Minikube, it is time to deploy a multi-node Kubernetes cluster. This paragraph describes how to deploy a multi-node Kubernetes cluster on an environment running CentOS as the Operating System.</p> <p>One final note before we start: do not openly connect this deployed cluster to the internet as securing the Kubernetes cluster is out of scope for this blog.</p> <p><strong>Prerequisites</strong></p> <p>The commands in this prerequisite chapter have to be executed on each of the Kubernetes cluster nodes individually.</p> <p>If you're using AWS EC2 nodes, make sure to enable the 'extra’ repository:</p> <pre><code class="language-bash"># On AWS EC2 enable the 'extra' repository containing git, docker, etc. yum-config-manager --enable rhui-REGION-rhel-server-extras </code></pre> <p>To get started, we need to disable SELinux as well as memory swapping on all nodes:</p> <pre><code class="language-bash"># Disable SELinux setenforce 0 sed -i '/^SELINUX./ { s/enforcing/disabled/; }' /etc/selinux/config # Disable swap swapoff -a sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab </code></pre> <p>Additionally, we also need to enable bridged networking for Kubernetes:</p> <pre><code class="language-bash"># Set iptables cat &#x3C; /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system </code></pre> <p>And, as the last step of the prerequisites, we need to install Docker to run containers:</p> <pre><code class="language-bash"># Install docker yum install -y docker # Launch Docker and enable it on system boot systemctl start docker systemctl enable docker </code></pre> <p><strong>Kubernetes Installation</strong></p> <p>The following commands also have to be executed on each of the individual cluster nodes.</p> <p>Add the Kubernetes repository, and install the Kubernetes tools—kubelet, kubeadm, and kubectl:</p> <pre><code class="language-bash"># Install kubernetes repo cat &#x3C; /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF # Install Kubernetes and start it yum install -y kubelet kubeadm kubectl systemctl start kubelet systemctl enable kubelet </code></pre> <p>At this point, we have the Kubernetes packages installed on all cluster nodes. We can now start configuring Kubernetes. To do so, run the following commands on only one (1) single cluster node.</p> <p>On the first node of the cluster, initialize the Kubernetes master by running kubeadm init. Please note that this task might take minutes to complete, as it will pull in all required containers.</p> <pre><code class="language-bash"># Initialize Kubernetes master # Validate the ip-address of the node: hostname --ip-address # If the ip address in the above command is correct, run the following. # Otherwise manually provide the correct address for apiserver-advertise-address kubeadm init --pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=$(hostname --ip-address) # The kubeadm command will take a few minutes and it will print a 'kubeadm join' # command once completed. Make sure to capture and store this 'kubeadm join' # command as it is required to add other nodes to the Kubernetes cluster </code></pre> <p>Once the cluster is initialized, we can copy the generated configuration file (admin.conf) to the home directory ($HOME/.kube/config) for easy cluster administration using the kubectl cli:</p> <pre><code class="language-bash"># Deploy kube config mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config </code></pre> <p>To allow the pods and containers to communicate over the network with each other, a cluster network is required to set up. Flannel is one of the various cluster networking solutions that we will use in this blog. For more information on Kubernetes networking, visit: <a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/">https://kubernetes.io/docs/concepts/cluster-administration/networking/</a></p> <pre><code class="language-bash"># Install Flanner for network # Doc: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#44-joining-your-nodes kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml </code></pre> <p>Per default,Kubernetes doesn't run pods on the master node as that could potentially result in a resource as well as security conflict. Pods might require such a large amount of system resources that the Kubernetes master might be negatively impacted. For single node clusters, however, (in case of testing, etc.), you can enforce pods to run on the master node as follows:</p> <pre><code class="language-bash"># Allow pods on master (not recommended for production clusters) kubectl taint nodes --all node-role.kubernetes.io/master- </code></pre> <p>It will take a couple of minutes for all containers to start. Use the following command to validate that all pods are running prior to continuing:</p> <pre><code class="language-bash"># Validate all pods are running kubectl get pods --all-namespaces </code></pre> <p>You can manage Kubernetes completely via the command line tool kubectl, but having a visual and graphical user interface to manage the cluster state can be very useful as well. To do so, let’s deploy the Kubernetes Dashboard:</p> <pre><code class="language-bash"> # Deploy Dashboard web ui # https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/ kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml </code></pre> <p>As the Kubernetes dashboard is also running in a Docker container, we need to modify its networking to access the dashboard from the outside world:</p> <pre><code class="language-bash"># Edit the dashboard to be open to the world (for demos only!) # Change type from 'ClusterIP' to 'NodePort' kubectl -n kube-system edit service kubernetes-dashboard </code></pre> <p>Kubernetes user administration is out of scope for this blog post. Instead, we will be allowing the default kube-system user to become a cluster administrator user. Again: not for production clusters!</p> <pre><code class="language-bash"># Make default user part of cluster admin # not for production clusters, for demos only! kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin </code></pre> <p>To log on into the Kubernetes dashboard, a login token is required. To obtain the login ticket:</p> <pre><code class="language-bash"># Get the login token kubectl describe serviceaccount default -n kube-system kubectl describe secret default-token -n kube-system </code></pre> <p>Alternatively, it is possible to open the dashboard without login required (once again: not required for production systems). Simply click the 'skip' button in the Kubernetes dashboard login page after applying the following:</p> <pre><code class="language-bash"># Allow opening the k8 dashboard without login (not for production clusters, for demos only!) cat &#x3C; k8auth.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard labels: k8s-app: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system EOF kubectl create -f k8auth.yaml </code></pre> <p>Pods on Kubernetes will, by default, open networking ports in the 30000+ range. To get the Kubernetes Dashboard port number, execute the following:</p> <pre><code class="language-bash"># Get the port that kubernetes dashboard runs on (should be a port in 30000+ range) kubectl -n kube-system get service kubernetes-dashboard </code></pre> <p>Launch the Kubernetes dashboard in your favorite internet browser:</p> <pre><code class="language-bash"># Open browser and connect to the Kubernetes Dashboard https://: </code></pre> <p>If you run into networking issues when connecting to the Dashboard, try using the Kubernetes proxy to connect to the Kubernetes internal networking:</p> <pre><code class="language-bash"># If unable to connect to Dashboard, try using the Kubernetes proxy: kubectl proxy # With proxy running, open the Dashboard using the following url: http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ </code></pre> <p>With the master node up and running, it is possible to add additional nodes to your Kubernetes cluster. To do so, use the 'kubeadm join …' command, as noted earlier in this blog. Please note, however, that the kubeadm command uses security tokens to authenticate itself with the master node. These tokens will expire after 24 hours, after which a new token has to be generated as explained below:</p> <pre><code class="language-bash"># Add additional nodes to the cluster (if required) using the earlier noted kubeadm join command kubeadm join … # On Master, show all nodes part of the cluster: kubectl get nodes # In case the token to join has expired, create a new token: # On Master, list the existing tokens: kubeadm token list # On Master, if there are no valid tokens, create a new token and list it: kubeadm token create kubeadm token list # Join additional nodes in the cluster with the newly created token, e.g.,: kubeadm join 172.16.1.125:6443 --discovery-token-unsafe-skip-ca-verification --token 5d4164.15b01d9af2e64824 </code></pre> <p>That's it: you now have a multi-node Kubernetes environment running!</p> <p><strong>Troubleshooting and Reset</strong></p> <p>When running into issues, use the following command to print logging information:</p> <pre><code class="language-bash"># Troubleshooting journalctl -xeu kubelet </code></pre> <p>To remove a node from the cluster:</p> <pre><code class="language-bash"># On master, remove a node from the cluster (hard) kubectl get nodes kubectl delete nodes # On the removed node, reset and uninstall ubernetes installation kubeadm reset yum erase kube* -y </code></pre> <h2>MapR Volume Driver Plugin for Kubernetes</h2> <p>The MapR Volume Driver Plugin for Kubernetes allows running anyDocker container from Docker Hub on a Kubernetes cluster where MapR is the persistent data store for the container. Deployment of the MapR Volume Driver Plugin is very straightforward and will work both on the previously described Minikube and on the CentOS Kubernetes environments.</p> <p>Make sure to check the latest documentation of the Volume Driver Plugin for any changes: <a href="https://docs.datafabric.hpe.com/62/PersistentStorage/kdf_installation.html">https://docs.datafabric.hpe.com/62/PersistentStorage/kdf_installation.html</a>.</p> <p>The MapR Volume Driver Plugin (like any other Kubernetes configuration and deployment) consists of various so-called yaml files to configure and deploy pods and containers. The yaml files for the MapR Volume Driver Plugin can be found on the public <a href="https://package.mapr.com/">package.mapr.com</a> repository:</p> <pre><code class="language-bash"># Locate and check the latest version of the MapR Volume Driver Plugin: https://package.mapr.com/tools/KubernetesDataFabric/ # To download the version 1.0.0 files, for example: wget https://package.mapr.com/tools/KubernetesDataFabric/v1.0.0/kdf-namespace.yaml wget https://package.mapr.com/tools/KubernetesDataFabric/v1.0.0/kdf-rbac.yaml wget https://package.mapr.com/tools/KubernetesDataFabric/v1.0.0/kdf-plugin-centos.yaml wget https://package.mapr.com/tools/KubernetesDataFabric/v1.0.0/kdf-provisioner.yaml </code></pre> <p>With the yaml files downloaded, it is required to specify the ip address of the Kubernetes master in the Volume Driver Plugin yaml configuration:</p> <pre><code class="language-bash"># Configure the MapR Kubernetes storage plugin to point to the Kubernetes Master: vi kdf-plugin-centos.yaml - name : KUBERNETES_SERVICE_LOCATION value: "changeme!:6443" # Set the KUBERNETES_SERVICE_LOCATION ip to match your Kubernetes master node </code></pre> <p>The final step is to deploy the Kubernetes configuration files to launch the MapR Volume Driver Plugin. We will start with creating the "mapr-system" namespace (kdf-namespace.yaml) to run the Volume Driver Plugin in. Additionally, we set the role-based access control (kdf-rbac.yaml) so that containers on Kubernetes can access the MapR Volume Driver Plugin. Finally, we will deploy the MapR Volume Driver Plugin.</p> <p>Simply use the kubectl command line tool to load the yaml files into Kubernetes:</p> <pre><code class="language-bash"># Launch the various yaml files to deploy the MapR Volume Driver Plugin kubectl create -f kdf-namespace.yaml kubectl create -f kdf-rbac.yaml kubectl create -f kdf-plugin-centos.yaml kubectl create -f kdf-provisioner.yaml </code></pre> <p>That's it! Check your Kubernetes Dashboard to validate the deployment status by navigating to the overview of the 'mapr-system' namespace.</p> <p>To remove the MapR Volume Driver Plugin from your Kubernetes cluster:</p> <pre><code class="language-bash"># Remove the MapR Volume Driver Plugin: kubectl delete -f kdf-plugin-centos.yaml kubectl delete -f kdf-provisioner.yaml kubectl delete -f kdf-rbac.yaml kubectl delete -f kdf-namespace.yaml </code></pre> <h2>Additional Resources:</h2> <ul> <li> <p><a href="/blog/kubernetes-tutorial-how-to-install-and-deploy-applications-at-scale-on-k">Part 1 in this series</a></p> </li> <li> <p><a href="https://kubernetes.io/docs/setup/independent/install-kubeadm/">https://kubernetes.io/docs/setup/independent/install-kubeadm/</a></p> </li> <li> <p><a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/">https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/</a></p> </li> <li> <p><a href="https://mapr.com/docs/home/PersistentStorage/kdf_overview.html">https://mapr.com/docs/home/PersistentStorage/kdf_overview.html</a></p> </li> </ul>https://developer.hpe.com/open-source-contributor-explains-how-kubedirector-empowers-data-intensive-apps/https://developer.hpe.com/open-source-contributor-explains-how-kubedirector-empowers-data-intensive-apps/Thu, 18 Mar 2021 09:52:53 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/3/kartik-blog-small-1616160879068.jpg" alt="Kartik Mathur"></p> <p>As a leading global, <strong>edge-to-cloud platform-as-a-service company</strong>, Hewlett Packard Enterprise (HPE) prides itself in employing team members who share one common purpose: to advance the way people live and work. In this blog series, you’ll get to meet a number of them as I interview some of the <a href="https://www.hpe.com/us/en/open-source.html">open source</a> experts who make up the HPE team.</p> <p>After having graduated with his MS in computer science from Indiana University Bloomington with a research focus on parallel computing and distributed systems, Kartik Mathur started his professional career at AMD. He then worked at several technology startup companies, including BlueData, which was acquired by HPE in 2019. Kartik is currently a Master Technologist at HPE and leads the MLOps initiatives on the HPE Ezmeral Software Platform. He has a keen interest in container orchestration for large scale data processing clusters and is a major contributor to the KubeDirector open source project.</p> <h2>How did you get started with open source technologies?</h2> <p>I first started contributing to open source for a project that lets you query Cassandra tables using sparkSQL. But honestly, almost any piece of software that I’ve written or consumed was based on open source projects or libraries in some capacity. In my role at BlueData, I first used KubeDirector to orchestrate Big Data and ML pipelines, but then I started contributing as well. I found it so engaging that I’m currently the leading contributor and maintainer for the KubeDirector project.</p> <h2>What makes KubeDirector so special?</h2> <p>KubeDirector empowers application developers to deploy their applications as a custom resource without having to implement a full-blown Kubernetes Operator. It decouples the operator boiler plate using application intelligence.</p> <p>Basically, KubeDirector works as a custom controller for generic applications. It uses standard Kubernetes (K8s) facilities of custom resources and API extensions to implement stateful scaleout application clusters. This is a unique approach that enables the transparent integration with K8s user/resource management, as well as existing K8s clients and tools.</p> <p>KubeDirector is unique since it has a rich catalog of complex stateful applications as part of the open-source codebase. And we are constantly adding more and more applications that developers can use as a template/example to onboard their application of choice on Kubernetes. Without KubeDirector, this would be an intimidating task with a huge learning curve, especially for implementing Day2 operations for their applications, like scaling in and out.</p> <h2>How do HPE customers benefit from KubeDirector?</h2> <p>Over the course of the last few years, Kubernetes has become the de-facto standard for orchestrating containerized applications. It’s worked quite well for stateless applications; less so for stateful applications, such as those focused on AI, machine learning, and big data analytics.</p> <p>When containers were first introduced as a way to package microservices, they were designed to be entirely stateless and ephemeral. A container would spin up, do its job, and then disappear, without leaving any record of what happened while it was running. Stateful applications save data to persistent disk storage for use by the server, clients, and other applications. An example of a stateful application is a database or key-value store to which data is saved and retrieved by other applications. There are tools, such as Statefulset and Persistent Volumes, which help developers build stateful applications on Kubernetes, but it all becomes quite difficult to manage as an application scales.</p> <p>Since KubeDirector provides an application-agnostic deployment pattern, it enables developers to run non-cloud native stateful applications on Kubernetes without modifying any code. It makes it easier to deploy data-intensive distributed applications for AI and analytics use cases, such as Hadoop, Spark, Kafka, TensorFlow, etc., on Kubernetes. HPE Ezmeral customers automatically receive the benefits of KubeDirector because it’s integrated into the HPE Ezmeral Software Platform, enabling them to more easily develop AI and ML applications.</p> <h2>What’s up next for KubeDirector?</h2> <p>The next big ticket item for me to work on is to tighten up the security. I’m currently looking at implementing Istio awareness for Kubedirector application endpoints, providing virtual endpoints to secure the microservices communication between them using JSON Web tokens (JWT).</p> <h2>Is there any advice you’d like to give others who might follow in your footsteps?</h2> <p>Robert Noyce, co-founder of Intel Corporation, once said “Knowledge shared is power multiplied.” I’m a huge fan of this quote. I feel as though open source projects are the biggest testament to the power of knowledge sharing. It gives me immense pleasure in being able to learn and collaborate with people worldwide, as being a part of the open source community helps me grow as an engineer and lift others as well.</p> <p>To learn more about the open source projects that HPE is involved with, please visit <a href="https://www.hpe.com/us/en/open-source.html">our website</a>. Interested in exploring what HPE offers for developers and data scientists? Check out our <a href="https://developer.hpe.com/">HPE DEV site</a> for a ton of articles, workshops, tutorials, and other resources.</p>https://developer.hpe.com/boost-your-analytics-factory-into-hyperdrive/https://developer.hpe.com/boost-your-analytics-factory-into-hyperdrive/Tue, 09 Mar 2021 13:03:33 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ezmeral-day-picture-1615293804678.png" alt=""></p> <p>You know that there’s a treasure trove of value hidden in the data streaming all around you. But how do you capture it, analyze it, and extract that golden nugget of information that impacts your applications? More importantly, how do you do this when the data is constantly changing? Critical applications need to be able to make decisions in real-time or risk the possibility of failure.</p> <p>In today’s data-driven world, being able to distill data into meaningful insights is key. But the deployment of analytics and data science in today’s organizations tends to be spotty and run through uncoordinated efforts. Organizations are looking for a unified experience that spans across environments. In the free, <a href="https://hpe.events.cube365.net/hpe/hpeezmeral">HPE Ezmeral \\ Analytics Unleashed</a> virtual event being held on March 17, 2021, experts will show you how you can industrialize your data to accelerate your data-driven transformation.</p> <p>This 90-minute event, packed with multiple sessions, talks, and demos, will focus on the top challenges many companies face as they attempt to harness the power of their data, and how data scientists are using HPE Ezmeral to overcome these challenges. Experts will demo three real-world Day in the Life examples and discuss best practices in scaling artificial intelligence (AI) to meet the needs of the enterprise.</p> <p>To start the day, Kirk Borne, globally known data scientist and thought leader on all things data-at-edge, will share his perspective on a few key questions many in the industry are asking. He’ll also offer his thoughts on how the HPE Ezmeral portfolio can help. Afterwards, Kumar Sreekanti, CTO &#x26; Head of Software and Robert Christiansen, VP of Strategy, Office of the CTO, will give you an inside scoop on how the HPE Ezmeral portfolio helps ensure data is always accessible, scalable, and secure. Later, you’ll hear from three customers, Ericsson, DXC, and ORock, who will describe how they each are able to unlock the value of their data. Some of the topics they’ll cover include containerizing apps in complex enterprise environments, scaling data intensive workloads with speed and security, and managing legacy transformations.</p> <p>The Day in the Life Demo Showcase will focus on three scenarios:</p> <p>• A Day in the Life of Data, which follows the arc of data as it is intelligently stretched and stitched across the HPE Ezmeral Data Fabric.</p> <p>• A Day in the Life of a Data Scientist, where you’ll get to see a demo on how HPE Ezmeral enables a data scientist to self-serve data, spin-up infrastructure, and leverage analytic toolkits.</p> <p>• A Day in the Life of an IT Admin, where you’ll be walked through basic infrastructure set up so you can see how HPE Ezmeral simplifies the management of infrastructure for analytics.</p> <p>By seeing how others have overcome the challenges of harnessing the power of data through analytics, artificial intelligence, and machine learning, you’ll be able to bring back ideas that will work for your organization. So, take advantage of this free event. There’s no registration required. Simply join us for <a href="https://hpe.events.cube365.net/hpe/hpeezmeral">HPE Ezmeral \\ Analytics Unleashed</a> on March 17th, 2021 at 8 a.m. PT / 4 p.m. GMT.</p> <p>We’ll also be sharing details on how HPE is collaborating with technology partners, like Dataiku and Run:AI through the HPE Ezmeral marketplace – a one-stop shop where customers can come to explore, learn, engage, and deploy with technology partners and open source projects. Join the ISV Ecosystem team and our partners to see how you can leverage a wide variety of different validated partner tools to improve the quality and reliability of your end-to-end solutions.</p> <p>For more details on the event, check out our blog post on <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/Virtual-event-Data-is-your-pot-of-gold-Time-to-unleash-its-value/ba-p/7123545#.YEZsf51Kg2x">HPE Ezmeral: Uncut</a>. Check out the <a href="https://developer.hpe.com/platform/hpe-ezmeral-container-platform/home">HPE Ezmeral Container Platform - now known as HPE Ezmeral Runtime Enterprise</a> on <a href="https://developer.hpe.com/">HPE DEV</a> for more information on the industry’s first enterprise-grade container platform designed to deploy both cloud-native and non cloud-native applications using Kubernetes – running on bare-metal or virtualized infrastructure, on any public cloud, and at the edge. And for more information on the HPE Ezmeral Data Fabric, check out its <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home">platform page here</a>. It enables you to run the right application at the right time in the right place on the right data.</p> <p>Finally, if you want to get some real hands-on experience with the HPE Ezmeral Container Platform, take our <strong>free</strong> <a href="https://learn.ezmeral.software.hpe.com/path/kubernetes-stateful-apps/container-platform-on-demand-developer-demos">Workshop-on-Demand</a>. This is an on-demand training course that leverages Jupyter Notebooks to help you really get to know the technology. You take the course on your own time and interact with our team on Slack if you have any questions. Just <a href="https://slack.hpedev.io/">join us on slack</a> and reach out for the dedicated <a href="https://hpedev.slack.com/archives/C01B60X8SSD">#hpe-workshop-on-demand Channel</a>.</p>https://developer.hpe.com/how-big-data-is-reducing-costs-and-improving-outcomes-in-health-care/https://developer.hpe.com/how-big-data-is-reducing-costs-and-improving-outcomes-in-health-care/Tue, 09 Mar 2021 12:19:17 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2016-06-07T07:00:00.000Z", "tags": "machine-learning, use-case, healthcare" </code></pre> <hr> <h2>The Motivation for Big Data</h2> <p>Health care costs are driving the demand for big-data driven Healthcare applications. U.S. health care spending has outpaced GDP growth for the past several decades and exceeds spending in any other developed country. Despite being more expensive, according to the Organisation for Economic Co-operation and Development (OECD), the US Health System ranks last among eleven countries on measures of access, equity, quality, efficiency, and healthy lives. Standards and incentives for the digitizing and sharing of healthcare data along with improvements and decreasing costs in storage and parallel processing on commodity hardware, are causing a <a href="http://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/the-big-data-revolution-in-us-health-care">big data revolution in health care</a> with the goal of <a href="http://www.nationalacademies.org/hmd/Reports/2012/Best-Care-at-Lower-Cost-The-Path-to-Continuously-Learning-Health-Care-in-America.aspx">better care at lower cost.</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/life-expectancy-1-1611296455217.png" alt=""></p> <h2>Value Based Care</h2> <p>A goal of the Affordable Care Act is to improve health care through the meaningful use of health information technology in order to:</p> <ul> <li> <p>Improve healthcare quality and coordination so that outcomes are consistent with current professional knowledge</p> </li> <li> <p>Reduce healthcare costs, reduce avoidable overuse</p> </li> <li> <p>Provide support for reformed payment structures</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/improved-health-care-2-1611296469408.png" alt=""></p> <p>Health insurance companies, such as Medicare and Medicaid, are shifting from fee-for-service compensation to value-based data-driven incentives that reward high-quality, cost-effective patient care and demonstrate meaningful use of electronic health records.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/improve-health0care-3-1611296489579.png" alt=""></p> <h2>Health Care Data</h2> <p>Unstructured data forms about 80% of information in the healthcare industry and is growing exponentially. Getting access to this unstructured data—such as output from medical devices, doctor’s notes, lab results, imaging reports, medical correspondence, clinical data, and financial data—is an invaluable resource for improving patient care and increasing efficiency.</p> <p>Examples of healthcare data sources that will benefit from big data and analytics:</p> <ul> <li> <p>Claims: are the documents providers submit to insurance companies to get paid. A key component of the <a href="http://www.edibasics.com/edi-resources/document-standards/hipaa/">Health Insurance Portability and Accountability Act (HIPAA)</a> is the establishment of national standards for electronic healthcare transactions in order to improve efficiency by encouraging the widespread use of Electronic Document Interchange (EDI) between healthcare providers and insurance companies. Claim transactions include International Classification of Diseases (ICD) diagnostic codes, medications, dates, provider IDs, the cost, etc.</p> </li> <li> <p>Electronic Health/Medical Record data (EHR or EMR): Medicare and Medicaid EHR <a href="https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/index.html?redirect=/EHRincentivePrograms/">incentive programs</a> were established to encourage professionals and hospitals to adopt and demonstrate meaningful use of certified EHR technology. EHRs facilitate a comprehensive sharing of data with other providers and medical applications. EHRs contain the data from the delivery of healthcare, which includes diagnosis, treatment, prescriptions, lab tests, and radiology. <a href="http://www.hl7.org/implement/standards/index.cfm?ref=nav">Health Level Seven International (HL7)</a> provides standards for the exchange, integration, sharing, and retrieval of electronic health record data.</p> </li> <li> <p>Pharmaceutical R&#x26;D: Clinical Trials Data, Genomic Data.</p> </li> <li> <p>Patient behavior and sentiment data.</p> </li> <li> <p>Medical Device Data: Patient sensor data from the home or hospital.</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/health-data-inputs-4-1611296501518.png" alt=""></p> <h2>Big Data Trends in Healthcare</h2> <p>There is a move toward evidence-based medicine, which involves making use of all clinical data available and factoring that into clinical and advanced analytics. Capturing and bringing all of the information about a patient together gives a more complete view for insight into care coordination and outcomes-based reimbursement, population health management, and patient engagement and outreach.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/converged-data-platform-mapr-5-1611296512091.png" alt=""></p> <h2>Example Healthcare Big Data Use Cases</h2> <h3>Reducing Fraud Waste and Abuse with Big Data Analytics</h3> <p>The cost of fraud, waste and abuse in the healthcare industry is a key contributor to spiraling health care costs in the United States, but big data analytics can be a game changer for health care fraud. The Centers for Medicare and Medicaid Services prevented more than $210.7 million in healthcare fraud in one year using predictive analytics. UnitedHealthcare transitioned to a predictive modeling environment based on a Hadoop big data platform, in order to identify inaccurate claims in a systematic, repeatable way and generated a 2200% return on their big data/advanced technology.</p> <p>The key to identifying fraud is the ability to store and go back in history to analyze large unstructured datasets of historical claims and to use machine-learning algorithms to detect anomalies and patterns.</p> <p>Healthcare organizations can analyze patient records and billing to detect anomalies such as a hospital’s overutilization of services in short time periods, patients receiving healthcare services from different hospitals in different locations simultaneously, or identical prescriptions for the same patient filled in multiple locations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/machine-learning-in-health-care-6-1611296525640.png" alt=""></p> <p>The Centers for Medicare and Medicaid Services uses predictive analytics to assign risk scores to specific claims and providers, to identify billing patterns, and claim aberrancies difficult to detect by previous methods. Rules-based models flag certain charges automatically. Anomaly models raise suspicion based on factors that seem improbable. Predictive models compare charges against a fraud profile and raise suspicion. Graph models raise suspicion based on the relations of a provider; fraudulent billers are often organized as tight networks.</p> <h3>Predictive Analytics to Improve Outcomes</h3> <p>Initiatives such as meaningful use are accelerating the adoption of Electronic Health Records and the volume and detail of patient information is growing rapidly. Being able to combine and analyze a variety of structured and unstructured data across multiple data sources aids in the accuracy of diagnosing patient conditions, matching treatments with outcomes, and predicting patients at risk for disease or readmission.</p> <p>Predictive modeling over data derived from EHRs is being used for early diagnosis and is reducing mortality rates from problems such as congestive heart failure and <a href="http://www.healthcareitnews.com/news/data-analytics-strategy-slashes-sepsis-death-rates">sepsis.</a> Congestive Heart Failure (CHF) accounts for the most health care spending. The earlier it is diagnosed the better it can be treated, avoiding expensive complications, but early manifestations can be easily missed by physicians. A machine learning example from Georgia Tech demonstrated that machine-learning algorithms could look at many more factors in patients’ charts than doctors and, by adding additional features, there was a substantial increase in the ability of the model to distinguish people who have CHF from people who don’t.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/uncovering-features-7-1611296534274.png" alt=""></p> <p>Predictive modeling and machine learning on large sample sizes, with more patient data, can uncover nuances and patterns that couldn’t be previously uncovered. <a href="http://www.datapine.com/blog/big-data-examples-in-healthcare/">Optum Labs has collected EHRs</a> of over 30 million patients to create a database for predictive analytics tools that will help doctors make Big Data-informed decisions to improve patients’ treatment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/health-care-model-building-and-scoring-8-1611296546479.png" alt=""></p> <h3>Real-time Monitoring of Patients</h3> <p>Healthcare facilities are looking to provide more proactive care to their patients by constantly monitoring patient vital signs. The data from these various monitors can be analyzed in real time and send alerts to care providers so they know instantly about changes in a patient’s condition. Processing real-time events with machine learning algorithms can provide physicians’ insights to make lifesaving decisions and allow for effective interventions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/medical-device-to-data-streaming-9-1611296558176.png" alt=""></p> <h2>Big Data Architecture for Healthcare: What do we need to do? And how do we do this at scale?</h2> <p>We need to collect the data, process the data, store the data, and finally serve the data for analysis, machine learning, and dashboards.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/health-care-data-architecture-10-1611296569137.png" alt=""></p> <h3>Data Ingestion with NFS</h3> <p>The Network File System (NFS) protocol provides remote access to shared disks across networks. An NFS-enabled server can share directories and files with clients, allowing users and programs to access files on remote systems as if they were stored locally.</p> <p>Unlike other Hadoop distributions that only allow cluster data import or import as a batch operation, MapR lets you mount the cluster itself via NFS so that your applications can read and write data directly. The MapR Distributed File and Object Store enables direct file modification and multiple concurrent reads and writes via POSIX semantics. An NFS-mounted cluster allows easy data ingestion of data sources, such as files, images, etc., from other machines leveraging standard Linux commands, utilities, applications, and scripts.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/data-sources-collection-storage-11-1611296581906.png" alt=""></p> <p>From your MapR Cluster, you can move data to and from more expensive storage using NFS. For example, you can move processed hot data to a relational database or Data Warehouse and you can also move off colder data into lower cost Hadoop storage.</p> <h3>Streaming Data Ingestion with the Kafka API</h3> <p>As more and more healthcare solutions require real-time analytics and fast moving data, ingesting data into the system using event streaming will become critical. MapR Event Store is a new distributed messaging system that enables producers and consumers to exchange events in real time via the Apache Kafka 0.9 API. Topics are logical collections of messages that organize events into categories.</p> <p>Topics are partitioned, spreading the load for parallel messaging across multiple servers, which provides for faster throughput and scalability.</p> <p>Messages are not deleted from topics when read and topics can have multiple different consumers. This allows processing of the same messages by different consumers for different purposes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/kafka-api-producers-and-consumers-12-1611296593323.png" alt=""></p> <h3>Batch Processing</h3> <p>Batch processing is for processing bulk loads of data gathered over a period where a fast response time is not critical; for example, EDI claims gathered over a day and submitted together in a file for processing overnight.</p> <p>Apache Hive is an open source Hadoop application for data warehousing. It offers a simple way to apply structure to large amounts of unstructured data, and then perform batch SQL-like queries on that data.</p> <p>Apache Spark is a next generation distributed parallel processing framework that provides a rich set of APIs for machine learning, graph processing, SQL. Spark is much faster than MapReduce for iterative algorithms, because Spark tries to keep things in memory, whereas MapReduce involves more reading and writing from disk.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/batch-processing-collect-to-processing-data-13-1611296604993.png" alt=""></p> <h3>Stream Processing</h3> <p>Spark Streaming brings Spark's APIs to stream processing, letting you write streaming jobs the same way you write batch jobs. Other popular options for Stream processing are Apache Flink and Apache Storm.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/stream-processing-streaming-and-processing-14-1611296616672.png" alt=""></p> <h3>Storing in a NoSQL Database</h3> <p>For Storing lots of Data we need a data store that supports fast writes and scales. MapR Database was designed to scale due to the fact that data that is accessed together is stored together.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/storing-in-nosql-rdbms-to-mapr-db-15-1611296626000.png" alt=""></p> <p>With MapR Database, data is automatically distributed or partitioned across the cluster by key range. Each server is the source for a subset of data. Grouping the data by row key provides for really fast read and writes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/group-by-row-for-fast-read-writes-16-1611296635923.png" alt=""></p> <p><strong>MapR Database has 2 APIs:</strong></p> <ul> <li> <p>JSON API for storing document models.</p> </li> <li> <p>HBase API for wide column data models (typical for time series data).</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/mapr-db-apis-17-1611296648222.png" alt=""></p> <h2>Serving the Data</h2> <p>End applications like dashboards, business intelligence tools and other applications use the processed data. The output can also be stored back in our database for further processing later.</p> <p>Apache Drill enables self-service data exploration on big data with a schema-free SQL query engine. Drill offers the following benefits:</p> <ul> <li> <p>Drill can read from all kinds of data.</p> </li> <li> <p>Drill is optimized for interactive applications, and thus is designed to process petabytes of data and trillions of records in seconds.</p> </li> <li> <p>Drill can be used by data analysts, in conjunction with tools like Tableau, for fast visualizations.</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/big-data-architecture-components-18-1611296658517.png" alt=""></p> <p>All of the components of the architecture we just discussed can run on the same cluster with the MapR Data Platform. There are several advantages of integrating Hadoop, Spark, real-time database capabilities, global event streaming, and scalable enterprise storage:</p> <ul> <li> <p>Maintaining only one cluster means less infrastructure to provision, manage, and monitor for security, reliability, and performance, dramatically lowering both hardware and operational costs.</p> </li> <li> <p>Having producers and consumers on the same cluster means fewer delays related to copying and moving data between clusters, and between applications.</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/mapr-health-care-architecture-19-1611296669267.png" alt=""></p> <h2>Use Case Example Architectures</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/data-lake-architecture-20-1611296679971.png" alt=""></p> <p>The National Institutes for Health built a data lake to combine separate institute data sets, in order to have all the information at one location where it could be shared and manipulated.</p> <p>UnitedHealthcare IT used Hadoop as the basic data framework and built a single platform equipped with the tools needed to analyze information generated by claims, prescriptions, plan participants and contracted care providers, and associated claim review outcomes.</p> <h3>Streaming System of Record for Healthcare</h3> <p>Liaison Technologies provides cloud-based solutions to help organizations integrate, manage and secure data across the enterprise. One vertical solution they provide is for the healthcare and life sciences industry, which comes with two challenges–meeting HIPAA compliance requirements and the proliferation of data formats and representations. With MapR Event Store, the data lineage portion of the compliance challenge is solved because the stream becomes a system of record by being an infinite, immutable log of each data change.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/streaming-system-of-record-health-care-21-1611296690735.png" alt=""></p> <p>To illustrate the latter challenge, a patient record may be consumed in different ways—a document representation, a graph representation, or search—by different users, such as pharmaceutical companies, hospitals, clinics, physicians, etc. By streaming data changes in real-time to the MapR Database HBase, MapR Database JSON document, graph, and search databases, users always have the most up-to-date view of data in the most appropriate format. Further, by implementing this service on the MapR Data Platform, Liaison is able to secure all of the data components together, avoiding data and security silos that alternate solutions require.</p> <h3>Genome Processing</h3> <p>The Novartis team chose Hadoop and Apache Spark to build a workflow system that allows them to integrate, process and analyze diverse data for Next Generation Sequencing (NGS) research while being responsive to advances in the scientific literature.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/genome-processing-22-1611296700172.png" alt=""></p> <p>The ability to capture, share, and store huge amounts of electronic healthcare data and transactions, along with advances in technology allowing the storage and fast processing of big data on commodity hardware, is transforming the healthcare industry by improving outcomes and reducing costs.</p> <p><strong>References and More Information:</strong></p> <p><a href="http://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/the-big-data-revolution-in-us-health-care">The Big Data Revolution in Health Care</a></p> <p><a href="http://www.nationalacademies.org/hmd/Reports/2012/Best-Care-at-Lower-Cost-The-Path-to-Continuously-Learning-Health-Care-in-America.aspx">Better Care at Lower Cost the Path to Continuously Learning Health Care</a></p>https://developer.hpe.com/from-pig-to-spark-an-easy-journey-to-spark-for-apache-pig-developers/https://developer.hpe.com/from-pig-to-spark-an-easy-journey-to-spark-for-apache-pig-developers/Tue, 09 Mar 2021 11:42:12 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Philippe de Cuzey", "publish": "2016-07-13T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>As a data analyst that primarily used Apache Pig in the past, I eventually needed to program more challenging jobs that required the use of Apache Spark, a more advanced and flexible language. At first, Spark may look a bit intimidating, but this blog post will show that the transition to Spark (especially PySpark) is quite easy.</p> <p>However, I'm not advocating that you move from Apache Pig to Spark in all cases. Pig is a wonderful language. It's simple yet efficient when it comes to transforming data through projections and aggregations, and the productivity of Pig can't be beat for standard Map/Reduce jobs.</p> <h2>Apache Pig has great features, but …</h2> <p>I like to think of Pig as a high-level Map/Reduce commands pipeline. As a former SQL programmer, I find it quite intuitive and, in my organization, our Hadoop jobs are still mostly developed in Pig.</p> <p>Pig has a lot of qualities: it is stable, scales very well, and integrates natively with the Hive metastore HCatalog. By describing each step atomically, it minimizes conceptual bugs that you often find in complicated SQL code.</p> <p>But sometimes, Pig has some limitations that makes it a poor programming paradigm to fit your needs. The three main limitations are:</p> <ol> <li> <p>Pig is a pipeline and doesn’t offer loops or code indirections (IF..THEN) which can sometimes be mandatory in your code.</p> </li> <li> <p>As beautifully stated in <a href="https://databricks.com/blog/2014/03/20/apache-spark-a-delight-for-developers.html">an article by Jai Ranganathan and Matei Zaharia</a>:</p> </li> </ol> <blockquote> <p><em>While scripting frameworks like Apache Pig provide many high-level operators as well, Spark allows you to access these operators in the context of a full programming language—thus, you can use control statements, functions, and classes as you would in a typical programming environment.</em></p> </blockquote> <ol start="3"> <li>Finally, a third Pig limitation is related to input data formats: although Pig is good with CSV and HCatalog, it seems a bit less comfortable with reading and processing some other data formats like JSON (through JsonLoader), whereas Spark integrates them natively.</li> </ol> <h2>Give Apache Spark a try</h2> <p>Time to take a dip in Spark! Pig and Spark share a common programming model that makes it easy to move from one to the other. Basically, you work through immutable transformations identified by an alias (Pig) or an RDD variable (Spark). Transformations are usually projections (maps), filters, or aggregations like GroupBy, sorts, etc.</p> <p>This common programming approach means that, for a Pig developer, the learning curve to Spark is fairly quick.</p> <p>PySpark is quite a natural choice for the data analyst who already has some Python basic skills, but the code would be similar in another flavor of Spark, such as Java or Scala.</p> <h2>A complete example</h2> <p>As an illustration, let’s take an example of a Pig script that loads a log file, filters it for a specific day, calculates the number of log entries grouped by item, and adds the item description from another file:</p> <pre><code class="language-python">/* load a log file of user sessions. Filter for a specific date and count entries per item */ f0 = LOAD 'logfile' using PigStorage('\t') AS (log_date:chararray, item_id:chararray, some_stuff:chararray); f1 = FILTER f0 BY log_date == '20160515'; f2 = FOREACH f1 GENERATE item_id; f3 = GROUP f2 BY item_id; f4 = FOREACH f3 GENERATE group AS item_id, COUNT(f2) AS nb_entries; /* add item name */ item1 = LOAD 'item' using PigStorage('\t') AS (item_id:chararray, item_name:chararray); join1 = JOIN f4 BY item_id LEFT, item1 BY item_id; result = FOREACH join1 GENERATE f4::item_id, item_name, nb_entries; STORE result INTO 'result_file' USING PigStorage('\t'); </code></pre> <p>The code is fairly simple, and each step performs one transformation.</p> <p>Now in Spark, we start with raw Spark using low-level RDDs to show similarities with Pig code. In the code, things are detailed one alias at a time, but obviously production code would be more compact.</p> <h2>Raw Spark (using RDD)</h2> <pre><code class="language-python">conf **=** SparkConf() sc **=** SparkContext(conf=conf) f0 **=** sc.textFile('logfile').map(**lambda** x: x.split('\t')) f1 **=** f0.filter(**lambda** x: x[0] == '20160515') f3 **=** f1.groupBy(**lambda** (log_date, item_id, some_stuff): item_id) f4 **=** f3.map (**lambda** (item_id, iterable): (item_id, len(iterable))) # add item name item1 **=** sc.textFile('item').map(**lambda** x: x.split('\t')) # no need to set the key item_id on both parts before performing the join, # It's already on first place on each part. join1 **=** f4.leftOuterJoin(item1) result **=** join1.map(**lambda** (item_id, (nb_entries, item_name)): (item_id, item_name, str(nb_entries))) # creating a line of tab separated fields, and save it in the result file result_to_store **=** result.map (**lambda** record : '\t'.join(record)) result_to_store.saveAsTextFile('result_file') </code></pre> <p>We can see here a similar code outline between Pig and Spark, which makes it easier for a Pig developer to start coding in Spark. One drawback, however, is that for relatively simple operations like this, Pig is still more productive than Spark, even if execution time is better (but not astoundingly better) with Spark.</p> <p>Now that we are getting familiar with this low-level RDD, code could be improved by using DataFrames and SparkSQL. The previous code could be rewritten in a more readable form:</p> <h2>Spark with DataFrames and SparkSQL</h2> <pre><code class="language-python">conf **=** SparkConf() sc **=** SparkContext(conf=conf) sqlContext **=** SQLContext(sc) f0 **=** sc.textFile('logfile').map(**lambda** x: x.split('\t')) fpFields = [ \ StructField('log_date', StringType(), True), \ StructField('item_id', StringType(), True), \ StructField('some_stuff', StringType(), True) \ ] fpSchema **=** StructType(fpFields) df_f0 **=** sqlContext.createDataFrame(f0, fpSchema) df_f0.registerTempTable('log') f1_df **=** sqlContext.sql( "SELECT log.item_id, count(*) AS nb_entries \ FROM log \ WHERE log_date **=** '20160515'\ GROUP BY item_id" ) f1_df.registerTempTable('log_agg') # items dataframe item1 **=** sc.textFile('item').map(**lambda** x: x.split('\t')) itemFields **=** [ \ StructField('item_id', StringType(), True), \ StructField('item_name', StringType(), True) \ ] itemSchema **=** StructType(itemFields) df_item1 **=** sqlContext.createDataFrame(item1, itemSchema) df_item1.registerTempTable('item') result **=** sqlContext.sql( 'SELECT log_agg.item_id, item_name, format_number(nb_entries, 0) \ FROM log_agg \ LEFT OUTER JOIN item ON log_agg.item_id = item.item_id' ) result_to_store **=** result.rdd \ .map (**lambda** record : '\t'.join(record)) result_to_store.saveAsTextFile('result_file') </code></pre> <p>I’m sure there are even more compact and elegant ways to do it in Spark SQL, but this is the outline.</p> <p>Now we have named fields, type safety, and compact SQL code that is more readable by a data analyst. Productivity has increased, and this is a better alternative to Pig.</p> <p>The drawback is that each piece of SQL is now a black box that can be only tested as a whole, which can prove tricky if the result differs from the expected or if execution time is slow. It is then up to the developer to design steps that are still readable and could be executed as individual units of code.</p> <h2>Loading data from Hive metastore HCatalog</h2> <p>If our data would have been stored in Hive HCatalog, all the DataFrame metadata would be inherited from the metastore and the Spark code would have been even simpler:</p> <pre><code class="language-python">conf **=** SparkConf() sc **=** SparkContext(conf=conf) sqlContext = HiveContext(sc) f1_df **=** sqlContext.sql( "SELECT item_id, count(*****) AS nb_entries \ FROM my_db.log \ WHERE log_date **=** '20160515' \ GROUP BY item_id" ) f1_df.registerTempTable('log_agg') result **=** sqlContext.sql( "SELECT log_agg.item_id, item_name, format_number(nb_entries, 0) \ FROM log_agg \ LEFT OUTER JOIN my_db.item item ON log_agg.item_id **=** item.item_id" ) result_to_store = result.rdd \ .map (**lambda** record : '\t'.join(record)) result_to_store.saveAsTextFile(outputFileName) </code></pre> <p>Now, this is a more compact and readable piece of code :)</p> <p>Let's push the advantage a bit further in favor of Spark: user-defined functions.</p> <h2>User-defined functions</h2> <p>As stated previously, in Spark there is obviously no need for UDFs; you would just write the function as a Python method:</p> <p>in Pig:</p> <pre><code class="language-python">/* the **function** below has been written and deployed in a jar file */ DEFINE myFancyUdf com.mydomain.myfunction1; ... log1 = FOREACH log0 GENERATE field1, myFancyUdf (field1l); </code></pre> <p>In Spark:</p> <pre><code class="language-python">**def** myFancyUdf(f1): someStuff **return** result log1 = log0.map (**lambda** field1: (field1, myFancyUdf(field1)) </code></pre> <h2>More advanced topics</h2> <p>In this section, let's take a look at more powerful features of Pig in Spark through two examples:</p> <h2>Map-side joins</h2> <p>One handy feature of Pig is map-side joins, where one of the tables to join is small enough to be sent to each worker to take part in the Map job (not requiring the more expensive Reduce job). This is conveniently performed by using the “replicated” hint on the <code>JOIN</code>.</p> <p>Imagine that in our previous example, the ‘item’ table is small enough to fit in memory. The <code>join1</code> alias becomes:</p> <pre><code class="language-python">join1 = JOIN f4 BY item_id, item1 BY item_id USING ‘replicated; result = FOREACH join1 GENERATE f4::item_id, item_name, nb_entries; </code></pre> <p>In Spark this is performed quite easily with <a href="http://spark.apache.org/docs/latest/rdd-programming-guide.html#broadcast-variables">broadcast variables</a>:</p> <pre><code class="language-python"># broadcast items item_bc **=** sc.broadcast(item.collect()) ''' gets item name from its id ''' **def** getItemName (item_id_to_match): # we know there will be only one result, so we take the first from the list (id, name) **=** filter(**lambda** (id, name): id **==** item_id_to_match, item_bc.value)[0] </code></pre> <p>The item table is broadcasted on each worker node. The <code>getItemName()</code> function then finds in the broadcasted table which record holds a given <code>item_id</code> and returns its name. This function is called in the map side of the Spark job, for each record processed.</p> <p>The complete code now looks like:</p> <pre><code class="language-python"> ''' gets item name from its id ''' **def** getItemName (item_id_to_match): # we know there will be only one result, so we take the first from the (id, name) **=** filter(**lambda** (id, name): id **==** item_id_to_match, item_bc.value)[0] return name f1_df **=** sqlContext.sql( "SELECT item_id, count(*****) AS nb_entries \ FROM my_db.log \ WHERE log_date **=** '20160515' \ GROUP BY item_id" ) item_df **=** sqlContext.sql( "SELECT item_id, item_name \ FROM my_db.item" ) item_bc **=** sc.broadcast(item_df.rdd.collect()) result **=** f1_df.rdd.map (**lambda****=** result.map (**lambda** record : '\t'.join(record)) result_to_store.saveAsTextFile('result_file') </code></pre> <h2>Window function: get n first occurrences of a sorted list of Grouped By items</h2> <p>It is sometimes required to find the top-n first records of a table, grouped by a common feature. From the log files of our example, let's get, for each item, the 10 most recent records (in SQL this would be a windowing function like <code>PARTITION BY</code>).</p> <p>In Pig, this can be accomplished with a piece of code like:</p> <pre><code class="language-python">f0 = LOAD ‘logfile’ using PigStorage('\t') AS (log_date:char array, item_id:chararray, some_stuff:chararray); f1 = GROUP f0 BY item_id; f2 = FOREACH f1 { o = ORDER f0 BY log_date DESC; l = LIMIT o 10; GENERATE FLATTEN(l) AS (log_date, item_id, some_stuff); } </code></pre> <p>In Spark it’s also feasible, either with low-level RDD stuff or with SparkSQL <a href="https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html">Windowing capabilities</a>.</p> <p>Let’s start with the RDD low-level solution:</p> <pre><code class="language-python"># create a tuple with the key for the GroupBy f1 **=** f0.map (**lambda** (log_date, item_id, some_stuff): (item_id, (log_date, some_stuff))) f2 **=** f1.groupByKey() # result of the GroupBy is a tuple (item_id, iterable over grouped items) # we sort the iterable according to log_date and retain only first 10 elements f3 **=** f2.map (**lambda** (item_id, iter1): (item_id, sorted(list(iter1), key=lambda (log_date, item_id, some_stuff):log_date, reverse=True)[:10])) # transform tuples of (item_id, [(log_date, item_id, some_stuff), ...]) into tuples of (log_date, item_id, some_stuff) f4 = f3.flatMapValues(**lambda** x:x) \ .map (**lambda** (item_id, (log_date, some_stuff)):(log_date, item_id, some_stuff) </code></pre> <p>It's not very elegant, but it does the job.</p> <p>Then the SparkSQL solution:</p> <pre><code class="language-python">f1_df **=** sqlContext.sql( 'SELECT \ log_date, \ item_id, \ some_stuff \ FROM ( \ SELECT \ log_date, \ item_id, \ some_stuff, \ dense_rank() OVER (PARTITION BY item_id ORDER BY log_date DESC) as rank \ FROM my_db.log) tmp \ WHERE rank &#x3C;**=** 10') f2 = f1_df.rdd.map (**lambda** row: (row.log_date, row.item_id, row.some_stuff)) </code></pre> <p>Much better!</p> <h2>Conclusion</h2> <p>I have voluntarily excluded from this blog post some interesting topics such as deploying, debugging, execution monitoring, dynamic resource allocation, partition and split size tuning, sampling, etc. The goal of this particular blog post is to show Pig developers how to start coding in Spark; I hope that from this perspective, you find it is helpful.</p>https://developer.hpe.com/spark-streaming-and-twitter-sentiment-analysis/https://developer.hpe.com/spark-streaming-and-twitter-sentiment-analysis/Tue, 09 Mar 2021 09:59:08 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-04-19T07:00:00.000Z", "tags": ["apache-spark","use-case"] </code></pre> <hr> <p>This blog post is the result of my efforts to show to a coworker how to get the insights he needed by using the streaming capabilities and concise API of Apache Spark. In this blog post, you'll learn how to do some simple, yet very interesting, analytics that will help you solve real problems by analyzing specific areas of a social network.</p> <p>Using a subset of a Twitter stream was the perfect choice to use in this demonstration, since it had everything we needed: an endless and continuous data source that was ready to be explored.</p> <h2>Spark Streaming, Minimized</h2> <p>Spark Streaming is very well explained <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html"><em>here</em></a>, so we are going to skip some of the details about the Streaming API and move on to setting up our app.</p> <h2>Setting Up Our App</h2> <p>Let’s see how to prepare our app before doing anything else.</p> <pre><code class="language-scala">val config = new SparkConf().setAppName("twitter-stream-sentiment") val sc = new SparkContext(config) sc.setLogLevel("WARN") val ssc = new StreamingContext(sc, Seconds(5)) System.setProperty("twitter4j.oauth.consumerKey", "consumerKey") System.setProperty("twitter4j.oauth.consumerSecret", "consumerSecret") System.setProperty("twitter4j.oauth.accessToken", accessToken) System.setProperty("twitter4j.oauth.accessTokenSecret", "accessTokenSecret") val stream = TwitterUtils.createStream(ssc, None) </code></pre> <p>Here, we have created the Spark Context <strong><em>sc</em></strong> and set the log level to <em>WARN</em> to eliminate the noisy log Spark generates. We also created a Streaming Context <strong><em>ssc</em></strong> using <strong><em>sc</em></strong>. Then we set up our Twitter credentials (before doing this we needed to follow <a href="https://iag.me/socialmedia/how-to-create-a-twitter-app-in-8-easy-steps/">these steps</a>) that we got from the Twitter website. <em>Now the real fun starts</em>.</p> <h2>What is Trending Right Now on Twitter?</h2> <p>It is easy to find out what is trending on Twitter at any given moment; it is just a matter of counting the appearances of each tag on the stream. Let’s see how Spark allows us to do this operation.</p> <pre><code class="language-scala">val tags = stream.flatMap { status => status.getHashtagEntities.map(_.getText) } tags.countByValue() .foreachRDD { rdd => val now = org.joda.time.DateTime.now() rdd.sortBy(_._2) .map(x => (x, now)) .saveAsTextFile(s"~/twitter/$now") } </code></pre> <p>First, we got the tags from the Tweets, counted how many times it (a tag) appeared, and sorted them by the count. After that, we persisted the result in order to point Splunk (or any other tool for that matter) to it. We could build some interesting dashboards using this information in order to track the most trending hashtags. Based on this information, my coworker could create campaigns and use these popular tags to attract a bigger audience.</p> <h2>Analyzing Tweets</h2> <p>Now we want to add functionality to get an overall opinion of what people think about a set of topics. For the sake of this example, let’s say that we want to know the <strong><em>sentiment</em></strong> of Tweets about <strong>Big Data</strong> and <strong>Food</strong>, two very unrelated topics.</p> <p>There are several APIs for analyzing sentiments from Tweets, but we are going to use an interesting library from <strong>The Stanford Natural Language Processing Group</strong> in order to extract the corresponding <strong><em>sentiments</em></strong>.</p> <p>In our <strong><em>build.sbt</em></strong> file we need to add the corresponding dependencies.</p> <pre><code class="language-scala">libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1" libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1"classifier "models" </code></pre> <p>Now, we need to select only those Tweets we really care about by filtering the <strong><em>stream</em></strong> using certain <em>hashtag (#)</em>. This filtering is quite easy, thanks to a unified Spark API.</p> <p>Let’s see how.</p> <pre><code class="language-scala">val tweets = stream.filter { t => val tags = t.getText.split(" ").filter(_.startsWith("#")).map(_.toLowerCase) tags.contains("#bigdata") &#x26;&#x26; tags.contains("#food") } </code></pre> <p>Here, we get all tags in each Tweet, checking that it has been tagged with**<em>#bigdata</em>** and <strong><em>#food</em></strong>.</p> <p>Once we have our Tweets, extracting the corresponding sentiment is quite easy. Let’s define a function that extracts the sentiment from the Tweet’s content so we can plug it in our pipeline.</p> <pre><code class="language-scala">def detectSentiment(message: String): SENTIMENT_TYPE </code></pre> <p>We are going to use this function, assuming it does what it should, and we will put its implementation at the end, since it's not the focus of this post. In order to get an idea of how it works, let's build some tests around it.</p> <pre><code class="language-scala">it("should detect not understood sentiment") { detectSentiment("")should equal (NOT_UNDERSTOOD) } it("should detect a negative sentiment") { detectSentiment("I am feeling very sad and frustrated.")should equal (NEGATIVE) } it("should detect a neutral sentiment") { detectSentiment("I'm watching a movie")should equal (NEUTRAL) } it("should detect a positive sentiment") { detectSentiment("It was a nice experience.")should equal (POSITIVE) } it("should detect a very positive sentiment") { detectSentiment("It was a very nice experience.")should equal (VERY_POSITIVE) } </code></pre> <p>These tests should be enough to show how <strong><em>detectSentiment</em></strong> works.</p> <p>Let’s see an example.</p> <pre><code class="language-scala">val data = tweets.map { status => val sentiment = SentimentAnalysisUtils.detectSentiment(status.getText) val tags = status.getHashtagEntities.map(_.getText.toLowerCase) (status.getText, sentiment.toString, tags) } </code></pre> <p>Here, <strong><em>data</em></strong> represents a <em>DStream</em> of Tweets we want, the associated sentiment, and the hashtags within the Tweet (here we should find the tags we used to filter).</p> <h2>SQL Interoperability</h2> <p>Now we want to cross reference the sentiment data with an external dataset that we can query using SQL. For my coworker, it makes a lot of sense to be able to <strong><em>join</em></strong> the Twitter stream with his other dataset.</p> <p>Let’s take a look at how we could achieve this.</p> <pre><code class="language-scala">val sqlContext = new SQLContext(sc) import sqlContext.implicits._ data.foreachRDD { rdd => rdd.toDF().registerTempTable("sentiments") } </code></pre> <p>We have transformed our stream into a different representation (a <strong><em>DataFrame</em></strong>), which is also backed by all Spark concepts (resilient, distributed, very fast) and exposed it as a table so my coworker can use his beloved SQL to query different sources.</p> <p>The table <em>sentiment</em> (that we defined from our DataFrame) will be queried as any other table in his system. Another possibility is that we could query other data sources (Cassandra, Xmls, or our own binary formatted files) using Spark SQL and cross them with the stream.</p> <p>You can find out more information about this topic <a href="https://medium.com/@anicolaspp/apache-spark-as-a-distributed-sql-engine-4373e254e0f9#.55n08p6w4"><em>here</em></a> and <a href="https://medium.com/@anicolaspp/extending-our-spark-sql-query-engine-5f4a088de986#.9jm66wp3o"><em>here</em></a>.</p> <p>An example of querying a DataFrame is shown next.</p> <pre><code class="language-scala">sqlContext.sql("select * from sentiments").show() </code></pre> <h2>Windowed Operations</h2> <p>Spark Streaming has the ability to look back in the stream, a functionality most streaming engines lack (if they do have this functionality, it's very hard to implement).</p> <p>In order to implement a windowed operation, you'll need to <em>checkpoint</em> the stream, but this is an easy task. You'll find more information about this <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing"><em>here</em></a>.</p> <p>Here's a small example of this kind of operation:</p> <pre><code class="language-scala">tags    .window(Minutes(1)) . (...) </code></pre> <h2>Conclusion</h2> <p>Even though our examples are quite simple, we were able to solve a real life problem using Spark. We now have the ability to identify trending topics on Twitter, which helps us both target and increase our audience. At the same time, we are able to access different data sets using a single set of tools such as SQL.</p> <p>Very interesting results came back from <strong><em>#bigdata</em></strong> and <strong><em>#food</em></strong> at the same time. Perhaps people Tweet about big data at lunch time—who knows?</p>https://developer.hpe.com/2021-March-04/https://developer.hpe.com/2021-March-04/Thu, 04 Mar 2021 06:00:00 GMThttps://developer.hpe.com/spire-maintainer-agustn-martnez-fay-reveals-his-passion-for-information-/https://developer.hpe.com/spire-maintainer-agustn-martnez-fay-reveals-his-passion-for-information-/Tue, 02 Mar 2021 14:43:49 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/agustin-martinez-fayo-1614696396460.PNG" alt="agustin martinez fayo"></p> <p>As a leading global, <strong>edge-to-cloud platform-as-a-service company</strong>, Hewlett Packard Enterprise (HPE) prides itself in employing team members who share one common purpose: to advance the way people live and work. In this blog series, you’ll get to meet a number of them as I interview some of the <a href="https://www.hpe.com/us/en/open-source.html">open source</a> experts on our team.</p> <p>Agustín Martínez Fayó is a principal engineer for SPIRE, the <a href="https://www.cncf.io/">Cloud Native Computing Foundation's</a> (CNCF) open source project that provides the ability to securely identify software systems in dynamic and heterogeneous environments. He’s a graduate of the Universidad Tecnológica Nacional in Information Systems Engineering and comes to HPE through its recent acquisition of Scytale where he helped to design and implement solutions to connect cloud and container-based services with on-premises services, extending existing identity providers to the cloud. Previously in his career, he worked on the development of database vulnerability assessment software.</p> <h2>Can you tell me a little about the SPIRE project and why it’s special?</h2> <p><a href="https://www.hpe.com/us/en/software/spiffe-spire-open-source.html#:~:text=SPIRE%20is%20an%20open%2Dsource,a%20wide%20variety%20of%20environments.&#x26;text=The%20open%2Dsource%20SPIFFE%20and,between%20multiple%20clouds%20and%20clusters.">SPIRE</a> is an open source project hosted by the Cloud Native Computing Foundation (CNCF). It implements the <a href="https://spiffe.io/">SPIFFE</a> (Secure Production Identity Framework for Everyone) standard to securely identify software systems in dynamic and heterogeneous environments. It does so through the use of platform-agnostic cryptographic identities in an automated way. In essence, SPIRE exposes the SPIFFE Workload API, which can attest running software systems and issue SPIFFE IDs and SVIDs to them. This allows two workloads to establish trust between each other.</p> <p>Both SPIFFE and SPIRE were <a href="https://www.infoq.com/news/2020/06/spire-identity-framework/">recently accepted into the CNCF Incubator</a> to provide a standard and tooling for establishing trust between software services without necessarily using secrets or network-based security controls. The projects enable organizations to deploy consistent, fine-grained cross-service authentication via a “dial-tone” API across heterogeneous environments. Similar to being able to just pick up a phone and connect with anyone because the system just knows how to do it, the Workload API offers a standard way for authentication when connecting with other workloads, no matter where they are or the infrastructure on which they are running. This enables zero trust architectures by delivering continuously attested service identity across cloud, container, and on-premise enterprise IT infrastructures.</p> <h2>How did you get involved with SPIFFE/SPIRE?</h2> <p>I tend to be pretty passionate about information security. As I was off looking for new challenges, the SPIFFE project was just getting started. Being able to contribute to SPIFFE and SPIRE right from the start looked like the perfect opportunity to engage with a community of security experts and be able to contribute to the development of software that would really help improve the security posture across organizations.</p> <h2>What are some of the things you’d like to work on in regards to this project?</h2> <p>Continuing to maintain the SPIRE project is important to me. There are a lot of ongoing efforts and proposals to enhance and extend the SPIRE capabilities that will help make it a suitable solution in more environments and use cases. As a maintainer of the project, I would like to help drive those efforts. It's great to have the opportunity to contribute to the growth and adoption of these projects.</p> <h2>Is there anything else you’d like to share with our readers?</h2> <p>Yes, actually. If you’re interested in learning more about SPIFFE, the standard for service identity, and SPIRE, the reference implementation for SPIFFE, you might want to check out the book <a href="https://spiffe.io/book/"><em>Solving the Bottom Turtle</em></a>. The book distills the experience of renowned security experts to provide a deep understanding of the identity problem and how to solve it. I think you’ll find it very informative and interesting.</p> <p>To learn more about the open source projects that HPE is involved with, please visit our <a href="https://www.hpe.com/us/en/open-source.html">website</a>. Interested in exploring what HPE offers for developers and data scientists? Check out our <a href="https://developer.hpe.com/">HPE DEV site</a> for a ton of articles, workshops, tutorials, and other resources.</p>https://developer.hpe.com/spark-streaming-with-hbase/https://developer.hpe.com/spark-streaming-with-hbase/Fri, 19 Feb 2021 07:00:12 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2015-09-04T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>This post will help you get started using <a target='\_blank' href='http://spark.apache.org/streaming/'>Apache Spark Streaming</a> with HBase on MapR. Spark Streaming is an extension of the core Spark API that enables continuous data stream processing.</p> <h2>What is Spark Streaming?</h2> <p>First of all, what is streaming? A data stream is an unbounded sequence of data arriving continuously. Streaming divides continuously flowing input data into discrete units for processing. Stream processing is low latency processing and the analysis of streaming data. Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data. Spark Streaming is for use cases that require a significant amount of data to be quickly processed as soon as it arrives. Example real-time use cases are:</p> <ul> <li>Website monitoring</li> <li>Network monitoring</li> <li>Fraud detection</li> <li>Web clicks</li> <li>Advertising</li> <li>IoT sensors</li> </ul> <p>Spark Streaming supports data sources such as HDFS directories, TCP sockets, Kafka, Flume, Twitter, etc. Data Streams can be processed with Spark’s core APIS, DataFrames SQL, or machine learning APIs, and can be persisted to a filesystem, HDFS, databases, or any data source offering a Hadoop OutputFormat.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream1-blog-1613718331029.png" alt=""></p> <h2>How Spark Streaming Works</h2> <p>Streaming data is continuous and needs to be batched to process. Spark Streaming divides the data stream into batches of <em>x</em> seconds called Dstreams, which internally is a sequence of RDDs. Your Spark Application processes the RDDs using Spark APIs, and the processed results of the RDD operations are returned in batches.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream2-blog-1613718342309.png" alt=""></p> <h2>The Streaming Application Example Architecture</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream3-blog-1613718349956.png" alt=""></p> <p>The Spark Streaming example code does the following:</p> <ul> <li>Reads streaming data</li> <li>Processes the streaming data</li> <li>Writes the processed data to an HBase Table</li> </ul> <p>Other Spark example code does the following:</p> <ul> <li>Reads HBase Table data written by the streaming code</li> <li>Calculates daily summary statistics</li> <li>Writes summary statistics to the HBase table Column Family stats</li> </ul> <h2>Example data set</h2> <p>The Oil Pump Sensor data comes in as comma separated value (csv) files dropped in a directory. Spark Streaming will monitor the directory and process any files created in that directory. (As stated before, Spark Streaming supports different streaming data sources. For simplicity, this example will use files.) Below is an example of the csv file with some sample data:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream4-blog-1613718358120.png" alt=""></p> <p>We use a Scala case class to define the Sensor Schema corresponding to the sensor data csv files, and a parseSensor function to parse the comma separated values into the sensor case class.</p> <pre><code class="language-scala">// schema for sensor data case class Sensor(resid: String, date: String, time: String, hz: Double, disp: Double, flo: Double, sedPPM: Double, psi: Double, chlPPM: Double) object Sensor { // function to parse line of csv data into Sensor class def parseSensor(str: String): Sensor = { val p = str.split(",") Sensor(p(0), p(1), p(2), p(3).toDouble, p(4).toDouble, p(5).toDouble, p(6).toDouble, p(7).toDouble, p(8).toDouble) } … } </code></pre> <h2>HBase Table Schema</h2> <p>The HBase Table Schema for the streaming data is as follows:</p> <ul> <li>Composite row key of the pump name date and time stamp</li> </ul> <p>The Schema for the daily statistics summary rollups is as follows:</p> <ul> <li>Composite row key of the pump name and date</li> <li>Column Family stats</li> <li>Columns for min, max, avg.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream5-blog-1613718366157.png" alt=""></p> <p>The function below converts a sensor object into an HBase Put object, which is used to insert a row into HBase.</p> <pre><code class="language-scala">val cfDataBytes = Bytes.toBytes("data") object Sensor { . . . // Convert a row of sensor object data to an HBase put object def convertToPut(sensor: Sensor): (ImmutableBytesWritable, Put) = { val dateTime = sensor.date + " " + sensor.time // create a composite row key: sensorid_date time val rowkey = sensor.resid + "_" + dateTime val put = new Put(Bytes.toBytes(rowkey)) // add to column family data, column data values to put object put.add(cfDataBytes, Bytes.toBytes("hz"), Bytes.toBytes(sensor.hz)) put.add(cfDataBytes, Bytes.toBytes("disp"), Bytes.toBytes(sensor.disp)) put.add(cfDataBytes, Bytes.toBytes("flo"), Bytes.toBytes(sensor.flo)) put.add(cfDataBytes, Bytes.toBytes("sedPPM"), Bytes.toBytes(sensor.sedPPM)) put.add(cfDataBytes, Bytes.toBytes("psi"), Bytes.toBytes(sensor.psi)) put.add(cfDataBytes, Bytes.toBytes("chlPPM"), Bytes.toBytes(sensor.chlPPM)) return (new ImmutableBytesWritable(Bytes.toBytes(rowkey)), put) } } </code></pre> <h2>Configuration for Writing to an HBase Table</h2> <p>You can use the <a target='\_blank' href='https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.html'>TableOutputFormat</a> class with Spark to write to an HBase table, similar to how you would write to an HBase table from MapReduce. Below, we set up the configuration for writing to HBase using the <code>TableOutputFormat</code> class.</p> <pre><code class="language-scala"> val tableName = "sensor" // set up Hadoop HBase configuration using TableOutputFormat val conf = HBaseConfiguration.create() conf.set(**TableOutputFormat.OUTPUT_TABLE**, tableName) val jobConfig: jobConfig = new JobConf(conf, this.getClass) jobConfig.setOutputFormat(classOf[**TableOutputFormat**]) jobConfig.set(**TableOutputFormat**.OUTPUT_TABLE, tableName) </code></pre> <h2>The Spark Streaming Example Code</h2> <p>These are the basic steps for Spark Streaming code:</p> <ol> <li>Initialize a Spark StreamingContext object.</li> <li>Apply transformations and output operations to DStreams.</li> <li>Start receiving data and processing it using <code>streamingContext.start()</code>.</li> <li>Wait for the processing to be stopped using <code>streamingContext.awaitTermination()</code>.</li> </ol> <p>We will go through each of these steps with the example application code.</p> <h2>Initializing the StreamingContext</h2> <p>First we create a <a target='\_blank' href='http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.StreamingContext'>StreamingContext</a>, the main entry point for streaming functionality, with a 2 second <a target='\_blank' href='http://spark.apache.org/docs/latest/streaming-programming-guide.html#setting-the-right-batch-interval'>batch interval</a>. (In the code boxes, comments are in grey)</p> <pre><code class="language-scala">val sparkConf = new SparkConf().setAppName("HBaseStream") // create a StreamingContext, the main entry point for all streaming functionality val ssc = new StreamingContext(sparkConf, Seconds(2)) </code></pre> <p>Next, we use the <code>StreamingContext textFileStream(directory)</code> method to create an input stream that monitors a Hadoop-compatible file system for new files and processes any files created in that directory.</p> <pre><code class="language-scala">// create a DStream that represents streaming data from a directory source val linesDStream = ssc.textFileStream("/user/user01/stream") </code></pre> <p>The linesDStream represents the stream of data, each record is a line of text. Internally a DStream is a sequence of RDDs, one RDD per batch interval.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream6-blog-1613718375057.png" alt=""></p> <h2>Apply Transformations and Output Operations to DStreams</h2> <p>Next, we parse the lines of data into Sensor objects, with the map operation on the linesDStream.</p> <pre><code class="language-scala">// parse each line of data in linesDStream into sensor objects val sensorDStream = linesDStream.map(Sensor.parseSensor) </code></pre> <p>The map operation applies the <code>Sensor.parseSensor</code> function on the RDDs in the linesDStream, resulting in RDDs of Sensor objects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream7-blog-1613718382720.png" alt=""></p> <p>Next, we use the DStream <a target='\_blank' href='https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/streaming/dstream/DStream.html'>foreachRDD</a> method to apply processing to each RDD in this DStream. We filter the sensor objects for low psi to create alerts, then we write the sensor and alert data to HBase by converting them to Put objects, and using the PairRDDFunctions <a target='\_blank' href='https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/rdd/PairRDDFunctions.html#saveAsHadoopDataset(org.apache.hadoop.mapred.JobConf)'>saveAsHadoopDataset</a> method, which outputs the RDD to any Hadoop-supported storage system using a Hadoop Configuration object for that storage system (see Hadoop Configuration for HBase above).</p> <pre><code class="language-scala">// for each RDD. performs function on each RDD in DStream sensorRDD.foreachRDD { rdd => // filter sensor data for low psi val alertRDD = rdd.filter(sensor => sensor.psi &#x3C; 5.0) // convert sensor data to put object and write to HBase Table CF data rdd.map(Sensor.convertToPut).saveAsHadoopDataset(jobConfig) // convert alert to put object write to HBase Table CF alerts rdd.map(Sensor.convertToPutAlert).saveAsHadoopDataset(jobConfig) } </code></pre> <p>The sensorRDD objects are converted to put objects, and then written to HBase.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream8-blog-1613718390677.png" alt=""></p> <h2>Start Receiving Data</h2> <p>To start receiving data, we must explicitly call <code>start()</code> on the StreamingContext, then call <code>awaitTermination</code> to wait for the streaming computation to finish.</p> <pre><code class="language-scala"> // Start the computation ssc.start() // Wait for the computation to terminate ssc.awaitTermination() </code></pre> <h2>Spark Reading from and Writing to HBase</h2> <p>Now we want to read the HBase sensor table data, calculate daily summary statistics and write these statistics to the stats column family.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream9-blog-1613718398619.png" alt=""></p> <p>The code below reads the HBase table sensor table psi column data, calculates statistics on this data using <a target='\_blank' href='https://spark.apache.org/docs/0.6.2/api/core/spark/util/StatCounter.html'>StatCounter</a>, and then writes the statistics to the sensor stats column family.</p> <pre><code class="language-scala"> // configure HBase for reading val conf = HBaseConfiguration.create() conf.set(TableInputFormat.INPUT_TABLE, HBaseSensorStream.tableName) // scan data column family psi column conf.set(TableInputFormat.SCAN_COLUMNS, "data:psi") // Load an RDD of (row key, row Result) tuples from the table val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result]) // transform (row key, row Result) tuples into an RDD of Results val resultRDD = hBaseRDD.map(tuple => tuple._2) // transform into an RDD of (RowKey, ColumnValue)s , with Time removed from row key val keyValueRDD = resultRDD. map(result => (Bytes.toString(result.getRow()). split(" ")(0), Bytes.toDouble(result.value))) // group by rowkey , get statistics for column value val keyStatsRDD = keyValueRDD. groupByKey(). mapValues(list => StatCounter(list)) // convert rowkey, stats to put and write to hbase table stats column family keyStatsRDD.map { case (k, v) => convertToPut(k, v) }.saveAsHadoopDataset(jobConfig) </code></pre> <p>The diagram below shows that the output from <code>newAPIHadoopRDD</code> is an RDD of row key, result pairs. The <code>PairRDDFunctions saveAsHadoopDataset</code> saves the Put objects to HBase.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream10-blog-1613718405899.png" alt=""></p> <h2>Software</h2> <ul> <li>You can download the <strong>code and data</strong> to run these examples from here: <ul> <li>Code: <a target='\_blank' href='https://github.com/caroljmcdonald/SparkStreamingHBaseExample'><a href="https://github.com/caroljmcdonald/SparkStreamingHBaseExample">https://github.com/caroljmcdonald/SparkStreamingHBaseExample</a></a></li> </ul> </li> </ul> <h2>Running the Application</h2> <p>You can run the code as a standalone application.</p> <p>Here are the steps summarized:</p> <ol> <li> <p>Log into MapR using userid user01, password mapr.</p> </li> <li> <p>Build the application using maven.</p> </li> <li> <p>Copy the jar file and data file to your home directory /user/user01 using scp.</p> </li> <li> <p>Run the streaming app:</p> <pre><code class="language-scala"> /opt/mapr/spark/spark-1.3.1/bin/spark-submit --driver-class-path `hbase classpath` --class examples.HBaseSensorStream sparkstreamhbaseapp-1.0.jar </code></pre> </li> <li> <p>Copy the streaming data file to the stream directory:<br> <code>cp sensordata.csv /user/user01/stream/</code></p> </li> <li> <p>Read data and calculate stats for one column:</p> <pre><code class="language-scala"> /opt/mapr/spark/spark-1.3.1/bin/spark-submit --driver-class-path `hbase classpath` --class examples.HBaseReadWrite sparkstreamhbaseapp-1.0.jar </code></pre> </li> <li> <p>Calculate stats for whole row:</p> <pre><code class="language-scala"> /opt/mapr/spark/spark-1.3.1/bin/spark-submit --driver-class-path `hbase classpath` --class examples.HBaseReadRowWriteStats sparkstreamhbaseapp-1.0.jar </code></pre> </li> </ol> <h2>Summary</h2> <p>This concludes the tutorial on Spark Streaming with HBase. You can find more information here:</p> <h2>References and More Information:</h2> <ul> <li><a target='\_blank' href='http://spark.apache.org/docs/latest/streaming-programming-guide.html'>Apache Spark Streaming Programming guide</a></li> <li><a target='\_blank' href='http://shop.oreilly.com/product/0636920028512.do'>Learning Spark O'Reilly Book</a></li> <li><a target='\_blank' href='https://databricks.com/blog/2015/07/30/diving-into-spark-streamings-execution-model.html'>Databricks Spark Streaming</a></li> </ul>https://developer.hpe.com/getting-started-with-mapr-event-store/https://developer.hpe.com/getting-started-with-mapr-event-store/Fri, 19 Feb 2021 06:51:45 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Tugdual Grall", "publish": "2016-03-10T08:00:00.000Z", "tags": "streaming" </code></pre> <hr> <p>MapR Event Store is a distributed messaging system for streaming event data at scale. It is integrated into the MapR converged platform. MapR Event Store uses the Apache Kafka API, so if you’re already familiar with Kafka, you’ll find it particularly easy to get started with MapR Event Store.</p> <p>Although MapR Event Store generally uses the <a target='\_blank' href='http://kafka.apache.org/documentation.html#introduction'>Apache Kafka programming model</a>, there are a few key differences. For instance, there is a new kind of object in the MapR file-system called, appropriately enough, a stream. Each stream can handle a huge number of topics, and you can have many streams in a single cluster. Policies such as time-to-live or ACEs (Access Control Expressions) can be set at the stream level for convenient management of many topics together. If you already have Kafka applications, it’s easy to migrate them over to MapR Event Store. In this blog post we describe how to run a simple application we originally wrote for Kafka using MapR Event Store instead.  </p> <h2>Sample Programs</h2> <p>As mentioned above, MapR Event Store uses <a target='\_blank' href='http://kafka.apache.org/documentation.html#api'>Kafka API 0.9.0</a>, which means it is possible to reuse the same application with minor changes. Before diving into a concrete example, let’s take a look at what has to be changed:</p> <ol> <li> <p>The topic names change from "<code>topic-name</code>" to "<code>/stream-name:topic-name</code>", as MapR organizes the topics in streams for management reasons (security, TTL, etc.).</p> </li> <li> <p>The producer and consumer configuration parameters that are not used by MapR Event Store are automatically ignored, so no change here.</p> </li> <li> <p>The producer and consumer applications are using jars from MapR rather than the Apache Kafka jars.</p> </li> </ol> <p>You can find a complete application on the <a target='\_blank' href='https://github.com/mapr-demos/mapr-streams-sample-programs'>Sample Programs for MapR Event Store</a> page. It’s a simple copy that includes minor changes of the <a target='\_blank' href='https://github.com/mapr-demos/kafka-sample-programs'>Sample Programs for Kafka 0.9 API</a> project.</p> <h2>Prerequisites</h2> <p>You will need basic Java programming skills, as well as access to:</p> <ul> <li>A running MapR Cluster</li> <li><a target='\_blank' href='https://maven.apache.org/'>Apache Maven 3.0</a> or later</li> <li>Git to clone the <a target='\_blank' href='https://github.com/mapr-demos/mapr-streams-sample-programs'><a href="https://github.com/mapr-demos/mapr-streams-sample-programs">https://github.com/mapr-demos/mapr-streams-sample-programs</a></a> repository</li> </ul> <h2>Running Your First MapR Event Store Application</h2> <h3>Step 1: Create the stream</h3> <p>A <em>stream</em> is a collection of topics that you can manage together by:</p> <ol> <li> <p>Setting security policies that apply to all topics in that stream</p> </li> <li> <p>Setting a default number of partitions for each new topic that is created in the stream</p> </li> <li> <p>Setting a time-to-live for messages in every topic in the stream.</p> </li> </ol> <p>Run the following command, as <code>mapr</code> user, on your MapR cluster:</p> <pre><code class="language-bash">$ maprcli stream create -path /sample-stream </code></pre> <p>By default, the produce and consume topic permissions are defaulted to the creator of the streams — the Unix user you are using to run the maprcli command. It is possible to configure the permission by editing the streams. For example, to make all of the topics available to anybody (public permission), you can run the following command:</p> <pre><code class="language-bash">$ maprcli stream edit -path /sample-stream -produceperm p -consumeperm p -topicperm p </code></pre> <h3>Step 2: Create the topics</h3> <p>We need two topics for the example program, which we can be created using <code>maprcli</code>:</p> <pre><code class="language-bash">$ maprcli stream topic create -path /sample-stream  -topic fast-messages $ maprcli stream topic create -path /sample-stream  -topic summary-markers </code></pre> <p>These topics can be listed using the following command:</p> <pre><code class="language-bash">$ maprcli stream topic list -path /sample-stream topic            partitions  logicalsize  consumers  maxlag  physicalsize fast-messages    1           0            0          0       0 summary-markers  1           0            0          0       0 </code></pre> <p>Note that the program will automatically create the topic if it does not already exist. For your applications, you should decide whether it is better to allow programs to automatically create topics simply by virtue of having mentioned them or whether it is better to strictly control which topics exist.</p> <h3>Step 3: Compile and package the example programs</h3> <p>Go back to the directory where you have the example programs and build an example program.</p> <pre><code class="language-bash">$ cd .. $ mvn package ... </code></pre> <p>The project creates a jar with all the external dependencies ( <code>./target/mapr-streams-examples-1.0-SNAPSHOT-jar-with-dependencies.jar</code>).</p> <p>Note that you can build the project with the Apache Kafka dependencies as long as you do not package them into your application when you run and deploy it. This example has a dependency on the MapR Event Store client instead, which can be found in the <code>mapr.com</code> maven repository.</p> <pre><code class="language-scala">   &#x3C;repositories>        &#x3C;repository>            &#x3C;id>mapr-maven&#x3C;/id>            &#x3C;url>http://repository.mapr.com/maven&#x3C;/url>            &#x3C;releases>&#x3C;enabled>true&#x3C;/enabled>&#x3C;/releases>            &#x3C;snapshots>&#x3C;enabled>false&#x3C;/enabled>&#x3C;/snapshots>        &#x3C;/repository>    &#x3C;/repositories>    ...        &#x3C;dependency>            &#x3C;groupId>org.apache.kafka&#x3C;/groupId>            &#x3C;artifactId>kafka-clients&#x3C;/artifactId>            &#x3C;version>0.9.0.0-mapr-1602&#x3C;/version>            &#x3C;scope>provided&#x3C;/scope>        &#x3C;/dependency>   ... </code></pre> <h3>Step 4: Run the example producer</h3> <p>You can install the MapR Client and run the application locally or copy the jar file onto your cluster (any node). If you are installing the MapR Client, be sure you also install the MapR Kafka package using the following command on CentOS/RHEL:</p> <pre><code class="language-bash">`yum install mapr-kafka` </code></pre> <pre><code class="language-bash">$ scp ./target/mapr-streams-examples-1.0-SNAPSHOT-jar-with-dependencies.jar mapr@&#x3C;YOUR_MAPR_CLUSTER>:/home/mapr </code></pre> <p>The producer will send a large number of messages to <code>/sample-stream:fast-messages</code> along with occasional messages to <code>/sample-stream:summary-markers</code>. Since there isn't any consumer running yet, nobody will receive the messages. If you compare this with the Kafka example used to build this application, the topic name is the only change to the code. Any MapR Event Store application will need the MapR Client libraries. One way to make these libraries available is to add them to the application classpath using the <code>/opt/mapr/bin/mapr classpath</code> command. For example:</p> <pre><code class="language-bash">$ java -cp $(mapr classpath):./mapr-streams-examples-1.0-SNAPSHOT-jar-with-dependencies.jar com.mapr.examples.Run producer Sent msg number 0 Sent msg number 1000 ... Sent msg number 998000 Sent msg number 999000 </code></pre> <p>The only important difference here between an Apache Kafka application and MapR Event Store application is that the client libraries are different. This causes the MapR Producer to connect to the MapR cluster to post the messages, and not to a Kafka broker.</p> <h3>Step 5: Start the example consumer</h3> <p>In another window, you can run the consumer using the following command:</p> <pre><code class="language-bash">$ java -cp $(mapr classpath):./mapr-streams-examples-1.0-SNAPSHOT-jar-with-dependencies.jar com.mapr.examples.Run consumer 1 messages received in period, latency(min, max, avg, 99%) = 20352, 20479, 20416.0, 20479 (ms) 1 messages received overall, latency(min, max, avg, 99%) = 20352, 20479, 20416.0, 20479 (ms) 1000 messages received in period, latency(min, max, avg, 99%) = 19840, 20095, 19968.3, 20095 (ms) 1001 messages received overall, latency(min, max, avg, 99%) = 19840, 20479, 19968.7, 20095 (ms) ... 1000 messages received in period, latency(min, max, avg, 99%) = 12032, 12159, 12119.4, 12159 (ms) &#x3C;998001 1000="" 12095="" 19583="" 999001="" messages="" received="" overall,="" latency(min,="" max,="" avg,="" 99%)="12032," 20479,="" 15073.9,="" (ms)="" in="" period,="" 12095,="" 12064.0,="" 15070.9,="" (ms)&#x3C;="" pre=""> </code></pre> <p>Note that there is a latency listed in the summaries for the message batches. This is because the consumer wasn't running when the messages were sent to MapR Event Store, and thus it is only getting them much later, long after they were sent.</p> <h2>Monitoring your topics</h2> <p>At any time you can, use the maprcli tool to get some information about the topic. For example:</p> <pre><code class="language-bash">$ maprcli stream topic info -path /sample-stream -topic fast-messages -json </code></pre> <p>The <code>-json</code> option is used to get the topic information as a JSON document.</p> <h2>Cleaning up</h2> <p>When you are done playing, you can delete the stream and all associated topics using the following command:</p> <pre><code class="language-bash">$ maprcli stream delete -path /sample-stream </code></pre> <h2>Conclusion</h2> <p>Using this example built from an Apache Kafka application, you have learned how to write, deploy, and run your first MapR Event Store application. As you can see, the application code is really similar, and only a few changes need to be made (such as changing the topic names). This means it is possible to easily deploy your Kafka applications on MapR and reap the benefits of all the features of MapR Event Store, such as advanced security, geographically distributed deployment, very large number of topics, and much more. This also means that you can immediately use all of your Apache Kafka skills on a MapR deployment.</p>https://developer.hpe.com/real-time-streaming-data-pipelines-with-apache-apis-kafka-spark-streamin/https://developer.hpe.com/real-time-streaming-data-pipelines-with-apache-apis-kafka-spark-streamin/Fri, 19 Feb 2021 06:29:39 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2016-04-22T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>Many of the systems we want to monitor happen as a stream of events. Examples include event data from web or mobile applications, sensors, or medical devices.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag01-1613716304443.png" alt=""></p> <p>Real-time analysis examples include:</p> <ul> <li>Website monitoring</li> <li>Network monitoring</li> <li>Fraud detection</li> <li>Web clicks</li> <li>Advertising</li> <li>IoT sensors</li> </ul> <p>Batch processing can give great insights into things that happened in the past, but it lacks the ability to answer the question of "what is happening right now?”</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag02-1613716314047.png" alt=""></p> <p>It is becoming important to process events as they arrive for real-time insights, but high performance at scale is necessary to do this. In this blog post, I'll show you how to integrate Apache Spark Streaming, MapR Database, and MapR Event Store for fast, event-driven applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag03-1613716321972.png" alt=""></p> <p><strong>Example Use Case</strong></p> <p>Let's go over an example which generates lots of data and needs real-time preventive alerts. Remember what happened back in 2010 with BP in the Gulf of Mexico oil spill?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag04-1613716329966.png" alt=""></p> <p>The example use case we will look at here is an application that monitors oil wells. Sensors in oil rigs generate streaming data, which is processed by Spark and stored in HBase for use by various analytical and reporting tools. We want to store every single event in HBase as it streams in. We also want to filter for, and store, alarms. Daily Spark processing will store aggregated summary statistics.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag05-1613716341142.png" alt=""></p> <p><strong>What do we need to do? And how do we do this with high performance at scale?</strong></p> <p>We need to collect the data, process the data, store the data, and, finally, serve the data for analysis, machine learning, and dashboards.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag06-1613716353632.png" alt=""></p> <p><strong>Streaming Data Ingestion</strong></p> <p>Spark Streaming supports data sources such as HDFS directories, TCP sockets, Kafka, Flume, Twitter, etc. In our example, we will use MapR Event Store for Apache Kafka, a new distributed messaging system for streaming event data at scale. MapR Event Store enables producers and consumers to exchange events in real time via the Apache Kafka 0.9 API. MapR Event Store integrates with Spark Streaming via the Kafka direct approach.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag07-1613716362975.png" alt=""></p> <p>MapR Event Store (or Kafka) topics are logical collections of messages. Topics organize events into categories. Topics decouple producers, which are the sources of data, from consumers, which are the applications that process, analyze, and share data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag08-1613716380545.png" alt=""></p> <p>Topics are partitioned for throughput and scalability. Partitions make topics scalable by spreading the load for a topic across multiple servers. Producers are load balanced between partitions and consumers can be grouped to read in parallel from multiple partitions within a topic for faster performance. Partitioned parallel messaging is a key to high performance at scale.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag09-1613716516266.png" alt=""></p> <p>Another key to high performance at scale is minimizing time spent on disk reads and writes. Compared with older messaging systems, Kafka and MapR Event Store eliminated the need to track message acknowledgements on a per-message, per-listener basis. Messages are persisted sequentially as produced, and read sequentially when consumed. These design decisions mean that non sequential reading or writing is rare, and allow messages to be handled at very high speeds. MapR Event Store performance scales linearly as servers are added within a cluster, with each server handling more than 1 million messages per second.</p> <p><strong>Real-time Data Processing Using Spark Streaming</strong></p> <p>Spark Streaming brings Spark's APIs to stream processing, letting you use the same APIs for streaming and batch processing. Data streams can be processed with Spark’s core APIs, DataFrames, GraphX, or machine learning APIs, and can be persisted to a file system, HDFS, MapR XD, MapR Database, HBase, or any data source offering a Hadoop OutputFormat or Spark connector.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag10-1613716527479.png" alt=""></p> <p>Spark Streaming divides the data stream into batches of x seconds called Dstreams, which internally is a sequence of RDDs, one for each batch interval. Each RDD contains the records received during the batch interval.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag11-1613716535163.png" alt=""></p> <p>Resilient distributed datasets, or RDDs, are the primary abstraction in Spark. An RDD is a distributed collection of elements, like a Java Collection, except that it’s spread out across multiple nodes in the cluster. The data contained in RDDs is partitioned and operations are performed in parallel on the data cached in memory. Spark caches RDDs in memory, whereas MapReduce involves more reading and writing from disk. Here again, the key to high performance at scale is partitioning and minimizing disk I/O.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag12-1613716543034.png" alt=""></p> <p>There are two types of operations on DStreams: transformations and output operations.</p> <p>Your Spark application processes the DStream RDDs using Spark transformations like map, reduce, and join, which create new RDDs. Any operation applied on a DStream translates to operations on the underlying RDDs, which, in turn, applies the transformation to the elements of the RDD.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag13-1613716550447.png" alt=""></p> <p>Output operations write data to an external system, producing output in batches.</p> <p>Examples of output operations are saveAsHadoopFiles, which saves to a Hadoop-compatible file system, and saveAsHadoopDataset, which saves to any Hadoop-supported storage system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag14-1613716557963.png" alt=""></p> <p><strong>Storing Streaming Data Using HBase</strong></p> <p>For storing lots of streaming data, we need a data store that supports fast writes and scales.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag15-1613716565379.png" alt=""></p> <p>With MapR Database (HBase API), a table is automatically partitioned across a cluster by key range, and each server is the source for a subset of a table. Grouping the data by key range provides for really fast read and writes by row key.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag16-1613716573353.png" alt=""></p> <p>Also with MapR Database, each partitioned subset or region of a table has a write and read cache. Writes are sorted in cache, and appended to a Write-Ahead Log (WAL HBase); writes and reads to disk are always sequential; recently read or written data and cached column families are available in memory and all of this provides for really fast read and writes.</p> <p>With a relational database and a normalized schema, query joins cause bottlenecks with lots of data. MapR Database and a de-normalized schema scales because data that is read together is stored together.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag17-1613716581199.png" alt=""></p> <p>So how do we collect, process, and store real-time events with high performance at scale? The key is partitioning, caching, and minimizing time spent on disk reads and writes for:</p> <ul> <li>Messaging with MapR Event Store</li> <li>Processing with Spark Streaming</li> <li>Storage with MapR Database</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag18-1613716588993.png" alt=""></p> <p><strong>Serving the Data</strong></p> <p>End applications like dashboards, business intelligence tools, and other applications use the processed event data. The processing output can also be stored back in MapR Database, in another Column Family or Table, for further processing later.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag19-1613716597526.png" alt=""></p> <p><strong>Example Use Case Code</strong></p> <p>Now we will step through the code for a MapR Event Store producer sending messages, and for Spark Streaming processing the events and storing data in MapR Database.</p> <p><strong>MapR Event Store Producer Code</strong></p> <p>The steps for a producer sending messages are:</p> <ol> <li> <p>Set producer properties</p> <ul> <li>The first step is to set the KafkaProducer configuration properties, which will be used later to instantiate a KafkaProducer for publishing messages to topics.</li> </ul> </li> <li> <p>Create a KafkaProducer</p> <ul> <li>You instantiate a KafkaProducer by providing the set of key-value pair configuration properties, which you set up in the first step. Note that the KafkaProducer&#x3C;K,V> is a Java generic class. You need to specify the type parameters as the type of the key-value of the messages that the producer will send.</li> </ul> </li> <li> <p>Build the ProducerRecord message</p> <ul> <li>The ProducerRecord is a key-value pair to be sent to Kafka. It consists of a topic name to which the record is being sent, an optional partition number, and an optional key and a message value. The ProducerRecord is also a Java generic class, whose type parameters should match the serialization properties set before. In this example, we instantiate the ProducerRecord with a topic name and message text as the value, which will create a record with no key.</li> </ul> </li> <li> <p>Send the message</p> <ul> <li>Call the send method on the KafkaProducer passing the ProducerRecord, which will asynchronously send a record to the specified topic. This method returns a Java Future object, which will eventually contain the response information. The asynchronous send() method adds the record to a buffer of pending records to send, and immediately returns. This allows sending records in parallel without waiting for the responses and allows the records to be batched for efficiency.</li> </ul> </li> <li> <p>Finally, call the close method on the producer to release resources. This method blocks until all requests are complete.</p> </li> </ol> <p>The code is shown below:</p> <pre><code class="language-java">import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; String topic="/streams/pump:warning"; public static KafkaProducer producer; Properties properties = new Properties(); Properties.put("Value.serializer","org.apache.kafka.common.serialization.StringSerializer"); producer= new KafkaProducer&#x3C;String,String>(properties); String txt = "sample msg"; ProducerRecord&#x3C;String, String> rec = new ProducerRecord&#x3C;String, String>(topic, txt); producer.send(rec); producer.close(); </code></pre> <p><strong>Spark Streaming Code</strong></p> <p>These are the basic steps for Spark Streaming code:</p> <ol> <li> <p>Initialize a Spark StreamingContext object. Using this context, create a DStream.</p> <ul> <li> <p>We use the KafkaUtils createDirectStream method to create an input stream from a Kafka or MapR Event Store topic. This creates a DStream that represents the stream of incoming data, where each record is a line of text.</p> <pre><code class="language-java">val ssc = new StreamingContext(sparkConf, Seconds(5)) cal dStream = KafkaUtils.createDirectStream[String, String](ssc, kafkaParams, topicsSet) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag20-1613716623810.png" alt=""></p> </li> </ul> </li> <li> <p>Apply transformations (which create new DStreams)</p> <ul> <li> <p>We parse the message values into Sensor objects, with the map operation on the dStream. The map operation applies the Sensor.parseSensor function on the RDDs in the dStream, resulting in RDDs of Sensor objects.</p> <p>Any operation applied on a DStream translates to operations on the underlying RDDs. The map operation is applied on each RDD in the dStream to generate the sensorDStream RDDs.</p> <pre><code class="language-java">val sensorDStream = dStream.map(_._2).map(parseSensor) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag21-1613716640674.png" alt=""></p> <p>The oil pump sensor data comes in as strings of comma separated values. We use a Scala case class to define the Sensor schema corresponding to the sensor data, and a parseSensor function to parse the comma separated values into the sensor case class.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag22-1613716657923.png" alt=""></p> <pre><code class="language-java">case class Sensor(resid: String, date: String, time: String, hz: Double, disp: Double, flo: Double, sedPPM: Double, psi: Double, ch1PPM: Double) def parseSensor(str: String): Sensor = { val p = str.split(",") Sensor(p(0), p(1), p(2), p(3).toDouble, p(4).toDouble, p(5).toDouble, p(6).toDouble, p(7).toDouble, p(8).toDouble) } </code></pre> <p>Next, we use the DStream foreachRDD method to apply processing to each RDD in this DStream. We register the DataFrame as a table, which allows us to use it in subsequent SQL statements. We use an SQL query to find the max, min, and average for the sensor attributes.</p> <pre><code class="language-java">// for Each RDD sensorDStream.foreachRDD { rdd => val sqlContext = SQLContext.getOrCreate(rdd.sparkContext) rdd.toDF().registerTempTable("sensor") val res = sqlContext.sql( "SELECT resid. date, max(hz) as maxhz, min(hz) as minhz, avg(hz) as avghz, max(disp) as maxdisp, min(disp) as mindisp, avg(disp) as avgdisp, max(flo) as maxflo, min(flo) as minflo, avg(flo) as avgflo, max(psi) as maxpsi, min(psi) as minpsi, avg(psi) as avgpsi, FROM sensor GROUP BY resid,date") res.show() } </code></pre> <p>Here is example output from the query which shows the max, min, and average output from our sensors.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag24-1613716769080.png" alt=""></p> </li> </ul> </li> <li> <p>And/or Apply output operations</p> <ul> <li> <p>The sensorRDD objects are filtered for low psi, the sensor and alert data is converted to Put objects, and then written to HBase, using the saveAsHadoopDataset method. This outputs the RDD to any Hadoop-supported storage system using a Hadoop Configuration object for that storage system.</p> <pre><code>rdd.map(Sensor.convertToPut).saveAsHadoopDataset(jobConfig) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag25-1613716782608.png" alt=""></p> </li> </ul> </li> <li> <p>Start receiving data and processing it. Wait for the processing to be stopped.</p> <ul> <li> <p>To start receiving data, we must explicitly call start() on the StreamingContext, then call awaitTermination to wait for the streaming computation to finish.</p> <pre><code class="language-java">sensorDStream.foreachRDD { rdd => . . . } // Start the computation scc.start() // Wait for the coputation to terminate scc.awaitTermination() </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/image27-1613716793452.png" alt=""></p> </li> </ul> </li> </ol> <p><strong>HBase Table schema</strong></p> <p>The HBase Table Schema for the streaming data is as follows:</p> <ul> <li>Composite row key of the pump name date and time stamp</li> </ul> <p>The Schema for the daily statistics summary rollups is as follows:</p> <ul> <li>Composite row key of the pump name and date</li> <li>Column Family stats</li> <li>Columns for min, max, avg.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/sparkstream5-blog-1613716819138.png" alt=""></p> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform. There are several advantages of having MapR Event Store on the same cluster as all the other components. For example, maintaining only one cluster means less infrastructure to provision, manage, and monitor. Likewise, having producers and consumers on the same cluster means fewer delays related to copying and moving data between clusters, and between applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/imag26-1613716839874.png" alt=""></p> <p><strong>Software</strong></p> <p>You can download the code, data, and instructions to run this example from here:</p> <p>Code: <a target='\_blank' href='https://github.com/caroljmcdonald/mapr-streams-sparkstreaming-hbase'><a href="https://github.com/caroljmcdonald/mapr-streams-sparkstreaming-hbase">https://github.com/caroljmcdonald/mapr-streams-sparkstreaming-hbase</a></a></p> <p><strong>Summary</strong></p> <p>In this blog post, you learned how the MapR Data Platform integrates Hadoop and Spark with real-time database capabilities, global event streaming, and scalable enterprise storage.</p> <p><strong>References and More Information:</strong></p> <ul> <li><a href="https://learn.ezmeral.software.hpe.com/">Free Online training </a></li> <li><a href="/blog/8nDR4EW79KclyzwBwR5z/getting-started-with-mapr-event-store">Getting Started with MapR Event Store Blog</a></li> <li><a target='\_blank' href='http://spark.apache.org/docs/latest/streaming-programming-guide.html'>Apache Spark Streaming Programming Guide</a></li> </ul>https://developer.hpe.com/how-to-get-started-with-spark-streaming-and-mapr-event-store-using-the-k/https://developer.hpe.com/how-to-get-started-with-spark-streaming-and-mapr-event-store-using-the-k/Fri, 19 Feb 2021 06:14:14 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2017-05-15T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p><em>The telecommunications industry is on the verge of a major transformation through the use of advanced analytics and big data technologies like the MapR Data Platform.</em></p> <p>This post will help you get started using Apache Spark Streaming for consuming and publishing messages with MapR Event Store and the Kafka API. Spark Streaming is an extension of the core Spark API that enables continuous data stream processing. MapR Event Store is a distributed messaging system for streaming event data at scale. MapR Event Store enables producers and consumers to exchange events in real time via the Apache Kafka 0.9 API. MapR Event Store integrates with Spark Streaming via the Kafka direct approach. This post is a simple "how to" example. If you are new to Spark Streaming and the Kafka API you might want to read these first:</p> <ul> <li><a href="/blog/PkVBvojrpwSOLo5J5xzM/real-time-streaming-data-pipelines-with-apache-apis-kafka-spark-streamin"><u>Real-Time Streaming Data Pipelines with Apache APIs: Kafka, Spark Streaming, and HBase</u></a></li> <li><a href="/blog/8nDR4EW79KclyzwBwR5z/getting-started-with-mapr-event-store"><u>Getting Started with MapR Event Store</u></a></li> <li><a href="/blog/oogwLnyPqLuMyBK6KzV8/spark-streaming-with-hbase"><u>Spark Streaming with HBase</u></a></li> </ul> <h2><strong>Example Use Case Data Set</strong></h2> <p>The example data set consists of aggregated mobile network data generated by the Telecom Italia cellular network over the city of Milano and Trento. The data measures the location and level of interaction of the users with the mobile phone network based on mobile events that occurred on the mobile network over the course of two months in 2013. The projects in the challenge used this data to provide insights, identify and predict mobile phone-based location activity trends and patterns of a population in a large metropolitan area.</p> <h2>**The Data Set Schema **</h2> <ol> <li><strong>Square id</strong>: id of the location in the city grid</li> <li><strong>Time interval</strong>: the beginning of the time interval</li> <li><strong>Country code</strong>: the phone country code</li> <li><strong>SMS-in activity</strong>: SMS received inside the Square id</li> <li><strong>SMS-out activity</strong>: SMS sent inside the Square id</li> <li><strong>Call-in activity</strong>: received calls inside the Square id</li> <li><strong>Call-out activity</strong>: issued calls inside the Square id</li> <li><strong>Internet traffic activity</strong>: internet traffic inside the Square id</li> </ol> <p>The Data Records are in TSV format.</p> <h2><strong>Example Use Case Code</strong></h2> <p>First, you import the packages needed to integrate MapR Streams (Now called MapR Event Store) with Spark Streaming and Spark SQL.</p> <p>In order for Spark Streaming to read messages from MapR Event Store, you need to import classed from <em>org.apache.spark.streaming.kafka.v09</em>. <br> In order for Spark Streaming to write messages to MapR Event Store, you need to import classes from <em>org.apache.spark.streaming.kafka.producer</em>.  </p> <h2><strong>Parsing the Data Set Records</strong></h2> <p>A Scala <code>CallDataRecord</code> case class defines the schema corresponding to the TSV records. The <code>parseCallDataRecord</code> function parses the tab separated values into the CallDataRecord case class.</p> <h2><strong>Spark Streaming Code</strong></h2> <p>These are the basic steps for the Spark Streaming Consumer Producer code:</p> <ol> <li>Configure Kafka Consumer Producer properties.</li> <li>Initialize a Spark StreamingContext object. Using this context, create a DStream which reads message from a Topic.</li> <li>Apply transformations (which create new DStreams).</li> <li>Write messages from the transformed DStream to a Topic.</li> <li>Start receiving data and processing. Wait for the processing to be stopped.</li> </ol> <p>We will go through each of these steps with the example application code.</p> <h2><strong>1) Configure Kafka Consumer Producer properties</strong></h2> <p>The first step is to set the KafkaConsumer and KafkaProducer configuration properties, which will be used later to create a DStream for receiving/sending messages to topics. You need to set the following parameters:</p> <ul> <li>Key and value deserializers: for deserializing the message.</li> <li>Auto offset reset: to start reading from the earliest or latest message.</li> <li>Bootstrap servers: this can be set to a dummy host:port since the broker address is not actually used by MapR Event Store.</li> </ul> <p>​For more information on the configuration parameters, <a href="https://docs.datafabric.hpe.com/52/MapR_Streams/differences_in_configuration_parameters_for_producers_and_consumers.html">see the MapR Event Store documentation.</a></p> <h2><strong>2) Initialize a Spark StreamingContext object.</strong></h2> <p>We use the KafkaUtils createDirectStream method with a StreamingContext object, the Kafka configuration parameters, and a list of topics to create an input stream from a MapR Event Store topic. This creates a DStream that represents the stream of incoming data, where each message is a key value pair. We use the DStream map transformation to create a DStream with the message values.</p> <h2><strong>3) Apply transformations (which create new DStreams)</strong></h2> <p>Next we use the DStream <a target='\_blank' href='https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/streaming/dstream/DStream.html'>foreachRDD</a> method to apply processing to each RDD in this DStream.  We parse the message values into CallDataRecord objects, with the map operation on the DStream, then we convert the RDD to a DataFrame, which allows you to use <a target='\_blank' href='http://spark.apache.org/docs/latest/sql-programming-guide.html'>DataFrames and SQL</a> operations on streaming data. </p> <h2><strong>4) Write messages from the transformed DStream to a Topic</strong></h2> <p>The CallDataRecord RDD objects are grouped and counted by the squareId. Then this sendToKafka method is used to send the messages with the squareId and count to a topic.</p> <h2><strong>5) Start receiving data and processing it. Wait for the processing to be stopped.</strong></h2> <p>To start receiving data, we must explicitly call start() on the StreamingContext, then call awaitTermination to wait for the streaming computation to finish.</p> <h2><strong>Software</strong></h2> <p>You can download the code, data, and instructions to run this example from here: Code: <a target='\_blank' href='https://github.com/caroljmcdonald/mapr-streams-spark'><u><a href="https://github.com/caroljmcdonald/mapr-streams-spark">https://github.com/caroljmcdonald/mapr-streams-spark</a></u></a></p> <h2><strong>Summary</strong></h2> <p>In this blog post, you learned how to integrate Spark Streaming with MapR Event Store to consume and produce messages using the Kafka API.</p> <h2><strong>References and More Information:</strong></h2> <ul> <li>This use case example builds upon an example from the book <a target='\_blank' href='https://www.apress.com/us/book/9781484214800'><u>Pro Spark Streaming By Zubair Nabi</u></a>, which contains other interesting Spark Streaming examples.</li> <li><a href="https://docs.datafabric.hpe.com/52/Spark/Spark_IntegrateMapRStreams.html"><u>Integrate Spark with MapR Event Store Documentation</u></a></li> <li><a href="https://learn.ezmeral.software.hpe.com/"><u>Free Online training on MapR Event Store, Spark</u></a></li> <li><a target='\_blank' href='http://spark.apache.org/docs/latest/streaming-programming-guide.html'><u>Apache Spark Streaming Programming Guide</u></a></li> </ul> <p><em>This blog was originally published on September 6, 2016.</em></p>https://developer.hpe.com/mapr-database-spark-connector-with-secondary-indexes-support/https://developer.hpe.com/mapr-database-spark-connector-with-secondary-indexes-support/Fri, 19 Feb 2021 05:57:39 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Nicolas A Perez"], "publish": "2019-03-08T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/image2-1613714507521.jpg" alt=""></p> <p>MapR Data Platform offers significant advantages over any other tool on the big data space. MapR Database is one of the core components of the platform, and it offers state-of-the-art capabilities that blow away most of the NoSQL databases out there.</p> <p>An important add-on to MapR Database is the ability to use, for writing and querying, Apache Spark through the <a href="https://docs.datafabric.hpe.com/61/Spark/SparkConnectorsMapRDB.html"><em><strong>Connector for Apache Spark</strong></em></a>. Using this connector comes in very handy, since it can read and write from Spark to MapR Database, using the different Spark APIs, such as RDDs, <a href="https://docs.datafabric.hpe.com/60/Spark/SparkSQLandDataFrames.html">DataFrames</a>, and Streams.</p> <p>Using the connector, we can issue queries like the following one:</p> <pre><code class="language-scala">val df: DataFrame = sparkSession.loadFromMapRDB("/tmp/user_profiles", someSchema) </code></pre> <p>The resulting type is a <code>DataFrame</code> that we can use as any other DataFrame from any other source, as we normally do in Spark.</p> <p>But when using the provided connector for Apache Spark, and if we filter our dataset out, problems may start to emerge as any filter that is being applied to a field that is part of an index will not be used by the connector to optimize the reading from MapR Database. For instance, let's look at the following query:</p> <pre><code class="language-scala">val df = sparkSession.loadFromMapRDB("/tmp/user_profiles") val filteredDF = df.filter("first_name = 'Bill'") </code></pre> <p>The filter is being pushed down, so MapR Database does the filtering and only sends back the data that complies with the filter, reducing the amount of data transferred between MapR Database and Spark. However, if the field <em><strong>first_name</strong></em> is part of an index, the index is ignored and the table is fully scanned, trying to find the rows that comply with the filter resulting in a non-optimized query.</p> <p>By having an index on a field, we expect to use it so queries on that field are optimized, ultimately speeding up the computation. The provided connector for Apache Spark is simply not using the index capabilities of MapR Database to optimize the reading from MapR Database.</p> <h2>Necessity</h2> <p>Our team, MapR Professional Services, knows that filtering using MapR Database secondary indexes makes a huge difference in performance. Since many of our customers actually try to take advantages of this feature (secondary indexes), we have taken different approaches in order to force the use of the indexes when using Spark.</p> <p>In another blog post, "<a href="/blog/GJVN37RWmoumz0L2LM8V/how-to-use-secondary-indexes-in-spark-with-open-json-application-interfa"><em><strong>How to Use Secondary Indexes in Spark with OJAI</strong></em></a>," a fellow coworker explains some ways to overcome the issue on hand.</p> <p>Even when we take some shortcuts, we have to give up some of the nice constructs the default connector has, such as <code>.loadFromMapRDB(...)</code>. Even though this solution is not scalable, we can use some of these ideas, which aim generalizing the concept that can be used for general purpose computation with Spark in a generic approach.</p> <h2>An Independent Connector</h2> <p>In the past, I have extended Apache Spark in <em><strong>so</strong></em> many ways. I have written my own <a href="https://hackernoon.com/extending-our-spark-sql-query-engine-5f4a088de986"><em><strong>Custom Data Sources</strong></em></a> and most recently a <a href="https://hackernoon.com/spark-custom-stream-sources-ec360b8ae240"><em><strong>Custom Streaming Source for Spark Structured Streams</strong></em></a>.</p> <p>Once again, I have sailed into the adventure of writing my own Spark data source, but this time for MapR Database, so we can leverage the full advantage of secondary indexes while keeping the same API that the current MapR Database Connector for Apache Spark already has.</p> <p>By the end of this post, we will be able to write a query in the following way while fully using secondary indexes:</p> <pre><code class="language-scala">val schema = StructType(Seq(StructField("_id", StringType), StructField("uid", StringType))) val data = sparkSession .loadFromMapRDB("/user/mapr/tables/data", schema) .filter("uid = '101'") .select("_id") data.take(3).foreach(println) </code></pre> <h2>Spark Data Sources, Version 2</h2> <p>The following data source implementation uses Spark 2.3.1 and uses the data source API V2.</p> <p>Let's start by looking at the things we need.</p> <ol> <li><strong>ReadSupportWithSchema</strong>, which allows us to create a DataSourceReader.</li> <li><strong>DataSourceReader</strong>, which allows us to get the schema for our data, while we specify how to create a <strong>DataReaderFactory</strong>.</li> <li><strong>SupportsPushDownFilters</strong>, which allows us to intercept the query filters, so we can push them down to MapR Database.</li> <li><strong>SupportsPushDownRequiredColumns</strong>, which allows us to intercept the query projections, so we can push them down to MapR Database.</li> </ol> <p>Let's start by implementing <code>ReadSupportWithSchema</code>.</p> <pre><code class="language-scala">class Reader extends ReadSupportWithSchema { override def createReader(schema: StructType, options: DataSourceOptions): DataSourceReader = { val tablePath = options.get("path").get() new MapRDBDataSourceReader(schema, tablePath) } } </code></pre> <p>As we can see, we simply get the table path and the schema we want to use when reading from MapR Database. Then we pass them to <code>MapRDBDataSourceReader</code>.</p> <h2>MapRDBDataSourceReader</h2> <p><code>MapRDBDataSourceReader</code> implements <code>DataSourceReader</code>, and we are also mixing in <code>SupportsPushDownFilters</code> and <code>SupportsPushDownRequiredColumns</code> to indicate that we want to push filters and projections down to MapR Database.</p> <p>Let's look at each piece separately, so we can understand them better.  </p> <pre><code class="language-scala">class MapRDBDataSourceReader(schema: StructType, tablePath: String) extends DataSourceReader with SupportsPushDownFilters with SupportsPushDownRequiredColumns { private var projections: Option[StructType] = None override def readSchema(): StructType = ??? override def pushFilters(filters: Array[Filter]): Array[Filter] = ??? override def pushedFilters(): Array[Filter] = ??? override def pruneColumns(requiredSchema: StructType): Unit = ??? override def createDataReaderFactories(): util.List[DataReaderFactory[Row]] = ??? } </code></pre> <p>The <code>projections</code> variable will hold the schema we want to project, if any. In case we don't explicitly project fields by doing <code>.select</code>, we will project all the fields on the <code>schema</code> variable.</p> <p><code>readSchema</code> works in conjunction with <code>projections</code> and <code>pruneColumns</code>. If in our Spark query we specify a <code>select</code>, then the selected fields are passed to <code>pruneColumns</code>, and those are the only fields we will bring from MapR Database.</p> <pre><code class="language-scala">private var projections: Option[StructType] = None override def readSchema(): StructType = projections match { case None => schema case Some(fieldsToProject) => fieldsToProject } override def pruneColumns(requiredSchema: StructType): Unit = projections = Some(requiredSchema) </code></pre> <p><code>pushFilters</code> indicates what filters we have specified in the <code>where</code> or <code>filter</code> clause in our Spark query. Basically, we have to decide which of those we want to push down to MapR Database; the other ones will be applied by Spark after the data is in memory.</p> <pre><code class="language-scala">private var supportedFilters: List[Filter] = List.empty override def pushFilters(filters: Array[Filter]): Array[Filter] = filters.partition(isSupportedFilter) match { case (supported, unsupported) => supportedFilters = supported.toList unsupported } override def pushedFilters(): Array[Filter] = supportedFilters.toArray private def isSupportedFilter(filter: Filter) = filter match { case _: And => true case _: Or => true case _: IsNull => true case _: IsNotNull => true case _: In => true case _: StringStartsWith => true case _: EqualTo => true case _: LessThan => true case _: LessThanOrEqual => true case _: GreaterThan => true case _: GreaterThanOrEqual => true case _ => false } </code></pre> <p>In the snippet above, the series of filters we are pushing down match the filters handled by the official connector, so we provide the same functionality as the official connector at this level.</p> <p><code>createDataReaderFactories</code> creates a list of data readers that actually do the heavy work of reading from our source, MapR Database. In our case here, we are getting the table information and creating a reader for each region/partition in the table, so we can take advantage of the parallelism offered by MapR Database.</p> <pre><code class="language-scala">override def createDataReaderFactories(): util.List[DataReaderFactory[Row]] = com.mapr.db.MapRDB .getTable(tablePath) .getTabletInfos .zipWithIndex .map { case (descriptor, idx) => logTabletInfo(descriptor, idx) MapRDBTabletInfo(idx, descriptor.getLocations, descriptor.getCondition.asJsonString) } .map(createReaderFactory) .toList private def createReaderFactory(tabletInfo: MapRDBTabletInfo) = new MapRDBDataPartitionReader( tablePath, supportedFilters, readSchema(), tabletInfo, hintedIndexes) } </code></pre> <h2>MapRDBDataPartitionReader</h2> <p>We are almost done, yet the most important part is about to come.</p> <p>The <code>MapRDBDataPartitionReader</code> is where we actually build the MapR Database query and execute it in our MapR Database table. Notice that we are passing the table we are going to read from, and the filters and projections we want to push down, along with the partition each particular reader will be reading from. Remember that we are creating multiple instances of this class; each will read from a different MapR Database region/partition.</p> <pre><code class="language-scala">class MapRDBDataPartitionReader(table: String, filters: List[Filter], schema: StructType, tabletInfo: MapRDBTabletInfo, hintedIndexes: List[String] ) extends DataReaderFactory[Row] { override def createDataReader(): DataReader[Row] = ??? } </code></pre> <p>Now we need to connect to MapR Database by opening a connection and creating a document store object.</p> <pre><code class="language-scala">class MapRDBDataPartitionReader(table: String, filters: List[Filter], schema: StructType, tabletInfo: MapRDBTabletInfo, hintedIndexes: List[String] ) extends DataReaderFactory[Row] { import org.ojai.store._ @transient private lazy val connection = DriverManager.getConnection("ojai:mapr:") @transient private lazy val store: DocumentStore = connection.getStore(table) override def createDataReader(): DataReader[Row] = ??? } </code></pre> <p><code>query</code> creates the final command to be sent to MapR Database. This task is a matter of applying the query condition and the projections to our <code>connection</code> object.</p> <pre><code class="language-scala">private def query = { val condition = buildQueryConditionFrom(filters)(connection) val query = connection .newQuery() .select(schema.fields.map(_.name): _*) // push projections down .where(condition) // push filters down .build() query } } </code></pre> <p>The <code>buildQueryConditionFrom</code> method reads the Spark filters and transforms them into OJAI filters with the corresponding data types; this is where we push the filters down.</p> <p><em>It is very important to notice that since we are using <strong>OJAI</strong>, it will automatically use any secondary indexes for fields that are part of the filters we are applying. Make sure you check the output at the end of this post.</em></p> <p><code>documents</code> is a stream of data coming from MapR Database, based on <code>query</code>.</p> <pre><code class="language-scala">@transient private lazy val documents = { val queryResult = store.find(query) println(s"QUERY PLAN: ${queryResult.getQueryPlan}") queryResult.asScala.iterator } </code></pre> <p><code>createDataReader</code> uses the stream we have created (<code>documents</code>) to do the actual reading and returning of the data back to Spark.</p> <pre><code class="language-scala">override def createDataReader(): DataReader[Row] = new DataReader[Row] { override def next(): Boolean = documents.hasNext override def get(): Row = { val document = ParsableDocument(documents.next()) val values = schema .fields .foldLeft(List.empty[Any])((xs, field) => document.get(field) :: xs) .reverse Row.fromSeq(values) } override def close(): Unit = { store.close() connection.close() } } </code></pre> <p>Notice that <code>ParsableDocument(document).get(field)</code> handles the transformation from OJAI types back to Spark types. We support all OJAI types, except for <em>Interval</em>. Types are transformed recursively, so if we have a <em>Map</em> that has another <em>Map</em> inside with <em>Arrays</em> of <em>Ints</em>, we've got you covered.</p> <h2>Using Our Connector</h2> <p>At this point, we are ready to plug in our custom data source into Spark in the following way:</p> <pre><code class="language-scala">sparkSession .read .format("com.github.anicolaspp.spark.sql.Reader") .schema(schema) .load(path) </code></pre> <p>This allows us to use our own way to read from MapR Database, so that any filter being applied that is part of a secondary index on the physical table will be used to optimize the reading.</p> <h2>Syntax</h2> <p>In order to maintain a similar API to the one offered by the default MapR Database Connector, we added some syntax to our library in the following way:</p> <pre><code class="language-scala">object MapRDB { implicit class ExtendedSession(sparkSession: SparkSession) { def loadFromMapRDB(path: String, schema: StructType): DataFrame = { sparkSession .read .format("com.github.anicolaspp.spark.sql.Reader") .schema(schema) .load(path) } } } </code></pre> <p>Notice that our <code>loadFromMapRDB</code> method requires a <code>schema</code> to be passed in. This is a small difference from the official connector that supports schema inference. However, this is a design decision, since we know that most of the time we have the schema available. On the other hand, we know that inferring the schema does not always work correctly on the official connector.</p> <p>We can now use our connector in the same way we used the default/official connector.</p> <pre><code class="language-scala">val schema = StructType(Seq(StructField("_id", StringType), StructField("uid", StringType))) val data = sparkSession .loadFromMapRDB("/user/mapr/tables/data", schema) .filter("uid = '101'") .select("_id") data.take(3).foreach(println) </code></pre> <h2>Using MapR Database Secondary Indexes</h2> <p>When we run the code above, the <strong>TRACE</strong> output from <strong>OJAI</strong> looks similar to the following:</p> <pre><code class="language-scala">QUERY PLAN: {"QueryPlan":[ [{ "streamName":"DBDocumentStream", "parameters":{ "queryConditionPath":false, "indexName":"uid_idx", "projectionPath":[ "uid", "_id" ], "primaryTable":"/user/mapr/tables/data" } } ] ]} </code></pre> <p>Notice that it automatically uses the index called <code>uid_idx</code>, which is an index for the field <code>uid</code> that at the same time is the field being used in the Spark filter.</p> <h2>Conclusions</h2> <p>MapR Database is a powerful tool that runs as part of the MapR Data Platform. The Spark Connector offers an interesting way to interact with MapR Database, since it allows us to use all Spark constructs at scale when working with this NoSQL system. However, sometimes the default connector falls short because it does not use the secondary index capabilities of MapR Database when we need them the most.</p> <p>On the other hand, our implementation mimics the Connector API and ensures that the implemented Spark data source uses MapR Database secondary indexes, since it relies on pure OJAI queries that are able to support secondary indexes out of the box.</p> <hr> <p><em>Our library code can be found here: <a href="https://github.com/anicolaspp/MapRDBConnector">MapRDBConnector</a>.</em></p> <p><em>You can get the binaries directly from Maven Central:</em></p> <pre><code class="language-scala">&#x3C;dependency> &#x3C;groupId>com.github.anicolaspp&#x3C;/groupId> &#x3C;artifactId>maprdbconnector_2.11&#x3C;/artifactId> &#x3C;version>1.0.2&#x3C;/version> &#x3C;/dependency> </code></pre> <p>Or using sbt:</p> <pre><code class="language-scala">libraryDependencies += "com.github.anicolaspp" % "maprdbconnector_2.11" % "1.0.2" </code></pre> <hr> <p><em>Disclaimer: This is an independent effort to improve querying MapR Database. This library is not a substitute for the official <a href="https://docs.datafabric.hpe.com/61/Spark/SparkConnectorsMapRDB.html"><strong>Connector for Apache Spark</strong></a> offered by MapR as part of its distribution.</em></p> <p>This blog post was originally published on <a href="https://hackernoon.com/mapr-db-spark-connector-with-secondary-indexes-df41909f28ea?sk%3D3dd8eb1038b07bfbc11ae35c37f60743">Medium</a>.</p>https://developer.hpe.com/using-python-with-apache-spark/https://developer.hpe.com/using-python-with-apache-spark/Sat, 13 Feb 2021 07:12:32 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Jim Scott", "publish": "2015-11-23T08:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>Apache Spark is awesome. <a target='\_blank' href='https://www.python.org/'>Python</a> is awesome. This post will show you how to use your favorite programming language to process large datasets quickly.</p> <h2>Why Python?</h2> <p>Python has become one of the major programming languages, joining the pantheon of essential languages like C, C++, and HTML. Why has it become so popular? Because Guido van Rossum designed it as a teaching tool, making Python easy to learn.</p> <p>But it’s more than just easy. It’s also super useful due to its “batteries included” standard library. Python comes with a number of modules for interacting with the operating system, searching text with regular expressions, accessing the Internet, and just about anything else you could think of. You can download lots more or roll your own by interfacing with a C library.</p> <p>Since it’s a dynamic, interpreted language, you don’t have to declare variables or deal with memory management bugs like you do with C.</p> <p>For this reason, Python appeals to experienced programmers as well as beginners. Google uses it extensively.</p> <p>While you can use Scala, which Spark is built upon, there are good reasons to use Python. More people will likely be familiar with Python than with Scala, which will flatten the learning curve.</p> <p>Actually installing Spark is beyond the scope of this tutorial. This post will assume you have Python on your machine. Python is standard on most Linux/Unix distributions and Mac OS X. You can easily install Python from its homepage if you’re running Windows or just want a more recent version.</p> <p>You can launch the interactive Python shell for Spark with the command ./bin/pyspark from the Spark directory.</p> <p>The Spark equivalent of “Hello, world” is a word count. Here it is using Spark on Python, borrowed from the Apache Spark <a target='\_blank' href='http://spark.apache.org/'>homepage:</a></p> <pre><code class="language-python">text_file = spark.textFile("hdfs://...") text_file.flatMap(lambda line: line.split()) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a+b) </code></pre> <p>This creates a text_file object, splits it into lines, counts all the words in the line, and adds them back together.</p> <h2>Transformations</h2> <p>The defining feature of Apache Spark is its Resilient Distributed Datasets, or RDDs. These RDDs represent the data as immutable collections across all of the nodes in a Spark cluster. Operations are made out of transformations which are chained together. This means that all the operations on your data are nondestructive, allowing the system to recover from failures.</p> <p>Transformations can be applied to RDDs, thusly generating more RDDs. One of the more popular transformations available is <code>filter()</code>, which applies a function as an argument to all the values in your data and returns only the values that return true.</p> <p>To transform an external text file into an RDD, just use the command <code>MyFile = sc.TextFile(“data.txt”)</code> where MyFile is the name you want to use and “data.txt” is the name of your file. <code>map()</code> is similar but applies the values to every value. <code>flatMap()</code>, seen in the earlier example, does the same thing but returns Seq instead of a value.</p> <p>Here are some other transformations:</p> <ul> <li><code>reduceByKey()</code> takes a function as an argument and aggregates the data, such as adding values together.</li> <li><code>groupByKey()</code> takes the values and turns them into an iterable object.</li> <li><code>distinct()</code> removes duplicates from an RDD.</li> </ul> <h2>Using Collections</h2> <p>You can use the <code>collect()</code> action to get all the data in an RDD and turn it into an array. This lets you apply an action to your data.</p> <p>You can also parallelize an array for use on a Spark cluster.</p> <pre><code class="language-python">data = [1, 2, 3, 4, 5] distData = sc.parallelize(data) </code></pre> <h2>Actions</h2> <p>While transformations create new RDDs, actions give you some kind of result. The <code>collect()</code> and <code>sc.parallelize()</code> are two examples of actions we’ve seen.</p> <p>Here are some common actions:</p> <ul> <li><code>count()</code> counts the number of elements in an RDD.</li> <li><code>take()</code> fetches the first n elements as an argument from an RDD.</li> <li><code>foreach()</code> applies a function to each element in an RDD.</li> <li><code>saveAsTextFile()</code> saves an RDD into a text file in the specified path.</li> </ul> <h2>Conclusion</h2> <p>Python is a powerful programming language that’s easy to code with. Combined with Apache Spark, you have a powerful, easy way to process Big Data either in real time or with scripts. The MapR distribution gives you everything you need to process Big Data in your favorite language. The <a target='\_blank' href='http://spark.apache.org/documentation.html'>Apache Spark documentation</a> will give you more info on how to do so.</p>https://developer.hpe.com/getting-started-with-mapr-client-container/https://developer.hpe.com/getting-started-with-mapr-client-container/Sat, 13 Feb 2021 06:58:44 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Tugdual Grall", "publish": "2017-02-07T06:00:00.000Z", "tags": "streaming" </code></pre> <hr> <h2>Introduction</h2> <p>The MapR Persistent Application Client Container (PACC) is a Docker-based container image that includes a container-optimized MapR client. The PACC provides secure access to MapR Data Platform services, including MapR XD, MapR Database, and MapR Event Store. The PACC makes it fast and easy to run containerized applications that access data in MapR. </p> <p>In this article, you will learn how you can easily deploy and run your MapR application in the container.</p> <h2>About the application</h2> <p>This sample application is made of two services:</p> <ul> <li>"MapR Sensor" is a Java application that captures the information about the computer (CPU, Memory, ...) and publishes the data on a topic every 500ms.</li> <li>"MapR Web Application" is a Web application, powered by Jetty, that prints the messages from this topic into a Web page. Jetty HTTP Request log files are saved into MapR XD. This is interesting to have a centralized log management but also to run some processing or analytics jobs using Apache Spark or Drill.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/1-1613199562954.png" alt=""></p> <h2>Prerequisite</h2> <ul> <li>Docker 1.12.5 or later</li> <li>Apache Maven 3.x or later</li> <li>A Git client</li> </ul> <h2>Build and Deploy the Application using Docker</h2> <p>Clone the application from Github using the following command:</p> <pre><code class="language-bash">$ git clone https://github.com/mapr-demos/mapr-pacc-sample </code></pre> <p>Let's look at the project structure:</p> <pre><code>mapr-pacc-sample ├── pom.xml ├── sensor-service │ ├── Dockerfile │ ├── docker.env │ ├── pom.xml │ ├── run.sh │ └── src └── webserver-service ├── Dockerfile ├── docker.env ├── pom.xml ├── run.sh └── src </code></pre> <p>The project is made of two modules: sensor-service and webserver-service, part of the Apache Maven project.</p> <p>Each of the projects contains a Dockerfile defining a custom image.</p> <h2>Create Custom Docker Files</h2> <p>The Docker files should be configured to:</p> <ol> <li>Install and configure the MapR Client. This is done by building the container from the MapR PACC image.</li> <li>Deploy the Java application. This is done by copying the Jar into the container.</li> <li>Run the Java application. This is done by simply calling the Java command.</li> </ol> <p><strong>Docker file for Sensor/Producer</strong></p> <p>Open the file /mapr-pacc-sample/sensor-service/Dockerfile:</p> <pre><code class="language-bash">FROM maprtech/pacc:5.2.0_2.0_centos7 # Create a directory for your MapR Application and copy the Application RUN mkdir -p /usr/share/mapr-apps/ COPY ./target/sensor-service-1.0-SNAPSHOT.jar /usr/share/mapr-apps/sensor-service.jar COPY run.sh /usr/share/mapr-apps/run.sh RUN chmod +x /usr/share/mapr-apps/run.sh CMD ["start", "/usr/share/mapr-apps/run.sh", "/apps/sensors:computer"] </code></pre> <p><strong>Docker file for Web/Consumer</strong></p> <p>Open the file /mapr-pacc-sample/webserver-service/Dockerfile:</p> <pre><code class="language-bash">FROM maprtech/pacc:5.2.0_2.0_centos7 EXPOSE 8080 # Create a directory for your MapR Application and copy the Application RUN mkdir -p /usr/share/mapr-apps/ COPY ./target/webserver-service-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/share/mapr-apps/webserver-service.jar COPY run.sh /usr/share/mapr-apps/run.sh RUN chmod +x /usr/share/mapr-apps/run.sh CMD ["start", "/usr/share/mapr-apps/run.sh", "/apps/sensors:computer" , "/mapr/my.cluster.com/apps/logs/"] </code></pre> <p>The docker files are based on the MapR pacc:5.2.0_2.0_centos7 image. The application will automatically inherit from the various components installed by this container: MapR Database and Streams Client, POSIX Client for Containers, and Java 8.</p> <p>Both Docker files are defined with the following steps:</p> <ul> <li>Create a new directory /usr/share/mapr-apps/ to deploy the application </li> <li>Copy the application from maven target directory into this directory</li> <li>Copy the run.sh file to this folder and make it executable</li> <li>For the Web service the HTTP port 8080 is exposed</li> <li>The run.sh is automatically started using the CMD command</li> </ul> <p>The run.sh script will create /apps/logs using mount point and will start the Java application when the container starts.</p> <p>Now that you understand how to create your Dockerfile based on MapR PACC, let's build and run the application services.</p> <p><strong>Build and Run Docker images</strong></p> <p>Before starting the application, you must create a new MapR Event Store and new folder in MapR Distributed File and Object Store. Execute the following steps on MapR cluster:</p> <p><strong>1- Create a MapR Stream and Topic</strong></p> <p>On your MapR cluster, using a terminal window, run the following commands to create the /apps/sensors:computer topic:</p> <pre><code class="language-bash">$ maprcli stream create -path /apps/sensors -produceperm p -consumeperm p -topicperm p $ maprcli stream topic create -path /apps/sensors -topic computer </code></pre> <p><strong>2- Monitor the MapR Stream Topic</strong></p> <p>Start the Kafka Consumer Console to monitor the messages in the /apps/sensors:computer topic; in the terminal window of your MapR Cluster run:</p> <pre><code class="language-bash">$ /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-consumer.sh --new-consumer --bootstrap-server this.will.be.ignored:9092 --topic /apps/sensors:computer </code></pre> <blockquote> <p>Note: You'll need to install mapr-kafka package on cluster to execute kafka-console-*.sh commands.</p> </blockquote> <p><strong>Application "Sensor Producer"</strong></p> <p><strong>1- Build the new customer image</strong></p> <p>Build a new custom image for the sensor producer using the following commands:</p> <pre><code class="language-bash">$ cd sensor-service $ mvn clean package $ docker build -t mapr-sensor-producer . </code></pre> <p><strong>2- Run the new container</strong></p> <p>Run the container with the following command:</p> <pre><code class="language-bash">$ docker run -it -e MAPR_CLDB_HOSTS=192.168.99.18 -e MAPR_CLUSTER=my.cluster.com -e MAPR_CONTAINER_USER=mapr --name producer -i -t mapr-sensor-producer </code></pre> <p>This command creates a new container based on the mapr-sensor-producer image that we just built. The command use the following mandatory variables:</p> <ul> <li>The name of the container is producer</li> <li>MAPR_CLDB_HOSTS: the list of CLDB hosts of your MapR cluster</li> <li>MAPR_CLUSTER: the name of the cluster</li> <li>MAPR_CONTAINER_USER : the user that will be used to run the application</li> </ul> <p>These two variables are used to configure the MapR Client embedded in the container.</p> <p>The Java application is automatically started by Docker, and you should see messages in the Kafka Console.</p> <p>You can start/stop the container, using the following command:</p> <pre><code class="language-bash">$ docker start producer $ docker stop producer </code></pre> <p><strong>Application "Web Consumer"</strong></p> <p><strong>1- Build the new customer image</strong></p> <p>Build a new custom image for the sensor producer, using the following commands:</p> <pre><code class="language-bash">$ cd webserver-service $ mvn clean package $ docker build -t mapr-web-consumer . </code></pre> <p><strong>2- Run the new container</strong></p> <p>Run the container with the following command:</p> <pre><code class="language-bash">$ docker run -it --privileged --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse -e MAPR_CLDB_HOSTS=192.168.99.18 -e MAPR_CLUSTER=my.cluster.com -e MAPR_CONTAINER_USER=mapr -e MAPR_MOUNT_PATH=/mapr -p 8080:8080 --device /dev/fuse --name web -i -t mapr-web-consumer </code></pre> <p>This command creates a new container based on the mapr-web-consumer image that we just built. The command use the following variables:</p> <ul> <li>The name of the container is web</li> <li>The -p 8080:8080 is used to map the HTTP port from the container to your host</li> <li>--device /dev/fuse is used to add a fuse client to the container</li> <li>--privileged --cap-add SYS_ADMIN --cap-add SYS_RESOURCE are needed for the /dev/fuse to work</li> <li>MAPR_CLDB_HOSTS: the list of CLDB hosts of your MapR cluster</li> <li>MAPR_CLUSTER: the name of the cluster</li> <li>MAPR_CONTAINER_USER : the user that will be used to run the application</li> <li>MAPR_MOUNT_PATH: the mount path of MapR FUSE client, providing direct access to MapR XD</li> </ul> <p>This container starts the Java application that is an embedded Jetty. You can access the Web application from your host, using <code>http://localhost:8080</code>.</p> <p>This application is a simple web page that show the last messages published to the topic. This page is automatically refreshed every 5 seconds.</p> <p>The Jetty server is saving the HTTP Request logs directly in MapR XD, using the Fuse client; you can look at the logs in the /apps/logs folder of your MapR cluster.</p> <p>You can start/stop the container, using the following command:</p> <pre><code class="language-bash">$ docker start web $ docker stop web </code></pre> <h2>Conclusion</h2> <p>In this article, you have learned how to use the MapR PACC to deploy and run applications. The security on the MapR cluster used by this sample has not been enabled; if your cluster is secured, you can use additional MapR configuration to authenticate your services using security token.</p>https://developer.hpe.com/chapel-technical-lead-brad-chamberlain-opens-up-about-open-source/https://developer.hpe.com/chapel-technical-lead-brad-chamberlain-opens-up-about-open-source/Fri, 12 Feb 2021 11:26:55 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/brad-chamberlain-for-blog-1613129198318.jpg" alt="brad chamberlain for blog"></p> <p>As a leading global, edge-to-cloud Platform-as-a-Service company, Hewlett Packard Enterprise (HPE) prides itself in employing team members who share one common purpose: to advance the way people live and work. Because of this, HPE boasts some of the finest <a href="https://www.hpe.com/us/en/open-source.html">open source</a> engineering talent. In this blog series, you’ll get to meet a number of them as I interview some of the open source experts who make up our team.</p> <p>Dr. Bradford Chamberlain is the principal engineer for the <a href="https://chapel-lang.org/">Chapel parallel programming language</a>. His research interests center around parallel computation, particularly with respect to programming languages, compilers, algorithms, applications, and scalability. Brad comes to HPE through its recent acquisition of Cray Inc.</p> <h3>What attracted you to open source technologies?</h3> <p>I like open source because I tend to be a community-minded person. I like the idea of creating technologies that can span organizations and draw upon the expertise of people from a variety of backgrounds. And given that I benefit greatly from other open-source efforts, I really like the notion of giving back in a way that, hopefully, others will find similarly valuable.</p> <p>When code is open source, users have more confidence that a project can garner enough interest to live on, even if you, as the original developer, move on to pursue new endeavors. Users can continue to make changes, if needed, even if you aren’t there. For me, this is one of the many reasons that open source technologies are so attractive.</p> <h3>How did you first get involved with open source?</h3> <p>I got started with open source when I was working on the ZPL programming language while pursuing my Ph.D. in Computer Science &#x26; Engineering at the University of Washington. ZPL was open source only in the sense that we released our source code to users—the development itself was done in an internal repository. With the Chapel programming language that I work on now, our notion of open source extends to the entire development process. As an example, our project is <a href="https://github.com/chapel-lang/chapel">hosted at GitHub</a>.</p> <p>In both cases, the choice to use open source was very much for practical reasons. It’s generally quite challenging to get people to try any new programming language; making a language that’s proprietary and closed source only makes it more difficult. In contrast, when a language is free and open source, people are far more willing to give it a try, due in large part to the flexibility afforded by open source. It also has the practical benefit of dramatically simplifying our release process given all the architectures, operating systems, and compilers that Chapel is compatible with.</p> <h3>Can you tell me a little more about your work on Chapel and why you’re so excited about it?</h3> <p>Chapel is a programming language designed to make parallel programming on supercomputers far more productive than current approaches. However, it is also designed for portability such that Chapel programs can be developed and run on laptops, commodity clusters, or the cloud, for anyone who doesn’t have easy access to a supercomputer. In practice, I do most of my Chapel programming on a Mac laptop with a high degree of confidence that, once I’ve got it working, I can run it on an HPE Cray supercomputer without any problems.</p> <p>Chapel is also designed to appeal to programmers of all expertise levels. We strive to make it so easy to use that essentially every programmer could program a high-performance computing (HPC) system. Today, programming at such scales tends to require expertise in a number of fairly specialized and explicit programming notations, like C/C++, MPI, OpenMP, and CUDA. With Chapel, we strive to combine the ease-of-use of a language like Python with the performance and scalability you’d expect from conventional HPC techniques.</p> <h3>What’s currently on your list of things to do?</h3> <p>My team’s current and upcoming projects that I’m most excited about include compiling Chapel to GPUs, refactoring our compiler for speed and flexibility, and optimizing Chapel’s performance for HPE Cray EX. I’m also excited to engage with our community more as we modernize our website using some open source technologies that are new to me (Jekyll and Rouge). That said, my most immediate task is helping to wrap up our Spring 2021 release, due out in March.</p> <h3>As a developer, what you create says a lot about who you are. What impression do you hope you leave with others?</h3> <p>When people think of me, I hope they envision someone who is passionate about making supercomputers as straightforward and fun to program as laptops are, yet without sacrificing performance or scalability. I also strive to be as approachable and affable as my own mentors were, and hope that comes through to people.</p> <p>To learn more about the open source projects that HPE is involved with, please visit our <a href="https://www.hpe.com/us/en/open-source.html">open source page</a>. Interested in exploring what HPE offers for developers and data scientists? Check out our <a href="https://developer.hpe.com/">HPE DEV site</a> for a ton of articles, workshops, tutorials, and other resources.</p>https://developer.hpe.com/accessing-ilo-redfish-apis-and-hpe-oneview-apis-on-ansible-awx/https://developer.hpe.com/accessing-ilo-redfish-apis-and-hpe-oneview-apis-on-ansible-awx/Tue, 09 Feb 2021 03:40:59 GMT<p>There has been a growing demand in infrastructure management automation and an adaption toward infrastructure-as-code (IAC) with Ansible and AWX. Backed by Red Hat, Ansible has become one of the most popular IAC toolings for its simple and easy to understand coding style, masterless and agentless design, and the ability to create custom playbooks and roles for providing extra support to process automation. <a href="https://github.com/ansible/awx">AWX</a>, on the other hand, is the open-sourced version of Ansible Tower, which along with a set of tools, provides a web-based graphical user interface hub for consuming Ansible playbooks.</p> <p>Both the <a href="https://github.com/HewlettPackard/python-ilorest-library">HPE Python iLO REST Library</a> and the <a href="https://github.com/HewlettPackard/oneview-ansible">HPE OneView SDK for Ansible</a> do not come bundled with AWX. The AWX project does provide instructions on <a href="https://github.com/ansible/awx/blob/devel/docs/custom_virtualenvs.md#managing-custom-python-dependencies">managing custom Python dependencies</a> on AWX. This blog post is to share the process that we took to set up a custom Python environment for the Python iLO REST Library and the HPE OneView SDK in order to access the iLO Redfish APIs and the HPE OneView APIs from an AWX job.</p> <h1>Ansible and AWX setup on localhost</h1> <p>Ansible and AWX setup instructions can be found <a href="https://github.com/ansible/awx/blob/devel/INSTALL.md#installing-awx">here</a> on GitHub. If running behind proxies, make sure the proxy parameters, such as <em>http_proxy</em>, <em>https_proxy</em>, and <em>no_proxy</em> are configured accordingly in the installation inventory file. Once installation completes, the Ansible command becomes available on the localhost, and AWX runs as a containerized application, as shown here:</p> <pre><code class="language-bash">bash-4.4# ansible --help usage: ansible [-h] [--version] [-v] [-b] [--become-method BECOME_METHOD] [--become-user BECOME_USER] [-K] [-i INVENTORY] [--list-hosts] [-l SUBSET] [-P POLL_INTERVAL] [-B SECONDS] [-o] [-t TREE] [-k] [--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER] [-c CONNECTION] [-T TIMEOUT] [--ssh-common-args SSH_COMMON_ARGS] [--sftp-extra-args SFTP_EXTRA_ARGS] [--scp-extra-args SCP_EXTRA_ARGS] [--ssh-extra-args SSH_EXTRA_ARGS] [-C] [--syntax-check] [-D] [-e EXTRA_VARS] [--vault-id VAULT_IDS] [--ask-vault-pass | --vault-password-file VAULT_PASSWORD_FILES] [-f FORKS] [-M MODULE_PATH] [--playbook-dir BASEDIR] [-a MODULE_ARGS] [-m MODULE_NAME] pattern </code></pre> <pre><code class="language-bash">[root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 79248db66699 ansible/awx:17.0.1 "/usr/bin/tini -- /u…" 5 days ago Up 5 days 8052/tcp awx_task 11e9c78d53cf ansible/awx:17.0.1 "/usr/bin/tini -- /b…" 5 days ago Up 5 days 0.0.0.0:80->8052/tcp awx_web 46c529f34016 postgres:12 "docker-entrypoint.s…" 5 days ago Up 5 days 5432/tcp awx_postgres 75c8e09ad2de redis "docker-entrypoint.s…" 5 days ago Up 5 days 6379/tcp awx_redis </code></pre> <h1>Installing the HPE libraries on AWX</h1> <p>The installation of the HPE Python iLO REST library and the HPE OneView Ansible library can be achieved by the following three steps:</p> <ol> <li> <p>Log in to the <em>awx_task</em> container. Create a custom Python virtual environment and install the HPE Python iLO REST library and the HPE OneView Ansible library using the Python package manager.</p> </li> <li> <p>Log in to the <em>awx_web</em> containers. Create a new virtual environment with the same name as the one newly created in the <em>awx_task</em> container, and then install these two HPE Python libraries again using the Python package manager.</p> </li> <li> <p>Lastly, add the new Python environment to the <em>custom_virtualenvs</em> in AWX through its REST APIs.</p> </li> </ol> <p>Keep in mind to create the virtual environments first in <em>awx_web</em> and <em>awx_task</em> before adding it to the AWX custom virtual environment. The following sections talk more about each of the steps.</p> <h2>Create new custom virtual environments on <em>awx_task</em> and <em>awx_web</em></h2> <p>First, access the container BASH shell with the command <em>Docker exec -it</em> . For example, accessing the <em>awx task</em> shell :</p> <pre><code class="language-bash">[root@localhost ~]# docker exec -it awx_task /bin/bash bash-4.4# </code></pre> <p>Within the container, create a new Python virtual environment. For this example, the virtual environment is created at <em>/opt/hpeAutomation/venv</em>:</p> <pre><code class="language-bash">bash-4.4# mkdir -p /opt/hpeAutomation/ bash-4.4# chmod 0755 /opt/hpeAutomation/ bash-4.4# python3 -m venv /opt/hpeAutomation/venv </code></pre> <h2>Install the HPE libraries on <em>awx_task</em> and <em>awx_web</em></h2> <p>On each of the AWX containers, proceed as follows:</p> <ol> <li>YUM install the pre-requisites gcc:</li> </ol> <pre><code class="language-bash">bash-4.4# yum install gcc -y </code></pre> <ol start="2"> <li>Install the psutil Python module and the HPE libraries:</li> </ol> <pre><code class="language-bash">bash-4.4# /opt/hpeAutomation/venv/bin/pip3 install psutil bash-4.4# /opt/hpeAutomation/venv/bin/pip3 install ansible hpOneView hpICsp python-ilorest-library bash-4.4# git clone https://github.com/HewlettPackard/oneview-ansible.git bash-4.4# cd oneview-ansible bash-4.4# cp library/*.py /opt/hpeAutomation/venv/lib/python3.6/site-packages/ansible/modules/remote_management/oneview/ bash-4.4# cp library/module_utils/oneview.py /opt/hpeAutomation/venv/lib/python3.6/site-packages/ansible/module_utils </code></pre> <h2>Configure the <em>custom_virtualenvs</em> in AWX</h2> <p>Once finished installing the Python modules to <em>awx_task</em> and <em>awx_web</em> , the last step is to add the newly created virtual environment to <em>custom_virtualenvs</em> in AWX. This can be done with a HTTP PATCH to the AWX:</p> <pre><code class="language-bash">[root@localhost ~]# curl -X PATCH http://AWX_admin_username:AWX_adminpassword@AWX_ip_address/api/v2/settings/system/ \ -d '{"CUSTOM_VENV_PATHS": ["/var/lib/awx/venv/ansible", "/opt/hpeAutomation/"]}' \ -H 'Content-Type:application/json' </code></pre> <p>You can verify the <em>custom_virtualenvs</em> in AWX with a HTTP GET request to <em>/api/v2/config/</em> , as shown below:</p> <pre><code class="language-bash">[root@localhost ~]# curl -u AWX_admin_username:AWX_admin_password http://AWX_IP_address/api/v2/config/ { "time_zone": "UTC", "license_info": { "license_type": "open", "valid_key": true, "subscription_name": "OPEN", "product_name": "AWX" }, ... .. . "custom_virtualenvs": [ "/var/lib/awx/venv/ansible/", "/opt/hpeAutomation/venv/" ] } </code></pre> <p>Once completed, the custom virtual environment becomes available as an <em>Ansible Environment</em> in the <em>AWX Projects</em> and the Python libraries become accessible by the Job Templates in the project. <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/pic1-1612842533941.png" alt="project"> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/pic2-1612842545279.png" alt="template"></p> <p>There you have it. The AWX is now ready to run jobs for HPE OneView and iLO.</p>https://developer.hpe.com/event-driven-microservices-on-the-mapr-data-platform/https://developer.hpe.com/event-driven-microservices-on-the-mapr-data-platform/Fri, 05 Feb 2021 07:05:46 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Rachel Silver", "publish": "2016-09-27T07:00:00.000Z", "tags": "use-cases" </code></pre> <hr> <p>MapR is pleased to announce support for event-driven microservices on the MapR Data Platform. In this blog post, I’d like to explain what this means and how it fits into our bigger idea of “convergence.”</p> <h2>What are event-driven microservices?</h2> <p>Microservices are simple, single-purpose applications that work in unison via lightweight communications, such as data streams. They allow you to more easily manage segmented efforts to build, integrate, and coordinate your applications in ways that have traditionally been impossible with monolithic applications.</p> <p>By breaking up the pieces of a large application and isolating them into smaller microservice apps, you introduce agility, as these can typically be built and maintained by small, often cross-functional teams. And, they offer flexibility by promoting reuse across different solutions.</p> <p>Event-driven microservices leverage event streaming engines like MapR Event Store for Apache Kafka as the communications vehicles between them. And by converging file, database, and streaming services using our publish-and-subscribe framework, we can enable you to run analytical workloads using a combination of recent and historical data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/operational-analytics-in-mapr-converged-platform-1612508715159.png" alt=""></p> <h2>What advantages do microservices provide?</h2> <p>The logical and functional isolation of services provided by our microservices support is ideal for all complex workflows, but for machine learning training in particular. This is because it’s a natural infrastructure for tracking the different outputs of evolving application versions.</p> <p><strong>To illustrate this, let’s take a look at an example using data versioning and snapshots:</strong></p> <p>During your application development life cycle, enhancements to your code will result in different outputs, and these outputs are important to preserve during the development life cycle to compare results and verify improvement.</p> <p>MapR Volumes are logical partitions in your cluster that can contain databases, files, and streams. Thus, each application version output can be directed to a specific volume with the associated output data. And, in a microservices architecture, all versions can be deployed in parallel to make live comparisons and ensure a more graceful upgrade process.</p> <p>In addition, input data can be organized in a volume and then actively preserved using a snapshot. This creates an immutable copy of the data that can be used as the basis for ongoing testing against future versions of your application. You can keep enhancing your application and run it against a known data set to ensure you can identify changes that are a direct result of your code changes, not due to changes in the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/snapshots-in-app-development-1612508723835.png" alt=""></p> <p>Different versions of database records, files, and event data need to be tracked and managed together in a streaming environment.</p> <p>Ready for a real-world example? The diagram below shows the high-level architecture of the converged application blueprint.</p> <p>This was built to be specific to stock trading data analysis, but the concepts apply to any environment that deals with combining real-time streams &#x26; historical data. This application is a great example of how you can process a high-speed stream of incoming data and enable both operational and analytical workloads on a single converged cluster. Take a look!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/blueprint-arch-final-1612508815719.jpg" alt="App Blueprint"></p> <h2>How does this all work?</h2> <p>An integrated publish-and-subscribe framework to support event-driven applications.</p> <p>The foundation of our microservices offering is our low latency messaging system. It’s adaptable, scalable, and allows you to leverage your converged platform to integrate data-in-motion and data-at-rest to support real-time applications.</p> <p>It’s a remarkably versatile framework allowing communication pipelines in hybrid cloud microservice architectures, between local applications, and among Docker containers. Built-in resource multi-tenancy allow you to run both processing and messaging services in the same cluster and on the same nodes.</p> <p>MapR Event Store consumers will automatically load-balance across partitions, enabling the application to scale linearly with increasing data rates, and the stream can be queried directly with the results integrated with the output of any microservice app in the pipeline.</p> <h2>Simplified Microservices Monitoring and Management</h2> <p>As more organizations adopt these microservices architectures, they will need better tools for monitoring and management. The MapR Data Platform has already accomplished much of this with the following:</p> <ul> <li>Comprehensive monitoring of resource usage using MapR Monitoring components</li> <li>Support for containerized applications in Docker</li> <li>Continuous high availability and multi-master disaster recovery capabilities</li> <li>Unified security with access control expressions for stream access and analytics</li> </ul>https://developer.hpe.com/kubernetized-machine-learning-and-ai-using-kubeflow/https://developer.hpe.com/kubernetized-machine-learning-and-ai-using-kubeflow/Fri, 05 Feb 2021 07:00:42 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Rachel Silver", "publish": "2018-08-28T07:00:00.000", "tags": "nosql" </code></pre> <hr> <p>In my previous blog, <a href="/blog/EEVLz2X9vmSmZg6k6Dqx/end-to-end-machine-learning-using-containerization">End-to-End Machine Learning Using Containerization</a>, I covered the advantages of doing machine learning using microservices and how containerization can improve every step of the workflow by providing:</p> <ul> <li>Personalized Development Environments</li> <li>Agile Training Capabilities</li> <li>Microservices Frameworks for Model Serving</li> </ul> <p>Today, I'd like to talk about an example open source framework called <a href="https://www.kubeflow.org/">KubeFlow</a>. The KubeFlow infrastructure provides the means to deploy best-of-breed open source systems for machine learning to any cluster running Kubernetes, whether on-premises or in the cloud.</p> <h2>Introduction to KubeFlow</h2> <p>As Suzy Visvanathan states in her blog, <a href="/blog/JM9k0E924rtRj1QgQYnM/containers-kubernetes-and-mapr-the-time-is-now">Containers, Kubernetes, and MapR: The Time is Now</a>,  "Kubernetes has won the container orchestration war."</p> <p>While the early playing field was rife with competitors, like Mesosphere Marathon, Google Kubernetes, Docker Swarm, OpenStack Magnum, and VMware Photon, it's become clear that Kubernetes is now the industry's de facto standard. And, as a result, an ecosystem of tools has begun to emerge around Kubernetes, similarly to how it did when Hadoop first emerged from <a href="https://www.apache.org/">Apache</a>.</p> <p>As it began and remains in the Hadoop ecosystem, the Kubernetes ecosystem is starting out as a conglomerate of occasionally integrated and interrelated tools intended for use by data scientists and data engineers. The advantage so far for Kubernetes, in this regard, has been the ability to deploy pre-built offerings from container registries, allowing tools to be easily downloaded (‘pulled') and deployed on systems, without the traditional install pain around compiling from source that was frequently present in Hadoop ecosystem projects.</p> <p>And this is sufficient for simple deployments of single containers running isolated processes. But, in most cases, users want to scale workflows up and down, using multiple containers to run parallel processes. In order to do this, templatized offerings and the ability to easily deploy them are needed. The most common way that this is managed in Kubernetes is by using <a href="https://docs.helm.sh/developing_charts/">Helm Charts</a>, <a href="https://coreos.com/operators/">Operators</a>, or ksonnets, which are collections of YAML files that describe a deployment template such that it's reproducible and can be used to generate interconnected pods of containers on demand.</p> <p>What KubeFlow does is make all of this functionality a bit more user-friendly by providing some of the commonly used machine learning projects as pre-built templatized offerings (ksonnets) that are pretested to integrate together in one Kubernetes <a href="https://kubernetes.io/docs/tasks/administer-cluster/namespaces-walkthrough/">namespace</a> –not unlike our <a href="https://docs.datafabric.hpe.com/60/MapREcoSystemPacks.html">MapR Ecosystem Packs</a>. The initial list is based off of a common <a href="https://www.tensorflow.org/">TensorFlow</a> deployment pattern and has been opened up, or <a href="https://en.wikipedia.org/wiki/Democratization_of_technology">'democratized,'</a> to support other engines and modules.</p> <p>Here are a few of the offerings that are available in KubeFlow, but the list is always growing:</p> <ul> <li><a href="http://jupyter.org/index.html">Jupyter Notebook</a>: an open source web-based data science notebook</li> <li><a href="https://github.com/kubeflow/katib">Katib</a>: a hyperparameter tuning framework</li> <li><a href="https://www.kubeflow.org/docs/guides/components/tftraining/">TensorFlow training</a>: support for TensorFlow training jobs on Kubernetes</li> <li><a href="https://www.kubeflow.org/docs/guides/components/pytorch/">PyTorch training</a>: support for PyTorch training jobs on Kubernetes</li> <li><a href="https://www.kubeflow.org/docs/guides/components/seldon/">Seldon serving</a>: a model serving framework for Kubernetes that uses Istio</li> </ul> <h2>Integrating MapR with KubeFlow</h2> <p>MapR and KubeFlow are a very natural fit. Both are modeled on the concept of a namespace but use it to manage separate and complementary functions.</p> <p>In MapR, the global namespace is the key to unified data access and allows the joining of data across any divide, whether it be geographical or architectural. The MapR Global Namespace allows read/write access to any dataset to which the user has access, as if it were a local resource. This enables data security and isolation at the user, team, and tenant levels, and <a href="https://docs.datafabric.hpe.com/60/SecurityGuide/Authentication.html">MapR-SASL tickets</a> are used to securely authenticate users.</p> <p>In KubeFlow, a <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">Kubernetes namespace</a> is used to manage cluster compute resources, Kubernetes objects (e.g., pods), and application/job deployments. Namespaces are logical entities that are used to isolate and represent cluster compute resources and jobs at the user and tenant level. <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Kubernetes Secrets</a> are used to authenticate users and can be set up to synchronize with MapR-SASL tickets for seamless integration with platform security.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/image2-1612508426888.png" alt=""></p> <p>In both cases, namespaces are used for access control and to logically isolate tenant processes and data, which is ideal for multi-tenant organizations looking to easily manage security and performance. These namespaces complement and integrate with each other very nicely, leaving the end user with a seamless experience and the DataOps teams with a simple architecture to manage.</p> <h2>Conclusion</h2> <p>Containerization and a microservices architecture are critical across the entire data science workflow from prototyping to monitoring models in production. KubeFlow is a possible solution that does a really nice job of solving administrative and infrastructure problems while still allowing users to select their own tools. And, with MapR, these workflows can benefit from a best-of-breed data platform to speed the time from sandbox to production.</p>https://developer.hpe.com/end-to-end-machine-learning-using-containerization/https://developer.hpe.com/end-to-end-machine-learning-using-containerization/Fri, 05 Feb 2021 06:50:53 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Rachel Silver", "publish": "2018-06-28T10:00:00.000", "tags": "mapr-platform" </code></pre> <hr> <p>Lately, we've been talking a lot about containerization and how Kubernetes and MapR can pair up to enhance the productivity of your data science teams and decrease the time to insights. In this multi-part blog series, I will start with a high-level overview of why Kubernetes and containerization are appealing for data science environments. In a later iteration, I will provide an example of a framework that enables Kubernetized data science on your MapR cluster.</p> <p>Earlier this year, we released the MapR Volume Driver for Kubernetes, which enabled MapR customers to use Kubernetes clusters as extensions of their MapR computing space. This volume plugin provides the ability to mount directories from the MapR global namespace easily to Kubernetes pods, enabling stateful applications to run using your data in place.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/kubernetes-1612507940753.jpg" alt="Kubernetes"></p> <p>While the <a href="/blog/VqVzX3gAzrT7p5PzPAZA/event-driven-microservices-on-the-mapr-data-platform">microservices approach</a> is useful for all types of applications, it's particularly well-suited for the data science life cycle, starting with the exploration phase through to model deployment. Let's look at how it could benefit each phase:</p> <h2>Personalized Development Environments</h2> <p>The need to experiment in isolation is paramount for data scientists during the exploration phase when experimentation with different algorithms is an important part of developing the executable code used for training models. New, innovative, and domain-specific libraries are emerging every day, and data scientists need the ability to experiment with them in an agile way that does not run afoul of IT policy.</p> <p>Traditionally, client applications have been enabled using edge nodes. But this paradigm has not worked well for data science as, frequently, the data science teams will be comparing the results from multiple algorithms. The overhead to manage these libraries with all of their dependencies in a shared or distributed environment can be very painful for IT. This makes the potential for an isolated experimental environment very appealing for all sides – especially since security is handled around the perimeter of the container.</p> <p>The ability to run multiple containers in parallel, each containing and running their own set of libraries and tools, dramatically reduces friction between IT and data scientists and serves to enhance the productivity of data science teams overall.</p> <h2>Training the Trainer: Separating Compute from Storage</h2> <p>Traditionally, compute was kept very close to the data and, frequently, this required that data be moved or replicated across many cluster silos to accomplish a variety of analytical tasks. There's a lot of overhead involved in moving data, so, frequently, less superior tools were adopted to minimize this cost.</p> <p>The ability to meaningfully separate compute and storage is a breakthrough that allows developers to easily adjust and test different compute footprints. One example would be moving long-running CPU-based workflows to a GPU to get faster results as research or business priorities change.</p> <p>While this can be useful in the exploration and development phase, it's critical to the training phase. This is the phase when the executable code, typically developed in a data science notebook, is submitted to a scheduler to run against a larger dataset. The output of this job is typically a model, and the faster this training job completes, the more frequently the data scientist can tune their parameters to update and refine their model.</p> <p>Where containerization really benefits the training workflow is by allowing you to peg your containerized training code to whatever compute resource that you choose, while not requiring you to move any data. In MapR, this is done in Kubernetes using FUSE via the volume plugin. Because this is a POSIX-compliant interface, any Python algorithm can speak to your distributed data store as if it were stored in the container itself.</p> <p>This sort of flexibility can really make a difference in getting models out of the sandbox and into production quickly.</p> <h2>Machine Learning Models as Microservices</h2> <p>Microservices have been described as simple, single-purpose applications that work in unison via lightweight communications, such as data streams. They've traditionally enabled developers to more easily build, integrate, and manage their applications in an agile way that had typically been impossible with monolithic applications.</p> <p>Data science models are typically integrated into applications in order to generate insights, and containerization frameworks are fundamentally <a href="/blog/0N796xBvYxcyGq8Yo35N/event-driven-microservices-architecture-patterns-and-examples">microservices architectures</a>. In this capacity, model deployment architectures benefit greatly from microservices frameworks, because microservices frameworks are typically intended to accommodate functionally and logically isolated applications, running in parallel. This becomes very useful in model deployment scenarios when A/B testing is used, models need to be updated or replaced in place, or inter-model routing can benefit from a streaming data fabric or services mesh.</p> <h2>In Conclusion</h2> <p>Data science workflows benefit from containerization in every phase of the pipeline from exploration, training, and deploying models into production.</p> <p>Checkout the next iteration of this series, <a href="/blog/Oj0pNxBE3JsJB02E2KOj/kubernetized-machine-learning-and-ai-using-kubeflow">Kubernetized Machine Learning and AI Using KubeFlow</a>, where we'll dive deep into a new Kubernetes framework that supports end-to-end machine learning and data science.</p>https://developer.hpe.com/containers-kubernetes-and-mapr-the-time-is-now/https://developer.hpe.com/containers-kubernetes-and-mapr-the-time-is-now/Fri, 05 Feb 2021 06:41:32 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suzy Visvanathan", "publish": "2018-03-06T10:45:00.000", "tags": "mapr-platform" </code></pre> <hr> <p>If you have been following the world of containers, you already know that Kubernetes has won the container orchestration war. Docker was instrumental in commercializing the use of containers as a method to deploy applications. However, Docker and Kubernetes only address a minor percentage of total applications. They struggle to support stateful applications that require persisted data. With the announcement today, MapR changes all that by enabling Kubernetes to support the containerization of all applications.</p> <p>Containers allow an application to be packaged with a skeleton of its dependencies, giving it lightweight attributes, which ensures that the application can remain independent of the environment, infrastructure, and configuration. Containers, by nature, are stateless, which is a term to indicate that containers retain knowledge of data only during their lifecycle. Containers quickly became a developer's preferred platform, but it wasn't long before customers started thinking of deploying applications in production.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/kubernetes-list-1612507221998.png" alt="Kubernetes List"></p> <p>Containers in production bring forth a completely different set of requirements than containers in development. The biggest set of requirements is around data. The stateless nature of containers runs into significant challenges when relying on data that needs to be persisted across sessions or shared across applications.</p> <p>Customers have various issues to consider, resulting in a steep learning curve. The checklist will vary from customer to customer, due in part to the complexity and disparity in environments. Docker, which has become synonymous with containers, even though containers have been in use for a long time, does not necessarily offer solutions for all of these aspects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/standard-1612507235714.png" alt="Management de facto standard"></p> <p>If you consider management and orchestration, many solutions quickly cropped up to fill the gap of container management. Mesosphere Marathon, Google Kubernetes, Docker Swarm, OpenStack Magnum, and VMware Photon were all fighting for a piece of the container management pie. Initially, it appeared that Mesos would come out on top, but it has been quickly overtaken by Kubernetes, as is evident in the survey done by CNCF.</p> <p>Kubernetes is fast becoming the container orchestration and management de facto standard. A quick primer on Kubernetes summarizes its benefits and many of the reasons for its growth in popularity.</p> <ul> <li>Offers intelligent and balanced scheduling of containers</li> <li>Automates creation, deletion, and movement of containers</li> <li>Assists with easy scaling of containers</li> <li>Offers monitoring and self-healing abilities</li> </ul> <p>Leading public cloud vendors enable many aspects of services on Kubernetes. Kubernetes itself only manages and orchestrates containers but does not organically offer all of the aspects required for an organization to successfully deploy in production. The MapR integration with Kubernetes can assist customers through their journey of deploying containers in production.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/mapr-cdp-1612507248730.png" alt="MapR CDP"></p> <p>The MapR Data Platform (MCDP) provides an ideal platform for containerized applications. MCDP is built on the foundation of scale, high availability, and versatility in hosting different types of applications and across different environments. The distinctive aspect of the MapR Platform lies in how it combines a distributed data store that offers enterprise storage features to support all applications, including those running in containers, with a full-fledged, robust, big data analytics ecosystem. MapR addresses the aspects required for production deployments, starting with high availability, policy-driven data placement, and extending all the way to sound disaster recovery strategy. These are fundamental requirements that a data fabric must address. MapR customers build such data fabrics to capture, store, process, and analyze petabytes of data.</p> <p>For Kubernetes-based container deployments, MapR brings its data fabric that extracts and exposes the underlying storage capacity as persistent storage volumes to Kubernetes. Customers can now run applications in containers managed by Kubernetes and take advantage of the benefits of the MapR platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/versatility-1612507262825.png" alt="Versatility"></p> <p>MapR achieves this versatility by starting with a few basics:</p> <ul> <li>Vast scalability</li> <li>Global namespace</li> <li>Support for diverse data</li> <li>Native multi-temperature management</li> <li>Cloud-grade reliability</li> <li>Multi-tenancy and security</li> <li>Naturally analytics-ready</li> <li>Built-in operational application capability</li> <li>Reliable global pub/sub stream transport</li> <li>Capable of spanning from edge to edge</li> </ul> <p>Armed with these strong capabilities, MapR easily and naturally expands to varying use cases, applications, and organizations.</p> <p>With Kubernetes and containers starting to play a major role in many of these use cases where MapR has distinct complementing features already baked into its platform, it would be remiss of organizations not to take advantage of such a powerful combined solution.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/mapr-data-fabric-kubernetes-1612507274729.png" alt="MapR Data Fabric for Kubernetes"></p> <p>This unique position of MapR Data Fabric for Kubernetes can assist with an enterprise organization's journey with containers and deliver a powerful impact on day-to-day business that will have a long-standing benefit. The time is now to take advantage of MapR Data Fabric for Kubernetes.</p>https://developer.hpe.com/kafka-rest-proxy-performance-tuning-for-mapr-event-store/https://developer.hpe.com/kafka-rest-proxy-performance-tuning-for-mapr-event-store/Fri, 05 Feb 2021 06:32:34 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Mathieu Dumoulin", "publish": "2017-04-04T12:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>MapR Event Store is a “Kafka-esque” message streaming system which, similarly to Apache Kafka, provides very high throughput performance combined with low message latency and high reliability. Unique to MapR Event Store, however, is a broker-less design that vastly simplifies configuration and increases reliability, in addition to providing replication capabilities that enable some pretty cool use cases.</p> <p>With MEP 2.0, the MapR Data Platform adds a Kafka REST Proxy server. This upgrade opens MapR Event Store to use any language that supports REST API calls over HTTP, which is to say, virtually all modern languages. For example, Python and the requests module work really well.</p> <p>But is the Kafka REST Proxy able to access the tremendous performance potential of MapR Event Store at the same level as its primary Java API?</p> <p>In this post, I’d like to go over a few performance objectives and provide some guidance to help data engineers get the most out of this very useful technology.</p> <h2>The default case</h2> <p>We should start with some good news. MapR Event Store is very fast and is shipped by default with settings that should provide enough performance for most applications.</p> <h2>Fix very high latency for single API call (with CURL)</h2> <p>You have a shiny new MapR 5.2 cluster installed with all the bells and whistles. Everything works great, and you get around to wanting to give MapR Event Store a try. With the REST Proxy, this is a piece of cake.</p> <pre><code class="language-bash">curl -X POST -H "Content-Type:application/vnd.kafka.json.v1+json" --data '{"records":[{"value":{"foo":"bar"}}]}' "http://demo1:8082/topics/%2Fstreams%2Ftest%3Atopic1" </code></pre> <p>And the response takes about 3 seconds to come back. This very high latency is because of the default streams buffer time value of 3000ms.</p> <p>To fix, add the following to the kafka-rest.properties file (in /opt/mapr/kafka-rest/kafka-rest-<version>/config):</p> <pre><code class="language-bash">consumer.request.timeout.ms=125 streams.buffer.max.time.ms=125 </code></pre> <p>Reference: <a href="https://docs.datafabric.hpe.com/62/Kafka/REST-config-parameters.html">https://docs.datafabric.hpe.com/62/Kafka/REST-config-parameters.html</a></p> <p><strong>Beware of high CPU if the timeout is very low</strong></p> <p>Lowering the value of this property seems to correlate to much higher CPU utilization. When the value is 0, one or two of my cores get pegged to 100%. Above about 125ms, the impact to CPU utilization isn’t noticeable, at least to something like top.</p> <h2>About the URL for the topic</h2> <p><code>“/%2Fstreams%2Ftest%3Atopic1”</code> in the URL is because MapR Event Store includes a path and topic (i.e. /path/to/stream:topic) and that’s going to need to be URL encoded or else it won’t work.</p> <p>It’s possible to avoid this by setting a default stream, adding the following property to kafka-rest.properties:</p> <pre><code class="language-bash">streams.default.streams=/streams/test </code></pre> <p>In that case, the above example URL would simplify to <code>“http://demo1:8082/topics/topic1”.</code></p> <p>Reference: <a href="https://docs.datafabric.hpe.com/62/Kafka/REST-get-topic-metadata.html">https://docs.datafabric.hpe.com/62/Kafka/REST-get-topic-metadata.html</a></p> <h2>Increase Throughput Performance</h2> <p><strong>Number of topics and partitions</strong></p> <p>MapR Event Store is fast by default and handles a lot, albeit not everything, automatically. Some performance tuning comes from design considerations and just aren’t up to the streams messaging system at all.</p> <p><strong>Partitions > topics</strong></p> <p>Pros</p> <ul> <li>Throughput should be good, and data spread out evenly across the cluster</li> <li>Easier to create and use, less moving parts</li> </ul> <p>Cons</p> <ul> <li>Finding data specific to a particular object/event type/location will require scanning through more data, which will be slower.</li> </ul> <p><strong>Topics >> partitions</strong></p> <p>Pros</p> <ul> <li>It’s very efficient to get data from a specific object/event type/location if they are all stored in their own stream.</li> <li>A very high number of streams (hundreds of thousands or even millions) will naturally spread across the cluster and will spread out well on all nodes of the cluster.</li> </ul> <p>Cons</p> <ul> <li>The consumer needs to specify a regex pattern to pick all (or a group of) data. This may come at a performance penalty compared to a single topic with many partitions.</li> <li>Stream split is a relatively heavy operation, and it could trigger high load as new topics are created after the initial creation of topics is done.</li> </ul> <p>Of course, one could also decide to use an intermediate solution, in which there are lots of topics and each topic has some number of partitions. The way to decide is to consider how the application is going to be used and where flexibility is needed. In any case, the default number of partitions for new topics is one, so that’s something to change for sure.</p> <p>How to create streams with a custom number of partitions:</p> <pre><code class="language-python">stream create -path Stream Path [ -ttl Time to live in seconds. default:604800 ] [ -autocreate Auto create topics. default:true ] [ -defaultpartitions Default partitions per topic. default:1 ] $> maprcli stream create -path /streams/test -defaultpartitions 10 </code></pre> <p>As a rule of thumb, try to keep about 10 partitions per node per topic.</p> <p><strong>Session keep-alive and record arrays</strong></p> <p>To get the highest throughput, it’s going to be important to reduce overhead to maximize the CPU/network resources that do useful work moving your bits around. Here are some findings from recent engagements with customers using MapR Event Store in pilot and production projects:</p> <p><strong>Use an array of records as payload</strong></p> <p>Instead of producing a single record on each API call, push an array of records.</p> <p><strong>Bad:</strong></p> <pre><code class="language-python">{"value":{"foo":"bar"}} </code></pre> <p><strong>Good:</strong></p> <pre><code class="language-python">{"records":[ {"value":{"foo1":"bar1"}},{"value":{"foo2":"bar2"}} ,… ]} </code></pre> <p>Getting the best performance will require some experimentation to find the balance between how frequently to make calls vs. how many records to pack into each call.</p> <p>Our own experience shows that the Proxy can handle as much as 280MB/s on very large (100-200KB) message sizes. Internal tests demonstrate modest 5 node AWS clusters that are able to handle millions of small (1-200B) messages per second.</p> <p>There is no substitute for experimentation, given variability of data set, throughput, and cluster hardware resources as well as the business requirements of a specific use case.</p> <p><strong>Reuse a session to push data into the REST Proxy</strong></p> <p>We’ve found significant gains from switching from single, isolated POST calls to multiple calls within the same session.</p> <p>Here is an example with Python and the excellent requests module:</p> <p><strong>Bad:</strong></p> <pre><code class="language-python">def produce(payload): headers = {'Content-Type':'application/vnd.kafka.binary.v1+json'} r = requests.post('http://gw1:8082/topics/test', headers=headers, json=payload) </code></pre> <p><strong>Good:</strong></p> <pre><code class="language-python">def send_messages(url, payload): session = requests.Session() headers = {'Content-Type':'application/vnd.kafka.binary.v1+json'} while not is_done: response = session.post(url, headers=headers, data=payload) </code></pre> <p><strong>Tuning the embedded Jetty server</strong></p> <p>One of the resources that limits the throughput performance of the Kafka REST Proxy is CPU resource. Well, it turns out that the Proxy is running the Jetty 9 server in embedded mode. It is possible to do some tuning at that level.</p> <p>There is a good article about tuning the operating system (of both load generator and server) and load generators and jetty for high load in Jetty server. For sure, we cannot tune Jetty as it's embedded. But have a look at the following link. You can certainly tune the following meetings for high load:</p> <ul> <li>TCP buffer sizes</li> <li>Queue sizes for connection listening queue</li> <li>Port range at the load generator side, so it won’t starve on parts during high load</li> </ul> <p>Reference: <a href="http://wiki.eclipse.org/Jetty/Howto/High_Load">http://wiki.eclipse.org/Jetty/Howto/High_Load</a></p> <p><strong>How to increase the memory buffer</strong></p> <p>It is possible to tune the “buffer.memory” parameter. Its default value is 32m. However, this setting cannot exceed the total memory that the producer is going to use. At the end of the day, the kaka-rest is a JVM process.</p> <p>Without changing any parameters, the Kafka REST API uses 256m of memory at most. Therefore, the “buffer.memory” parameter cannot exceed this value. How come 256m? See the kaka-rest-run-class script (in /opt/mapr/kafka-rest/kakfa-rest-<version>/bin). It says the following:</p> <pre><code class="language-python"># Memory options if [ -z "$KAFKAREST_HEAP_OPTS" ]; then KAFKAREST_HEAP_OPTS="-Xmx256M" Fi </code></pre> <p>So, if you want to increase “<code>buffer.memory</code>” beyond 256m, provide the KAFKAREST_HEAP_OPTS value accordingly.</p> <p><strong>Waste-of-time parameters</strong></p> <p>The producer throughput of a single Kafka REST Proxy doesn't scale by increasing the “producer.threads” parameter. We tried to set it to 20, 50, 500, and even 10,000, but there were no visible performance differences.</p> <p>According to <a href="https://github.com/confluentinc/kafka-rest/issues/181">https://github.com/confluentinc/kafka-rest/issues/181</a>, it is not used in Kafka REST code, and the Kafka REST Proxy that runs on MapR is largely identical to the Confluent implementation, only with the libraries changed to MapR libraries. Our implementation shares this known issue for now.</p> <h2>Cluster Architecture</h2> <p><strong>Run the Proxy on dedicated server(s)</strong></p> <p>A great way to ensure optimal performance for performance-critical use cases is to use one or more dedicated servers for the Kafka REST Proxy. Instead of installing it on a shared cluster node, you can install the MapR Client on a separate server and install the REST Proxy there.</p> <p>To boost performance further, add additional servers and put them behind a load balancer. From the Client to the cluster, ensure that the network connectivity is as fast as can be afforded, since MapR will take advantage of all the network interfaces on the node automatically.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture1-1612507057820.png" alt="Kafka REST Proxy"></p> <p><strong>Run two or more Proxy processes on a dedicated node</strong></p> <p>This can be done by running the other server on a different port (e.g. 8083 instead of the default 8082). Given a server with enough physical cores, such as a two-socket design, this strategy can further increase the throughput.</p> <p>Note that running two proxy processes on a single server will not scale linearly the throughput. Our testing, in one instance, showed throughput to increase from 1,580 msg/s to 2,660 msg/s, good for close to a 70% increase.</p> <h2>About message size</h2> <p>The performance characteristics of MapR Event Store and the Kafka REST Proxy change depending on the message size. Very small messages will be handled faster than very large messages. Your design should take this difference into consideration and favor smaller messages.</p> <p>Keep in mind that the largest message size that can be handled very efficiently is about 100KB. Larger messages will come at some cost in peak performance, with a maximum best practice size of 2MB. Smaller messages are super-efficiently handled, so those are always fine.</p> <p>Given the large sweet spot, we’d advise favoring development simplicity and not worrying about it too much until individual messages get over about 100KB in size.</p> <h2>Do's and Don'ts</h2> <ul> <li>DO choose your performance targets based on the business needs and the use case.</li> <li>DO monitor the CPU, memory, and network load of the server running the Kafka REST Proxy.</li> <li>DO consider your design (cluster architecture, topics vs. partitions) before changing parameters.</li> <li>DO use a session if throughput is important.</li> <li>Do favor lots of smaller messages.</li> <li>DON'T change default parameters without a clear performance goal (latency, throughput, lower CPU usage, etc.).</li> <li>DON’T create too large messages (2MB+).</li> </ul> <h2>Some Additional Resources</h2> <p><strong>Script to measure throughput in MapR Event Store</strong></p> <pre><code class="language-python">#!/bin/bash STREAM="/streams/stream1" TOPIC="test" function sum_of_offset { maprcli stream topic info -path $STREAM -topic $TOPIC -json | awk -F':|,' '/maxoffset/ {n+=$2} END {print n}' 2> /dev/null } function epoch_ms { date +%s%3N } date +%T,%3N o=$(sum_of_offset); t=$(epoch_ms) while true do prev_o=$o; prev_t=$t o=$(sum_of_offset); t=$(epoch_ms) echo "$(date +%T,%3N) $((($o - $prev_o)*1000/($t - $prev_t))) msg/s" done </code></pre> <p><em>Thank you to Vince Gonzalez, Akihiko Kusanagi, Ted Dunning and Muthu Lalapet for their contributions to this blog post.</em></p>https://developer.hpe.com/a-functional-approach-to-logging-in-apache-spark/https://developer.hpe.com/a-functional-approach-to-logging-in-apache-spark/Fri, 05 Feb 2021 05:32:01 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-07-28T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <h2>A Functional Approach to Logging in Apache Spark</h2> <p>Logging in Apache Spark is very easy to do, since Spark offers access to a logobject out of the box; only some configuration setups need to be done. In a <a href="/blog/0NBjLpX5VAF3JKoDEqOo/how-to-log-in-apache-spark">previous post</a>, we looked at how to do this while identifying some problems that may arise. However, the solution presented might cause some problems when you are ready to collect the logs, since they are distributed across the entire cluster. Even if you utilize YARN log aggregation capabilities, there will be some contentions that might affect performance, or you could end up with log interleaves that corrupt the nature of the log itself.</p> <p>In this blog post, I will demonstrate how to solve these problems by taking a different, more functional approach.</p> <h2>The Monad Writer</h2> <p>I do not intend to go over the details about monads or the Monad Writer, so if you would like to learn more, please read “<a target='\_blank' href='https://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html'>Functors, Applicatives, And Monads In Pictures</a>” which is very informative about this topic.</p> <p>Just to put things in context, let’s say that the monad writer (<em>writer</em>) is a container that holds the current value of a computation in addition to the history (log) of the value (set of transformation on the value).</p> <p>Because the <em>writer</em> has monadic properties, it allows us to do functional transformations, and we will soon see how everything sticks together.</p> <h2>A Simplistic Log</h2> <p>The following code demonstrates a simplistic log.</p> <pre><code class="language-scala">object app { def main(args: Array[String]) { val log = LogManager.getRootLogger log.setLevel(Level.WARN) val conf = new SparkConf().setAppName("demo-app") val sc = new SparkContext(conf) log.warn("Hello demo") val data = sc.parallelize(1 to 100000) log.warn("I am done") } } </code></pre> <p>The only thing to note is that logging is actually happening on the Spark driver, so we don’t have synchronization or contention problems. However, everything starts to get complicated once we start distributing our computations.</p> <p>The following code won’t work (read this <a href="/blog/0NBjLpX5VAF3JKoDEqOo/how-to-log-in-apache-spark"><em>previous post</em></a> to know why)</p> <pre><code class="language-scala">val log = LogManager.getRootLogger val data = sc.parallelize(1 to 100000) data.map { value => log.info(value) value.toString } </code></pre> <p>A solution to this was also presented in the <a href="/blog/0NBjLpX5VAF3JKoDEqOo/how-to-log-in-apache-spark">previous post</a>, but it requires extra work to manage the logs.</p> <p>Once we start logging on each node of the cluster, we need to go to each node and collect each log file in order to make sense of whatever is in the logs. Hopefully, you are using some kind of tool to help you with this task, such as Splunk, Datalog, etc. However, you still need to know how to get those logs into your system.</p> <h2>Our Data Set</h2> <p>Our data set is a collection of the class “Person” that is going to be transformed while keeping an unified log of the operations on our data set.</p> <p>Let’s suppose we want our data set to get loaded, filter each person who is less than 20 years old, and finally, extract his/her name. This is a very silly example, but it will demonstrate how the logs are produced. You could replace these computations, but the idea of building a unified log will remain.</p> <h2>Getting the Writer</h2> <p>In order to use the <a target='\_blank' href='https://typelevel.org/projects/'>TypeLevel / Cats</a> library to import the monad writer, we add the following line to our build.sbt file.</p> <p><code>libraryDependencies += "org.typelevel" %% "cats" % "0.6.1"</code></p> <h2>Playing with our Data</h2> <p>Now, let’s define the transformations we are going to use. First, let’s load the data.</p> <pre><code class="language-scala">def loadPeopleFrom(path: String)(implicit sc: SparkContext) = s"loading people from $path" ~> sc.textFile(path) .map(x => User(x.split(",")(0), x.split(",")(1).toInt)) </code></pre> <p>In here, the ~> operation is defined via implicit conversions as follows:</p> <pre><code class="language-scala">implicit class toW(s: String) { def ~>[A](rdd: RDD[A]): Writer[List[String], RDD[A]] = Writer(s :: Nil, rdd) } </code></pre> <p>If you look closely, our loading operation is not returning an RDD; in fact, it returns the monad writer that keeps track of the logs.</p> <p>Let’s define the filter that we want to apply over the collection of users.</p> <pre><code class="language-scala">def filter(rdd: RDD[User])(f: User => Boolean) = "filtering users" ~> rdd.filter(f) </code></pre> <p>Again, we are applying the same function (~>) to keep track of this transformation.</p> <p>Lastly, we define the mapping, which follows the same pattern we just saw.</p> <pre><code class="language-scala">def mapUsers(rdd: RDDUser])(prefix: String): Writer[List[String], RDD[String]] = "mapping users" ~> rdd.map(p => prefix + p.name) </code></pre> <h2>Putting it together</h2> <p>So far we have only defined our transformations, but we need to stick them together. Scala for is a very convenient way to work with monadic structures. Let’s see how.</p> <pre><code class="language-scala">val result = for { person &#x3C;-&#x3C; span=""> loadPeopleFrom("~/users_dataset/")(sc) filtered &#x3C;-&#x3C; span=""> filter(person)(_.age &#x3C; 20) namesWithPrefix &#x3C;-&#x3C; span=""> mapUsers(filtered)("hello") } yield namesWithPrefix val (log, rdd) = result.run </code></pre> <p>Please note that the result is of the type:<code>Writer[List[String], RDD[String]]</code>.</p> <p>Calling result.run will give us the <code>log: List[String]</code> and the final computation is expressed by <code>rdd: RDD[String]</code>.</p> <p>At this point, we could use Spark logger to write down the log generated by the chain of transformations. Note that this operation will be executed on the Spark master, which implies that one log file will be created that contains all of the log information. We are also removing potential contention problems during the log writes. In addition, we are not locking the log file, which avoids performance issues by creating and writing to the file in a serial way.</p> <h2>Conclusion</h2> <p>In this blog post, I’ve shown you how to improve how to log on Apache Spark by using the Monad Writer. This functional approach allows you to distribute the creation of logs along with your computations, which is something that Spark does very well. However, instead of writing the logs on each worker node, you are collecting them back to the master to write them down.</p> <p>This mechanism has certain advantages over the previous implementation. You can now control exactly how and when your logs are going to be written down, you can boost performance by removing IO operations on the worker nodes, you can remove synchronization issues by writing the logs in a serial way, and you can avoid the hazard of fishing logs across your entire cluster.</p> <p>This post was originally published <a target='\_blank' href='https://medium.com/hackernoon/how-to-log-in-apache-spark-a-functional-approach-e48ffbbd935b#.87l91o1r3'>here.</a></p>https://developer.hpe.com/how-to-use-secondary-indexes-in-spark-with-open-json-application-interfa/https://developer.hpe.com/how-to-use-secondary-indexes-in-spark-with-open-json-application-interfa/Fri, 05 Feb 2021 05:25:49 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Ranjit Lingaiah"], "publish": "2019-02-12T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <h2>Introduction</h2> <p>Starting with MapR 6.0, MapR Database supports secondary indexes on fields in JSON tables.  </p> <p>Indexes provide flexibility to access data stored in MapR Database JSON. Secondary indexes provide efficient access to a wide range of queries on MapR Database JSON tables.</p> <p>By default, there is only one index on <code>_id</code> column; if applications query any other column, it can result in a full table scan to extract data from the underlying JSON tables. Secondary indexes solve this limitation by reducing the number of documents that applications would have to read from large tables. These indexes can be used with OJAI API, MapR Database JSON REST API, or MapR Drill, but not from Spark.</p> <p>In this blog post, we will look into how Spark can use OJAI API to leverage secondary indexes.</p> <h2>How to Use Secondary Indexes in Spark?</h2> <p>OJAI is the API to interface with MapR Database JSON. Most applications build using OJAI for filtering and sorting, which leverages secondary indexes to improve query response times. Here we will see how we can use this API in Spark to leverage secondary indexes.</p> <h2>Here are the steps:</h2> <ol> <li>Create a JSON table (user-info) with some JSON documents. One of the fields from this table will be used to look up fields from another table. The sample program, below, ingests the data into the table.</li> </ol> <pre><code class="language-bash">// Create JSON table $ maprcli table create -path /tmp/user-info -tabletype json // After the data is ingested into the table $ echo "find /tmp/user-info" | mapr dbshell ==================================================== *                  MapR Database Shell                   * * NOTE: This is a shell for JSON table operations. * ==================================================== Version: 6.0.1-mapr MapR Database Shell maprdb mapr:> find /tmp/user-info {"_id":"101","address":{"Pin":{"$numberLong":95985},"city":"sunnyvale","street":"35 town way"},"dob":{"$dateDay":"1987-05-04"},"interests":["squash","comics","movies"]} {"_id":"102","address":{"Pin":{"$numberLong":95652},"city":"san jose","street":"305 city way"},"dob":{"$dateDay":"1976-01-09"},"interests":["cricket","sketching"]} 2 document(s) found. </code></pre> <ol start="2"> <li>Create another JSON table (data-table) with JSON documents, with one of the fields (uid) in this table matching the field in the table created in step #1, and ingest sample data.</li> </ol> <pre><code class="language-bash">// Create JSON table $ maprcli table create -path /tmp/data-table -tabletype json //Sample data for querying $ cat data-type.json {"_id":"1","uid":"101","first_name":"tom"} {"_id":"2","uid":"102","first_name":"john"} {"_id":"3","uid":"103","first_name":"sam"} {"_id":"4","uid":"104","first_name":"thomas"} {"_id":"5","uid":"105","first_name":"david"} {"_id":"6","uid":"106","first_name":"robert"} {"_id":"7","uid":"107","first_name":"william"} {"_id":"8","uid":"108","first_name":"michael"} {"_id":"9","uid":"109","first_name":"bill"} {"_id":"10","uid":"110","first_name":"jarred"} // Put the sample data in hdfs $ hadoop fs -put data-table.json /tmp // Ingest the sample data into the JSON table $ mapr importJSON -idfield '_id' -mapreduce true -src /tmp/data-table.json -dst /tmp/data-table </code></pre> <ol start="3"> <li>Create a secondary index on data-table field uid.</li> </ol> <pre><code class="language-bash">$ maprcli table index add -path /tmp/data-table -index uid_idx -indexedfields uid </code></pre> <ol start="4"> <li> <p><a href="https://docs.datafabric.hpe.com/60/MapR-DB/Indexes/ExaminingOJAIQueryPlan.html">Enable log tracing</a> by setting the property "log4j.logger.com.mapr.ojai.store.impl=TRACE, stdout" in <code>log4j.properties</code>, located in <code>/opt/mapr/conf</code> directory. This step is optional; it is used to see if the OJAI query plan used the secondary index.  This is not recommended for production clusters, as it will generate lot of log data.</p> </li> <li> <p>The complete sample program is listed below. In this sample program, the <code>getDocuments()</code> method invokes the OJAI API to leverage the secondary index and returns an RDD.</p> </li> </ol> <pre><code class="language-scala">/* Copyright (c) 2009 &#x26; onwards. MapR Tech, Inc., All rights reserved */ package com.mapr.demo.spark.ojai.secondaryindex import com.fasterxml.jackson.annotation.{JsonIgnoreProperties, JsonProperty} import com.mapr.db.spark.impl.OJAIDocument import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.sql.SparkSession import org.ojai.types.ODate import com.mapr.db.spark.{field, _} import org.ojai.store.DriverManager import scala.collection.mutable.ListBuffer object SparkOjaiApplication { val userInfo = "/tmp/user-info" val dataTable = "/tmp/data-table" def main(args: Array[String]): Unit = { val spark = SparkSession.builder.appName("Spark-OJAI Secondary Index Application").master("local[*]").getOrCreate() val allUsers = spark.sparkContext.parallelize(getUsers()) val sc = spark.sparkContext //Save users to JSON table allUsers.saveToMapRDB(userInfo, createTable = false ) //Load all the people from the JSON table val allUsersInfo = sc.loadFromMapRDB(userInfo) //Extract JSON documents using secondary index val documentsRDD = allUsersInfo.mapPartitions(getDocuments) // print a few documents documentsRDD.take(3).foreach(println(_)) System.out.println("Number of documents extracted:" + documentsRDD.count()) } //Invokes OJAI api to query JSON documents using seconary index. def getDocuments(iterator: Iterator[OJAIDocument]): Iterator[String] = { val connection = DriverManager.getConnection("ojai:mapr:") val store = connection.getStore(dataTable) val dm = ListBuffer[String]() iterator.foreach(r => { val qs = "{\"$eq\": {\"uid\":\"%s\"}}".format(r.getDoc.getId.getString) System.out.println("Finding documents for qs:" + qs); val query = connection.newQuery().select("_id") //This option is not required. OJAI client makes the determination to use secondary index. // Since the sample data set is small, I'm enabling this option to use secondary index. .setOption(com.mapr.ojai.store.impl.OjaiOptions.OPTION_USE_INDEX, "uid_idx") .where(qs).build() val iterator = store.find(query).iterator() if (iterator.hasNext) { dm += iterator.next().asJsonString() } }) //Close the Document Store store.close() //Close the OJAI connection connection.close() dm.toIterator } // User documents. The _id field of users is used to query the user in the data-table. def getUsers(): Array[Person] = { val users: Array[Person] = Array( Person("101", ODate.parse("1976-1-9"), Seq("cricket", "sketching"), Map("city" -> "san jose", "street" -> "305 city way", "Pin" -> 95652)), Person("102", ODate.parse("1987-5-4"), Seq("squash", "comics", "movies"), Map("city" -> "sunnyvale", "street" -> "35 town way", "Pin" -> 95985)) ) users } } @JsonIgnoreProperties(ignoreUnknown = true) case class Person (@JsonProperty("_id") id: String, @JsonProperty("dob") dob: ODate, @JsonProperty("interests") interests: Seq[String], @JsonProperty("address") address: Map[String, Any]) </code></pre> <ol start="6"> <li>To build the sample program, clone the Git repo and use Maven to build the program.</li> </ol> <pre><code class="language-bash">$ git clone https://github.com/ranjitreddy2013/spark-using-ojai-secondary-index-example $ cd spark-using-ojai-secondary-index-example $ mvn clean install </code></pre> <ol start="7"> <li>To run, copy spark-ojai-secondaryindex-1.0-SNAPSHOT.jar from target folder to an edge node or cluster node and submit to the cluster using <code>spark-submit</code>.</li> </ol> <pre><code class="language-bash">/opt/mapr/spark/spark-2.2.1/bin/spark-submit --class com.mapr.demo.spark.ojai.secondaryindex.SparkOjaiApplication --master yarn --deploy-mode client --driver-java-options "-Dlog4j.configuration=file:///opt/mapr/conf/log4j.properties" --conf "spark.yarn.executor.memoryOverhead=1G"  --executor-memory 2G --num-executors 1 --executor-cores 1 /home/mapr/spark-ojai-secondaryindex-1.0-SNAPSHOT.jar </code></pre> <ol start="8"> <li>The sample output is shown below. In addition to this output, there will be DEBUG and TRACE logs.</li> </ol> <pre><code class="language-scala">Finding  documents for qs:{"$eq": {"uid":"101"}} Finding  documents for qs:{"$eq": {"uid":"102"}} {"_id":"1","first_name":"tom","uid":"101"} {"_id":"2","first_name":"john","uid":"102"} Number of documents extracted:2 </code></pre> <ol start="9"> <li>Verify the logs if the secondary index is used by the OJAI query plan. Note the <code>indexName</code> used by the OJAI query plan.</li> </ol> <pre><code class="language-bash">2019-01-11 10:49:35,876 TRACE com.mapr.ojai.store.impl.OjaiDocumentStore logQueryPlan Executor task launch worker for task 204: Ojai Query Plan: '[{"streamName":"DBDocumentStream","parameters":{"queryConditionPath":true,"indexName":"uid_idx","projectionPath":["_id"],"primaryTable":"/tmp/data-table"}},{"streamName":"RowkeyLookup","parameters":{"condition":"(uid = "88d800cf-39c9-482f-856c-486090c3de2c")","primaryTable":"/tmp/data-table"}}]' 2019-01-28 12:04:33,087 TRACE com.mapr.ojai.store.impl.OjaiDocumentStore logQueryPlan Executor task launch worker for task 201: Ojai Query Plan: '[{"streamName":"DBDocumentStream","parameters":{"queryConditionPath":true,"indexName":"uid_idx","projectionPath":["_id"],"primaryTable":"/tmp/data-table"}}]' </code></pre> <h2>Conclusion:</h2> <p>In this blog post, you learned how to use OJAI secondary indexes from Spark.</p> <h2>Additional Resources:</h2> <ul> <li><a href="https://docs.datafabric.hpe.com/61/MapR-DB/JSON_DB/getting_started_json_ojai_build_java_app.html">Developing OJAI Applications</a></li> <li><a href="https://docs.datafabric.hpe.com/60/MapR-DB/Indexes/DeterminingSIUsage.html">Determining Secondary Index Usage</a></li> <li><a href="https://docs.datafabric.hpe.com/60/MapR-DB/Indexes/admin-adding-indexes.html">Adding Secondary Indexes on JSON Tables</a></li> </ul>https://developer.hpe.com/setting-up-spark-dynamic-allocation-on-mapr/https://developer.hpe.com/setting-up-spark-dynamic-allocation-on-mapr/Fri, 05 Feb 2021 05:19:26 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Tugdual Grall", "publish": "2016-11-03T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>Apache Spark can use various cluster managers to execute applications (Standalone, YARN, Apache Mesos). When you install Apache Spark on MapR, you can submit an application in Standalone mode or by using YARN.</p> <p>This blog post focuses on YARN and dynamic allocation, a feature that lets Spark add or remove executors dynamically based on the workload. You can find more information about this feature in this presentation from Databricks:</p> <ul> <li><a target='\_blank' href='https://www.slideshare.net/databricks/dynamic-allocation-in-spark'>Dynamic Allocation in Spark</a></li> </ul> <p>Let’s see how to configure Spark and YARN to use dynamic allocation (that is disabled by default).</p> <h2>Prerequisites</h2> <ul> <li>MapR Data Platform cluster</li> <li>Apache Spark for MapR installed</li> </ul> <p>The example below is for MapR 5.2 with Apache Spark 1.6.1; you just need to adapt the version to your environment.</p> <h2>Enabling Dynamic Allocation in Apache Spark</h2> <p>The first thing to do is to enable dynamic allocation in Spark. To do this, you need to edit the Spark configuration file on each Spark node</p> <pre><code class="language-bash">/opt/mapr/spark/spark-1.6.1/conf/spark-defaults.conf </code></pre> <p>and add the following entries:</p> <pre><code class="language-bash">spark.dynamicAllocation.enabled = true spark.shuffle.service.enabled = true spark.dynamicAllocation.minExecutors = 5 spark.executor.instances = 0 </code></pre> <p>You can find additional configuration options in the <a target='\_blank' href='http://spark.apache.org/docs/1.6.1/configuration.html#dynamic-allocation'>Apache Spark Documentation</a>.</p> <h2>Enabling Spark External Shuffle for YARN</h2> <p>Now you need to edit the YARN configuration to add information about Spark Shuffle Service. Edit the following file on each YARN node:</p> <pre><code class="language-bash">/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-site.xml </code></pre> <p>and add these properties:</p> <pre><code class="language-bash">&#x3C;property> &#x3C;name>yarn.nodemanager.aux-services&#x3C;/name> &#x3C;value>mapreduce_shuffle,mapr_direct_shuffle,spark_shuffle&#x3C;/value> &#x3C;/property> &#x3C;property> &#x3C;name>yarn.nodemanager.aux-services.spark_shuffle.class&#x3C;/name> &#x3C;value>org.apache.spark.network.yarn.YarnShuffleService&#x3C;/value> &#x3C;/property> </code></pre> <h2>Add Spark Shuffle to YARN classpath</h2> <p>Spark Shuffle service must be added to the YARN classpath. The jar is located in the Spark distribution:</p> <pre><code class="language-bash">/opt/mapr/spark/spark-1.6.1/lib/spark-1.6.1-mapr-1605-yarn-shuffle.jar </code></pre> <p>To do this, add the jar in the following folder on each node:</p> <pre><code class="language-bash">/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib </code></pre> <p>You can either copy the file or create a symlink:</p> <pre><code class="language-bash">$ ln -s /opt/mapr/spark/spark-1.6.1/lib/spark-1.6.1-mapr-1605-yarn-shuffle.jar /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib </code></pre> <h2>Restart YARN</h2> <p>Since you have changed the YARN configuration, you must restart your node managers using the following command:</p> <pre><code class="language-bash">$ maprcli node services -name nodemanager -action restart -nodes [list of nodes] </code></pre> <h2>Submitting a Spark Job</h2> <p>Your MapR cluster is now ready to use Spark dynamic allocation. This means that when you submit a job, you do not need to specify any resource configuration. For example:</p> <pre><code class="language-bash">/opt/mapr/spark/spark-1.6.1/bin/spark-submit \ --class com.mapr.demo.WordCountSorted \ --master yarn \ ~/spark-examples-1.0-SNAPSHOT.jar \ /mapr/my.cluster.com/input/4gb_txt_file.txt \ /mapr/my.cluster.com/user/mapr/output/ </code></pre> <p>Note that you can still specify the resources, but in this case, the dynamic allocation will not be used for this specific job. For example:</p> <pre><code class="language-bash">/opt/mapr/spark/spark-1.6.1/bin/spark-submit \ --class com.mapr.demo.WordCountSorted \ --master yarn \ --num-executors 3 --executor-memory 1G \ ~/spark-examples-1.0-SNAPSHOT.jar \ /mapr/my.cluster.com/input/4gb_txt_file.txt \ /mapr/my.cluster.com/user/mapr/output/ </code></pre> <p>In this blog post, you learned how to set up Spark dynamic allocation on MapR.</p>https://developer.hpe.com/the-challenges-of-sharing-gpus-and-how-to-solve-them/https://developer.hpe.com/the-challenges-of-sharing-gpus-and-how-to-solve-them/Wed, 03 Feb 2021 15:54:07 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/pinkish-image-4-run-ai-post-1612367610553.JPG" alt="pinkish image 4 run ai post"></p> <p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Whether purchasing GPUs for on-premise machines or renting them in the cloud, GPUs for AI can be expensive. Organizations want to get the most out of their on-premise GPUs and use GPUs in the cloud as efficiently as possible. The HPE Ezmeral team and <a href="http://www.run.ai/">Run:AI</a> recently worked together in a number of customer engagements to help researchers take better advantage of their GPU resources. This post offers some of our takeaways and includes resources that can help you get started in doing the same.</p> <p>Though many <a href="https://www.zdnet.com/article/facebooks-latest-giant-language-ai-hits-computing-wall-at-500-nvidia-gpus/">articles</a> cite examples of the <a href="https://syncedreview.com/2018/05/17/ai-doubling-its-compute-every-3-5-months/">insatiable demand</a> for compute resources, in practice, GPUs are often underutilized. When Run:AI starts work with a new customer, we typically see a GPU utilization rate of between 25 and 30 percent. IT is typically surprised by this – they assume that resources are being fully utilized. But, if you think about it, it’s actually quite logical:</p> <ul> <li> <p>GPUs tend to be idle during non-work hours (e.g. nights, weekends).</p> </li> <li> <p>They can also be idle during work breaks (e.g. coffee breaks, lunch).</p> </li> <li> <p>They can be idle when a researcher is building a model (e.g. developing a model in a Jupyter notebook). Note: A Jupyter Notebook is a classic example. A user working with a Jupyter Notebook usually alternates between writing code, executing it on the GPU, and examining the results. The GPU is kept idle for long periods of time during this process.</p> </li> <li> <p>They can even be idle during the execution of a GPU-consuming application, e.g. training workloads. This is because the application has some work to do on the CPU as well and wait for I/O. Note: Most applications have CPU and I/O work in between launching GPU kernels. The GPU utilization of a deep-learning model running solely on a GPU can be much less than 100%.</p> </li> </ul> <p>Increasing GPU utilization and minimizing idle times can drastically reduce costs and help achieve model accuracy faster. To do this, one needs to improve the sharing of GPU resources.</p> <h3>Sharing a GPU is complex</h3> <p>Applications running on the same GPU share its memory. Every byte allocated by one application leaves one less byte for other applications to use. The only way for multiple applications to run simultaneously is to cooperate with one another. Otherwise, applications can easily, and even mistakenly, impact each other.</p> <p>In addition, many applications arrogantly assume they are the only ones running on that GPU, and allocate the entire GPU memory upfront by default (e.g. TensorFlow). This is a common paradigm when using an external processing unit (in addition to the CPU). Code modifications are required to change this default behavior (e.g. allow growth in TensorFlow) and might impact the application performance (i.e. due to fragmentation). This might be even impossible in some cases; for example, when executing sealed Docker images without access to the source code they contain.</p> <p>When running multiple containers on the same GPU, cooperation is much less a legitimate requirement, as containers should not be aware of each other, and certainly should not be accessible to one another.</p> <p>When multiple users are supposed to share a GPU, they require coordination between one another, making this a logistical issue as well and not only technical.</p> <p>All the above makes sharing a GPU between applications, and containers in particular, inconvenient and not scalable or dynamic.</p> <h3>What do we mean by “dynamic”?</h3> <p>Ideally, sharing a GPU would work like this; for me to have the resources I need, when I need them, and for you to have what you need, when you need them. That would require dynamic allocation of resources.</p> <p>Sharing the same GPU between multiple applications and containers has two requirements.</p> <ol> <li> <p>The first requirement is to be able to modify the application to use only a portion of the GPU memory (i.e. code changes and manual tweaks). This might seem easy but not every user can do so, as it might require deep knowledge with the internals of the application and how to configure it. Sometimes it might not even be possible - as explained above, an example of this would be when receiving a docker image without controlling what application is running inside and with which configuration.</p> </li> <li> <p>The second requirement is to decide how much GPU memory should be used by each and every application. This might be relatively easy for a single user to do as there is only a single person who needs to decide. A team might also be able to do so, but in a very inefficient way.</p> </li> </ol> <p>In instances where a team is involved, the team can decide on a strict policy in which each member gets an equal, static share of the GPU. For example, in a team of three members, each one would be given exactly one third of the GPU memory.</p> <p>This might sound satisfactory but the truth is that the GPU would be underutilized, based on the scenarios we outlined above. If, at any time, one of the team members did not use his or her share of the GPU memory, it would still be left unused. Additionally, the team members would never be able to use more than their share without breaking their agreement and risking applications of other members with Out Of Memory failures.</p> <p>This unused GPU memory could have been allocated by another team member, allowing him or her to run more GPU-consuming applications (e.g. larger deep learning models).</p> <h3>The answer: Share GPUs by enabling access to fractions of GPUs</h3> <p>A fractional GPU system, such as one built by Run:AI, transparently gives data science and AI engineering teams the ability to run multiple workloads simultaneously on a single GPU. Virtualized logical GPUs have their own memory and computing space that containers can use and access as if they were self-contained processors. This enables several deep learning workloads to run in containers side-by-side on the same GPU without interfering with each other.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/gpu-orchestration-dashboard-1612369520951.png" alt="gpu orchestration dashboard"></p> <p>Fractional GPU capabilities enable simplified sharing of single and multiple GPUs. In the figure above, you can see that more than 70% of the 160 pooled GPUs are fully utilized. Researchers have maximized cluster utilization, and can see that there are still idle GPUs available to any researchers who need them. This is one way <a href="https://www.hpe.com/us/en/software/marketplace/runai.html">HPE Ezmeral Container Platform and Run:AI</a> are able to help customers bring AI solutions to market faster. By maximizing the utilization of their GPU clusters, customers can build and train concurrent AI models without resource limitations. We’ve <a href="https://docs.run.ai/Researcher/Walkthroughs/walkthrough-fractions/">shared some resources here</a> to help you get started with these concepts.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/run-ai-gpu-orchestration-architecture-2-1612435001011.jpg" alt="run ai gpu orchestration architecture 2"></p> <p>As shown in the picture above, Run:AI GPU orchestration solution creates a virtualization and acceleration layer over GPU resources that manage granular scheduling, prioritization, and allocation of compute power for the HPE Ezmeral Container Platform. Run:AI provides a dedicated batch scheduler, running on top of HPE Ezmeral Container Platform to manage GPU-based workloads. Find out more about the Run:AI GPU orchestration solution running on top of HPE Ezmeral Container Platform, including how you can get a free trial of the solution, by visiting the [HPE Ezmeral Marketplace] (<a href="https://www.hpe.com/us/en/software/marketplace/runai.html">https://www.hpe.com/us/en/software/marketplace/runai.html</a>).</p> <p>Keep coming back to the <a href="/blog">HPE DEV blog</a> site for more interesting articles and tutorials on related topics.</p>https://developer.hpe.com/2021-February-03/https://developer.hpe.com/2021-February-03/Wed, 03 Feb 2021 06:00:00 GMThttps://developer.hpe.com/devsecops-what-it-is-and-where-its-going/https://developer.hpe.com/devsecops-what-it-is-and-where-its-going/Tue, 02 Feb 2021 11:03:18 GMT<p><strong>Editor’s note: This article was originally posted on HPE Enterprise.nxt on February 2, 2021</strong></p> <hr> <p>Experts basically agree on the definition of DevSecOps, but there isn't a full consensus on how to do it and where it is leading.</p> <p>While the practices that drive DevOps are more than a decade old, integrating security with DevOps, often termed <a href="https://resources.whitesourcesoftware.com/blog-whitesource/devsecops">DevSecOps</a>, remains a challenge. What does DevSecOps mean within an organization? What does successful DevSecOps look like? And can a company improve its DevSecOps practices so security can scale as the organization scales?</p> <p>Like DevOps itself, DevSecOps developed organically in the community of developers. It didn't come down as standards from on high; rather, developers shared ideas about best practices. Consequently, there isn't one agreed-upon definition of what DevSecOps is, how one implements it, and where current DevSecOps trends are going.</p> <p>In 2020, at the KubeCon conference, a media panel comprised of technical leaders in the DevSecOps space sought to tackle the question on how to implement DevSecOps in the enterprise. Panel members illustrated where there was and was not consensus.</p> <p><img src="/img/devsecops_-what-it-is-and-where-it-s-going.png" alt="Block Text" title="Block text"></p> <h3>DevSecOps: Building apps better for the bottom line</h3> <p>In defining DevSecOps, there was considerable agreement among the panelists—with some caveats. For instance, Emily Fox, DevOps security lead at the U.S. Department of Defense, said DevSecOps is about how DevOps organizations move to a security-first risk posture. What that means, she said, is building services with self-sustaining microperimeters, vetted by "trust but verify" defense-in-depth strategies and interactions with the services around them, within their workload environments.</p> <p>Nicolas Chaillan, U.S. Air Force chief software officer, agreed, except for security being first. "I want to be careful with 'first,'" Chaillan said. "We don't build software just to be secured. Security needs to be baked in and security needs to be a central piece of the process, but I don't think it's first. I think it's a continuous cycle."</p> <p>He added, "I certainly agree with continuous monitoring and zero trust enforcement. For us, that means behavior detection and monitoring zero trust enforcement down to the function layer—really reducing the attack surface and continuously monitoring."</p> <p>According to Peter Bosch, distinguished engineer at Cisco Systems, when it comes to successful DevSecOps, security team members must be adequately integrated with development teams. As a developer "long before DevSecOps was a term," Bosch recalled being stuck writing code and then waiting for three months for a firewall rule to be opened so it could function properly. By integrating those teams, making sure the security team becomes part of the application delivery process, the speed of application development goes up and the robustness of applications increases, he said.</p> <p>"You will not run into your <a href="https://owasp.org/www-project-top-ten/">OWASP Top 10 security issues</a> as frequently," Bosch added. "And you will start making applications that actually contribute better to the bottom line."</p> <p>Sunil James, senior director at Hewlett Packard Enterprise, also saw DevSecOps as a way to move development efforts forward securely and quickly. When done correctly, DevSecOps enables enterprises to move at "cloud speed," he said.</p> <p>"The impetus for this [DevOps] movement is cloud. And cloud materially drives the cadence of our ability to deliver," James noted. "While not many organizations are delivering at the scale and the pace of a Google or a Netflix, they'd like to. They'd like to get to that point because they can then bring value to their customers faster and faster."</p> <p>And that's what DevSecOps provides: In addition to a continuous way to deliver secure development and security policies deeper into the application development and operations, DevSecOps also helps "create the consistency and speed with which you can actually offer these capabilities," James said.</p> <h3>DevSecOps and regulatory compliance</h3> <p>DevSecOps isn't just about software code security, but also about maintaining compliance with various industry and government regulations. As Fox explained, teams must contend with many different types of compliance, including legal compliance, compliance that mandates how data is handled, security policy, and control compliance. Each compliance requirement, she said, helps ensure that organizations can secure their mission, systems, applications, and datasets.</p> <p>Still, "we have a lot of challenges in this space because cloud native is continually evolving and security is constantly playing catch up," Fox said. "Therefore, the security tooling to allow us to do compliance in a secure, automated, easy-to-understand fashion is still behind and leaves a lot of room for improvement."</p> <p>Fortunately, better tools are on the way. For one, there is the <a href="https://pages.nist.gov/OSCAL/">Open Security Controls Assessment Language (OSCAL) from NIST</a>, which aims to provide a standardized and automated way to publish, implement, and assess security controls. Chaillan noted that the U.S. Department of Defense is partnering with NIST to implement OSCAL development in its environment and to help automate some of the department's security controls.</p> <p>Still, the sheer size and complexity of the Defense Department's environment make it challenging. "If you try to reach a complete analysis of your stack every time you make a change, it's very difficult," Chaillan said.</p> <p>However, by slicing the technology stack into layers, one can automate the mapping of <a href="https://nvd.nist.gov/800-53">NIST Special Publication 800-53</a> controls with OSCAL. "I think there is something [to this approach]," he said. "Of course, it's very nascent, and there are very few tools that support OSCAL today, so we need companies to bring that capability as a tool."</p> <h3>Automating security and compliance with 'policy first'</h3> <p>Another way to help DevOps teams succeed at integrating security more tightly into their workflow is to place security controls directly within development environments and continuous pipelines as part of a so-called policy-first initiative. As Fox explained, when developers attempt to commit a project, such in-line controls can help ensure, for example, that secrets aren't part of the package.</p> <p>One of the most straightforward ways to potentially stop that from happening is with pre-commit hooks. Fox said she expects to see policies like these baked into developer tools. Such checks are then reinforced with additional tests throughout the development pipeline to ensure that workloads being deployed remain compliant.</p> <p>In addition to NIST's work on the <a href="https://www.openpolicyagent.org/">Open Policy Agent</a>, other standards have been developed to help organizations enable a policy-first footing. One such set of standards is <a href="https://developer.hpe.com/platform/spiffe-and-spire-projects/home">SPIFFE (Secure Production Identity Framework For Everyone) and SPIRE</a> (a software implementation of the SPIFFE API). "It's been happening around SPIFFE and other standards," James said.</p> <p>Essentially, SPIFFE is a workload API that creates trust between workloads and system actions. Because SPIFFE is an API and eliminates manual key generation and distribution, authentication such as Kerberos and OAuth can be fully automated within cloud workloads. SPIRE is a software implementation of the SPIFFE API and can be integrated with call providers, middleware layers, hardware trust mechanisms, and more.</p> <p>According to James, the approaches to policy-first DevOps are moving toward allow/deny lists that help create scalable, automated mechanisms that can determine what something is and what it will be allowed to do and then create a centralized framework to manage the policy over time.</p> <p>James added that the policy-first mindset isn't optional in today's fast-paced environment. "When you're looking at organizations scaling their use of microservices—whether they are going from five, to 10, to 100, to whatever the ultimate number might be—they're not going to be able to get their hands around [management challenges] unless they have a policy-first mindset," he said.</p> <p>"We've seen these patterns before DevSecOps, though. We've seen policy-first in other domains, such as application security and information security. And I think there are a lot of lessons learned that are going to find their way back into the cloud-native landscape."</p> <h3>Lessons for leaders</h3> <ul> <li>When done right, DevSecOps enables enterprises to make secure software at "cloud speed."</li> <li>The security policy-first mindset isn't optional in today's fast-paced environment.</li> <li>DevSecOps isn't just about software code security but also about maintaining compliance with various industry and government regulations.</li> </ul> <br /> <p><u><strong>About the author:</strong></u></p> <p>George V. Hulme is an award-winning journalist and internationally recognized information security and business technology writer. He has covered business, technology, and IT security topics for more than 20 years. His work has appeared in CSOOnline, ComputerWorld, InformationWeek, and dozens of other technology publications. He is also a founding editor at DevOps.com</p>https://developer.hpe.com/how-to-integrate-custom-data-sources-into-apache-spark/https://developer.hpe.com/how-to-integrate-custom-data-sources-into-apache-spark/Fri, 29 Jan 2021 05:41:49 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-05-10T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>Streaming data is a hot topic these days, and Apache Spark is an excellent framework for streaming. In this blog post, I'll show you how to integrate custom data sources into Spark.</p> <p>Spark Streaming provides the ability to stream from a variety of sources while using the same concise API for accessing data streams, performing SQL queries, or creating machine learning algorithms. These abilities make Spark a preferable framework for streaming (or any type of workflow) applications, since we can use all aspects of the framework.</p> <p>The challenge is figuring out how to integrate custom data sources into Spark so we can leverage its power without needing to change to more standard sources. It might seem logical to change, but in some cases it is just not possible or convenient to do so.</p> <h2>Streaming Custom Receivers</h2> <p>Spark offers different extension points, as we could see when we extended the Data Source API for our example here in order to integrate our custom data store into Spark SQL.</p> <p>In this example, we are going to do the same, but we are also going to extend the streaming API so we can stream from <strong>anywhere</strong>.</p> <p>In order to implement our custom receiver, we need to extend the Receiver[A] class. Note that it has type annotation, so we can enforce type safety on our DStream from the streaming client side point of view.</p> <p>We are going to use this custom receiver to stream orders that one of our applications sent over a socket.</p> <p>The structure of the data traveling through the network looks like this:</p> <pre><code>1 5 1 1 2 2 1 1 2 1 1 4 1 1 2 2 1 2 2 </code></pre> <p>We first receive the order ID and the total amount of the order, and then we receive the line items of the order. The first value is the item ID, the second is the order ID, (which matches the order ID value) and then the cost of the item. In this example, we have two orders. The first one has four items and the second has only one.</p> <p>The idea is to hide all of this from our Spark application, so what it receives on the DStream is a complete order defined on a stream as follows:</p> <pre><code class="language-scala">val orderStream: DStream[Order] = ..... </code></pre> <p>At the same time, we are also using the receiver to stream our custom streaming source. Even though it sends the data over a socket, it will be quite complicated to use the standard socket stream from Spark, since we will not be able to control how the data is coming in. In addition, we have the problem of conforming orders on the application itself. This could be very complicated, since, once we are in the app space, we are running in parallel, and it is hard to sync all of this incoming data. However, in the receiver space it is easy to create orders from the raw input text.</p> <p>Let’s take a look at what our initial implementation looks like.</p> <pre><code class="language-scala">case class Order(id: Int, total: Int, items: List[Item] = null) case class Item(id: Int, cost: Int) class OrderReceiver(host: String, port: Int) extends Receiver[Order](StorageLevel.MEMORY_ONLY) { override def onStart(): Unit = { println("starting...") val thread = new Thread("Receiver") { override def run() {receive() } } thread.start() } override def onStop(): Unit = stop("I am done") def receive() = .... } </code></pre> <p>Our OrderReceiver extends Receiver[Order] which allows us to store an Order (type annotated) inside Spark. We also need to implement the onStart() and onStop() methods. Note that onStart() creates a thread so it is non-blocking, which is very important for proper behavior.</p> <p>Now, let’s take a look at the receive method. This is where the magic really happens.</p> <pre><code class="language-scala">def receive() = { val socket = new Socket(host, port) var currentOrder: Order = null var currentItems: List[Item] = null val reader = new BufferedReader(new InputStreamReader (socket.getInputStream(), "UTF-8")) while (!isStopped()) { var userInput = reader.readLine() if (userInput == null) stop("Stream has ended") else { val parts = userInput.split(" ") if (parts.length == 2) { if (currentOrder != null) { store(Order(currentOrder.id, currentOrder.total, currentItems)) } currentOrder = Order(parts(0).toInt, parts(1).toInt) currentItems = List[Item]() } else { currentItems = Item(parts(0).toInt, parts(1).toInt) :: currentItems } } } } </code></pre> <p>Here, we create a socket and point it to our source. Then we simply start reading from it until a stop command has been dispatched or our socket has no more data on it. Note that we are reading the same structure we have defined previously (how our data is being sent). Once we have completely read an Order, we call store(…) so it gets saved into Spark.</p> <p>There is nothing left to do here but to use our receiver in our application, which looks like this:</p> <pre><code class="language-scala">val config = new SparkConf().setAppName("streaming") val sc = new SparkContext(config) val ssc = new StreamingContext(sc, Seconds(5)) val stream: DStream[Order] = ssc.receiverStream(new OrderReceiver(port)) </code></pre> <p>Note how we have created the stream using our custom OrderReceiver (the val stream has been annotated only for clarity but it is not required). From now on, we use the stream (DString[Order]) as any other stream we have used in any other application.</p> <pre><code class="language-scala">stream.foreachRDD { rdd => rdd.foreach(order => { println(order.id)) order.items.foreach(println) } } </code></pre> <h2>Summary</h2> <p>Spark Streaming comes in very handy when processing sources that generate endless data. You can use the same API that you use for Spark SQL and other components in the system, but it is also flexible enough to be extended to meet your particular needs.</p>https://developer.hpe.com/containers-vs-vms-a-5-minute-guide-to-understanding-their-differences/https://developer.hpe.com/containers-vs-vms-a-5-minute-guide-to-understanding-their-differences/Fri, 29 Jan 2021 05:33:44 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suzy Visvanathan", "publish": "2018-05-16T10:45:00.000", "tags": "mapr-platform" </code></pre> <hr> <p>Almost every organization is investing in or thinking of investing in containers. Containers are transitioning from being a hype to becoming a tangible entity in which applications are deployed, but that doesn’t mean virtual machines are suddenly out of date or not needed. While there are many articles highlighting the technical comparisons between how a container and VM image differ, articulating the advantages of one over the other for one’s business needs, and when to use which, requires a closer look.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/containers-vs-vms-wide-1611898725434.jpg" alt="Containers vs. VMs"></p> <h2>What’s in the package</h2> <p>Containers take only what the application needs and share system resources like OS, CPU, and memory, thereby making them easier to deploy. VMs, on the other hand, take up a lot of system resources within each image, which translates to a lot of memory and CPU cycles. Because of their agile nature, containers are the deployment of choice in development and testing environments.</p> <p>Containers tend to run on one operating system, whereas the virtual hypervisor can run on any operating system. If you have one or a few consistent sets of applications on a single operating system, consider containers. If you have diverse applications with varying operating systems, consider VMs.</p> <h2>It’s all about the money</h2> <p>Since containers don’t package system resources as much as VMs, you can run at least twice (or more) the number of applications on the same server with containers than if you were to run them with VMs. This advantage maximizes resource usage and brings down operating costs. Agile development and testing speeds up time to market with containers, more so than if done with VMs.</p> <p>Both container and VM sprawl are a real problem that administrators face; however, because of the elasticity and portability of containers, they can easily be doubled or tripled to run the same set of applications, as opposed to VMs. So, while you can package more applications on a single host server with containers, running more containers than you need will end up consuming more resources.</p> <p>Put simply, if you are looking to develop applications or run a single or a handful of applications in multiple instances and resource footprint is a concern, consider containers. If you are looking to run multiple applications and resource footprint can be fluid, consider VMs.</p> <h2>About that security...</h2> <p>Security, by and large, has been the single biggest problem around containers. Containers, by their very nature of sharing OS, require root access, which makes the data vulnerable and at risk for unauthorized access. While there are several workarounds for this issue, they are quite lengthy and need to be thought out in detail. VMs, on the other hand, have a very robust, rich set of security services that make them attractive for sensitive data and for production environments. VMs also have a very mature ecosystem in terms of network, storage, data protection, and recovery that can make them better for production environments.</p> <h2>Forecast is mostly cloudy</h2> <p>Cloud solutions have made huge strides in the services they offer. However, the cost of using the cloud is not always significantly lower – as one might expect. The elasticity of containers allows organizations to create containers on demand and tear them down when done. Scaling up and down your services on cloud by spawning new containers is easier and more cost-effective than it is with VMs. Containers in cloud also allow you to use only the minimum cloud resources you need for your service, thereby keeping your subscription costs down.</p> <p>To keep it simple, if you have a heavy development, test, or integration environment, switch to containers. If you have multiple applications with varying characteristics requiring a secure environment, remain on VMs. If you are looking to deploy in the cloud or offer services in the cloud, standardizing the deployment in containers will be a good idea. Eventually, it will be prudent to envision and plan for both containers and VMs to coexist in both data centers and in the cloud, since each has its own benefits and challenges.</p>https://developer.hpe.com/kafka-connect-and-kafka-rest-api-on-mapr-streaming-just-became-a-whole-l/https://developer.hpe.com/kafka-connect-and-kafka-rest-api-on-mapr-streaming-just-became-a-whole-l/Fri, 29 Jan 2021 05:29:20 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ankur Desai", "publish": "2016-12-09T06:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>In my previous blog post <a href="/blog/LOV2B97WzAiAzmYlY17y/real-time-event-streaming-what-are-your-options">Real-Time Event Streaming: What Are Your Options?</a>, I explained the three major components of a streaming architecture. Most streaming architectures have three major components – producers, a streaming system, and consumers. Producers (such as Apache Flume) publish event data into a streaming system after collecting it from the data source, transforming it into the desired format, and optionally filtering, aggregating, and enriching it. The streaming or messaging system (such as Apache Kafka or MapR Event Store) takes the data published by the producers, persists it, and reliably delivers it to consumers. Consumers are typically stream processing engines (such as Apache Spark) that subscribe to data from streams and manipulate or analyze that data to look for alerts and insights. Furthermore, once the data is processed, it may need to be persisted in a database or a file for future use by downstream applications.</p> <p>The following diagram illustrates the typical streaming architecture:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture1-1611898227918.png" alt=""></p> <p>However, as streaming becomes more pervasive, we are looking to simplify this architecture and, at the same time, make it more agile. Enter Kafka Connect and the Kafka REST API. The following diagram illustrates a new, simple, agile way of setting up your streaming:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture2-1611898241971.png" alt=""></p> <h2>Kafka Connect: Easily connect common data systems with Kafka</h2> <p>Kafka Connect provides pre-built connectors that allow legacy data stores (such as databases and data warehouses) and modern data stores (such as HDFS) to connect with Kafka. This connection eliminates the need of building a custom “producer” or “consumer” application to help these data systems to publish/subscribe to Kafka. It also eliminates the need for a third party data collector that provides connectors to these data stores.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture3-1611898250133.png" alt=""></p> <p>Kafka Connect provides a convenient, reliable connection to the most common data stores. It helps to ingest data into Kafka as well as push data from Kafka into the most commonly used data systems. Moreover, to eliminate the need of custom producer apps, it allows pull-based ingestion of data, supporting sources that don't know how to push. Similarly, to eliminate custom consumer apps, it allows push-based export of data from Kafka, supporting data systems that don't know how to pull data from Kafka. As Kafka Connect continues to mature, more connectors will be created, opening up a large range of sources and sinks that can connect to Kafka out of the box.</p> <h2>Kafka REST Proxy: Connect with Kafka using HTTP</h2> <p>New age data sources such as sensors, mobile devices, etc., know how to communicate using HTTP. However, they often do not have enough computing resources to run a Kafka producer application and a Kafka client. This deficiency is why the Kafka REST API is a game changer. It allows these devices to publish/subscribe to Kafka topics easily, which makes the architecture much more agile. Any device that can communicate using HTTP can now communicate directly with Kafka. This development has massive implications in simplifying IoT architectures. Any car, thermostat, machine sensor, etc., can now communicate directly with Kafka.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture4-1611898257677.png" alt=""></p> <p>The Kafka REST API eliminates intermediate data collectors and simplifies the architecture by directly connecting the data sources with Kafka. Any programming language in any runtime environment can now connect with Kafka using HTTP. This ability gives developers the freedom to use the development framework of their choice and connect with Kafka using simple REST APIs, which reduces the time-to-market for streaming applications.</p> <h2>MapR Data Platform: Further simplifying the architecture</h2> <p>The MapR Platform further simplifies the streaming architecture by providing event streaming, stream processing, and persistence (both database and files) on one single platform, in one system, in one cluster. You can connect the data sources with MapR Event Store - which is a more secure, reliable, and performant replacement for Kafka - using the Kafka REST API or Kafka Connect. All the components of your streaming architecture will be available on MapR, within one platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/picture5-1611898264954.png" alt=""></p> <p>MapR Event Store uses the same APIs as Kafka (0.9), which means that applications built with Kafka as the messaging system can be easily ported over to MapR Event Store and vice versa. Without the converged platform, event streaming, stream processing, and persistence would run as separate systems that would need to be connected. Connected systems require cross-cluster data movement, which introduces additional latency. They also require more hardware, since resources cannot be shared across siloed systems and have higher administration cost. The MapR Data Platform eliminates these problems by providing one single system for all components (except the original data sources) of the streaming architecture.</p>https://developer.hpe.com/real-time-event-streaming-what-are-your-options/https://developer.hpe.com/real-time-event-streaming-what-are-your-options/Fri, 29 Jan 2021 05:22:57 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ankur Desai", "publish": "2016-04-14T07:00:00.000Z", "tags": "open-source" </code></pre> <hr> <p>With the Internet of Things expected to bring billions of devices online, a lot of people are excited about the potential value of event streaming, that is, ingesting and analyzing lots of real-time data for immediate decision-making. But streaming also introduces new concepts and components that need a closer look. This blog post is intended to provide an introduction to the components of a typical streaming architecture and various options available at each stage.</p> <h2>Three Components of a Streaming Architecture</h2> <p>Most streaming architectures have three major components – producers, a streaming system, and consumers.</p> <p><strong>A producer</strong> is a software-based system that is connected to the data source. Producers publish event data into a streaming system after collecting it from the data source, transforming it into the desired format, and optionally filtering, aggregating, and enriching it.</p> <p><strong>The streaming system</strong> takes the data published by the producers, persists it, and reliably delivers it to consumers.</p> <p><strong>Consumers</strong> are typically stream processing engines that subscribe to data from streams and manipulate or analyze that data to look for alerts and insights. There are lots of options to choose from, and more are on the way. Let’s look at what your options are for each stage.</p> <h3><strong>Stage 1: Producers</strong></h3> <p>Data producers collect the data from data sources, convert it to the desired format, and publish the data into streaming platforms such as <a target='\_blank' href='http://kafka.apache.org/'>Apache Kafka</a> and MapR Event Store. Apache Flume is commonly used as a producer to Kafka. StreamSets is an up-and-coming data collector that may be worth a look.</p> <p><strong><a target='\_blank' href='https://flume.apache.org/'>Apache Flume</a></strong> is a distributed system for efficiently collecting, aggregating, and moving large amounts of data. Flume has a source and sink architecture. A Flume source collects the event data from the data sources. A Flume sink puts the event into an external repository, which is often a streaming system like Apache Kafka or MapR Event Store.</p> <p><strong><a target='\_blank' href='https://streamsets.com/product/'>StreamSets Data Collector</a></strong> is open source software for the development and operation of complex data flows. It provides a graphical IDE for building ingest pipelines. StreamSets can help you connect with Kafka without writing a single line of code. StreamSets Data Collector includes out-of-the-box connectors for Kafka and many other sources and destinations.</p> <h3><strong>Stage 2: Streaming System</strong></h3> <p>Two event transport systems that can easily scale to deliver billions of events per second are Apache Kafka and MapR Event Store. The ability to linearly scale to deliver billions of events per second differentiates Kafka and MapR Event Store from traditional messaging queues like <a target='\_blank' href='http://www.tibco.com/products/automation/enterprise-messaging/enterprise-message-service'>Tibco EMS</a> and <a target='\_blank' href='http://www-03.ibm.com/software/products/en/ibm-mq'>IBM MQ</a>. Kafka and MapR Event Store both use a publish-subscribe model in which the data producer is the publisher and the data consumer is the subscriber.</p> <p><strong>Apache Kafka</strong> is great at handling large volumes of data. You can set up a cluster as a data backbone, which gives you great scalability. You can then easily expand the cluster as needed without downtime. Kafka also stores messages on disk and replicates them within the cluster to reduce the risk of data loss when you encounter a hardware failure.</p> <p><strong>MapR Event Store for Apache Kafka</strong> is like Kafka – in fact, it uses <a target='\_blank' href='http://kafka.apache.org/documentation.html'>the Kafka 0.9 API</a> – but it has certain enterprise features that provide additional support for very large and geographically diverse networks where data integrity is essential. MapR Event Store is integrated with the MapR Data Platform, which combines file storage, database services, and processing frameworks in a single cluster. That means batch, interactive, and stream processing engines all have direct access to event streams, which reduces data movement and ensures consistency.</p> <h3><strong>Stage 3: Consumers (Processing)</strong></h3> <p>MapR Event Store and Kafka can deliver data from a wide variety of sources, at IoT scale. It’s then up to the processing engines to do something with it. Four important engines to know about include Apache Spark Streaming, Apache Flink, Apache Storm, and Apache Apex.</p> <p><strong>Spark Streaming</strong> is a built-in component of Apache Spark. Spark Streaming can consume event streams from MapR Event Store, Kafka, and many other systems. By being a built-in component of Spark, Spark Streaming runs in-memory, and allows you to run ad-hoc queries on stream data. Spark Streaming can be more accurately described as “micro-batching,” or processing small amounts of batch information in quick bursts.</p> <p><strong><a target='\_blank' href='http://storm.apache.org/'>Apache Storm</a></strong> is another popular event processing engine. Unlike Spark, Storm is a pure real-time event-based analytics engine, which makes it most useful in situations in which each event needs to be processed instantaneously. Storm actually processes each event as soon as it is delivered.</p> <p><strong><a target='\_blank' href='https://flink.apache.org/'>Apache Flink</a></strong> works in-memory and is notable for its speed and scalability. Similar to Storm, it is a pure real-time event-based processing engine. The differences between the two are quite technical, having to do with such things as the way each ensures data reliability, the programming languages they support, and the capabilities of their respective APIs. Flink supports the Apache Storm API, to make the transition from Storm easy for developers familiar with Storm.</p> <p><strong><a target='\_blank' href='http://apex.incubator.apache.org/'>Apache Apex</a></strong> is a YARN-native platform that unifies stream and batch processing. It lowers the expertise required to write big data applications by providing a simple API that enables users to write or reuse generic Java code.</p> <p>You can see that the enthusiasm over real-time processing is being met with a host of technologies. And the landscape is constantly evolving.</p>https://developer.hpe.com/scaling-with-kafka-common-challenges-solved/https://developer.hpe.com/scaling-with-kafka-common-challenges-solved/Fri, 29 Jan 2021 05:13:45 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Will Ochandarena", "publish": "2016-05-05T07:00:00.000Z", "tags": "streaming" </code></pre> <hr> <p>I attended several Kafka Summit sessions held in San Francisco given by Kafka users and talked to many more at the booth and in the hallway. The general consensus is the Kafka model and API is well suited to building horizontally-scalable, asynchronous data pipelines for data integration and stream processing. That said, the companies that are operating these systems at scale – billions of events per day or multiple data centers – described a consistent set of challenges.</p> <p>We launched MapR Event Store, a publish-subscribe event streaming system built on top of the MapR platform, exposing the Kafka API. By leveraging the strong MapR foundation that supports our distributed file and object store (MapR XD) and database (MapR Database), both of which effortlessly scale to petabytes of data and thousands of nodes across multiple data centers, MapR Event Store inherently avoids several of the challenges described by the companies operating it at scale. In this blog, I’d like to highlight a few of these challenges and how MapR Event Store overcomes them.</p> <p><strong>Topic Balancing</strong></p> <p>The most universal pain point I heard had to do with how Kafka balances topic partitions between cluster nodes. Because Kafka assumes all partitions are equal in terms of size and throughput, a common occurrence is for multiple “heavy” partitions to be placed on the same node, resulting in hot spotting and storage imbalances. To overcome this, these companies devote a lot of resources to monitoring and manually intervene each time an issue is found to migrate partitions between nodes.  </p> <p>Rather than pin partitions to a single node, MapR Event Store splits partitions into smaller linked objects, called “partitionlets”, that are spread among the nodes in the cluster. As data is written to the cluster, the active partitionlets (those handling new data) are dynamically balanced according to load, minimizing hotspotting.</p> <p><strong>Infinite Topic Persistence</strong></p> <p>Another common issue that was talked about was handling long-term persistence of streaming data. As mentioned above, Kafka partitions are pinned to a single node, meaning they can’t outgrow the storage capacity of that node.  Because of this, a common design pattern is to shovel streaming data from Kafka into HDFS for long-term persistence, also known as the Lamba architecture.  This creates huge complications around reprocessing of old data (either for new use cases or fixing bugs on existing streaming apps), as it forces companies to write two versions of each app – one for processing from Kafka, the other from HDFS.  </p> <p>The MapR partitionlet approach described above also eliminates this issue, as it allows all historical data to be stored in a single system – MapR Event Store. This allows streaming apps, new or existing, to simply “scrollback to 0” and reprocess months or even years of historical data.</p> <p><strong>Global Replication</strong></p> <p>LinkedIn gave a presentation to a packed room called “More Clusters, More Problems” on designing for multi-datacenter. They listed several design challenges, to name a few -</p> <ul> <li>Establishing tiered cluster types (local and aggregate) to avoid forming topology loops that infinitely replicate data.</li> <li>Ensuring topic names are globally unique due to lack of namespacing.</li> <li>Ensuring identical partition configuration on both ends to prevent re-ordering.</li> <li>No ability to handle failover of applications in a disaster recovery scenario, as message offsets aren’t synchronized between clusters.</li> </ul> <p>By building global replication into the platform, MapR Event Store overcomes all of the challenges above and more.</p>https://developer.hpe.com/exploring-data-fabric-and-containers-in-hpe-devs-new-munch-learn-monthly/https://developer.hpe.com/exploring-data-fabric-and-containers-in-hpe-devs-new-munch-learn-monthly/Thu, 28 Jan 2021 18:22:24 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/munch-learn-feb-2021-1611858129501.jpg" alt="munch learn feb 2021"></p> <p>HPE DEV Munch &#x26; Learn technical talks are designed to provide attendees with an opportunity to learn more about today’s most popular technologies. These free, 60-minute informative meetups are sponsored by the HPE DEV Community and provide the unique opportunity to connect with some of the industry’s leading technologists.</p> <h3>January’s inaugural talk on data fabric with Ted Dunning and Ellen Friedman</h3> <p>In the January 27th Munch &#x26; Learn session, over 160 technologists attended to listen to Ellen Friedman and Ted Dunning explore the impact of a unifying data fabric and how it works. Ellen moderated the session, starting with an invitation to make these informal meetups fun, interactive, and informative. She introduced Ted Dunning, HPE Ezmeral Data Fabric CTO, who started with a business-level look at three different use cases and then quickly moved to a deeper technical level to examine how data infrastructure played a role in each one.</p> <p>Ted showed a variety of processes that are handled efficiently by the data platform rather than having to be coded over and over again into applications. The examples included:</p> <ul> <li>Event stream replication from many edge sources to the core data center</li> <li>How data infrastructure handles massive scale of data mirrored from edge to core in the case of autonomous car manufacture</li> <li>How a unifying data infrastructure supports true multitenancy for a large customer</li> </ul> <p>The data infrastructure used is the HPE Ezmeral Data Fabric, a software solution based on technology from the acquisition of MapR Technologies by HPE and now part of the HPE Ezmeral Software Portfolio. This ability to run very different applications – including AI and analytics, modern and legacy, containerized and non-containerized – together on the same data system is a central theme of their free, newly published O’Reilly ebook: <em><a href="https://www.hpe.com/us/en/resources/software/ai-and-analytics-systems.html">AI and Analytics at Scale: Lessons from Real-World Production Systems</a></em>.</p> <p>During the talk, Ted acknowledged that people may be accepting certain trade-offs between scale, resilience, performance, flexibility, or cost that aren’t actually necessary. He pointed out that, when you release yourself from the limitations imposed by some data infrastructure technologies, you can avoid these tradeoffs, and then proceeded to dive into technical detail on how the HPE Ezmeral Data Fabric makes this possible.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ellen-munch-and-learn-2-1612197191552.jpg" alt="ellen munch and learn 2"></p> <p>While listening to the session, attendees participated in polls, had the opportunity to ask questions. In keeping with the Munch and Learn theme several of the attendees had an opportunity to share pictures of their favorite “munchies” on the accompanying Slack channel.</p> <h3>February talk on containers with Tom Phelan and Nigel Poulton</h3> <p>During the next session, February 24th, Nigel Poulton hosted Tom Phelan, HPE Fellow and CTO of the HPE Ezmeral Runtime Enterprise, who discussed container architectures and how they can leverage the Kubernetes Container orchestrator to deploy and manage stateful, as well as microservice-based, applications.</p> <p>As part of the discussion, Tom described how containers require less system resources than traditional virtual machine environments, allowing applications to be deployed more easily and run on different operating systems and hardware platforms. Tom also explained how the HPE Ezmeral Machine Learning Operations package helps support collaboration between data scientists.</p> <p>To view what’s up for our next Munch &#x26; Learn session, check out our <a href="/blog/munch-and-learn">schedule</a>.</p>https://developer.hpe.com/top-trends-machine-learning-microservices-containers-kubernetes-cloud-to/https://developer.hpe.com/top-trends-machine-learning-microservices-containers-kubernetes-cloud-to/Fri, 22 Jan 2021 06:42:26 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-02-28T12:00:00.000", "tags": "use-cases" </code></pre> <hr> <p>Developers, data scientists, and IT operations are working together to build intelligent apps with new technologies and architectures because of the flexibility, speed of delivery, and maintainability that they make possible. This post will go over some top trending technologies, such as machine learning, containers, Kubernetes, event streams (Kafka API), DataOps, and cloud to edge computing, which are driving this revolution.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/industries-1611298009552.png" alt=""></p> <p><strong>AI, Machine Learning, Deep Learning</strong></p> <p>Predictive machine learning uses algorithms to find patterns in data and then uses a model that recognizes those patterns to make predictions on new data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/predictive-ml-1611298021253.png" alt=""></p> <p>Why is this so hot? Analytical technology has changed dramatically over the last decade, with more powerful and less expensive distributed computing across commodity servers, streaming analytics, and improved machine learning technologies, enabling companies to store and analyze both far more data and many different types of it. According to <a href="https://www.gartner.com/newsroom/id/3812063">Gartner</a>, over the next few years, virtually every app, application, and service will incorporate some level of AI or machine learning.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/ml-examples-1611298030328.png" alt=""></p> <p><strong>Microservices</strong></p> <p>The <a href="https://martinfowler.com/articles/microservices.html">microservice architectural style</a> is an approach to developing an application as a suite of small, independently deployable services built around specific business capabilities.</p> <p>A monolithic application puts all of its functionality into a single process; scaling requires replicating the whole application, which has limitations. With microservices, functionality is put into separate services, allowing these services to be distributed and replicated across servers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/microservices-1611298039137.png" alt=""></p> <p><a href="http://ostatic.com/blog/q-a-maprs-jack-norris-on-the-impact-of-microservices">A microservices approach is well-aligned to a typical big data deployment</a>. You can gain modularity, extensive parallelism, and cost-effective scaling by deploying services across many commodity hardware servers. Microservices modularity facilitates independent updates/deployments and helps to avoid single points of failure, which can help prevent large-scale outages.</p> <p><strong>Event-Driven Microservices</strong></p> <p>A common architecture pattern combined with microservices is event sourcing using an append-only publish subscribe event stream such as MapR Event Streams (which provides a Kafka API).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/microservices-with-cdp-1611298050238.png" alt=""></p> <p>MapR Event Store provides high performance messaging, which can scale to very high throughput levels, easily delivering millions of messages per second on modest hardware. The publish/subscribe Kafka API provides decoupled communications, wherein producers don't know who subscribes, and consumers don't know who publishes, making it easy to add new listeners or new publishers without disrupting existing processes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/kafka-api-integration-1611298061346.png" alt=""></p> <p>When you combine these messaging capabilities with the simple concept of microservices, you can greatly enhance the agility with which you build, deploy, and maintain complex data pipelines. Pipelines are constructed by simply chaining together multiple microservices, each of which listens for the arrival of some data, performs its designated task, and optionally publishes its own messages to a topic.</p> <p>Take, for example, an online shopping application's item rating functionality, as shown in the image below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/online-shopping-application-1611298073294.png" alt=""></p> <p>This could be decomposed into the following microservices:</p> <ul> <li>a service publishes "Rate Item" events to a Topic</li> <li>a service reads from the stream and persists a materialized view of the ratings in a NoSQL document datastore</li> <li>a browse item ratings service reads from the NoSQL document datastore</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/rating-functionality-decomposed-1611298085852.png" alt=""></p> <p>With event-driven microservices, new functionality can easily be added by deploying new services; for example recommendations, predictive services, and fraud detection service, as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/deploying-new-services-1611298100670.png" alt=""></p> <p>Development teams can deploy new services or service upgrades more frequently and with less risk, because the production version does not need to be taken offline. Both versions of the service simply run in parallel, consuming new data as it arrives and producing multiple versions of output. Both output streams can be monitored over time; the older version can be decommissioned when it ceases to be useful.</p> <p><strong>Event Streams and Machine Learning Logistics</strong></p> <p>Combining event streams with machine learning can handle the logistics of machine learning in a flexible way by:</p> <ul> <li>Making input and output data available to independent consumers</li> <li>Managing and evaluating multiple models and easily deploying new models</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/event-streams-and-ml-logistics-1611298114104.png" alt=""></p> <p>Architectures for these types of applications are discussed in more detail in the <a href="https://www.scribd.com/document/435442431/Spark2018eBook-pdf">eBook</a> <em>Machine Learning Logistics</em>, <em>Streaming Architecture</em>, and <em>Microservices and Containers</em>.</p> <p><strong>Containers</strong></p> <p>A <a href="https://www.docker.com/what-container">container image</a> packages an entire runtime environment: an application, plus all its dependencies, libraries and other binaries, and configuration files needed to execute the application. Compared to virtual machines, containers have similar resources and isolation benefits, but are more lightweight, because containers virtualize the operating system instead of the hardware. Containers are more portable and efficient; they take up less space, use far fewer system resources, and can be spun up in seconds.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/containers-1611298125788.png" alt=""></p> <p>To learn more about containers, click <a href="https://www.docker.com/what-container">here</a></p> <p><strong>DevOps and Containers</strong></p> <p>Similar to how the agile software development movement broke down the handoff between business requirements, development, and testing, <a href="https://en.wikipedia.org/wiki/DevOps">DevOps</a> breaks down silos between developers and operations with a collaborative process.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/devops-and-containers-1611298138209.png" alt=""></p> <p>To learn more about DevOps, click [here] (<a href="https://en.wikipedia.org/wiki/DevOps">https://en.wikipedia.org/wiki/DevOps</a>)</p> <p>Containers provide greater efficiency for developers: instead of waiting for operations to provision machines, DevOps teams can quickly package an application into a container and deploy it easily and consistently across different platforms, whether a laptop, a private data center, a public cloud, or hybrid environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/with-and-without-containers-1611298147637.png" alt=""></p> <p><strong>Containers and Microservices</strong></p> <p>Containers are perfect for microservices; each service can be packaged, and each instance deployed as a container, providing the following benefits:</p> <ul> <li>Services can be isolated to specific resources</li> <li>Each container can be health checked</li> <li>Containers can be started upon demand and stopped independently of each other</li> </ul> <p><strong>Container and Cloud</strong></p> <p>The National Institute of Standards and Technology <a href="https://martinfowler.com/bliki/CloudComputing.html">defines a cloud</a> as access to a pool of computing resources that can be rapidly provisioned and made available with four deployment models: private, community, public, and hybrid. With containers, developers can deploy their microservices directly into production without porting efforts. This ability to deploy across different platforms is destined to become much more important in the emerging hybrid IT environment, in which infrastructure is a combination of existing legacy systems, on-premises and off-premises, private cloud and public cloud.</p> <p><strong>Orchestration of Containers and Cloud</strong></p> <p><a href="https://kubernetes.io/"><strong>Kubernetes</strong></a> has been a big step toward making containers mainstream. Kubernetes automates "container orchestration": deployment, scaling, and management of containerized applications.</p> <p>Kubernetes introduced a high-level abstraction layer called a "pod" that enables multiple containers to run on a host machine and share resources without the risk of conflict. A pod can be used to define shared services, like a directory or storage, and expose it to all the containers in the pod.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/kubernetes-1611298156941.png" alt=""></p> <p>This simplifies the management of machines and services, enabling a single administrator to manage thousands of containers running simultaneously.</p> <p>Kubernetes allows you to orchestrate across on-site deployments to public or private clouds and to hybrid deployments in-between. On-premises computation also is moving quickly to containerized orchestration, and when you can interchangeably schedule services anywhere, you have real revolution.</p> <p><strong>DataOps</strong></p> <p>Just as the broader IT world has embraced the concept of DevOps, which uses new technologies and processes to brings application developers and operations together in a cohesive and mutually beneficial manner, the data world today is moving toward DataOps. DataOps is an emerging practice utilized by large organizations with teams of data scientists, developers, and other data-focused roles that train machine learning models and deploy them to production. The goal of using a DataOps methodology is to create an agile, self-service workflow that fosters collaboration and boosts creativity while respecting data governance policies. A DataOps practice supports cross-functional collaboration and fast time-to-value. It is characterized by processes as well as the use of enabling technologies, such as the MapR Data Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/dataops-1611298166619.png" alt=""></p> <p>Combining microservices, containers, and event streams with DataOps makes managing and evaluating multiple models and easily deploying new models more efficient and agile.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/continuous-model-deployment-1611298180998.png" alt=""></p> <p><strong>IoT, Edge Computing, Machine Learning, and the Cloud</strong></p> <p>From automobile manufacturers to oil and gas companies, businesses across the globe seek to derive real business value from outcomes like predicting equipment failures, avoiding accidents, improving diagnostics, and more. There is a growing requirement for edge computing, which brings analytics and machine learning models close to IoT data sources. What makes <a href="https://www.cbronline.com/feature/edge-computing-artificial-intelligence-iot">Edge different</a> is the ability to enable real-time analytics, leveraging local compute for running and feeding machine learning models. In the world of IoT, fast analytics is essential for anomaly detection, fraud detection, aircraft monitoring, oil rig monitoring, manufacturing monitoring, utility monitoring, and health sensor monitoring, where alerts may need to be acted upon rapidly. Imagine how, if machine learning had detected the BP valve pressure anomaly before the Deepwater Horizon explosion in the Gulf of Mexico, <a href="https://en.wikipedia.org/wiki/Deepwater_Horizon_explosion">the largest environmental disaster in U.S. history could have been avoided</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/cloud-to-the-edge-1611298192469.png" alt=""></p> <p><a href="https://www.gartner.com/newsroom/id/3812063">Cloud to the Edge</a>, also called <a href="https://www.cbronline.com/in-depth/iot-fog-computing-cio">Fog</a>, is one of the Gartner's top technology trends for 2018, in which a cloud service-oriented model is combined with edge computing for distributed processing that spans the continuum between the cloud and edge. Ted Dunning, Chief Application Architect at MapR, predicts that we will see a full-scale data fabric extend right to the edge next to devices, and, in some cases, we will see threads of the fabric extend right into the devices themselves.</p> <p><strong>Conclusion</strong></p> <p>A confluence of several different technology shifts have dramatically changed the way that applications are being built. The combination of machine learning, event-driven microservices, containers, DataOps, and cloud to edge computing is accelerating the development of next-generation intelligent applications, which are taking advantage of modern computational paradigms, powered by modern computational infrastructure. The MapR Data Platform integrates global event streaming, real-time database capabilities, and scalable enterprise storage with a collection of data processing and analytical engines to power this new generation of data processing pipelines and intelligent applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/mapr-cdp-1611298203271.png" alt=""></p> <p><strong>Want to learn more?</strong></p> <ul> <li><a href="https://thenewstack.io/microservices-running-containers-need-streaming-platform/">Why Microservices Running in Containers Need a Streaming Platform</a></li> <li><a href="https://www.gartner.com/newsroom/id/3812063">Gartner Identifies the Top 10 Strategic Technology Trends for 2018</a></li> <li><a href="https://www.gartner.com/smarterwithgartner/gartner-top-10-strategic-technology-trends-for-2018/">Gartner Top 10 Strategic Technology Trends for 2018</a></li> </ul>https://developer.hpe.com/association-rule-mining-not-your-typical-data-science-algorithm/https://developer.hpe.com/association-rule-mining-not-your-typical-data-science-algorithm/Fri, 22 Jan 2021 06:28:06 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Kirk Borne", "publish": "2014-04-28T07:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p>Many machine learning algorithms that are used for data mining and data science work with numeric data. And many algorithms tend to be very mathematical (such as Support Vector Machines). But, association rule mining is perfect for categorical (non-numeric) data and it involves little more than simple counting! That’s the kind of algorithm that <a href="http://codecapsule.com/2010/04/15/efficient-counting-mapreduce/">MapReduce is really good at</a>, and it can also lead to some really interesting discoveries.</p> <p>Association rule mining is primarily focused on finding frequent co-occurring associations among a collection of items. It is sometimes referred to as “Market Basket Analysis”, since that was the original application area of association mining. The goal is to find associations of items that occur together more often than you would expect from a random sampling of all possibilities. The classic example of this is the famous Beer and Diapers association that is often mentioned in data mining books. The story goes like this: men who go to the store to buy diapers will also tend to buy beer at the same time. Let us illustrate this with a simple example.</p> <p>Suppose that a store’s retail transactions database includes the following information:</p> <ul> <li>There are 600,000 transactions in total.</li> <li>7,500 transactions contain diapers (1.25 percent)</li> <li>60,000 transactions contain beer (10 percent)</li> <li>6,000 transactions contain <u>both</u> diapers and beer (1.0 percent)</li> </ul> <p>If there was no association between beer and diapers (i.e., they are statistically independent), then we expect only 10% of diaper purchasers to also buy beer (since 10% of all customers buy beer). However, we discover that 80% (=6000/7500) of diaper purchasers also buy beer. This is a factor of 8 increase over what was expected – that is called Lift, which is the ratio of the observed frequency of co-occurrence to the expected frequency. This was determined simply by counting the transactions in the database. So, in this case, the association rule would state that diaper purchasers will also buy beer with a Lift factor of 8.</p> <p>In statistics, Lift is simply estimated by the ratio of the joint probability of two items x and y, divided by the product of their individual probabilities:  Lift = P(x,y)/[P(x)P(y)]. If the two items are statistically independent, then P(x,y)=P(x)P(y), corresponding to Lift = 1 in that case. Note that anti-correlation yields Lift values less than 1, which is also an interesting discovery – corresponding to mutually exclusive items that rarely co-occur together.</p> <p>The above simple example was made up, and it is very rare in real world cases to have Lift factors as high as 8. But, there was a case where it did happen. That case was discovered by Walmart in 2004 when a series of hurricanes crossed the state of Florida. After the first hurricane, there were several more hurricanes seen in the Atlantic Ocean heading toward Florida, and so <a href="http://www.nytimes.com/2004/11/14/business/yourmoney/14wal.html">Walmart mined their massive retail transaction database</a> to see what their customers really wanted to buy prior to the arrival of a hurricane. They found one particular item that increased in sales by a factor of 7 over normal shopping days. That was a huge Lift factor for a real-world case. That one item was not bottled water, or batteries, or beer, or flashlights, or generators, or any of the usual things that we might imagine. The item was <a href="http://www.hurricaneville.com/pop_tarts.html">strawberry pop tarts</a>! One could imagine lots of reasons why this was the most desired product prior to the arrival of a hurricane – pop tarts do not require refrigeration, they do not need to be cooked, they come in individually wrapped portions, they have a long shelf life, they are a snack food, they are a breakfast food, kids love them, and we love them. Despite these “obvious” reasons, it was a still a huge surprise! And so Walmart stocked their stores with tons of strawberry pop tarts prior to the next hurricanes, and they sold them out. That is a win-win: Walmart wins by making the sell, and customers win by getting the product that they most want.</p> <p>Another example of association mining was provided to me by a colleague of mine at George Mason University. He is a professor of geoinformation systems and earth science. He used this algorithm to examine the characteristics of hurricanes (internal wind speed, atmospheric pressure in the eye of the hurricane, wind shear, rainfall amounts, direction and propagation speed of the hurricane, etc.​), and he found a strong association between the final strength (category) of the hurricane and the values of those different characteristics. He was able to predict hurricane intensification and its ultimate strength more accurately with association mining than the standard hurricane model used by the national hurricane center. That was an amazing application of an algorithm that was initially developed for retail store transaction mining.</p> <p>There was an equally impressive scientific application that I encountered several years ago when I was working at NASA. Every summer we would have student interns working with us. These students were usually college undergraduates, and they were always very bright. In one of those summers, we had a student who was not yet a senior in high school. He heard me give a lunch talk on data mining, which I presented to the full cohort of summer interns that year. He was working on a project with a NASA space physicist to try to predict when solar energetic particles would reach the earth after the occurrence of a major solar storm on the Sun. He decided to try association mining. What he did was very clever. Similar to the hurricane example mentioned above, he collected characteristics of the solar storms on the Sun and geomagnetic events around the earth (as measured by NASA’s satellites) to look for predictive patterns. But the special thing that he did was to look at time-shifted data values. For example, he compared events on the Sun with geo events with time lags of 1 hour, 2 hours, 3 hours, 4 hours, etc. in order to see when the peak correlation (association!) occurred. He found it – the strongest geomagnetic effects were measured around the earth at approximately 2-3 hours after the solar storm. His NASA mentor called me into his office to show me the amazing discovery by this high school junior, using the simple techniques that I taught him in my lunchtime seminar. We were all quite impressed!</p> <p>The above examples illustrate two very useful approaches when mining your own big data collections: (1) search for rare and unusual co-occurring associations of non-numeric items (which then makes for powerful predictive analytics); and (2) if you have time-based data, consider the effects of introducing a time lag in your data mining experiments to see if the strength of the correlation reaches it peak some time later​. My final example of association rule mining was discovered many years ago by a major electronics store that sold video cameras and video (VHS) players/recorders. They mined their retail customer database and found that customers who buy a VHS player/recorder tend to come back to the store about 3-4 months later to buy a video camera (camcorder). The store then used this information to send discount coupons for camcorders to all of its customers who bought VHS player/recorders a few months earlier, in order to bring those customers back into the store to purchase a camcorder. Apparently, this customer engagement program worked!  And its success was due to association rule mining. With the massive quantities of big data that are now available, and with powerful technologies to perform analytics on those data, one can only imagine what surprising and useful associations are waiting to be discovered that can boost your bottom line. Start counting!</p>https://developer.hpe.com/an-inside-look-at-the-components-of-a-recommendation-engine/https://developer.hpe.com/an-inside-look-at-the-components-of-a-recommendation-engine/Fri, 22 Jan 2021 06:01:22 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2015-04-09T07:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p>Recommendation engines help narrow your choices to those that best meet your particular needs. In this post, we’re going to take a closer look at how all the different components of a recommendation engine work together. We’re going to use collaborative filtering on movie ratings data to recommend movies. The key components are a collaborative filtering algorithm in <a target='\_blank' href='http://mahout.apache.org/'>Apache Mahout</a> to build and train a machine learning model and search technology from <a target='\_blank' href='https://www.elastic.co/elasticsearch/'>Elasticsearch</a> to simplify deployment of the recommender.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/recommendation-engine-video-1611295605722.png" alt=""></p> <h2>What is Recommendation?</h2> <p>Recommendation is a class of machine learning that uses data to predict a user's preference for or rating of an item.  Recommender systems are used in industry to recommend:</p> <ul> <li>Books and other products (e.g. Amazon)</li> <li>Music (e.g. Pandora)</li> <li>Movies (e.g. Netflix)</li> <li>Restaurants (e.g. Yelp)</li> <li>Jobs (e.g. LinkedIn)</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/netflix-recommendation-engine-1611295620220.png" alt=""></p> <p>The recommender relies on the following observations:</p> <ol> <li>Behavior of users is the best clue to what they want.</li> <li>Co-occurrence is a simple basis that allows Apache Mahout to compute significant indicators of what should be recommended.</li> <li>There are similarities between the weighting of indicator scores in output of such a model and the mathematics that underlie text retrieval engines.</li> <li>This mathematical similarity makes it possible to exploit text-based search to deploy a Mahout recommender using a search engine like Elasticsearch.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/recommendation-engine-architecture-1611295634594.png" alt=""></p> <h2>Architecture of the Recommendation Engine</h2> <p>The architecture of the recommendation engine is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/architecture-recommendation-engine-1611295647852.png" alt=""></p> <ol> <li>Movie information data is reformatted and then stored in Elasticsearch for searching.</li> <li>An item-similarity algorithm from Apache Mahout is run with user movie ratings data to create recommendation indicators for movies. These indicators are added to the movie documents in Elasticsearch.  </li> <li>Searches of a user's preferred movies among the indicators of other movies will return a list of new films sorted by relevance to the user's taste.</li> </ol> <h2>Collaborative Filtering with Mahout</h2> <p>A Mahout-based collaborative filtering engine looks at what users have historically done and tries to estimate what they might likely do in the future, if given a chance. This is accomplished by looking at a history of which items users have interacted with. In particular, Mahout looks at how items co-occur in user histories.  Co-occurrence is a simple basis that allows Apache Mahout to compute significant indicators of what should be recommended. Suppose that Ted likes movie A, B, and C. Carol likes movie A and B. To recommend a movie to Bob, we can note that since he likes movie B and since Ted and Carol also liked movie B, movie A is a possible recommendation. Of course, this is a tiny example. In real situations, we would have vastly more data to work with.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/recommendation-grid-1611295660227.png" alt=""></p> <p>In order to get useful indicators for recommendation, Mahout’s ItemSimilarity program builds three matrices from the user history:</p> <p><strong>1. History matrix:</strong>  contains the interactions between users and items as a user-by-item binary matrix.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/history-matrix-1611295671003.png" alt=""></p> <p><strong>2. Co-occurrence matrix:</strong>  transforms the history matrix into an item-by-item matrix, recording which items co-occur or appear together in user histories.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/co-occurrence-matrix-1611295696721.png" alt=""></p> <p>In this example movie A and movie B co-occur once, while movie A and movie C co-occur twice. The co-occurrence matrix cannot be used directly as recommendation indicators because very common items will tend to occur with lots of other items simply because they are common.  </p> <p><strong>3. Indicator matrix:</strong> The indicator matrix retains only the anomalous (interesting) co-occurrences that will serve as clues for recommendation. Some items (in this case, movies) are so popular that almost everyone likes them, meaning they will co-occur with almost every item, which makes them less interesting (anomalous) for recommendations.  Co-occurrences that are too sparse to understand are also not anomalous and thus are not retained. In this example, movie A is an indicator for movie B.    </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/indicator-matrix-1611295706790.png" alt=""></p> <p>Mahout runs multiple MapReduce jobs to calculate the co-occurrences of items in parallel. (Mahout 1.0 runs on Apache Spark). Mahout’s ItemSimilarityJob uses the log likelihood ratio test (LLR) to determine which co-occurrences are sufficiently anomalous to be of interest as indicators. The output gives pairs of items with a similarity greater than the threshold you provide.</p> <p>The output of the Mahout ItemSimilarity job gives items that identify interesting co-occurrences, or that indicate recommendation, for each item. For example, the Movie B row shows Movie A is indicated, and this means that liking Movie A is an indicator that you will like Movie B.  </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/indicator-matrix-2-1611295716322.png" alt=""></p> <h2>Elasticsearch Search Engine</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/elasticsearch-search-engine-1611295728868.png" alt=""></p> <p>Elasticsearch is an open-source search engine built on top of Apache Lucene™, a full-text search engine library. Full-text search uses precision and recall to evaluate search results:</p> <ul> <li>Precision = proportion of top-scoring results that are relevant</li> <li>Recall = proportion of relevant results that are top-scoring</li> </ul> <p>Elasticsearch stores documents, which are made up of different fields. Each field has a name and content. Fields can be indexed and stored to allow documents to be found by searching for content found in fields.</p> <p>For our recommendation engine, we store movie meta data such as id, title, genre, and also movie recommendation indicators, in a JSON document:</p> <pre><code class="language-json">{ "id": "65006", "title": "Electric Horseman", "year": "2008", "genre": ["Mystery","Thriller"] } </code></pre> <p>The output row from the indicator matrix that identified significant or interesting co-occurrence is stored in the Elasticsearch movie document indicator field. For example, since Movie A is an indicator for Movie B, we will store Movie A in the indicator field in the document for Movie B. That means that when we search for movies with Movie A as an indicator, we will find Movie B and present it as a recommendation.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2021/1/recommendation-matrix-1-1611295740359.png" alt=""></p> <p>Search engines are optimized to find a collection of fields by similarity to a query. We will use the search engine to find movies with the most similar indicator fields to a query.</p>https://developer.hpe.com/hpe-infosight-northbound-apis-bringing-the-power-of-infosight-home/https://developer.hpe.com/hpe-infosight-northbound-apis-bringing-the-power-of-infosight-home/Wed, 20 Jan 2021 11:20:19 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/infosight-1611162508158.jpg" alt="infosight"></p> <p>There is just so much to keep tabs on, isn’t there? There are the tools you use to monitor your Data Center. Then, there are the support &#x26; case management and other IT service management (ITSM) &#x26; BI tools that you are using such as Service Now, not to mention the complexity introduced when your infrastructure stack consists of products from different companies.  It costs you time and effort to constantly have to manually consolidate information from all of these different places. And, with so many tools each having their own stack, your IT environment can grow quite siloed and you can miss out on critical information related to your infrastructure, which can cost you a lot of money. This can all be quite nightmarish and those of us who developed HPE InfoSight understand that.</p> <p>Using HPE InfoSight as an integral part of your cloud-based AI Ops and monitoring solution stack, you now have the ability to pull all the critical information regarding the health and wellness of your HPE infrastructure directly from InfoSight into any of your ITSM/BI/Home grown tools such as Service Now. You can use it to apply your own business logic &#x26; rules to optimize your work and the environment, quickly and easily. HPE InfoSight does this through harnessing the power of REST APIs in our new offering: “InfoSight North Bound APIs for Nimble Storage”. This product is built on top of the case automation system and integrated with the Wellness Dashboard in HPE InfoSight.</p> <p><strong>The current offering:</strong></p> <p>HPE InfoSight North Bound APIs for Nimble Storage consists of a set of two public API endpoints for programmatic access to HPE InfoSight Wellness Issue details, enabling synchronization with service desks and other operational tools. </p> <ol> <li><strong>/Issues:</strong> Retrieve a list of wellness issues with filters for severity, date range, etc.</li> <li><strong>/Issue:</strong> Get details for a specific wellness issue</li> </ol> <p>It also provides a complementary Service Now Integration Configuration Pack to easily integrate the APIs for those of you who use Service Now. </p> <p><strong>How to access the APIs:</strong></p> <p>It is super easy and fast to integrate these APIs into your environment.</p> <ol> <li>Log in to <a href="https://infosight.hpe.com/">https://infosight.hpe.com</a>. Note: Admin role is required.</li> <li>Select the correct Organization and navigate to Settings/API Access</li> <li>Add a new Application, select the Wellness API, and generate a Client Key and Client Secret pair. These will be required when making Wellness API calls.</li> <li>Integrate Wellness APIs into your application.</li> </ol> <p>You can learn more about the technical specifications by checking out our <a href="https://infosight.hpe.com/InfoSight/media/cms/active/public/pubs_HPE_infosight_wellness_spec.pdf">technical specification document</a>.</p> <p><strong>What’s in store for the future:</strong></p> <p>Right now, this offering is available for Nimble Storage and allows access to Wellness Information. We plan on extending this offering to other partner products, as well as add performance and asset metrics, in the future.</p> <p>Go check out these APIs. We would love to hear what you think. Connect with us on <a href="https://app.slack.com/client/T5SNJCC7K/GLWKH9CG5">HPE DEV Slack workspace</a> to share your feedback.</p>https://developer.hpe.com/making-ai-a-reality/https://developer.hpe.com/making-ai-a-reality/Fri, 15 Jan 2021 07:48:29 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/data-drive-ai-1610724616865.JPG" alt="data drive ai"></p> <p><em>Main body of this blog post was originally published on <a href="https://www.h2o.ai/blog/making-ai-a-reality/">H2O.ai blog</a>. Published here with permission.</em></p> <p>Do you want to make AI a part of your company? You can’t just mandate AI. But you can lead by example. </p> <p>All too often, especially in companies new to AI and machine learning, team leaders may be tasked by their managers to “start using AI” without having clear goals set about how AI may help the business or how to build an effective AI team. It’s understandable that executives want to take advantage of the great potential value they see that AI and machine learning are delivering for other companies, but successfully putting AI to work in your own enterprise requires more than passing down the order to do it. Whether you are the executive choosing to add AI to your business or a team leader responsible for making it so, be aware that there are <a href="https://www.h2o.ai/blog/in-a-world-wh">practical steps to put in place</a> before you plunge into hiring someone to make models. One of the first practical steps to develop and implement AI is to build a data-aware culture within your organization. </p> <h2>Data-Driven Choices</h2> <p>One of the challenges in developing serious data skills is to get buy-in about the importance of data-driven decisions. People with extensive experience in their particular business sector hold valuable insights – knowledge that is important to the success of AI projects built to address essential business goals. But this experience may result in people relying on “gut feeling” about business processes, predicted performance and the potential for new lines of business. A key step in building the data-aware culture needed to support AI systems is to develop an appreciation among stakeholders in your organization <strong>for what data can tell you.</strong>  Show - rather than tell - how data can augment, reinforce or refute their experienced view or gut feeling. </p> <p>One way to do this is to <a href="https://www.h2o.ai/blog/the-benefits-of-budget-allocation-with-ai-driven-marketing-mix-models/">make use of data in making your own decisions</a> and to provide transparency to your teams about how data is influencing your decisions. Develop the habit of asking, “What do we know and <strong>how do we know it?</strong>” This habit may involve collecting more metrics about essential business processes than you already do, but more often it means making use of the data you already have. Help people within your organization to see data as a way to expand understanding, not just as a report card on their own performance. </p> <p>Data sources have proliferated in the last few years – data is literally everywhere. This variety of data provides a rich resource for AI and machine learning. You also can raise awareness within your organization of the wide range of data sources you have available and how they can inform you not only about your own business processes but also about the behavior of your customers. </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ellen-ai-picture1-1610696869470.png" alt="ellen ai picture1"></p> <p>Using automated, machine-based learning systems makes it feasible to make decisions based on data at a level and speed that may not be practical for humans to do. This, in turn, lets you take advantage of how data tells you about conditions in the world around you, and how they affect your business.  It also is the best way to inform you about how your business needs to adjust to changes in the world, a situation underlined dramatically by the <a href="https://www.h2o.ai/covid-19/">COVID pandemic</a>. </p> <p>Why is it important to get buy-in from stakeholders? One of the biggest barriers to building effective AI systems is to set the business context for AI in realistic ways. Getting buy-in from stakeholders is important to ensure cooperation in <a href="https://www.h2o.ai/webinars/?commid=433866">defining business goals for AI</a> and for ensuring adequate resources are assigned to develop, implement and maintain AI systems. </p> <h2>AI is a Team Sport</h2> <p>Getting up-to-speed with AI may require hiring talented data scientists or selecting the right company to provide data science as a service. But whether you build an in-house data science team of AI and machine learning experts or work with external talent, there’s more to building an AI system than just the specialists who code and train the models.   </p> <p>Who needs to be on your list of AI talent? Data engineers are a critical part of the success of AI and machine learning projects. In fact, it’s essential to budget a major part of the time and resources allotted to a particular AI project to the effort needed to handle the logistics, from data preparation for training sets to <a href="https://www.h2o.ai/blog/deploying-models-to-maximise-the-impact-of-machine-learning-part-1/">deployment and management of AI models.</a> The following figure shows the relative effort of model building to all the other parts of a machine learning project.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ellen-ai-picture2-1610696879470.png" alt="ellen ai picture2"></p> <p>This requirement for many steps surrounding the specific model building process creates a need for a range of skills beyond the ability to work with algorithms. It’s important, then, to think of AI as a team sport. </p> <h2>DataOps helps bring AI to life</h2> <p>Making AI practical and profitable for your business requires the cooperation and collaboration of people with different skills. One of the best ways to achieve this is through a DataOps approach in which you assemble a cross-skill team that has members collectively focused on a shared goal. That style of work improves intra-team communication and avoids the sense that asking someone with a particular specialized skill to do their part is “asking a favor” or imposing on their time. Instead, people are more apt to work together efficiently when they share a goal and understand what needs to be done. </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ellen-ai-picture3-1610696885974.png" alt="ellen ai picture3"></p> <p>Technology can help with this team approach as well. Tools that improve <a href="https://www.h2o.ai/blog/interview-with-patrick-hall-machine-learning-h2o-ai-machine-learning-interpretability/">machine learning interpretability</a> make it easier for data scientists to communicate clearly to others how models are making decisions and how data is influencing those decisions. The <a href="https://www.hpe.com/us/en/software/marketplace/h2o.html">H2O Driverless AI</a> platform is an example of technology that not only makes it easier to develop AI models but also easier to explain them. </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/ellen-ai-picture4-1610696892273.png" alt="ellen ai picture4"></p> <h2>Next Steps</h2> <p>Building effective AI systems also requires efficient access to data. AI is only as good as the data used to train its models, and <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a> provides multi-API direct global access to data. H2O Driverless AI, for example, can access data stored in the HPE Ezmeral Data Fabric <em>directly,</em> without having to copy it out to another system. Furthermore, HPE Ezmeral Data Fabric is the core data layer in the <a href="https://www.hpe.com/us/en/software/ezmeral-runtime.html">HPE Ezmeral Runtime Enterprise</a> for cloud-native and distributed non-cloud native applications. Containerization of AI applications is a big advantage in being able to train, deploy and run AI models in predictable environments that you can easily control. H2O Driverless AI has been tested and <a href="https://www.hpe.com/us/en/software/marketplace/h2o.html">validated for compatibility</a> with the HPE Ezmeral Container platform.</p> <p> </p> <p>Another offering in the HPE Ezmeral Software Portfolio, <a href="https://www.hpe.com/us/en/solutions/machine-learning-operations.html">HPE Ezmeral ML Ops</a>, also works together with H2O Driverless AI to make data science more efficient through automatic machine learning.  </p> <p> </p> <p>Find out more about these AI-enabling technologies, including how you can try a 21-day free trial of H2O Driverless AI, by visiting the <a href="https://www.hpe.com/us/en/software/marketplace/h2o.html">HPE Ezmeral Marketplace</a>.</p> <p> </p> <p>For additional information about building a data-aware culture, as well as other practical steps in bringing AI into your business, read the free ebook <em><a href="https://www.h2o.ai/resources/ebook/practical-advice-for-making-ai-part-of-your-companys-future/">Practical Advice for Making AI Part of Your Company’s Future</a>.</em></p> <p> </p> <p>And please check out the <a href="https://developer.hpe.com/">HPE DEV blog</a> for more articles on developer-focused topics.</p>https://developer.hpe.com/synchronized-volume-snapshots-for-distributed-workloads-on-kubernetes/https://developer.hpe.com/synchronized-volume-snapshots-for-distributed-workloads-on-kubernetes/Thu, 14 Jan 2021 18:10:48 GMT<p>Typically, Persistent Volume Claims on Kubernetes are treated as a singular entity completely decoupled from your workload. The actual physical location doesn't really matter. But what if you wanted an atomic operation where all Persistent Volume Claims that make up an application in a microservice architecture need to be atomically protected to ensure referential integrity? Would you stop the application, sequence the operation or take a shotgun approach and hope for the best?</p> <p>In this blog post, we'll use the HPE CSI Driver for Kubernetes to create Volume Groups that allow users to group Persistent Volume Claims together and use those Volume Groups to perform CSI Volume Snapshots through Snapshot Groups.</p> <p>In other storage infrastructure management systems, the term "Volume Groups" is usually referred to as "Consistency Groups". It is the industry standard to create volume snapshots with referential integrity. This capability was introduced in the HPE CSI Driver for Kubernetes v1.4.0 and more information about the release may be found on <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-v1-4-0-with-expanded-ecosystem-and/ba-p/7118180">Around The Storage Block</a>.</p> <h1>TL;DR</h1> <p>A variant of the demonstrative steps below has been captured in a screencast that is available on YouTube. If you prefer watching and listening instead of reading, please go ahead and watch the screencast.</p> <p>Just don’t forget to come back to read the "Learn more" section at the end of this article for important information.</p> <p><a href="https://youtu.be/zUj-bJ_KqHU" title="Synchronize Volume Snapshots for Distributed Workloads using the HPE CSI Driver for Kubernetes"><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/untitled-1610648109128.png" alt="Synchronize Volume Snapshots for Distributed Workloads using the HPE CSI Driver for Kubernetes"></a> Watch on <a href="https://youtu.be/zUj-bJ_KqHU">YouTube</a>!</p> <h1>Prerequisites</h1> <p>The examples we're going to walk through require that the HPE CSI Driver for Kubernetes v1.4.0 or later has been installed along with the CSI external snapshotter. Examples also assume a <code>VolumeSnapshotClass</code> named "hpe-snapshot" exists on the cluster.</p> <ul> <li>Install the HPE CSI Driver using the <a href="https://artifacthub.io/packages/helm/hpe-storage/hpe-csi-driver">Helm Chart</a></li> <li>Enable <a href="https://scod.hpedev.io/csi_driver/using.html#enabling_csi_snapshots">CSI snapshots</a></li> <li>Add a <a href="https://scod.hpedev.io/csi_driver/deployment.html#add_a_hpe_storage_backend">HPE storage backend</a></li> <li>Create a <a href="https://scod.hpedev.io/csi_driver/using.html#using_csi_snapshots">VolumeSnapshotClass</a> and of course a <a href="https://scod.hpedev.io/csi_driver/using.html#base_storageclass_parameters">StorageClass</a></li> </ul> <p>No particular parameters are needed in either the <code>VolumeSnapshotClass</code> or <code>StorageClass</code> but the backend <code>Secret</code> is assumed to be named "hpe-backend" and reside in the "hpe-storage" <code>Namespace</code>.</p> <p>In the examples below, we're using HPE Nimble Storage. Any Container Storage Provider (CSP) will work that supports <code>VolumeGroups</code> and <code>SnapshotGroups</code>.</p> <h1>Pick an application</h1> <p>In order to illustrate the fact that multiple snapshots are being created, either pick an application that requires multiple volumes or deploy a microservice stack comprised of multiple stateful applications. In this example we'll use WordPress from the <code>bitnami/wordpress</code> Helm Chart.</p> <p>We're using the following "values" file for the deployment:</p> <pre><code class="language-yaml">mariadb: architecture: replication </code></pre> <p>Add the Bitnami repo:</p> <pre><code class="language-markdown">helm repo add bitnami https://charts.bitnami.com/bitnami </code></pre> <p>Install the WordPress chart:</p> <pre><code class="language-markdown">helm install my-wordpress bitnami/wordpress -f wp-values.yaml </code></pre> <p>Once deployed, there should be three <code>PersistentVolumeClaims</code> on the cluster.</p> <pre><code class="language-markdown">kubectl get pvc -o name persistentvolumeclaim/data-my-wordpress-mariadb-primary-0 persistentvolumeclaim/data-my-wordpress-mariadb-secondary-0 persistentvolumeclaim/my-wordpress </code></pre> <p>Adding content to the WordPress site being deployed is completely optional as restore procedures are not covered in this tutorial. However, if you're interested in performing a restore from the <code>VolumeSnapshots</code> we're creating, please see the CSI snapshots tutorial in this <a href="/blog/PklOy39w8NtX6M2RvAxW/hpe-csi-driver-for-kubernetes-snapshots-clones-and-volume-expansion">HPE DEV blog post</a>.</p> <h1>Volume Groups</h1> <p>The <code>CustomResourceDefinition</code> (CRD) that users interact with is called a <code>VolumeGroup</code>. In order to facilitate the creation of <code>VolumeGroups</code> a Kubernetes cluster administrator needs to create another <code>CRD</code> called a <code>VolumeGroupClass</code>.</p> <p>The Volume Group Provisioner (depicted below) is a Kubernetes CSI sidecar container that performs a number of duties to facilitate volume grouping.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/groups-1610651079691.png" alt="picture3"></p> <p>The <code>VolumeGroupContent</code> <code>CRD</code> is managed solely by the Volume Group Provisioner.</p> <p>Let's start by creating a <code>VolumeGroupClass</code>. The remaining YAML in this tutorial is assumed to be pasted into a terminal verbatim, like this:</p> <pre><code class="language-markdown">kubectl create -f- &#x3C;hit ENTER> &#x3C; paste the YAML content > ^D &#x3C;hit CTRL+D on a new line> </code></pre> <p>Let's begin:</p> <pre><code class="language-yaml">--- apiVersion: storage.hpe.com/v1 kind: VolumeGroupClass metadata: name: my-volume-group-class provisioner: csi.hpe.com deletionPolicy: Delete parameters: description: "HPE CSI Driver for Kubernetes Volume Group" csi.hpe.com/volume-group-provisioner-secret-name: hpe-backend csi.hpe.com/volume-group-provisioner-secret-namespace: hpe-storage </code></pre> <p>At this stage nothing has been created on the backend storage system. The next step is to create a blank <code>VolumeGroup</code>.</p> <pre><code class="language-yaml">--- apiVersion: storage.hpe.com/v1 kind: VolumeGroup metadata: name: my-volume-group spec: volumeGroupClassName: my-volume-group-class </code></pre> <p>Now, logging in to the backend HPE Nimble Storage array, we'll see a new Volume Collection has been created.</p> <pre><code class="language-markdown">Nimble OS $ volcoll --list --------------------+---------------+------------------------------------------- Volume Collection Application Owned By Name Synchronization --------------------+---------------+------------------------------------------- volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 none nva-grp </code></pre> <p>Inspecting further:</p> <pre><code class="language-markdown">Nimble OS $ volcoll --info volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 Name: volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 Description: HPE CSI Driver for Kubernetes Volume Group Owned by: nva-grp Application synchronization: none Application server: N/A Application ID: N/A Cluster name: N/A Service name: N/A VMware vCenter hostname: N/A VMware vCenter username: N/A VMware vCenter password: N/A Backup agent hostname: N/A Backup agent username: N/A Backup agent password: N/A Associated volumes: none Associated pinned volumes: none Snapshot collection count: 0 Created: Jan 13 2021 16:40:48 Last configuration change: Jan 13 2021 16:40:48 Replication Type: Periodic Snapshot Synchronous Replication State: N/A Synchronous Replication Last In Sync: N/A Synchronous Replication Resync %: N/A </code></pre> <p>Adding members to the <code>VolumeGroup</code> is done by adding annotations to <code>PersistentVolumeClaims</code>. This can be done in a number of ways. Either the claims may be created with the annotation, using the <code>kubectl patch</code> command and the <code>kubectl annotate</code> command. I prefer using the <code>kubectl annotate</code> command. Let's do it.</p> <pre><code class="language-markdown">kubectl annotate pvc/data-my-wordpress-mariadb-primary-0 \ pvc/data-my-wordpress-mariadb-secondary-0 \ pvc/my-wordpress \ csi.hpe.com/volume-group=my-volume-group </code></pre> <p>Annotations doesn't need to be created all at once. They can be added (or removed) individually.</p> <p>At this stage you can inspect the <code>VolumeGroup</code> on the Kubernetes cluster or the Volume Collection on the array to confirm that <code>PersistentVolumeClaims</code> have been added to the <code>VolumeGroup</code>.</p> <pre><code class="language-markdown">kubectl get -o yaml volumegroup/my-volume-group \ -o 'jsonpath={.spec.persistentVolumeClaimNames[*]}' </code></pre> <p>Next, let's create some atomic <code>VolumeSnapshots</code>.</p> <h1>Snapshot Groups</h1> <p>The <code>SnapshotGroup</code> <code>CRD</code> is primarily what users interact with. The Snapshot Group Snapshotter depicted below carries out all the backend work and populates Kubernetes with the necessary <code>CRDs</code>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/snaps-1610647869530.png" alt="picture4"></p> <p>In a similar fashion to <code>VolumeGroupClasses</code> a <code>SnapshotGroupClass</code> needs to be created by an administrator.</p> <pre><code class="language-yaml">--- apiVersion: storage.hpe.com/v1 kind: SnapshotGroupClass metadata: name: my-snapshot-group-class snapshotter: csi.hpe.com deletionPolicy: Delete parameters: csi.hpe.com/snapshot-group-snapshotter-secret-name: hpe-backend csi.hpe.com/snapshot-group-snapshotter-secret-namespace: hpe-storage </code></pre> <p>Now, users may create <code>SnapshotGroups</code>. Upon creation, the <code>SnapshotGroups</code> will automatically be populated by <code>VolumeSnapshots</code> by the Snapshot Group Snapshotter. Let's create a <code>SnapshotGroup</code> and see.</p> <pre><code class="language-yaml">--- apiVersion: storage.hpe.com/v1 kind: SnapshotGroup metadata: name: my-snapshot-group-1 spec: source: kind: VolumeGroup apiGroup: storage.hpe.com name: my-volume-group snapshotGroupClassName: my-snapshot-group-class volumeSnapshotClassName: hpe-snapshot </code></pre> <p>While unrelated to this tutorial, the "hpe-snapshot" <code>VolumeSnapshotClass</code> should already exist on the cluster as described in the prerequisites above. The <code>.spec.source</code> indicates our previously created <code>VolumeGroup</code> to snapshot. Now, let's check for <code>VolumeSnapshots</code>.</p> <pre><code class="language-markdown">kubectl get volumesnapshots -o name volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-data-my-wordpress-mariadb-primary-0 volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-data-my-wordpress-mariadb-secondary-0 volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-my-wordpress </code></pre> <p>Presto! Further, logging in to the backend array, we can now see a new Snapshot Collection with populated entries.</p> <pre><code class="language-markdown">Nimble OS $ snapcoll --list --------------------+---------------------------------------+-------+----------- Volume Collection Snapshot Collection Num Replication Name Name Snaps Status --------------------+---------------------------------------+-------+----------- volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 snapshot-fde72bb7-6633-4e6f-841c-3efaa0444710 3 N/A </code></pre> <p>A few points to note.</p> <ul> <li><code>SnapshotGroups</code> may be deleted from the cluster and will in turn delete the <code>VolumeSnapshots</code> and backend Snapshot Collection</li> <li><code>VolumeSnapshots</code> may be used to perform restores (clone) from as any other <code>VolumeSnapshot</code></li> </ul> <h1>Learn more</h1> <p>This is just the tip of the iceberg when it comes to data management innovation from HPE. Stay tuned to the <a href="https://developer.hpe.com/">HPE Developer Community</a> to learn more about upcoming features and capabilities.</p> <ul> <li>Explore the HPE CSI Driver for Kubernetes <a href="https://github.com/hpe-storage/csi-driver">on GitHub</a></li> <li>Check out the release announcement on <a href="https://community.hpe.com/t5/Around-the-Storage-Block/HPE-CSI-Driver-for-Kubernetes-v1-4-0-with-expanded-ecosystem-and/ba-p/7118180">Around The Storage Block</a></li> <li>Install the latest HPE CSI Driver using the <a href="https://artifacthub.io/packages/helm/hpe-storage/hpe-csi-driver">Helm Chart from ArtifactHub.io</a></li> <li>Visit the <a href="https://scod.hpedev.io">HPE Storage Container Orchestration Documentation</a> (SCOD) portal for everything pertaining to the HPE CSI Driver</li> <li>Sign up for a <a href="/hackshack/workshops">HPE DEV Hack Shack Workshop</a> to learn about CSI on the HPE Ezmeral Container Platform</li> </ul> <p>HPE is eager to learn what our customers and partners are doing with Kubernetes and data management. Join the HPE DEV Slack community to share your thoughts and engage with the team. Sign up at <a href="http://slack.hpedev.io">slack.hpedev.io</a> and sign in at <a href="http://hpedev.slack.com">hpedev.slack.com</a>, we hang out in #nimblestorage, #3par-primera and #kubernetes.</p>https://developer.hpe.com/streaming-data-pipeline-to-transform-store-and-explore-healthcare-datase/https://developer.hpe.com/streaming-data-pipeline-to-transform-store-and-explore-healthcare-datase/Thu, 14 Jan 2021 05:59:02 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-02-27T12:00:00.000", "tags": "use-cases" </code></pre> <hr> <p>In the past, big data was interacted with in batch on a once-a-day basis. Now data is dynamic and data driven businesses need instant results from continuously changing data. Data Pipelines, which combine real-time Stream processing with the collection, analysis and storage of large amounts of data, enable modern, real-time applications, analytics and reporting.</p> <p>This post is based on a recent workshop I helped develop and deliver at a large health services and innovation company's analytics conference. This company is combining streaming data pipelines with data science on top of the MapR Data Platform to improve healthcare outcomes, improve access to appropriate care, better manage cost, and reduce fraud, waste and abuse.</p> <p>In this post we will:</p> <ul> <li>Use Apache Spark streaming to consume <a href="https://www.cms.gov/openpayments/">Medicare Open payments</a> data using the Apache Kafka API</li> <li>Transform the streaming data into JSON format and save to the MapR Database document database.</li> <li>Query the MapR Database JSON table with Apache Spark SQL, Apache Drill, and the Open JSON API (OJAI) and Java.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/example-streamline-processing-pipeline-1610604239860.png" alt="Example Streamline Processing Pipeline"></p> <h2>Example Use Case Data Set</h2> <p>Since 2013, <a href="https://www.cms.gov/openpayments/">Open Payments</a> is a federal program that collects information about the payments drug and device companies make to physicians and teaching hospitals for things like travel, research, gifts, speaking fees, and meals.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/facts-about-open-payments-data-1610604259299.png" alt="Facts About Open Payments Data"></p> <p>A large Health payment dataset, JSON, Apache Spark, MapR Event Store, and MapR Database are an interesting combination for a health analytics workshop because:</p> <ul> <li>JSON is an open-standard and efficient format that is easy for computer languages to manipulate. Newer standards for exchanging healthcare information such as <a href="https://www.hl7.org/fhir/overview.html">FHIR</a> are easier to implement because they use a modern suite of API technology, including JSON.</li> <li><a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">Apache Spark SQL, Dataframes, and Datasets</a> make it easy to load, process, transform, and analyze JSON data. MapR Event Store is a distributed messaging system for streaming event data at scale. MapR Event Store integrates with Spark Streaming via the Kafka API.</li> <li>MapR Database, a high performance NoSQL database, supports JSON documents as a native data store. MapR Database makes it easy to store, query and build applications with JSON documents. The Spark connector makes it easy to build real-time or batch pipelines between your JSON data and MapR Database and leverage Spark within the pipeline.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/mapr-db-table-1610604274960.png" alt="MapR Database table"></p> <h2>How Do You Build a Data Pipeline that handles millions of events in Real Time at Scale?</h2> <p>A common data pipeline architecture pattern is event sourcing using an append only publish subscribe event stream such as MapR Event Streams (which provides a Kafka API). MapR Event Store Topics are logical collections of events, which organize events into categories and decouple producers from consumers, making it easy to add new producers and consumers. Topics are partitioned for throughput and scalability, producers are load balanced and consumer can be grouped to read in parallel. MapR Event Store can scale to very high throughput levels, easily delivering millions of messages per second using very modest hardware.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/kafka-api-1610604290555.png" alt="Kafka API"></p> <h2>Processing Streaming Data with Spark</h2> <p>Apache Spark Streaming is an extension of the core Spark API that enables continuous data stream processing. Data streams can be processed with Spark's core, SQL, GraphX, or machine learning APIs, and can be persisted to a file system, HDFS, MapR XD, MapR Database, HBase, or any data source offering a Hadoop OutputFormat or Spark connector. Stream processing of events is useful for filtering, transforming, creating counters and aggregations, correlating values, joining streams together, machine learning, and publishing to a different topic for pipelines.</p> <p>MapR Event Streams integrates with Spark Streaming via the Kafka direct approach. The MapR Database OJAI Connector for Apache Spark enables you to use <a href="https://docs.datafabric.hpe.com/62/Spark/SavingDStreamMapRDB.html">MapR Database as a sink for Apache Spark Data Streams</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/application-1610604303183.png" alt="Application"></p> <p>The incoming data is in CSV format, an example is shown below:</p> <pre><code class="language-markdown">"NEW","Covered Recipient Physician",,,,"132655","GREGG","D","ALZATE",,"8745 AERO DRIVE","STE 200","SAN DIEGO","CA","92123","United States",,,"Medical Doctor","Allopathic &#x26; Osteopathic Physicians|Radiology|Diagnostic Radiology","CA",,,,,"DFINE, Inc","100000000326","DFINE, Inc","CA","United States",90.87,"02/12/2016","1","In-kind items and services","Food and Beverage",,,,"No","No Third Party Payment",,,,,"No","346039438","No","Yes","Covered","Device","Radiology","StabiliT",,"Covered","Device","Radiology","STAR Tumor Ablation System",,,,,,,,,,,,,,,,,"2016","06/30/2017" </code></pre> <p>There are a lot of fields in this data that we will not use; we will parse the following fields:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/parse-fields-1610604316662.png" alt="Parse Fields"></p> <p>And transform them into the following JSON document for storing in MapR Database:</p> <pre><code class="language-json">{ "_id" :"317150_08/26/2016_346122858", "physician_id" :"317150", "date_payment" :"08/26/2016", "record_id" :"346122858", "payer" :"Mission Pharmacal Company", "amount" :9.23, "Physician_Specialty" :"Obstetrics &#x26; Gynecology", "Nature_of_payment" :"Food and Beverage" } </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/transform-1610604333567.png" alt="Transform"></p> <p>Spark Kafka Consumer Producer Code</p> <blockquote> <p>Note: Code snippets are shown here. You can download the complete code and instructions from the GitHub link at the end of this post.</p> </blockquote> <h2>Parsing the Data Set Records</h2> <p>A Scala Payment case class defines the schema corresponding to the CSV data that we are interested in. The parsePayment function parses a line of comma separated values into the Payment case class.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/parsing-data-set-records-1610604356991.png" alt="Parsing the Data Set Records"></p> <p>A PaymentwId class defines the JSON document schema for MapR Database.</p> <p>In order to save the JSON objects to MapR Database, we need to define the_id field, which is the row key and primary index for MapR Database. The parsePaymentwID function creates an object with the id equal to a combination of the physician id, the date, and the record id. Since MapR Database row keys are partitioned and sorted by row key when inserted, the payment documents will be grouped by physician and date in MapR Database. This will give really fast queries, aggregations and sorting by physician id and date. We will also look at secondary indexes later in this post.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/parsing-data-set-records-2-1610604371777.png" alt="Parsing the Data Set Records"></p> <h3>Spark Streaming Code</h3> <p>We use the KafkaUtils createDirectStream method with Kafka configuration parameters to create an input stream from a MapR Event Store topic. This creates a DStream that represents the stream of incoming data, where each message is a key value pair. We use the DStream map transformation to create a DStream with the message values, and then another map transformation with the parsePaymentwID function to create a DStream of PaymentwID objects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/spark-streaming-code-1-1610604388626.png" alt="Spark Streaming Code"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/spark-streaming-code-2-1610604403278.png" alt="Spark Streaming Code"></p> <p>The output of the paymentDStream.print(3) is shown below</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/output-paymentdstream-1610604421331.png" alt="output of the paymentDStream.print(3)"></p> <p>For storing lots of streaming data, we need a data store that supports fast writes and scales. The MapR Database Spark Connector DStream saveToMapRDB method performs a parallel partitioned bulk insert of JSON PaymentwID objects into MapR Database:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/mapr-db-spark-connector-dstream-savetomaprdb-method-1610604444456.png" alt="MapR Database Spark Connector DStream saveToMapRDB method"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/save-to-mapr-db-json-1610604463458.png" alt="Save to MapR Database JSON"></p> <h3>MapR Database</h3> <p>One of the challenges when you are processing lots of data is where do you want to store it? With MapR Database (HBase API or JSON API), a table is automatically partitioned into tablets across a cluster by key range, providing for scalable and fast reads and writes by row key.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/scalable-fast-reads-writes-row-key-1610604478573.png" alt="Scalable and Fast Reads and Writes by Row Key"></p> <p>The Spark MapR Database Connector leverages the Spark <a href="https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html">DataSource API</a>. The connector architecture has a connection object in every Spark Executor, allowing for distributed parallel writes, reads, or scans with MapR Database tablets.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/connection-in-spark-executor-1610604492674.png" alt="Connection in Every Spark Executor"></p> <h2>Querying MapR Database JSON with Spark SQL</h2> <p>The Spark MapR Database Connector enables users to perform complex SQL queries and updates on top of MapR Database using a Spark Dataset, while applying critical techniques such as Projection and filter pushdown, custom partitioning, and data locality.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/querying-application-1610604507438.png" alt="Querying Application"></p> <p>A Spark <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset">Dataset</a> is a distributed collection of data. Dataset is a newer interface, which provides the benefits of strong typing, the ability to use powerful lambda functions, efficient object serialization/deserialization , combined with the benefits of Spark SQL's optimized execution engine.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/dataset-1610604520672.png" alt="Dataset"></p> <p>A DataFrame is a Dataset organized into named columns Dataset[Row]. (In Spark 2.0, the DataFrame APIs merged with <a href="https://databricks.com/blog/2016/01/04/introducing-apache-spark-datasets.html">Datasets</a> APIs.)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/unified-apache-spark-1610604532100.png" alt="Unified Apache Spark"></p> <h3>Loading data from MapR Database into a Spark Dataset</h3> <p>To <a href="https://docs.datafabric.hpe.com/62/Spark/LoadDataFromMapRDBasDataset.html">load data from a MapR Database JSON</a> table into an Apache Spark Dataset, we first define the Scala class and <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">Spark StructType </a>matching the structure of the JSON objects in the MapR Database table.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/load-data-from-mapr-db-json-1610604545905.png" alt="load data from a MapR Database JSON"></p> <p>Next, we invoke the loadFromMapRDB method on a SparkSession object, providing the tableName , schema and case class. This will return a Dataset of PaymentwId objects:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/dataset-paymentwid-objects-1610604560801.png" alt="Dataset of PaymentwId objects"></p> <p>Explore and query the Payment data with Spark SQL</p> <p>Datasets provide a domain-specific language for structured data manipulation in Scala, Java, and Python. Below are some examples in scala. The Dataset show() action displays the top 20 rows in a tabular form.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/20-rows-tabular-form-1610604575925.png" alt="20 rows in a tabular form"></p> <p>What are the top 5 nature of payments by count?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/5-nature-payments-by-count-1610604589757.png" alt="What are the top 5 nature of payments by count"></p> <p>What is the count of Payers with payment amounts > $1000?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/payments-greater-than-1000-1610604608196.png" alt="Payers with payment amounts > $1000"></p> <p>You can register a Dataset as a temporary table using a given name and then run Spark SQL. Here are some example Spark SQL queries on the payments dataset:</p> <p>What are the top 5 physician specialties by amount with count?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/5-physician-speicialties-1610604623826.png" alt="5 physician specialties by amount with count"></p> <h1>Querying the Data with Apache Drill</h1> <p>Apache Drill is an open source, low-latency query engine for big data that delivers interactive SQL analytics at petabyte scale. Drill provides a massively parallel processing execution engine, built to perform distributed query processing across the various nodes in a cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/apache-drill-1610604648035.png" alt="Apache Drill"></p> <p>With Drill, you can use SQL to interactively query and join data from files in JSON, Parquet, or CSV format, Hive, and NoSQL stores, including HBase, MapR Database, and Mongo, without defining schemas. MapR provides a <a href="https://package.mapr.com/tools/MapR-JDBC/MapR_Drill/">Drill JDBC</a> driver that you can use to connect Java applications, BI tools, such as SquirreL and Spotfire, to Drill. Below is an snippit of Java code for querying MapR Database using Drill and JDBC:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/java-code-querying-1610604663404.png" alt="Java code for querying MapR Database using Drill and JDBC"></p> <p>Partial output for this query is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/partial-output-for-query-1610604688017.png" alt="Partial output for this query"></p> <p>Below are some examples SQL queries using the Drill shell.</p> <p>What are the top 5 physicians by total amount?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/5-physicians-by-total-amount-1610604702767.png" alt="top 5 physicians by total amount"></p> <p>What are the distinct payers in the Payments table?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/distinct-payers-payments-table-1610604719490.png" alt="distinct payers in the Payments table"></p> <p>Follow the instructions in the github code README to <a href="https://docs.datafabric.hpe.com/62/Drill/optimizing_queries_with_indexes.html">add a secondary index to MapR Database</a> and try more queries.</p> <h2>Querying with the Open JSON API (OJAI)</h2> <p>Below is a Java example of using the <a href="https://docs.datafabric.hpe.com/62/MapR-DB/JSON_DB/QueryingWithOJAI.html">OJAI Query interface to query documents in a MapR Database JSON</a> table:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/java-example-1610604732007.png" alt="Java example"></p> <p>Partial output for this query is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/partial-output-1610604746301.png" alt="Partial output"></p> <h2>Summary</h2> <p>In this blog post, you've learned how to consume streaming Open Payments CSV data, transform to JSON, store in a document database, and explore with SQL using Apache Spark, MapR Event Store MapR Database, OJAI, and Apache Drill.</p> <p><strong>Code</strong></p> <p>You can download the code and data to run these examples from <a href="https://github.com/mapr-demos/mapr-es-db-spark-payment">here</a>. Refer to the README for complete instructions to run the code.</p> <p><strong>Running the Code</strong></p> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/mapr-cdp-1610604758846.png" alt="MapR-CDP"></p> <p>This example was developed using the <a href="https://docs.datafabric.hpe.com/62/MapRContainerDevelopers/MapRContainerDevelopersOverview.html">MapR 6.0 container for developers</a>, a docker container that enables you to create a single node MapR cluster. The container is lightweight and designed to run on your laptop. Refer to the code README <a href="https://github.com/mapr-demos/mapr-es-db-spark-payment">here</a> for instructions on running the code.</p>https://developer.hpe.com/predicting-forest-fires-with-spark-machine-learning/https://developer.hpe.com/predicting-forest-fires-with-spark-machine-learning/Thu, 14 Jan 2021 05:41:09 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ian Downard", "publish": "2018-07-26T10:45:00.000", "tags": "use-case" </code></pre> <hr> <p>Anytime you have lat/long coordinates, you have an opportunity to do data science with k-means clustering and visualization on a map. This is a story about how I used geo data with k-means clustering on a topic that has affected me personally - wildfires!</p> <p>Every summer wildfires become front of mind for thousands of people who live in the West, Pacific Northwest, and Northern Rockies regions of the United States. Odds are, if you don't see the flames first hand, you will probably see smoke influenced the weather, road closures, and calls for caution by local authorities.</p> <p>I've lived in Oregon for about 10 years. In that time I've had more than one close encounter with a forest fire. The summer of 2017 was especially bad. A fire in the Columbia River Gorge blew smoke and ash through my neighborhood. Earlier in the year, I crossed paths with firefighters attempting to control a fire in steep rugged terrain in southern Washington. I was stunned to see the size of their equipment and trucks so badass they could be in a Mad Max movie.</p> <p>Fire fighting is expensive! <a href="https://www.usda.gov/media/press-releases/2017/09/14/forest-service-wildland-fire-suppression-costs-exceed-2-billion">Wildland fire suppression costs exceeded $2 billion in 2017</a>, making it the most expensive year on record for the U.S. Forest Service. Fires also have a tendency to explode in size. It's not at all unusual for fires to grow by 50,000 acres in one day when winds are high and the terrain is steep. Take a moment to conceptualize those numbers. $2 billion! More than 50,000 acres, burned in one day!</p> <h2>A Beckoning to Data Science</h2> <p>I focused my study on optimizing against the following two constraints:</p> <ol> <li>The cost it takes to move water tanks and other heavy equipment.</li> <li>The propensity for fires to explode in size if not suppressed quickly.</li> </ol> <p>Both of these things can be minimized by staging equipment as close as possible to where fires are likely to occur.</p> <h2>The Problem</h2> <p>Minimize the cost and time to respond to fires by staging firefighting assets as close as possible to where fires are likely to occur.</p> <h2>The Solution</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image10-1610603073723.png" alt=""></p> <p>My goal is to predict where forest fires are prone to occur by partitioning the locations of past burns into clusters whose centroids can be used to optimally place water tanks and heavy fire fighting equipment as near as possible to where fires are likely to occur. The k-means clustering algorithm is aptly suited for this purpose.</p> <h2>The Data</h2> <p>The U.S. Forest Service provides datasets that describe forest fires that have occurred in Canada and the United States since the year 2000. That data can be downloaded from the Forest Service at <a href="https://fsapps.nwcg.gov/gisdata.php">https://fsapps.nwcg.gov/gisdata.php</a>. For my purposes, this dataset is provided in an inconvenient <a href="http://doc.arcgis.com/en/arcgis-online/reference/shapefiles.htm">shapefile</a> format. It needs to be transformed to CSV in order to be easily analyzed with Apache Spark (no pun intended). Also, the records after 2008 have a different schema than prior years, so, after converting the shapefiles to CSV, they'll need to be ingested into Spark using separate user-defined schemas.</p> <p>By the way, this complexity is typical. Raw data is hardly ever suitable for machine learning without cleansing. The process of cleaning and unifying messy data sets is called data wrangling, and it frequently comprises the bulk of the effort involved in real-world machine learning.</p> <h2>The Science</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image7-1610603093049.png" alt=""></p> <p>The data wrangling that precedes machine learning typically involves writing expressions in R, SQL, Scala, and/or Python to join and transform sampled datasets. Often, getting these expressions right involves a lot of trial and error. Ideally, you want to test those expressions without the burden of compiling and running a full program. Data scientists have embraced web-based notebooks such as Apache Zeppelin for this purpose because they allow you to interactively transform datasets and let you know right away if what you're trying to do will work properly.</p> <p>The Zeppelin notebook I wrote for this study contains a combination of Bash, Python, Scala, and Angular code. A screenshot of the Zeppelin notebook I created for this study is shown <a href="https://github.com/iandow/iandow.github.io/blob/master/img/firenotebook.png">here</a> and the notebook file can be downloaded <a href="https://raw.githubusercontent.com/iandow/iandow.github.io/master/_includes/Forest%20Fire%20Prediction.json">here</a>. Essentially, I used Zeppelin to accomplish the following tasks:</p> <ol> <li>Download historical forest fire data</li> <li>Transform raw data into a desirable format</li> <li>Identify lat/long feature columns for k-means clustering</li> <li>Load data into Spark ML and train a k-means model</li> <li>Map the clusters showing regions where fires have been concentrated in the past</li> </ol> <p>The code and documentation I wrote for this study is available here:</p> <p><a href="https://github.com/mapr-demos/mapr-sparkml-streaming-wildfires">https://github.com/mapr-demos/mapr-sparkml-streaming-wildfires</a></p> <h2>Code Walkthrough</h2> <p>Here's the bash code I use to download the dataset:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image2-1610603101302.png" alt=""></p> <p>Here's the Python code I used to convert the downloaded datasets to CSV files:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image9-1610603111027.png" alt=""></p> <p>Here's the Scala code I use to ingest the CSV files and train a k-means model with Spark libraries:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image12-1610603119677.png" alt=""></p> <p>The resulting cluster centers are shown below. Where would you stage fire fighting equipment?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image3-1610603128020.jpg" alt=""></p> <p>These circles were calculated by analyzing the locations for fires that have occurred in the past. These points can be used to help stage fire fighting equipment as near as possible to regions prone to burn, but how do we know which staging area should respond when a new forest fire starts? We can use our previously saved model to answer that question. The Scala code for that looks like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image4-1610603136768.png" alt=""></p> <h2>Operationalizing This Model as a Real-Time "Fire Response" App</h2> <p>The previous code excerpt shows how the model we developed could be used to identify which fire station (i.e., centroid) should be assigned to a given wildfire. We could operationalize this as a real-time fire response application with the following ML pipeline:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image1-1610603349597.png" alt=""></p> <p>Most machine learning applications are initially architected with a synchronous pipeline like the one shown above, but there are limitations to this simplistic approach. Since it is only architected for a single model, your options are limited when it comes to the following:</p> <ul> <li>How do you A/B test different versions of your model?</li> <li>How do you load balance inference requests?</li> <li>How do you process inference requests with multiple models optimized for different objectives (e.g., speed versus accuracy)?</li> </ul> <p>In order to do these things, the model must be a modular component in the pipeline, and model results should rendezvous at a point where they can be compared, monitored, and selected based upon user-defined criteria. This design pattern can be achieved with an architecture called the rendezvous architecture.</p> <h2>The Rendezvous Architecture</h2> <p>The rendezvous architecture is a machine learning pipeline that allows multiple models to process inference requests and rendezvous at a point where user-defined logic can be applied to choose which ML result to return to the requester. Such logic could say, "Give me the fastest result," or "give me the highest confidence score after waiting 10 seconds." The rendezvous point also gives us a point where models can be monitored and requests can be captured where model results significantly disagree with each other.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image5-1610603365540.png" alt=""></p> <p>Note the emphasis on streams. Streams buffer requests in an infinite, resilient, and replayable queue. This makes it easy to hotswap models and scale ML executors in a microservices fashion. It also guarantees traceability for every inference request and response.</p> <p>If you'd like to learn more about the rendezvous architecture, read the highly recommended Machine Learning Logistics by Ted Dunning and Ellen Friedman. It was published in 2017 and <a href="https://www.oreilly.com/library/view/machine-learning-logistics/9781491997628/">is available as a free downloadable eBook</a>.</p> <h2>Conclusion</h2> <p>This was a story about how I used geo data with k-means clustering that relates to a topic that deeply affects a lot of people - wildfires! Anytime you have lat / long coordinates you have an opportunity to do data science with k-means clustering and visualization on a map. I hope this example illustrates the basics of k-means clustering and also gives some perspective on how machine learning models can be operationalized in production scenarios using streaming interfaces.</p>https://developer.hpe.com/spark-custom-streaming-sources/https://developer.hpe.com/spark-custom-streaming-sources/Thu, 14 Jan 2021 05:34:57 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Nicolas A Perez"], "publish": "2019-03-14T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>Apache Spark is one of the most versatile big data frameworks out there. The ability to mix different kinds of workloads, in memory processing and functional style, makes it desirable for anyone coming to code in the data processing world.</p> <p>One important aspect of Spark is that it has been built for extensibility. Writing new connectors for the <strong>RDD</strong> API or extending the <strong>DataFrame/Dataset</strong> API allows third parties to integrate with Spark with ease. Most people will use one of the built-in APIs, such as Kafka for streams processing or JSON/CVS for file processing. However, there are times where we need more specific implementations, closer to us. For example, we might have a proprietary database we use in our company, and there will not be a connector for it. We can simply write one, as we explained in this previous post <a href="/blog/XvlK6AnLW6cRQAzVL8XL/spark-data-source-api-extending-our-spark-sql-query-engine"><em><strong>Spark Data Source API. Extending Our Spark SQL Query Engine</strong></em></a>.</p> <p>Starting with Spark 2.0, we could create sources from streams, which gave life to the <em>Spark Structured Streaming API</em>. As we would imagine, there are some built-in streaming sources, Kafka being one of them, alongside <strong>FileStreamSource, TextSocketSource</strong>, etc.</p> <p>Using the new <em>Structured Streaming API</em> should be preferred over the old <strong>DStream</strong> API. However, the same problem as before presents again. How can we extend this new API, so we can use our own streaming sources? The answer to this question is in this blog post.</p> <h2>Extensibility Points</h2> <p>Let's start by reviewing the main components that we need to touch on in order to create our own streaming source.</p> <p>First of all, <strong>StreamSourceProvider</strong> is what indicates what source will be used as the stream reader.</p> <p>Secondly, <strong>DataSourceRegister</strong> will allow us to register our source within Spark, so it becomes available to the stream processing.</p> <p>Thirdly, <strong>Source</strong> is the interface that we need to implement, so we provide streaming source-like behavior.</p> <h2>Our Streaming Source</h2> <p>For the sake of this post, we will implement a rather easy streaming source, but the same concepts apply to any streaming source that you need to implement on your own.</p> <p>Our streaming source is called <strong>InMemoryRandomStrings</strong>. It basically generates a sequence of random strings and their length, which are viewed as a <strong>DataFrame</strong> of pairs.</p> <p>Since we want to keep it simple, we will store the batches in memory and discard them when the process is done. <strong>InMemoryRandomStrings</strong> is not fault-tolerant, since data is generated at the processing time in contrast to the built-in Kafka source, where data actually lives in a Kafka cluster. In most real-case scenarios, our data is consistently stored in advance systems that keep it secured and consistent; MapR Event Store for Apache Kafka and MapR Database are just a couple of these examples.</p> <p>We can start by defining our <strong>StreamSourceProvider</strong>, which defines how our <strong>Source</strong> is created.</p> <p>The class <strong>DefaultSource</strong> is our <strong>StreamSourceProvider</strong>, and we need to implement the two required functions, <strong>sourceSchema</strong> and <strong>createSource</strong>.</p> <pre><code class="language-scala">class DefaultSource extends StreamSourceProvider with DataSourceRegister { override def sourceSchema(sqlContext: SQLContext, schema: Option[StructType], providerName: String, parameters: Map[String, String]): (String, StructType) = { (shortName(), InMemoryRandomStrings.schema) } override def createSource(sqlContext: SQLContext, metadataPath: String, schema: Option[StructType], providerName: String, parameters: Map[String, String]): Source = { new InMemoryRandomStrings(sqlContext) } override def shortName(): String = "InMemoryRandomStrings" } </code></pre> <p><strong>InMemoryRandomStrings.schema</strong> is the fixed schema we are going to use for the example, but the schema can be dynamically passed in.</p> <p>The <strong>createSource</strong> function then returns an instance of <strong>InMemoryRandomStrings</strong> that is our actual <strong>Source</strong>.</p> <h2>InMemoryRandomStrings</h2> <p>Now, let's see <strong>InMemoryRandomStrings</strong> code in parts, so we can focus on all the details.</p> <pre><code class="language-scala">class InMemoryRandomStrings(sqlContext: SQLContext) extends Source { override def schema: StructType = InMemoryRandomStrings.schema override def getOffset: Option[Offset] = ??? override def getBatch(start: Option[Offset], end: Offset): DataFrame = ??? override def commit(end: Offset): Unit = ??? override def stop(): Unit = ??? } object InMemoryRandomStrings { lazy val schema = StructType(List(StructField("value", StringType), StructField("ts", LongType))) } </code></pre> <p><code>schema</code> returns the schema that our source uses; in our case, we know that the schema is fixed.</p> <p><code>getOffset</code> should return the latest offset seen by our source.</p> <pre><code class="language-scala">class InMemoryRandomStrings(sqlContext: SQLContext) extends Source { private var offset: LongOffset = LongOffset(-1) override def schema: StructType = InMemoryRandomStrings.schema override def getOffset: Option[Offset] = this.synchronized { println(s"getOffset: $offset") if (offset.offset == -1) None else Some(offset) } override def getBatch(start: Option[Offset], end: Offset): DataFrame = ??? override def commit(end: Offset): Unit = ??? override def stop(): Unit = ??? } object InMemoryRandomStrings { lazy val schema = StructType(List(StructField("value", StringType), StructField("ts", LongType))) } </code></pre> <p>Notice that we added a variable called <code>offset</code> that will keep track of the seen data. Then, we return <code>None</code> if our source has never seen any data, <code>Some(offset)</code> otherwise.</p> <p>Now, let's see how our source can produce some data; we will use a running thread for it. Please, notice the <strong>dataGeneratorStartingThread</strong> function.</p> <pre><code class="language-scala">class InMemoryRandomStrings(sqlContext: SQLContext) extends Source { private var offset: LongOffset = LongOffset(-1) private var batches = collection.mutable.ListBuffer.empty[(String, Long)] private val incrementalThread = dataGeneratorStartingThread() override def schema: StructType = InMemoryRandomStrings.schema override def getOffset: Option[Offset] = this.synchronized { println(s"getOffset: $offset") if (offset.offset == -1) None else Some(offset) } override def getBatch(start: Option[Offset], end: Offset): DataFrame = ??? override def commit(end: Offset): Unit = ??? override def stop(): Unit = incrementalThread.stop() private def dataGeneratorStartingThread() = { val t = new Thread("increment") { setDaemon(true) override def run(): Unit = { while (true) { try { this.synchronized { offset = offset + 1 val value = Random.nextString(Random.nextInt(5)) batches.append((value, offset.offset)) } } catch { case e: Exception => println(e) } Thread.sleep(100) } } } t.start() t } } object InMemoryRandomStrings { lazy val schema = StructType(List(StructField("value", StringType), StructField("ts", LongType))) } </code></pre> <p>In here, we have added a thread that generates random values and increments the offset while storing the value and offset on an internal buffer. The thread starts running as soon as our source is created. The <code>stop</code> function stops the running thread.</p> <p>At this point, we are only two functions away from our goal.</p> <p><code>getBatch</code> returns a <strong>DataFrame</strong> back to Spark with data within the passed offset range.</p> <pre><code class="language-scala">override def getBatch(start: Option[Offset], end: Offset): DataFrame = this.synchronized { val s = start.flatMap(LongOffset.convert).getOrElse(LongOffset(-1)).offset + 1 val e = LongOffset.convert(end).getOrElse(LongOffset(-1)).offset + 1 println(s"generating batch range $start ; $end") val data = batches .par .filter { case (_, idx) => idx >= s &#x26;&#x26; idx &#x3C;= e } .map { case (v, _) => (v, v.length) } .seq val rdd = sqlContext .sparkContext .parallelize(data) .map { case (v, l) => InternalRow(UTF8String.fromString(v), l.toLong) } sqlContext.sparkSession.internalCreateDataFrame(rdd, schema, isStreaming = true) } </code></pre> <p>We can see that we are getting the data from our internal buffer so that the data has the corresponding indexes. From there, we generate the <strong>DataFrame</strong> that we then send back to Spark.</p> <p>Finally, <code>commit</code> is how Spark indicates to us that it will not request offsets less or equal to the one being passed. In other words, we can remove all data from our internal buffer with an offset less than or equal to the one passed to <code>commit</code>. In this way, we can save some memory and avoid running out of heap space.</p> <pre><code class="language-scala">override def commit(end: Offset): Unit = this.synchronized { val committed = LongOffset.convert(end).getOrElse(LongOffset(-1)).offset val toKeep = batches.filter { case (_, idx) => idx > committed } batches = toKeep } </code></pre> <p>Now, we have completed our source; the entire code is the following:</p> <pre><code class="language-scala">class InMemoryRandomStrings(sqlContext: SQLContext) extends Source { private var offset: LongOffset = LongOffset(-1) private var batches = collection.mutable.ListBuffer.empty[(String, Long)] private val incrementalThread = dataGeneratorStartingThread() override def schema: StructType = InMemoryRandomStrings.schema override def getOffset: Option[Offset] = this.synchronized { println(s"getOffset: $offset") if (offset.offset == -1) None else Some(offset) } override def getBatch(start: Option[Offset], end: Offset): DataFrame = this.synchronized { val s = start.flatMap(LongOffset.convert).getOrElse(LongOffset(-1)).offset + 1 val e = LongOffset.convert(end).getOrElse(LongOffset(-1)).offset + 1 println(s"generating batch range $start ; $end") val data = batches .par .filter { case (_, idx) => idx >= s &#x26;&#x26; idx &#x3C;= e } .map { case (v, _) => (v, v.length) } .seq val rdd = sqlContext .sparkContext .parallelize(data) .map { case (v, l) => InternalRow(UTF8String.fromString(v), l.toLong) } sqlContext.sparkSession.internalCreateDataFrame(rdd, schema, isStreaming = true) } override def commit(end: Offset): Unit = this.synchronized { val committed = LongOffset.convert(end).getOrElse(LongOffset(-1)).offset val toKeep = batches.filter { case (_, idx) => idx > committed } batches = toKeep } override def stop(): Unit = incrementalThread.stop() private def dataGeneratorStartingThread() = { val t = new Thread("increment") { setDaemon(true) override def run(): Unit = { while (true) { try { this.synchronized { offset = offset + 1 val value = Random.nextString(Random.nextInt(5)) batches.append((value, offset.offset)) } } catch { case e: Exception => println(e) } Thread.sleep(100) } } } t.start() t } } object InMemoryRandomStrings { lazy val schema = StructType(List(StructField("value", StringType), StructField("ts", LongType))) } </code></pre> <h2>Using Our Custom Source</h2> <p>Now, we need to plug in our source into the Spark Structured Streaming API by indicating the correct format to be used.</p> <pre><code class="language-scala">val r = sparkSession .readStream .format("com.github.anicolaspp.spark.sql.streaming.DefaultSource") .load() </code></pre> <p>In here, we use the regular <code>.readStream</code> API and specify that the stream format is our implementation of <code>StreamSourceProvide</code>, that is: <em><strong>com.github.anicolaspp.spark.sql.streaming.DefaultSource.</strong></em></p> <p>Now we can query our streaming source as any other <strong>DataFrame</strong>.</p> <pre><code class="language-scala">r.createTempView("w") sparkSession .sql("select ts, count(*) as c from w group by ts order by ts, c desc") .writeStream .format("console") .outputMode(OutputMode.Complete()) .start() .awaitTermination() </code></pre> <p>The output will look similar to this:</p> <pre><code class="language-markdown">------------------------------------------- Batch: 3 ------------------------------------------- +---+---+ | ts| c| +---+---+ | 0| 81| | 1| 78| | 2| 74| | 3| 82| | 4| 80| +---+---+ </code></pre> <p>What we see is a continuous aggregation of the data generated by our source.</p> <h2>Conclusions</h2> <p>Apache Spark is the way to go when processing data at scale. Its features outperform almost any other tool out there. Also, it can be extended in many different ways, and as we can see, we can write our own data sources and streaming sources, so they can be plugged into our Spark code with ease.</p> <p>Originally posted January 14, 2019, <a href="https://hackernoon.com/spark-custom-stream-sources-ec360b8ae240">here</a>.</p> <p><em><strong>The entire project and source code can be found here: <a href="https://github.com/anicolaspp/SparkStreamSources">SparkStreamSources</a>.</strong></em></p> <p><em>Happy Coding.</em></p>https://developer.hpe.com/journey-science-in-telecom-take-customer-experience-to-the-next-level/https://developer.hpe.com/journey-science-in-telecom-take-customer-experience-to-the-next-level/Thu, 14 Jan 2021 05:27:21 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ronald van Loon", "publish": "2017-06-03T12:00:00.000", "tags": "mapr" </code></pre> <hr> <p>Journey Science, being derived from connected data from different customer activities, has become pivotal for the telecommunications industry, providing the means to drastically improve the customer experience and retention. It has the ability to link together scattered pieces of data and enhance a telco business’s objectives. Siloed approaches are becoming obsolete – take call centers as an example – there is only so much that you can do with data from only one system.</p> <p>By using insights from customer journey analytics, telco businesses can better measure the user experience, and make informed decisions for refining it. The data not only allows them to take a proactive approach towards customer satisfaction, but enables the prediction of future failures as well. With customer journey analytics, you can evaluate the touchpoints to journeys, and revamp your strategies to better cater to customers’ needs.</p> <p>In the telecom industry, it is difficult for a business to effectively manage the massive volume of data with the existing systems and technology. There are several aspects where telecom companies need to make improvements, such as reduce costs, improve customer experience, increase conversion rates, and many more. To do so, they need to derive meaning from the collected data by finding connections among them. This linked data is also known as journeys. Journeys provide you with relevant data that enable you to make well-grounded business decisions by looking at customer transactions as a whole and determining where direct improvements are needed.</p> <h2>Customer Journey Analytics is Transforming Telecommunications</h2> <p>Many leading telco businesses are embracing the Journey Science concept and deem it to be the best way to make greater impact on the target audience. One good way to better understand digital journeys is through a multi-channel, end-2-end, view. Journey Sciences, at its best, provides enhanced data accessibility and increased analytics agility, and helps in weaving together disparate pieces of data. This makes it possible for telco businesses to link together structured and unstructured data back to their strategic objectives, and quickly modify them to ensure they cope with the evolving customer demands. However, in order to get insight into customer experience through journey analytics, it is critical to focus not only on the individual moments but the customers’ end-to-end experiences as well.</p> <h2>Customer Experience Boost</h2> <p>The main benefit of customer journey analytics for telco companies is that it enables them to better recognize customer needs and assess their satisfaction level. While most people think Journey Science is all about marketing, it mainly focuses on the services domain. For example, a customer seeking technical support for their device has multiple paths to resolution. Journey Science enables businesses to evaluate each step of the journey experience and figure out the critical points that could negatively impact customer experience. With this kind of information, businesses can develop strategies to overcome hurdles customers face on all such touchpoints, resulting in improved customer experience.</p> <h2>Improving Customer Journeys through Transparency</h2> <p><em>Connecting the Dots</em></p> <p>For improving customer experience, it is essential to connect all the data down to the individual customer level to fully understand the required changes. For telco businesses to completely understand customer journeys, they must gather data from many different channels and track the individual journey the customer experiences. Typically, more than 50 percent of customers make multi-channel journeys; meaning, in order to understand their behavior, establishing connection among all the data is extremely important. Because of the deep roots of technology in today’s common lifestyle, many journeys start from digital channels, but eventually go into a human channel for completion.</p> <p><em>Utilizing Aggregate and Raw Data</em></p> <p>Apart from giving a complete picture of customer journeys, the analytics let you tap into different levels of aggregation, allowing you to view raw data as well. With journey mapping, telco businesses can benefit from both in-depth data points and aggregated data sets. Since a single customer journey can compile hundreds of thousands of data points, having aggregated views makes it much easier to pinpoint and prioritize the problematic areas. On the other hand, some journeys may yield unclear results, for example, unusual behavior of a customer on a webpage. In such a case, having access into the raw data renders the ability to focus on one key area and get invaluable insights.</p> <p><em>Making Changes through Data Availability</em></p> <p>Effective utilization of data from customer journey analytics allows telco to revamp their strategy as well as make smaller improvements on a continuous basis. Getting immediate feedback regarding a certain change is critical for understanding its impact. You can determine whether the intended results will be realized, or if you should scale-up or sustain the change. However, a manual, project-based approach that only provides an overview of the required data will not be enough to transform journeys successfully. Instead, you should opt for an agile, iterative, analytic approach that uses continuous data availability.</p> <p>It wouldn’t be wrong to say that all those ad-hoc, manual, project-based approaches using snapshots of data have severe limitations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/picture1-1610602144488.jpg" alt=""></p> <p><em>Better data accessibility to more than 18 telco raw data sources as a prerequisite</em></p> <h2>How the Customer Journey differs in both Fixed and Mobile Telco</h2> <p><em>Mobile (mobile data usage, subscriptions, charges, and mobile data access)</em></p> <p>Several small customer journeys can be linked together to make improvements to a mobile telco operation. One great way is through customer engagement, i.e. moving down to individualized journeys of each customer instead of mass-segmentation. Journey Science opens doors for mobile telco companies to take personalization up a notch and provide customized recommendations based on the journeys of each customer. You should also utilize real-time context to enhance customer engagement for better results.</p> <p>Mobile customer experience is comprised of several touchpoints where a subscriber interacts with a service provide agent – it can be during retail, billing, customer support, visible marketing campaigns, and others. Consider three customers below that have 3 different journeys to perform the same action.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/picture2-1610602154635.jpg" alt=""></p> <p><em>Fixed line providers (phone, internet, entertainment)</em></p> <p>Fixed line providers have an additional interaction channel with field technicians being deployed to customers’ homes for service. These field service appointments are a major part of customer experience and often have significant variability for different customers. Consider the following journey which involves multiple appointments, agent phone calls, and delays:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/picture3-1610602163126.jpg" alt=""></p> <p><em>Improve key journeys for fixed telco’s</em></p> <h2>Journey Science is Moving towards Predictive Analytics</h2> <p>The Journey Science concept is increasingly becoming popular across the telco industry, as it greatly benefits by assessing journeys of individual customers and allows them to develop customized strategies. Moreover, it allows telco businesses to anticipate the potential pitfalls leading to negative customer experience and prevent it altogether. By tapping into the data from customer journey, telco can streamline their operations and provide a better, more satisfying experience to their customers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/picture4-1610602170940.jpg" alt=""></p> <p><em>Derived value from Customer Journey data by Journey Science &#x26; Journey Analytics</em></p> <p>In today’s world, customer satisfaction is the keystone for success in every industry, including telco. Businesses should turn to the Journey Science movement, and optimize their processes by carefully analyzing customer journeys and making improvements accordingly. Effective utilization of customer journey analytics leads to better redesigning efforts, ultimately reducing costs, enhancing customer experience, and stretching bottom-line.</p> <p><em>This article was originally published on February 1, 2017 at <a href="http://www.datasciencecentral.com/profiles/blogs/journey-science-in-telecom-take-customer-experience-to-the-next">Data Science Central</a>. Follow Ronald van Loon on <a href="https://www.linkedin.com/in/ronald-van-loon-5411a/">LinkedIn</a> or <a href="https://twitter.com/Ronald_vanLoon">Twitter</a>.</em></p>https://developer.hpe.com/introducing-the-hpe-gl20-iot-gateway/https://developer.hpe.com/introducing-the-hpe-gl20-iot-gateway/Tue, 12 Jan 2021 10:36:54 GMT<img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/h4-1610447678726.jpg" width="80%" height="80%"> <p>HPE IoT Gateways enable organizations to rapidly acquire, analyze and take action on real-time data as it’s being collected for additional analysis at a later stage. Bringing computing and analytics close to the edge accelerates the speed of your decision-making and reduces the chance of lost opportunities or a missed red flag. In this post, I’ll be discussing the HPE GL20 IoT Gateway, a fun device for the edge. The HPE GL20 has an 8-bit digital input/output (DIO) capability designed specifically to address IoT needs. For detailed product information, check out the <a href="https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=c04884769">Quick Spec</a>.</p> <p>In this post, I would like to focus on the DIO capability and demonstrate how to develop using the digital input/output pin. After reading this tutorial, you will understand how to work with the DIO feature using Python. Two GitHub repositories shown below are highly related to this blog. Make sure you can access these:</p> <p>• <a href="https://github.com/helloezmeral/HPE-GL20-GPIO">GitHub – helloezmeral/HPE-GL20-GPIO</a></p> <p>• <a href="https://github.com/helloezmeral/HPE-GL20-gRPC">GitHub - helloezmeral/HPE-GL20-gRPC</a></p> <p>The first repository contains information used for controlling input/output pins using Python directly, and the second repository is about wrapping the code into a gRPC service and preparing a docker image for developers to access DIO on the HPE GL20 easily.</p> <blockquote> <p><strong>Note:</strong> gRPC is a high-performance, open source universal RPC framework. This open source remote procedure call system was initially developed at Google in 2015.</p> </blockquote> <p>Here we go. Let’s have some fun!</p> <h2>Potential use case</h2> <p>Digital input/output pins tend to be important for edge applications because they need to interact with microcontrollers, like IoT sensors or actuators, which may not have the ability to communicate wirelessly. It can be very handy to work with DIO. You can connect the output pin to LEDs for indication, relays, or even interrupt pins on microcontrollers to initiate an edge process.</p> <p>For example, you can design an IoT device that sleeps all the time to reserve battery power and wake it up using output pins. Input pins on an HPE GL20 can be used to measure the environment or used as a hardware control button. A touch sensor that sends a signal high when touching the board, or a PIR sensor that sends a signal high when human passed by, can be easily integrated with HPE GL20. For example, if you want to design an edge device that detects any human that passes by and takes a photo of them and sends back to the cloud, HPE GL20 is a great device to use for achieving this target.</p> <h2>Demo</h2> <p>To illustrate how to use the DIO feature on an HPE GL20, I set up a touch sensor as the input to the HPE GL20 and two LEDs as the output. See the picture below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/h3-1610447669565.png" alt="h3"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/h2-1610447657582.jpg" alt="h2"></p> <p>DIO is enabled by an i2c device internally. Thus, we need to install an i2c environment before we can use it. In Bash, you may need to install with <code>sudo apt install i2c-tools</code>. In Python, you may need to install with <code>sudo pip3 install smbus2</code>. You'll need to determine which bus the i2c-gpio chip is connected to. Run <code>sudo i2cdetect -l</code> to obtain the bus number. After installing the complete package, you can start downloading the file <strong><em>pyGL20.py</em></strong> into the same folder as your Python script.</p> <pre><code class="language-bash">sudo apt install i2c-tools sudo pip3 install smbus2 sudo i2cdetect -l wget https://raw.githubusercontent.com/helloezmeral/HPE-GL20-GPIO/master/pyGL20.py </code></pre> <p>Before writing the script, we need to know how the circuit is routed internally. The setting of input pin is pull-up, which means that when nothing is connected to the pin, it will return a high signal. The output pin is configured as open collector, which means it controls the pins connect to the ground or not. In technical terms, this is referred to as output low or high-impedance. Now you can start writing your code. The syntax of writing GPIO code is highly similar to Arduino code. First, import the module with <code>from pyGL20 import GPIO</code>. Next, initiate the i2C bus communication channel. If your GPIO chip is connected to i2c-0, you should write <code>IO = GPIO(0)</code>. Once this is done, you can control all pins with commands like <code>IO.digitalWrite(IO.PIN6, True)</code> to set pin6 output high, or <code>IO.digitalRead(IO.PIN0)</code> to read the level of pin0. In the Arduino world, you can see signature <code>setup()</code> and <code>loop()</code> functions. You can do the same thing here with the <code>while(True):</code> loop. Another useful feature in Arduino is <code>delay(1000)</code>. This command creates a 1000ms delay of the processor. In Python, you can do a similar thing with importing the time module and calling <code>time.sleep(1)</code> for sleeping one second.</p> <pre><code class="language-python"># blink.py from pyGL20 import GPIO import time """ Please run :: sudo i2cdetect -l :: to find the coresponding i2c-x bus of "SMBus I801 adapter at f040" replace the number x of the below variable. """ IO = GPIO(0) # This means i2c-0 while True: print(IO.digitalWrite(IO.PIN6, True)) time.sleep(0.1) print(IO.digitalWrite(IO.PIN(6), False)) # both work time.sleep(0.1) </code></pre> <p>You can find more information about this in this repository: <a href="https://github.com/helloezmeral/HPE-GL20-GPIO">https://github.com/helloezmeral/HPE-GL20-GPIO</a></p> <h2>Using gRPC service</h2> <p>To further assist you in writing a GPIO application, I have prepared a docker-compose file that you can use to deploy a gRPC server to control digital I/O pins within a second. gRPC is a modern open-source high performance RPC framework. With this service installed, you can control the GPIO of an HPE GL20 over the network. I have prepared all necessary files below in a Github release which includes a docker-compose yaml file and two gRPC code files. <strong><em>GL20_pb2.py</em></strong> contains generated request and response classes and <strong>_ GL20_pb2_grpc.py_</strong> contains our generated client and server classes.<br> I have prepared all necessary files <a href="https://github.com/helloezmeral/HPE-GL20-gRPC/releases/tag/v0.9">here</a> in a GitHub release which includes a docker-compose yaml file and two gRPC code files.</p> <p>First, set up the gRPC service on the HPE GL20. Download the file <strong><em>docker-compose.yaml</em></strong> and run it as <code>docker-compose up</code>/<code>docker-compose up -d</code> (i.e.: flag d represents detached mode). Hooray! You just deployed a gRPC service for controlling the HPE GL20 GPIO.</p> <p>The next step prepares the required file for the client call. <strong><em>GL20_pb_2.py</em></strong> and <strong><em>GL20_pb2_grpc.py</em></strong> is required since it contains code used to define the calling procedure and information exchange. The syntax of controlling the GPIO pins is nearly the same as the first one. You can read the example code in gRPC folder of the repository shown below.</p> <p>You can read the example code in gRPC folder of the GitHub repository <a href="https://github.com/helloezmeral/HPE-GL20-gRPC">here</a>. This is the docker image for gRPC microservice for you to use the GPIO of the GL20.</p> <h2>Summary</h2> <p>In this tutorial, I have shown you two ways to write a simple Python script to access HPE GL20 DIO. The first way utilizes Bash utility named <strong><em>i2c-tools</em></strong> and Python module called <strong><em>smbus2</em></strong> to access the DIO directly. This is suitable for those who want to take all control of your code. I have also prepared a second way for you with an easy-to-use gRPC service docker image exposing DIO over network. You can control DIO within the same device or in the cloud remotely.</p> <p>I just wanted to note that using <code>time.sleep(1)</code> as a delay is not always a wise option when it comes to writing an embedded application. During delay, Python does nothing but wait one second, which causes inefficiency in your code. Concurrency in Python like threading or multiprocessing is recommended. However, this really isn't today's topic. If you want to learn more about this, you might want to check out these blog posts on Python Concurrency.</p> <p>• <a href="/blog/understanding-concurrency-in-python-part-1-threading">HPE Developer | Understanding Concurrency in Python Part 1 – Threading</a><br> • <a href="/blog/understanding-concurrency-in-python-part-2-multiprocessing">HPE Developer | Understanding Concurrency in Python Part 2 – Multiprocessing</a><br> • <a href="/blog/understanding-concurrency-in-python-part-3-asyncio">HPE Developer | Understanding Concurrency in Python Part 3 – Asyncio</a><br> • <a href="https://docs.python.org/3/library/concurrent.futures.html">concurrent.futures — Launching parallel tasks</a></p> <p>Another quick note for you: sometimes you might require serial/USART communications with other devices. A Python module called <strong><em>PySerial</em></strong> is handy for dealing with serial communications.</p> <p>There are numerous features on HPE GL20. I'll just stop here in my discussion of the digital input/output capability of HPE GL20. I hope you enjoyed my tutorial on how to use DIO with the HPE GL20 and find many great ways to apply this feature. If you have any questions regarding this blog post, you can drop a message to me. My email is: <a href="mailto:cenz@hpe.com">cenz@hpe.com</a> . Or reach me via <a href="https://hpedev.slack.com/">HPE DEV slack workspace</a>: <strong><em>@hpe.cenz</em></strong>. Happy hacking.</p>https://developer.hpe.com/architecting-the-worlds-largest-biometric-identity-system-the-aadhaar-ex/https://developer.hpe.com/architecting-the-worlds-largest-biometric-identity-system-the-aadhaar-ex/Thu, 07 Jan 2021 23:06:37 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Michele Nemschoff", "publish": "2015-02-13T08:00:00.000Z", "tags": "use-case" </code></pre> <hr> <p>Dr. Pramod Varma, Chief Architect and Technology Advisor to Unique Identification Authority of India (UIDAI), gave an informative talk titled “Architecting World's Largest Biometric Identity System - Aadhaar Experience” at the Strata + Hadoop World 2014 conference held in New York.</p> <p>Dr. Varma began his talk by talking about why the Aadhaar project was created. In India, the inability to prove one’s identity is one of the biggest barriers that prevents the poor from accessing benefits and subsidies. India is a country with 1.2 billion residents in over 640,000 villages. The Indian government spends $50 billion on direct subsidies (food coupons for rice, cooking gas, etc.) every year. Both public and private agencies in India require proof of identity before providing services or benefits to those living in India.</p> <p>Until the introduction of the Aadhaar program, there was no verifiable identity number program that both residents and agencies could use. As a result, every time Indian residents tried to receive benefits, they had to undergo an arduous personal identification process. What made it even more difficult was that the various service providers had different document and information requirements. This made it especially hard for India’s poor residents, who often lacked documentation and found it difficult to access services.</p> <p>The <a target='\_blank' href='http://en.wikipedia.org/wiki/Unique_Identification_Authority_of_India'><u>Unique Identification (Aadhaar) project</u></a> was created in order to provide every resident of India with a unique identification number that can be used to access a variety of services and benefits. The project enables residents in India to receive food coupons, receive cooking gas deliveries, open checking accounts, apply for loans, insurance, pensions, property deeds, etc. In addition, the program makes it possible for the Indian government to make sure that welfare benefits go directly to the right person.</p> <p>They are using MapR to build the project’s biometric database (the largest in the world), which can verify a person’s identity within 200 milliseconds. The database includes an iris scan, digital fingerprints, a digital photo, and text-based data for every resident.</p> <p>The uniqueness of an Aadhaar identity makes it possible to eliminate fake and duplicate accounts, and the online authentication system provides a mechanism for the paperless, electronic and instantaneous verification of a person’s identity. Currently, over 690 million Aadhaars (IDs) have been issued so far, with 10 million new IDs issued every 10 days. The target is to complete 1 billion IDs by 2015.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/aadhaar-number-id-1610061218903.png" alt="Aadhaar Project ID Number"></p> <p><strong>Architecture at a Glance</strong></p> <p>The entire technology architecture behind Aadhaar is based on principles of openness, linear scalability, strong security, and most importantly vendor neutrality. The backbone of the Aadhaar technology was developed using the following principles:</p> <p><strong>Open architecture</strong> – Building the Aadhaar system with true openness meant that they relied on open standards to ensure interoperability; the platform approach with open APIs made it possible for the ecosystem to build on top of Aadhaar APIs; vendor neutrality was ensured across the application components by using open and standard interfaces. The identity system was designed to work with any device, any form factor, and on any network.</p> <p><strong>Design for scale</strong> – The Aadhaar system is expected to issue more than 1.2 billion identities, and will continue to grow as the resident population expands. Since every new enrollment requires biometric de-duplication across the entire system, every component needs to scale to very large volumes. This meant that the system needed to be able to handle hundreds of millions of transactions across billions of records doing hundreds of trillions of biometric matches every day. In addition all online services such as Aadhaar authentication, e-KYC services, and update services must work with high availability and sub-second performance. In order to achieve such massive scalability, the program established network and data center load balancing and a multi-location distributed architecture for horizontal scale.</p> <p><strong>Data Security</strong> – The security and privacy of one’s data is a foundation of the Aadhaar system. The system uses 2048-bit PKI encryption and tamper detection using HMAC in order to ensure that no one can decrypt and misuse the data. Resident data and raw biometrics are always kept encrypted, even within UIDAI data centers. In addition, the system does not keep track of any transactional data.</p> <p><strong>Enrollment</strong></p> <p>Since the Aadhaar system launched four years ago, the Aadhaar platform has grown in capability. Over 690 million Aadhaar numbers have been issued so far using the system, making it the largest biometric identity repository in the world, resulting in 600+ trillion biometric matches every day. They are on target to complete 1 billion Aadhaars by 2015.</p> <p>The amount of biometric data that is collected per person is approximately 3-5MB per person, which maps to a total of 10-15 petabytes of data.</p> <p>60,000-80,000 small laptops that include the installed Aadhaar system are used in remote villages. Laptops, generators, chairs and tables are transported to the villages via donkeys, and the systems are then set up in each camp. There are currently 150,000 certified operators and supervisors who are trained and certified to operate the enrollment station. On average, each station enrolls about 50 people per day, resulting in approximately 1 million new enrollments every day.</p> <p><strong>Authentication</strong></p> <p>The Aadhaar authentication system provides multi-factor authentication, based on “what you have” (something the user uniquely has, such as a mobile phone or laptop that accesses email, etc.) and “who you are” using resident fingerprints patterns, iris scans, a signature, or handwriting. By combining one or more factors, the resident’s authentication could be strengthened. In addition, Authentication User Agency (AUA)-specific factors such as ATM cards or smart cards or passwords may also be used in conjunction with Aadhaar authentication to further strengthen user authentication.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/aadhaar-project-server-provider-1610061236041.png" alt="Aadhaar project server provider"></p> <p><em>Authentication Overview</em></p> <p><strong>Aadhaar in Action</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/aadhaar-action-1-1610061249888.png" alt="Aadhaar in action"></p> <p><strong>Payments through micro ATMs</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/aadhaar-action-2-1610061264885.png" alt="Aadhaar project in action"></p> <p><strong>Future Plans</strong></p> <p>Future plans call for Aadhaar to be used for digital signatures, electronic documents and digital locker services. Also, Aadhaar could also be used for college/university certificates, as well as credit registries. Dr. Varma emphasized that the Aadhaar system is a great example of using Hadoop technology to make a difference to every resident of India. The Aadhaar identity platform and Aadhaar-enabled applications are helping a billion people in India to participate in the digital economy, and benefit from government, public and private sector services that are tailored to them.</p>https://developer.hpe.com/apache-spark-as-a-distributed-sql-engine/https://developer.hpe.com/apache-spark-as-a-distributed-sql-engine/Thu, 07 Jan 2021 22:53:50 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-03-17T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>SQL has been here for awhile and people like it. However, the engines that power SQL have changed with time in order to solve new problems and keep up with demands from consumers.</p> <p>Traditional engines such as Microsoft SQL Server had some problems with scalability that they have solved with time and cloud-based solutions. On the other hand, others have been built from the ground up to work in a distributed environment so they can put performance at the top of their priority list.</p> <p>There is not a tool for all use cases. In fact, we believe that tools are built with use cases in mind, to solve a specific problem. Then they evolve to a more mature stage where they can be used to solve many other problems.</p> <p>In a traditional SQL environment, the data is represented by tables and the relationships between them, but this representation is sometimes not enough, so new tools have been developed to solve this. We can find everywhere organizations that don’t use relational databases; instead, they prefer to go to the non-SQL ones.</p> <h2>Hadoop</h2> <p>In the Hadoop world, we have a variety of different query engines; each of them has its own particularities, and they each solve a wide variety of problems.</p> <p>In any Hadoop distribution, we can find Apache Hive, a SQL-like tool that offers data warehouse infrastructure and capabilities for big data queries and analysis.</p> <p>Depending on the Hadoop distribution, we can also find Apache Impala and Apache Drill. All of them offer more or less the same capabilities, sharing a common goal. We can use SQL or SQL-like languages to query data stored in Hadoop. They also have their own limitations and advantages that you should be aware of.</p> <h2>Apache Spark</h2> <p>Apache Spark is a lightning-fast cluster computing that can be deployed in a Hadoop cluster or stand alone mode. It can also be used as a SQL engine like the others we mentioned. Spark, however, offers some advantages over the previous ones.</p> <p>Spark exposes APIs for different languages such as Scala, Java, Python, and R. This makes it accessible by many types of people, such as developers, data scientists, and those with statistics experience.</p> <p>Interactive algorithms are easily implemented in Spark, especially machine learning ones.</p> <p>Let’s walk through an example of how to use Spark as a SQL engine.</p> <h2>Exploring Our Data Source</h2> <p>Our data set is a simple folder with a few terabytes in CSV-formatted files, and each file is about 40MB each. The size of the files does not affect the performance, because they are stored in a MapR cluster. MapR take cares of the <a href="https://blog.cloudera.com/the-small-files-problem/">Hadoop small file problem</a> as I explain in <a href="https://medium.com/hackernoon/how-mapr-improves-our-productivity-and-simplify-our-design-2d777ab53120">this post</a>.</p> <p>Because we are using MapR, copying files to the cluster is quite easy, since we have mounted a volume to our local file system.</p> <p>In order to mount the MapR volume, we run this command:</p> <pre><code class="language-bash">sudo mount_nfs -o "hard,nolock" 10.21.112.209:/mapr/mapr.domain.com/datalake /Users/anicolaspp/mapr/ </code></pre> <p>Now, if we run POSIX commands again in our local folder, they will in fact be executed in the MapR cluster.</p> <h2>Preparing the Environment for Auto Schema Discovery</h2> <p>We are going to create a Spark application using Scala that will allow us to execute SQL statements over our data stored in the MapR Distribution.</p> <p>In <a href="https://medium.com/@anicolaspp/sbt-scala-and-spark-6a57c0a2623a">this post</a> I explained how to create an application in Spark and the previous steps we need to follow.</p> <p>Our app class will look as follows:</p> <pre><code class="language-scala">/** * Created by anicolaspp. */ import org.apache.spark import org.apache.spark._ import org.apache.spark.sql.hive.HiveContext import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 import org.apache.spark.sql.{Row, SQLContext} import org.apache.spark.sql.types.{StringType, StructField, StructType} object app { def main(args: Array[String]) { val conf = new SparkConf().setAppName("testing") val sc = new SparkContext(conf) val sql = new HiveContext(sc) sql.setConf("hive.server2.thrift.port", "10001") val delimiter = "\t" val data = sc.textFile("datalake/myTestDataFolder/") val headers = data.first.split(delimiter) val schema = StructType(headers.map(h => StructField(h, StringType))) val rowRDD = data.map(p => Row.fromSeq(p.split(delimiter))) val dataFrame = sql.createDataFrame(rowRDD, schema) dataFrame.registerTempTable("someTableName") HiveThriftServer2.startWithContext(sql) while (true) { Thread.`yield`() } } } </code></pre> <p>Let’s review our code.</p> <p>First, we create the Spark Context based on a Config object.</p> <pre><code class="language-scala">val conf = new SparkConf().setAppName("testing") val sc = new SparkContext(conf) val sql = new HiveContext(sc) </code></pre> <p>Then, we set the thrift port to avoid conflicts with other components such as Hive.</p> <pre><code class="language-scala">sql.setConf("hive.server2.thrift.port", "10001") </code></pre> <p>Now, we set our CSV delimiter that in this case is the tab character. We also set the location of our data set by creating a Resilient Distributed Dataset (RDD) using the Spark Context (sc).</p> <pre><code class="language-scala">val delimiter = "\t" val data = sc.textFile("datalake/myTestDataFolder/") </code></pre> <p>At this point, we want to be able to serve our data without worrying about the schema of our file; we want a self-service BI environment as I explained <a href="https://medium.com/hackernoon/how-mapr-improves-our-productivity-and-simplify-our-design-2d777ab53120">here</a>. Using the headers from our data files, we can create the schema automatically, so we don’t have to worry about schema changes in the future. Once we have the schema, we create a DataFrame that we are going to expose in order to be queried using SQL.</p> <pre><code class="language-scala">val headers = data.first.split(delimiter) val schema = StructType(headers.map(h => StructField(h, StringType))) val rowRDD = data.map(p => Row.fromSeq(p.split(delimiter))) val dataFrame = sql.createDataFrame(rowRDD, schema) </code></pre> <p>The only part missing is the one that registers our data set as a table in the Hive meta store; we do that by doing:</p> <pre><code class="language-scala">dataFrame.registerTempTable("someTableName") HiveThriftServer2.startWithContext(sql) </code></pre> <p>We have a loop just to keep our app alive. Note that RDD transformations are lazy and they will be only executed when a query is submitted for execution.</p> <h2>Deploying Our Application</h2> <p>We build and test our app using SBT, and the resulting .jar can be copied to the cluster in the same way we copy files in our local file system.</p> <p>cp pathToOurJar/app.jar /Users/anicolaspp/mapr/testing</p> <p>Remember this is possible because we have previously mounted a MapR volume in our local file system.</p> <p>Now we need to submit our application in the cluster, and we do that by using the spark-submit command. Detailed documentation about submitting Spark applications can be found on the <a href="https://spark.apache.org/docs/latest/running-on-yarn.html">Spark website</a>.</p> <p>In our cluster, we run:</p> <pre><code class="language-bash">/spark-submit --master yarn /mapr/mapr.domain.com/datalake/testing/testing_2.10-1.0.jar </code></pre> <p>Our application should start running on YARN as we indicated when submitting it.</p> <p>Our SQL engine is ready to be queried, so let’s move forward and test it out.</p> <h2>SQL Clients</h2> <p>An easy way to test our SQL engine is to run <a href="https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93NewCommandLineShell">beeline</a>, a command line tool that works as an SQL client.</p> <p>We can find beeline in the Spark bin folder. To start it, we type ./beeline.</p> <p>Within beeline, we need connect to the end point we have defined in our application, so we run:</p> <pre><code class="language-bash">!connect jdbc:hive2://localhost:10001 </code></pre> <p>We should be ready to run SQL statements, but let’s verify we can see the table we registered.</p> <pre><code class="language-bash">show tables; </code></pre> <p>Spark SQL will return a table with the registered tables including the one we registered in our application (someTableName).</p> <p>In the same way, we can connect using other clients such as Microstrategy or Tableau. We have tried both and they both can build and execute queries on tables registered by Spark applications. We can also combine different sources (Spark SQL, MS SQL Server, Hive, Impala, etc.) which gives us the flexibility of combining relational sources with non-relational data.</p> <p>Spark SQL performs quite well and often better than the other providers in Hadoop, but be aware that performance can be degraded under certain conditions and use cases.</p> <h2>Why Apache Spark</h2> <p>Certainly, Spark SQL offers some of the functionalities that other tools have within Hadoop. However, the possibility of exploring complex data sets is rather unique to Spark, since we can code custom serialization / deserialization processes in our application. Using Spark SQL, we can connect to any data source and present it as tables to be consumed by SQL clients. This is as easy as changing how we ready the data in those sources by changing our serializer in our application.</p> <h2>Endings</h2> <p>There are very useful tools that we can use within Hadoop to query data in an SQL fashion and all of them have their advantages. The Spark SQL module from Apache Spark offers some flexibility that others lack while keeping performance as one of the main priorities.</p> <p>Spark is not the only tool you can use, but we strongly advise that you include it in big data solutions where SQL statements are to be executed. You might need to use a mix of different tools, but Spark should be an important part of the system you are building.</p> <p><a href="https://medium.com/@anicolaspp/apache-spark-as-a-distributed-sql-engine-4373e254e0f9">View the original.</a></p>https://developer.hpe.com/2021-January-06/https://developer.hpe.com/2021-January-06/Wed, 06 Jan 2021 06:00:00 GMThttps://developer.hpe.com/enabling-python-3-with-opensslfips-on-microsoft-windows/https://developer.hpe.com/enabling-python-3-with-opensslfips-on-microsoft-windows/Mon, 21 Dec 2020 08:48:53 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/fips-compliant-1611244310749.png" alt=""></p> <p><strong>Introduction</strong></p> <p>Federal Information Processing Standard (FIPS) are a set of encryption algorithms and is mandatory in all computer systems and software used by non-military American government agencies, government contractors and vendors who work with the agencies. When new software is developed, it needs to be FIPS-compliant. Thus, there is a need to enable Python with FIPS, but the default Python package comes without FIPS as shown in screenshot below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/openssl_init-1609929682543.PNG" alt=""></p> <p>This blog will explain, step-by-step, how to enable Python 3 with the OpenSSL/FIPS standard on a Microsoft Windows platform so that any new software compiled out of it, is FIPS-compliant.</p> <p><strong>Prerequisites</strong></p> <ul> <li>Cygwin environment</li> <li>Python 3</li> <li>Visual Studio 2017</li> </ul> <p><strong>STEPS</strong></p> <p><strong>Step 1</strong> Download the OpenSSL and FIPS source from <a href="http://www.openssl.org">http://www.openssl.org</a> and the Python 3 source from <a href="http://www.python.org">http://www.python.org</a></p> <p><strong>Step 2</strong> Convert all symlinks in the archive to regular files. This is done in Linux, Cygwin environment.</p> <p>Untar</p> <pre><code class="language-bash">$ tar -zxvf openssl-fips-2.0.16.tar.gz $ tar -zxvf openssl-1.0.2u.tar.gz $ tar -zxvf Python-3.8.6.tgz </code></pre> <p>Zip -9</p> <pre><code class="language-bash">$ zip -9 -r openssl-fips-2.0.16.zip openssl-fips-2.0.16 $ zip -9 -r openssl-1.0.2u.zip openssl-1.0.2u $ zip -9 -r Python-3.8.6.zip Python-3.8.6 </code></pre> <p>Finally, unzip in Windows to the c:\work\ folder.</p> <p><strong>Step 3</strong> Install NASM for x64 and copy the contents of C:\Program Files\NASM to the C:\work\openssl-1.0.2u\ folder.</p> <p>Note: NASM is a netwide assembler program and can be downloaded from [<a href="https://www.nasm.us/pub/nasm/releasebuilds">https://www.nasm.us/pub/nasm/releasebuilds</a>]</p> <p><strong>Step 4</strong> Build the FIPS module using the VS 2015 Native Tools Command prompt.</p> <pre><code class="language-bash">> cd openssl-fips-2.0.16 > ms\do_fips </code></pre> <p>Rename the folder “out32dll” to “lib”.</p> <p>Rename the folder “util” to “bin”.</p> <p>Move “fips_standalone_sha1.exe” from “lib” to “bin”.</p> <p>This is done so that OpenSSL can compile in the next steps.</p> <p><strong>Step 5</strong> Build OpenSSL module using the VS 2015 Native Tools Command prompt.</p> <pre><code class="language-bash">cd openssl-1.0.2u perl Configure VC-WIN64A no-zlib no-idea no-mdc2 no-rc5 no-ssl2 no-ssl3 fips --with-fipslibdir=C:\usr\local\ssl\fips-2.0 ms\do_win64a nmake -f ms\nt.mak all nmake -f ms\nt.mak install </code></pre> <p>The openssl.exe and libeay32.dll and ssleay32.dll files are generated be in the C:\usr\local\ssl\bin\ folder.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221151001541-1608549015693.png" alt=""></p> <p><strong>Step 6</strong> Unzip Python in Windows and create an 'externals' directory at the root (c:\work\Python-3.8.6) folder.</p> <p>Under 'externals', create a directory for openssl-1.0.2u and copy all the contents of c:\usr\local\ssl\ to this directory.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221152015627-1608548808308.png" alt=""></p> <p><strong>Step 7</strong> Under 'externals', create a directory for openssl-bin-1.0.2u/amd64 and copy all the files from c:\work\openssl-1.2.u\out32dll to this directory.</p> <p>Also, copy all the files from c:\work\openssl-1.2.u\inc32 to the openssl-bin-1.0.2u/amd64/include directory.</p> <p><strong>Step 8</strong> Patch/Add/Modify these codes to files as shown below under Python-3.8.6 source (c:\work\Python-3.8.6).</p> <p>Lib\ssl.py:</p> <pre><code class="language-python">try: from _ssl import FIPS_mode, FIPS_mode_set print('successful import') except ImportError as e: print('error in importing') print(e) </code></pre> <p>Modules/_ssl.c:</p> <pre><code class="language-python">static PyObject * _ssl_FIPS_mode_impl(PyObject *module) { return PyLong_FromLong(FIPS_mode()); } static PyObject * _ssl_FIPS_mode_set_impl(PyObject *module, int n) { if (FIPS_mode_set(n) == 0) { _setSSLError(ERR_error_string(ERR_get_error(), NULL) , 0, __FILE__, __LINE__); return NULL; } Py_RETURN_NONE; } static PyMethodDef PySSL_methods[] = { _SSL__TEST_DECODE_CERT_METHODDEF _SSL_RAND_ADD_METHODDEF _SSL_RAND_BYTES_METHODDEF _SSL_RAND_PSEUDO_BYTES_METHODDEF _SSL_RAND_EGD_METHODDEF _SSL_RAND_STATUS_METHODDEF _SSL_GET_DEFAULT_VERIFY_PATHS_METHODDEF _SSL_ENUM_CERTIFICATES_METHODDEF _SSL_ENUM_CRLS_METHODDEF _SSL_TXT2OBJ_METHODDEF _SSL_NID2OBJ_METHODDEF _SSL_FIPS_MODE_METHODDEF _SSL_FIPS_MODE_SET_METHODDEF {NULL, NULL} /* Sentinel */ }; </code></pre> <p>Modules/clinic/_ssl.c.h:</p> <pre><code class="language-python">PyDoc_STRVAR(_ssl_FIPS_mode__doc__, "FIPS Mode"); #define _SSL_FIPS_MODE_METHODDEF \ {"FIPS_mode", (PyCFunction)_ssl_FIPS_mode, METH_NOARGS, _ssl_FIPS_mode__doc__}, static PyObject * _ssl_FIPS_mode_impl(PyObject *module); static PyObject * _ssl_FIPS_mode(PyObject *module, PyObject *Py_UNUSED(ignored)) { return _ssl_FIPS_mode_impl(module); } PyDoc_STRVAR(_ssl_FIPS_mode_set_doc__, "FIPS Mode Set"); #define _SSL_FIPS_MODE_SET_METHODDEF \ {"FIPS_mode_set", (PyCFunction)_ssl_FIPS_mode_set, METH_O, _ssl_FIPS_mode_set_doc__}, static PyObject * _ssl_FIPS_mode_set_impl(PyObject *module, int n); static PyObject * _ssl_FIPS_mode_set(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; int n; if (!PyArg_Parse(arg, "i:FIPS_mode_set", &#x26;n)) { goto exit; } return_value = _ssl_FIPS_mode_set_impl(module, n); exit: return return_value; } </code></pre> <p>The above code is used to enable functions ssl.FIPS_mode() and ssl.FIPS_mode_set().</p> <p>Reference: <a href="https://stackoverflow.com/questions/49493537/how-to-implement-fips-mode-and-fips-mode-set-in-python-3-6s-ssl-module">https://stackoverflow.com/questions/49493537/how-to-implement-fips-mode-and-fips-mode-set-in-python-3-6s-ssl-module</a>.</p> <p><strong>Step 9</strong> Modify PCbuild/openssl.props as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/openssl-1608656128730.PNG" alt=""></p> <p><strong>Step 10</strong> Open PCbuild/python.props and change entries as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221155609284-1608549075946.png" alt=""></p> <p><strong>Step 11</strong> Open Python Solution under PCbuild/pcbuild.sln in VS 2017/2015.</p> <p>Change the link settings of _hashlib and _ssl projects under Python as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221155243791-1608549098426.png" alt=""></p> <p>Change compile settings _hashlib and _ssl projects as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221155403908-1608549116108.png" alt=""></p> <p><strong>Step 12</strong> Now, build _hashlib.pyd and _ssl.pyd in VS 2017/2015.</p> <p><strong>Step 13</strong> Copy these built pyd files to a Python binary installation directory c:\python38\DLLs folder.</p> <pre><code>Copy ssl.py to c:\python38\Lib folder. Copy OpenSSL DLLs (libeay32.dll and ssleay32.DLL) to the c:\python38\DLLs folder. </code></pre> <p><strong>Step 14</strong> Start Python and use these commands to check the OpenSSL/FIPS version.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image-20201221161218408-1608549142246.png" alt=""></p> <p>Voila...now Python 3.8 has OpenSSL with FIPS.</p> <p><strong>Summary</strong></p> <p>In this blog, I have covered the following steps in regards to enabling Python 3 with FIPS.</p> <ul> <li>Download the required packages.</li> <li>Compile both OpenSSL and FIPS and link them both.</li> <li>Make the required changes to Python source before linking to the new OpenSSL and compile.</li> <li>Copy the newly generated binaries to the installed Python location.</li> <li>Test the version and check if FIPS is enabled.</li> </ul> <p>I hope this blog is useful to the entire developer community!! Make sure you check out other blog posts on <a href="/blog">HPE DEV</a> for more useful tutorials.</p>https://developer.hpe.com/best-practices-on-migrating-from-a-data-warehouse-to-a-big-data-platform/https://developer.hpe.com/best-practices-on-migrating-from-a-data-warehouse-to-a-big-data-platform/Wed, 16 Dec 2020 07:00:29 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Michael Farnbach", "publish": "2016-10-24T07:00:00.000Z", "tags": "use-cases" </code></pre> <hr> <p>Offloading cold or unused data and ETL workloads from a data warehouse to Hadoop/big data platforms is a very common starting point for enterprises beginning their big data journey. Platforms like Hadoop provide an economical way to store data and do bulk processing of large data sets; hence, it’s not surprising that cost is the primary driver for this initial use case.</p> <p>What do these projects look like when they are actually implemented? In this post, we’ll take a look at the different factors to think about, we’ll provide a methodology for implementing data warehouse offloads, and demonstrate how things translate in a Hadoop/big data world. In the traditional data warehouse world, people are very used to sequencing tasks and workflows. Data has to be extracted from source systems, transformed, and then loaded into the target, i.e., data warehouses. </p> <p>In the traditional data warehousing world, structure and schemas are essential, which lead to clearly defined transformations. In the Hadoop and big data world, data doesn’t need to be stored as a structured format. New tools work without schema, or apply schema on read, or are optimized for columnar, key value pair and document databases as such. There is no real extract and loading—it’s all about the transformations that occur after the data lands in the cluster. When offloading from a data warehouse, both data and transformations are being moved. <strong>Data lifecycle</strong> is an important topic with three main areas to consider: data ingest, data integrations, and data delivery.</p> <ol> <li> <p><strong>Data Ingest</strong>: When it comes to data ingestion, what’s important is to map out your existing data flows to understand what modifications might be necessary in the Hadoop architecture. MapR offers various alternatives when it comes to ingest. You can use the unique NFS capability that comes only with the MapR Platform to ingest data into the cluster. On the storage side, it’s important to understand if data needs to be partitioned by day, for example, and whether updates will be incremental or full rewrites. When it comes to transformations, the big difference in the Hadoop world is that these occur after the fact and the critical step of defining a schema for transforming data is not a requirement.</p> </li> <li> <p><strong>Data Integrations</strong>: In the traditional world of data warehousing, customers have often built their data models using a star schema methodology, or a 3NF, or perhaps a mix of both. These techniques provide a compact relational understanding of the data and comprise a centralized data model. This can be leveraged in the Hadoop architecture and one can build data microservices on top of this which can be denormalized, cubed, or otherwise aggregated and interpreted for specific applications.</p> </li> <li> <p><strong>Data Delivery:</strong> At some point within the big data journey, customers will want to have to some sort of OLAP-like capabilities and build cubes to surface data easily to end users. Using tools from the broad Hadoop ecosystem, these data “microservices” can be built on top of streaming and batch models, using SQL and full programming languages.</p> </li> </ol> <p>The below visual shows how the data lifecycle is accomplished and aids in offloading data and transformations to a Hadoop-based environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/data-lifecycle-mapr-1608102123491.png" alt="MapR - Data Lifecycle"></p> <p>Another key topic to address is around <strong>data structures.</strong> There are potential decisions to be made on data models regarding architecture in the context of data warehouse offloads.</p> <ol> <li> <p>Some of the unique capabilities offered in the MapR Platform with MapR Database can help facilitate key-based lookups, even through SQL as an example.</p> </li> <li> <p>When it comes to delimited files, the MapR Platform can work directly on these, removing the need to architect metadata for these files, which can then be put in compressed indexed formats like Avro and Parquet to speed up regular reporting and exploration queries.</p> </li> <li> <p>JSON is increasingly becoming a key format for nested data and is very flexible. Handling JSON data is a key strength for the MapR Platform. In the data warehousing world, it’s common to find 2D table structures and various aggregations to put nested dimensional data into different entities. These can be interpreted differently by different functions within an organization. Restructuring these entities provides a distinct challenge in a pure relational data warehousing data flow, but can be handled more easily by using JSON in the MapR Platform.</p> </li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/billing-strategy-etl-offload-mapr-1608102141072.png" alt="MapR - Billing Strategy - ETL Offload"></p> <p>Above is an example of a customer in the telecommunications space that migrated portions of their data warehouse workload to a MapR cluster. You can see that they are benefiting from both a performance and price perspective.</p> <p>When it finally comes to data migration, there are a couple of different perspectives to consider and where the MapR Platform can help this effort.</p> <ol> <li> <p>Existing stored procedures implemented in a current data warehouse implementation can be moved over with reasonable effort, though this will require some development efforts to cover for platform specific features and SQL compatibility.</p> </li> <li> <p>On the systems architecture front, components can be ported to different toolsets altogether (e.g., SQL to Pig) but with the ability to retain their interfaces to other parts of the data workflow. Re-architecting at this level can improve development speed and manageability, while helping facilitate a more intuitive and efficient interface to other processes in the data workflow.</p> </li> </ol> <h2>Summary</h2> <p>After the data migration and transformation efforts are done, there are quite a few downstream benefits. New analytical tools and methods can be applied to derive new business insights. Use cases such as customer 360 and deeper analytics on existing business processes can be made available to business stakeholders and improve operational efficiency. As seen in the customer example above, there are cost savings and performance improvements to be gained as well. Clearly, data warehouse migration and offload initiatives can benefit not only the bottom line but also the top line. For more information on solutions MapR offers in this area, we encourage you to check out the data warehouse optimization area on our website.</p>https://developer.hpe.com/cloud-vs-on-premises-what-are-the-best-options-for-deploying-microservic/https://developer.hpe.com/cloud-vs-on-premises-what-are-the-best-options-for-deploying-microservic/Wed, 16 Dec 2020 06:55:47 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Jim Scott", "publish": "2018-04-18T11:00:00.000", "tags": "cloud-computing" </code></pre> <hr> <p>Microservices are great. Don't just take my word for it: there are many sources saying the same thing, and the industry is transitioning to this model. Containers are a key piece of the success of microservices. Once a microservice is built and put into a container, the task of deploying the application comes next. For the sake of simplicity, I am going to focus on the type of infrastructure to deploy services. Deployment options fall into four basic models, which can be <strong>mixed and matched</strong> as needed.</p> <h2>On-Premises Infrastructure</h2> <p>On-premises infrastructure has been the dominant enterprise computing model for more than 50 years. Organizations maintain their own equipment and software in a captive data center with full control over all aspects of processing, scheduling, administration, and maintenance. Many organizations in regulated industries have had no choice but to use an on-premises model because of the need to tightly control data and document processes. However, the cost and significant capital expense involved with building and maintaining on-premises infrastructure is prompting many organizations to shift some or all of their workloads to more-flexible cloud options. On-premises computing won't go away anytime soon, however. Legacy equipment and applications may be incompatible with a cloud environment, and organizations that want to protect investments in hardware and software may choose to maintain on-premises investments for years until depreciation cycles have run their course and applications can be redeveloped.</p> <h2>Public Cloud</h2> <p>Public cloud makes resources, such as processors, memory, operating systems, applications, and storage, available over the public internet on a pay-per-usage basis. Think of it as a computer in the sky. Public cloud is like using a local server, but the server is virtualized and managed elsewhere by a cloud provider with a high degree of automation.</p> <p>Organizations use public cloud for a variety of reasons, but the most popular are flexibility, scalability, and ease of administration. Public cloud instances can be launched with a few mouse clicks and just as easily taken down when no longer needed. Developers and end-users can, in many cases, deploy their own cloud instances without approval from IT and its accompanying delays. Billing is usually based upon usage, which gives organizations accountability and flexibility to pay only for the resources they use. Public cloud instances can be scaled up or down with relative ease, and many cloud providers offer best-of-breed automation tools to make administration easy. Public cloud is also an excellent platform for developing applications that will "live" in the cloud, such as those meant for use on mobile devices or with services that are exposed via APIs.</p> <h2>Private Cloud</h2> <p>For organizations that want the flexible automation benefits of public cloud but need to keep resources on-premises for control or compliance reasons, private cloud is a popular alternative. This model provides the same scalability, automation, and flexibility advantages of public cloud and on-premises environments that can be physically secured and tightly managed. Private clouds can be built using existing data center equipment architecture or licensed from public cloud providers, which could deliver what is essentially a version of their existing services in a secure environment. True private cloud is more than just virtualization. The research firm Wikibon <a href="https://wikibon.com/true-private-cloud-will-begin-shipping-to-the-market-in-2016/">defines</a> it as encompassing converged architecture, virtualized software and hardware, self-service provisioning, orchestration/automation, and a single point of control.</p> <h2>Hybrid Cloud</h2> <p>When you combine a public and private cloud, you get a hybrid cloud. This architecture combines both models in a manner that is seamless and that permits workloads to easily move back and forth. This gives organizations a combination of control and flexibility that can be adjusted to the situation. Hybrid architecture preserves existing hardware and software investments while giving companies the flexibility to move applications to the cloud as resources and budgets permit. Not all applications can be moved easily, and some may continue to live for a long time in private data centers. In those cases, organizations may opt for a "cloud bursting" approach, in which demand spills over to a duplicate or compatible cloud application as needed. This reduces the need to add on-premises infrastructure that sits idle much of the time. There are even cloud-cloud options, in which applications move back and forth between multiple public clouds.</p> <h2>The Plethora of Options</h2> <p>Perhaps the biggest challenge facing IT with respect to cloud is the obvious realization that there is no one single cloud. Rather, there are many, and most enterprises will deploy applications to several of them simultaneously. The challenge is one of orchestration and integration of data across various clouds.</p> <p>For example, consider the app/dev environment. Ideally, a lot of prototyping and testing would be done in the public cloud, with its scale-on-demand capabilities that make it easy for developers to get this vital work done without imposing on internal resources. Most organizations still choose to retain their most sensitive data on premises, however. Yet to complete the app/dev process from development to test to production, data must flow securely and seamlessly among these different environments in what is known as the hybrid IT world (a combination of on-premises and off-premises, public and private cloud as well as use of traditional on-premises non-cloud clusters).</p> <p>What is sorely needed to make all this happen is a distributed data platform and processing model that scales easily across all locations and environments. In other words, a converged data platform.</p>https://developer.hpe.com/spark-data-source-api-extending-our-spark-sql-query-engine/https://developer.hpe.com/spark-data-source-api-extending-our-spark-sql-query-engine/Wed, 16 Dec 2020 06:52:16 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019 may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-03-23T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>In my last post, <a href="/blog/xArv3gJz67Tl0Rv0kLVn/apache-spark-as-a-distributed-sql-engine">Apache Spark as a Distributed SQL Engine</a>, we explained how we could use SQL to query our data stored within Hadoop. Our engine is capable of reading CSV files from a distributed file system, auto discovering the schema from the files and exposing them as tables through the Hive meta store. All this was done to be able to connect standard SQL clients to our engine and explore our dataset without manually define the schema of our files, avoiding ETL work.</p> <p>Spark provides a framework that can be extended and we will push its capabilities even further by extending some of its functionalities.</p> <h2>Spark Data Source API</h2> <p>The Data Source API allows us to manage structured data in any format. Spark already has some standard structures built in such as Avro and Parquet, yet third parties have created new readers for CSV, JSON and others by extending this API. Today we are going to create our own.</p> <p>We have two reasons to extend the API.</p> <p>First, we want a library that is capable of reading our legacy format and transform our current data source into a new one that is easier to use.</p> <p>Second, we want to share this library across all our applications that use our data avoiding complex packaging of applications that need to be shared in order to achieve the same goal.</p> <h2>The Data Source</h2> <p>Our data source consists in a collection of files where each file is an entity by itself. For the sake of this example, we have defined a simple format where each file is a text file containing the information of a user, each field by line. Let’s see an example of a file.</p> <pre><code class="language-markdown">Pepe 20 Miami Cube </code></pre> <p>This file represents a user called ‘Pepe’ who is 20 years old, lives in Miami and was born in Cuba.</p> <p>In the real world, the format can be as complicated as we want, but the process we are going to explain will not change.</p> <p>Each file has the same format and we have millions of them. We also want to expose them to be queried in SQL.</p> <h2>Our Implementation</h2> <p>In order to extend the Data Source API, we need to implement certain classes from the Spark framework, so our custom reader can be loaded and used.</p> <p>Let’s start by creating a Spark application as the entry point to our example. We can do this by following the post <a href="https://medium.com/@anicolaspp/sbt-scala-and-spark-6a57c0a2623a#.yzj69ycnz">SBT, Scala and Spark</a>.</p> <p>The first thing we need to do once the app has been created is to link the correct Spark libraries. We are going to be running the examples on Spark 1.5.1 and our sbt file is defined as follow.</p> <pre><code class="language-markdown">name := "spark-datasource" version := "1.0" scalaVersion := "2.11.7" libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.5.1" libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "1.5.1" </code></pre> <h2>Creating Our Schema</h2> <p>The starting extension point of the Data Source API is the RelationProvider class. The RelationProvider class will be used to create the necessary relations of our data.</p> <p>We also need to mix the SchemaRelationProvider trait, which allows us to create the schema that we want.</p> <p>We need to create a class named DefaultSource and Spark will look for it in a given package. The DefaultSource class will extend RelationProvider and mix SchemaRelationProvider</p> <p>Our code so far looks as follow:</p> <pre><code class="language-scala">class DefaultSource extends RelationProvider with SchemaRelationProvider {  override def createRelation(sqlContext: SQLContext, parameters: Map[String, String])    : BaseRelation = {    createRelation(sqlContext, parameters, null)  }  override def createRelation(sqlContext: SQLContext, parameters: Map[String, String]    , schema: StructType)    : BaseRelation = {    parameters.getOrElse("path", sys.error("'path' must be specified for our data."))    return new LegacyRelation(parameters.get("path").get, schema)(sqlContext)  } } </code></pre> <p>In the code, we are basically creating a LegacyRelation object, which defined the Relation we want to create. Think about a relation like a collection of tuples with a known schema.</p> <p>Let’s see how our Relation class is implemented.</p> <pre><code class="language-scala">class LegacyRelation(location: String, userSchema: StructType) (@transient val sqlContext: SQLContext)  extends BaseRelation       with Serializable {  override def schema: StructType = {    if (this.userSchema != null) {      return this.userSchema    }    else {      return StructType(Seq(StructField("name", StringType, true),                             StructField("age", IntegerType, true)))    }  } } </code></pre> <p>Here we are overriding the schema function so it returns the schema we want. In this example, we know the schema of our data, but in here, we could do anything we want to obtain the required schema. If the data were CSV, we could infer the schema using the headers of the file or do any other operations we need.</p> <p>Notice that we only want the name and age fields instead of the entire content of our entities.</p> <p>The next step is to test that we are getting the correct schema and we can do this by adding the following code to our app.</p> <pre><code class="language-scala">object app {  def main(args: Array[String]) {    val config = new SparkConf().setAppName("testing provider")    val sc = new SparkContext(config)    val sqlContext = new SQLContext(sc)    val df = sqlContext              .read              .format("com.nico.datasource.dat")              .load("/Users/anicolaspp/data/")       df.printSchema()  } } </code></pre> <p>This code creates a SparkContext and an SQLContext from it. Using the SQLContext we set the format by passing the package name(Remember Spark will look at this package for the DefaultSource class). Then we load the data in the specified path using our provider into a DataFrame.</p> <pre><code class="language-scala">df.printSchema() </code></pre> <p>This will print the schema we defined and the output should look as follows.</p> <pre><code class="language-markdown">root |-- name: string (nullable = true) |-- age: integer (nullable = true) </code></pre> <p>At this point, we only have created the schema we want, but there is nothing that says how to ready the data and how to structure it into our defined schema.</p> <h2>Reading Data Into Our Schema</h2> <p>In order to read from our data source, our LegacyRelation class needs to mix the TableScan trait. TableScan has a method we need to implemented with the following signature:</p> <pre><code class="language-scala">def buildScan(): RDD[Row] </code></pre> <p>The method buildScan should return all rows from our data source. In our particular case, each row will be the selected content of each file. Let’s take a look at our implementation of the buildScan.</p> <pre><code class="language-scala">override def buildScan(): RDD[Row] = {    val rdd = sqlContext                .sparkContext                .wholeTextFiles(location)                .map(x => x._2)    val rows = rdd.map(file => {      val lines = file.split("\n")      Row.fromSeq(Seq(lines(0), lines(1)))    })    rows  } </code></pre> <p>Here we are using the wholeTextFiles method that reads the entire file (each file is an entity), reads the first two lines (the only fields we want) and creates a row from each of them. The result is a collection of rows where each row is created using only the part of the file we care about.</p> <p>This will be enough to modify our app so it prints out the content of our data source. The app now looks as follows.</p> <pre><code class="language-scala">object app {  def main(args: Array[String]) {    val config = new SparkConf().setAppName("testing provider")    val sc = new SparkContext(config)    val sqlContext = new SQLContext(sc)    val df = sqlContext              .read              .format("com.nico.datasource.dat")              .load("/Users/anicolaspp/data/")       df.show()  } } </code></pre> <p>Even though we are reading the desired format into a data frame, there is no information about the field types of our data. Our schema definition supports different data types, yet we are not enforcing them.</p> <p>Let’s modify our buildScan method so it infers the type information when creating each row.</p> <pre><code class="language-scala">override def buildScan(): RDD[Row] = {    val schemaFields = schema.fields    val rdd = sqlContext                .sparkContext                .wholeTextFiles(location)                .map(x => x._2)    val rows = rdd.map(file => {      val lines = file.split("\n")      val typedValues = lines.zipWithIndex.map {        case (value, index) => {          val dataType = schemaFields(index).dataType          castValue(value, dataType)        }    nbsp;  }      Row.fromSeq(typedValues)    })    rows  }   private def castValue(value: String, toType: DataType) = toType match {    case _: StringType      => value    case _: IntegerType     => value.toInt  } </code></pre> <p>Here, the only change is that we are casting each value read from our files into its correct type, inferred from the schema.fields object. In our particular case we are only interested that name is a String and age an Integer, but again, we could be very creative at this point.</p> <p>Now, our final LegacyRelation class will look as follows.</p> <pre><code class="language-scala">class LegacyRelation(location: String, userSchema: StructType)  (@transient val sqlContext: SQLContext)  extends BaseRelation      with TableScan with Serializable {  override def schema: StructType = {    if (this.userSchema != null) {      return this.userSchema    }    else {      return StructType(Seq(StructField("name", StringType, true),                             StructField("age", IntegerType, true)))    }  }  private def castValue(value: String, toType: DataType) = toType match {    case _: StringType      => value    case _: IntegerType     => value.toInt  }  override def buildScan(): RDD[Row] = {    val schemaFields = schema.fields    val rdd = sqlContext              .sparkContext              .wholeTextFiles(location)              .map(x => x._2)    val rows = rdd.map(file => {      val lines = file.split("\n")      val typedValues = lines.zipWithIndex.map{        case (value, index) => {          val dataType = schemaFields(index).dataType          castValue(value, dataType)        }      }      Row.fromSeq(typedValues)    })    rows  } </code></pre> <p>Now we can load our data into a DataFrame and register it to be used by SQL clients as we explain in our previous post. Our app is as simple as shown below.</p> <pre><code class="language-scala">object app {  def main(args: Array[String]) {    val config = new SparkConf().setAppName("testing provider")    val sc = new SparkContext(config)    val sqlContext = new SQLContext(sc)    val df = sqlContext              .read              .format("com.nico.datasource.dat")              .load("/Users/anicolaspp/data/")       df.registerTempTable("users")    sqlContext.sql("select name from users").show()  } } </code></pre> <p>We have shown enough to read a custom format into a data frame so we can take advantage from the DataFrame API, yet more can be done.</p> <p>The Data Source API not only offers functionalities for reading data, but also to write it in a custom format. This functionality is very powerful if we want to transform a data set from one format to another one. Let’s see how we add these capabilities to our existing driver.</p> <h2>Writing a Formatter</h2> <p>Let’s suppose we want to save our data so it can be read from other standard systems. We are going to load our custom data source and create a CSV-like output from it.</p> <p>In order to support save calls from the API, our DefaultSource class has to mix with the CreatableRelationProvider trait. This trait has a method called createRelation we need to implement. Let’s take a look at it.</p> <pre><code class="language-scala">override def createRelation(sqlContext: SQLContext, mode: SaveMode,     parameters: Map[String, String], data: DataFrame): BaseRelation = {    saveAsCsvFile(data, parameters.get("path").get)    createRelation(sqlContext, parameters, data.schema)  }  def saveAsCsvFile(data: DataFrame, path: String) = {    val dataCustomRDD = data.rdd.map(row => {      val values = row.toSeq.map(value => value.toString)      values.mkString(",")    })    dataCustomRDD.saveAsTextFile(path)  } </code></pre> <p>We are basically saving our data frame as a CSV-like file and then returning a relation with a known schema.</p> <p>The saveAsCsvFile method is creating a RDD[String] with our data formatted as CSV, then it saves it to the given path. For simplicity we did not include the headers in our output files, but remember we can do whatever we need to output the data in the format we require.</p> <p>The entire code of our DefaultSource class is the following.</p> <pre><code class="language-scala">class DefaultSource extends RelationProvider     with SchemaRelationProvider     with CreatableRelationProvider {  override def createRelation(sqlContext: SQLContext,     parameters: Map[String, String]): BaseRelation = {        createRelation(sqlContext, parameters, null)  }  override def createRelation(sqlContext: SQLContext,     parameters: Map[String, String], schema: StructType): BaseRelation = {        parameters.getOrElse("path", sys.error("'path' must be specified for CSV data."))        return new LegacyRelation(parameters.get("path").get, schema)(sqlContext)  }  def saveAsCsvFile(data: DataFrame, path: String) = {    val dataCustomRDD = data.rdd.map(row => {      val values = row.toSeq.map(value => value.toString)      values.mkString(",")    })    dataCustomRDD.saveAsTextFile(path)  }  override def createRelation(sqlContext: SQLContext, mode: SaveMode,     parameters: Map[String, String], data: DataFrame): BaseRelation = {        saveAsCsvFile(data, parameters.get("path").get)        createRelation(sqlContext, parameters, data.schema)  } } </code></pre> <p>In order to save our original data as CSV-like format, we modify our app as follow.</p> <pre><code class="language-scala">object app {  def main(args: Array[String]) {    val config = new SparkConf().setAppName("testing provider")    val sc = new SparkContext(config)    val sqlContext = new SQLContext(sc)    val df = sqlContext              .read              .format("com.nico.datasource.dat")              .load("/Users/anicolaspp/data/")       df.write      .format("com.nico.datasource.dat")      .save("/Users/anicolaspp/data/output")  } } </code></pre> <p>Note that every time we read/write our data, we need to specify the package name where our DefaultSource class is located.</p> <p>We now can package our library and include it in any project we need to use the data source we described. Many other libraries are being created to support all possible formats we can imagine and now you can create your own to contribute to the community or just to be used in your own projects.</p> <h2>Endings</h2> <p>We have seen how to load data from a custom format into data frames using the Spark Data Source API. We also reviewed the classes involved in the process, especially how Spark uses our DefaultSource from our package to perform the required operations. We also implemented an output formatter so our data frames can be saved, as we like to.</p> <p>There is much more we can do with the Data Source API, but finding the right documentation has been quite hard in my experience. I believe that better documentation could be created, specifically for those parts of the API that are very useful when extending them.</p> <p>Even though our example shows how to extend the Data Source API to support a simple format, it can be modified to read and write more complex types such as binary encoded entities.</p> <p>The ability to integrate our own data types into Spark makes it one of the top frameworks for data processing out there.</p> <p>In the Hadoop world we can find a lot of tools that share goals and functionalities, but none of them is as flexible and versatile as Spark. This makes Spark very desirable in this field. If we are interested in a processing framework capable of work under limitless circumstances, then Apache Spark is the way to go.</p> <p><a href="https://medium.com/@anicolaspp/extending-our-spark-sql-query-engine-5f4a088de986">View the original</a></p>https://developer.hpe.com/analyzing-flight-delays-with-apache-spark-graphframes-and-mapr-database/https://developer.hpe.com/analyzing-flight-delays-with-apache-spark-graphframes-and-mapr-database/Wed, 16 Dec 2020 06:33:59 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-11-16T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>Apache Spark GraphX made it possible to run graph algorithms within Spark. GraphFrames integrates GraphX and DataFrames and makes it possible to perform Graph pattern queries without moving data to a specialized graph database.</p> <p>This blog will help you get started using Apache Spark GraphFrames Graph Algorithms and Graph Queries with MapR Database JSON document database. We will begin with an overview of Graph and GraphFrames concepts, then we will analyze a real flight dataset for January-August 2018 stored in a MapR Database table.  </p> <p>Graphs provide a powerful way to analyze the connections in a Dataset. GraphX is the Apache Spark component for graph-parallel and data-parallel computations, built upon a branch of mathematics called graph theory. It is a distributed graph processing framework that sits on top of the Spark core. GraphX brings the speed and scalability of parallel, iterative processing to graphs for big datasets. It partitions graphs that are too large to fit in the memory of a single computer among multiple computers in a cluster. In addition, GraphX partitions vertices independently of edges, which avoids the load imbalance often suffered when putting all the edges for a vertex onto a single machine.</p> <p>With Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster: Spark SQL and the Dataset/DataFrame APIs provide ease of use, space efficiency, and performance gains with Spark SQL's optimized execution engine. GraphFrames extends Spark GraphX to provide the DataFrame API, making the analysis easier to use, more efficient, and simplifying data pipelines.</p> <h2>Overview of Some Graph Concepts</h2> <p>A graph is a mathematical structure used to model relations between objects. A graph is made up of vertices and edges that connect them. The vertices are the objects, and the edges are the relationships between them.  </p> <p>A regular graph is a graph where each vertex has the same number of edges. An example of a regular graph is Facebook friends. If Ted is a friend of Carol, then Carol is also a friend of Ted.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image7-1608101013070.png" alt=""></p> <p>A <strong>directed graph</strong> is a graph where the edges have a direction associated with them. An example of a directed graph is a Twitter follower. Carol can follow Oprah without implying that Oprah follows Carol.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image18-1608101023047.png" alt=""></p> <h2>GraphFrame Property Graph</h2> <p>Spark GraphFrames support graph computation with a distributed property graph. A property graph is a directed multigraph, which can have multiple edges in parallel. Every edge and vertex has user-defined properties associated with it. The parallel edges allow multiple relationships between the same vertices.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image28-1608101031846.png" alt=""></p> <p>With GraphFrames, vertices and edges are represented as DataFrames, which adds the advantages of querying with Spark SQL and support for DataFrame data sources like Parquet, JSON, CSV, and also MapR Database with the MapR Database Spark Connector.</p> <h2>Graph Algorithms verses Graph Queries</h2> <p>Graph analysis comes in two forms:  graph algorithms and graph pattern queries. Let’s look at an example of each and how GraphFrames integrates the two.</p> <h2>PageRank Graph Algorithm</h2> <p>The breakthrough for the creators of the Google search engine was to create the PageRank graph algorithm, which represents pages as nodes and links as edges and measures the importance of a page by the number and rank of linking pages, plus the number and rank of each of the linking pages. The PageRank algorithm is useful for measuring the importance of a vertex in a graph. Example of this could be a Twitter tweeter with lots of important followers or an airport with lots of connections to other airports with lots of connections.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image2-1608101039047.png" alt=""></p> <p>Many graph algorithms such as PageRank, shortest path, and connected components repeatedly aggregate properties of neighboring vertices.  These Algorithms can be implemented as a sequence of steps where vertices pass message functions to their neighboring vertices and then the aggregate of the message functions is calculated at the destination vertex.</p> <p>You can visualize the PageRank algorithm as each page sending a message with it’s rank of importance to each page it points to. In the beginning, each page has the same rank equal to the ratio of the number of pages.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image1-1608101046388.png" alt=""></p> <p>The messages are aggregated and calculated at each vertex with the sum of all of the incoming messages becoming the new page rank. This is calculated iteratively so that links from more important pages are more important.</p> <h2>Graph Motif Queries  </h2> <p>Graph motifs are recurrent patterns in a graph. Graph queries search a graph for all occurrences of a given motif or pattern. As an  example, to recommend who to follow on Twitter, you could search for patterns where A follows B and B follows C, but A does not follow C. Here is a  GraphFrames Motif Query for this pattern, to find the edges from a to b and b to c  for which there is no edge from a to c:</p> <pre><code class="language-markdown">graph.find("(a)-[]->(b); (b)-[]->(c); !(a)-[]->(c)") </code></pre> <p> (vertices are denoted by parentheses ( ), while edges are denoted by square brackets [ ] )</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image24-1608101055589.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image3-1608101062281.png" alt=""></p> <p>GraphFrames integrate graph algorithms and graph queries, enabling optimizations across graph and Spark SQL queries, without requiring data to be moved to specialized graph databases.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image12-1608101070628.png" alt=""></p> <h2>Graph Examples</h2> <p>Examples of connected data that can be represented by graphs include:</p> <p><strong>Recommendation Engines:</strong> Recommendation algorithms can use graphs where the nodes are the users and products, and their respective attributes and the edges are the ratings or purchases of the products by users. Graph algorithms can calculate weights for how similar users rated or purchased similar products.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image11-1608101079964.png" alt=""></p> <p><strong>Fraud:</strong> Graphs are useful for fraud detection algorithms in banking, healthcare, and network security. In healthcare, graph algorithms can explore the connections between patients, doctors, and pharmacy prescriptions. In banking, graph algorithms can explore the relationship between credit card applicants and phone numbers and addresses or between credit cards customers and merchant transactions. In network security, graph algorithms can explore data breaches.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image13-1608101087917.png" alt=""></p> <p>These are just some examples of the uses of graphs. Next, we will look at a specific example, using Spark GraphFrames.</p> <h2>Example Flight Scenario</h2> <p>As a starting simple example, we will analyze 3 flights; for each flight, we have the following information:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image9-1608101096479.png" alt=""></p> <table> <thead> <tr> <th>Originating Airport</th> <th>Destination Airport</th> <th>Distance</th> </tr> </thead> <tbody> <tr> <td>SFO</td> <td>ORD</td> <td>1800 miles</td> </tr> <tr> <td>ORD</td> <td>DFW</td> <td>800 miles</td> </tr> <tr> <td>DFW</td> <td>SFO</td> <td>1400 miles</td> </tr> </tbody> </table> <p>In this scenario, we are going to represent the airports as vertices and flight routes as edges. For our graph, we will have three vertices, each representing an airport. The vertices each have the airport code as the ID, and the city as a property:</p> <p><strong>Vertex Table for Airports</strong></p> <table> <thead> <tr> <th>id</th> <th>city</th> </tr> </thead> <tbody> <tr> <td>SFO</td> <td>San Francisco</td> </tr> <tr> <td>ORD</td> <td>Chicago</td> </tr> <tr> <td>DFW</td> <td>Texas</td> </tr> </tbody> </table> <p>The edges have the Source ID, the Destination ID, and the distance as a property.</p> <p><strong>Edges Table for Routes</strong></p> <table> <thead> <tr> <th>Src</th> <th>Dst</th> <th>Distance</th> <th>Delay</th> </tr> </thead> <tbody> <tr> <td>SFO</td> <td>ORD</td> <td>1800</td> <td>40</td> </tr> <tr> <td>ORD</td> <td>DFW</td> <td>800</td> <td>0</td> </tr> <tr> <td>DFW</td> <td>SFO</td> <td>1400</td> <td>10</td> </tr> </tbody> </table> <p><strong>Launching the Spark Interactive Shell with GraphFrames</strong></p> <p>Because GraphFrames is a separate package from Spark, start the Spark shell, specifying the GraphFrames package as shown below: </p> <pre><code class="language-bash">$SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.6.0-spark2.3-s_2.11 </code></pre> <p><strong>Define Vertices</strong></p> <p>First, we will import the DataFrames, GraphX, and GraphFrames packages.</p> <pre><code class="language-scala">import org.apache.spark._ import org.apache.spark.graphx._ import org.apache.spark.sql._ import org.apache.spark.sql.functions._ import org.apache.spark.sql.types._ import org.apache.spark.sql.types.StructType import org.graphframes._ import spark.implicits._ </code></pre> <p>We define airports as vertices. A vertex DataFrame must have an ID column and may have multiple attribute columns. In this example, each airport vertex consists of:</p> <ul> <li>Vertex ID→ id</li> <li>Vertex Property → city</li> </ul> <p><strong>Vertex Table for Airports</strong></p> <table> <thead> <tr> <th>id</th> <th>city</th> </tr> </thead> <tbody> <tr> <td>SFO</td> <td>San Francisco</td> </tr> </tbody> </table> <p>We define a DataFrame with the above properties, which will be used for the vertices in the GraphFrame.</p> <pre><code class="language-scala">// create  vertices  with ID and Name case class Airport(id: String, city: String) extends Serializable val airports=Array(Airport("SFO","San Francisco"),Airport("ORD","Chicago"),Airport("DFW","Dallas Fort Worth")) val vertices = spark.createDataset(airports).toDF vertices.show --- result: --- +---+-----------------+ | id|             city| +---+-----------------+ |SFO|    San Francisco| |ORD|          Chicago| |DFW|Dallas Fort Worth| +---+-----------------+ </code></pre> <p><strong>Define Edges</strong></p> <p>Edges are the flights between airports. An edge DataFrame must have src and dst columns and may have multiple relationship columns. In our example, an edge consists of:</p> <ul> <li>Edge origin ID → src</li> <li>Edge destination ID → dst</li> <li>Edge property distance → dist</li> <li>Edge property delay→ delay</li> </ul> <p><strong>Edges Table for Flights</strong></p> <table> <thead> <tr> <th>id</th> <th>src</th> <th>dst</th> <th>dist</th> <th>delay</th> </tr> </thead> <tbody> <tr> <td>SFO_ORD_2017-01-01_AA</td> <td>SFO</td> <td>ORD</td> <td>1800</td> <td>40</td> </tr> </tbody> </table> <p>We define a DataFrame with the above properties, which will be used for the edges in the GraphFrame.</p> <pre><code class="language-scala">// create  flights with srcid, destid , distance case class Flight(id: String, src: String,dst: String, dist: Double, delay: Double) val flights=Array(Flight("SFO_ORD_2017-01-01_AA","SFO","ORD",1800, 40),Flight("ORD_DFW_2017-01-01_UA","ORD","DFW",800, 0),Flight("DFW_SFO_2017-01-01_DL","DFW","SFO",1400, 10)) val edges = spark.createDataset(flights).toDF edges.show --- result: --- +--------------------+---+---+------+-----+ |                  id|src|dst|  dist|delay| +--------------------+---+---+------+-----+ |SFO_ORD_2017-01-0...|SFO|ORD|1800.0| 40.0| |ORD_DFW_2017-01-0...|ORD|DFW| 800.0|  0.0| |DFW_SFO_2017-01-0...|DFW|SFO|1400.0| 10.0| +--------------------+---+---+------+-----+ </code></pre> <p><strong>Create the GraphFrame</strong></p> <p>Below, we create a GraphFrame by supplying a vertex DataFrame and an edge DataFrame. It is also possible to create a GraphFrame with just an edge DataFrame; then the vertices will be equal to the unique src and dst ids from the edge DataFrame.</p> <pre><code class="language-scala">// define the graph val graph = GraphFrame(vertices, edges) // show graph vertices graph.vertices.show +---+-----------------+ | id|             name| +---+-----------------+ |SFO|    San Francisco| |ORD|          Chicago| |DFW|Dallas Fort Worth| +---+-----------------+ // show graph edges graph.edges.show --- result: --- +--------------------+---+---+------+-----+ |                  id|src|dst|  dist|delay| +--------------------+---+---+------+-----+ |SFO_ORD_2017-01-0...|SFO|ORD|1800.0| 40.0| |ORD_DFW_2017-01-0...|ORD|DFW| 800.0|  0.0| |DFW_SFO_2017-01-0...|DFW|SFO|1400.0| 10.0| +--------------------+---+---+------+-----+ </code></pre> <h2>Querying the GraphFrame</h2> <p>Now we can query the GraphFrame to answer the following questions:</p> <p><strong>How many airports are there?</strong></p> <pre><code class="language-scala">// How many airports? graph.vertices.count --- result: --- Long = 3 </code></pre> <p><strong>How many flights are there between airports?</strong></p> <pre><code class="language-scala">// How many flights? graph.edges.count --- result: --- = 3 </code></pre> <p><strong>Which flight routes are greater than 1000 miles in distance?</strong></p> <pre><code class="language-scala">// routes >  1000 miles distance? graph.edges.filter("dist > 800").show +--------------------+---+---+------+-----+ |                  id|src|dst|  dist|delay| +--------------------+---+---+------+-----+ |SFO_ORD_2017-01-0...|SFO|ORD|1800.0| 40.0| |DFW_SFO_2017-01-0...|DFW|SFO|1400.0| 10.0| +--------------------+---+---+------+-----+ </code></pre> <p>The GraphFrames triplets put all of the edge, src, and dst columns together in a DataFrame.</p> <pre><code class="language-scala">// triplets = src edge dst graph.triplets.show --- result: --- +--------------------+--------------------+--------------------+ |                 src|                edge|                 dst| +--------------------+--------------------+--------------------+ | [SFO,San Francisco]|[SFO_ORD_2017-01-...|       [ORD,Chicago]| |       [ORD,Chicago]|[ORD_DFW_2017-01-...|[DFW,Dallas Fort ...| |[DFW,Dallas Fort ...|[DFW_SFO_2017-01-...| [SFO,San Francisco]| +--------------------+--------------------+--------------------+ </code></pre> <p><strong>What are the longest distance routes?</strong></p> <pre><code class="language-scala">// print out longest routes graph.edges   .groupBy("src", "dst")   .max("dist")   .sort(desc("max(dist)")).show +---+---+---------+ |src|dst|max(dist)| +---+---+---------+ |SFO|ORD|   1800.0| |DFW|SFO|   1400.0| |ORD|DFW|    800.0| +---+---+---------+ </code></pre> <p>Next we will analyze flight delays and distances using the real flight data.</p> <h2>Loading and Querying Flight Data with Spark DataFrames and MapR Database JSON</h2> <p>The Spark MapR Database Connector enables users to perform complex SQL queries and updates on top of MapR Database using a Spark Dataset, while applying critical techniques such as projection and filter pushdown, custom partitioning, and data locality.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image5-1608101104461.png" alt=""></p> <p>With MapR Database, a table is automatically partitioned into tablets across a cluster by key range, providing for scalable and fast reads and writes by row key. In this use case, the row key, the id, starts with the origin, destination airport codes, so the table is automatically partitioned and sorted by the edge src, dst with Atlanta first.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image15-1608101113190.png" alt=""></p> <p>The Spark MapR Database Connector leverages the Spark <a href="https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html">DataSource API</a>. The connector architecture has a connection object in every Spark Executor, allowing for distributed parallel writes, reads, or scans with MapR Database tablets (partitions).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image29-1608101121307.png" alt=""></p> <p>MapR Database supports JSON documents as a native data store, making it easy to store, query, and build applications with JSON documents. For the flights MapR Database table we have the following JSON schema. We will focus on the highlighted id, the src, dst, depdelay, and distance attributes. The MapR Database table will be automatically partitioned and sorted by the rowkey or id, which consists of the src, dst, date, carrier and flight number.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image4-1608101128380.png" alt=""></p> <h2>Loading The Flight MapR Database Table Data into a Spark Dataset</h2> <p>In our use case Edges are the flights between airports. An edge must have src and dst columns and can have multiple relationship columns. In our example, an edge consists of:</p> <table> <thead> <tr> <th>id</th> <th>src</th> <th>dst</th> <th>distance</th> <th>depdelay</th> <th>carrier</th> <th>crsdephour</th> </tr> </thead> <tbody> <tr> <td>SFO_ORD_2017-01-01_AA</td> <td>SFO</td> <td>ORD</td> <td>1800</td> <td>40</td> <td>AA</td> <td>17</td> </tr> </tbody> </table> <p>Below, we define the flight schema, corresponding to a row in the MapR Database JSON Flight Table .</p> <pre><code class="language-scala">// define the Flight Schema case class Flight(id: String,fldate: String,month:Integer, dofW: Integer, carrier: String, src: String,dst: String, crsdephour: Integer, crsdeptime: Integer, depdelay: Double, crsarrtime: Integer, arrdelay: Double, crselapsedtime: Double, dist: Double) val schema = StructType(Array(     StructField("id", StringType, true),     StructField("fldate", StringType, true),     StructField("month", IntegerType, true),     StructField("dofW", IntegerType, true),     StructField("carrier", StringType, true),     StructField("src", StringType, true),     StructField("dst", StringType, true),     StructField("crsdephour", IntegerType, true),     StructField("crsdeptime", IntegerType, true),     StructField("depdelay", DoubleType, true),     StructField("crsarrtime", IntegerType, true),     StructField("arrdelay", DoubleType, true),     StructField("crselapsedtime", DoubleType, true),     StructField("dist", DoubleType, true)   )) </code></pre> <p>To <a href="https://docs.datafabric.hpe.com/62/Spark/LoadDataFromMapRDBasDataset.html">load data from a MapR Database JSON</a> table into an Apache Spark Dataset, we invoke the loadFromMapRDB method on a SparkSession object, providing the tableName, schema, and case class. This returns a Dataset of Flight objects:</p> <pre><code class="language-scala">import com.mapr.db._ import com.mapr.db.spark._ import com.mapr.db.spark.impl._ import com.mapr.db.spark.sql._ val df = spark.sparkSession.loadFromMapRDB\[Flight](tableName, schema) flights.show --- result: ---  </code></pre> <h2>Define Vertices</h2> <p>We define airports as vertices. Vertices can have properties or attributes associated with them. For each airport, we have the following information:</p> <p><strong>Vertex Table for Airports</strong></p> <table> <thead> <tr> <th>id</th> <th>city</th> <th>state</th> </tr> </thead> <tbody> <tr> <td>SFO</td> <td>San Francisco</td> <td>CA</td> </tr> </tbody> </table> <p>Note that our dataset contains only a subset of the airports in the USA; below are the airports in our dataset shown on a map.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image23-1608101155741.png" alt=""></p> <p>Below, we read the airports information into a DataFrame from a JSON file.</p> <pre><code class="language-scala">// create  airports DataFrame val airports = spark.read.json("maprfs:///data/airports.json") airports.createOrReplaceTempView("airports") airports.show --- result: --- +-------------+-------+-----+---+ |         City|Country|State| id| +-------------+-------+-----+---+ |      Chicago|    USA|   IL|ORD| |     New York|    USA|   NY|LGA| |       Boston|    USA|   MA|BOS| |      Houston|    USA|   TX|IAH| |       Newark|    USA|   NJ|EWR| |       Denver|    USA|   CO|DEN| |        Miami|    USA|   FL|MIA| |San Francisco|    USA|   CA|SFO| |      Atlanta|    USA|   GA|ATL| |       Dallas|    USA|   TX|DFW| |    Charlotte|    USA|   NC|CLT| |  Los Angeles|    USA|   CA|LAX| |      Seattle|    USA|   WA|SEA| +-------------+-------+-----+---+ </code></pre> <h2>Create the Property Graph</h2> <p>Again, in this scenario, we are going to represent the airports as vertices and flights as edges.  Below, we create a GraphFrame by supplying a vertex DataFrame and an edge DataFrame. The airports and flights Dataframes are available as the graph.edges and graph.vertices. Since GraphFrame vertices and edges are stored as DataFrames, many queries are just DataFrame (or SQL) queries.</p> <pre><code class="language-scala">// define the graphframe val graph = GraphFrame(airports, df) // filter graph vertices graph.vertices.filter("State='TX'").show +-------+-------+-----+---+ |   City|Country|State| id| +-------+-------+-----+---+ |Houston|    USA|   TX|IAH| | Dallas|    USA|   TX|DFW| +-------+-------+-----+---+ </code></pre> <h2>Querying the GraphFrame</h2> <p>Now we can query the GraphFrame to answer the following questions:</p> <p><strong>How many airports are there?</strong></p> <pre><code class="language-scala">// How many airports? val numairports = graph.vertices.count --- result: ---  Long = 13 </code></pre> <p><strong>How many flights are there?</strong></p> <pre><code class="language-scala">// How many flights val numflights = graph.edges.count --- result: --- // Long = 282628 </code></pre> <p><strong>Which flight routes have the longest distance?</strong></p> <pre><code class="language-scala">// show the longest distance routes graph.edges .groupBy("src", "dst") .max("dist") .sort(desc("max(dist)")).show(4) --- result: --- +---+---+---------+ |src|dst|max(dist)| +---+---+---------+ |MIA|SEA|   2724.0| |SEA|MIA|   2724.0| |BOS|SFO|   2704.0| |SFO|BOS|   2704.0| +---+---+---------+ </code></pre> <p><strong>Which flight routes have the highest average delays?</strong></p> <pre><code class="language-scala">graph.edges .groupBy("src", "dst") .avg("depdelay") .sort(desc("avg(delay)")).show(5) --- result: --- +---+---+------------------+ |src|dst|     avg(depdelay)| +---+---+------------------+ |ATL|EWR|25.520159946684437| |DEN|EWR|25.232164449818622| |MIA|SFO|24.785953177257525| |MIA|EWR|22.464104423495286| |IAH|EWR| 22.38344914718888| +---+---+------------------+ </code></pre> <p><strong>Which flight hours have the highest average delays?</strong></p> <pre><code class="language-scala">graph.edges .groupBy("crsdephour") .avg("depdelay") .sort(desc("avg(delay)")).show(5) --- result: --- +----------+------------------+                                                 |crsdephour|        avg(delay)| +----------+------------------+ |        18| 24.24118415324336| |        19|23.348782771535582| |        21|19.617375231053604| |        16| 19.30346232179226| |        17| 18.77857142857143| +----------+------------------+ </code></pre> <p><strong>What are the longest delays for flights that are greater than 1500 miles in distance?</strong></p> <pre><code class="language-scala">// flights >  1500 miles distance ordered by delay graph.edges.filter("dist > 1500") .orderBy(desc("depdelay")).show(3) --- result: ---  </code></pre> <p><strong>What is the average delay  for delayed flights departing from Atlanta?</strong></p> <pre><code class="language-scala">graph.edges.filter("src = 'ATL' and depdelay > 1") .groupBy("src", "dst") .avg("depdelay").sort(desc("avg(depdelay)")).show --- result: --- +---+---+------------------+ |src|dst|     avg(depdelay)| +---+---+------------------+ |ATL|EWR|  58.1085801063022| |ATL|ORD| 46.42393736017897| |ATL|DFW|39.454460966542754| |ATL|LGA| 39.25498489425982| |ATL|CLT| 37.56777108433735| |ATL|SFO| 36.83008356545961| +---+---+------------------+ </code></pre> <p><strong>Projection and filter push down into MapR Database</strong></p> <p>You can see the physical plan for a DataFrame query by calling the explain method shown below. Here we see projection and filter push down, which means that the scanning of the src, dst and depdelay columns and the filter on the depdelay column are pushed down into MapR Database, which means that the scanning and filtering will take place in MapR Database before returning the data to Spark. Projection pushdown minimizes data transfer between MapR Database and the Spark engine by omitting unnecessary fields from table scans. It is especially beneficial when a table contains many columns. Filter pushdown improves performance by reducing the amount of data passed between MapR Database and the Spark engine when filtering data.</p> <pre><code class="language-scala">graph.edges.filter("src = 'ATL' and depdelay > 1") .groupBy("src", "dst") .avg("depdelay").sort(desc("avg(depdelay)"))&#x3C;span style="color: red;">.explain&#x3C;/span> == Physical Plan == \*(3) Sort [avg(depdelay)#273 DESC NULLS LAST], true, 0 +- Exchange rangepartitioning(avg(depdelay)#273 DESC NULLS LAST, 200)    +- \*(2) HashAggregate(keys=[src#5, dst#6],          functions=[avg(depdelay#9)])       +- Exchange hashpartitioning(src#5, dst#6, 200)          +- \*(1) HashAggregate(keys=[src#5, dst#6],             functions=[partial_avg(depdelay#9)])             +- \*(1) Filter (((isnotnull(src#5) &#x26;&#x26;                 isnotnull(depdelay#9)) &#x26;&#x26;                             (src#5 = ATL)) &#x26;&#x26; (depdelay#9 > 1.0))                +- \*(1) Scan MapRDBRelation(/user/mapr/flighttable &#x3C;span style="color: red;">[src#5,dst#6,depdelay#9] PushedFilters: [IsNotNull(src), IsNotNull(depdelay), EqualTo(src,ATL), GreaterThan(depdelay,1.0)]&#x3C;/span> </code></pre> <p><strong>What are the worst hours for delayed flights departing from Atlanta?</strong></p> <pre><code class="language-scala">graph.edges.filter("src = 'ATL' and delay > 1")  .groupBy("crsdephour")  .avg("delay")  .sort(desc("avg(delay)")).show(5) --- result: ---  </code></pre> <p><strong>What are the four most frequent flight routes in the data set? or What is the count of flights for all possible flight routes, sorted?</strong> (Note: we will use the DataFrame returned later)</p> <pre><code class="language-scala">val flightroutecount=graph.edges  .groupBy("src", "dst")  .count().orderBy(desc("count")).show(4) --- result: --- +---+---+-----+ |src|dst|count| +---+---+-----+ |LGA|ORD| 4442| |ORD|LGA| 4426| |LAX|SFO| 4406| |SFO|LAX| 4354| +---+---+-----+ </code></pre> <p><strong>result in  a bar chart:</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image14-1608101187466.png" alt=""></p> <h2>Vertex Degrees</h2> <p>The degree of a vertex is the number of edges that touch the vertex. GraphFrames provides vertex inDegree, outDegree, and degree operations, which determine the number of incoming edges, outgoing edges, and total edges. Using GraphFrames the degree operation we can answer the following question.</p> <p><strong>Which airports have the most incoming and outgoing flights?</strong></p> <pre><code class="language-scala">graph.degrees.orderBy(desc("degree")).show(3) --- result: --- +---+------+ | id|degree| +---+------+ |ATL| 60382| |ORD| 64386| |LAX| 53733| +---+------+ </code></pre> <p><strong>result in  a bar chart:</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image16-1608101195455.png" alt=""></p> <h2>PageRank</h2> <p>Another GraphFrames query is PageRank, which is based on the Google PageRank algorithm.</p> <p>PageRank measures the importance of each vertex in a graph, by determining which vertices have the most edges with other vertices. In our example, we can use PageRank to determine which airports are the most important, by measuring which airports have the most connections to other airports with lots of connections. We have to specify the probability tolerance, which is the measure of convergence. Note that the results are similar to the degrees operation, but the algorithm is different.</p> <p><strong>What are the most important airports, according to PageRank?</strong></p> <pre><code class="language-scala">// use pageRank val ranks = graph.pageRank.resetProbability(0.15).maxIter(10).run() ranks.vertices.orderBy($"pagerank".desc).show() --- result: --- +-------------+-------+-----+---+------------------+ |         City|Country|State| id|          pagerank| +-------------+-------+-----+---+------------------+ |      Chicago|    USA|   IL|ORD| 1.421132695625478| |      Atlanta|    USA|   GA|ATL|1.3389970164746383| |  Los Angeles|    USA|   CA|LAX|1.2010647369509115| |       Dallas|    USA|   TX|DFW|1.1270726146978445| |       Denver|    USA|   CO|DEN|1.0590628954667447| |San Francisco|    USA|   CA|SFO| 1.024613545715222| |     New York|    USA|   NY|LGA|0.9449041443648624| |       Boston|    USA|   MA|BOS|0.8774889102400271| |       Newark|    USA|   NJ|EWR|0.8731704325953235| |        Miami|    USA|   FL|MIA|0.8507611366339813| |      Houston|    USA|   TX|IAH|0.8350494969577277| |    Charlotte|    USA|   NC|CLT|0.8049025258215664| |      Seattle|    USA|   WA|SEA|0.6417798484556717| +-------------+-------+-----+---+------------------+ </code></pre> <h2>Message Passing via AggregateMessages</h2> <p>Many important graph algorithms are iterative algorithms, since properties of vertices depend on properties of their neighbors, which depend on properties of their neighbors. Pregel is an iterative graph processing model, developed at Google, which uses a sequence of iterations of messages passing between vertices in a graph. GraphFrames provides aggregateMessages, which implements an aggregation message-passing API, based on the Pregel model.  GraphFrames aggregateMessages sends messages between vertices and aggregates message values from the neighboring edges and vertices of each vertex.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image19-1608101204300.png" alt=""></p> <p>The code below shows how to use aggregateMessages to compute the average flight delay by the originating airport. The flight delay for each flight is sent to the src vertex, then the average is calculated at the vertices.</p> <pre><code class="language-scala">import org.graphframes.lib.AggregateMessages val AM = AggregateMessages val msgToSrc = AM.edge("depdelay") val agg = { graph.aggregateMessages   .sendToSrc(msgToSrc)       .agg(avg(AM.msg).as("avgdelay"))   .orderBy(desc("avgdelay"))   .limit(5) } agg.show() --- result: --- +---+------------------+ | id|          avgdelay| +---+------------------+ |EWR|17.818079459546404| |MIA|17.768691978431264| |ORD|  16.5199551010227| |ATL|15.330084535057185| |DFW|15.061909338459074| +---+------------------+ </code></pre> <h2>Motif Find for Graph Pattern Queries</h2> <p>Motif finding searches for structural patterns in a graph. In this example, we want to find flights with no direct connection. First, we create a subgraph from the flightroutecount DataFrame that we created earlier, which gives us a subgraph with all the possible flight routes. Then, we do a find on the pattern shown here to search for flights from a to b and b to c, that do not have a flight from a to c.  Finally, we use a DataFrame filter to remove duplicates. This shows how Graph queries can be easily combined with DataFrame operations like filter.</p> <p><strong>What are the flight routes with no direct connection?</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image8-1608101212457.png" alt=""></p> <pre><code class="language-scala">val subGraph = GraphFrame(graph.vertices, flightroutecount) val res = subGraph  .find("(a)-[]->(b); (b)-[]->(c); !(a)-[]->(c)")  .filter("c.id !=a.id”) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image26-1608101221169.png" alt=""></p> <h2>Shortest Path Graph Algorithm</h2> <p>Shortest path computes the shortest paths from each vertex to the given sequence of landmark vertices. Here, we search for the shortest path from each airport to LGA. The results show that there are no direct flights from LAX, SFO, SEA, and EWR to LGA (the distances greater than 1).</p> <pre><code class="language-scala">val results = graph.shortestPaths.landmarks(Seq("LGA")).run() +---+----------+ | id| distances| +---+----------+ |IAH|[LGA -> 1]| |CLT|[LGA -> 1]| |LAX|[LGA -> 2]| |DEN|[LGA -> 1]| |DFW|[LGA -> 1]| |SFO|[LGA -> 2]| |LGA|[LGA -> 0]| |ORD|[LGA -> 1]| |MIA|[LGA -> 1]| |SEA|[LGA -> 2]| |ATL|[LGA -> 1]| |BOS|[LGA -> 1]| |EWR|[LGA -> 2]| +---+----------+ </code></pre> <h2>Breadth First Search  Graph Algorithm</h2> <p>Breadth-first search (BFS) finds the shortest path from beginning vertices to end vertices. The beginning and end vertices are specified as DataFrame expressions, maxPathLength specifies the limit on the length of paths. Here we see that there are no Direct flights between LAX and LGA.</p> <pre><code class="language-scala">val paths = graph.bfs.fromExpr("id = 'LAX'")  .toExpr("id = 'LGA'")  .maxPathLength(1).run() paths.show() +----+-------+-----+---+ |City|Country|State| id| +----+-------+-----+---+ +----+-------+-----+---+ </code></pre> <p>Here we set the maxPathLength to 2. The results show some flights connecting through IAH for flights from LAX to LGA.</p> <pre><code class="language-scala">val paths = graph.bfs.fromExpr("id = 'LAX'")  .toExpr("id = 'LGA'")  .maxPathLength(2).run().limit(4) paths.show() </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image17-1608101229234.png" alt=""></p> <p>You can combine motif searching with DataFrames operations. Here we want to find connecting flights between LAX and LGA using a Motif find query. We use a Motif query to search for the pattern of a flying to b, connecting through c, then we use a DataFrame filter on the the results for A=LAX and C=LGA. The results show some flights connecting through IAH for flights from LAX to LGA.</p> <pre><code class="language-scala">graph.find("(a)-[ab]->(b); (b)-[bc]->(c)") .filter("a.id = 'LAX'") .filter("c.id = 'LGA'").limit(4) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image6-1608101238194.png" alt=""></p> <p>Combining a Motif find with DataFrame operations, we can narrow these results down further, for example flights with the arrival flight time before the departure flight time, and/or with a specific carrier.</p> <h2>Summary</h2> <p>GraphFrames provides a scalable and easy way to query and process large graph datasets, which can be used to solve many types of analysis problems. In this chapter, we gave an overview of the GraphFrames graph processing APIs. We encourage you to try out GraphFrames in more depth on some of your own projects.</p> <p>All of the components of the use case we just discussed can run on the same cluster with the MapR Data Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image27-1608101246907.jpg" alt=""></p> <h2>Code</h2> <ul> <li>You can download the code and data to run these examples from here: <a href="https://github.com/mapr-demos/mapr-spark2-ebook">https://github.com/mapr-demos/mapr-spark2-ebook</a></li> </ul>https://developer.hpe.com/apache-spark-packages-from-xml-to-json/https://developer.hpe.com/apache-spark-packages-from-xml-to-json/Fri, 11 Dec 2020 03:26:34 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-08-23T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>The Apache Spark community has put a lot of effort into extending Spark. Recently, we wanted to transform an XML dataset into something that was easier to query. We were mainly interested in doing data exploration on top of the billions of transactions that we get every day. XML is a well-known format, but sometimes it can be complicated to work with. In Apache Hive, for instance, we could define the structure of the schema of our XML and then query it using SQL.</p> <p>However, it was hard for us to keep up with the changes on the XML structure, so the previous option was discarded. We were using Spark Streaming capabilities to bring these transactions to our cluster, and we were thinking of doing the required transformations within Spark. However, the same problem remained, as we had to change our Spark application every time the XML structure changed.</p> <p>There must be another way!</p> <p>There is an Apache Spark package from the community that we could use to solve these problems. In this blog post, I'll walk you through how to use an Apache Spark package from the community to read any XML file into a DataFrame.</p> <p>Let’s load the Spark shell and see an example:</p> <pre><code class="language-Scala">./spark-shell — packages com.databricks:spark-xml_2.10:0.3.3 </code></pre> <p>In here, we just added the XML package to our Spark environment. This of course can be added when writing a Spark app and packaging it into a jar file.</p> <p>Using the package, we can read any XML file into a DataFrame. When loading the DataFrame, we could specify the schema of our data, but this was our main concern in the first place, so we will let Spark infer it. The inference of the DataFrame schema is a very powerful trick since we don’t need to know the schema anymore so it can change at any time.</p> <p>Let’s see how we load our XML files into a DataFrame:</p> <pre><code class="language-Scala">val df = sqlContext .read .format("com.databricks.spark.xml") .option("rowTag", "OrderSale") .load("~/transactions_xml_folder/") df.printSchema </code></pre> <p>Printing the DataFrame schema gives us an idea of what the inference system has done.</p> <pre><code class="language-Scala">root |-- @ApplicationVersion: string (nullable = true) |-- @BusinessDate: string (nullable = true) |-- @Change: double (nullable = true) |-- @EmployeeId: long (nullable = true) |-- @EmployeeName: string (nullable = true) |-- @EmployeeUserId: long (nullable = true) |-- @MealLocation: long (nullable = true) |-- @MessageId: string (nullable = true) |-- @OrderNumber: long (nullable = true) |-- @OrderSourceTypeId: long (nullable = true) |-- @PosId: long (nullable = true) |-- @RestaurantType: long (nullable = true) |-- @SatelliteNumber: long (nullable = true) |-- @SpmHostOrderCode: string (nullable = true) |-- @StoreNumber: long (nullable = true) |-- @TaxAmount: double (nullable = true) |-- @TaxExempt: boolean (nullable = true) |-- @TaxInclusiveAmount: double (nullable = true) |-- @TerminalNumber: long (nullable = true) |-- @TimeZoneName: string (nullable = true) |-- @TransactionDate: string (nullable = true) |-- @TransactionId: long (nullable = true) |-- @UTCOffSetMinutes: long (nullable = true) |-- @Version: double (nullable = true) |-- Items: struct (nullable = true) | |-- MenuItem: struct (nullable = true) | | |-- #VALUE: string (nullable = true) | | |-- @AdjustedPrice: double (nullable = true) | | |-- @CategoryDescription: string (nullable = true) | | |-- @DepartmentDescription: string (nullable = true) | | |-- @Description: string (nullable = true) | | |-- @DiscountAmount: double (nullable = true) | | |-- @Id: long (nullable = true) | | |-- @PLU: long (nullable = true) | | |-- @PointsRedeemed: long (nullable = true) | | |-- @Price: double (nullable = true) | | |-- @PriceLessIncTax: double (nullable = true) | | |-- @PriceOverride: boolean (nullable = true) | | |-- @ProductivityUnitQuantity: double (nullable = true) | | |-- @Quantity: long (nullable = true) | | |-- @TaxAmount: double (nullable = true) | | |-- @TaxInclusiveAmount: double (nullable = true) |-- OrderTaxes: struct (nullable = true) | |-- TaxByImposition: struct (nullable = true) | | |-- #VALUE: string (nullable = true) | | |-- @Amount: double (nullable = true) | | |-- @ImpositionId: long (nullable = true) | | |-- @ImpositionName: string (nullable = true) |-- Payments: struct (nullable = true) | |-- Payment: struct (nullable = true) | | |-- #VALUE: string (nullable = true) | | |-- @AccountIDLast4: string (nullable = true </code></pre> <p>At this point, we could use any SQL tool to query our XML using Spark SQL. Please read this post <a href="https://medium.com/@anicolaspp/apache-spark-as-a-distributed-sql-engine-4373e254e0f9#.w77z4ml3r">Apache Spark as a Distributed SQL Engine</a> to learn more about Spark SQL. Going a step further, we could use tools that can read data in JSON format. Having JSON datasets are especially useful if you have something like Apache Drill.</p> <p>As we could expect, with Spark we can do any kind of transformations, but there is no need to write a fancy JSON encoder because Spark already supports these features. Let’s convert our DataFrame to JSON and save it our file system.</p> <pre><code class="language-Scala">val jsons = df.toJSON jsons.saveAsTextFile("~/json_folder/") </code></pre> <p>When applying the JSON function to the DataFrame, we get <code>an RDD[String]</code> with the JSON representation of our data. Then we save the RDD as a plain text file. Now, we could use Drill to read and query our new dataset and of course, we can always go back to Spark if we need to do something more complicated operations / transformations.</p> <h2>Conclusion</h2> <p>Transforming our dataset from XML to JSON is an easy task in Spark, but the advantages of JSON over XML are a big deal. We now can rest assured that XML schema changes are not going to affect us at all. We have removed ourselves from the burden of changing our application for every XML change. We can also use powerful tools to query our JSON dataset such as Apache Drill in a schema free fashion while our clients can report on our data using SQL.</p>https://developer.hpe.com/types-of-machine-learning-part-2-in-the-intro-to-aiml-series/https://developer.hpe.com/types-of-machine-learning-part-2-in-the-intro-to-aiml-series/Wed, 09 Dec 2020 07:38:36 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Saira Kennedy", "publish": "2018-09-28T07:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p>Based on MapR Academy course, <a href="https://learn.ezmeral.software.hpe.com/bus-introduction-to-artificial-intelligence-and-machine-learning">Introduction to Artificial Intelligence and Machine Learning</a></p> <p>In this post – second in the Intro to AI/ML Series – we discuss the different methods of machine learning and some of the most common algorithms available for your projects. <a href="/blog/kkLZ0mNpXqhQMZLRJQqr/artificial-intelligence-and-machine-learning-what-are-they-and-why-are-t">Read the first blog in this series here</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image12-1607499687612.jpg" alt=""></p> <h2>Types of Machine Learning</h2> <p>There are a few different types of machine learning, but they generally fall into these main groups. Supervised and unsupervised learning are the primary learning types, along with semi-supervised learning.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image8-1607499695694.jpg" alt=""></p> <p>Other methods sit in the middle or outskirts of these methodologies, such as reinforcement learning, which describes a machine that creates a cyclical learning cycle as it continuously trains itself from its own results. These results are then fed back into itself as input data. We won't be going into much detail on these other techniques, but further information can be found online.</p> <p>First, let's understand what differentiates each of these learning types from each other and how they work.</p> <h2>Supervised Learning: Defined</h2> <p>Supervised learning uses labeled data to train machines to learn the relationships between given inputs and outputs.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image4-1607499702919.jpg" alt=""></p> <p>A label is a known description given to objects in the data, which trains the machine on what to look for. Labels also provide the structure of the algorithm output, as any result must be one of these labels. Therefore, you can think of labels as a schema, defining the possible output that we want the machine to look for.</p> <p>Think of supervised learning as the algorithm to use when data scientists have labeled input data and when the type of behavior to predict is known. We want the machine to learn the patterns used to classify this data and apply those patterns to classify new data.</p> <h2>How Supervised Learning Works</h2> <ul> <li>First, labeled or classified data is loaded into the system. The preparation of labeled data makes this step the most time-consuming, as it is often done by a human trainer.</li> <li>The model is trained and connections to inputs and outputs are made.</li> <li>As new data is introduced, the algorithm is applied.</li> <li>Output is categorized data.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image9-1607499710729.jpg" alt=""></p> <p>In the previous post, where we provided the cat example on labeled data, the trained labels include ears, nose, tail, paws, and cat, which the algorithm then applies to presented data, in this case an image of a cat, and returns the results of known output as "cat," yes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image10-1607499718508.jpg" alt=""></p> <h2>Pros and Cons of Supervised Learning</h2> <p>Supervised learning always has a clear objective and can be easily measured for accuracy. The training of the machine is also tightly controlled, which leads to very specific behavioral outcomes.</p> <p>On the downside, it is often very labor-intensive, as all data needs to be labeled before the model is trained, which can take hundreds of hours of specialized human effort. The costs can become astronomical. This creates an overall slower training process and may also limit the data that it can work with.</p> <table> <thead> <tr> <th>Pros</th> <th>Cons</th> </tr> </thead> <tbody> <tr> <td>Very clear objective</td> <td>Often labor-intensive</td> </tr> <tr> <td>Easy to measure accuracy</td> <td>Limited data to work with</td> </tr> <tr> <td>Controlled training</td> <td>Limited insights</td> </tr> </tbody> </table> <p>Finally, insights may be more limited, as the predicted behavior is described in advance. There is no freedom for the machine to explore other possibilities, as we will see with unsupervised learning.</p> <p>In supervised learning, there are primarily two categories of algorithms: classification and regression.</p> <p>A classification algorithm organizes input data as belonging to one of several predefined classes. This algorithm is the most useful for providing categorical results that fit within the predefined labels. It is very effective with well-calculated if-then rules and distinguishes one class of objects from another.</p> <table> <thead> <tr> <th>Type</th> <th>Algorithm or Task</th> </tr> </thead> <tbody> <tr> <td>Supervised</td> <td>Classification (used to predict a categorical result)</td> </tr> <tr> <td>Supervised</td> <td>Regression (used to predict the output value given the input value)</td> </tr> </tbody> </table> <h2>Supervised Algorithms</h2> <h2>Classification: Used to predict a categorical result</h2> <p>Some common use cases for classification algorithms include credit card fraud detection and email spam detection, both of which are binary classification problems, meaning there are only two possible output values. Data is labeled, for example, as fraud/non-fraud or spam/non-spam.</p> <p>Generally, if the question we are asking of a model is open-ended or if the potential answers are not categorical, then we aren't dealing with a classification problem, but more likely a regression one.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image13-1607499726836.jpg" alt=""></p> <p>A regression algorithm attempts to predict the output value given the input value. Regression problems are predictive of a continuous numerical, as opposed to categorical, result.</p> <p>Think of this continuous value as a range or average, something that is estimating the relationship between variables.</p> <h2>Regression: Used to predict the output value given the input value</h2> <p>For example, this type of algorithm can be used to determine how profitable a credit card model is. It is also used to predict customer or employee churn models.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image2-1607499735202.jpg" alt=""></p> <p>Regression algorithms determine the strength of correlation between two attributes, allowing you to find a predictive range of likelihood.</p> <h2>Supervised Algorithms Table</h2> <p>This table depicts some of the most common algorithms used with supervised learning types. It is important to understand that many machine learning data models will use more than one, and sometimes many, different algorithms for a project.</p> <table> <thead> <tr> <th>Type</th> <th>Algorithm or Task</th> </tr> </thead> <tbody> <tr> <td>Classification</td> <td>Naïve Bayes</td> </tr> <tr> <td>Classification</td> <td>Logistic Regression</td> </tr> <tr> <td>Classification</td> <td>Support Vector Machines (SVMs)</td> </tr> <tr> <td>Regression</td> <td>Linear Regression</td> </tr> <tr> <td>Both</td> <td>Decision Trees/Random Forest</td> </tr> <tr> <td>Both</td> <td><em>k</em>-Nearest Neighbors (<em>k</em>-NN)</td> </tr> <tr> <td>Both</td> <td>Gradient Boosting Algorithms</td> </tr> </tbody> </table> <h2>Unsupervised Learning: Defined</h2> <p>While supervised learning involves having labeled data to find input-output relationships during the training phase, unsupervised learning has no knowledge of the output label. In this type of ML, the machine finds groups and patterns in the data on its own, and there is no specific outcome or target to predict.</p> <p>Think of unsupervised learning as the algorithm to use when we don't know how to classify the data and we want the machine to classify or group it for us.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image11-1607499742282.jpg" alt=""></p> <h2>How Unsupervised Learning Works</h2> <p>Here are the steps of how the unsupervised learning algorithm works:</p> <ul> <li>First, unlabeled raw data is loaded into the system.</li> <li>Next, the algorithm analyzes the data.</li> <li>It looks for patterns on its own.</li> <li>Then, it identifies and groups patterns of behavior and provides output results.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image1-1607499749885.jpg" alt=""></p> <h2>Pros and Cons of Unsupervised Learning</h2> <p>Compared to supervised learning, unsupervised learning projects are much faster to implement, as no data labeling is required. In this regard, it uses fewer human resources. It also interprets data on its own and has the potential to provide unique, disruptive insights for a business to consider.</p> <p>However, unsupervised learning can be difficult to measure for accuracy because there is no expected result to compare it to. It can require more experimentation and tuning to get meaningful results.</p> <p>Lastly, unsupervised learning does not natively handle high-dimensional data, or massively large datasets with considerable variance, well. This is known as the curse of dimensionality. In some cases, the dimensions, or number of variables, may need to be reduced for it to work effectively. This requires human-intensive data cleansing.</p> <table> <thead> <tr> <th>Pros</th> <th>Cons</th> </tr> </thead> <tbody> <tr> <td>Very fast to start</td> <td>Difficult to measure accuracy</td> </tr> <tr> <td>Disruptive insights</td> <td>Requires more experience</td> </tr> <tr> <td></td> <td>Curse of dimensionality</td> </tr> </tbody> </table> <h2>Common Use Cases for Unsupervised Algorithms</h2> <p>Let's take a look at a common use case example using cluster analysis. Cluster analysis has the goal of organizing raw data into related groups and is often used for anomaly detection.</p> <p>A security company uses it to identify unusual patterns in network traffic, indicating potential signs of a security breach or intrusion.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image5-1607499758343.jpg" alt=""></p> <h2>Unsupervised Learning Use Case – Anomaly Detection</h2> <p>Recall the steps of how this type of algorithm works.</p> <ul> <li>First, the security company streams in raw network traffic data.</li> <li>Next, the algorithm analyzes the data on its own and looks for unusual patterns.</li> <li>Then, it identifies patterns of behavior as either normal or suspect.</li> <li>When suspect behavior is identified, the output is provided and the company is notified.</li> </ul> <p>With this example using anomaly detection, a scatter plot may return results looking something like the image below. The green dots indicate behavior that is grouped together as normal, and the red dots show the potential outliers that are sent back as suspect.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image14-1607499766291.jpg" alt=""></p> <p>This table depicts some unsupervised learning algorithms. The most common algorithm here is <em>k</em>-means, for cluster analysis, which is what we've just focused on with our security use case example on anomaly detection.</p> <table> <thead> <tr> <th>Type</th> <th>Algorithm or Task</th> </tr> </thead> <tbody> <tr> <td>Unsupervised</td> <td>_k_-Means: Cluster Analysis</td> </tr> <tr> <td>Unsupervised</td> <td>Association Rule Learning</td> </tr> <tr> <td>Unsupervised</td> <td>Dimensionality Reduction Techniques (PCA, SVD)</td> </tr> </tbody> </table> <h2>Semi-Supervised Learning: Defined</h2> <p>Semi-supervised learning includes a combination of supervised and unsupervised learning types together. Usually, this means that only a part of the provided input data is labeled, which the machine is trained on. It then learns to create additional labels and classifiers for raw data, on its own, which in turn gets added back to the original training data set.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image6-1607499774519.jpg" alt=""></p> <h2>How Semi-Unsupervised Learning Works</h2> <p>As a combination of the previous two learning types, let's look at how a common self-training algorithm, from the semi-supervised learning method, works:</p> <ul> <li>First, an initial set of labeled input training data is loaded into the system.</li> <li>The model is trained on the data. Then, a new data set of unlabeled data is presented.</li> <li>The algorithm infers new labels and classifiers to apply to the new data. High-confidence data, or data that scores well based on the algorithm, is added back to the original labeled data set. From here, the machine progressively adapts and learns in an iterative process.</li> <li>In some cases, when the labels and the rule-based engine conflicts, a human is needed for verification.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image15-1607499782353.jpg" alt=""></p> <h2>Semi-Supervised Algorithms Table</h2> <p>This table depicts some semi-supervised learning algorithms.</p> <table> <thead> <tr> <th>Type</th> <th>Algorithm or Task</th> </tr> </thead> <tbody> <tr> <td>Semi-Supervised</td> <td>Self-Training Algorithms</td> </tr> <tr> <td>Semi-Supervised</td> <td>Generative Model – Gaussian Mixture Model</td> </tr> <tr> <td>Semi-Supervised</td> <td>Graph-Based Algorithms – Label Propagation</td> </tr> </tbody> </table> <h2>Check your Knowledge</h2> <p>Classify the items listed below as a supervised, semi-supervised, or unsupervised learning method:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image16-1607499789317.png" alt=""></p> <p>Answer Key:</p> <p>Works on both labeled data and raw data: Semi-supervised</p> <p>Easiest data preparation method: Unsupervised</p> <p>Only uses labeled input data: Supervised</p> <p>Infers patterns on its own: Unsupervised</p> <p>Output is predefined: Supervised</p> <p>Can be used to automate data labeling: Semi-supervised</p> <h2>More Resources:  Machine Learning Libraries</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/12/image3-1607499796807.jpg" alt=""></p> <h2>Where do we go from here</h2> <p>Keep your eyes out for the next post in this series, discussing real world use cases for AI and ML.</p>https://developer.hpe.com/2020-December-01/https://developer.hpe.com/2020-December-01/Tue, 01 Dec 2020 06:00:00 GMThttps://developer.hpe.com/introducing-hpe-nimble-storage-sdk-for-go/https://developer.hpe.com/introducing-hpe-nimble-storage-sdk-for-go/Wed, 25 Nov 2020 18:25:52 GMT<p>In the cloud-native, API-first economy, the infrastructure platform relevance increases when gauged against how well it integrates with others. The Go Programming Language is considered to be the language of choice for the cloud and the building microservices. Many large, recognized open source projects use Go, such as Kubernetes, Docker, InfluxDB and Terraform. Some of the most successful companies in the new API economy use Go, including Uber, Netflix, Google, and of course, Hewlett Packard Enterprise (HPE). Go was invented by Google and designed to scale concurrency across networked systems and support large codebases, which many of the successful projects can attest to.</p> <p>Today HPE introduces the HPE Nimble Storage SDK for Go. Built on top of the robust REST API available in NimbleOS, developers may take advantage of the full suite of APIs to build robust applications utilizing HPE Nimble Storage features and capabilities.</p> <h1>Example usage</h1> <p>There are some more elaborate examples available <a href="https://github.com/hpe-storage/nimble-golang-sdk/tree/master/examples">in the GitHub repo</a>, but let's see a how a complete example program could work. Instructions on how to install Go on your computer is <a href="https://golang.org/doc/install">available here</a>.</p> <p>Save this file to <code>createvolume.go</code>:</p> <pre><code class="language-go">package main import ( "fmt" "flag" "github.com/hpe-storage/nimble-golang-sdk/pkg/client/v1/nimbleos" "github.com/hpe-storage/nimble-golang-sdk/pkg/service" ) func main() { // Input parameters groupParam := flag.String("g", "myarray", "Nimble group `IP/hostname`") usernameParam := flag.String("u", "admin", "Nimble array `username`") passwordParam := flag.String("p", "admin", "Nimble array `password`") volumeNameParam := flag.String("v", "myvol", "Volume `name` to create") volumeSizeParam := flag.Int64("s", 5120, "Volume `size in MB`") // Parse flags flag.Parse() // Create new group service groupService, err := service.NewNsGroupService( *groupParam, *usernameParam, *passwordParam, "v1", true) if err != nil { fmt.Printf("Unable to connect to group, %+v\n", err.Error()) return } // Logout when finished defer groupService.LogoutService() // Set mandatory volume attributes newVolume := &#x26;nimbleos.Volume{ Name: volumeNameParam, Size: volumeSizeParam, } // Assign volume service instance volSvc := groupService.GetVolumeService() // Create volume volume, err := volSvc.CreateVolume(newVolume) if err != nil { fmt.Printf("Failed to create volume, %+v\n", err.Error()) return } // Volume created fmt.Printf("Volume \"%s\" created (%s)\n", *volume.Name, *volume.ID) } </code></pre> <p>Go is a compiled language, so, in order to run the example, it needs to be compiled:</p> <pre><code class="language-go">go build createvolume.go </code></pre> <p>Now, we have the ability to create volumes from a remote command-line:</p> <pre><code class="language-markdown">./createvolume -h Usage of ./createvolume: -g IP/hostname Nimble group IP/hostname (default "myarray") -p password Nimble array password (default "admin") -s size in MB Volume size in MB (default 5120) -u username Nimble array username (default "admin") -v name Volume name to create (default "myvol") </code></pre> <p>Let's create a new volume:</p> <pre><code class="language-markdown">./createvolume -g myarray1.example.com -u admin -p admin -v myvol1 -s 10240 Volume "myvol1" created (06412618225c784ec900000000000000000000000b) </code></pre> <p>And that is how simple it is with only a few lines of code to create a custom CLI for a common operation.</p> <blockquote> <p><strong>Note:</strong> It's extremely bad practice to use a password on the command-line, please use other means to supply the password to the SDK in a production scenario.</p> </blockquote> <h1>Available endpoints</h1> <p>Each of the API endpoints available on the REST service is available to the SDK and is mapped from the plural version of the endpoint to the singular version of the service. For example, <code>v1/volumes</code> is accessible on <code>myService.GetVolumeService()</code> (<code>myService</code> being a <code>GroupService</code> object).</p> <p>These are the currently available services in the SDK.</p> <pre><code class="language-markdown">AccessControlRecordService MasterKeyService ActiveDirectoryMembershipService NetworkConfigService AlarmService NetworkInterfaceService ApplicationCategoryService PerformancePolicyService ApplicationServerService PoolService ArrayService ProtectionScheduleService AuditLogService ProtectionTemplateService ChapUserService ProtocolEndpointService ControllerService ReplicationPartnerService DiskService ShelfService EventService SnapshotCollectionService FibreChannelConfigService SnapshotService FibreChannelInitiatorAliasService SoftwareVersionService FibreChannelInterfaceService SpaceDomainService FibreChannelPortService SubnetService FibreChannelSessionService TokenService FolderService UserGroupService GroupService UserService InitiatorGroupService VersionService InitiatorService VolumeCollectionService JobService VolumeService </code></pre> <p>The list of services will grow as new versions of NimbleOS introduce new features.</p> <h1>Start writing code</h1> <p>The SDK is available immediately and we encourage feedback and collaboration. Check out the following resources to learn more.</p> <ul> <li>HPE Nimble Storage SDK for Go <a href="https://github.com/hpe-storage/nimble-golang-sdk">on GitHub</a></li> <li><a href="https://tour.golang.org/">Learn Go</a> from the official language creators</li> <li>Learn more about HPE Nimble Storage on <a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">the HPE DEV platform page</a></li> </ul> <p>The HPE Nimble Storage team hangs out on the HPE DEV Slack community in <a href="https://hpedev.slack.com/archives/C7TTAHRUN">#NimbleStorage</a>. Sign up at <a href="https://slack.hpedev.io">slack.hpedev.io</a> and login to the Slack at <a href="https://hpedev.slack.com">hpedev.slack.com</a>. Please tell us what you're building!</p>https://developer.hpe.com/apache-spark-machine-learning-tutorial/https://developer.hpe.com/apache-spark-machine-learning-tutorial/Wed, 25 Nov 2020 03:08:41 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2016-02-22T08:00:00.000Z", "update": "2019-02-20T08:00:00.000Z", "tags": ["machine-learning","tutorial"] </code></pre> <hr> <p>In this blog post, we will give an introduction to machine learning and deep learning,  and we will go over the main Spark machine learning algorithms and techniques with some real-world use cases. The goal is to give you a better understanding of what you can do with machine learning. Machine learning is becoming more accessible to developers, and data scientists work with domain experts, architects, developers, and data engineers, so it is important for everyone to have a better understanding of the possibilities. Every piece of information that your business generates has potential to add value. This overview is meant to provoke a review of your own data to identify new opportunities.</p> <p>With Apache Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster:</p> <ul> <li>the Spark SQL and the Dataset/DataFrame APIs provide ease of use, space efficiency, and performance gains with Spark SQL's optimized execution engine.  </li> <li>Spark ML provides a uniform set of high-level APIs, built on top of DataFrames with the goal of making machine learning scalable and easy.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image16-1606274834703.jpg" alt=""></p> <h2>What Is Machine Learning?</h2> <p>Machine learning uses algorithms to find patterns in data and then uses a model that recognizes those patterns to make predictions on new data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image4-1606274844591.jpg" alt=""></p> <p>There are typically two phases in machine learning:</p> <ul> <li>Data Discovery: The first phase involves analysis on historical data to build and train the machine learning model.</li> <li>Analytics Using the Model: The second phase uses the model in production on new data.</li> </ul> <p>In production, models need to be continuously monitored and updated with new models when needed.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image21-1606274852952.jpg" alt=""></p> <p>In general, machine learning may be broken down into two types: supervised, unsupervised, and in between those two. Supervised learning algorithms use labeled data; unsupervised learning algorithms find patterns in unlabeled data. Semi-supervised learning uses a mixture of labeled and unlabeled data. Reinforcement learning trains algorithms to maximize rewards based on feedback.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image24-1606274861604.jpg" alt=""></p> <h2>Three Common Categories of Techniques for Machine Learning</h2> <p>Three common categories of machine learning techniques are classification, clustering, and collaborative filtering.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image25-1606274869972.jpg" alt=""></p> <ul> <li>Classification: Gmail uses a machine learning technique called classification to designate if an email is spam or not, based on the data of an email: the sender, recipients, subject, and message body. Classification takes a set of data with known labels and learns how to label new records based on that information.</li> <li>Clustering: Google News uses a technique called clustering to group news articles into different categories, based on title and content. Clustering algorithms discover groupings that occur in collections of data.</li> <li>Collaborative Filtering: Amazon uses a machine learning technique called collaborative filtering (commonly referred to as recommendation) to determine which products users will like, based on their history and similarity to other users.</li> </ul> <h2>Supervised Learning: Classification and Regression</h2> <p>Supervised algorithms use labeled data in which both the input and target outcome, or label, are provided to the algorithm.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image13-1606274878211.jpg" alt=""></p> <p>Supervised learning is also called predictive modeling or predictive analytics, because you build a model that is capable of making predictions.</p> <p>Some examples of predictive modeling are classification and regression. Classification identifies which category an item belongs to (e.g., whether a transaction is fraud or not fraud), based on labeled examples of known items (e.g., transactions known to be fraud or not).  Logistic regression predicts a probability (e.g., the probability of fraud). Linear regression predicts a numeric value (e.g., the amount of fraud).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image23-1606274887440.jpg" alt=""></p> <h2>Classification and Regression Example</h2> <p>Classification and regression take a set of data with known labels and predetermined features and learns how to label new records based on that information. Features are the "if questions" that you ask. The label is the answer to those questions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image6-1606274895544.jpg" alt=""></p> <h3>Regression Example</h3> <p>Let's go through an example of car insurance fraud:</p> <ul> <li>What are we trying to predict? <ul> <li>This is the label: the amount of fraud</li> </ul> </li> <li>What are the "if questions" or properties that you can use to predict? <ul> <li>These are the features: to build a classifier model, you extract the features of interest that most contribute to the classification.</li> <li>In this simple example, we will use the claimed amount.</li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image7-1606274904051.jpg" alt=""></p> <p>Linear regression models the relationship between the Y "Label" and the X "Feature," in this case the relationship between the amount of fraud and the claimed amount. The coefficient measures the impact of the feature, the claimed amount, and on the label, the fraud amount.</p> <p>Multiple linear regression models the relationship between two or more "Features" and a response "Label." For example, if we wanted to model the relationship between the amount of fraud and the age of the claimant, the claimed amount, and the severity of the accident, the multiple linear regression function would look like this:</p> <p>   Y<sub>i</sub> = β<sub>0</sub> + β<sub>1</sub>X<sub>1</sub> + β<sub>2</sub>X<sub>2</sub> + · · · + β<sub>p</sub> X<sub>p</sub> + Ɛ</p> <p><strong>Amount Fraud = intercept + (coefficient1 * age) + (coefficient2 * claimed Amount) + (coefficient3 * severity) + error.</strong></p> <p>The coefficients measure the impact on the fraud amount of each of the features.</p> <p>Some examples of linear regression include:</p> <ul> <li>Given historical car insurance fraudulent claims and features of the claims, such as age of the claimant, claimed amount, and severity of the accident, predict the amount of fraud.</li> <li>Given historical real estate sales prices and features of houses (square feet, number of bedrooms, location, etc.), predict a house's price.</li> <li>Given historical neighborhood crime statistics, predict crime rate.</li> </ul> <h3>Classification Example</h3> <p>Let's go through an example of debit card fraud:</p> <ul> <li>What are we trying to predict? <ul> <li>This is the label: probability of fraud</li> </ul> </li> <li>What are the "if questions" or properties that you can use to make predictions? <ul> <li>Is the amount spent today > historical average?</li> <li>Are there transactions in multiple countries today?</li> <li>Are the number of transactions today > historical average?</li> <li>Are the number of new merchant types today high compared to the last 3 months?</li> <li>Are there multiple purchases today from merchants with a category code of risk?</li> <li>Is there unusual signing activity today, compared to historically using pin?</li> <li>Are there new state purchases compared to the last 3 months?</li> <li>Are there foreign purchases today compared to the last 3 months?</li> </ul> </li> </ul> <p>To build a classifier model, you extract the features of interest that most contribute to the classification.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image18-1606274916889.jpg" alt=""></p> <p>Logistic regression measures the relationship between the Y "Label" and the X "Features" by estimating probabilities using a <a href="https://en.wikipedia.org/wiki/Logistic_function">logistic function</a>. The model predicts a probability, which is used to predict the label class.</p> <p>Some examples of classification include:</p> <ul> <li>Given historical car insurance fraudulent claims and features of the claims, such as age of the claimant, claimed amount, and severity of the accident, predict the probability of fraud.</li> <li>Given patient characteristics, predict the probability of congestive heart failure.</li> <li>Credit card fraud detection (fraud, not fraud)</li> <li>Credit card application (good credit, bad credit)</li> <li>Email spam detection (spam, not spam)</li> <li>Text sentiment analysis (happy, not happy)</li> <li>Predicting patient risk  (high risk patient, low risk patient)</li> <li>Classifying a tumor (malignant, not malignant)</li> </ul> <h2>Spark Supervised Algorithms Summary</h2> <p>Classification</p> <ul> <li>Logistic regression</li> <li>Decision tree classifier</li> <li>Random forest classifier</li> <li>Gradient-boosted tree classifier</li> <li>Multilayer perceptron classifier</li> <li>Linear Support Vector Machine</li> <li>Naive Bayes</li> </ul> <p>Regression</p> <ul> <li>Linear regression</li> <li>Generalized linear regression</li> <li>Decision tree regression</li> <li>Random forest regression</li> <li>Gradient-boosted tree regression</li> <li>Survival regression</li> <li>Isotonic regression</li> </ul> <h2>Unsupervised Learning         </h2> <p>Unsupervised learning, also sometimes called descriptive analytics, does not have  labeled data provided in advance. These algorithms discover similarities, or regularities, in the input data.  An example of unsupervised learning is grouping similar customers, based on purchase data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image22-1606274925028.jpg" alt=""></p> <h3>Clustering</h3> <p>In clustering, an algorithm classifies inputs into categories by analyzing similarities between input examples.  Some clustering use cases include:</p> <ul> <li>Search results grouping</li> <li>Grouping similar customers</li> <li>Grouping similar patients</li> <li>Text categorization</li> <li>Network Security Anomaly detection (anomalies find what is <strong>not</strong> similar, which means the outliers from clusters)</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image12-1606274934428.jpg" alt=""></p> <p>The <em>k</em>-means algorithm groups observations into <em>k</em> clusters in which each observation belongs to the cluster with the nearest mean from its cluster center.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image17-1606274943600.jpg" alt=""></p> <p>An example of clustering is a company that wants to segment its customers in order to better tailor products and offerings. Customers could be grouped on features such as demographics and purchase histories. Clustering with unsupervised learning is often combined with supervised learning in order to get more valuable results. For example, in a banking customer 360 use case, customers were first clustered based on answers to a survey. The customer groups were analyzed and then labeled with customer personas. Next, the persona labels were linked by customer ID with customer features, such as types of accounts and purchases. Finally, supervised machine learning was applied and tested with the labeled customers, allowing it to link the survey customer personas with their banking actions and provide insights.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image3-1606275231189.jpg" alt=""></p> <h3>Frequent Pattern Mining, Association, Co-Occurrence, Market Basket Recommendations</h3> <p>Frequent pattern or association rule mining finds frequent co-occurring associations among a collection of items, such as products often purchased together. A famous story about association rule mining is the "beer and diaper" story. An analysis of behavior of grocery shoppers discovered that men who buy diapers often also buy beer.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image2-1606275329553.png" alt=""></p> <p><a href="https://www.nytimes.com/2004/11/14/business/yourmoney/what-walmart-knows-about-customers-habits.html">Walmart mined their massive retail transaction database</a> to see what their customers really wanted to buy prior to the arrival of a hurricane. They found one particular item which had an increase in sales by a factor of 7 over normal shopping days, a huge lift factor for a real-world case. The item was not bottled water, batteries, beer, flashlights, generators, or any of the usual things that you might imagine: it was strawberry pop tarts!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image20-1606274973982.png" alt=""></p> <p>Another example is from Target, which analyzed that when a woman starts buying scent-free lotion, vitamin supplements, and a combination of some other items, it signals she could be pregnant. Unfortunately, Target sent a coupon for baby items to a teenager whose father questioned why she was receiving such coupons.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image5-1606274983558.jpg" alt=""></p> <p>Co-occurrence analysis is useful for:</p> <ul> <li>Store layouts</li> <li>Determining which products to put on specials, promotions, coupons, etc.</li> <li>Identifying healthcare patients, like mine cohorts</li> </ul> <h3>Collaborative Filtering</h3> <p>Collaborative filtering algorithms recommend items (this is the filtering part) based on preference information from many users (this is the collaborative part). The collaborative filtering approach is based on similarity; people who liked similar items in the past will like similar items in the future. The goal of a collaborative filtering algorithm is to take preferences data from users and create a model that can be used for recommendations or predictions. Ted likes movies A, B, and C. Carol likes movies B and C. We take this data and run it through an algorithm to build a model. Then, when we have new data, such as Bob likes movie B, we use the model to predict that C is a possible recommendation for Bob.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image9-1606274992738.jpg" alt=""></p> <h2>Spark Unsupervised Algorithms Summary</h2> <p>Clustering</p> <ul> <li><em>k</em>-means</li> <li>Latent Dirichlet allocation (LDA)</li> <li>Gaussian mixture model (GMM)</li> </ul> <p>Collaborative Filtering</p> <ul> <li> Alternating least squares (ALS)</li> </ul> <p>Frequent Pattern Mining</p> <ul> <li>FP-Growth Algorithm</li> </ul> <h2>Deep Learning</h2> <p>Deep learning is the name for multilayered neural networks, which are networks composed of several "hidden layers" of nodes between the input and output.  There are many variations of neural networks, which you can learn more about on this <a href="http://www.asimovinstitute.org/neural-network-zoo/">neural network cheat sheet.</a>  Improved algorithms, GPUs, and massively parallel processing (MPP) have given rise to networks with thousands of layers.  Each node takes input data and a weight and outputs a confidence score to the nodes in the next layer, until the output layer is reached, where the error of the score is calculated.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image15-1606275001688.jpg" alt=""></p> <p>With <a href="https://en.wikipedia.org/wiki/Backpropagation">backpropagation</a> inside of a process called <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a>, the errors are sent back through the network again and the weights are adjusted, improving the model. This process is repeated thousands of times, adjusting a model's weights in response to the error it produces, until the error can't be reduced anymore.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image26-1606275012401.jpg" alt=""></p> <p>During this process, the layers learn the optimal features for the model, which has the advantage that features do not need to be predetermined. However, this has the disadvantage that the model's decisions are not explainable. Because explaining the decisions can be important, researchers are developing <a href="http://www.sciencemag.org/news/2017/07/how-ai-detectives-are-cracking-open-black-box-deep-learning">new ways to understand the black box of deep learning</a>.</p> <p>There are different variations of deep learning algorithms, which can be used to build data-driven applications, such as the following:</p> <ul> <li>Deep Neural Networks for improved traditional algorithms <ul> <li>Finance: enhanced fraud detection through identification of more complex patterns</li> <li>Manufacturing: enhanced identification of defects, based on deeper anomaly detection</li> </ul> </li> <li>Convolutional Neural Networks for images <ul> <li>Retail: in-store activity analysis of video to measure traffic</li> <li>Satellite images: labeling terrain, classifying objects</li> <li>Automotive: recognition of roadways and obstacles</li> <li>Healthcare: diagnostic opportunities from x-rays, scans, etc.</li> <li>Insurance: estimating claim severity, based on photographs</li> </ul> </li> <li>Recurrent Neural Networks for sequenced data <ul> <li>Customer satisfaction: transcription of voice data to text for NLP analysis</li> <li>Social media: real-time translation of social and product forum posts</li> <li>Photo captioning: search archives of images for new insights</li> <li>Finance: Predicting behavior based via time series analysis (also enhanced recommendation systems)</li> </ul> </li> </ul> <h2>Deep Learning with Spark</h2> <p>Deep learning libraries or frameworks that can be leveraged with Spark include:</p> <ul> <li>BigDL</li> <li>Spark Deep Learning Pipelines</li> <li>TensorFlowOnSpark</li> <li>dist-keras</li> <li>H2O Sparkling Water</li> <li>PyTorch</li> <li>Caffe</li> <li>MXNet</li> </ul> <h2>USING SPARK ML</h2> <p>With Apache Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster:</p> <ul> <li>the Spark SQL and the Dataset/DataFrame APIs provide ease of use, space efficiency, and performance gains.</li> <li>Spark ML provides a uniform set of high-level APIs, built on top of DataFrames with the goal of making machine learning scalable and easy. Having ML APIs built on top of DataFrames provides the scalability of partitioned data processing with the ease of SQL for data manipulation.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image8-1606275020336.jpg" alt=""></p> <p>You can use a Spark ML Pipeline to pass your data through transformers in order to extract the features, an estimator to produce a model, and an evaluator to evaluate the model.  </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image19-1606275029231.jpg" alt=""></p> <h2>Summary</h2> <p>A confluence of several different technology shifts have dramatically changed machine learning applications. The combination of distributed computing, streaming analytics, and machine learning is accelerating the development of next-generation intelligent applications, which are taking advantage of modern computational paradigms, powered by modern computational infrastructure. The MapR Data Platform integrates global event streaming, real-time database capabilities, and scalable enterprise storage with a collection of data processing and analytical engines to power this new generation of data processing pipelines and intelligent applications.</p>https://developer.hpe.com/demystifying-ai-machine-learning-and-deep-learning/https://developer.hpe.com/demystifying-ai-machine-learning-and-deep-learning/Wed, 25 Nov 2020 02:31:24 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2017-08-16T12:00:00.000", "tags": ["machine-learning","deep learning"] </code></pre> <hr> <p>Deep learning, machine learning, artificial intelligence - all buzzwords and representative of the future of analytics. In this post we will explain what is machine learning and deep learning at a high level with some real world examples. In future posts we will explore vertical use cases. The goal of this is not to turn you into a data scientist, but to give you a better understanding of what you can do with machine learning. Machine learning is becoming more accessible to developers, and Data scientists work with domain experts, architects, developers and data engineers, so it is important for everyone to have a better understanding of the possibilities. Every piece of information that your business generates has potential to add value. This and future posts are meant to provoke a review of your own data to identify new opportunities.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mlexamples-1606272628062.png" alt=""></p> <h2>What is Artificial Intelligence?</h2> <p>Throughout the history of AI the definition has been continuously redefined. AI is an umbrella term, the idea started in the 50s, machine learning is a subset of AI and deep learning is a subset of ML.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/ai-1606272643389.png" alt=""></p> <p><em>Image reference: <a href="https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/">AI/ML/DL</a></em></p> <p>In 1985, when I was a student interning at the NSA, AI was also a very hot topic. At the NSA I even took an MIT video (VCR) class on AI, which was about expert systems. Expert systems capture an expert's knowledge in a rules engine. Rules engines have wide use in industries such as finance and healthcare, and more recently for event processing, but when data is changing, rules can become difficult to update and maintain. Machine learning has the advantage that it learns from the data, and it can give data driven probabilistic predictions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/system-1606272654657.png" alt="Expert System"></p> <p><em>Image reference: <a href="https://www.pcmag.com/encyclopedia/term/42865/expert-system">expert-system</a></em></p> <p><a href="https://www.foxbusiness.com/features/7-tips-for-machine-learning-success">According to Ted Dunning</a>, it is better to use precise terminology, like Machine Learning or Deep Learning instead of the word AI, because before we get something to work well, we call it AI, afterwards, we always call it something else. AI is better used as a word for the next frontier.</p> <h2>How Has Analytics Changed in the Last 10 Years?</h2> <p>According to <a href="https://hbr.org/2017/06/how-analytics-has-changed-in-the-last-10-years-and-how-its-stayed-the-same">Thomas Davenport in the HBR</a>, analytical technology has changed dramatically over the last decade, with more powerful and less expensive distributed computing across commodity servers, streaming analytics, and improved machine learning technologies, enabling companies to store and analyze both far more data and many different types of it.</p> <p>Traditionally data was stored on a RAID system, sent to a multi-core server for processing and sent back for storage, which caused a bottleneck on data transfer, and was expensive. With file and table storage like MapR XD and MapR Database, data is distributed across a cluster, and Hadoop technologies like MapReduce, Pig, and Hive send the computing task to where the data resides.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/xd-1606272673595.png" alt="MapR XD"></p> <p>Technologies like Apache Spark speed up parallel processing of distributed data even more with iterative algorithms by caching data in memory across iterations and using lighter weight threads.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-1606272692913.png" alt="Apache Spark"></p> <p>MapR Event Store, a new distributed messaging system for streaming event data at scale, combined with Stream processing like Apache Spark streaming, or Apache Flink, speed up parallel processing of real time events with machine learning models.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/maprstreams-1606272706000.png" alt="MapR Event Store"></p> <p>Graphical Processing Units (GPUs) have sped up multi-core servers for parallel processing. A GPU has a massively parallel architecture consisting of thousands of smaller, more efficient cores designed for handling multiple tasks simultaneously. Whereas a CPU consists of a few cores optimized for sequential serial processing. <a href="https://www.kdnuggets.com/2017/06/deep-learning-demystifying-tensors.html">In terms of potential performance, the evolution from the Cray-1 to today’s clusters with lots of GPU’s is roughly a million times what was once the fastest computer on the planet at a tiny fraction of the cost</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/gpu-1606272719191.png" alt="GPU"></p> <p><em>Image reference: <a href="http://www.nvidia.com/object/what-is-gpu-computing.html">AI Computing</a></em></p> <h2>What is Machine Learning?</h2> <p>Machine learning uses algorithms to find patterns in data, and then uses a model that recognizes those patterns to make predictions on new data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/ml-1606272731259.png" alt="ML"></p> <p>In general, machine learning may be broken down into two types: supervised, unsupervised, and in between those two. Supervised learning algorithms use labeled data, unsupervised learning algorithms find patterns in unlabeled data. Semi-supervised learning uses a mixture of labeled and unlabeled data. Reinforcement learning trains algorithms to maximize rewards based on feedback.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/ml2-1606272742434.png" alt="ML"></p> <h2>Supervised Learning</h2> <p>Supervised algorithms use labeled data in which both the input and target outcome, or label, are provided to the algorithm.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/sl-1606272757444.png" alt="Supervised Learning"></p> <p>Supervised Learning is also called predictive modeling or predictive analytics, because you build a model that is capable of making predictions. Some examples of predictive modeling are classification and regression. Classification identifies which category an item belongs to (for example whether a transaction is fraud or not fraud), based on labeled examples of known items (for example transactions known to be fraud or not). Logistic regression predicts a probability, for example the probability of fraud. Linear regression predicts a numeric value, for example the amount of fraud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/fraud-1606272770701.png" alt="Fraud"></p> <p>Some examples of Classification include:</p> <ul> <li>credit card fraud detection (fraud, not fraud)</li> <li>credit card application (good credit, bad credit)</li> <li>email spam detection (spam, not spam)</li> <li>text sentiment analysis (happy, not happy)</li> <li>Predicting patient risk (high risk patient, low risk patient)</li> <li>classifying a tumor as malignant or not</li> </ul> <p>Some examples of logistic regression (or other algorithms) include:</p> <ul> <li>given historical car insurance fraudulent claims and features of the claims such as age of the claimant, claimed amount, severity of the accident, predict the probability of fraud.</li> <li>given patient characteristics predict the probability of congestive heart failure.</li> </ul> <p>Some examples of linear regression include:</p> <ul> <li>Given historical car insurance fraudulent claims and features of the claims such as age of the claimant, claimed amount, severity of the accident, predict the amount of fraud.</li> <li>Given historical real estate sales prices and features of houses(square feet, number of bedrooms, location..) predict a house’s price.</li> <li>Given historical neighborhood crime statistics, predict crime rate.</li> </ul> <p>There are other Supervised and Unsupervised learning Algorithms <a href="https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html">shown below</a>, which we won’t go over, but we will look at one example of each in more detail.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/algorithm-1606272784378.png" alt="Algorithms"></p> <p><em>Image reference: <a href="https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html">Supervised and Unsupervised learning Algorithms</a></em></p> <h2>Classification Example Debit Card Fraud</h2> <p>Classification takes a set of data with known labels and pre-determined features and learns how to label new records based on that information. Features are the “if questions” that you ask. The label is the answer to those questions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/duck-1606272802082.png" alt="Duck"></p> <p>Let’s go through an example of Debit Card Fraud:</p> <ul> <li>What are we trying to predict? <ul> <li>Whether a debit card transaction is fraud or not</li> <li>Fraud is the Label: True or False</li> </ul> </li> <li>What are the “if questions” or properties that you can use to make predictions? <ul> <li>Is the amount spent today > historical average?</li> <li>Are there transactions in multiple countries today?</li> <li>Are the number of transactions today > historical average?</li> <li>Are the number of new merchant types today high compared to the last 3 months?</li> <li>Are there multiple purchases today from merchants with a category code of risk?</li> <li>Is there unusual signing activity today compared to using historically using pin?</li> <li>Are there new state purchases compared to the last 3 months?</li> <li>Are there foreign purchases today compared to the last 3 months?</li> </ul> </li> </ul> <p>To build a classifier model, you extract the features of interest that most contribute to the classification.</p> <h2>Decision Trees</h2> <p>Decision trees create a model that predicts the class or label, based on several input features. Decision trees work by evaluating a question containing a feature at every node and selecting a branch to the next node, based on the answer. A possible decision tree for predicting debit card fraud is shown below. The feature questions are the nodes, and the answers “yes” or “no” are the branches in the tree to the child nodes. (Note that a real tree would have more nodes).</p> <ul> <li>Q1: Is the amount spent in 24 hours > average <ul> <li>Yes</li> <li>Q2: Are there multiple purchases today from risky merchants?</li> <li>Yes Fraud 90%</li> <li>Not Fraud 50%</li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/decisiontree-1606272815273.png" alt="Decision Tree"></p> <p>Decision trees are popular because they are easy to visualize and explain. The accuracy of models can be improved by combining algorithms with ensemble methods. An ensemble example is Random forest, which combines multiple random subsets of decision trees.</p> <h2>Unsupervised Learning</h2> <p>Unsupervised learning, also sometimes called descriptive analytics, does not have labeled data provided in advance. These algorithms discover similarities, or regularities in the input data. An example of unsupervised learning is grouping similar customers, based on purchase data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/ul-1606272827176.png" alt="Unsupervised Learning"></p> <h2>Clustering</h2> <p>In clustering, an algorithm classifies inputs into categories by analyzing similarities between input examples. Some clustering use cases include:</p> <ul> <li>search results grouping</li> <li>grouping similar customers</li> <li>grouping similar patients</li> <li>Text categorization</li> <li>Network Security Anomaly detection (finds what is not similar, the outliers from clusters)</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/cluster-1606272838221.png" alt="Clustering"></p> <p>The K-means algorithm groups observations into K clusters in which each observation belongs to the cluster with the nearest mean from its cluster center.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/k-means-1606272852036.png" alt="K-means"></p> <p>An example of clustering is a company that wants to segment its customers in order to better tailor products and offerings. Customers could be grouped on features such as demographics and purchase histories. Clustering with unsupervised learning is often combined with Supervised learning in order to get more valuable results. For example in this banking customer 360 use case, customers were first segmented based on answers to a survey. The customer groups were analyzed and labeled with customer personas. These labels were then linked by customer Id with features such as types of accounts and purchases. Finally supervised machine learning was applied and tested with the labeled customers, allowing to link the survey customer personas with their banking actions and provide insights.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/survey-1606272868644.png" alt="Surveys"></p> <h2>Deep Learning</h2> <p>Deep learning is the name for multilayered neural networks, which are networks composed of several “hidden layers” of nodes between the input and output. There are many variations of neural networks, which you can learn more about on this <a href="https://www.asimovinstitute.org/neural-network-zoo/">neural network cheat sheet</a>. Improved algorithms, GPUs and massively parallel processing (MPP), have given rise to networks with thousands of layers. Each node takes input data and a weight and outputs a confidence score to the nodes in the next layer, until the output layer is reached where the error of the score is calculated. With <a href="https://en.wikipedia.org/wiki/Backpropagation">backpropagation</a> inside of a process called <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a>, the errors are sent back through the network again and the weights are adjusted improving the model. This process is repeated thousands of times, adjusting a model’s weights in response to the error it produces, until the error can’t be reduced any more.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/process-1606272882881.png" alt="Process"></p> <p>During this process the layers learn the optimal features for the model, which has the advantage that features do not need to be predetermined. However this has the disadvantage that the model’s decisions are not explainable. Because explaining the decisions can be important, researchers are developing <a href="https://www.sciencemag.org/news/2017/07/how-ai-detectives-are-cracking-open-black-box-deep-learning">new ways to understand the black box of deep learning</a>.</p> <p>There are different variations of Deep Learning Algorithms, which can be used to build data-driven applications such as the following:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/qss-1606272896542.png" alt="Deep Learning QSS"></p> <p><em>Deep Neural Networks for Improved Traditional Algorithms</em></p> <ul> <li>Finance: Enhanced Fraud Detection through identification of more complex patterns</li> <li>Manufacturing: Enhanced identification of defects based on deeper anomaly detection</li> </ul> <p><em>Convolutional Neural Networks for images</em></p> <ul> <li>Retail: in-store activity analysis of video to measure traffic</li> <li>Satellite images: labeling terrain, classifying objects</li> <li>Automotive: recognition of roadways and obstacles</li> <li>Healthcare: diagnostic opportunities from x-rays, scans, etc.</li> <li>Insurance: estimating claim severity based on photographs</li> </ul> <p><em>Recurrent Neural Networks for sequenced data</em></p> <ul> <li>Customer satisfaction: transcription of voice data to text for NLP analysis</li> <li>Social media: real-time translation of social and product forum posts</li> <li>Photo captioning: search archives of images for new insights</li> <li>Finance: Predicting behavior based via time series analysis (also enhanced recommendation systems</li> </ul> <h2>Additional Resources and References</h2> <ul> <li><a href="https://www.foxbusiness.com/features/7-tips-for-machine-learning-success">7 tips for Machine Learning Success</a></li> <li><a href="https://www.forbes.com/sites/bernardmarr/2016/09/30/what-are-the-top-10-use-cases-for-machine-learning-and-ai/">The Top 10 AI And Machine Learning Use Cases Everyone Should Know About</a></li> <li><a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/">Visual Introduction to machine learning</a></li> <li><a href="https://francesco-ai.medium.com/a-brief-history-of-ai-baf0f362f5d6">A Brief History of AI</a></li> <li><a href="https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/how-artificial-intelligence-can-deliver-real-value-to-companies">How artificial intelligence can deliver real value to companies</a></li> </ul>https://developer.hpe.com/best-practices-for-migrating-your-apps-to-containers-and-kubernetes/https://developer.hpe.com/best-practices-for-migrating-your-apps-to-containers-and-kubernetes/Wed, 25 Nov 2020 02:26:04 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suzy Visvanathan", "publish": "2018-05-15T10:45:00.000", "tags": "mapr-platform" </code></pre> <hr> <p>I was having a discussion with a potential customer. As I listened, I realized that this was a company going through a massive digital transformation. They were not doing it in bits and pieces but rather through a massive overhaul. The discussion inevitably centered around containers, managing container-based workloads, and the steps they are taking to make this transformation happen.</p> <p>That led me to ask: “What should an organization know, consider, and do in order to migrate to containers?” This response may not be specific to a digital transformation but, in general, applies to the considerations one must take in order to adopt containers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/containers-wide-1606271200572.png" alt=""></p> <p>This blog assumes the reader has a fair knowledge of containers, Kubernetes, and some of their unique value adds.</p> <p>Many organizations made architecture, infrastructure, and solution decisions a long time ago and have followed the “if not broken, don’t fix it” model. Understandably so, since keeping up with technology trends came with a cost – investment in new skill sets, procurement of the latest, usually more expensive equipment, and, more importantly, disruption to business. For example, when VMware came out with the concept of virtualization, it took significant commitment from organizations to move from a physical to a virtual environment.</p> <p>A few years down the road, Amazon (and then Microsoft and Google) made public cloud and a subscription-based cost model very attractive for deploying on infrastructure managed by someone else. This required a significant change in mindset – to educate oneself in the benefits of infrastructure as a service and then to adopt cloud-native architecture in order to move to the cloud.</p> <p>Of all the technology trends introduced by innovators, Docker – with their containers technology – made it easy to be adopted, since it circumvented the complications of prior inventions by following the open source model. Open source brought containers more quickly to the enterprise and is propelling the maturity of containers, which is noticeably faster than the journey that VMs or cloud technology underwent. But with the proliferation of containers, managing them became a problem. This prompted several vendors to come up with solutions, with Kubernetes largely becoming the de facto container management platform.</p> <p>Cloud Native Computing Foundation (CNCF), which spearheads the development of open source projects like Kubernetes, has been busy releasing improvements at a rapid pace to keep up with the pace of adoption.</p> <h2>Move legacy applications to containers? Or not?</h2> <p>Containers allow applications to be broken into smaller manageable microservices. Each microservice is self-sufficient and can be changed and updated on its own without the need to touch the other services. For instance, if a revision or an update needs to be made, only the affected services need be changed and compiled instead of the entire application having to be recompiled. Kubernetes can be used to schedule and manage these individual services. In order to benefit from the full benefits of containers and Kubernetes, assess your legacy applications to see if they can be broken into modules. A very classic example is that of how Uber uses thousands of microservices to improve scalability and reliability.</p> <p>In his book <a href="https://www.academia.edu/41522528/A_Practical_Guide_to_Microservices_and_Containers_Mastering_the_Cloud_Data_and_Digital_Transformation">A Practical Guide to Microservices and Containers</a>, Jim Scott, Director of Enterprise Strategy at MapR, draws out the intricate details of scaling and microservices by citing a simple web application.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/microservices-1606271160940.jpg" alt=""></p> <p>Microservices sounds very fancy and not all legacy applications can be broken down into smaller modules. Following these simple steps may help greatly toward the journey:</p> <ol> <li> <p>Not adding anything more to legacy applications. Starting fresh is definitely a viable choice, especially if the decision has been made to fully convert to containers. Rewriting legacy applications into microservices architecture must also be considered.</p> </li> <li> <p>If legacy applications cannot be broken down into modules, the simplest thing to do may just be to enclose the application in a single container.</p> </li> <li> <p>Just splitting applications into containers doesn’t make it scalable automatically. Proper planning is needed to determine how these individual containers are to be run. Kubernetes creates containers in Pods and offers a DaemonSet, which is an automated way to create Pods of containers as more server nodes are added. Using such features to scale with microservices needs to be considered upfront.</p> </li> <li> <p>If certain applications are dependent on certain performance characteristics, pinning containers to certain hardware may be required. Kubernetes offers a feature, called stateful sets, which will allow containers to be locked to underlying infrastructure. If planned carefully, certain Pods carrying services can be spread across different performing servers to get the optimal environment.</p> </li> <li> <p>If applications cannot simply be broken down, it may be easier and cheaper to just write a new.</p> </li> </ol> <h2>Designing for containers</h2> <p>When designing new applications to run in containers, one key aspect to keep in mind is the concept of decoupling. Tightly coupling an application and its data together may be a non-starter. Keeping the characteristics of a container in mind, separating the data from the application’s dependencies from the start will speed up things greatly. Making use of cloud-native architecture and tools, along with a microservices approach, will make it easy to make the transition.</p> <h2>Decide the environment</h2> <p>Decide, before starting, whether these containers will be run in the data center or in the cloud. Running them in the cloud offers certain advantages when the decision to choose the right servers and everything that comes with maintaining an on-premises cluster is not needed. Expanding and shrinking in the cloud with containers reduces cost and speeds up deployment.</p> <p>However, there is a value-add in running containers in data centers. Containers and Kubernetes are still a concept that is being familiarized in many organizations. Deploying them in small phases, in a controlled environment, will greatly assist in understanding the benefits of containers before “shifting” everything to containers and cloud.</p> <p>One aspect every organization must understand is that digital transformation is not an end but a journey that never really ends. With careful planning and understanding of the environment, concept, and benefits, and by using tips as described in this blog, organizations can successfully embark on their digital transformation.</p> <p>MapR offers a robust data platform to deploy applications in containers and Kubernetes. The MapR Data Platform can be run in on-premises data centers and across clouds, enabling disparate environments to be considered.</p>https://developer.hpe.com/the-5-minute-guide-to-understanding-the-significance-of-apache-spark/https://developer.hpe.com/the-5-minute-guide-to-understanding-the-significance-of-apache-spark/Wed, 25 Nov 2020 02:19:48 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nitin Bandugula", "publish": "2015-06-23T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>In this blog, I’d like to talk about the differences between Apache Spark and MapReduce, why it’s easier to develop on Spark, and the top five use cases.</p> <h2>So what is Spark?</h2> <p>Spark is another execution framework. Like MapReduce, it works with the filesystem to distribute your data across the cluster, and process that data in parallel. Like MapReduce, it also takes a set of instructions from an application written by a developer. MapReduce was generally coded from Java; Spark supports not only Java, but also Python and Scala, which is a newer language that contains some attractive properties for manipulating data.</p> <h2>What are the key differences between Spark and MapReduce?</h2> <ol> <li> <p><strong>Spark tries to keep things in memory, whereas MapReduce keeps shuffling things in and out of disk.</strong> MapReduce inserts barriers, and it takes a long time to write things to disk and read them back. Hence MapReduce can be slow and laborious. The elimination of this restriction makes Spark orders of magnitude faster. For things like SQL engines such as Hive, a chain of MapReduce operations is usually needed, and this requires a lot of I/O activity. On to disk, off of disk—on to disk, off of disk. When similar operations are run on Spark, Spark can keep things in memory without I/O, so you can keep operating on the same data quickly. This results in dramatic improvements in performance, and that means Spark definitely moves us into at least the interactive category. For the record, there are some benefits to MapReduce doing all that recording to disk — as recording everything to disk allows for the possibility of restarting after failure. If you’re running a multi-hour job, you don’t want to begin again from scratch. For applications on Spark that run in the seconds or minutes, restart is obviously less of an issue.</p> </li> <li> <p><strong>It’s easier to develop for Spark</strong>. Spark is much more powerful and expressive in terms of how you give it instructions to crunch data. Spark has a Map and a Reduce function like MapReduce, but it adds others like Filter, Join and Group-by, so it’s easier to develop for Spark. In fact, Spark provides for lots of instructions that are a higher level of abstraction than what MapReduce provided. You can think more about how you want the data processed, rather than about how to cajole MapReduce into doing what you want. This might not seem that important, until you look at this: <a href="https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Example%3A+WordCount+v2.0">MapReduce-Wordcount</a>. This is the code to calculate a count of words in a text file, done in MapReduce (not Spark). It’s over 100 lines of code, and fairly unintuitive. The equivalent in Spark is found on this page: <a href="https://spark.apache.org/examples.html">Spark Examples</a> (look for the Word Count example). It’s four lines versus over 100. If you’re trying to do risk calculations on Wall Street, which one are you going to choose? Same thing goes for someone writing a new analytics application or a new query engine. It’s a no-brainer.</p> </li> </ol> <p>Related to this theme of ease of development, Spark is more intelligent about how it operates on data. Spark supports lazy evaluation. Normally we don’t like anything to be lazy, but in this case, lazy evaluation means that if you tell Spark to operate on a set of data, it listens to what you ask it to do, writes down some shorthand for it so it doesn’t forget, and then does absolutely nothing. It will continue to do nothing, until you ask it for the final answer.</p> <p>Why is this great? Because often work magically goes away. This is a bit like when you were in high school, and your mom came in to ask you to do a chore (“fetch me some milk for tonight’s dinner”). Your response: say that you were going to do it, then keep right on doing what you were already doing. Sometimes your mom would come back in and say she didn’t need the chore done after all (“I substituted water instead”). Magic, work saved! Sometimes the laziest finish first.</p> <p>Spark is the same. It waits until you’re done giving it operators, and only when you ask it to give you the final answer does it evaluate, and it always looks to limit how much work it has to do. Suppose you first ask Spark to filter a petabyte of data for something—say, find you all the point of sale records for the Chicago store—then next you ask for it to give you just the first result that comes back. This is a really common thing to do. Sometimes a data analyst just wants to see a typical record for the Chicago store. If Spark were to run things explicitly as you gave it instructions, it would load the entire file, then filter for all the Chicago records, then once it had all those, pick out just the first line for you. That’s a huge waste of time and resources. Spark will instead wait to see the full list of instructions, and understand the entire chain as a whole. If you only wanted the first line that matches the filter, then Spark will just find the first Chicago POS record, then it will emit that as the answer, and stop. It’s much easier than first filtering everything, then picking out only the first line.</p> <p>Now, you could write your MapReduce jobs more intelligently to similarly avoid over-using resource, but it’s much more difficult to do that. Spark makes this happen automatically for you. Normally, software like Hive goes into contortions to avoid running too many MapReduce jobs, and programmers write very complex and hard-to-read code to force as much as possible into each Map and Reduce job. This makes development hard, and makes the code hard to maintain over time. By using Spark instead, you can write code that describes how you want to process data, not how you want the execution to run, and then Spark “does the right thing” on your behalf to run it as efficiently as possible. This is the same thing a good high-level programming language does: it raises the abstraction layer, letting the developer talk more powerfully and expressively, and does the work behind the scenes to ensure it runs as fast as possible.</p> <ol> <li><strong>Spark also adds libraries for doing things like machine learning, streaming, graph programming and SQL</strong> (see image below). This also makes things much easier for developers. These libraries are integrated, so improvements in Spark over time provide benefits to the additional packages as well. Most data analysts would otherwise have to resort to using lots of other unrelated packages to get their work done, which makes things complex. Spark’s libraries are designed to all work together, on the same piece of data, which is more integrated and easier to use. Spark streaming in particular provides a way to do real-time stream processing. The aforementioned Apache Storm project was designed to do this kind of work, but Spark is much easier to develop for than Storm. Spark will enable developers to do real-time analysis of everything from trading data to web clicks, in an easy to develop environment, which tremendous speed.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-core-stack-db-1606270847095.jpg" alt=""></p> <p><strong>So to summarize, Spark is promising to speed up application development by 10-100x, make applications more portable and extensible, and make the actual application run 100x faster.</strong> This should make it clear why people are excited.</p> <h2>How does someone develop an application for Spark?</h2> <p>Spark is a set of libraries. You can program to those libraries from three programming languages today: Java, Python, and a newer language called Scala. There aren’t a lot of Scala developers today, while there are millions of Java and Python developers. But Scala is better designed to work with Spark, and it offers the greatest reduction in the number of lines of code to stand up an application. Many complex applications that were hundreds of lines can be re-written in Scala in less than a hundred lines. The design methodology of Scala is more congruent with that of Spark than any other language. Scala also compiles down to the same bytecode that a Java Virtual Machine (JVM) executes, so any existing code you’ve got in Java can be used by Scala. This is one of Scala’s biggest wins: it gives you the best of both worlds. Scala offers first-class support for integrating Java; in fact, much of Scala is actually directly reliant on Java. All the Java code you’ve already got working can be repurposed. And the really important code that represents the critical part of the data-crunching application can be re-written in Scala in a much smaller form factor that is much easier for developers to read, repurpose and maintain. Note that with Java 8.0 supporting lambda expressions, a Java developer can become lot more productive without switching to Scala completely.</p> <p>The libraries provided by Spark were discussed previously. The developer community is excited about Spark because everything is integrated with Spark. If you wanted to do applications with MapReduce, you had a bunch of problems. First, MapReduce pretty much had to be done with Java. That’s not the case with Spark: Python and Scala are first-class citizens. Second, you had to marry up MapReduce with other technologies. Wanted machine learning? You’ve got to separately integrate something like Mahout, H2O, or Oryx to get things done, and you’ve got to figure out how it works, and how to bolt it on. Wanted a graph database, with inbuilt tools for graph analytics? Well, again, you’ve got to select from Giraph, TitanDB, neo4j, or some other technology. The point is that the integration of all these parts would definitely not be seamless; each of them wants to be used in its own way.</p> <p>Spark offers a different model. You get SQL, machine learning, graph analytics and streaming in a single set of libraries that all work together with the Spark core. You can manipulate the same datasets with all of these. And when Spark core gets improvements, all of the libraries also improve. Integration is much easier, applications are far easier to maintain, costs go down, developers are happier. Most important for the field teams to understand: If a company developing applications is going to make a bet on a single foundation for your applications, Spark is looking like the best choice right now.</p> <p>Spark does not replace Hadoop. You still need a single data layer, preferably one that is hyper-scalable and extremely fast, and that’s where MapR comes in. MapR makes Spark faster, more scalable, and more reliable.</p> <h2>What are the Spark use cases?</h2> <p>Databricks (a company founded by the creators of Apache Spark) lists the following cases for Spark:</p> <ol> <li>Data integration and ETL</li> <li>Interactive analytics or business intelligence</li> <li>High performance batch computation</li> <li>Machine learning and advanced analytics</li> <li>Real-time stream processing</li> </ol> <p>Tons of people are doing data integration and ETL on MapReduce, as well as batch computation, machine learning and batch analytics. But these things are going to be much faster on Spark. Interactive analytics and BI are possible on Spark, and the same goes for real-time stream processing. So some of the new use cases are just the old use cases, done faster, while some are totally new. There are some things that just couldn’t have been done with acceptable performance on MapReduce.</p> <h2>Summary</h2> <p>In this blog post, you’ve learned about the key differences between Spark and MapReduce, why it’s easier to develop on Spark, and the top five use cases.</p>https://developer.hpe.com/event-driven-microservices-architecture-patterns-and-examples/https://developer.hpe.com/event-driven-microservices-architecture-patterns-and-examples/Thu, 19 Nov 2020 00:06:45 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2017-02-08T06:00:00.000Z", "tags": "nosql,microservices" </code></pre> <hr> <p>In this blog we will discuss some patterns that are often used in microservices applications that need to scale:</p> <ul> <li>Event Stream</li> <li>Event Sourcing</li> <li>Polyglot Persistence</li> <li>Memory Image</li> <li>Command Query Responsibility Separation</li> </ul> <h2>The Motivation</h2> <p><a target='\_blank' href='https://eng.uber.com/service-oriented-architecture/'>Uber</a>, <a target='\_blank' href='https://www.infoq.com/presentations/scale-gilt/'>Gilt</a> and others have moved from a monolithic to a microservices architecture because they needed to scale. A monolithic application puts all of its functionality into a single process, meaning that scaling requires replicating the whole application, which has limitations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture1-1605744773149.png" alt=""></p> <p>Sharing normalized tables in a clustered RDBMS does not scale well because distributed transactions and joins can cause concurrency bottlenecks.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture2-1605744781148.png" alt=""></p> <p>The <a target='\_blank' href='https://martinfowler.com/articles/microservices.html'>microservice architectural style</a> is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities.  <a target='\_blank' href='https://ostatic.com/blog/q-a-maprs-jack-norris-on-the-impact-of-microservices'>A microservices approach is well aligned to a typical big data deployment</a>.  You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many commodity hardware servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single points of failure, which can help prevent large-scale outages. </p> <h2>Event Stream</h2> <p>When moving from a monolithic to a microservices architecture  a common architecture pattern is event sourcing  using an append only event stream such as Kafka or MapR Event Store (which provides a Kafka 0.9 API). With MapR Event Store (or Kafka) events are grouped into logical collections of events called Topics. Topics are partitioned for parallel processing. You can think of a partitioned Topic like a queue, events are delivered in the order they are received.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture3-1605744789588.png" alt=""></p> <p>Unlike a queue, events are persisted, even after they are delivered they remain on the partition, available to other consumers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture4-1605744796164.png" alt=""></p> <p>Older messages are automatically deleted based on the Stream’s time-to-live setting. If the setting is 0 then they will never be deleted.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture5-1605744806846.png" alt=""></p> <p>Messages are not deleted from Topics when read, and topics can have multiple different consumers. This allows processing of the same messages by different consumers for different purposes. Pipelining is also possible where a consumer enriches an event and publishes it to another topic.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture6-1605744813964.png" alt=""></p> <h2>Event Sourcing</h2> <p><a target='\_blank' href='https://martinfowler.com/eaaDev/EventSourcing.html'>Event Sourcing</a> is an architectural pattern in which the state of the application is determined by a sequence of events each of which is recorded in an append-only Event store or Stream. As an example, imagine that each “event” is an incremental update to an entry in a database. In this case, the state of a particular entry is simply the accumulation of events pertaining to that entry. In the example below the Stream persists the queue of all deposit and withdrawal events, and the database table persists the current account balances.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture7-1605744821094.png" alt=""></p> <p>Which one of these, the Stream or the Database, makes a better system of record? The events in the Stream can be used to reconstruct the current account balances in the Database, but not the other way around. Database replication actually works by suppliers writing changes to a change log, and consumers applying the changes locally. Another well known example of this is a source code version control system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture8-1605744828551.png" alt=""></p> <p>With a Stream, events can be re-played to create a new view, index, cache, <a target='\_blank' href='https://martinfowler.com/bliki/MemoryImage.html'>memory image</a>, or materialized view of the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture9-1605744835216.png" alt=""></p> <p>The Consumer simply reads the messages from the oldest to the latest to create a new View of the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture10-1605744842963.png" alt=""></p> <p>There are several advantages for modeling application state with streams:</p> <ul> <li><strong>Lineage</strong>: to ask how did BradA’s balance get so low?</li> <li><strong>Auditing</strong>:  it gives an audit trail, who deposited/withdrew from account id BradA? This is how accounting transactions work.</li> <li><strong>Rewind</strong>: to see what the status of the accounts were last year.</li> <li><strong>Integrity</strong>: can I trust the data hasn’t been tampered with? <ul> <li>yes because Streams are immutable.</li> </ul> </li> </ul> <p><a href="https://docs.datafabric.hpe.com/62/MapR_Streams/replicating_streams.html">The Replication of MapR Event Store</a> gives a powerful testing and debugging technique. A replica of a Stream can be used to replay a version of events for testing or debugging purposes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture11-1605744849895.png" alt=""></p> <h2>Different databases and schemas for different needs</h2> <p>There are lots of databases out there. Each <a target='\_blank' href='https://martinfowler.com/bliki/PolyglotPersistence.html'>use different technologies depending on how the data is used</a>, optimized for a type of write or read pattern:  graph query, search, document. What if you need to have the same set of data for different databases, for different types of queries coming in? The Stream can act as the distribution point for multiple databases, each one providing a different read pattern.  All changes to the application state are persisted to an event store, which is the system of record. The event store provides a rebuilding state by re-running the events in the stream.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture12-1605744856503.png" alt=""></p> <p>Events funnel out to databases, which are consumers of the stream. Polyglot persistence provides different specialized materialized views.</p> <h2>CQRS</h2> <p><a target='\_blank' href='https://martinfowler.com/bliki/CQRS.html'>Command and Query Responsibility Segregation (CQRS)</a> is a pattern that separates the read model and Queries from the write model and Commands often using event sourcing.  Let’s look at how an online shopping  application’s item rating functionality could be separated using the CQRS pattern.  The functionality, shown below in a monolithic application, consists of users rating items they have bought, and browsing item ratings while shopping.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture13-1605744863999.png" alt=""></p> <p>In the CQRS design shown below we isolate and separate the Rate Item write “command” from the Get Item Ratings read “query” using event sourcing.  Rate Item events are published to a Stream. A handler process reads from the stream and persists a materialized view of the ratings for an item in a NoSQL document-style database.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture14-1605744871613.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture15-1605744879088.png" alt=""></p> <h2>NoSQL and De-normalization</h2> <p>With MapR Database, a table is automatically partitioned across a cluster by key range, and each server is the source for a subset of a table. Grouping the data by key range provides for really fast read and writes by row key. With MapR Database you design your schema so that the data that is read together is stored together.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture16-1605744887363.png" alt=""></p> <p>Often with MapR Database, you de-normalize or store in one table what would be multiple tables in a normalized relational database. If your entities exist in a one-to-many relationship, it’s possible to model it in MapR Database HBase as a single row or MapR Database JSON as a single document. In the example below, the item and related ratings are stored together and can be read together with a single get on the indexed row key. This makes the reads a lot faster than joining tables together.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/17b-1605744895310.png" alt=""></p> <h2>Event Sourcing: New Uses of Data</h2> <p>An advantage of using an Event Stream for the rate item and other shopping related events is shown here. This design lets us use this data more broadly. Raw or enriched events can be stored in inexpensive storage such as MapR XD. Historical ratings data can be used to build a machine learning model for recommendations. Having a long retention time for data in the queue is also very useful. For example, that data could be processed to build a collection of shopping transaction histories stored in a data format such as Parquet that allows very efficient querying. Other processes might use historical data and streaming shopping related events with machine learning to <a target='\_blank' href='https://www.forbes.com/sites/bernardmarr/2015/11/10/big-data-a-game-changer-in-the-retail-sector/?sh=5884f9339f37'>predict shopping trends</a>, to detect fraud, or to build a real-time display of where transactions are happening.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture18-1605744754814.png" alt=""></p> <h2>Fashion Retailer’s Event Driven Architecture</h2> <p>A major fashion retailer wanted to increase in-season agility and inventory discipline in order to react to demand changes and reduce markdowns. The Event driven solution architecture is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture19-1605744904534.png" alt=""></p> <ul> <li>Weather, world events, and logistical data is collected in real time via MapR Event Store, allowing for real time analysis of potential logistical impacts, and rerouting of inventory.</li> <li>Apache Spark is used for batch and streaming analytics processing, and machine learning for predicting supply chain disruptions, and product recommendations.</li> <li>Data is stored in MapR Database providing scalable, fast reads and writes. Apache Drill is used for interactive exploration and preprocessing of the data with a schema-free SQL query engine.</li> <li>ODBC with Drill provides support for existing BI tools.</li> <li>MapR’s Enterprise capabilities provide for global data center replication.</li> </ul> <h2>Summary</h2> <p>In this blog post, we discussed event driven microservice architecture using the following design patterns: <a target='\_blank' href='https://martinfowler.com/eaaDev/EventSourcing.html'>Event Sourcing,</a> <a target='\_blank' href='https://martinfowler.com/bliki/CQRS.html'>Command Query Responsibility Separation</a>, and <a target='\_blank' href='https://martinfowler.com/bliki/PolyglotPersistence.html'>Polyglot Persistence</a>. All of the components of the architectures we discussed can run on the same cluster with the MapR Data Platform. </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture20-1605744911661.png" alt=""></p> <h2>References and More Information</h2> <ul> <li><a target='\_blank' href='https://www.eweek.com/cloud/10-advantages-to-building-enterprise-applications-with-microservices'>10 Advantages to Building Enterprise Applications with Microservices</a></li> <li><a target='\_blank' href='https://ostatic.com/blog/q-a-maprs-jack-norris-on-the-impact-of-microservices'>MapR's Jack Norris on the Impact of Microservices</a></li> <li><a target='\_blank' href='https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html'>Turning the database upside down</a></li> <li><a target='\_blank' href='http://milinda.pathirage.org/kappa-architecture.com/'>Kappa Architecture</a></li> <li><a target='\_blank' href='https://www.oreilly.com/library/view/making-sense-of/9781492042563/'>Making Sense of Stream Processing</a></li> <li><a target='\_blank' href='https://www.infoq.com/presentations/uber-stream-processing/'>Stream processing in Uber</a></li> <li><a target='\_blank' href='http://highscalability.com/blog/2015/1/26/paper-immutability-changes-everything-by-pat-helland.html'>Immutability Changes Everything</a></li> <li><a target='\_blank' href='https://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/'>NoSQL Data Modeling Techniques</a></li> </ul>https://developer.hpe.com/how-spark-runs-your-applications/https://developer.hpe.com/how-spark-runs-your-applications/Wed, 18 Nov 2020 23:48:48 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-10-31T07:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p>Recall that your Spark application runs as a set of parallel tasks. In this blog post, we will go over how Spark translates Dataset transformations and actions into an execution model.  </p> <p>With Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster.</p> <p>In order to understand how your application runs on a cluster, an important thing to know about Dataset transformations is that they fall into two types, narrow and wide, which we will discuss first, before explaining the execution model.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image18-1605743734060.png" alt=""></p> <h2>Narrow and Wide Transformations  </h2> <p>As a review, transformations create a new Dataset from an existing one.  Narrow transformations do not have to move data between partitions when creating a new dataset from an existing one. Some example narrow transformations are <code>filter</code> and <code>select</code>, which are used in the example below to retrieve flight information for the carrier "AA":</p> <pre><code class="language-scala">// select and filter are narrow transformations df.select($"carrier",$"origin",  $"dest", $"depdelay", $"crsdephour").filter($"carrier" === "AA" ).show(2) result: +-------+------+----+--------+----------+ |carrier|origin|dest|depdelay|crsdephour| +-------+------+----+--------+----------+ |     AA|   ATL| LGA|     0.0|        17| |     AA|   LGA| ATL|     0.0|        13| +-------+------+----+--------+----------+ </code></pre> <p>Multiple narrow transformations can be performed on a Dataset in memory, in a process called pipelining, making narrow transformations very efficient.</p> <p>Wide transformations cause data to be moved between partitions when creating a new Dataset, in a process called the shuffle.  With wide transformation shuffles, data is sent across the network to other nodes and written to disk, causing network and disk I/O, and making the shuffle a costly operation. Some example wide transformations are <code>groupBy</code>, <code>agg</code>, <code>sortBy</code>, and <code>orderBy</code>. Below is a wide transformation to count the number of flights by carrier.</p> <pre><code class="language-scala">df.groupBy("carrier").count.show result: +-------+-----+ |carrier|count| +-------+-----+ |     UA|18873| |     AA|10031| |     DL|10055| |     WN| 2389| +-------+-----+ </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image12-1605743742466.png" alt=""></p> <h2>The Spark Execution Model</h2> <p>The Spark execution model can be defined in three phases: creating the logical plan, translating that into a physical plan, and then executing the tasks on a cluster.</p> <p>You can view useful information about your Spark jobs in real time in a web browser with this URL: http://<driver-node>:4040. For Spark applications that have finished, you can use the Spark history server to see this information in a web browser at this URL: http://<server-url>:18080.  Let’s walk through the three phases and the Spark UI information about the phases, with some example code.</p> <h2>The Logical Plan</h2> <p>In the first phase, the logical plan is created. This is the plan that shows which steps will be executed when an action gets applied. Recall that when you apply a transformation on a Dataset, a new Dataset is created. When this happens, that new Dataset points back to the parent, resulting in a lineage or directed acyclic graph (DAG) for how Spark will execute these transformations.  </p> <h2>The Physical Plan</h2> <p>Actions trigger the translation of the logical DAG into a physical execution plan. The Spark Catalyst query optimizer creates the physical execution plan for DataFrames, as shown in the diagram below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image9-1605743750828.png" alt=""></p> <p><em>(Image reference: Databricks)</em></p> <p>The physical plan identifies resources, such as memory partitions and compute tasks, that will execute the plan.</p> <h2>Viewing the Logical and Physical Plan</h2> <p>You can see the logical and physical plan for a Dataset by calling the <code>explain(true)</code> method. In the code below, we see that the DAG for df2 consists of a <code>FileScan</code>, a <code>Filter</code> on <code>depdelay</code>, and a <code>Project</code> (selecting columns).</p> <pre><code class="language-scala">import org.apache.spark.sql.types._ import org.apache.spark.sql._ import org.apache.spark.sql.functions._ var file = "maprfs:///data/flights20170102.json" case class Flight(_id: String, dofW: Long, carrier: String, origin: String, dest: String, crsdephour: Long, crsdeptime: Double, depdelay: Double,crsarrtime: Double, arrdelay: Double, crselapsedtime: Double, dist: Double) extends Serializable val df = spark.read.format("json").option("inferSchema", "true").load(file).as[Flight] val df2 = df.filter($"depdelay" > 40) df2.take(1) result: Array[Flight] = Array(Flight(MIA_IAH_2017-01-01_AA_2315, 7,AA,MIA,IAH,20,2045.0,80.0,2238.0,63.0,173.0,964.0)) df2.explain(true) result: == Parsed Logical Plan == 'Filter ('depdelay > 40) +- Relation[_id#8,arrdelay#9,…] json == Analyzed Logical Plan == _id: string, arrdelay: double… Filter (depdelay#15 > cast(40 as double)) +- Relation[_id#8,arrdelay#9…] json == Optimized Logical Plan == Filter (isnotnull(depdelay#15) &#x26;&#x26; (depdelay#15 > 40.0)) +- Relation[_id#8,arrdelay#9,…] json == Physical Plan == *Project [_id#8, arrdelay#9,…] +- *Filter (isnotnull(depdelay#15) &#x26;&#x26; (depdelay#15 > 40.0))    +- *FileScan json [_id#8,arrdelay#9,…] Batched: false, Format: JSON, Location: InMemoryFileIndex[maprfs:///..], </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image11-1605743758827.png" alt=""></p> <p>You can see more details about the plan produced by Catalyst on the web UI SQL tab (http://<driver-node>:4040/SQL/).  Clicking on the query description link displays the DAG and details for the query.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image19-1605743768630.png" alt=""></p> <p>In the code below, after the <code>explain</code>, we see that the physical plan for df3 consists of a <code>FileScan</code>, <code>Filter</code>, <code>Project</code>, <code>HashAggregate</code>, <code>Exchange</code>, and <code>HashAggregate</code>. The <strong>Exchange</strong> is the shuffle caused by the <code>groupBy</code> transformation. Spark performs a hash aggregation for each partition before shuffling the data in the Exchange. After the exchange, there is a hash aggregation of the previous sub-aggregations. Note that we would have an in-memory scan instead of a file scan in this DAG, if df2 were cached.  </p> <pre><code class="language-scala">val df3 = df2.groupBy("carrier").count df3.collect result: Array[Row] = Array([UA,2420], [AA,757], [DL,1043], [WN,244]) df3.explain result: == Physical Plan == *HashAggregate(keys=[carrier#124], functions=[count(1)]) +- Exchange hashpartitioning(carrier#124, 200) +- *HashAggregate(keys=[carrier#124], functions=[partial_count(1)])       +- *Project [carrier#124]          +- *Filter (isnotnull(depdelay#129) &#x26;&#x26; (depdelay#129 > 40.0)) +- *FileScan json [carrier#124,depdelay#129] </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image6-1605743777932.png" alt=""></p> <p>Clicking on the SQL tab link for this query displays the DAG below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image5-1605743786007.png" alt=""></p> <h2>Executing the Tasks on a Cluster</h2> <p>In the third phase, the tasks are scheduled and executed on the cluster. The scheduler splits the graph into stages, based on the transformations. The narrow transformations (transformations without data movement) will be grouped (pipe-lined) together into a single stage. The physical plan for this example has two stages, with everything before the exchange in the first stage.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image16-1605743813015.png" alt=""></p> <p>Each stage is comprised of tasks, based on partitions of the Dataset, which will perform the same computation in parallel.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image17-1605743851321.png" alt=""></p> <p>The scheduler submits the stage task set to the task scheduler, which launches tasks via a cluster manager. These phases are executed in order, and the action is considered complete when the final phase in a job completes. This sequence can occur many times when new Datasets are created.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image4-1605743864900.png" alt=""></p> <p>Here is a summary of the components of execution:</p> <ul> <li><strong>Task:</strong> a unit of execution that runs on a single machine</li> <li><strong>Stage:</strong> a group of tasks, based on partitions of the input data, which will perform the same computation in parallel</li> <li><strong>Job:</strong> has one or more stages</li> <li><strong>Pipelining:</strong> collapsing of Datasets into a single stage, when Dataset transformations can be computed without data movement</li> <li><strong>DAG:</strong> Logical graph of Dataset operations</li> </ul> <h2>Exploring the Task Execution on the Web UI</h2> <p>Here is a screenshot of the web UI Jobs tab, after running the code above. The Jobs page gives you detailed execution information for active and recently completed Spark jobs. It gives you the performance of a job and also the progress of running jobs, stages, and tasks. In this example, Job Id 2 is the job that was triggered by the <code>collect</code> action on df3.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image2-1605743873603.png" alt=""></p> <p>Clicking the link in the Description column on the Jobs page takes you to the Job Details page. This page gives you details on the progress of the job, stages, and tasks.  We see this job consists of 2 stages, with 2 tasks in the stage before the shuffle and 200 in the stage after the shuffle.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image15-1605743880620.png" alt=""></p> <p>The number of tasks correspond to the partitions: after reading the file in the first stage, there are 2 partitions; after a <code>shuffle</code>, the default number of partitions is 200. You can see the number of partitions on a Dataset with the <code>rdd.partitions.size</code> method shown below.</p> <pre><code class="language-scala">df3.rdd.partitions.size result: Int = 200 df2.rdd.partitions.size result: Int = 2 </code></pre> <p>Under the Stages tab, you can see the details for a stage by clicking on its link in the description column.  </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image8-1605743887018.png" alt=""></p> <p>Here we have summary metrics and aggregated metrics for tasks and aggregated metrics by executor.  You can use these metrics to identify problems with an executor or task distribution. If your task process time is not balanced, then resources could be wasted.</p> <p>The Storage tab provides information about persisted Datasets. The Dataset is persisted if you called Persist or Cache on the Dataset, followed by an action to compute on that Dataset. This page tells you which fraction of the Dataset’s underlying RDD is cached and the quantity of data cached in various storage media. Look at this page to see if important Datasets are fitting into memory. You can also click on the link to view more details about the persisted Dataset. If you no longer need a cached Dataset, you can call <code>Unpersist</code> to uncache it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image7-1605743893255.png" alt=""></p> <p>Try caching df2, performing an action, then seeing how this gets persisted on the storage tab and how it changes the plan and execution time for df3 on the job details page. Notice how the execution time is faster after caching.</p> <pre><code class="language-scala">df2.cache df2.count df3.collect </code></pre> <p>Notice how the first stage is skipped in job4, when df2 is cached and df3 collect is executed again.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image14-1605743900253.png" alt=""></p> <p>The Environment tab lists all the active properties of your Spark applicationenvironment. Use this page when you want to see which configuration flags are enabled.  Only values specified through spark-defaults.conf, SparkSession, or the command line will be displayed here. For all other configuration properties, the default value is used.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image3-1605743907003.png" alt=""></p> <p>Under the Executors tab, you can see processing and storage for each executor:</p> <ul> <li><strong>Shuffle Read Write Columns:</strong> shows size of data transferred between stages</li> <li><strong>Storage Memory Column:</strong> shows the current used/available memory</li> <li><strong>Task Time Column:</strong> shows task time/garbage collection time</li> </ul> <p>Use this page to confirm that your application has the amount of resources you were expecting.  You can look at the thread call stack by clicking on the thread dump link.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image20-1605743914411.png" alt=""></p> <h2>Summary</h2> <p>In this post, we discussed the Spark execution model, and we explored task execution on the Spark Web UI. This understanding of how Spark runs your applications is important when debugging, analyzing, and tuning the performance of your applications.</p>https://developer.hpe.com/real-time-credit-card-fraud-detection-with-apache-spark-and-event-stream/https://developer.hpe.com/real-time-credit-card-fraud-detection-with-apache-spark-and-event-stream/Wed, 18 Nov 2020 23:37:40 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2016-05-03T07:00:00.000Z", "tags": "apache-spark, " </code></pre> <hr> <p>In this post we are going to discuss building a real time solution for credit card fraud detection.</p> <p>There are 2 phases to Real Time Fraud detection:</p> <ul> <li>The first phase involves analysis and forensics on historical data to build the machine learning model.  </li> <li>The second phase uses the model in production to make predictions  on live events. </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-1-1605742738606.jpg" alt=""></p> <h2>Building the Model</h2> <p><strong>Classification</strong></p> <p>Classification is a family of supervised machine learning algorithms that identify which category an item belongs to (for example whether a transaction is fraud  or not fraud), based on labeled examples of known items (for example transactions known to be fraud or not). Classification takes a set of data with known labels and pre-determined features and learns how to label new records based on that information.  Features are the “if questions” that you ask. The label is the answer to those questions. In the example below, if it walks, swims, and quacks like a duck, then the label is "duck".</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-2-1605742750342.jpg" alt="recommendation engine with ducks"></p> <p>Let’s go through an example of car insurance fraud:</p> <ul> <li>What are we trying to predict?    <ul> <li>This is the Label: The Amount of Fraud</li> </ul> </li> <li>What are the  “if questions” or properties that you can use to predict ? <ul> <li>These are the Features, to build a classifier model, you extract the features of interest that most contribute to the classification.</li> <li>In this simple example we will use the the claimed amount.</li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-3-1605742767766.jpg" alt="fraud detection with spark"></p> <p>Linear regression models the relationship between the Y “Label” and the X “Feature”,  in this case the relationship between the amount of fraud and the claimed amount.  The coefficient measures the impact of the feature, the claimed amount, on the label, the fraud amount.  </p> <p>Multiple linear regression models the relationship between two or more “Features” and a response “Label”.  For example  if we wanted to model the relationship between the amount of fraud and the the age of the claimant, the claimed amount, and the severity of the accident, the multiple linear regression function would look like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-4-1605742785380.jpg" alt="fraud detection algorithm "></p> <p><strong>AmntFraud = intercept+ coeff1 * age + coeff2 * claimedAmnt + coeff3 * severity + error.</strong>   </p> <p>The coefficients measure the impact on the fraud amount  of each of the features.</p> <p>Let’s take credit card fraud as another example:</p> <ul> <li>Example Features: transaction amount, type of merchant, distance from and time since last transaction .</li> <li>Example Label:  Probability of Fraud</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-5-1605742798937.jpg" alt="fraud detection with spark"></p> <p>Logistic regression measures the relationship between the Y “Label”  and the X “Features” by estimating probabilities using a <a target='\_blank' href='https://en.wikipedia.org/wiki/Logistic_function'>logistic function</a>. The model  predicts a probability  which is used to predict the label class.</p> <ul> <li>Classification: identifies which category (eg fraud or not fraud)</li> <li>Linear Regression: predicts a value (eg amount of fraud)</li> <li>Logistic Regression: predicts a probability (eg probability of fraud)</li> </ul> <p>Linear and Logistic Regression are just a couple of algorithms used in machine learning, there are many more as shown <a target='\_blank' href='http://scikit-learn.org/stable/tutorial/machine_learning_map/'>in this cheat sheet</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-6-1605742814936.png" alt="fraud detection with spark"></p> <h2>Feature Engineering</h2> <p>Feature engineering is the process of transforming raw data into inputs for a machine learning algorithm. Feature engineering is extremely dependent on the type of use case and potential data sources.  </p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-7-1605742828673.jpg" alt=""></p> <p>(reference <a target='\_blank' href='http://shop.oreilly.com/product/0636920028512.do'>Learning Spark</a>)</p> <p>Looking more in depth at the credit card fraud example for feature engineering,  our goal is to distinguish normal card usage from fraudulent card usage.</p> <ul> <li>Goal: we are looking for someone using the card other than the cardholder</li> <li>Strategy: we want to design features to measure the differences between recent and historical activities.</li> </ul> <p>For a credit card transaction we have features associated with the transaction, features associated with the card holder, and features derived from transaction history. Some examples of each are shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-8-1605742836968.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9-1605742844958.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9a-1605742851836.png" alt=""></p> <h2>Model Building Workflow</h2> <p>A typical supervised machine learning workflow has the following steps:</p> <ul> <li>Feature engineering to transform historical data into feature and label inputs for a machine learning algorithm.</li> <li>Split the data into two parts, one for building the model and one for testing the model.</li> <li>Build the model with the training features and labels</li> <li>Test the model with the test features to get predictions. Compare the test predictions to the test labels.</li> <li>Loop until satisfied with the model accuracy: <ul> <li>Adjust the model fitting parameters, and repeat tests.</li> <li>Adjust the features and/or machine learning algorithm and repeat tests.</li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9b-1605742860922.jpg" alt=""></p> <h2>Read Time Fraud Detection Solution in Production</h2> <p>The figure below shows the high level architecture of a real time fraud detection solution, which is capable of high performance at scale.  Credit card transaction events are delivered through the MapR Event Store messaging system, which supports the Kafka .09 API. The events are processed and checked for Fraud by Spark Streaming using Spark Machine Learning with the deployed model.  MapR XD, which supports the posix NFS API  and  HDFS API, is used for storing event data. MapR Database a NoSql database which supports the HBase API, is used for storing and providing fast access to credit card holder profile data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9c-1605742870860.jpg" alt=""></p> <h2>Streaming Data Ingestion</h2> <p>MapR Event Store is a new distributed messaging system which enables producers and consumers to exchange events in real time via the Apache Kafka 0.9 API. MapR Event Store topics are logical collections of messages which organize events into categories. In this solution there are 3 categories:</p> <ul> <li>Raw Trans: raw credit card transaction events.</li> <li>Enriched: credit card transaction events enriched with card holder features,  which were predicted to be not fraud.</li> <li>Fraud Alert: credit card transaction events enriched with card holder features which were predicted to be fraud.</li> </ul> <p>Topics are partitioned, spreading the load for parallel messaging across multiple servers,  which provides for faster throughput and scalability.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9d-1605742879420.jpg" alt=""></p> <h2>Real-time Fraud Prediction Using Spark Streaming</h2> <p>Spark Streaming lets you use the same Spark APIs for streaming and batch processing, meaning that well modularized Spark functions written for the offline machine learning can be re-used for the real time machine learning.</p> <p>The data flow for the real time fraud detection using Spark Streaming is as follows:</p> <ol> <li>Raw events come into Spark Streaming as DStreams,  which internally is a sequence of RDDs.  RDDs are like a Java Collection, except that the data elements contained in RDDs are partitioned across a cluster. RDD operations are performed in parallel on the data cached in memory, making the iterative algorithms often used in machine learning much faster for processing lots of data.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9e-wide-1605857312275.png" alt=""></p> <ol start="2"> <li>The credit card transaction data is parsed to get the features associated with the transaction.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9f-1605742904521.jpg" alt=""></p> <ol start="3"> <li>Card holder features and profile history are read from MapR Database using the account number as the row key.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9g-1605742912597.png" alt=""></p> <ol start="4"> <li>Some derived features are re-calculated with the latest transaction data.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9h-1605742920965.png" alt=""></p> <ol start="5"> <li> <p>Features are run with the model algorithm to produce fraud prediction scores.</p> </li> <li> <p>Non fraud events enriched with derived features are published to the enriched topic. Fraud events with derived features are published to the fraud topic.</p> </li> </ol> <h2>Storage of Credit Card Events</h2> <p>Messages are not deleted from Topics when read, and topics can have multiple different consumers, this allows processing of the same messages by different consumers for different purposes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9i-1605742927931.png" alt=""></p> <p>In this solution, MapR Event Store consumers read and store all raw events, enriched events, and alarms to MapR XD for future analysis, model training and updating. MapR Event Store consumers read enriched events and Alerts to update the Card holder features in MapR Database. Alerts events are also used to update Dashboards in real time.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9j-1605742935016.jpg" alt=""></p> <h2>Rapid Reads and Writes with MapR Database</h2> <p>With MapR Database (HBase API), a table is automatically partitioned across a cluster by key range, and each server is the source for a subset of a table. Grouping the data by key range provides for really fast read and writes by row key.</p> <p>Also with MapR Database each partitioned subset or region of a table has a write and read cache. Recently read or written data and cached column families are available in memory; all of this provides for really fast read and writes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9k-1605742944012.jpg" alt="fraud detection with Hadoop, NoSQL, streaming"></p> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform. There are several advantages of having MapR Event Store on the same cluster as all the other components. For example, maintaining only one cluster means less infrastructure to provision, manage, and monitor. Likewise, having producers and consumers on the same cluster means fewer delays related to copying and moving data between clusters, and between applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/spark-fraud-9l-1605742956048.jpg" alt="fraud detection software stack"></p> <h2>Summary</h2> <p>In this blog post, you learned how the MapR Data Platform integrates Hadoop and Spark with real-time database capabilities, global event streaming, and scalable enterprise storage.</p> <p>References and More Information:</p> <ul> <li><a href="https://learn.ezmeral.software.hpe.com/">Free Online training</a></li> <li><a target='\_blank' href='http://spark.apache.org/docs/latest/streaming-programming-guide.html'>Apache Spark Streaming Programming Guide</a></li> <li>Fraud Analytics Using Descriptive, Predictive, and Social Network Techniques: A Guide to Data Science for Fraud Detection Book, by Wouter Verbeke; Veronique Van Vlasselaer; Bart Baesens</li> <li>Learning Spark Book, By Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia</li> </ul>https://developer.hpe.com/artificial-intelligence-and-machine-learning-what-are-they-and-why-are-t/https://developer.hpe.com/artificial-intelligence-and-machine-learning-what-are-they-and-why-are-t/Thu, 12 Nov 2020 08:03:23 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information</h2> <pre><code>"authorDisplayName": "Saira Kennedy", "publish": "2018-09-21T07:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p>Editor's Note: This post is based on the MapR Academy course, <a href="https://learn.ezmeral.software.hpe.com/bus-introduction-to-artificial-intelligence-and-machine-learning"><em>Introduction to Artificial Intelligence and Machine Learning</em></a>.</p> <div style="padding:75% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/279683304?title=0&byline=0&portrait=0" style="position:absolute;top:0;left:0;width:100%;height:100%;" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe></div> <p>This post will give you a basic background on artificial intelligence and machine learning. It's a good place to gain an intro-level, working understanding of the categories, how they fit together, and their differences.</p> <p>In this post, we begin by defining the differences between artificial intelligence and machine learning and what these terms mean. In future posts, we will discuss the different methods of machine learning and some of the most common algorithms available for your projects. After that, you'll learn about real-world use cases, utilizing this technology, and the unique value MapR solutions provide for machine learning endeavors.</p> <h2>Where do AI and ML fit in data science?</h2> <p>Before we get into the finer details of artificial intelligence and machine learning, let's see how it fits in the larger world of data science. Because this field is rapidly changing, some people may be confused or disagree about the overall landscape and terms being used in the industry, so let's clarify how we will be defining them in this blog.</p> <p>Think of a series of Russian nesting dolls, but now imagine them as futuristic robot dolls, like the ones pictured below. With this analogy, we can have multiple robot dolls nested within each level.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image6-1605168354596.jpg" alt=""></p> <p>With these nesting robot dolls, version 2.0, the largest doll represents the entire field of data science. The second doll represents artificial intelligence, and the next doll represents machine learning. A fourth doll, for deep learning, can also be nested within the machine learning doll, but we won't be going into much depth on that topic in this post.</p> <h2>What is Data Science?</h2> <p>Both artificial intelligence and machine learning nest under the largest doll of data science, whose purpose is to extract insights from data. Data science analyzes large amounts of data to deliver value and give businesses a competitive edge across all industries. As an example, retail businesses analyze buyer habits to better target recommendations and promotions to their customers.</p> <p>In the growing world of big data, it is important to have an effective data science strategy to help make informed business decisions. All these fields have become more prominent, as the attempts to meet the growing demand for finding more efficient ways to extract value from data at scale has increased.</p> <p>Using artificial intelligence to accomplish these goals is a natural outgrowth of the big data movement.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image13-1605168362957.jpg" alt=""></p> <h2>What is Artificial Intelligence?</h2> <p>Artificial intelligence describes a machine that is capable of imitating and performing intelligent human behavior. Some of these tasks could include problem-solving and decision-making or specific activities requiring acute perception, recognition, or translation abilities.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image11-1605168371076.jpg" alt=""></p> <p>Remember from the robot nesting doll example that machine learning is nested within artificial intelligence; therefore, all machine learning counts as AI, but not all AI counts as machine learning. Other robot dolls within classic AI, without machine learning, would include expert systems using symbolic artificial intelligence and AI Planning.</p> <h2>Symbolic Artificial Intelligence</h2> <p>Symbolic artificial intelligence is one of the first approaches to AI, which is based on the assertion that human intelligence can be achieved through the manipulation of symbols. This is the basis for physical symbol systems, also called formal systems, which center around three basic concepts that follow human thinking abilities:</p> <ul> <li>Symbols, like the plus sign as the physical joining of two perpendicular lines, are first encoded in our brains.</li> <li>Thoughts are the structures, like the plus sign means to add things together.</li> <li>The manipulation process is the act of thinking, or applying the symbol and its structure together, like when we use the plus sign in a mathematical equation for one plus two equals three.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image15-1605168382441.jpg" alt=""></p> <p>To understand this concept more clearly, let's take a look at a couple of specific examples. Physical Symbol System examples include algebra, formal logic, and even chess.</p> <h2>Physical Symbol Example:  Algebra</h2> <p>In algebra, the numbers and mathematical operators of plus, x, equals, et cetera, are the symbols. The equations and formulas are the expressions, and the calculation is the manipulated expression response.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image1-1605168391109.jpg" alt=""></p> <h2>Physical Symbol System Example:  Formal Logic</h2> <p>With formal logic problems, words like "if," "or," and "not," are the symbols. The structures are true or false statements, and the process manipulation are the rules of logical deduction, which result in the final expression. So, we could say "If a primarily healthy adult has a fever, then they may have the flu."</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image5-1605168398975.jpg" alt=""></p> <h2>Physical Symbol System Example:  Chess</h2> <p>And in games, such as chess, the defined number of pieces are the symbols, and the legal chess moves are the structures. The manipulated expressions are the resulting positions of the pieces on the board after each move.</p> <p>This AI approach states that machines are capable of mimicking this behavior. Though interest in this approach has faded over time, it led to the development of expert systems, which are widely considered to be one of the first successful forms of artificial intelligence.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image10-1605168407973.jpg" alt=""></p> <h2>Expert Systems</h2> <p>An expert system is a computer system that is designed to solve problems by imitating the decision-making abilities of a human expert. This system uses two subsystems: a knowledge base and an inference engine.</p> <p>Input data is presented to an expert system for training. This data is then reasoned through production, or If-Then, rules. Together, the data and reasoning production rules create the knowledge base of an expert system.</p> <p>The inference engine applies the rules to the data and facts in the knowledge base, and then deduces new facts from it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image2-1605168415913.jpg" alt=""></p> <h2>AI Planning</h2> <p>Automated planning and scheduling, also known as AI Planning, is another branch of classic AI. AI Planning can be done in known environments, and it describes a system that coordinates strategies or action sequences from an initial state, in order to achieve a specified goal state. The actions may be executed by autonomous robots, intelligent agents, or unmanned vehicles, or a combination of them.</p> <p>This field has such a wide variety of project scopes with varying complexity that the level of programming effort and the human resources required for AI Planning became too much for most organizations to support.</p> <p>Today, machine learning has taken over this field, as it offers a much more agile approach.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image3-1605168423145.jpg" alt=""></p> <h2>Artificial Intelligence Examples</h2> <p>As we now realize, artificial intelligence varies greatly in its potential complexity.</p> <p>It includes both piles of if-then statements, as with the simple rule-based, expert systems used in classic AI, along with more complex statistical models that use learning algorithms to generate predictions.</p> <p>Then, there is also the Hollywood version of AI: super-fancy computer systems, specialized robots, and advanced androids.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image7-1605168431287.jpg" alt=""></p> <p>It may all seem a bit campy, but every day we get closer and closer to this type of reality. Mainly, this is because we are now teaching machines how to learn, and grow, on their own.</p> <h2>What is Machine Learning?</h2> <p>Now that we know about artificial intelligence, how about machine learning? This is where AI really starts to get interesting.</p> <p>Machine learning describes machines that are taught to learn and make decisions by examining large amounts of input data. It makes calculated suggestions and/or predictions based on analyzing this information and performs tasks that are considered to require human intelligence. This includes activities like speech recognition, translation, visual perception, and more.</p> <p>The field of machine learning also encompasses the area of deep learning. The key difference between machine learning and artificial intelligence is the term "learning."</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image9-1605168439805.jpg" alt=""></p> <h2>Why is Machine Learning Important?</h2> <p>Machines learn and provide intelligent insights through a sophisticated use of learning algorithms.</p> <p>To provide business value, the machine is trained to learn patterns from data and then can proceed autonomously on new and changing data. This creates a dynamic feedback loop, which allows it to efficiently generate more models to gain further insights, even more accurately, without requiring additional resources or human interaction.</p> <p>With continuous advancement in this field, machines are becoming increasingly self-healing, self-organizing, and self-architecting, seamlessly producing greater value for businesses.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image12-1605168447402.png" alt=""></p> <h2>What is Deep Learning?</h2> <p>Also known as artificial neural networks, deep learning is one of the most talked about sub-areas of machine learning.</p> <p>Deep learning performs machine learning in a hierarchy of layers, where the output of decisions from one layer feeds into the next layer. This model is loosely patterned after the brain's neural networks and has been setting new records of accuracy when applied to sound and image recognition.</p> <p>The term "deep" describes the number of layers in a network and some go deeper than others by using many layers, versus just one layer.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/image4-1605168455124.jpg" alt=""></p> <h2>Where do we go from here?</h2> <p>Next in this series, we'll discuss the different learning methods used in machine learning, such as supervised, unsupervised, and semi-supervised types, along with some of the most common algorithms available for your projects.</p> <p>For more information on this course, please visit: <a href="https://learn.ezmeral.software.hpe.com/bus-introduction-to-artificial-intelligence-and-machine-learning"><em>Introduction to Artificial Intelligence and Machine Learning</em></a>.</p>https://developer.hpe.com/performance-tuning-of-an-apache-kafkaspark-streaming-system-telecom-case/https://developer.hpe.com/performance-tuning-of-an-apache-kafkaspark-streaming-system-telecom-case/Thu, 12 Nov 2020 07:48:12 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Mathieu Dumoulin", "publish": "2017-05-31T12:00:00.000", "tags": "apache-spark" </code></pre> <hr> <h2><em>Real-world case study in the telecom industry</em></h2> <p><a href="/blog/MPxYKrwlr5SZQGoLnZ2X/performance-tuning-of-an-apache-kafkaspark-streaming-system">In a previous post</a>, I pointed out how we were successfully able to accelerate an Apache Kafka/Spark Streaming/Apache Ignite application and turn a development prototype into a useful, stable streaming application – one that actually exceeded the performance goals set for the application. In this post, I’ll cover how we were able to tune a Kafka/Spark Streaming system and run it stably, without backing up under maximum production load.</p> <p>Many of the lessons learned during this project would also apply to a similar system implemented using the MapR Data Platform. However, as we’ll explain later, a lot of the issues could have been avoided entirely, or at least greatly mitigated by using a converged platform instead of a multi-cluster approach.</p> <p>The MapR Data Platform is the only currently available production-ready implementation of such a platform as of this writing.</p> <h2>Goal of the System</h2> <p>The Kafka/Spark Streaming system aims to provide better customer support by providing their support staff with always up-to-date call quality information for all their mobile customers.</p> <p>Mobile customers, while making calls and using data, connect to the operator’s infrastructure and generate logs in many different systems. Three specific logs were identified that, if correlated with each other, give visibility in the actual quality of service experienced by each individual customer. The three logs were selected because they can be correlated through a simple relational database-like join operation.</p> <p>For improving customer support, the quality of call information needs to be kept updated in near to real time; otherwise, it has no value. This has led, down the road, to building a streaming architecture rather than a batch job. The data volume at production load reaches several GB/s, generated by several million mobile customers, 24 hours a day, 365 days a year. Performance and stability at that scale is required for the system to reach production.</p> <h2>Project SLA Goals</h2> <p>The application has clear performance requirements based on the known worst-case throughput of the input data. This log data is generated by real-world use of the services of the company. If the application is to be useful at all, as a real-time streaming application, it must be able to handle this data without getting behind.</p> <p>In term of numbers, the goal is to handle up to 3GB/min of input data. For this large mobile operator, such throughput represents about 150-200,000 events/second. Ordinarily, the throughput is about half of that value or 1.5GB/min and 60,000-80,000 events/second.</p> <h2>Data Sources</h2> <p>The raw data source are the logs of three remote systems, labeled A, B, and C here, where the log from A comprises about 84-85% of the entries, the log from B about 1-2%, and the log from C about 14-15%. The fact that the data is unbalanced is one of the (many) sources of difficulty in this application.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture1-1605167436242.png" alt=""></p> <p>The raw data is ingested into the system by a single Kafka producer into Kafka running on 6 servers. The producer reads the various logs and adds each log's records into its own topic. As there are three logs, there are three Kafka topics.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture2-1605167445109.png" alt=""></p> <p>The data is consumed by a Spark Streaming application, which picks up each topic, does a simple filter to cut out unnecessary fields, a map operation to transform the data, and then a foreachRDD operation (each micro-batch generates an RDD in Spark Streaming) that saves the data to Ignite and to HDFS as Hive tables for backup.</p> <p>A second batch Spark application runs once per hour on the data stored in-memory in Ignite to join the records from the three separate logs into a single table. The batch job has a maximum data size of about 100GB. The cluster CPU resources should be sufficient to process this amount of data in one hour or less.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture3-1605167453170.png" alt=""></p> <p>Ignite stores 3 hours’ worth of data at all time to account for calls that begin in one hour and end in the hour getting processed, as well as calls that begin in the target hour and end in the next one. The telecom operator judges that calls that are so long they aren’t captured in this scheme can be ignored, as they are very rare.</p> <p>It’s worth noting that a better all-streaming architecture could have avoided the whole issue with the intermediate representation in the first place. An illustrative, real-world case with more time and thought upfront can make the entire project end faster than just rushing headlong into coding the first working solution that comes to mind.</p> <h2>System Hardware and Software: At the Bleeding Edge of Open Source Big Data</h2> <p>The cluster has a lot of CPU and memory resources. It has 12 nodes of enterprise-grade servers, each equipped with two E5 Xeon CPUs (16 physical cores), 256GB memory, and eight 6TB spinning HDD (2 for OS in RAID 1). Each server has one 10GbE network interface.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture4-1605167461409.png" alt=""></p> <p>The technology stack selected for this project are centered around Kafka 0.8 for streaming the data into the system, <a href="https://ignite.apache.org/">Apache Spark 1.6</a> for the ETL operations (essentially a bit of filter and transformation of the input, then a join), and the use of Apache Ignite 1.6 as an in-memory shared cache to make it easy to connect the streaming input part of the application with joining the data. Backup is done to HDFS, as Hive ORC tables are also used to serve as a just-in-case backup for Ignite and to serve future need for other analytics use cases (none at the time).</p> <p>The Spark applications are both coded in Scala 2.10 and <a href="http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers">Kafka’s direct approach</a> (no receivers). Apache Ignite has a really nice Scala API with a magic <a href="https://ignite.apache.org/features/igniterdd.html">IgniteRDD</a> that can allow applications to share in-memory data, a key feature for this system to reduce coding complexity.</p> <p>The cluster is running Apache Hadoop's HDFS as a distributed storage layer, with resources managed by Mesos 0.28. Finally, HBase is used as the ultimate data store for the final joined data. It will be queried by other systems outside the scope of this project. The cluster design with all relevant services is shown in the table above.</p> <h2>Performance Issues</h2> <p>The original system had several issues:</p> <ol> <li>Performance</li> </ol> <ul> <li>First Spark Streaming job is not stable</li> <li>Second Spark batch job can’t process 1 hour of data before the next hour of data arrives</li> </ul> <ol start="2"> <li>Stability: The application crashes under load</li> </ol> <p>A Spark Streaming application is said to be stable if the processing time of each micro-batch is less than or equal to that micro-batch time. In this case, the application processes each 30 seconds of data in as much as 6 minutes. We need a 12x speedup.</p> <p>Second, there is a batch process to join data one hour at a time that was targeted to run in 30 minutes but was taking over 2 hours to complete.</p> <p>Third, the application was randomly crashing after running for a few hours. Stability of such a complex, fully open-source stack should never be assumed. Rather, it is the result of a constant effort by the team to better understand the system. We can expect that there will still be a lot of learning required to keep the system up and running once it is moved to production as well.</p> <h2>Performance Tuning</h2> <p>In my opinion, all performance and stability issues stem from the terrible idea of management to push a very good POC project developed on AWS into production on some on-premises hardware. It’s hard to believe, but they fully expected the POC code to run as-is on a production system it was never tested on.</p> <p>Regardless, the task was set, and we had only a few short days to identify what could be done and get the system up to production speed. Final QA testing of the system was barely 1 week away, and management wasn’t in the mood to accept delays. We got to work...</p> <h2>First target: Improve Spark Streaming Performance</h2> <p>At maximum load, the Spark Streaming application is taking between 4.5 to 6 minutes for each micro-batch of 30 seconds. We need to find 9-12x speedup worth of improvements.</p> <p>Spark has a lot of moving parts, but it will always be true that fast algorithms beat tweaking the configuration. In this case, there is nothing to get from the code; it’s all very parallelizable with no obvious issues, like doing two computations separately when they could be combined or any O(n^2) loop-in-another loop issues. The job is nothing more than a filter and a map.</p> <p>What we need to determine, then, is whether the job is indeed being processed in parallel to make the most of all those CPU cores. In a Spark Streaming job, Kafka partitions map 1 to 1 with Spark partitions.</p> <h2>Increase Parallelism: Increase Number of Partitions in Kafka</h2> <p>A quick check of the Spark UI shows 36 partitions. As each server has 6 physical disks, I assume the choice of partitioning was selected by the formula node * physical disks = partition count per topic. Quickly checking online reveals that partitioning is quite a bit more complex than that and the formula to decide on partition number isn’t from any known Kafka best practices guide. (Ref: <a href="https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/">https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/</a> )</p> <p>The input data was unbalanced and most of the application processing time was spent processing Topic 1 (with 85% of the throughput). Kafka partitions are matched 1:1 with the number of partitions in the input RDD, leading to only 36 partitions, meaning we can only keep 36 cores busy on this task. To increase the parallelism, we need to increase the number of partitions. What we did was split topic 1 into 12 topics each with 6 partitions for a total of 72 partitions. The way it was done was a simple modification to the producer to evenly divide the data from the first log into 12 topics instead of just one. Zero code needed to be modified on the consumer side.</p> <p>We also right-sized the number of partitions for the two other topics, in proportion to their relative importance in the input data, so we set topic 2 to two partitions and topic 3 to eight partitions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/completed-jobs-1605256886579.png" alt=""></p> <p><em>Running more tasks in parallel. Before tuning, each stage always had 36 partitions!</em></p> <h2>Fix RPC Timeout Exceptions</h2> <p>When looking at the application logs, we could see a lot of RPC timeout exceptions. We do a web search and find what we believe is the relevant JIRA (<a href="https://issues.apache.org/jira/browse/SPARK-14140">SPARK-14140 in JIRA</a>). The recommended fix is to increase the spark.executor.heartbeatInterval from 10s (default) to 20s.</p> <p>I think this could be caused by nodes getting busy from disk or CPU spikes because of Kafka, Ignite, or garbage collector pauses. Since Spark runs on all nodes, the issue was random (see the cluster services layout table in the first section).</p> <p>The configuration change fixed this issue completely. We haven’t seen it happen since. (Yay!)</p> <h2>Increase Driver and Executor Memory</h2> <p>Out of memory issues and random crashes of the application were solved by increasing the memory from 20g per executor to 40g per executor as well as 40g for the driver. Happily, the machines in the production cluster were heavily provisioned with memory. This is a good practice with a new application, since you don’t know how much you will need at first.</p> <p>The issue was difficult to debug with precision and reliable information, since the Spark UI reports very little memory consumption. In practice, as this setting is easy to change, we empirically settled on 40g being the smallest memory size for the application to run stably.</p> <h2>Right Size the Executors</h2> <p>The original application was running only 3 executors with 72 total cores. We configured the application to run with 80 cores with a maximum of 10 cores per executor, for a total of 8 executors. Note that with 16 real cores per node on a 10 node cluster, we’re leaving plenty of resources for Kafka brokers, Ignite, and HDFS/NN to run on.</p> <h2>Increase the Batch Window from 30s to 1m</h2> <p>The data is pushed into Kafka by the producer as batches every 30s, as it is gathered by FTP batches from the remote systems. Such an arrangement is common in telecom applications due to a need to deal with equipment and systems from a bewildering range of manufacturers, technology, and age.</p> <p>This meant that the input stream was very spiky, when looking at the processing time from the Spark UI’s streaming tab.</p> <p>Increasing the window to 1m allowed us to smooth out the input and gave the system a chance to process the data in 1 minute or less and still be stable.</p> <p>To make sure of it, the team had a test data which simulated the known worst-case data, and with the new settings, the Spark Streaming job was now indeed stable. We also tried it on real production data, and everything looked good. Win!</p> <h2>Drop Requirement to Save Hive Tables to HDFS</h2> <p>Discussion with the project managers revealed that Hive was not actually part of the requirements for the streaming application! Mainly, this is because the other analytics, mostly SQL requests, could be serviced from the data in HBase.</p> <p>Considering the goal of the system, the worst-case scenario for missing data is that a customer's call quality information cannot be found... which is already the case. In other words, the consequence of data loss is not negative; rather, the consequence of gaining data is additional insights. If the great majority of the data is processed and stored, the business goals can be reached.</p> <p>There wasn’t much point in saving the data to Hive mid-flight for increased fault-tolerance either, as once the data is in Ignite, it’s safe even if the Spark application crashes. This made Ignite an even more critical part of the application, despite it having some issues of its own. It was a difficult decision that we made entirely due to the advanced stage of the project. As we’ll explain in more detail in the conclusion, the architecture itself was problematic, and it’s not time to play with architecture when you’re a week or two from production.</p> <h2>Spark Performance Tuning Results</h2> <p>The Spark Streaming application finally became stable, with an optimized runtime of 30-35s.</p> <p>As it turns out, cutting out Hive also sped up the second Spark application that joins the data together, so that it now ran in 35m, both now well within the project requirements.</p> <p>With improvements from the next part, the final performance of the Spark Streaming job went down in the low 20s range, for a final speedup of a bit over 12 times.</p> <h2>Second target: Improve System Stability</h2> <p>We had to work quite hard on stability. Several strategies were required, as we will explain below.</p> <h2>Make the Spark Streaming Application Stable</h2> <p>The work we did to fix the performance had a direct impact on system stability. If both applications are stable themselves and running on right-sized resources, then the system has the best chance to be stable overall.</p> <h2>Remove Mesos and Use Spark Standalone</h2> <p>The initial choice of Mesos to manage resources was forward-looking, but ultimately we decided to drop it from the final production system. At the onset, the plan was to have Mesos manage all the applications. But the team never could get Kafka and Ignite to play nice with Mesos, and so they were running in standalone mode, leaving only Spark to be managed by Mesos. Surely, with more time, there is little doubt all applications could be properly configured to work with Mesos.</p> <p>Proposing to remove Mesos was a bit controversial, as Mesos is much more advanced and cool than Spark running in standalone mode.</p> <p>But the issue with Mesos was twofold:</p> <ol> <li>Control over executor size and number was poor, a known issue (SPARK-5095) with Spark 1.6 and now fixed in Spark 2.X.</li> <li>Ignite and Kafka aren’t running on Mesos, only Spark is. Given the schedule pressure, the team had given up trying to get those two services running in Mesos.</li> </ol> <p>Mesos can only ever allocate resources well if it controls resources. In the case of this system, Kafka and Ignite are running outside of Mesos’ knowledge, meaning it’s going to assign resources to the Spark applications incorrectly.</p> <p>In addition, it’s a single purpose cluster, so we can live with customizing the sizing of the resources for each application with a global view of the system’s resources. There is little need for dynamic resource allocations, scheduling queues, multi-tenancy, and other buzzwords.</p> <h2>Change the Ignite Memory Model</h2> <p>It is a known issue that when the Heap controlled by the JVM gets very big (> 32GB), the cost of garbage collection is quite large. We could indeed see this when the join application runs, where the stages with 25GB shuffle had some rows with spikes in GC time from 10 seconds range up to more than a minute.</p> <p>The initial configuration of Ignite was to run ONHEAP_TIERED, with 48GB worth of data cached on heap, then overflow drops to 12GB of off-heap memory. That setting was changed to the OFFHEAP_TIERED model. While slightly slower due to serialization cost, OFFHEAP_TIERED doesn't rely on the JVM’s garbage collection. It still runs in memory, so we estimated it would be a net gain.</p> <p>With this change, the run time for each batch dutifully came down by about five seconds, from 30 seconds down to about 25 seconds. In addition, successive batches tended to have much more similar processing time, with a delta of 1-3 seconds, whereas it would vary by over 5 to 10 seconds, previously.</p> <h2>Update the Ignite JVM Settings</h2> <p>We followed the recommended JVM options as found in Ignite documentation’s performance tuning section (<a href="http://apacheignite.gridgain.org/docs/jvm-and-system-tuning">http://apacheignite.gridgain.org/docs/jvm-and-system-tuning</a>).</p> <h2>Improve the Spark Code</h2> <p>Some parts of the code assumed reliability, like queries to Ignite, when in fact there was possibility of the operations failing. These problems can be fixed in the code, which now handles exceptions more gracefully, though there is probably work left to increase the robustness of the code. We can only find these spots by letting the application run now.</p> <h2>Reassign ZooKeeper to Nodes 10-12</h2> <p>Given the cluster is of medium size, it’s worth spreading the services as much as possible. We moved the ZooKeeper services from nodes 1-3 to nodes 10-12.</p> <h2>Final System Architecture</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture6-1605167479768.png" alt=""></p> <h2>Conclusion</h2> <p>Tuning this application took about 1 week of full-time work. The main information we used was Spark UI and Spark logs, easily accessible from the Spark UI. The view of Jobs and Stages as well as the streaming UI are really very useful.</p> <h2>Essential Takeaways</h2> <ul> <li>Migrating a streaming application from a prototype on AWS to an on-site cluster requires schedule time for testing</li> <li>Not testing the AWS prototype with realistic data was a big mistake</li> <li>Including many “bleeding-edge” OSS components (Apache Ignite and Mesos) with expectations of very high reliability is unrealistic</li> <li>A better architecture design could have simplified the system tremendously</li> <li>Tuning a Kafka/Spark Streaming application requires a holistic understanding of the entire system; it’s not just about changing parameter values of Spark. It’s a combination of the data flow characteristics, the application goals and value to the customer, the hardware and services, the application code, and then playing with Spark parameters.</li> <li>The MapR Data Platform would have cut the development time, complexity, and cost for this project.</li> </ul> <p>This project was a hell of a dive in the deep end of the pool for a telecom operator with very little experience with the open-source enterprise big data world. They should be applauded for ambition and desire to take up such a challenge with the goal of benefiting their customers. But a better choice of platform and application architecture could have made their life a lot easier.</p> <h2>A Converged Platform is the Correct Approach</h2> <p>In fact, the requirements for this project show the real-world business need for a state-of-the-art converged platform with a fast distributed file system, high performance key-value store for persistence and real-time, and high performance streaming capabilities.</p> <p>A MapR-based solution would have been a lot easier to build and maintain, absolutely for sure. Since MapR Event Store is built-in, there is one less cluster to manage (bye bye, Kafka brokers). The Spark application could run with the same code but without needing to rely on a speculative open-source project like Apache Ignite.</p> <p>Saving to MapR Database uses the same HBase API, so likely no code change there either, and you’re saving to a DB that’s built into the native C MapR XD, so that’s going to be super fast as well. Finally, sharing the resources is simplified by running only Spark on YARN or standalone-mode, while the platform is left to deal with the resource requirements of the MapR Event Store, MapR XD, and MapR Database with reliability and performance, guaranteed, since highly trained support engineers are available 24/7 to support every single part of this application.</p> <p>Given this system is heading into production for a telecom operator with 24/7 reliability expectation, I’d argue that built-in simplicity, performance, and support are pretty compelling and hopefully will be adopted by this customer for the next iteration of the system. (Stay tuned!)</p>https://developer.hpe.com/kubernetes-kafka-event-sourcing-architecture-patterns-and-use-case-examp/https://developer.hpe.com/kubernetes-kafka-event-sourcing-architecture-patterns-and-use-case-examp/Wed, 11 Nov 2020 07:02:01 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-05-01T11:00:00.000", "tags": "use-cases" </code></pre> <hr> <p>With the rapidly changing business and technology landscape of today, developers, data scientists, and IT operations are working together to build intelligent applications with new technologies and dynamic architectures because of the flexibility, speed of delivery, and maintainability that they make possible. This post will go over the technologies that are facilitating evolutionary architectures: containers, Kubernetes, and the Kafka API. Then we will look at some Kafka event sourcing architecture patterns and use case examples.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/containers-1605078458080.png" alt="Containers"></p> <h2>Containers Architecture</h2> <p>Containers simplify going from development to deployment, without having to worry about portability or reproducibility. Developers can package an application plus all its dependencies, libraries, and configuration files needed to execute the application into a container image. A container is a runnable instance of an image. Container images can be pulled from a registry and deployed anywhere the container runtime is installed: your laptop, servers on-premises, or in the cloud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/docker-host-1605078472533.png" alt="Docker_Host"></p> <p>image reference <a href="https://docs.docker.com/engine/docker-overview/#docker-architecture">https://docs.docker.com/engine/docker-overview/#docker-architecture</a></p> <p>Compared to virtual machines, containers have similar resources and isolation benefits, but are lighter in weight, because containers virtualize the operating system instead of the hardware. Containers are more portable and efficient, take up less space, use far fewer system resources, and can be spun up in seconds.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/vm-containers-1605078486231.png" alt="Virtual Machines and Containers"></p> <h2>Kubernetes Architecture</h2> <p><a href="https://kubernetes.io/">Kubernetes</a> provides a platform to configure, automate, and manage:</p> <ul> <li>intelligent and balanced scheduling of containers</li> <li>creation, deletion, and movement of containers</li> <li>easy scaling of containers</li> <li>monitoring and self-healing abilities</li> </ul> <p>A Kubernetes cluster is comprised of at least one master node, which manages the cluster, and multiple worker nodes, where containerized applications run using Pods. A Pod is a logical grouping of one or more containers, which are scheduled together and share resources. Pods enable multiple containers to run on a host machine and share resources, such as: storage, networking, and container runtime information.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/clusters-1605078500093.png" alt="Clusters"></p> <p>The Master node manages the cluster in this way:</p> <ul> <li>The API server parses the YAML configuration and stores the configuration in the etcd key value store.</li> <li>The etcd stores and replicates the current configuration and run state of the cluster.</li> <li>The scheduler schedules pods on worker nodes.</li> <li>The controller manager manages the state of non-terminating control loops, such as pod replicas.</li> </ul> <p>The <a href="https://martinfowler.com/articles/microservices.html">microservice architectural style</a> is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. A microservice approach is well aligned to containers and Kubernetes. You can gain modularity, extensive parallelism, and cost-effective scaling by deploying services across many nodes. Microservices modularity facilitates independent updates/deployments and helps to avoid single points of failure, which can help prevent large-scale outages.</p> <p>The MapR Data Fabric includes a natively integrated Kubernetes volume driver to provide persistent storage volumes for access to any data located on-premises, across clouds, and to the edge. Stateful applications can now be easily deployed in containers for production use cases, machine learning pipelines, and multi-tenant use cases.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mapr-cdp-1605078512665.png" alt="MapR CDP"></p> <h2>Event-Driven Microservices Architecture</h2> <p>Most business data is produced as a sequence of events, or an event stream: for example, web or mobile app interactions, sensor data, bank transactions, and medical devices all continuously generate events. Microservices often have an event-driven architecture, using an append-only event stream, such as Kafka or MapR Event Streams (which provides a Kafka API).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mapr-es-1605078524402.png" alt="MapR Event Store"></p> <p>With MapR Event Store (or Kafka), events are grouped into logical collections of events called "topics." Topics are partitioned for parallel processing. You can think of a partitioned topic like an event log, new events are appended to the end, and like a queue,events are delivered in the order they are received.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mapr-cluster-1605078538269.png" alt="MapR Cluster"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mapr-cluster-2-1605078550838.png" alt="MapR Cluster 2"></p> <p>Unlike a queue, events are not deleted after they are delivered, they remain on the partition, available to other consumers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/events-1605078564306.png" alt="Events"></p> <p>Older messages are automatically deleted, based on the stream's timetolive setting; if the setting is 0, then they will never be deleted.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/messages-1605078583134.png" alt="Messages"></p> <p>Messages are not deleted from topics when read, and topics can have multiple different consumers; this allows processing of the same messages by different consumers for different purposes. Pipelining is also possible, where a consumer enriches an event and publishes it to another topic.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/kafka-api-1605078598421.png" alt="Kafka API"></p> <p>MapR Event Store provides scalable high performance messaging, easily delivering millions of messages per second on modest hardware. The publish/subscribe Kafka API provides decoupled communications, making it easy to add new listeners or new publishers without disrupting existing processes.</p> <p>When you combine these messaging capabilities with the simple concept of microservices, you can greatly enhance the agility with which you build, deploy, and maintain complex data pipelines. Pipelines are constructed by simply chaining together multiple microservices, each of which listens for the arrival of some data, performs its designated task, and optionally publishes its own messages to a topic.</p> <h2>The Stream Is the System of Record</h2> <p><a href="https://martinfowler.com/eaaDev/EventSourcing.html">Event Sourcing</a> is an architectural pattern in which the state of the application is determined by a sequence of events, each of which is recorded in an append-only event store or stream. As an example, imagine that each "event" is an incremental update to an entry in a database. In this case, the state of a particular entry is simply the accumulation of events pertaining to that entry. In the example below, the stream persists the queue of all deposit and withdrawal events, and the database table persists the current account balances.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/entry-database-1605078612009.png" alt="Entry Database"></p> <p>Which one of these, the stream or the database, makes a better system of record? The events in the stream can be used to reconstruct the current account balances in the database, but not the other way around. Database replication actually works by suppliers writing changes to a change log, and consumers applying the changes locally.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/replication-1605078622856.png" alt="Replication"></p> <h2>Adding Microservices to a Bank Monolithic Application with Change Data Capture</h2> <p>Banks often have mainframe applications, which are expensive to run, difficult to update, and also difficult to completely replace. Let's look at how we could incrementally add event-driven microservices to a monolithic bank application, which consists of payment transactions and batch jobs for fraud detection, statements, and promotion emails.</p> <p>In the design shown below, payment transactions from the monolithic database commit log are published to a stream, which is set to never throw data away. The immutable event store (stream) becomes the system of record, with events processed by different data pipelines, based on the use case. Event data pipelines funnel out to <a href="https://martinfowler.com/bliki/PolyglotPersistence.html">polyglot persistence</a>, different data storage technologies, each one providing different materialized views: MapR Database HBase and MapR Database JSON document, graph, and search databases, so that microservices always have the most up-to-date view of their data in the most appropriate format. Using a different model for reading than for writing is the <a href="http://martinfowler.com/bliki/CQRS.html">Command Query Responsibility Separation</a> pattern.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/pattern-1605078634993.png" alt="Pattern"></p> <p>The event store provides for rebuilding state by re-running the events in the stream — this is the <a href="http://martinfowler.com/eaaDev/EventSourcing.html">Event Sourcing</a> pattern. Events can be reprocessed to create a new index, cache, or view of the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/reprocessing-events-1605078645854.png" alt="Reprocessing Events"></p> <p>The consumer simply reads from the oldest message to the latest to create a new view of the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/create-new-view-1605078659099.png" alt="Create New View"></p> <p>With the payment transactions now coming in as an event stream, real time fraud detection, using Spark Machine Learning and Streaming, could be added more easily than before, as shown in the data flow below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/stream-processing-1605078670497.png" alt="Stream Processing"></p> <p>Having a long retention time for events in the stream allows for more analysis and functionality to be added. For example, a materialized view of card location histories could be stored in a data format such as Parquet, which provides very efficient querying.</p> <p>Evolving the Architecture by Adding Events and Microservices</p> <p>With more event sources, stream processing and machine learning can be added to provide new functionality. Machine learning techniques across a wide range of interactions — including click stream, click through rates, call center reports, customer preferences, and purchase data — can be used to provide insights, such as: financial recommendations, predictions, alerts, and relevant offers. For example, web click stream analysis combined with purchase history can be used to segment customers who share behavioral affinities into groups, in order to better target advertisements. Lead events can be added to a stream when a customer clicks on targeted offers, triggering updates to the customer profile in MapR Database and automated campaigns to prospects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/microservices-1605078682511.png" alt="Microservices"></p> <h2>Healthcare Event Sourcing Examples</h2> <p>Now let's look at how a stream-first architecture has been implemented in healthcare. Data from hospitals, providers, and labs flow into the ALLOY Health Platform. MapR Event Store solves the data lineage problem of HIPAA compliance because the stream becomes a system of record by being an infinite, immutable log of each data change. Polyglot persistence solves the problem of storing multiple data formats. By streaming data changes in real time to the MapR Database HBase API/MapR Database JSON API, graph, and search databases, materialized views can be provided, explored, and analyzed for different use cases, such as population health queries and patient matching.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/sourcing-examples-1605078694102.png" alt="Sourcing Examples"></p> <h2>Other healthcare stream processing and machine learning data pipeline examples include:</h2> <ul> <li>Using predictive analytics on claims events to reduce fraud waste and abuse for healthcare payments.</li> <li>Using predictive analytics across multiple sources from over <a href="https://www.datapine.com/blog/big-data-examples-in-healthcare/">30 million patients</a> to: <ul> <li><a href="https://healthcare-conversation.com/2015/07/06/todays-predictive-analytics-should-provide-timely-actionable-intelligence/">provide doctors with timely, actionable intelligence</a> to aid in the accuracy of diagnosing patient conditions</li> <li>aid in the matching of treatments with outcomes</li> <li>predict patients at risk for disease or readmission</li> </ul> </li> </ul> <h2>Retail Event Sourcing Example</h2> <p>A major retailer wanted to increase in-season agility and inventory discipline in order to react to demand changes and reduce markdowns.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/retail-event-example-1605078705622.png" alt="Retail Event Example"></p> <p>Data is collected from point of sale transactions, inventory status and pricing, competitive intelligence, social media, weather, and customers (scrubbed of personal identification), allowing for a centralized analysis of correlations and patterns that are relevant to improving business. Big data algorithms analyze in-store and online purchases, Twitter trends, local sports events, and weather buying patterns to build innovative applications that personalize customer experience while increasing the efficiency of logistics. Point of sale transactions are analyzed to provide product recommendations or discounts based on which products were bought together or before another product. Predictive analytics is used to know what products sell more on particular days in certain kinds of stores, in order to reduce overstock and stay properly stocked on the most in-demand products, thereby helping to optimize the supply chain.</p> <h2>Conclusion</h2> <p>A confluence of several different technology shifts have dramatically changed the way that applications are being built. The combination of event-driven microservices, containers, Kubernetes, and machine learning data pipelines is accelerating the development of next-generation intelligent applications, which are taking advantage of modern computational paradigms, powered by modern computational infrastructure. The MapR Data Platform integrates global event streaming, real-time database capabilities, and scalable enterprise storage with a collection of data processing and analytical engines to power this new generation of data processing pipelines and intelligent applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/mapr-converged-1605078721345.png" alt="MapR CDP"></p>https://developer.hpe.com/kafka-vs-mapr-event-store-why-mapr/https://developer.hpe.com/kafka-vs-mapr-event-store-why-mapr/Wed, 11 Nov 2020 06:51:11 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ian Downard", "publish": "2017-01-11T08:00:00.000Z", "tags": "streaming" </code></pre> <hr> <p>A lot of people choose MapR as their core platform for processing and storing big data because of its advantages for speed and performance. MapR consistently performs faster than any other big data platform for all kinds of applications, including Hadoop, distributed file I/O, NoSQL data storage, and data streaming. In this post, I’m focusing on the latter to provide some perspective on how much better/faster/cheaper MapR Event Store can be compared to Apache Kafka as a data streaming technology.</p> <p>MapR Event Store for Apache Kafka is a cluster-based messaging system for streaming data at scale. It’s integrated into the MapR Data Platform and implements the Apache Kafka Java API so applications written for Kafka can also run on MapR Event Store. What differentiates the MapR Event Store technology from Kafka are its built-in features for global replication, security, multi-tenancy, high availability, and disaster recovery—all of which it inherits from the MapR Data Platform. From an operational perspective, these features make MapR Event Store easier to manage than Kafka, but there are speed advantages, too. I’ve been looking at this a lot lately, trying to understand where and why MapR Event Store outperforms Kafka. In this blog post, I will share with you how clearly <strong>MapR Event Store can transport a much faster stream of data, with much larger message sizes, and to far more topics than what can be achieved with Kafka</strong>.</p> <h2>Test Strategy</h2> <p>In this study, I wanted to compare Kafka and MapR Event Store as to how they perform “off the shelf” without the burden of tuning my test environment to perfectly optimize performance in each test scenario. So, I have pretty much stuck with the default settings for services and clients. The only exceptions are that I configured each Kafka topic with a replication factor of 3 and configured producers to send messages synchronously, since these are the default modes for MapR Event Store. I also disabled stream compression in order to control message sizes and measure throughput more precisely.</p> <h2>Test Configurations</h2> <p>I measured performance from both producer and consumer perspectives. However, consumers run faster than producers, so I focused primarily on the producer side since the throughput of a stream is bounded by the throughput of its producers. I used two threads in my producer clients so that message generation could happen in parallel with sending messages and waiting for acknowledgments. I used the following properties for producers and topics:</p> <pre><code class="language-markdown">acks = all batch.size = 16384 latency.ms = 0ms block.on.buffer.full = true compression = none default.replication.factor = 3 </code></pre> <p>My test environment consisted of three Ubuntu servers running Kafka 2.11-0.10.0.1 or MapR 5.2 on Azure VMs sized with the following specs:</p> <ul> <li>Intel Xeon CPU E5-2660 2.2 GHz processor with 16 cores</li> <li>SSD disk storage with 64,000 Mbps cached / 51,200 uncached max disk throughput</li> <li>112GB of RAM</li> <li>Virtual networking throughput between 1 and 2 Gbits/sec (I measured this quantitatively since I couldn’t easily find virtual network throughput specs from Microsoft).</li> </ul> <h2>Performance Metrics</h2> <p>Throughput, latency, and loss are the most important metrics measuring the performance of a message bus system. MapR Event Store and Kafka both guarantee zero loss through at-least-once semantics. MapR provides some advantages when it comes to latency, but typically both MapR Event Store and Kafka deliver messages sufficiently quick for real-time applications. For those reasons, I chose to focus on throughput in this study.</p> <p>Throughput is important because if an application generates messages faster than a message bus can consume and deliver them, then those messages must be queued. Queueing increases end-to-end latency and destabilizes applications when queues grow too large.</p> <p>Furthermore, throughput in Kafka and MapR Event Store is sensitive to the size of the messages being sent and to the distribution of those messages into topics. So, I analyzed those two attributes independently in order to measure how message size and stream topics affect throughput.</p> <h2>Throughput Performance</h2> <p>To measure producer throughput, I measured how fast a single producer could publish a sustained flow of messages to single topic with 1 partition and 3x replication. I ran this test for a variety of message sizes to see how that affects throughput. The results show MapR Event Store consistently achieving much higher throughput than Kafka and having a much higher capacity for handling large message sizes, as shown below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-tput-bytes-1605077662806.png" alt="Throughput MB/s"></p> <p>MapR Event Store doesn’t just send a faster volume of data than Kafka; it also has the capacity to send more records per second. We can see this by plotting throughput in terms of raw record count, as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-tput-records-1605077683740.png" alt="Throughput Msgs/s"></p> <p>I recorded these results with two different code bases. First, I used custom tests that I wrote using the Java unit test framework (JUnit), then I used the performance test scripts included with Kafka and MapR. These different approaches did not produce exactly the same results but they were close, as shown below. This correlation helps validate the conclusions stated above, that <strong>MapR Event Store can transport a larger volume of data and more frequent messages than Kafka</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-tput-v1v2-1605077697749.png" alt="Throughput Correlation"></p> <h2>How does MapR Event Store achieve more than 4x throughput than Kafka?</h2> <p>There are a lot of reasons why MapR Event Store is faster, and without getting too technical, I’ll mention just a few. First, the MapR Event Store client more efficiently flushes data to the MapR Event Store server. It spawns its own threads to do this work, whereas Kafka uses the client application threads directly to flush to a Kafka broker, which in many cases is limited to just a single thread.</p> <p>On the server side, MapR Event Store inherits efficient I/O patterns from the core MapR storage layer which keeps files coherent and clean so that I/O operations can be efficiently buffered and addressed to sequential locations on disk. Replication is more efficient, too, since the underlying MapR storage platform has distributed synchronous replication built in, along with other operational features that simply don’t exist in Kafka, such as snapshots, mirroring, quotas, access controls, etc.</p> <h2>Replicating this test</h2> <p>My JUnit tests for benchmarking Kafka and MapR Event Store is available at <a target='\_blank' href='https://github.com/iandow/kafka_junit_tests'><a href="https://github.com/iandow/kafka_junit_tests">https://github.com/iandow/kafka_junit_tests</a></a>. Here are the commands that I used to generate the data shown above:</p> <pre><code class="language-bash">git clone https://github.com/iandow/kafka_junit_tests cd kafka_junit_tests # Create a Kafka topic... /opt/kafka_2.11-0.10.0.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic t-00000 --config compression.type=uncompressed # or create a MapR Event Store topic. maprcli stream create -path /user/mapr/iantest -produceperm p -consumeperm p -topicperm p -defaultpartitions 1 -compression off # Then compile. mvn -e -Dtest=MessageSizeSpeedTest test # Test data will be saved in size-count.csv </code></pre> <p>You can also measure throughput using the performance test utilities included with Kafka and MapR. Here are the commands that I used to do that:</p> <p>Kafka script:</p> <p><a target='\_blank' href='https://gist.github.com/iandow/bf5df0f9b4f19e6a19aa5a7a93b7c81c'><a href="https://gist.github.com/iandow/bf5df0f9b4f19e6a19aa5a7a93b7c81c">https://gist.github.com/iandow/bf5df0f9b4f19e6a19aa5a7a93b7c81c</a></a></p> <p>MapR script:</p> <p><a target='\_blank' href='https://gist.github.com/iandow/0750185f1d3631301d476b426c109a50'><a href="https://gist.github.com/iandow/0750185f1d3631301d476b426c109a50">https://gist.github.com/iandow/0750185f1d3631301d476b426c109a50</a></a></p> <h2>Topic Scalability</h2> <p>Another major advantage that MapR Event Store holds over Kafka relates to how well it can handle large quantities of stream topics. Topics are the primary means of organizing stream data; however, there is overhead associated with categorizing streams into topics, and producer throughput is sensitive to that overhead. I quantified this by measuring how fast a single producer could publish a sustained flow of messages to an increasingly large quantity of topics. This is essentially a "fan-out" producer (illustrated below) and it is very common for fast data pipelines to use this pattern so that data can be more easily consumed downstream.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-fanout-1605077711198.png" alt="Fanout Producer"></p> <p>Each of the topics created for this scenario were configured with a single partition and 3x replication. Record size was held constant at 100 bytes.</p> <p>It’s clear from the following graph that <strong>MapR Event Store scales to a larger quantity of topics than Kafka</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-tput-topics-1605077727421.png" alt="Topic Scalability"></p> <h2>How does MapR Event Store handle so many more topics than Kafka?</h2> <p>A topic is just metadata in MapR Event Store; it does not introduce overhead to normal operations. MapR Event Store uses only one data structure for a stream, no matter how many topics it has, and the MapR storage system provides extremely fast and scalable storage for that data.</p> <p>On the other hand, Kafka represents each topic by at least one directory and several files in a general purpose file system. The more topics/partitions Kafka has the more files it creates. This makes it harder to buffer disk operations, perform sequential I/O, and it increases the complexity of what ZooKeeper must manage.</p> <h2>Replicating this test</h2> <p>This scenario can be run with another JUnit test from <a href="https://github.com/iandow/kafka_junit_tests">https://github.com/iandow/kafka_junit_tests</a>, as follows:</p> <pre><code class="language-bash">git clone https://github.com/iandow/kafka_junit_tests cd kafka_junit_tests # For MapR only, create the stream first: maprcli stream create -path /user/mapr/taq -produceperm p -consumeperm p -topicperm p -compression off mvn -e -Dtest= ThreadCountSpeedTest test # Test data will be saved in thread-count.csv </code></pre> <h2>Partition Scalability</h2> <p>Stream topics are often subdivided into partitions in order to allow multiple consumers to read from a topic simultaneously. Both Kafka and MapR Event Store allow topics to be partitioned, but partitions in MapR Event Store are much more powerful and easier to manage than partitions in Kafka. For example, Kakfa requires partitions to fit within the disk space of a single cluster node and cannot be split across machines. MapR Event Store is not limited by the storage capacity of any one node because the MapR storage system automatically grows (or shrinks) partitions across servers. I’ll talk more about these operational advantages later, but let’s consider the performance implications of partitioning now.</p> <p>ZooKeeper elects separate nodes to be leaders for each partition. Leaders are responsible for processing the client reads and writes for their designated partition. This helps load balance client requests across the cluster, but it complicates the work the ZooKeeper must do to keep topics synchronized and replicated. Leader election takes time and does not scale well. In my tests, I saw leader election take at least 0.1 seconds per partition and it ran serially. So, for example, it would take more than 10 seconds to configure a topic with 100 partitions, that is, if ZooKeeper didn’t crash, which it frequently did when I created topics with 100 or more partitions.</p> <p>In MapR Event Store, I had no problem streaming data to topics with thousands of partitions, as shown below. This graph shows the throughput for a producer sending synchronously to a 3x replicated topic subdivided into an increasingly large number of partitions. I could not run my test in Kafka beyond 400 partitions, so that line is cut short.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/streams-tput-partitions-1605077753761.png" alt="Partitioning Scalability"></p> <h2>Replicating this test</h2> <p>I used the performance scripts included with Kafka and MapR to generate the partition vs. throughput data shown above. Here is the script I used to run this test in Kafka:</p> <p><a target='\_blank' href='https://gist.github.com/iandow/625d783333a53b592f0381e6b37ee9ab'><a href="https://gist.github.com/iandow/625d783333a53b592f0381e6b37ee9ab">https://gist.github.com/iandow/625d783333a53b592f0381e6b37ee9ab</a></a></p> <p>That script will silently freeze if ZooKeeper fails, but it will continue once ZooKeeper starts again. So in another terminal, I simultaneously ran the following script to automatically restart ZooKeeper if it fails (which it is likely to do during this test):</p> <p><a target='\_blank' href='https://gist.github.com/iandow/2dc07bde132669706467e8ee45507561'><a href="https://gist.github.com/iandow/2dc07bde132669706467e8ee45507561">https://gist.github.com/iandow/2dc07bde132669706467e8ee45507561</a></a></p> <p>Here is the script I used to generate partitions vs. throughput data in MapR:</p> <p><a target='\_blank' href='https://gist.github.com/iandow/8074962f6205552c9cdc3fceccdd9793'><a href="https://gist.github.com/iandow/8074962f6205552c9cdc3fceccdd9793">https://gist.github.com/iandow/8074962f6205552c9cdc3fceccdd9793</a></a></p> <h2>Operational Advantages for MapR Event Store</h2> <p>Increasing throughput capacity and decreasing message latency can often be accomplished simply by adding nodes to your distributed messaging cluster. However, doing so costs money and complicates management, so essentially saying that MapR Event Store performs better than Kafka is another way of saying that operating a distributed messaging platform can be done with less hardware on MapR than with Kafka.</p> <p>However, unless you’re working on applications that scale to extreme lengths, then the challenges you face with Kafka are more likely to be operational rather than performance in nature. And this is where the MapR total cost of ownership really shines.</p> <p>Not only does MapR Event Store execute with higher performance, it also addresses major operational deficiencies in Kafka. Here are three examples relating to replication, scaling, and mirroring:</p> <ul> <li> <p>Kafka requires that the MirrorMaker processes be manually configured in order to replicate across clusters. Replication is easy to configure with MapR Event Store and supports unique capabilities for replicating streams across data centers and allowing streams to be updated in multiple locations at the same time.</p> </li> <li> <p>Kafka’s mirroring design simply forwards messages to a mirror cluster. The offsets in the source cluster are useless in the mirror, which means consumers and producers cannot automatically failover from one cluster to a mirror. MapR continuously transfers updated records for near real-time replication and preserves message offsets in all replicated copies.</p> </li> <li> <p>Kakfa requires partitions to fit within the disk space of a single cluster node and cannot be split across machines. This is especially risky, because ZooKeeper could automatically assign multiple large partitions to a node that doesn’t have space for them. You can move them manually, but that can quickly become unmanageable. MapR Event Store is not limited by the storage capacity of any one node because it distributes stream data across the cluster.</p> </li> </ul> <h2>Conclusion</h2> <p>MapR Event Store outperforms Kafka in big ways. I measured the performance of distributed streaming in a variety of cases that focused on the effects of message size and topic quantity, and I saw MapR Event Store transport a much faster stream of data, with much larger message sizes, and to far more topics than what could be achieved with Kafka on a similarly sized cluster. Although performance isn’t the only thing that makes MapR Event Store desirable over Kafka, it offers one compelling reason to consider it.</p>https://developer.hpe.com/configure-jupyter-notebook-for-spark-210-and-python/https://developer.hpe.com/configure-jupyter-notebook-for-spark-210-and-python/Thu, 05 Nov 2020 17:04:32 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Mathieu Dumoulin", "publish": "2017-07-07T12:00:00.000", "tags": "apache-spark" </code></pre> <hr> <p>I'll guess that many people reading this have spent time wrestling with a configuration to get Python and Spark to play nicely. Having gone through the process myself, I've documented my steps and will share my knowledge, hoping it will save some time and frustration for some of you.</p> <p>This article targets the latest releases of MapR 5.2.1 and the MEP 3.0 version of Spark 2.1.0. It should work equally well for earlier releases of MapR 5.0 and 5.1. In fact, I've tested this to work with MapR 5.0 with MEP 1.1.2 (Spark 1.6.1) for a customer.</p> <p>The version of Jupyter is 4.3. It seems like it changed quite a bit since the earlier versions and so most of the information I found in blogs were pretty outdated. Hence my having so much trouble getting everything working to my satisfaction.</p> <p>My goals:</p> <ul> <li>Run PySpark successfully from either a cluster node or an edge node</li> <li>Run python code in YARN distributed mode</li> <li>Have access to modules like numpy, scipy, pandas and others.</li> <li>Do all this using Jupyter in server mode that I access from my own laptop</li> </ul> <p>I'm leaving out Jupyter server mode security, which could be the topic of a future blog, potentially. I've implemented it before and found <a href="https://jupyter-notebook.readthedocs.io/en/latest/security.html">THE JUPYTER DOCUMENTATION</a> explains setting it up for encryption (HTTPS) and authentication to be pretty good.</p> <h2>Installing Python</h2> <p>Verify your version of Python:</p> <pre><code class="language-python">python --version </code></pre> <p>If it's Python 2.6.X, it's probably a good idea to use a recent build of Python 2.7 If it's Python 2.7.X, then you'll need to choose to use the system python or not.</p> <ul> <li>System python is easier to make work, it's already there and shared everywhere.</li> <li>Isolated separate python (anaconda or a separate python) is harder to get working but will provide a more consistent environment where each user can have their own (and only their own) modules installed.</li> </ul> <p>I will use Miniconda for Python 2.7 64 bits throughout. It works very well. Using Python 3 would be just the same, with the only difference being in terms of code and module compatibility. Either will work fine with Spark.</p> <p>Note: Python 3.6 doesn't work with Spark 1.6.1 See <a href="https://issues.apache.org/jira/browse/SPARK-19019">SPARK-19019</a>.</p> <h2>Installing Anaconda</h2> <p>There is a choice between Anaconda and Miniconda, as well as between python 2.7 and Python 3.6.</p> <p>Miniconda is very nice because the download is small and you only install what you need. Anaconda is very nice for having everything installed from the start, so all needed modules will be there from the start for most needs.</p> <p>Here, we show installing miniconda and Python 2.7 (64bits):</p> <pre><code class="language-bash">wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh bash Miniconda2-latest-Linux-x86_64.sh -b -p /opt/miniconda2 </code></pre> <p>To install it on all nodes at once, we recommend to check out Clustershell.</p> <pre><code class="language-bash">#copy the file to all nodes clush -ac Miniconda2-latest-Linux-x86_64.sh #install on all nodes at same time: clush -aB bash Miniconda2-latest-Linux-x86_64.sh -b -p /opt/miniconda2 </code></pre> <p><strong>Important</strong>: Get all nodes in same exact state, with python/anaconda installed <strong>exactly</strong> in the same location with all nodes having exactly the same modules installed. Miss that here and it guarantees weird errors that will be hard to diagnose.</p> <h2>Update Spark environment to use Python 2.7:</h2> <p>Add to <code>/opt/mapr/spark/spark-2.1.0/conf/spark-env.sh</code>:</p> <pre><code class="language-bash"> export PYSPARK_PYTHON=/opt/miniconda2/bin/python export PYSPARK_DRIVER_PYTHON=/opt/miniconda2/bin/python </code></pre> <p>Update file on all nodes:</p> <pre><code class="language-bash"># using clustershell to copy file ("c") to all nodes ("a") clush -ac /opt/mapr/spark/spark-2.1.0/conf/spark-env.sh </code></pre> <blockquote> <p>Note: this is known to work on previous MEP versions. I have also tested it with MEP 1.1.2 (Spark 1.6.1) and it worked very well. just use the correct path to Spark and it will work just fine.</p> </blockquote> <h2>Testing</h2> <p>For testing, let's use some Ebay auction data.</p> <p>Copy the data into the folder: <code>/user/mapr/data</code></p> <p>Start pyspark and run the following code:</p> <pre><code class="language-markdown">>>> auctionRDD = sc.textFile("/user/mapr/data/auctiondata.csv").map(lambda line:line.split(",")) >>> auctionRDD.first() [u'8213034705', u'95', u'2.927373', u'jake7870', u'0', u'95', u'117.5', u'xbox', u'3'] >>> auctionRDD.count() 10654 </code></pre> <p>Ok, so now we have a working pyspark shell!</p> <blockquote> <p>Note: Don't do this as <code>root</code> or as user <code>MAPR</code> on a production cluster. However, for tutorials, user <code>MAPR</code> is convenient as it is a superuser and you don't need to worry about file permissions on MapR.</p> </blockquote> <p>Errors:</p> <ul> <li><em>pyspark java.io.IOException: Cannot run program "python2.7": error=2, No such file or directory</em></li> </ul> <p>This error is because the driver and/or the executors can't find the python executable. It's fixed by setting the PYSPARK_PYTHON (and PYSPARK_DRIVER_PYTHON) variables in <code>spark-env.sh</code> (see above)</p> <h2>ipython Notebook</h2> <p>If you want to able to choose to use spark when launch ipython shell:</p> <ol> <li>Ensure SPARK_HOME env variable is defined.</li> </ol> <pre><code class="language-bash">export SPARK_HOME=/opt/mapr/spark/spark-2.1.0 </code></pre> <ol start="2"> <li>Install ipython with Anaconda</li> </ol> <pre><code class="language-bash">/opt/miniconda2/bin/conda install jupyter </code></pre> <ol start="3"> <li>Add a ipython profile named pyspark</li> </ol> <pre><code class="language-bash">ipython profile create pyspark </code></pre> <ol start="4"> <li>Add <code>~/.ipython/profile_pyspark/startup/00-pyspark-setup.py</code>:</li> </ol> <pre><code class="language-python">import os import system spark_home = os.environ.get('SPARK_HOME', None) if not spark_home: raise ValueError('SPARK_HOME environment variable is not set') sys.path.insert(0, os.path.join(spark_home, 'python')) sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.4-src.zip')) exec(open(os.path.join(spark_home, 'python/pyspark/shell.py')).read()) </code></pre> <ol start="5"> <li>Launch</li> </ol> <pre><code class="language-bash">/opt/miniconda2/bin/ipython --profile=pyspark </code></pre> <p>Try the sample code above. It should also work without issue.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/ipythonsamplecode-1604596105535.png" alt=""></p> <h2>Jupyter Notebook</h2> <p>Now on to Jupyter. In this case, we're looking to have the notebook run on an edge node (less ideally, on a cluster node) in server mode and access it from our development laptop.</p> <p>The following instructions assume the user <code>MAPR</code>, but should work equally well for any other user. For production use, never use <code>MAPR</code> user as it is a superuser with read-write access to all data in MapR.</p> <p>With Anaconda:</p> <pre><code class="language-bash">clush -aB /opt/miniconda2/bin/conda install jupyter -y </code></pre> <ol> <li>Allow remote login to a notebook:</li> </ol> <p>Generate a profile:</p> <pre><code class="language-bash"> /opt/miniconda2/bin/jupyter notebook --generate-config </code></pre> <pre><code>This generates the following file: </code></pre> <p><code>$HOME/.jupyter/jupyter_notebook_config.py</code> In this file, we're going to update the following setting: <code>c.NotebookApp.ip</code></p> <p>The setting for this is “c.notebookapp.ip”. The default value is ‘localhost’, meaning you can’t login remotely. Setting it to ‘*’ means it is accessible from anywhere, which is fine for development, but not so good for production.</p> <pre><code class="language-bash"> # c.NotebookApp.ip = '*' </code></pre> <p><strong>About Security:</strong></p> <pre><code>It's a good time to remind you about security. It's pretty easy to configure Jupyter to use https and have a password. See [Jupyter documentation](https://jupyter-notebook.readthedocs.io/en/latest/public_server.html#securing-a-notebook-server). Basic auth with https are a reasonable minimum security level for a production system. Important: the user which runs the notebook is meaningful, as that user’s permissions are used to access files on MapR. If you run it as user mapr, then everything is accessible as it’s a superuser account. For production, you want to run it as a less privileged user. </code></pre> <ol start="2"> <li>The startup script from the ipython step is helpful:</li> </ol> <pre><code class="language-bash"> [mapr@ip-10-0-0-180 ~]$ cat .ipython/profile_default/startup/00-default-setup.py import os import sys spark_home = os.environ.get('SPARK_HOME', None) if not spark_home: raise ValueError('SPARK_HOME environment variable is not set') sys.path.insert(0, os.path.join(spark_home, 'python')) sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.4-src.zip')) execfile(os.path.join(spark_home, 'python/pyspark/shell.py')) </code></pre> <p>This step is essential or else the kernel won't initialize properly. Alternatively, you can past the code above in the first cell to initialize pyspark first. Another alternative is to use the module <code>findspark</code>, which probably does something similar to this, but with less code.</p> <ol start="3"> <li>Add a PySpark Kernel. create the <code>kernel.json</code> file in the location as shown below:</li> </ol> <pre><code class="language-bash">[mapr@ip-10-0-0-20 ~]$ cat .ipython/kernels/pyspark/kernel.json { "display_name": "pySpark (Spark 2.1.0)", "language": "python", "argv": [ "/opt/miniconda2/bin/python", "-m", "ipykernel", "-f", "{connection_file}" ], "env": { "CAPTURE_STANDARD_OUT": "true", "CAPTURE_STANDARD_ERR": "true", "SEND_EMPTY_OUTPUT": "false", "SPARK_HOME": "/opt/mapr/spark/spark-2.1.0" } } </code></pre> <ol start="4"> <li>Start a notebook and have fun with Spark and Python!</li> </ol> <pre><code class="language-bash"> jupyter notebook --no-browser </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/jupyternotebook-nobrowser-1604596122293.png" alt=""></p> <p>Open your browser to the indicated link and... Success!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/browsersuccess-1604596151125.png" alt=""></p> <h2>Launch Jupyter notebook instead of pyspark</h2> <ol> <li>Update <code>$SPARK_HOME/conf/spark-env.sh</code>:</li> </ol> <pre><code class="language-bash"> [mapr@ip-10-0-0-20 ~]$ tail /opt/mapr/spark/spark-2.1.0/conf/spark-env.sh export PYSPARK_DRIVER_PYTHON=/opt/miniconda2/bin/jupyter # Setup env variable for Jupyter + Pyspark export PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser" </code></pre> <ol start="2"> <li>Launch pyspark, it will launch a Jupyter notebook</li> </ol> <pre><code class="language-bash"> $SPARK_HOME/bin/pyspark </code></pre> <h2>Thanks</h2> <p><a href="https://mengdong.github.io/2016/08/08/fully-armed-pyspark-with-ipython-and-jupyter/">Dong Meng's blog</a> proved to be a life saver. Check him out: <a href="https://mengdong.github.io">https://mengdong.github.io</a></p>https://developer.hpe.com/performance-tuning-of-an-apache-kafkaspark-streaming-system/https://developer.hpe.com/performance-tuning-of-an-apache-kafkaspark-streaming-system/Thu, 05 Nov 2020 16:39:38 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Mathieu Dumoulin", "publish": "2017-01-17T06:00:00.000Z", "tags": "apache-hive" </code></pre> <hr> <p><em>Real-world case study in the telecom industry</em></p> <p>Debugging a real-life distributed application can be a pretty daunting task. Most common Google searches don't turn out to be very useful, at least at first. In this blog post, I will give a fairly detailed account of how we managed to accelerate by almost 10x an Apache Kafka/Spark Streaming/Apache Ignite application and turn a development prototype into a useful, stable streaming application that eventually exceeded the performance goals set for the application.</p> <p>The lessons learned here are fairly general and extend easily to similar systems using MapR Event Store as well as Kafka.</p> <p>This project serves as a concrete case for the need of a converged platform, which integrates the full software stack to support the requirements of this system: real-time streams and big data distributed processing and persistence. The MapR Data Platform is the only currently available production-ready implementation of such a platform as of this writing.</p> <h2>Goal of the system</h2> <p>To meet the needs of the telecom company, the goal of the application is to join together the log data from three separate systems. When the data is joined, it becomes possible to correlate the network conditions to a particular call for any particular customer, thus allowing customer support to provide accurate and useful information to customers who are unsatisfied with their phone service. The application has great additional value if it can do this work in real time rather than as a batch job, since call quality information that is 6 hours old has no real value for customer service or network operations.</p> <p>Basically, this is a fairly straight-up ETL job that would normally be done as a batch job for a data warehouse but now has to be done in real time as a streaming distributed architecture.</p> <p>More concretely, the overall picture is to stream the input data from a remote server into a distributed cluster, do some data cleaning and augmentation, join the records from the three logs, and persist the joined data as a single table into a database.</p> <h2>The problems with the original system</h2> <p>The original system had several issues centered around performance and stability.</p> <p>First, the streaming application was not stable. In a Spark Streaming application, the stream is said to be stable if the processing time of each microbatch is equal to or less than the batch time. In this case, the streaming part of the application was receiving data in 30 second windows but was taking between 4.5-6 minutes to process.</p> <p>Second, there is a batch process to join data one hour at a time that was targeted to run in 30 minutes but was taking over 2 hours to complete.</p> <p>Third, the application was randomly crashing after running for a few hours.</p> <h2>The cluster hardware, software stack, and input data</h2> <p>The cluster hardware is pretty good, with 12 nodes of enterprise servers, each equipped with two E5 Xeon CPUs each with 16 physical cores, 256GB memory, and eight 6TB spinning HDD. The network is 10GB Ethernet.</p> <p>The technology stack selected for this project is centered around Kafka 0.8 for streaming the data into the system, Apache Spark 1.6 for the ETL operations (essentially a bit of filter and transformation of the input, then a join), and the use of <a target='\_blank' href='https://ignite.apache.org/'>Apache Ignite 1.6</a> as an in-memory shared cache to make it easy to connect the streaming input part of the application with joining the data. Apache Hive is also used to serve as a disk backup for Ignite in case of failure and for separate analytics application.</p> <p>The initial cluster was configured as follows:</p> <table> <thead> <tr> <th align="center"><strong>Node</strong></th> <th align="center"><strong>Zk</strong></th> <th align="center"><strong>NN</strong></th> <th align="center"><strong>HDFS</strong></th> <th align="center"><strong>Mesos</strong></th> <th align="center"><strong>Mesos Master</strong></th> <th align="center"><strong>Kafka</strong></th> <th align="center"><strong>Spark Worker</strong></th> <th align="center"><strong>Ignite</strong></th> </tr> </thead> <tbody> <tr> <td align="center">1</td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">2</td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">3</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">...</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">7</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">8</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">...</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> </tr> <tr> <td align="center">12</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> <td align="center"></td> <td align="center"></td> <td align="center">x</td> <td align="center">x</td> </tr> </tbody> </table> <p>The cluster is running Apache Hadoop's HDFS as a distributed storage layer, with resources managed by Mesos 0.28. Finally, HBase is used as the ultimate data store for the final joined data. It will be queried by other systems outside the scope of this project.</p> <p>The performance requirement of the system is to handle an input throughput of up to 3GB/min, or 150-200,000 events/second, representing the known peak data throughput, plus an additional margin. The ordinary throughput is about half of that value or 1.5GB/min and 60,000-80,000 events/second.</p> <p>The raw data source are the logs of three remote systems, labeled A, B, and C here: Log A comprises about 84-85% of the entries, Log B about 1-2%, and Log C about 14-15%. The fact that the data is unbalanced is one of the (many) sources of difficulty in this application.</p> <p>The Spark applications are both coded in Scala 2.10 and <a target='\_blank' href='http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers'>Kafka’s direct approach</a> (no receivers). Apache Ignite has a really nice Scala API with a magic <a target='\_blank' href='https://ignite.apache.org/features/igniterdd.html'>IgniteRDD</a> that can allow applications to share in-memory data, a key feature for this system to reduce coding complexity.</p> <h2>The application architecture</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture2-1604595435073.png" alt=""></p> <p>The raw data is ingested into the system by a single Kafka producer into Kafka running on 6 servers. The producer reads the various logs and adds each log's records into its own topic. As there are three logs, there are three Kafka topics. Each topic is split into 36 partitions. Most likely, there are 36 partitions because there are 6 nodes with each 6 disks assigned to HDFS, and Kafka documentation seems to recommend having about one partition per physical disk as a guideline.</p> <p>The data is consumed by a Spark Streaming application which picks up each topic and then does a simple filter to cut out unnecessary fields, a map operation to transform the data, and a foreachRDD operation (each micro-batch generates an RDD in Spark Streaming) that saves the data to Ignite and to Hive.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture3-1604595444583.png" alt=""></p> <p><em>The streaming app is very straightforward: map, filter, and foreach partition to save to Ignite</em></p> <p>A second "regular" Spark application runs on the data stored in-memory by Ignite to join the records from the three separate logs into a single table in batches of 1 hour. This job is done using Spark's DataFrame API, which is ideally suited to the task. The second part involves no more than 100GB worth of data, and the cluster hardware is properly sized to handle that amount of data.</p> <p>Three hours of data are accumulated into Ignite, because the vast majority of calls last for less than an hour, and we want to run the join on one hour’s worth of data at a time. Since some calls will start in one batch and finish in another, the system keeps three hours and only processes the middle one-hour batch, thus the join can succeed on close to 100% of the records.</p> <p>It’s worth noting that a better all-streaming architecture could have avoided the whole issue with the intermediate representation in the first place. An illustrative, real-world case, built with more time and thought up-front, can end the entire project faster, as opposed to rushing headlong into coding the first working solution that comes to mind.</p> <h2>Performance tuning</h2> <p>The main issues for these applications were caused by trying to run a development system's code, tested on AWS instances on a physical, on-premise cluster running on real data. The original developer was never given access to the production cluster or the real data.</p> <p>Apache Ignite was a huge source of problems, principally because it is such a new project that nobody had any real experience with it and also because it is not a very mature project yet.</p> <h2><strong>First target: Improve Spark Streaming performance</strong></h2> <p>The Spark Streaming application was running in about 4.5 minutes, and the project goal was to run in about 30 seconds. We needed to find 9x speedup worth of improvements, and due to time constraints, we couldn’t afford to change any code!</p> <p>The system had to be ready for production testing within a week, so the code from the architecture and algorithm point of view was assumed to be correct and good enough that we could reach the performance requirement only with tuning.</p> <p><strong>Fix RPC timeout exceptions</strong></p> <p>We found the correct solution from somebody having the same problem, as seen in <a target='\_blank' href='https://issues.apache.org/jira/browse/SPARK-14140'>SPARK-14140 in JIRA</a>. They recommend increasing the spark.executor.heartbeatInterval from 10s to 20s.</p> <p>I think this problem may be caused by nodes getting busy from disk or CPU spikes because of Kafka, Ignite, or garbage collector pauses. Since Spark runs on all nodes, the issue was random. (See the cluster services layout table in the first section.)</p> <p>The configuration change fixed this issue completely. We haven’t seen it happen since.</p> <p><strong>Increase driver and executor memory</strong></p> <p>Out of memory issues and random crashes of the application were solved by increasing the memory from 20g per executor to 40g per executor as well as 40g for the driver. Happily, the machines in the production cluster were heavily provisioned with memory. This is a good practice with a new application, since you don’t know how much you will need at first.</p> <p>The issue was difficult to debug with precision, lacking accurate information, since the Spark UI reports very little memory consumption. In practice, as this setting is easy to change, we empirically settled on 40g being the smallest memory size for the application to run stably.</p> <p><strong>Increase parallelism: increase number of partitions in Kafka</strong></p> <p>The input data was unbalanced, and most of the application processing time was spent processing Topic 1 (with 85% of the throughput). Kafka partitions are matched 1:1 with the number of partitions in the input RDD, leading to only 36 partitions, meaning we can only keep 36 cores busy on this task. To increase the parallelism, we need to increase the number of partitions. So we split topic 1 into 12 topics each, with 6 partitions, for a total of 72 partitions. We did a simple modification to the producer to divide the data evenly from the first log into 12 topics, instead of just one. Zero code needed to be modified on the consumer side.</p> <p>We also right-sized the number of partitions for the two other topics, in proportion to their relative importance in the input data, so we set topic 2 to 2 partitions and topic 3 to 8 partitions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture4-1604595452674.png" alt=""></p> <p><em>Running more tasks in parallel. Before tuning, each stage always had 36 partitions!</em></p> <p><strong>Right-size the executors</strong></p> <p>The original application was running only 3 executors with 72 total cores. We configured the application to run with 80 cores at a maximum of 10 cores per executor, for a total of 8 executors. Note that with 16 real cores per node on a 10-node cluster, we’re leaving plenty of resources for Kafka brokers, Ignite, and HDFS/NN to run on.</p> <p><strong>Increase the batch window from 30s to 1m</strong></p> <p>The data is pushed into Kafka by the producer as batches every 30s, as it is gathered by FTP batches from the remote systems. Such an arrangement is common in telecom applications due to a need to deal with equipment and systems from a bewildering range of manufacturers, technology, and ages.</p> <p>This meant that the input stream was very lumpy, as shown in the screenshot of Spark UI's Streaming tab:</p> <p>Increasing the window to 1m allowed us to smooth out the input and gave the system a chance to process the data in 1 minute or less and still be stable.</p> <p>To make sure of it, the team generated a test data, which simulated the known worst-case data, and with the new settings, the spark-streaming job was now indeed stable. The team was also able to switch easily between test data and the real production data stream as well as a throttle on the producers to configure how much data to let in to the system. This was extremely helpful to test various configurations quickly and see if we had made progress or not.</p> <p><strong>Drop requirement to save to Hive, only use Ignite</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/picture5-1604595575920.png" alt=""></p> <p>Discussion with the project managers revealed that Hive was not actually part of the requirements for the streaming application! Mainly, this is because the data in HBase could just as easily be used by the analytics; also, in the context of this application, each individual record doesn't actually need to be processed with a 100% guarantee.</p> <p>Indeed, in light of the goal of the system, the worse-case scenario for missing data is that a customer's call quality information cannot be found... which is already the case. In other words, the risk of data loss is not a deal-breaker, and the upside to gaining data is additional insights. As long as the great majority of the data is processed and stored, the business goals can be reached.</p> <p><strong>Results of all optimizations</strong></p> <p>The streaming application finally became stable, with an optimized runtime of 30-35s.</p> <p>As it turns out, cutting out Hive also sped up the second Spark application that joins the data together, so that it now ran in 35m, which meant that both applications were now well within the project requirements.</p> <p>With improvements from the next part, the final performance of the Spark Streaming job went down in the low 20s range, for a final speedup of a bit over 12 times.</p> <h2><strong>Second target: Improve System Stability</strong></h2> <p>We had to work quite hard on stability. Several strategies were required, as we will explain below.</p> <p><strong>Make the Spark Streaming application stable</strong></p> <p>The work we did to fix the performance had a direct impact on system stability. If both applications are stable themselves and running on right-sized resources, then the system has the best chance to be stable overall.</p> <p><strong>Remove Mesos and use Spark Standalone</strong></p> <p>The initial choice of Mesos to manage resources was forward-looking, but ultimately we decided to drop it from the final production system. At the onset, the plan was to have Mesos manage all the applications. But the team never could get Kafka and Ignite to play nice with Mesos, and so  they were running in standalone mode, leaving only Spark to be managed by Mesos. Surely, with more time, there is little doubt all applications could be properly configured to work with Mesos.</p> <p>Proposing to remove Mesos was a bit controversial, as Mesos is much more advanced and cool than Spark running in standalone mode.</p> <p>But the issue with Mesos was twofold:</p> <ol> <li>Control over executor size and number was poor, a known issue (<a target='\_blank' href='https://issues.apache.org/jira/browse/SPARK-5095'>SPARK-5095</a>) with Spark 1.6 and fixed in Spark 2.0.</li> <li>Ignite and Kafka weren’t running inside Mesos, just Spark. Because of schedule pressure, the team had given up on trying to get those two services running in Mesos.</li> </ol> <p>Mesos can only ever allocate resources well if it actually controls resources. In the case of this system, Kafka and Ignite are running outside of Mesos’ knowledge, meaning it’s going to assign resources to the Spark applications incorrectly.</p> <p>In addition, it’s a single-purpose cluster, so we can live with customizing the sizing of the resources for each application with a global view of the system’s resources. There is little need for dynamic resource allocations, scheduling queues, multi-tenancy, and other buzzwords.</p> <p><strong>Change the Ignite memory model</strong></p> <p>It is a known issue that when the heap controlled by the JVM gets very big (>32GB), the cost of garbage collection is quite large. We could indeed see this problem when the join application runs: the stages with 25GB shuffle had some rows with spikes in GC time, ranging from 10 seconds up to more than a minute.</p> <p>The initial configuration of Ignite was to run ONHEAP_TIERED with 48GB worth of data cached on heap, then overflow drops to 12GB of off-heap memory. That setting was changed to the OFFHEAP_TIERED model. While slightly slower due to serialization cost, OFFHEAP_TIERED doesn't result in big garbage collections. It still runs in memory, so we estimated it would be a net gain.</p> <p>With this change, the run time for each batch dutifully came down by about five seconds, from 30 seconds down to about 25 seconds. In addition, successive batches tended to have much more similar processing time with a delta of 1-3 seconds, whereas it would previously vary by over 5 to 10 seconds.</p> <p><strong>Update the Ignite JVM settings</strong></p> <p>We followed the recommended JVM options as found in Ignite documentation’s performance tuning section (<a target="_blank" href="https://apacheignite.readme.io/docs/jvm-and-system-tuning"><a href="https://apacheignite.readme.io/docs/jvm-and-system-tuning">https://apacheignite.readme.io/docs/jvm-and-system-tuning</a></a>).</p> <p><strong>Improve the Spark code</strong></p> <p>Some parts of the code assumed reliability, like queries to Ignite, when in fact there was a possibility of the operations failing. These problems can be fixed in the code, which now handles exceptions more gracefully, though there is probably work left to increase the robustness of the code. We can only find these spots by letting the application run now.</p> <p><strong>Reassign ZooKeeper to nodes 10-12</strong></p> <p>Given that the cluster is medium-sized, it’s worth spreading the services as much as possible. We moved the ZooKeeper services from nodes 1-3 to nodes 10-12.</p> <h2>Conclusion</h2> <p>Tuning this application took about 1 week of full-time work. The main information we used was Spark UI and Spark logs, easily accessible from the Spark UI. The view of Jobs and Stages as well as the streaming UI are really very useful.</p> <h2>What I learned</h2> <ul> <li>Migrating a streaming application from a prototype on AWS to an on-premise cluster requires schedule time for testing</li> <li>Not testing the AWS prototype with realistic data was a big mistake</li> <li>Including many “bleeding-edge” OSS components (Apache Ignite and Mesos) with expectations of very high reliability is unrealistic</li> <li>A better architecture design could have simplified the system tremendously</li> <li>Tuning a Kafka/Spark Streaming application requires a holistic understanding of the entire system. It’s not simply about changing the parameter values of Spark; it’s a combination of the data flow characteristics, the application goals and value to the customer, the hardware and services, the application code, and then playing with Spark parameters.</li> <li>MapR Data Platform would have cut the development time, complexity, and cost for this project.</li> </ul> <p>The project is a first for this particular telecom company, and they decided to go all-out on such an advanced, 100% open-source platform. They should be applauded for their pioneering spirit. But a better choice of platform and application architecture would have made their lives a lot easier.</p> <h2>The need for a converged big-data platform is now</h2> <p>In fact, the requirements for this project show the real-world business need for a state-of-the-art converged platform with a fast distributed files system, high-performance key-value store for persistence, and real-time streaming capabilities.</p> <p>A MapR solution could probably skip the requirement for a still speculative open-source project like Ignite, since the full software stack required by the architecture is already built-in and fully supported. Given this system is heading into production for a telecom operator with 24/7 reliability expectation, such an advantage is considerable.</p>https://developer.hpe.com/data-fabric-the-future-of-data-management/https://developer.hpe.com/data-fabric-the-future-of-data-management/Thu, 05 Nov 2020 16:35:05 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Karen Whipple", "publish": "2018-08-08T07:00:00.000", "tags": "" </code></pre> <hr> <p>"Data fabric" is a relatively new term, but its importance is not. For years, enterprises have struggled to integrate all their data into a single, scalable platform. A data fabric simply describes a comprehensive way to achieve that goal.</p> <p>And, with <a href="https://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm">data growing at a rate of 40 percent per year</a> (and not showing any signs of slowing down), achieving that goal is more important than ever.</p> <p>Let's learn more about what a data fabric is, why you should care, and how MapR is offering a data fabric that's more advanced than any other.</p> <h2>What Is Data Fabric?</h2> <p>As companies expand, use an increasing number of applications, and collect data into more and more silos, their need for a better data solution grows exponentially.</p> <p>When data is isolated and relegated to silos, it becomes stagnant and inaccessible. Legacy systems only make it more difficult to access data, and the result is lowered productivity and efficiency.</p> <p>Plus, it's not only silos that separate data — even state-of-the-art technologies like data centers and the cloud can inadvertently work to divide data into inconvenient clusters.</p> <p>The cloud is particularly troublesome because it makes it difficult for companies to leverage data from public clouds as well as that which is stored on-site.</p> <p>EnterpriseTech <a href="https://enterprisetech.com/2017/10/02/global-data-fabric-takes-diversity-data-types/">provides an excellent illustration</a> of this problem:</p> <blockquote> <p>"For example, you might have an analyst in Cleveland who needs to run a query on an exception report generated by a robotic arm in Munich, Germany. To detect anomalies, the data from that robotic arm needs to be consistent with your data warehouse. With global strong consistency, your data is consistent between the device – in this case the robotic arm – and the data center. The data fabric ensures this data is available and consistent regardless of where it originates and where it is accessed."</p> </blockquote> <p>Just like an actual piece of fabric, a data fabric can be placed over a wide expanse of space. That's the main idea behind data fabrics, which can encompass a wide variety of data locations and sources.</p> <p>What a data fabric does is enable the processing, management, analysis, and storage of almost any amount of data from a multitude of sources. The data fabric then enables applications and tools to access that data using an array of interfaces. It's also important to note that data fabrics leverage data in real time.  </p> <h2>Why Should You Care?</h2> <p>As we mentioned above, enterprises to whom data is important often find that their current systems are too slow, too divided, and inefficient.</p> <p>And, in larger companies, having multiple groups of people store and retrieve data in their own way results in a fragmented network of data that's far from unified.</p> <p>Unlike other solutions, a data fabric is not a band-aid fix. Rather, it's a permanent and scalable way to bring all your data under the umbrella of one unified platform.</p> <p>A data fabric is able to solve a number of problems, such as:</p> <ol> <li>Low data availability</li> <li>Little to no reliability in terms of storage and security</li> <li>Siloed data</li> <li>Poor scalability that's unable to adapt to different amounts of data</li> <li>Reliance on underperforming legacy systems</li> </ol> <h2>What Makes MapR's Data Fabric Unique?</h2> <p>MapR's Vice President of Technology Strategy, Crystal Valentine, puts it best in <a href="https://forbes.com/sites/danwoods/2017/11/14/mapr-why-data-fabric-is-now-vital-to-the-app-stack">an interview with Forbes</a>:</p> <blockquote> <p>"From our founding, MapR has had a singular focus on building a platform with the capabilities to support next-gen applications. Our vision of a future in which applications can leverage large amounts of distributed data in real time to provide enormous competitive advantages is quickly becoming reality, and our strong growth numbers are testament to the fact that that is resonating with our customers."</p> </blockquote> <p>Although other data fabric solutions are available, most focus on more basic functions, such as:</p> <ul> <li>Storing and retrieving data from various sources</li> <li>Accessing those sources, using one standard method or application program interface (API)</li> <li>Integrating data both within and across sources</li> <li>Applying processing or real-time analytics to data from any source</li> <li>Syncing data with cloud storage</li> <li>Providing backup and disaster recovery (DR) support</li> </ul> <p>MapR's data fabric goes above and beyond by allowing you to:</p> <ul> <li>Unify data silos</li> <li>Deploy anywhere</li> <li>Scale up</li> <li>Replicate data across data centers, edge devices, and cloud instances</li> <li>Search and explore data with Apache Drill</li> <li>Safely secure your data with a single security model with granular controls</li> <li>Easily move exabytes of data across edge, on-premises, and cloud deployments</li> <li>Extend IoT applications to remote locations</li> <li>Increase your data transfer speed</li> <li>Make faster decisions</li> <li>Effortlessly access data.</li> <li>Reduce both operational and capital expenses</li> </ul> <p>Plus, our roster of high-profile clients, who are winning at data with the MapR Data Fabric, speaks for itself. Our customers include Hewlett-Packard, SAP SE, NTT Security, Mediahub, and more.</p> <p>In <a href="https://dzone.com/articles/the-rise-of-the-data-fabric">an article for DZone</a>, Dan Kusnetzky says:</p> <blockquote> <p>"In my industry research, I've come across a number of definitions of the data fabric notion. So far, the one being put forward by MapR Technologies seems to be the most comprehensive. Furthermore, the company has been enjoying some level of success as it persuades customers to adopt its approach. The company appears to understand the importance of distributed processing in the era of mobile, cloud, and the fast proliferation of IoT and how all of these are important members of enterprise computing applications. They also appear to understand that these devices also represent potential threats and have discussed how to bring them in safely. The company has also considered where and how data should be stored so that it is available broadly, but still secure and reliable."</p> </blockquote> <p>Essentially, MapR goes beyond the basic requirements of a data fabric to provide our customers with a truly revolutionary product that makes data analysis, storage, and security fast, reliable, and easy.</p>https://developer.hpe.com/big-data-opportunities-for-telecommunications/https://developer.hpe.com/big-data-opportunities-for-telecommunications/Thu, 05 Nov 2020 16:27:29 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2017-05-09T00:00:00.000Z", "tags": "apache-spark" </code></pre> <hr> <p><em>The telecommunications industry is on the verge of a major transformation through the use of advanced analytics and big data technologies like the MapR Data Platform.</em></p> <h2><strong>The Motivation for Big Data</strong></h2> <p>With the rapid expansion of smart phones and other connected mobile devices, communications service providers (CSPs) need to rapidly process, store, and derive insights from the diverse volume of data traveling across their networks. Big data analytics can help CSPs improve profitability by optimizing network services/usage, enhancing customer experience, and improving security.  According to McKinsey, <a target="_blank" href="http://www.mckinsey.com/industries/telecommunications/our-insights/telcos-the-untapped-promise-of-big-data">the potential for Telcos to profit from applying data science effectively is substantial</a>. Examples include:</p> <ul> <li>Predicting the periods of heaviest network usage, and targeting steps to relieve congestion</li> <li>Identifying the customers most likely to defect, and targeting steps to prevent churn</li> <li>Identifying the customers most likely to have problems paying bills, and targeting steps to improve the recovery of payments</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/0-1605076495751.png" alt=""></p> <h2><strong>Big Data Use Cases In Telecom</strong></h2> <p>Telecommunication companies collect massive amounts of data from call detail records, mobile phone usage, network equipment, server logs, billing, and social networks, providing lots of information about their customers and network, but how can telecom companies use this data to improve their business?</p> <p>Most telecom use cases fall into these main categories: customer acquisition and retention, network services optimization, and security.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/1-1605076513981.png" alt=""></p> <h2><strong>Example Telecom Big Data Use Cases</strong></h2> <p><strong>Data-Driven Improvement of Services or Product</strong></p> <p>Telecoms need to share data between cell towers, users and processing centers and due to the sheer volume of this data, it is important to process it near the source and then efficiently transfer it to various data centers for further use. <u>MapR Event Store</u>, a new distributed messaging system, is uniquely effective to transport huge amounts of data and to make this data available with reliable geo-distributed replication across multiple data centers. With MapR Event Store, you can replicate streams in a master-slave, many-to-one, or multi-master configuration between thousands of geographically distributed clusters.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/2-1605076542289.png" alt=""></p> <p>One MapR customer is utilizing MapR Event Store to collect real-time data from all of its regional data centers and bring it to the HQ Central Data Center.</p> <p><strong>BEFORE</strong></p> <p>Before, the customer was using FTP to transfer data from antennas to regional data centers and to the HQ Central Data center, but the FTP transfer meant extreme latency throughout the data pipeline.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/3-1605076561221.png" alt=""></p> <p><strong>AFTER</strong></p> <p>Now data is collected at regional data centers with MapR Event Store and made available in real time to regional dashboards.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/4-1605076603434.png" alt=""></p> <p>MapR Event Store Topics at regional data centers are replicated in a many-to-one configuration to the HQ Central Data Center, making events available in real time to the HQ dashboard. This means they can now monitor global performance and react fast enough to improve customer services.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/11/5-1-1605076623532.png" alt=""></p> <p>Being able to process high throughput geo-distributed events in real time enables:</p> <ul> <li>Understanding how and where service issues are trending and how that is affecting customers.</li> <li><strong>Crowd-based antenna optimization</strong>: Monitor quickly changing network usage patterns, and reconfigure network support to handle short-term surges, such as heavy usage near a stadium during a sporting event.</li> <li><strong>Optimizing Services with Equipment Monitoring, Capacity Planning, and Preventative Maintenance</strong>: <ul> <li>Dropped calls</li> <li>Lack of network coverage, resulting in poor customer experience</li> <li>Bandwidth issues</li> <li>Poor download times</li> <li>Inordinate service wait times</li> <li>Switching, frequency utilization, capacity use</li> </ul> </li> </ul> <p>Analyzing these events in real time is the key to timely insights on network services in order to improve customer satisfaction.</p> <h2><strong>Customer 360</strong></h2> <p>Using data science in order to better understand and predict customer behavior is an iterative process, which involves:</p> <ol> <li>Data Discovery and Model Creation:</li> <li>Analysis of historical data.</li> <li>Identifying new data sources, which traditional analytics or databases are not using due to the format, size, or structure.</li> <li>Collecting, correlating, and analyzing data across multiple data sources.</li> <li>Knowing and applying the right kind of machine learning algorithms to get value out of the data.</li> <li>Using the Model in production to make predictions</li> <li>Data Discovery and updating the Model with new data.</li> </ol> <p>In order to understand the customer, a number of factors can be analyzed such as:</p> <ul> <li>Customer demographic data (age, marital status, etc.)</li> <li>Sentiment analysis of social media</li> <li>Customer usage patterns, geographical usage trends</li> <li>Calling-circle data</li> <li>Browsing behavior from clickstream logs</li> <li>Support call center statistics</li> <li>Historical data that shows patterns of behavior that suggest churn</li> </ul> <p>With this analysis, telecom companies can gain insights to predict and enhance the customer experience, prevent churn, and tailor marketing campaigns.</p> <h2><strong>Threat Detection</strong></h2> <p>Solutionary, a subsidiary of NTT Group, is a leader in Managed Security Services. They provide Threat Intelligence, Incident Response, Compliance and Vulnerability Management as a service to their clients. Their platform collects and correlates vast amounts of data from logs, endpoints, firewalls, and network devices.</p> <p>They needed to improve scalability as the data volume grew, but it was cost-prohibitive with their existing Oracle database solution. The old solution could not process the unstructured log data at scale and there were also major performance issues.</p> <p>They replaced their RDBMS solution with the MapR Data Platform to achieve scalability while still meeting reliability requirements. Their new solution combines machine learning algorithms, complex event processing, and predictive analytics to detect real-time security threats.</p> <p>All of the components of the use case architectures we just discussed can run on the same cluster with the MapR Data Platform, which provides advantages such as:</p> <ul> <li>Less complexity, fewer moving parts, fewer things to manage: converging multiple clusters for Streams/HBase/Spark/Hadoop into one cluster.</li> <li>"Joining" data sources into one core data mediation platform so that applications consume data in an easier way.</li> <li>Unified security.</li> <li>High reliability and high availability, replication from datacenter to datacenter.</li> <li>Multi-tenancy: MapR Event Store is able to have essentially unlimited topics for lots of tenants.</li> </ul> <p>Telecom is a classic example of the big data issues of huge volume and velocity of data flow, but CSPs also have high requirements for quick responses to events, security, and reliability. The use cases we just went over showed how telecom companies can not only address these requirements, but also profit from the huge amount of information in their data to improve their business.</p>https://developer.hpe.com/mapr-kubernetes-spark-and-drill-a-two-part-guide-to-accelerating-your-ai/https://developer.hpe.com/mapr-kubernetes-spark-and-drill-a-two-part-guide-to-accelerating-your-ai/Tue, 03 Nov 2020 16:17:29 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suresh Ollala", "publish": "2019-04-02T07:00:01.000Z", "tags": "mapr-platform" </code></pre> <hr> <p><em>Highly available, scalable, and elastic data platform for analytics and AI/ML</em></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image3-1604420243941.png" alt=""></p> <p>The early days of big data platforms focused on optimizing storage and compute on commodity hardware, mostly by software-level capabilities such as data locality support, better local caching, and effective data distribution. This worked well, but at a cost of flexibility.</p> <p>Things have changed over the last 5 years – hardware innovation in networking, CPU, and disk has made data locality no longer a bottleneck. A hybrid infrastructure of on-premises and cloud technology adoption forced innovation to scale storage and compute separately: AI/ML workloads require compute burst infrastructure, and cloud-native apps require flexibility of scheduling.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image1-1604420235871.png" alt=""></p> <h2>Scalability and Manageability</h2> <p>Separating storage from compute makes scaling infrastructure to match bursty and rapidly growing workloads as well as datasets more manageable. One can scale up or down strictly based on the workload and independent of each other. It also provides total flexibility in matching resources to the actual storage and compute capacity required at any given time. The separation of the two also reduces the complexity of managing multiple environments. For instance, production, development, and ad hoc analytics processes could run against the same dataset, thereby eliminating the administrative overhead of data replication and potential synchronization.</p> <h2>Agility</h2> <p>Decoupling storage and compute gives infrastructure greater agility. By relying on modern scalable storage systems, teams don't have to know the compute and storage capacity needed well in advance, freeing them from having to do a guesstimate, which results in either over-provisioning or under-provisioning the resources.</p> <p>In addition, with innovation in systems and cloud infrastructure, it's easier to choose storage-optimized, compute-optimized, and memory-optimized systems. This removes lock-in to particular set of machines or vendors. Flexible configurations enable admins to determine to what degree to optimize for cost vs. performance. For example, if a particular workload needs to be processed quickly and cost is not a key factor, then the admin can configure more nodes to speed up processing. If controlling cost is critical, then the admin can configure less nodes and utilize other cost-saving features, such as Auto Scaling and Spot Instances.</p> <h2>Lower TCO</h2> <p>Decoupling storage and compute can lead to lower total cost of ownership. Modern data infrastructure allows admins to configure the infrastructure for a pay-per-use model, so organizations only pay for storage space and point-in-time compute capacity. Today's organizations depend on a flexible infrastructure to speed up new initiatives, which requires low cost and faster infrastructure provisioning. DevOps culture is widely adapted to cut down on the red tape for system procurement and provisioning. It is important to note that organizations are looking for hybrid infrastructure, where cost is an important factor to consider along with consistent user and admin experience.</p> <h2>Evolution of Kubernetes as an Infrastructure Neutral Platform</h2> <p>Kubernetes has evolved to meet the needs of Developer, IT, and DevOps users.</p> <p>If you are a <strong>Developer</strong>, Kubernetes provides the necessary APIs that are needed today to build the next generation of applications, while at the same time supporting the more traditional ones. This lets developers focus on the app's functionality, instead of worrying about all the nitty-gritty details of how to manage the application. With Kubernetes, the developer experience is consistent, whether they work on their laptops or in a production environment. The application will not break when moved from a local machine to production, which saves the headache of debugging it.</p> <p><strong>IT Operations Perspective:</strong> System administrators like Kubernetes because it automates a lot of the mundane, boring, and error-prone operational tasks to keep applications running smoothly at scale. With Kubernetes, a server outage no longer keeps them up at night.</p> <h2>An Infrastructure Framework for Today</h2> <p>These days, developers are called on to write applications that run across multiple operating environments, including dedicated on-prem servers, virtualized private clouds, and public clouds such as AWS and Azure. Traditionally, applications and the tooling that support them have been closely tied to the underlying infrastructure, so it was costly to use other deployment models, despite their potential advantages. This meant that applications became dependent on a particular environment in several respects, including performance issues related to a specific network architecture, adherence to cloud provider-specific constructs, such as proprietary orchestration techniques, and dependencies on a particular back-end storage system.</p> <p>PaaS tries to get around these issues, but often at the cost of imposing strict requirements in areas like programming languages and application frameworks. Thus, PaaS is off limits to many development teams.</p> <p>Kubernetes eliminates infrastructure lock-in by providing core capabilities for containers without imposing restrictions. It achieves this through a combination of features within the Kubernetes platform, including Pods and Services.</p> <h2>Better Management Through Modularity</h2> <p>Containers allow applications to be decomposed into smaller parts with a clear separation of concerns. The abstraction layer provided for an individual container image allows us to fundamentally rethink how distributed applications are built. This modular approach enables faster development by smaller, more focused teams that are each responsible for specific containers. It also allows us to isolate dependencies and make wider use of well-tuned, smaller components.</p> <p>But this can't be achieved by containers alone; it requires a system for integrating and orchestrating these modular parts. Kubernetes achieves this in part using Pods – typically a collection of containers that are controlled as a single application. The containers share resources, such as file systems, kernel namespaces, and an IP address. By allowing containers to be collocated in this manner, Kubernetes removes the temptation to cram too much functionality into a single container image.</p> <p>The concept of a Service in Kubernetes is used to group together a collection of Pods that perform a similar function. Services can be easily configured for discoverability, observability, horizontal scaling, and load balancing.</p> <h2>Deploying and Updating Software at Scale</h2> <p>DevOps emerged as a method to speed the process of building, testing, and releasing software. Its corollary has been a shift in emphasis from managing infrastructure to managing how software is deployed and updated at scale. Most infrastructure frameworks don't support this model, but Kubernetes does, in part through Kubernetes Controllers. Thanks to controllers, it's easy to use infrastructure to manage the application lifecycle.</p> <p>The Deployment Controller simplifies a number of complex management tasks. For example:</p> <ul> <li><strong>Scalability.</strong> Software can be deployed for the first time in a scale-out manner across Pods, and deployments can be scaled in or out at any time.</li> <li><strong>Visibility.</strong> Identify completed, in-process, and failing deployments with status querying capabilities.</li> <li><strong>Time savings.</strong> Pause a deployment at any time and resume it later.</li> <li><strong>Version control.</strong> Update deployed Pods using newer versions of application images and roll back to an earlier deployment if the current version is not stable.</li> </ul> <p>Among other possibilities, Kubernetes simplifies a few specific deployment operations that are especially valuable to developers of modern applications. These include the following:</p> <ul> <li><strong>Horizontal autoscaling.</strong> Kubernetes automatically sizes a deployment's number of Pods based on the usage of specified resources (within defined limits).</li> <li><strong>Rolling updates.</strong> Updates to a Kubernetes deployment are orchestrated in "rolling fashion," across the deployment's Pods. These rolling updates are orchestrated while working with optional predefined limits on the number of Pods that can be unavailable and the number of spare Pods that may exist temporarily.</li> <li><strong>Canary deployments.</strong> A useful pattern when deploying a new version of a deployment is to first test the new deployment in production, in parallel with the previous version, and then scale up the new deployment while simultaneously scaling down the previous deployment.</li> </ul> <p>Kubernetes marks a breakthrough for DevOps because it allows teams to keep pace with the requirements of modern software development. In the absence of Kubernetes, teams have often been forced to script their own software deployment, scaling, and update workflows. Some organizations employ large teams to handle those tasks alone. Kubernetes allows us to derive maximum utility from containers and build cloud-native applications that can run anywhere, independent of cloud-specific requirements. This is clearly the efficient model for application development and operations we've been waiting for.</p> <h2>Why MapR is Best Suited for Kubernetes</h2> <p>The MapR Data Platform, time-tested across very large deployments, aligns with Kubernetes by offering several "add-on" features. MapR augments the capabilities of Kubernetes by offering heuristics on optimal CPU resources required by app containers. For instance, to run a Spark cluster, MapR offers a typical t-shirt sizing of small, medium, and large options with CPU and memory resources to pick from. This ability, combined with the auto scaling features of Kubernetes, lends end users the capability to create and scale compute, up and down intelligently. MapR also introduces the concept of a tenant, with CPU resources dedicated to it, wherein a tenant can be any app container – Spark, Drill, or a custom app. Multiple tenants with resource isolation can be efficiently run on the same platform with MapR data volumes mounted to tenants in an exclusive or shared mode.</p> <p>With these succinct capabilities, MapR along with Kubernetes offers ease of use and management, making it attractive for organizations to embrace Kubernetes and running applications in containers.</p>https://developer.hpe.com/better-complex-event-processing-at-scale-using-a-microservices-based-str/https://developer.hpe.com/better-complex-event-processing-at-scale-using-a-microservices-based-str/Tue, 03 Nov 2020 16:05:36 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Mathieu Dumoulin", "publish": "2017-01-09T06:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p><em>A microservice-based streaming architecture combined with an open source rule engine makes real-time business rules easy</em></p> <p>This post is intended as a detailed account of a project I have made to integrate an OSS business rules engine with a modern stream messaging system in the Kafka style. The goal of the project, better known as Complex Event Processing (CEP), is to enable real-time decisions on streaming data, such as in IoT use cases.</p> <p>After much writing, I’ve decided to split the post into two parts. In the first part, I’ll focus on explaining what is CEP, why it is useful, and explain the architectural solution and why we feel this is a good idea for a lot of useful production use cases.</p> <p>In <a href="https://developer.hpe.com/blog/real-time-smart-city-traffic-monitoring-using-microservices-based-streaming-architecture-part-2/">the second post</a>, I’ll show a concrete example based on a road traffic monitoring system and give as much detail as possible about how it was made.</p> <p>So without further ado, on to part 1!</p> <h2>Overview</h2> <p>As of 2015, the worldwide enterprise application software market is worth around 150 billion USD, according to Gartner Inc. It’s a huge market where one of the most common types of application revolves around applying some kind of business logic to data generated from various aspects of the business.</p> <p>These days, modern enterprise applications need to connect to ever more types of data sources, scale with the size of the data and the number of users, be reliable, and perform quickly. Long, custom application development cycles of one year or more are unappealing as business needs and conditions change, thus rendering the application obsolete before it is even put into production.</p> <p>In very large, country-wide, regional or global organisations, or organisations with exceptional data use in industries like finance, healthcare or IT, the needs stay the same, but must be met using big data technologies. This opens up a whole new class of difficulties that have made the cost of developing enterprise applications at scale extremely expensive, and it puts up very high barriers in terms of IT infrastructure and know-how requirements.</p> <p>So what is needed is a way to run business logic on data collected across a variety of sources, potentially at very large scales and ideally in real time, like an Internet of Things-type of application.</p> <h2>Understanding Complex Event Processing (CEP)</h2> <p>Complex event processing, or CEP for short, is not as complex at the name might suggest. Fundamentally, CEP is about applying business rules to streaming event data. Event data is simply data with a timestamp field. Examples of this kind of data might be log entries for a web server, receipts from purchases, or sensor data, all of which can be viewed as a constant stream of events. Applying rules on this streaming data enable useful actions to be taken in response.</p> <p>Here is an example for a smart home which has sensors at the doors, a smart WiFi router, and room movement detectors. With CEP streaming all the data into a home server, a user could make some rules like the following:</p> <ol> <li>If it's daytime and the door is closed and no phones are connected to the WiFi, set the house to “nobody home”</li> <li>If nobody is home and the door is unlocked, then lock the door and turn on the alarm</li> <li>If nobody is home and it's winter, lower the house temperature to 18C</li> <li>If nobody is home and it's summer, turn off the air conditioning</li> <li>If nobody is home and the door is unlocked by a family member, then turn off the alarm and set the house to “people are home'”</li> </ol> <p>Having a bunch of simple rules like these will quickly add up to a very smart home indeed. In fact, such capabilities are already available for purchase in several competing smart home "hub" devices that use common protocols to read information from compatible sensor devices around the house and then push actions back when some rules are met.</p> <p>This kind of example can be easily ported to many other domains. For example, in retail, purchase histories and beacons could be used to generate personalized, location-sensitive messages or coupons. In industrial applications, many machine tools could be operated and maintained more easily using a combination of relatively simple logical rules such as, "If the red button of this machine is lit, then it must be stopped."</p> <h2>CEP Rule-engine vs. Hand Coding</h2> <p>The engineers reading this so far are probably not very impressed, as streaming events apply simple rules.  A smart home use case such as the one described above could easily (well, to a point) be handled entirely by hand coding using Python and running on a old repurposed PC or even a Raspberry Pi.</p> <p>What are the parts of this type of project?</p> <ol> <li>Data ingest  </li> <li>Defining rules on the data</li> <li>Executing the rules  </li> <li>Taking action from rules when the conditions are met.</li> </ol> <p>Good software architecture calls for trying to make the parts most likely to change easy to change, at the cost of making other parts more difficult. What is the part most likely to change? Data ingest will only change when a new sensor is added, but a given sensor's data will not change suddenly. Executing rules in the abstract is always the same; what varies is the rule itself. Taking an action, once coded and working, doesn't really change, but it should be easy to add new actions over time.</p> <p>When the use cases starts to scale, and the number of rules increases, the rules processing engine efficiency starts to become important. Also, when the number of rules increases, making rules easy to edit is not just a "nice to have" feature, but a core requirement.</p> <p>Another often used argument is the separation of business logic from the SDLC. Business needs to move faster than software development. By using a rules engine, the two streams can move independently for the most part.</p> <h2>CEP is “Baked Into” IoT Applications</h2> <p>CEP is almost a requirement for any kind of IoT application such as smart homes, smart agriculture, Industry 4.0, or telecom data. It's a requirement in the sense that putting aside how the feature is implemented, IoT needs to apply rules to streaming event data. This is true whether it's at small scale in a single private home or at large scale across several factories scattered across the globe.</p> <p>An ideal design, based on what we just described, argues against a hand- coded solution and uses what is known as a "business rules processing engine." There are several which exist in the open source world, the most well known being Drools.</p> <h2>Drools: Open Source Business Rules Engine</h2> <p>Drools is an open source project developed under the JBoss umbrella of open source projects. It is a project with a long history of active development and it’s currently at version 6.5.0.Final with version 7 in beta. It is reasonably modern as it supports Java 8's vastly improved API.</p> <p>Drools has all the characteristics we are looking for in terms of a rules engine, with a well-defined DSL to define rules, and a rules engine based on the RETE algorithm that is well optimized and very fast. In addition, the documentation is thorough and there are a good number of books available to learn all about how to use this powerful framework.</p> <p>Finally, Drools comes with a GUI called Workbench that allows us to create and edit rules visually without any need for coding. This is a killer feature that puts the power of rules within the reach of business analysis.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture1-1604419877337.png" alt=""></p> <h2>Streaming Architecture Enables CEP for Big Data</h2> <p>A streaming architecture is a critical component to CEP. The entire point of CEP is to make decisions in (near) real-time over streaming data, as opposed to taking actions from analysis of historical data done as a batch process.</p> <p>CEP is all about agility and getting potentially complex behavior arising from the interaction of lots of simple rules all getting applied over the data, in memory in real time. A streaming, microservices-based architecture is becoming a standard for modern, large-scale architecture.</p> <p>I also presented a talk on this topic at Strata Singapore 2016. <a target="_blank" href="http://www.slideshare.net/mathieudumoulin2/cep-simplified-streaming-architecture-strata-singapore-2016">Please go take a look on Slideshare</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture2-1604419889212.png" alt=""></p> <p>In general terms, the solution will look like the graph above. Data sources, such as sensors, cash registers, or logs, are collected and with light ETL, are added to a stream. The data is then consumed by a program which simply passes the data as facts into the Drools KieSession. This is the in-memory workspace where the rule engine uses pattern matching to see what rules can fire based on the facts present in memory.</p> <p>In our proposed architecture, the rules reside in the Drools Workbench, a GUI rule editor which also serves as version control and as a repository for the rules to be deployed to production.</p> <p>The main benefit of this approach is to separate the process of maintaining the application itself completely independent from the process of editing rules that create value for the business. Engineers can be left with the clear task of making sure the system is performing well and is stable, while the business side can focus on the rules.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture3-1604419915057.png" alt=""></p> <p>In the diagram above, we can see how this may look more concretely with an implementation using a MapR cluster. It would be equally valid to use a Kafka cluster in its place for this particular application, although that would result in less potential for new use cases and an increased burden of system administration. The reason for this is that a Kafka cluster is strictly limited to supporting streaming, whereas using a cluster that is converged allows for additional use cases, both operational or analytical, right there on the same cluster.</p> <p>A key point here is the second arrow going <strong>back</strong> from the CEP Engine to the stream. It illustrates the important concept of using streams <strong>for input and output</strong> that is at the core of streaming architectures. That is also why Enterprise IT Systems is shown to get its data from the stream as well.</p> <p>The flow of data looks like so:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/picture4-1604419923661.png" alt=""></p> <p>Data flows from the data source to an Event Producer, which is just a stream producer or calls to a REST endpoint using the new <a target="_blank" href="https://github.com/confluentinc/kafka-rest">Kafka REST Proxy</a>. The REST proxy is also supported by MapR Event Store from MapR Ecosystem Pack 2.0.</p> <p>The CEP Engine can read data off the stream, and gets its rules from the Drools Workbench. From a streaming architecture point of view, the Drools Workbench and the CEP Engine are a unit, a single microservice, so to speak, as they are entirely self-contained and don’t have any external dependencies.</p> <p>As rules fire in the rules processing algorithm, some external actions will need to be taken. Those actions may be an insert or update of a table in a corporate DB, indexing to Elasticsearch to serve data to a Kibana dashboard, sending a notification. But instead of tightly coupling the systems together by making the call directly from the CEP Engine to the external system, we output the data from the CEP Engine back into another topic into the stream. Another microservice or application (like <a target="_blank" href="https://cdap.io/">Cask.co</a> or <a target="_blank" href="https://streamsets.com/">Streamsets</a>) will handle that flow.</p> <h2>In Conclusion</h2> <p>Complex Event Processing has been around for quite a while, but is now finally coming into its own. On the hardware side, services with a lot of memory are much more commonplace. On the software side, it’s possible to create a useful, production-grade CEP system entirely out of OSS, without needing to resort to expensive, custom-coded streaming applications.</p> <p>Combining a Kafka-style stream messaging system with Drools provides an organization with much needed agility in separating the very different tasks for creating and maintaining an enterprise streaming application and defining and editing business logic for real-time decisions.</p> <p>In <a href="https://developer.hpe.com/blog/real-time-smart-city-traffic-monitoring-using-microservices-based-streaming-architecture-part-2/">the next blog post</a>, we will cover a concrete use case that puts all this into practice and will show how such a system can be implemented using nothing more than Java, a MapR cluster, and the Drools Workbench running on a Wildfly application server.</p>https://developer.hpe.com/provisioning-secure-access-controls-in-mapr-database/https://developer.hpe.com/provisioning-secure-access-controls-in-mapr-database/Mon, 02 Nov 2020 06:21:10 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Jimit Shah", "publish": "2016-05-27T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>The ability to store and retrieve JSON documents using the OJAI standard has introduced a very powerful way to work with data in your MapR cluster. MapR Boolean Access Control Expressions (ACEs) provide a powerful, easy way to enforce authorization for providing secure access to data across the MapR Data Platform including MapR XD, MapR Event Store, and MapR Database.</p> <h2>MULTIPLE LEVELS</h2> <p>Access control using ACEs can be administered at multiple levels, separating administration and data access operations. Permissions for operations like creating a column family, deleting a column family, or setting ACEs can be controlled by setting ACEs as a part of table properties. If you are not familiar with the concept of a column family, just think of it as a logical grouping of data elements for setting a consistent policy (like permissions, TTL, etc.) on those elements. This construct exists in the MapR Database wide column data model as well, though the underlying implementation is completely different in the JSON document model. In MapR Database JSON, a JSON document can be stored as a composite of multiple JSON sub-documents which are part of different user-defined column families. More information about column families in MapR Database JSON can be found at the <a href="https://docs.datafabric.hpe.com/51/MapR-DB/JSON_DB/column_families_in_json_tables.html">Docs site</a>.</p> <p>Data access can be at column family and fieldpath levels within a table. In a JSON document, a field is the smallest unit that can be assigned a value which can be a scalar (integer, string, double, boolean etc) or a collection type (array, map). A fieldpath is a dot-separated set of fields. In fieldpath “address.home.street”, “address, “home” and “street” are individual fields—while “address” and “home” are map type, “street” is a scalar type.</p> <p>Both column family and fieldpath level permissions work in conjunction to decide the final access privileges to a single JSON fieldpath or sub-document.</p> <p>MapR Database supports three types of data access permissions:</p> <ul> <li>READ: used for managing permissions for get and scan operations</li> <li>WRITE: used for managing permissions for put and update operations</li> <li>TRAVERSE: used for managing permissions for READ and WRITE access</li> </ul> <p>While read and write permissions are independent of each other, traverse works in tandem with read and write permissions, which I’ll explain shortly. For now, let’s take a look at how the ACEs work at multiple levels.</p> <p>In the following illustration, a JSON document is represented as a hierarchical structure with sub-documents rooted at respective column families. In this case, the JSON document has three column families: ‘default’, ‘CF1’, ‘CF2’. ‘Default’ is the root of the document. Each column family contains the data of all the fields and their descendant fields under it. While the number of column families and how they’re laid out in a document is entirely configurable by the admin, in MapR Database the ‘default’ column family is always the root of the document.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-1-1604298011058.jpg" alt=""></p> <p>While the boundaries between the different subcomponents of a document are invisible to the developer using MapR Database, the way these components are laid out is important for the admin to administer security and ensure good performance.</p> <ul> <li>Permissions on column families are independent of each other. Permissions set on ‘default’ have no bearing on data access on ‘CF1’ or ‘CF2’. Likewise, permissions on ‘CF2’ have no effect on data access on ‘default’ or ‘CF1’.</li> <li>Different column families can have different access control configurations. This way, the admin can control which parts of the document are accessible to which user, group or role.</li> <li>Permissions set on a column family or fieldpath are applied to every row in the table that has data in that column family or fieldpath.</li> <li>The permissions of column families and fieldpaths work in conjunction: <ul> <li>Final Permission(Data in Column Family) = Permissions(Column Family) &#x26; Permissions(Fieldpath)</li> <li>If a user, group, or role doesn’t have permission to read data on CF1, then the user, group, or role cannot read any data under CF1.</li> <li>Likewise, if a user, group, or role has permission to read data on CF2, but has only permission to read a subset of fieldpaths in CF2, then the user, group, or role can read only that subset of fieldpaths.</li> </ul> </li> </ul> <h2>FINE-GRAINED</h2> <p>With support for native JSON documents, we have extended ACEs to provide the most fine-grained control over your data’s security. Now, admins can set read, write, and traverse permissions for any user, group, or role, to any part of a JSON document or a JSON subdocument, down to a single JSON fieldpath, with a scalar value. This is independent of access privileges to other parts of a document. All this is possible while preserving maximum space efficiency and performance.</p> <h2>TRAVERSE PERMISSION</h2> <p>In order to provide the flexibility in enforcing access to data in MapR Database JSON, we introduced a new type of permission, called the ‘TRAVERSE’ permission, which will allow the admin to allow access to some users to some parts of the JSON document while masking access to the rest of the document. As the name suggests, this new type of permission allows ‘traversing’ a JSON document to be able to discover its ‘structure’ or ‘self-describing schema’, without giving away access to any part of the data without explicit permission.</p> <ul> <li>The traverse permission has to be used with either a read or write permission. On its own, it has no effect on allowing or denying access. In a complex multi-level document, certain parts of the document are made accessible to certain users and groups while the rest are off limits and vice versa</li> <li>Traverse permission works in tandem with read/write permission in a hierarchical manner. If a traverse permission is set at a higher level in a JSON document, and a read/write permission is set at a lower level, then the read/write operation is allowed only on a fieldpath that terminates at or after the lower level.</li> <li>A good way of using the power of traverse permissions is to set it at the column family or at the highest field under a column family and then provide read/write access as desired at specific fieldpaths contained in the column family or top-level field.</li> <li>For instance, let’s say traverse permission for user ‘m7user1’ is set at fieldpath “a.b” and read permission for the same user is set at fieldpath “a.b.c.d”. For user ‘m7user1’, all read operations up to path “a.b.c” are disallowed and only read operations on fieldpaths “a.b.c.d” or beginning with fieldpath “a.b.c.d” are allowed. If no write permission has been set for the user ‘m7user1’ explicitly at the column family or the fieldpath level, no writes are allowed, even though traverse permission had been set at “a.b”.</li> <li>Now, let’s say we set a write permission for user ‘m7user1’ at “a.b.c” under a column family, all write operations are allowed at fieldpaths beginning with “a.b.c”, including “a.b.c” while all write operations at “”, “a” or “a.b” will be disallowed. Note that read operations will still be disallowed at “a.b.c” but write operations will be allowed for the user ‘m7user1’.</li> </ul> <p>Another example is a simplified version of a ‘Personnel Record’ that may be used in a company-wide people database. Different business functions—such as Finance, HR, or Engineering—would use the same database. Depending on which business function an employee belongs to, she may belong to one of the following group IDs (‘gid’) on the MapR cluster: ‘finance’, ‘engineering’ or ‘hr’.</p> <p><u>‘Personnel Record’ JSON Document</u></p> <pre><code class="language-json">{ _id: "ENG12345", address: { home: { street: "116 Severn Dr", city: "Mars", state: "CA", zip: 96066 }, work: { street: "5440 5th Ave", city: "Cranberry", state: "CA", zip: 95213 } }, dob: { month: "January", day: "30", year: "1989" }, name: { first: "John", last: "Doe }, photo: "&#x3C;binary:photo>", salary: 123456.00 sex: "male" }&#x3C;/binary:photo> </code></pre> <p>Depending on which gid a user belongs to, she may or may not have access to an employee’s information.</p> <ul> <li>Engineering employees may be able to look at only the name, work address, sex, home city and zip, day and month of birth</li> <li>Finance employees may be able to look at the name, home and work addresses, full date of birth, name, and read/write the salary.</li> <li>HR employees may be able to read/write at all the fields except for salary.</li> </ul> <p>The following illustration shows how the permissions may be configured. Imagine the JSON document laid out as a hierarchical structure like a tree. The ‘Personnel Database’ table has only one column family ‘default’. The admin ‘root’ manages permissions for the different groups - ‘hr’, ‘finance’ and ‘engineering’. The following is one possible way for the admin to set permissions:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-2-700-1604298029724.jpg" alt=""></p> <h2>COLUMN FAMILY PERMISSIONS</h2> <p>The admin can set independent read, write, and traverse permissions on the entire column family. When no column permissions are specified, all the columns (or fieldpaths in the JSON document) will inherit the same permissions as the column family they belong to</p> <p>In the illustration, no permissions have been set for the field ‘sex’. In that case, the field is readable by the user ‘root’, or anyone belonging to groups ‘hr’ or ‘finance’ while only a member of the ‘hr’ group can modify it.</p> <p>However, for the field ‘dob’, a column permission for read has been added so that anyone from ‘hr’, ‘finance’ or ‘engineering’ can read it. An additional column permission for read has been specified for ‘dob.year’ so that no one from ‘engineering’ can read it. Hence, the final read access can be computed as:</p> <ul> <li>Fieldpath ‘default: dob.day’ is readable by anyone in groups ‘hr, ‘finance’ and ‘engineering’</li> <li>Fieldpath ‘default: dob.month’ is readable by anyone in groups ‘hr, ‘finance’ and ‘engineering’</li> <li>Fieldpath ‘default: dob.year’ is readable by anyone in groups ‘hr’ and ‘finance’ but not by anyone in ‘engineering’</li> </ul> <p>No permission for write has been specified so the column family permission for write will prevail for ‘dob’ and its descendants. Hence, the final write access can be computed as:</p> <ul> <li>Only user ‘root’ or any user from group ‘hr’ can insert or update a value for any fieldpath originating at ‘dob’ in the default column family​</li> </ul> <h2>COLUMN PERMISSIONS</h2> <p>Like column family permissions, the admin can set independent read, write, and traverse permissions on a fieldpath or the predecessor of multiple fieldpaths.</p> <p>In the ‘Personnel Record’ illustration, individual column permissions have been set on fieldpaths having scalar values, or fieldpaths having multiple children as documents and/or scalar values.</p> <ul> <li>All fieldpaths beginning with ‘default:address.home’, except for ‘default:address.home.street’, are readable by anyone from ‘hr’, ‘finance’ and ‘engineering’, whereas only ‘hr’ and ‘finance’ can read data at ‘default:address.home.street’ fieldpath. Only a user with gid ‘hr’ can modify any field under ‘default:address.home’.</li> <li>On the other hand, all the fieldpaths under ‘default:address.work’ are readable by everyone in groups ‘hr’, ‘finance’ and ‘engineering’ while only users from ‘hr’ are allowed to modify them</li> <li>For fieldpath ‘default:salary’, only users from ‘finance’ can access the data at the fieldpath. No one from ‘hr’, ‘engineering’ or even the DB admin can read or update the value under ‘default:salary’. The column permissions at ‘default:salary’, in conjunction with column family permissions at ‘default’, effectively allow only users of group ‘finance’ to access the ‘salary’ fieldpath under ‘default’.</li> </ul> <p>While this illustration is by no means the most optimal way to administer security within a MapR Database JSON document, it demonstrates how a database administrator can use ACEs along with placement of correlated data in the same column family in a hierarchical fashion so that most coarse-grained permissions may be specified at the higher levels of the document, closer to the column family, and more specific permissions administered as one goes deeper in the hierarchy of information, along the different fieldpaths.</p> <h2>BEST PRACTICES</h2> <p>Let’s look at some really easy ways to begin using ACEs for administering access to JSON documents. In order to understand how ACE syntax works, we have some great docs over at <a href="https://docs.datafabric.hpe.com/51/SecurityGuide/ACEs.html">Docs</a> explaining the different parts of an ACE Boolean Expression and how to set it for column families and fieldpaths.</p> <p>I’ve categorized the most basic use cases for using ACEs. We’ll see how to use ACEs for READ access. The same methods can be applied to manage WRITE access.</p> <p>Let’s take the example of a taxi trip database. The following is a simplified version of how a trip record may look.</p> <p><u>Taxi Trip Record</u></p> <pre><code class="language-json">{ trip_id: "SFO12345", trip_info: { trip_source: { lat: 37.40308, long: -121.96983, address: { street: "4900 Marie P DeBartolo Way", city: "Santa Clara", state: "CA", zip: 95054 }, name: "Levi's Stadium" }, trip_destination: { lat: 37.418434, long: -121.94498, address: { street: "350 Holger Way", city: "San Jose", state: "CA", zip: 95134 }, name: "MapR Technologies" } }, rider_id: "m7user1", driver_id: "uberlyftcoexist", vehicle: { VIN: "VWW12345354564D234DD", license: "7GTZ113", make: "Toyota", model: "Camry", year: "2015", color: "black". photo: &#x3C; binary: jpg > }, billing: { merchant: "Visa", transactionKey: "12345678901234", amount: "10.45" }, reviews: { driver_review: { stars: 5, comment: "Safe, on time" }, rider_review: { stars: 4 } } } </code></pre> <p>For a taxi trip database table, the admin could lay out the column families as follows:</p> <ul> <li>‘default’ column family at root of the document</li> <li>‘trip_info’ column family at ‘trip_info’ JSON path</li> <li>‘billing_info’ column family at ‘billing’ JSON path</li> <li>‘reviews’ column family at ‘reviews’ JSON path</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-3-1604298037997.jpg" alt=""></p> <h2>INCLUDE FEW</h2> <p>In this use case, the admin wants to grant access to a document or part of a document to a limited set of users and/or groups.</p> <p>Let’s say the parties involved in a trip are ‘user_rider’, ‘user_driver’ and groups ‘billing’, ‘navigation’, ‘traffic’, ‘crm’, ‘cust_happiness’. Each group represents a team with a specific function in the taxi company’s business: ‘billing’ is responsible for processing and bookkeeping of payments; ‘navigation’ and ‘traffic’ are responsible for analyzing, computing routes and providing navigation to drivers; and ‘cust_happiness’ and ‘crm’ are responsible for customer satisfaction and addressing grievances.</p> <p>Example: The billing information should be visible only to the users in ‘billing’, while the reviews are relevant only for users in ‘cust_happiness’ and ‘crm’. The admin can set the permissions on ‘billing’ and ‘reviews’ column families to allow only ‘billing’ and ‘cust_happiness’ groups to have access to the data in respective column families.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-4-1604298045986.jpg" alt=""></p> <h2>EXCLUDE FEW</h2> <p>In this use case, the admin wants to deny access to a document or part of a document to a limited set of users and/or groups.</p> <p>Example: The ‘trip_id’ is an internal unique trip identifier that should be visible to everyone using the database but the ‘user_rider’ and ‘user_driver’. The admin can set permissions on the ‘trip_id’ field to exclude access to ‘user_rider’ and ‘user_driver’ and give access to all the users and groups to read or write ‘default’ column family.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-5-1604298055030.jpg" alt=""></p> <h2>1:1</h2> <p>In this use case, the admin wants to provide access to specific parts of a document to specific users and/or groups.</p> <p>Example: Let’s give read access to both ‘user_driver’ and ‘user_rider’ to each others’ reviews but also give them write access to reviews authored by them. The admin can set permissions on the ‘reviews’ column family to allow both the users to traverse the ‘reviews’ subdocument but provide only specific permissions to be writable for these users.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/aces-6-1604298064571.jpg" alt=""></p> <h2>CLOSING COMMENTS</h2> <p>MapR Database ACEs are a very flexible and powerful method for provisioning secure access control for your data residing on MapR Database. If you have questions about ACEs or MapR Database, feel free to ask your questions on <a href="https://community.datafabric.hpe.com/s/">Ezmeral Data Fabric Community.</a></p>https://developer.hpe.com/kubernetes-tutorial-how-to-install-and-deploy-applications-at-scale-on-k/https://developer.hpe.com/kubernetes-tutorial-how-to-install-and-deploy-applications-at-scale-on-k/Mon, 02 Nov 2020 06:12:48 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Martijn Kieboom", "publish": "2018-04-26T10:45:00.000", "tags": ["open-source-software","kubernetes"] </code></pre> <hr> <h2>Introduction</h2> <p>Containers are hot! But how do you manage hundreds or even thousands of containers in a production environment to support a 24/7 business? Various container management solutions have jumped into that business, but one is getting a lot of attention and adoption at the moment: Kubernetes.</p> <p>Originally designed by Google but now open-sourced, Kubernetes is being adopted by many commercial vendors, including Docker Enterprise, Red Hat OpenShift, and Mesosphere as well as all major cloud providers. So there are plenty of ways to manage your containers using Kubernetes.</p> <p>In this first out of three blog posts, we will look into what business benefits can be achieved by combining MapR with Kubernetes to run and manage your containers.</p> <h2>Why Containers</h2> <p>What we hear from customers in their journey toward making data actionable are the following challenges:</p> <ul> <li>Introducing new innovations or capabilities</li> <li>Maintaining business SLAs in a changing environment</li> <li>Using or introducing legacy services as a result of mergers and acquisitions</li> <li>Ongoing upgrade of applications/services</li> <li>Difficulty of packaging and distributing apps to end customers</li> </ul> <p>Introducing new innovations while maintaining existing business SLAs is a big challenge with often a conflict of interest.  IT organizations are mainly focusing on delivering existing SLAs and therefore push back can be experienced when the business wants to launch new innovative products.</p> <p>In addition, organizations are even more pressured when they acquire or merge businesses as that brings in the challenge of onboarding existing legacy systems.</p> <p>Finally, the technology updates are going faster than ever before. Upgrading applications and services is becoming more complex and challenging every day. This goes hand in hand with the complexity involved in how apps are being packaged and distributed to your end customers.</p> <p>In the following paragraphs, we will have a look at what MapR Technologies has to offer to overcome these challenges and really put your data into action.</p> <h2>MapR Volume Driver Plugin for Kubernetes</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/mapr-volume-driver-plugin-1604297556537.png" alt="MapR Volume Driver Plugin for Kubernetes"></p> <p>Let’s have a look at how we can combine existing applications with new innovative applications and services:</p> <p><strong>Applications</strong></p> <p>Placing applications and even microservices in container pods is a first step in making them flexible and agile. This allows us to distribute the application or service to where it runs best. It also allows physical separation of different types of applications. This way you can easily run classic applications and processes (for example, an ETL process) as well as an innovative machine learning application for image classification using Tensorflow on the same environment.</p> <p><strong>Compute - Kubernetes</strong></p> <p>Finally, Google’s Kubernetes is quickly gaining adoption as the container scheduler and orchestration solution to allow running applications and services anywhere. To maintain the agility and flexibility of the container-based applications running on Kubernetes, the MapR Kubernetes Volume Driver Plugin gives all applications and microservices seamless access to the MapR Data Platform.</p> <p><strong>Data Stores</strong></p> <p>The data required by these different applications and (micro)services can, however, be anywhere, as data nowadays is distributed geographically across multiple environments. From edge environments to a combination of private and public cloud, where the data actually is stored should be transparent to the applications and services.</p> <p>Data - MapR Data Platform</p> <p>That’s where the MapR Data Platform with its Global Namespace comes in. It virtually combines all MapR clusters into a single Global Data Fabric, providing applications and (micro)services seamless access to all data, irrespective of its physical location.</p> <h2>Business Benefits</h2> <p>Combining MapR with Kubernetes integration delivers the following business benefit to any organization:</p> <ul> <li>Faster innovation while running ongoing business and operations</li> <li>Flexible scaling (up and down) to accommodate business needs</li> <li>Easy integration of mergers and acquisitions</li> <li>Ease of maintaining and rolling out upgrades</li> <li>Ease of packaging and distributing apps to end customers</li> </ul> <p>To summarize the business benefits of the powerful combination of Kubernetes with the MapR Data Platform:</p> <p>Combining ongoing business operations with deploying new business innovations has never been easier. Scaling any application or service to accommodate ever-changing business and customer needs is simply a matter of scaling up or down the number of application container pods. Onboarding legacy services as part of mergers and acquisitions doesn’t have to stop you from innovating in parallel, rolling out new application and service results with quicker innovation and time to market. And finally, packaging and distributing applications and services to your customer allows you to adopt new technologies and innovation immediately.</p> <p><a href="/blog/kubernetes-tutorial-part-2-of-3-how-to-install-and-deploy-applications-at-scale-on-k8s">In the following blog post</a>, we will start deploying a Kubernetes cluster and load the MapR Volume Driver Plugin for Kubernetes to allow enabling these business benefits.</p>https://developer.hpe.com/2020-November-02/https://developer.hpe.com/2020-November-02/Mon, 02 Nov 2020 06:00:00 GMThttps://developer.hpe.com/hpe-dev-hack-shack-coding-challenges-are-you-ready-to-compete/https://developer.hpe.com/hpe-dev-hack-shack-coding-challenges-are-you-ready-to-compete/Fri, 30 Oct 2020 15:04:29 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-1604080946671.png" alt="coding challenge"></p> <p>I am really excited to announce that we have created a new community activity for you – the HPE DEV Hack Shack Coding Challenges. Typically we ran these challenges and physical events, but, because they became so popular, we are now providing them virtually. HPE DEV will post a coding challenge in the <a href="/hackshack/">Hack Shack</a> for the community to solve, offering cool prizes to those whose solution is chosen as the winner of the challenge. Submissions will be judged on technical achievement and completeness of the code and quiz answers. Winners will be notified and will be able to choose a prize from a given catalog.</p> <p>The idea is to offer a fun event to the community that shares knowledge. The challenges have been designed to be simple enough that most data scientists, developers, and DevOps engineers can work on them. In this blog post, I’ll cover the details of the coding challenges program, including instructions for participants.</p> <p>These challenges are being designed to offer students a hands-on method of understanding specific open source and HPE technologies guided by a subject matter expert (SME) and detailed instructions provided in a Jupyter Notebook format. Students will have access to other SMEs through Slack should any questions arise during their challenge session.</p> <h2>One challenge to get started, with more coming shortly</h2> <p>Many of these challenges will be extensions to technical workshops we’ve hosted in the past. You can find the video <a href="/hackshack/replays/0">replays of these workshops</a> on the <a href="/hackshack/">HPE DEV Hack Shack</a>. Through the workshops, students received hands-on experience with coding practices through an associated Jupyter Notebook. The HPE DEV Hack Shack challenges will utilize much of the same material, including the Jupyter Notebook. For those of you who are new to the topic being covered, it is recommended you watch the corresponding <a href="/hackshack/replays/0">video replay</a> first and then sign up for the challenge.</p> <p>During the mid-month of November, we will start with a single challenge:</p> <ul> <li>Deploy a front-end app on a Kubernetes cluster</li> </ul> <p>Students will have 4 hours in which to go through the challenge, which includes time to review the video replay of the corresponding technical workshop, follow the Jupyter Notebook instructions, and save their work to their local laptop should they want to do more work in their own environment or retake the challenge. They’ll also be able to connect with SMEs through off-line support.</p> <h2>How it works</h2> <p>To take the challenge, go to the <a href="/hackshack/">Hack Shack</a> and navigate to the <strong>Challenges</strong> page. From there, select which challenge you want to take and click on the <strong>Challenge me</strong> button:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-1-1604071470980.png" alt="coding challenge"></p> <p>At this point, the registration panel pops up. Enter the details requested and click on the <strong>Take on the challenge</strong> button. In a matter of just a few minutes, you’ll start your challenge.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-2-1604430877028.png" alt="coding challenge"></p> <p>By pressing the <strong>Take on the Challenge</strong> button, you initiate a back-end automated registration process. This process spawns a dedicated notebook environment for you and sends you a welcome email indicating that you have been registered in the database.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-3-1604071494820.png" alt="welcome email"></p> <p>Not long after, it sends you a second email providing a link to your challenge Jupyter Notebook, along with your StudentID and password.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-4-1604071501876.png" alt="credentials email"></p> <blockquote> <p><strong>IMPORTANT: Receipt of this email indicates that the challenge environment is ready for you to begin. Similar to our HPE DEV Workshops-on-Demand, you will have just 4 hours from the receipt of this second email to complete the challenge. It is recommended that you only register for a challenge when you know you will have the next 4 hours to work on it. We advise you to regularly save your work and download the Jupyter Notebook to refer to later should you not be able to finish the challenge within the given 4-hour time slot. If you cannot finish the challenge in that time, you will need to take the challenge again from the beginning.</strong></p> </blockquote> <p>When you click on the <strong>Start Challenge</strong> button found in your second email, it will bring you to a Sign In page where you will log into the challenge with your StudentID and the password provided in your second email.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-5-1604072032740.png" alt="jupyter sign in"></p> <p>Once you log in, open the challenge folder on the left by double-clicking on it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-6-1604072039116.png" alt="jupter notebook home screen"></p> <p>Each notebook generally has several sections. Start with the <strong>Challenge Overview</strong> and follow the instructions from there to work on the challenge and how to submit your work.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-7-1604367130983.png" alt="jupter notebook"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-8-1604367142859.png" alt="jupter notebook"></p> <h2>Don’t forget to save your work</h2> <p>One hour prior to the end of the 4-hour period, you will receive an email reminding you that your session is coming to a close and that you should download the challenge notebook if you anticipate using it in the future. At the end of every session, the environment is cleaned up automatically, so be sure to save your work.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-9-1604072055584.png" alt="reminder email"></p> <p>At the end of the challenge, you will receive a final email indicating that the challenge is over. In the email, you will also be asked to take a short survey. The results of this survey will help us improve how we offer the challenges in the future. Your feedback is very important to our being able to meet your needs, so we encourage you to take just a few minutes to fill it out.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/coding-challenge-10-1604072062041.png" alt="challenge end email"></p> <h2>How to get help</h2> <p>You’ll have access to SME assistance through our challenges Slack channel. We staff the channel to answer questions between 4 am and 4 pm EST Monday through Friday. You may wish to ensure you schedule the timing of your challenge between those hours if you think you’ll have questions or need additional help.</p> <p>There will be a limited number of seats available for each challenge. These seats will be filled on a first-come/first-serve basis. We look forward to the opportunity of offering these challenges to you and you can win cool prizes. Remember to check back on the <a href="/blog">HPE DEV blog</a> for any further updates.</p>https://developer.hpe.com/hpe-dev-hack-shack-expands-for-premier-kubernetes-event/https://developer.hpe.com/hpe-dev-hack-shack-expands-for-premier-kubernetes-event/Wed, 28 Oct 2020 19:02:50 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/kubecon-na-image-1603912007661.png" alt="kubecon na image"></p> <p>Mark your calendars for November 17-20, 2020! <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/">KubeCon | CloudNativeCon, North America</a> is coming, the <a href="http://cncf.io/">Cloud Native Computing Foundation’s (CNCF)</a> flagship event, and you won’t want to miss it! Because of its increased involvement with CNCF and commitment to open source technologies, Hewlett Packard Enterprise (HPE) is excited to upgrade its sponsorship to the Diamond level. This allows us to more deeply connect with technologists worldwide, offering them easy access to experts, technology, and tools to accelerate digital transformation.</p> <p>Solutions for security, data-centric workloads, and workflow challenges will be front and center, with lots of opportunities to engage in direct conversations with HPE subject matter experts. Make sure you attend the HPE speaker sessions and demos, and take advantage of the hands-on technical workshops in the HPE Developer Community Hack Shack. You’ll get to hear HPE speaker sessions on how to scale machine learning without compromising privacy and view a tutorial on how to use the Container Storage Interface (CSI) primitives with Kubernetes. Using machine learning to harness data and operationalize data pipelines using KubeDirector and other products in the HPE Ezmeral Software Portfolio will also be highlighted throughout the virtual booth. <a href="https://community.hpe.com/t5/hpe-ezmeral-uncut/hpe-showcasing-enterprise-edge-to-cloud-solutions-at-kubecon/ba-p/7107337">Check here for details on HPE’s overall presence at the event</a>.## Explore our virtual Hack Shack and see what’s new!</p> <p>The HPE DEV Community Hack Shack offers developers, designers, and data scientists the opportunity to connect with HPE subject matter experts and collaborate with them to accelerate innovation using open source and HPE technologies. It’s a unique place designed to give virtual events a more personal touch and extend the experience beyond the event.</p> <p>The HPE DEV team is pleased to now offer Workshops-on-Demand through the Hack Shack. In our technical workshops, you’ll learn about HPE and open source solutions and get hands-on experience with these technologies, including the HPE Ezmeral Container REST API.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/10/hack-shack-home-page-1604432949140.png" alt="hack shack home page"></p> <p>Here’s a snapshot of what you will find in the virtual Hack Shack:</p> <ul> <li><a href="/hackshack/workshops">WORKSHOPS:</a> Check out our Workshops-on-Demand. These are free, hands-on training sessions you can access at any time, from anywhere. Using a Jupyter Notebook environment, you’ll have the opportunity to gain hands-on experience across different technologies, like Grommet and the HPE Ezmeral Container Platform REST API.</li> <li><a href="/hackshack/challenges">CHALLENGES:</a> Our coding challenge offers an exciting opportunity to put what you’ve learned to the test and compete with others for prizes. This challenge will task you with deploying a sample front-end application on a CNCF-certified Kubernetes cluster running on HPE Ezmeral Container Platform using the API calls.</li> <li><a href="/hackshack/replays">REPLAYS:</a> Augmenting our Workshops-on-Demand, we’ve posted replays of many of the technical workshops we’ve offered live in the past. View them to learn more about the <a href="/hackshack/replays/1">HPE Ezmeral Container Platform</a>, <a href="/hackshack/replays/5">SPIFFE and SPIRE authentication</a>, and the <a href="/hackshack/replays/2">HPE Container Storage Interface</a> for Kubernetes.</li> <li><a href="/hackshack/ezmeral">HPE EZMERAL:</a> We’ve made it easy for you to find detailed information on the <a href="https://developer.hpe.com/platform/hpe-ezmeral-container-platform/home">HPE Ezmeral Container Platform</a>, <a href="https://developer.hpe.com/platform/hpe-ezmeral-data-fabric/home">HPE Ezmeral Data Fabric</a>, <a href="https://www.hpe.com/us/en/solutions/machine-learning-operations.html">HPE Ezmeral ML Ops</a>, along with other products in this innovative set of software that can be deployed on any cloud, on any hardware, and is 100% open source Kubernetes.</li> <li><a href="/hackshack/arcade">ARCADE:</a> In our arcade, along with a bunch of cool stickers, you’ll find Hack Shack Attack! Give our popular retro-style video game a try and compete with your friends for the highest score.</li> <li><a href="/hackshack/community">COMMUNITY:</a> We invite you to join and contribute your expertise in our blog forum or deliver an on-demand workshop. Connect with others in the community via <a href="https://slack.hpedev.io/">Slack</a> and <a href="https://twitter.com/HPE_Developer">Twitter</a> channels to start conversations and get answers to questions. And you can sign-up for our HPE DEV Newsletter to stay up-to-date on the newest blog posts and tutorials.</li> </ul> <h2>Prizes? Did someone say prizes?</h2> <p>Take our <a href="/hackshack/challenges">HPE DEV Hack Shack coding challenge</a> and you could win a prize! Our coding challenges are timed events where developers are asked to create their own code to achieve a defined function while taking specific parameters into account. For KubeCon | CloudNativeCon NA, we will be challenging you to deploy a sample front-end application on a CNCF-certified Kubernetes cluster running on HPE Ezmeral Container Platform using the API calls. Submissions will be judged on technical achievement and completeness of the code and quiz answers. For details on how to participate, <a href="/blog/hpe-dev-hack-shack-coding-challenges-are-you-ready-to-compete">please refer to this blog post</a>.</p> <h2>But wait… there’s more!</h2> <p>Visit the <a href="https://developer.hpe.com/">HPE DEV Community Portal</a> via the <a href="/hackshack/">Hack Shack Lobby</a>. There, you can access a treasure trove of information, including:</p> <ul> <li><a href="https://developer.hpe.com/platforms">PLATFORMS</a> – Access GitHub resources and software development kits (SDKs)</li> <li><a href="/blog">BLOG</a> – Learn through blog posts and tutorials</li> <li><a href="https://developer.hpe.com/projects">OPEN SOURCE</a> – Discover platforms, apps and contributions</li> <li><a href="https://developer.hpe.com/events">EVENTS</a> – Learn about upcoming events</li> <li><a href="https://developer.hpe.com/community">COMMUNITY</a> – Participate in and contribute to the community</li> </ul> <p>In other sections of the KubeCon | CloudNativeCon North America HPE virtual booth, you’ll explore HPE’s different software-enabling and application development technologies, including:</p> <ul> <li> <p><a href="https://spiffe.io/">The SPIFFE and SPIRE Projects</a>. With the recent acquisition of Scytale, HPE is the leading contributor for CNCF’s SPIFFE and SPIRE projects. These projects help organizations build a foundation for zero trust through the use of platform agnostic, cryptographic service identity.</p> </li> <li> <p><a href="https://www.hpe.com/us/en/ezmeral.html">The HPE Ezmeral Software Portfolio</a>, including the industry’s first enterprise-grade container platform for cloud-native and distributed non cloud-native applications. Built on open source Kubernetes, this unified container platform runs on any infrastructure; either on-premises, in multiple public clouds, in a hybrid model, or at the edge.</p> </li> <li> <p>The <a href="https://www.hpe.com/us/en/storage/intelligent-storage.html?chatsrc=ot-en&#x26;jumpid=ps_8r5mdg32xs_aid-520023673&#x26;gclid=Cj0KCQiAs67yBRC7ARIsAF49CdU6O6Hbaj1lwT8tcrU702BzRnZboWNQILTShb0cCk-eEk7nUjQ-yhMaAv4fEALw_wcB&#x26;gclsrc=aw.ds%22%20%5Ct%20%22_blank">intelligent data platform</a> from HPE, focused on persistent and ephemeral storage use cases for Kubernetes and cloud-native workloads in private, public and hybrid clouds. Visit the respective platform pages for an overview:</p> <ul> <li><a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">HPE Nimble Storage and HPE Cloud Volumes</a></li> <li><a href="https://developer.hpe.com/platform/hpe-3par-and-primera/home">HPE Primera</a></li> </ul> </li> </ul> <p>For details on HPE’s overall presence at KubeCon | CloudNativeCon North America, please <a href="https://community.hpe.com/t5/hpe-ezmeral-uncut/hpe-showcasing-enterprise-edge-to-cloud-solutions-at-kubecon/ba-p/7107337">refer to our community blog post here</a>.</p> <h2>Don't miss these other interesting HPE sessions</h2> <p>You can also find a number of other sessions highlighting HPE offerings at KubeCon | CloudNativeCon North America, including:</p> <ul> <li><strong><a href="https://sched.co/eoDy">Scaling machine learning without compromising privacy</a></strong>, a Keynote session presented by lead data scientist and Distinguished Technologist, Nanda Vijaydev of HPE, Thursday, November 19, 1:18pm - 1:23pm</li> <li><strong><a href="https://kccncna20.sched.com/event/f6jb?iframe=no">No More Moats: Protecting Your Cloud Native Infrastructure with Zero Trust</a></strong>, presented by HPE software engineer, Daniel Feldman, Tuesday, November 17, 1:00pm - 1:30pm</li> <li><strong><a href="https://kccncna20.sched.com/type/Tutorials/101+%28sessions+for+those+new+to+the+conference+overall+and%2For+beginners+to+the+conference+content">Introduction to using the CSI Primitives</a></strong>, presented by Tech Marketing Engineer and Master Technologist, Michael Mattsson of HPE, Friday, November 20, 5:05pm - 6:30pm</li> </ul> <p>Refer to the <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/">program schedule</a> for updates on times and locations.</p> <p>Virtual events can be challenging to engage and talk to the right people. Many who attend physical conferences come specifically to talk with industry and subject matter experts. That’s why the <a href="https://developer.hpe.com/community">HPE DEV community</a> has worked hard to deliver its <a href="/hackshack/">HPE DEV Community Hack Shack</a> virtual experience. We want to ensure you have a place where you can do just that. For questions or help, be sure to connect with us in our virtual room at the booth or on our <a href="https://slack.hpedev.io/">HPE DEV Slack Channel</a>.</p>https://developer.hpe.com/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-/https://developer.hpe.com/streaming-machine-learning-pipeline-for-sentiment-analysis-using-apache-/Wed, 28 Oct 2020 15:43:34 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Carol McDonald"], "publish": "2019-05-07T07:00:00.000Z", "tags": "machine-learning" </code></pre> <hr> <p>Text mining and analysis of social media, emails, support tickets, chats, product reviews, and recommendations have become a valuable resource used in almost all industry verticals to study data patterns in order to help businesses to gain insights, understand customers, predict and enhance the customer experience, tailor marketing campaigns, and aid in decision-making.</p> <p>Sentiment analysis uses machine learning algorithms to determine how positive or negative text content is. Example use cases of sentiment analysis include:</p> <ul> <li>Quickly understanding the tone from customer reviews <ul> <li>To gain insights about what customers like or dislike about a product or service</li> <li>To gain insights about what might influence buying decisions of new customers</li> <li>To give businesses market awareness</li> <li>To address issues early</li> </ul> </li> <li>Understanding stock market sentiment to gain insights for financial signal predictions</li> <li>Determining what people think about customer support</li> <li>Social media monitoring</li> <li>Brand/product/company popularity/reputation/perception monitoring</li> <li>Discontented customer detection monitoring and alerts</li> <li>Marketing campaign monitoring/analysis</li> <li>Customer service opinion monitoring/analysis</li> <li>Brand sentiment attitude analysis</li> <li>Customer feedback analytics</li> <li>Competition sentiment analytics</li> <li>Brand influencers monitoring</li> </ul> <p>Manually analyzing the abundance of text produced by customers or potential customers is time-consuming; machine learning is more efficient and with streaming analysis, insights can be provided in real time.</p> <p>This is the first in a series of blog posts, which discusses the architecture of a data pipeline that combines streaming data with machine learning and fast storage.  In this first part, we will explore sentiment analysis using Spark machine learning data pipelines. We will work with a dataset of Amazon product reviews and build a machine learning model to classify reviews as positive or negative.  In the <a href="/blog/streaming-ml-pipeline-for-sentiment-analysis-using-apache-apis-kafka-spark-and-drill-part-2">second part</a> of this tutorial, we will use this machine learning model with streaming data to classify documents in real time.  The second post will discuss using the saved model with streaming data to do real-time analysis of product sentiment, storing the  results in MapR Database, and making them rapidly available for Spark and Drill SQL.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image7-1603902952832.png" alt=""></p> <p>In this post, we will go over the following:</p> <ul> <li>Overview of classification and sentiment analysis concepts</li> <li>Building feature vectors from text documents</li> <li>Training a machine learning model to classify positive and negative reviews using logistic regression</li> <li>Evaluating and saving the machine learning model</li> </ul> <h2>Classification</h2> <p>Classification is a family of supervised machine learning algorithms that identify which category an item belongs to (such as whether an email is spam or not), based on labeled data (such as the email subject and message text). Some common use cases for classification include credit card fraud detection, email spam detection, and sentiment analysis.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image3-1603902962230.png" alt=""></p> <p>Classification takes a set of data with known labels and predetermined features and learns how to label new records, based on that information. Features are the properties that you can use to make predictions. To build a classifier model, you explore and extract the features that most contribute to the classification.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image15-1603902978389.png" alt=""></p> <p>Let's go through an example for sentiment analysis for text classification of positive or negative.</p> <ul> <li>What are we trying to predict? <ul> <li>In this example, the customer review ratings are used to label reviews as positive or not. A review with 4 to 5 stars is considered a positive review, and a review with 1 to 2 stars is considered a negative review.</li> </ul> </li> <li>What are the properties that you can use to make predictions? <ul> <li>The review text words are used as the features to discover positive or negative similarities in order to categorize customer text sentiment as positive or negative.</li> </ul> </li> </ul> <h2>Machine Learning Workflow</h2> <p>Using Machine Learning is an iterative process, which involves:</p> <ol> <li>Data discovery and model creation <ul> <li>Analysis of historical data</li> <li>Identifying new data sources, which traditional analytics or databases are not using, due to the format, size, or structure</li> <li>Collecting, correlating, and analyzing data across multiple data sources</li> <li>Knowing and applying the right kind of machine learning algorithms to get value out of the data</li> <li>Training, testing, and evaluating the results of machine learning algorithms to build a model</li> </ul> </li> <li>Using the model in production to make predictions</li> <li>Data discovery and updating the model with new data</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image18-1603902991041.png" alt=""></p> <h2>Feature Extraction</h2> <p>Features are the interesting properties in the data that you can use to make predictions. Feature engineering is the process of transforming raw data into inputs for a machine learning algorithm. In order to be used in Spark machine learning algorithms, features have to be put into feature vectors, which are vectors of numbers representing the value for each feature. To build a classifier model, you extract and test to find the features of interest that most contribute to the classification.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image8-1603902999171.png" alt=""></p> <h2>Apache Spark for Text Feature Extraction</h2> <p>The TF-IDF (Term Frequency–Inverse Document Frequency) feature extractors in <a href="http://spark.apache.org/docs/latest/ml-features.html#23tf-idf">SparkMLlib</a> can be used to convert text words into feature vectors. TF-IDF calculates the most important words in a single document compared to a collection of documents. For each word in a collection of documents, it computes:</p> <ul> <li>Term Frequency (TF), which is the number of times a word occurs in a specific document</li> <li>Document Frequency (DF), which is the number of times a word occurs in a collection of documents</li> <li>Term Frequency-Inverse Document Frequency (TF-IDF), which measures the significance of a word in a document (the word occurs a lot in that document, but is rare in the collection of documents)</li> </ul> <p>For example, if you had a collection of reviews about bike accessories, then the word 'returned' in a review would be more significant for that document than the word  'bike.'In the simple example below, there is one positive text document and one negative text document, with the word tokens 'love,''bike,' and 'returned' (after filtering to remove insignificant words like 'this' and 'I').  The TF, DF, and TF-IDF calculations are shown. The word 'bike' has a TF of 1 in 2 documents (word count in each document), a document frequency of 2 (word count in set of documents), and a TF-IDF of ½ (TF divided by DF).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image1-1603903007533.png" alt=""></p> <h2>Logistic Regression</h2> <p><a href="https://spark.apache.org/docs/latest/ml-classification-regression.html#23logistic-regression">Logistic regression is a popular method to predict a binary response.</a> It is a special case of generalized linear models that predicts the probability of the outcome. Logistic regression measures the relationship between the Y "Label" and the X "Features" by estimating probabilities using <a href="https://en.wikipedia.org/wiki/Logistic_regression">a logistic function.</a> The model predicts a probability, which is used to predict the label class.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image13-1603903017520.png" alt=""></p> <p>In our text classification case, logistic regression tries to predict the probability of a review text being positive or negative, given the label and feature vector of TF-IDF values.  Logistic regression finds the best fit weight for each word in the collection of text by multiplying each TF-IDF feature by a weight and passing the sum through a sigmoid function, which transforms the input x into the output y, a number between 0 and 1.  In other words, logistic regression can be understood as finding the <a href="http://logisticregressionanalysis.com/86-what-is-logistic-regression/">parameters that best fit:</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image16-1603903026176.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image19-1603903034716.png" alt=""></p> <p>Logistic regression has the following advantages:</p> <ul> <li>Can handle sparse data</li> <li>Fast to train</li> <li>Weights can be interpreted</li> <li>Positive weights will correspond to the words that are positive</li> <li>Negative weights will correspond to the words that are negative</li> </ul> <h2>Data Exploration and Feature Extraction</h2> <p>We will be using a dataset of Amazon sports and outdoor products review data, which you can download here: <a href="http://jmcauley.ucsd.edu/data/amazon/">http://jmcauley.ucsd.edu/data/amazon/</a>. The dataset has the following schema:<br> *<em>Italicized fields are for sentiment analysis</em></p> <p><code>reviewerID</code> - ID of the reviewer, e.g., A2SUAM1J3GNN3B<br> <code>asin</code> - ID of the product, e.g., 0000013714<br> <code>reviewerName</code> - name of the reviewer<br> <code>helpful</code> - helpfulness rating of the review, e.g., 2/3<br> *<code>reviewText</code> - <em>text of the review</em><br> *<code>overall</code> - <em>rating of the product</em><br> *<code>summary</code> - <em>summary of the review</em><br> <code>unixReviewTime</code> - time of the review (Unix time)<br> <code>reviewTime</code> - time of the review (raw)</p> <p>The dataset has the following JSON format:</p> <pre><code class="language-json">{ "reviewerID": "A1PUWI9RTQV19S", "asin": "B003Y5C132", "reviewerName": "kris", "helpful": [0, 1], "reviewText": "A little small in hind sight, but I did order a .30 cal box. Good condition, and keeps my ammo organized.", "overall": 5.0, "summary": "Nice ammo can", "unixReviewTime": 1384905600, "reviewTime": "11 20, 2013" } </code></pre> <p>In this scenario, we will use logistic regression to predict the label of positive or not, based on the following:</p> <p>Label :</p> <ul> <li>overall - rating of the product 4-5  = 1 Positive</li> <li>overall - rating of the product 1-2  =  0 Negative</li> </ul> <p>Features :</p> <ul> <li>reviewText + summary  of the review → TF-IDF features</li> </ul> <h2>USING THE SPARK ML PACKAGE</h2> <p>Spark ML provides a uniform set of high-level APIs, built on top of DataFrames with the goal of making machine learning scalable and easy. Having ML APIs built on top of DataFrames provides the scalability of partitioned data processing with the ease of SQL for data manipulation.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image9-1603903043455.png" alt=""></p> <p>We will use an ML Pipeline to pass the data through transformers in order to extract the features and an estimator to produce the model.</p> <ul> <li>Transformer: A transformer is an algorithm that transforms one <code>DataFrame</code> into another <code>DataFrame</code>. We will use transformers to get a <code>DataFrame</code> with a features vector column.</li> <li>Estimator: An estimator is an algorithm that can be fit on a <code>DataFrame</code> to produce a transformer. We will use a an estimator to train a model, which can transform input data to get predictions.</li> <li>Pipeline: A pipeline chains multiple transformers and estimators together to specify an ML workflow.</li> </ul> <h2>Load the Data from a File into a DataFrame</h2> <p>The first step is to load our data into a <code>DataFrame</code>. Below, we <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#23manually-specifying-options">specify the data source format and path to load into a <code>DataFrame</code></a>.  Next, we use the <code>withColumn</code> method to add a column combining the review summary with the review text, and we drop columns that are not needed.</p> <pre><code class="language-scala">import org.apache.spark._ import org.apache.spark.ml._ import org.apache.spark.sql._ var file ="/user/mapr/data/revsporttrain.json" val df0 = spark.read.format("json") .option("inferSchema", "true") .load(file) val df = df0.withColumn("reviewTS", concat($"summary", lit(" "),$"reviewText")) .drop("helpful") .drop("reviewerID") .drop("reviewerName") .drop("reviewTime") </code></pre> <p>The <code>DataFrame printSchema</code> displays the schema:</p> <pre><code class="language-scala">df.printSchema root |-- asin: string (nullable = true) |-- overall: double (nullable = true) |-- reviewText: string (nullable = true) |-- summary: string (nullable = true) |-- unixReviewTime: long (nullable = true) |-- reviewTS: string (nullable = true) </code></pre> <p>The <code>DataFrame show</code> method displays the first 20 rows or the specified number of rows:</p> <pre><code class="language-scala">df.show(5) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image5-1603903053652.png" alt=""></p> <h2>Summary Statistics</h2> <p>Spark DataFrames include some <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#23pyspark.sql.DataFrame">built-in functions</a> for statistical processing. The <code>describe()</code> function performs summary statistics calculations on all numeric columns and returns them as a <code>DataFrame</code>. Below, we analyze the product rating overall column:</p> <pre><code class="language-scala">df.describe("overall").show **result:** +-------+------------------+ |**summary**| **overall**| +-------+------------------+ | **count**| **200000**| | **mean**| **4.395105**| | **stddev**|**0.9894654790262587**| | **min**| **1.0**| | **max**| **5.0**| +-------+------------------+ </code></pre> <p>In the code below, we filter to remove neutral ratings (=3), then a Spark <a href="https://spark.apache.org/docs/2.2.0/ml-features.html#23bucketizer">Bucketizer</a> is used to add a label 0/1 column to the dataset for Positive (overall rating >=4) and not positive (overall rating &#x3C;4) reviews. Then, the resulting total counts are displayed. Grouping the data by the label column and counting the number of instances in each group shows that there are roughly 13 times as many positive samples as not positive samples.</p> <pre><code class="language-scala">val df1 = df.filter("overall !=3") val bucketizer = new Bucketizer() .setInputCol("overall") .setOutputCol("label") .setSplits(Array(Double.NegativeInfinity, 4.0, Double.PositiveInfinity)) val df2= bucketizer.transform(df1) df2.groupBy("overall","label").count.show **result:** +-------+-----+------+ |**overall**|**label**| **count**| +-------+-----+------+ | **2.0**| **0.0**| **6916**| | **5.0**| **1.0**|**127515**| | **1.0**| **0.0**| **6198**| | **4.0**| **1.0**| **43303**| +-------+-----+------+ </code></pre> <h2>Stratified Sampling</h2> <p>In order to ensure that our model is sensitive to the negative samples, we can put the two sample types on the same footing using stratified sampling. The DataFrames <code>sampleBy()</code> function does this when provided with fractions of each sample type to be returned. Here, we're keeping all instances of negative, but downsampling the  negative instances to 10%, then displaying the results.</p> <pre><code class="language-scala">val fractions = Map(1.0 -> .1, 0.0 -> 1.0) val df3 = df2.stat.sampleBy("label", fractions, 36L) df3.groupBy("label").count.show **result:** +-----+-----+ |label|count| +-----+-----+ | 0.0|13114| | 1.0|17086| +-----+-----+ </code></pre> <p>Below, the data is split into a training data set and a test data set: 80% of the data is used to train the model, and 20% will be used for testing.</p> <pre><code class="language-scala">// split into training and test dataset val splitSeed = 5043 val Array(trainingData, testData) = df3.randomSplit(Array(0.8, 0.2), splitSeed) </code></pre> <h2>Feature Extraction and Pipelining</h2> <p>The ML package needs the label and feature vector to be added as columns to the input <code>DataFrame</code>. We set up a pipeline to pass the data through transformers in order to extract the features and label.</p> <p>The <code>RegexTokenizer</code> takes an input text column and returns a <code>DataFrame</code> with an additional column of the text split into an array of words by using the provided regex pattern.   The <code>StopWordsRemover</code> filters out words which should be excluded, because the words appear frequently and don't carry as much meaning – for example, 'I,' 'is,' 'the.'</p> <p>In the code below, the <code>RegexTokenizer</code> will split up the column with the review and summary text into a column with an array of words, which will then be filtered by the <code>StopWordsRemover</code>:</p> <pre><code class="language-scala">val tokenizer = new RegexTokenizer() .setInputCol("reviewTS") .setOutputCol("reviewTokensUf") .setPattern("\\s+|[,.()\"]") val remover = new StopWordsRemover() .setStopWords(StopWordsRemover .loadDefaultStopWords("english")) .setInputCol("reviewTokensUf") .setOutputCol("reviewTokens") </code></pre> <p>An example of  the results of the <code>RegexTokenizer</code> and <code>StopWordsRemover</code>, taking as input column <code>reviewTS</code> and adding the <code>reviewTokens</code> column of filtered words, is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image11-1603903062165.png" alt=""></p> <table> <thead> <tr> <th align="left">reviewTS</th> <th align="left">reviewTokens</th> </tr> </thead> <tbody> <tr> <td align="left">resistance was good but quality wasn't So it worked well for a couple weeks, but during a lunge workout, it snapped on me. I liked it and thought it was a great product until this happened. I noticed small rips on the band. This could have been the issue.</td> <td align="left">Array(resistance, good, quality, worked, well, couple, weeks, lunge, workout, snapped, liked, thought, great, product, happened, noticed, small, rips, band, issue)</td> </tr> </tbody> </table> <p>A <code>CountVectorizer</code> is used to convert the array of word tokens from the previous step to vectors of word token counts.  The <code>CountVectorizer</code> is performing the TF part of TF-IDF feature extraction.</p> <pre><code class="language-scala">val cv = new CountVectorizer() .setInputCol("reviewTokens") .setOutputCol("cv") .setVocabSize(200000) </code></pre> <p>An example of  the results of the <code>CountVectorizer</code>, taking as input column <code>reviewTokens</code> and adding the <code>cv</code> column of vectorized word counts, is shown below.  In the <code>cv</code> column: 56004 is the size of the TF word vocabulary; the second array is the position of the word in the word vocabulary ordered by term frequency across the corpus; the third array is the count of the word (TF) in the <code>reviewTokens</code> text.</p> <table> <thead> <tr> <th align="left">reviewTokens</th> <th align="left">cv</th> </tr> </thead> <tbody> <tr> <td align="left">Array(resistance, good, quality, worked, well, couple, weeks, lunge, workout, snapped, liked, thought, great, product, happened, noticed, small, rips, band, issue)</td> <td align="left"><span style="overflow-wrap: anywhere;">(56004,[1,2,6,8,13,31,163,168,192,276,487,518,589,643,770,955,1194,1297,4178,19185],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])</span></td> </tr> </tbody> </table> <p>Below the <code>cv</code> column created by the <code>CountVectorizer</code> (the TF part of TF-IDF feature extraction) is the input for IDF.  IDF takes feature vectors created from the <code>CountVectorizer</code> and down-weights features which appear frequently in a collection of texts (the IDF part of TF-IDF feature extraction). The output <code>features</code> column is the TF-IDF features vector, which the logistic regression function will use.</p> <pre><code class="language-scala">// list of feature columns val idf = new IDF() .setInputCol("cv") .setOutputCol("features") </code></pre> <p>An example of the results of the IDF, taking as input column <code>cv</code> and adding the <code>features</code> column of vectorized TF-IDF, is shown below. In the <code>cv</code> column, 56004 is the size of the word vocabulary; the second array is the position of the word in the word vocabulary ordered by term frequency across the corpus; the third array is the TF-IDF of the word in the reviewTokens text.</p> <table> <thead> <tr> <th align="left">cv</th> <th align="left">features</th> </tr> </thead> <tbody> <tr> <td align="left"><span style="overflow-wrap: anywhere;"> (56004,[1,2,6,8,13,31,163,168,192,276,487,518,589,643,770,955,1194,1297,4178,19185],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])</span></td> <td align="left"><span style="overflow-wrap: anywhere;">(56004,[1,2,6,8,13,31,163,168,192,276,487,518,589,643,770,955,1194,1297,4178,19185],[1.3167453737971118,1.3189162538557524,1.5214341820160893,1.9425118863569042,2.052613811061827,2.3350290362765134,3.188779919701724,3.245760634740672,3.316430208091361,3.620260266951124,4.115700971877636,4.165254786332365,4.655788580192657,4.32745920096672,4.781242886345692,5.001914248514512,5.008106218762434,5.169529657918772,6.793673717742568,8.990898295078788])</span></td> </tr> </tbody> </table> <p>The final element in our pipeline is an estimator, a logistic regression classifier, which will train on the vector of labels and features and return a (transformer) model.</p> <pre><code class="language-scala">// create Logistic Regression estimator // regularizer parameters avoid overfitting val lr = new LogisticRegression() .setMaxIter(100) .setRegParam(0.02) .setElasticNetParam(0.3) </code></pre> <p>Below, we put the <code>Tokenizer</code>, <code>CountVectorizer</code>, IDF,  and Logistic Regression Classifier in a pipeline.  A pipeline chains multiple transformers and estimators together to specify an ML workflow for training and using a model.</p> <pre><code class="language-scala">val steps = Array( tokenizer, remover, cv, idf,lr) val pipeline = new Pipeline().setStages(steps) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image10-1603903070796.png" alt=""></p> <h2>TRAIN THE MODEL</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image17-1603903079716.png" alt=""></p> <p>Next, we train the logistic regression model <a href="https://spark.apache.org/docs/latest/ml-classification-regression.html#23logistic-regression">with elastic net regularization</a>. The model is trained by making associations between the input features and the labeled output associated with those features. The <code>pipeline.fit</code> method returns a fitted pipeline model.</p> <pre><code class="language-scala">val model = pipeline.fit(trainingData) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image6-1603903088693.png" alt=""></p> <p>Note: another option for training the model is to tune the parameters, using grid search, and select the best model, using k-fold cross validation with a Spark CrossValidator and a ParamGridBuilder.</p> <p>Next, we can get the <code>CountVectorizer</code> and <code>LogisticRegression</code> model from the fitted pipeline model, in order to print out the coefficient weights of the words in the text vocabulary (the word feature importance).</p> <pre><code class="language-scala">// get vocabulary from the CountVectorizer val vocabulary = model.stages(2) .asInstanceOf[CountVectorizerModel] .vocabulary // get the logistic regression model val lrModel = model.stages.last .asInstanceOf[LogisticRegressionModel] // Get array of coefficient weights val weights = lrModel.coefficients.toArray // create array of word and corresponding weight val word_weight = vocabulary.zip(weights) // create a dataframe with word and weights columns val cdf = sc.parallelize(word_weight) .toDF("word","weights") </code></pre> <p>Recall that logistic regression generates the coefficient weights of a formula to predict the probability of occurrence of the feature x (in this case, a word) to maximize the probability of the outcome Y, 1 or 0 (in this case, positive or negative text sentiment). The weights can be interpreted:</p> <ul> <li>Positive weights will correspond to the words that are positive</li> <li>Negative weights will correspond to the words that are negative</li> </ul> <p>Below, we sort the weights in descending order to show the most positive words.  The results show that 'great,' perfect,' 'easy,' 'works,' and 'excellent' are the most important positive words.</p> <pre><code class="language-scala">// show the most positive weighted words cdf.orderBy(desc("weights")).show(10) **result:** +---------+-------------------+ | **word**| **weight**| +---------+-------------------+ | **great**| **0.6078697902359276**| | **perfect**|**0.34404726951273945**| |**excellent**|**0.28217372351853814**| | **easy**|**0.26293906850341764**| | **love**|**0.23518819188672227**| | **works**| **0.229342771859023**| | **good**| **0.2116386469012886**| | **highly**| **0.2044040462730194**| | **nice**|**0.20088266981583622**| | **best**|**0.18194893152633945**| +---------+-------------------+ </code></pre> <p>Below, we sort the weights in ascending order to show the most negative words.The results show that 'returned,' 'poor,' 'waste,' and 'useless' are the most important negative words.</p> <pre><code class="language-scala">// show the most negative sentiment words cdf.orderBy("weights").show(10) **result:** +-------------+--------------------+ | **word**| **weight**| +-------------+--------------------+ | **returned**|**-0.38185206877117467**| | **poor**|**-0.35366409294425644**| | **waste**| **-0.3159724826017525**| | **useless**| **-0.2914292653060789**| | **return**| **-0.2724012497362986**| |**disappointing**| **-0.2666580559444479**| | **broke**| **-0.2656765359468423**| | **disappointed**|**-0.23852780960293438**| | **returning**|**-0.22432617475366876**| | **junk**|**-0.21457169691127467**| +-------------+--------------------+ </code></pre> <h2>Predictions and Model Evaluation</h2> <p>The performance of the model can be determined, using the test data set that has not been used for any training. We transform the test <code>DataFrame</code> with the pipeline model, which will pass the test data, according to the pipeline steps, through the feature extraction stage, estimate with the logistic regression model, and then return the label predictions in a column of a new <code>DataFrame</code>.</p> <pre><code class="language-scala">val predictions = model.transform(testData) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image4-1603903098021.png" alt=""></p> <p>The <code>BinaryClassificationEvaluator</code> provides a metric to measure how well a fitted model does on the test data. The default metric for this evaluator is the area under the ROC curve. The area measures the ability of the test to correctly classify true positives from false positives. A random predictor would have .5. The closer the value is to 1, the better its predictions are.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image14-1603903107176.png" alt=""></p> <p>Below, we pass the predictions <code>DataFrame</code> (which has a <code>rawPrediction</code> column and a label column) to the <code>BinaryClassificationEvaluator</code>, which returns .93 as the area under the ROC curve.  </p> <pre><code class="language-scala">val evaluator = new BinaryClassificationEvaluator() val areaUnderROC = evaluator.evaluate(predictions) result: 0.9350783400583272 </code></pre> <p>Below, we calculate some more metrics. The number of false/true positives and negative predictions is also useful:</p> <ul> <li>True positives are how often the model correctly predicts positive sentiment.</li> <li>False positives are how often the model incorrectly predicts positive sentiment..</li> <li>True negatives indicate how often the model correctly predicts negative sentiment.</li> <li>False negatives indicate how often the model incorrectly predicts negative sentiment.</li> </ul> <pre><code class="language-scala">val lp = predictions.select("label", "prediction") val counttotal = predictions.count() val correct = lp.filter($"label" === $"prediction").count() val wrong = lp.filter(not($"label" === $"prediction")).count() val ratioWrong = wrong.toDouble / counttotal.toDouble val lp = predictions.select( "prediction","label") val counttotal = predictions.count().toDouble val correct = lp.filter($"label" === $"prediction") .count() val wrong = lp.filter("label != prediction") .count() val ratioWrong=wrong/counttotal val ratioCorrect=correct/counttotal val truen =( lp.filter($"label" === 0.0) .filter($"label" === $"prediction") .count()) /counttotal val truep = (lp.filter($"label" === 1.0) .filter($"label" === $"prediction") .count())/counttotal val falsen = (lp.filter($"label" === 0.0) .filter(not($"label" === $"prediction")) .count())/counttotal val falsep = (lp.filter($"label" === 1.0) .filter(not($"label" === $"prediction")) .count())/counttotal val precision= truep / (truep + falsep) val recall= truep / (truep + falsen) val fmeasure= 2 * precision * recall / (precision + recall) val accuracy=(truep + truen) / (truep + truen + falsep + falsen) **result:** **counttotal: 6112.0** **correct: 5290.0** **wrong: 822.0** **ratioWrong: 0.13448952879581152** **ratioCorrect: 0.8655104712041884** **truen: 0.3417866492146597** **truep: 0.5237238219895288** **falsen: 0.044829842931937175** **falsep: 0.08965968586387435** **precision: 0.8538276873833023** **recall: 0.9211510791366907** **fmeasure: 0.8862126245847176** **accuracy: 0.8655104712041886** </code></pre> <p>Below, we print out the summary and review token words for the reviews with the highest probability of a negative sentiment:</p> <pre><code class="language-scala">predictions.filter($"prediction" === 0.0) .select("summary","reviewTokens","overall","prediction") .orderBy(desc("rawPrediction")).show(5) result: +--------------------+--------------------+-------+----------+ | summary| reviewTokens|overall|prediction| +--------------------+--------------------+-------+----------+ | Worthless Garbage!|[worthless, garba...| 1.0| 0.0| |Decent but failin...|[decent, failing,...| 1.0| 0.0| |over rated and po...|[rated, poorly, m...| 2.0| 0.0| |dont waste your m...|[dont, waste, mon...| 1.0| 0.0| |Cheap Chinese JUNK! |[cheap, chinese,....| 1.0| 0.0| +--------------------+--------------------+-------+----------+ </code></pre> <p>Below we print out the summary and review token words for the reviews with the highest probability of a positive sentiment:</p> <pre><code class="language-scala">predictions.filter($"prediction" === 1.0) .select("summary","reviewTokens","overall","prediction") .orderBy("rawPrediction").show(5) **result:** +--------------------+--------------------+-------+----------+ | summary| reviewTokens|overall|prediction| +--------------------+--------------------+-------+----------+ | great|[great, excellent...| 5.0| 1.0| |Outstanding Purchase|[outstanding, pur...| 5.0| 1.0| |A fantastic stov....|[fantastic, stov....| 5.0| 1.0| |Small But Delight...|[small, delightfu...| 5.0| 1.0| |Kabar made a good...|[kabar, made, goo...| 5.0| 1.0| +--------------------+--------------------+-------+----------+ </code></pre> <h2>Saving the Model</h2> <p>We can now save our fitted pipeline model to the distributed file store for later use in production. This saves both the feature extraction stage and the logistic regression model in the pipeline.</p> <pre><code class="language-scala">var dir = "/user/mapr/sentmodel/" model.write.overwrite().save(dir) </code></pre> <p>The result of saving the pipeline model is a JSON file for metadata and Parquet files for model data. We can reload the model with the load command; the original and reloaded models are the same:</p> <pre><code class="language-scala">val sameModel = org.apache.spark.ml.PipelineModel.load(modeldirectory) </code></pre> <h2>Summary</h2> <p>There are plenty of great tools to build classification models. Apache Spark provides an excellent framework for building solutions to business problems that can extract value from massive, distributed datasets.</p> <p>Machine learning algorithms cannot answer all questions perfectly. But they do provide evidence for humans to consider when interpreting results, assuming the right question is asked in the first place.</p> <h2>Code</h2> <p>All of the data and code to train the models and make your own conclusions, using Apache Spark, are located in GitHub; refer to GitHub "README" for more information about running the code.</p> <ul> <li><a href="https://github.com/caroljmcdonald/mapr-sparkml-sentiment-classification">https://github.com/caroljmcdonald/mapr-sparkml-sentiment-classification</a></li> </ul>https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/https://developer.hpe.com/fast-data-processing-pipeline-for-predicting-flight-delays-using-apache-/Wed, 21 Oct 2020 05:53:58 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>""authorDisplayName": "Carol McDonald", "publish": "2018-01-10T10:30:00.000Z", "tags": "use-cases" </code></pre> <hr> <p>According to <a href="https://hbr.org/2017/06/how-analytics-has-changed-in-the-last-10-years-and-how-its-stayed-the-same">Thomas Davenport in the HBR</a>, analytical technology has changed dramatically over the last decade, with more powerful and less expensive distributed computing across commodity servers, streaming analytics, and improved machine learning technologies, enabling companies to store and analyze both far more data and many different types of it. Werner Vogel stated in his recent <a href="https://www.infoq.com/news/2017/12/vogels-21st-century-architecture">keynote at AWS re:invent</a> that key technology drivers of today are data, the Internet of Things (IoT), and machine learning.</p> <p>Leveraging the huge amounts of data coming from the Internet of Things requires processing events in real time, applying machine learning to add value, and scalable fast storage. This is the first in a series of blogs, which discusses the architecture of an end-to-end application that combines streaming data with machine learning and fast storage. In this first post, I’ll help you get started using Apache Spark’s <a href="https://spark.apache.org/docs/latest/ml-pipeline.html">ML pipelines</a> with a <a href="https://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-classifier">Decision Tree Classifier</a> to predict flight delays.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/spark-rdd-based-api-1603259821463.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/machine-learning-logistics-and-streaming-architecture-1603259842484.png" alt=""></p> <h2>What is Machine Learning?</h2> <p>Machine learning uses algorithms to find patterns in data, and then uses a model that recognizes those patterns to make predictions on new data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/predictions-1603259860781.png" alt=""></p> <p>There are typically two phases in machine learning with real-time data:</p> <ul> <li>Data Discovery: The first phase involves analysis on historical data to build the machine learning model.</li> <li>Analytics Using the Model: The second phase uses the model in production on live events. (Note that Spark does provide some streaming machine learning algorithms, but you still often need to do an analysis of historical data.)</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/historical-data-1603259872698.png" alt=""></p> <p>In general, machine learning may be broken down into two-three types: supervised, unsupervised, and in between those two.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/machine-learning-1603259887263.png" alt=""></p> <p>Supervised algorithms use labeled data in which both the input and target outcome, or label, are provided to the algorithm.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/labeled-data-1603259898645.png" alt=""></p> <p>Supervised Learning is also called predictive modeling or predictive analytics, because you build a model that is capable of making predictions.</p> <p>Unsupervised learning algorithms find patterns in unlabeled data. Semi-supervised learning uses a mixture of labeled and unlabeled data. Reinforcement learning trains algorithms to maximize rewards based on feedback.</p> <h2>Three Categories of Techniques for Machine Learning</h2> <p>Three common categories of machine learning techniques are Classification, Clustering and Collaborative Filtering.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/three-common-catgories-1603259911430.png" alt=""></p> <ul> <li>Classification: Gmail uses a machine learning technique called classification to designate if an email is spam or not, based on the data of an email: the sender, recipients, subject, and message body. Classification takes a set of data with known labels and learns how to label new records based on that information.</li> <li>Clustering: Google News uses a technique called clustering to group news articles into different categories, based on title and content. Clustering algorithms discover groupings that occur in collections of data.</li> <li>Collaborative Filtering: Amazon uses a machine learning technique called collaborative filtering (commonly referred to as recommendation), to determine which products users will like based on their history and similarity to other users.</li> </ul> <p>In this example we will be using a supervised machine learning algorithm for classification of flight delays.</p> <h2>CLASSIFICATION</h2> <p>Classification is a family of supervised machine learning algorithms that identify which category an item belongs to (e.g., whether a transaction is fraud or not fraud), based on labeled examples of known items (e.g., transactions known to be fraud or not). Classification takes a set of data with known labels and pre-determined features and learns how to label new records based on that information. Features are the “if questions” that you ask. The label is the answer to those questions. In the example below, if it walks, swims, and quacks like a duck, then the label is "duck."</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/duck-1603259924086.png" alt=""></p> <p>Let’s go through an example for flight delays:</p> <ul> <li>What are we trying to predict? <ul> <li>Whether a flight will be delayed or not.</li> <li>Delayed is the Label: True or False</li> </ul> </li> <li>What are the “if questions” or properties that you can use to make predictions? <ul> <li>What is the Originating Airport?</li> <li>What is the Destination Airport?</li> <li>What is the Scheduled time of departure?</li> <li>What is the schedule time of arrival?</li> <li>What is the day of the week?</li> <li>What is the Airline Carrier?</li> </ul> </li> </ul> <h2>DECISION TREES</h2> <p>Decision trees create a model that predicts the label (or class) by evaluating a set of rules that follow an IF THEN ELSE…pattern. The If then ELSE feature questions are the nodes, and the answers True or false are the branches in the tree to the child nodes. A decision tree model estimates the minimum number of true/false questions needed, to assess the probability of making a correct decision. Below is an example of a simplified decision tree for flight delays:</p> <ul> <li>Q1: If the scheduled departure time &#x3C; 10:15 AM <ul> <li>T:Q2: If the originating airport is in the set {ORD, ATL,SFO} <ul> <li>T:Q3: If the day of the week is in {Monday, Sunday} <ul> <li>T:Delayed=1</li> <li>F: Delayed=0</li> </ul> </li> <li>F: Q3: If the destination airport is in the set {SFO,ORD,EWR} <ul> <li>T: Delayed=1</li> <li>F: Delayed=0</li> </ul> </li> </ul> </li> <li>F :Q2: If the originating airport is not in the set {BOS . MIA} <ul> <li>T:Q3: If the day of the week is in the set {Monday , Sunday} <ul> <li>T: Delayed=1</li> <li>F: Delayed=0</li> </ul> </li> <li>F: Q3: If the destination airport is not in the set {BOS . MIA} <ul> <li>T: Delayed=1</li> <li>F: Delayed=0</li> </ul> </li> </ul> </li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/decision-tree-1603259936871.png" alt=""></p> <h2>EXAMPLE USE CASE DATA SET</h2> <p>Our data is from <a href="http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&#x26;DB_Short_Name=On-Time">http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&#x26;DB_Short_Name=On-Time</a>. We are using flight information for January, February, March, April and May 2017. For each flight, we have the following information:</p> <table> <thead> <tr> <th><strong>Field</strong></th> <th><strong>Description</strong></th> <th><strong>Example Value</strong></th> </tr> </thead> <tbody> <tr> <td>id</td> <td>Unique identifier: composed of carrier code, date, origin, destination, flight number</td> <td>AA_2017-02-22_SFO_ORD_150</td> </tr> <tr> <td>dofW (Integer)</td> <td>Day of week (1=Monday 7=Sunday)</td> <td>1</td> </tr> <tr> <td>carrier (String)</td> <td>Carrier code</td> <td>AA</td> </tr> <tr> <td>origin(String)</td> <td>Origin Airport Code</td> <td>JFK</td> </tr> <tr> <td>dest(String</td> <td>Destination airport code</td> <td>LAX</td> </tr> <tr> <td>crsdephour(Integer)</td> <td>Scheduled departure hour</td> <td>9</td> </tr> <tr> <td>crsdeptime(Double)</td> <td>Scheduled departure time</td> <td>900.0</td> </tr> <tr> <td>depdelay (Double)</td> <td>Departure delay in minutes</td> <td>40.0</td> </tr> <tr> <td>crsarrtime (Double)</td> <td>Scheduled arrival time</td> <td>1230.0</td> </tr> <tr> <td>arrdelay (Double)</td> <td>Arrival delay minutes</td> <td>40.0</td> </tr> <tr> <td>crselapsedtime (Double)</td> <td>Elapsed time</td> <td>390.0</td> </tr> <tr> <td>dist (Double)</td> <td>Distance</td> <td>2475.0</td> </tr> </tbody> </table> <p>I have already cleaned up, limited the number of airports and carriers, and transformed the data into 2 JSON files, one for training and one for testing. (you can see the code for the cleanup in the github repository). The JSON file has the following format:</p> <pre><code class="language-json">{ "_id": "AA_2017-01-01_ATL_LGA_1678", "dofW": 7, "carrier": "AA", "origin": "ATL", "dest": "LGA", "crsdephour": 17, "crsdeptime": 1700, "depdelay": 0.0, "crsarrtime": 1912, "arrdelay": 0.0, "crselapsedtime": 132.0, "dist": 762.0 } </code></pre> <p>You can run the <a href="https://github.com/caroljmcdonald/spark-ml-flightdelay">code for this example</a> with MapR 5.2.1 or MapR 6.0 (which includes Spark 2.1).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/zepplin-notebook-1603259951128.png" alt=""></p> <h2>Load The Data From a File Into a Dataframe</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/scala-case-class-1603259961799.png" alt=""></p> <p>We use a Scala case class and <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema">Structype</a> to define the schema, corresponding to a line in the JSON data file.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/define-schema-for-json-file-data-1603259974595.png" alt=""></p> <p>Below we <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#manually-specifying-options">specify the data source, schema and class to load into a Dataset</a>. We load the data from January and February, which we will use for training the model. (Note that specifying the schema when loading data into a DataFrame will <a href="http://data-informed.com/6-steps-to-get-top-performance-from-the-changes-in-spark-2-0/">give better performance</a> than schema inference.}</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/read-data-json-file-into-dataset-1603259991766.png" alt=""></p> <p>The Dataframe show method displays the first 20 rows:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/dataframe-show-method-1603260003975.png" alt=""></p> <p>Here we load data from March and April which we will use for testing the model:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/march-april-data-1603260015870.png" alt=""></p> <h2>Summary Statistics</h2> <p>Spark DataFrames include some <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame">built-in functions</a> for statistical processing. The describe() function performs summary statistics calculations on all numeric columns and returns them as a DataFrame.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/perform-summary-statistics-1603260030384.png" alt=""></p> <h2>Data Exploration</h2> <p>We can use Spark SQL to explore the dataset. Here are some example queries using the Spark SQL:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/register-dataset-as-a-temporary-view-1603260043715.png" alt=""></p> <p>Below we display information for the longest departure delays:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/top-5-longest-departure-delays-1603260056308.png" alt=""></p> <p>Below we display the average departure delay by Carrier:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/average-departure-delay-1603260067246.png" alt=""></p> <p>We want to predict flight delays where depdelay > 40 minutes, so let’s explore this data. Below we see that United Airlines and Delta have the highest count of flight delays for Jan &#x26; Feb 2017 (the training set).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/count-of-departure-delays-by-carrier-1603260084783.png" alt=""></p> <p>In the query below we see that Monday and Sunday have the highest count of flight delays.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/count-of-departure-delays-by-day-of-the-week-1603260094738.png" alt=""></p> <p>In the query below we see that the hours between 13:00-19:00 have the highest count of flight delays.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/count-of-departure-delays-by-hour-of-day-1603260105210.png" alt=""></p> <p>In the query below we see that the origin airports Chicago and Atlanta have the highest count of flight delays.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/count-of-departure-delays-by-origin-1603260115273.png" alt=""></p> <p>In the query below we see that the destination airports San Francisco and Newark have the highest count of flight delays.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/count-of-departure-delays-by-destination-1603260125772.png" alt=""></p> <p>In the query below we see the count of departure delays by Origin and destination. The routes ORD->SFO and DEN->SFO have the highest delays, maybe because of weather in January and February? Adding weather to this dataset would give better results, but that is left as an exercise for the reader.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/delays-by-origin-destination-1603260137188.png" alt=""></p> <p>In the code below a Spark <a href="https://spark.apache.org/docs/2.2.0/ml-features.html#bucketizer">Bucketizer</a> is used to split the dataset into delayed and not delayed flights with a delayed 0/1 column. Then the resulting total counts are displayed. Grouping the data by the delayed field and counting the number of instances in each group shows that there are roughly 8 times as many not delayed samples as delayed samples.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/add-labels-for-delayed-flights-1603260149465.png" alt=""></p> <p>In the query below we see the count of not delayed (0=dark blue) and delayed (1=light blue) flights by origin.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/delays-by-origin-1603260162630.png" alt=""></p> <h2>Stratified Sampling</h2> <p>In order to ensure that our model is sensitive to the delayed samples we can put the two sample types on the same footing using stratified sampling. The DataFrames sampleBy() function does this when provided with fractions of each sample type to be returned. Here, we're keeping all instances of delayed, but downsampling the not delayed instances to 29%, then displaying the results</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/stratify-the-sampling-to-fewr-not-delayed-1603260173743.png" alt=""></p> <h2>FEATURES ARRAY</h2> <p>To build a classifier model, you extract the features that most contribute to the classification. In this scenario, we will build a tree to predict the label of delayed or not based on the following features:</p> <ul> <li>Label → delayed = 0 <ul> <li>Delayed = 1 if delay > 40 minutes</li> </ul> </li> <li>Features → {day of the week, scheduled departure time, scheduled arrival time, carrier, scheduled elapsed time, origin, destination, distance}</li> </ul> <table> <thead> <tr> <th><strong>delayed</strong></th> <th><strong>dofW</strong></th> <th><strong>crsdepTime</strong></th> <th><strong>crsArrTime</strong></th> <th><strong>carrier</strong></th> <th><strong>elapTime</strong></th> <th><strong>origin</strong></th> <th><strong>dest</strong></th> <th><strong>dist</strong></th> </tr> </thead> <tbody> <tr> <td>1.0/0.0</td> <td>1</td> <td>1015</td> <td>1230</td> <td>AA</td> <td>385.0</td> <td>JKF</td> <td>LAX</td> <td>2475.0</td> </tr> </tbody> </table> <p>In order for the features to be used by a machine learning algorithm, they must be transformed and put into Feature Vectors, which are vectors of numbers representing the value for each feature.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/reference-learning-spark-1603260186018.png" alt=""></p> <h2>USING THE SPARK ML PACKAGE</h2> <p>The <a href="http://spark.apache.org/docs/latest/ml-guide.html">ML package</a> is the newer library of machine learning routines. <a href="http://spark.apache.org/docs/latest/ml-pipeline.html#pipeline-components">Spark ML provides a uniform set of high-level APIs built on top of DataFrames.</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/ml-pipeline-1603260196123.png" alt=""></p> <p>We will use an ML Pipeline to pass the data through transformers in order to extract the features and an estimator to produce the model.</p> <ul> <li>Transformer: A Transformer is an algorithm which transforms one DataFrame into another DataFrame. We will use a transformer to get a DataFrame with a features vector column.</li> <li>Estimator: An Estimator is an algorithm which can be fit on a DataFrame to produce a Transformer. We will use a an estimator to train a model which can transform data to get predictions.</li> <li>Pipeline: A Pipeline chains multiple Transformers and Estimators together to specify a ML workflow.</li> </ul> <h2>Feature Extraction and Pipelining</h2> <p>The ML package needs the label and feature vector to be added as columns to the input dataframe. We set up a pipeline to pass the data through transformers in order to extract the features and label. We use a StringIndexer to encode a string columns to a column of number indices. We use a OneHotEncoder to map a number indices column to a column of binary vectors, with at most a single one-value. Encoding categorical features allows decision trees to treat categorical features appropriately, improving performance. An example of StringIndexing and OneHotEncoding for carrier is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/example-stringindexing-1603260208808.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/combination-stringindexer-onehotencoder-1603260221098.png" alt=""></p> <p>Below a Bucketizer is used to add a label of delayed 0/1. The VectorAssembler combines a given list of columns into a single feature vector column.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/vectorassembler-transformer-1603260233625.png" alt=""></p> <p>The result of running these transformers in a pipeline will be to add a label and features column to the dataset as shown below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/add-label-features-column-1603260245500.png" alt=""></p> <p>The final element in our pipeline is an estimator (a decision tree classifier), training on the vector of labels and features.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/create-decision-tree-estimator-1603260259278.png" alt=""></p> <p>Below we chain the indexers and tree in a Pipeline.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/chain-indexers-1603260267909.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/tree-pipeline-1603260278538.png" alt=""></p> <h2>TRAIN THE MODEL</h2> <p>We would like to determine which parameter values of the decision tree produce the best model. A common technique for model selection is k-fold cross validation, where the data is randomly split into k partitions. Each partition is used once as the testing data set, while the rest are used for training. Models are then generated using the training sets and evaluated with the testing sets, resulting in k model performance measurements. The model parameters leading to the highest performance metric produce the best model.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/ml-cross-validation-process-1603260289452.png" alt=""></p> <p>Spark ML supports k-fold cross validation with a transformation/estimation pipeline to try out different combinations of parameters, using a process called grid search, where you set up the parameters to test, and a cross validation evaluator to construct a model selection workflow.</p> <p>Below, we use a ParamGridBuilder to construct the parameter grid. We define an Evaluator, which will evaluate the model by comparing the test label column with the test prediction column. We use a CrossValidator for model selection.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/setup-crossvalidator-with-parameters-1603260303905.png" alt=""></p> <p>The CrossValidator uses the Estimator Pipeline, the Parameter Grid, and the Classification Evaluator to fit the training data set and returns a model.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/use-cross-validator-estimator-1603260318523.png" alt=""></p> <p>The CrossValidator uses the ParamGridBuilder to iterate through the maxDepth parameter of the decision tree and evaluate the models, repeating 3 times per parameter value for reliable results.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/crossvalidator-1603260329067.png" alt=""></p> <p>Next, we can get the best decision tree model, in order to print out the decision tree and feature importances. (Note that the OneHotEncoders increases the number of features. In order to understand this printout better I built a tree without the encoders, which gave a slightly lower accuracy)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/get-the-best-decision-tree-model-1603260338670.png" alt=""></p> <p>We find that the best tree model produced using the cross-validation process is one with a depth of 6. The toDebugString() function provides a print of the tree's decision nodes and final prediction outcomes at the end leaves. Below is a partial printout of the decision tree:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/treemodel-todebugstring-1603260350331.png" alt=""></p> <p>The features numbers correspond to the following:</p> <p><code>( 0=carrierIndexed, 1=destIndexed, 2=originIndexed, 3=dofW, 4=crsdephour, 5=crselapsedtime, 6=crsarrtime, 7=crsdeptime, 8=dist)</code></p> <p>Below we can see that the feature importance in order is</p> <ol> <li>scheduled departure time (feature 7)</li> <li>destination (feature 1)</li> <li>origin (feature 2)</li> <li>scheduled arrival time (feature 6)</li> <li>day of the week (feature 3)</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/featureimportances-1603260369952.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/features-importances-1603260389746.png" alt=""></p> <p>Decision trees are often used for feature selection because they provide an automated mechanism for determining the most important features (those closest to the tree root).</p> <h2>Predictions and Model Evaluation</h2> <p>The actual performance of the model can be determined using the test data set that has not been used for any training or cross-validation activities.</p> <p>We transform the test Dataframe with the model pipeline, which will tranform the features according to the pipeline, estimate and then return the label predictions in a column of a new dataframe.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/dataframe-1603260403535.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/get-predictions-from-test-dataset-1603260414077.png" alt=""></p> <p>The evaluator will provide us with the score of the predictions. Accuracy is measured by the area under the ROC curve. The area measures the ability of the test to correctly classify true positives from false positives. A random predictor would have .5 accuracy. The closer the value is to 1 the better its predictions are. In this case, the evaluation returns 81% precision.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/evaluator-predictions-1603260427433.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/evaluate-predictions-accuarcy-1603260438521.png" alt=""></p> <p>Below, we calculate some more metrics. The number of false/true positive and negative predictions is also useful:</p> <ul> <li>True positives are how often the model correctly predicted delayed flights.</li> <li>False positives are how often the model incorrectly predicted delayed flights.</li> <li>True negatives indicate how often the model correctly predicted not delayed flights.</li> <li>False negatives indicate how often the model incorrectly predicted not delayed flights.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/calculate-some-more-metrics-1603260455772.png" alt=""></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/calculate-some-more-metrics-2-1603260464963.png" alt=""></p> <h2>SAVE THE MODEL</h2> <p>We can now save our fitted Pipeline for later use with streaming events. This saves both the feature extraction stage and the decision tree model chosen by model tuning.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/save-model-to-file-system-1603260479132.png" alt=""></p> <p>The result of saving the pipeline model is a JSON file for metadata and a Parquet for model data. We can re-load the model with the load command , the original and re-loaded models are the same:</p> <p><code>val sameCVModel = CrossValidatorModel.load(“../cfModel")</code></p> <h2>Code</h2> <ul> <li>You can download the code and data to run these examples from here: <a href="https://github.com/caroljmcdonald/spark-ml-flightdelay">https://github.com/caroljmcdonald/spark-ml-flightdelay</a></li> <li><a href="https://github.com/caroljmcdonald/spark-ml-flightdelay/blob/master/notebooks/sparkmlpipelineflightdelays.json">Zeppelin Notebook for the code</a></li> </ul> <h2>Running the Code</h2> <p>All of the components of the use case architecture we just discussed can run on the same cluster with the MapR Data Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/mapr-cdp-1603260494864.png" alt=""></p>https://developer.hpe.com/hpe-firmware-updates-part-3-the-redfish-update-service/https://developer.hpe.com/hpe-firmware-updates-part-3-the-redfish-update-service/Mon, 19 Oct 2020 13:25:05 GMT<br> <p><big>This blog has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/firmware_updates/part3/firmware_update_part3">Server Management Portal</a></big></p> <br>https://developer.hpe.com/how-to-use-a-table-load-tool-to-batch-puts-into-hbasemapr-database/https://developer.hpe.com/how-to-use-a-table-load-tool-to-batch-puts-into-hbasemapr-database/Thu, 15 Oct 2020 06:28:44 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Terry He", "publish": "2015-04-22T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <p>Apache HBase is an in-Hadoop database that delivers wide-column schema flexibility with strongly consistent reads and writes. Clients can access HBase data through either a native Java API, a Thrift or REST gateway, or now through a C API, making it very easy to access data. MapR Database, yet another in-Hadoop database has the same HBase APIs, but provides enterprise-grade features for production deployments of HBase applications.</p> <p>Put, Get and Scan are some of the prominent programming APIs that get used in the context of HBase applications. For certain write-heavy workloads, Put operations can get slow, so batching these Put operations is a commonly used technique to increase the overall throughput of the system. The following program illustrates a table load tool, which is a great utility program that can be used for batching Puts into an HBase/MapR Database table. The program creates a simple HBase table with a single column within a column family and inserts 100000 rows with 100 bytes of data. The batch size for the Puts is set to 500 in this example.</p> <pre><code class="language-python">import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.Random; import java.util.zip.CRC32; import org.apache.hadoop.hbase.util.Pair; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.TableExistsException; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.util.Bytes; public class LoadTableMTBatch { static long uniqueSeed = System.currentTimeMillis(); static long[] count; static long[] latency; static int[] keySizes; public static long printPerNum = 10000; public static boolean noCRC = false; public static long keySize = 8; public static long startRow = 0; public static int batchSize = 500; public static int preSplit = 1; //Used as a demo - Not accurated key distribution public static boolean flush = false; public static boolean autoFlush = false; public static final String KEY_PREFIX="user"; public static final long startKey = 0L; public static final long endKey = 999999999999999L; public static final String HBASE_RESOURCE_NAME = "/opt/mapr/hbase/hbase-0.98.9/conf/hbase-site.xml"; public static String ZOOKEEPER_NODES = "localhost"; //Default to localhost, only needed for accessing HBase public static final Pair ZOOKEEPER_SETTINGS = new Pair( "hbase.zookeeper.quorum", ZOOKEEPER_NODES); public static void usage(String arg) { System.err.println("bad token: " + arg); System.err .println("loadMT -rows &#x3C;100000> -valuesize &#x3C;100 bytes=""> -debug -path -threads &#x3C;10> -batchSize &#x3C;500> -numCF &#x3C;1> -numC &#x3C;1> -preSplit &#x3C;1> -zookeeperNodes -AutoFlush -flush"); System.exit(1); } public static void main(String[] args) throws java.io.IOException { Configuration conf = HBaseConfiguration.create(); String tableName = null; long numRows = 100000; long numCF = 1; long numC = 1; long valueSize = 100; int numThreads = 10; boolean augment = false; for (int i = 0; i &#x3C; args.length; ++i) { if (args[i].equals("-rows")) { i++; if (i >= args.length) usage(args[i]); numRows = Long.parseLong(args[i]); } else if (args[i].equals("-path")) { i++; if (i >= args.length) usage(args[i]); tableName = args[i]; } else if (args[i].equals("-debug")) { conf.set("fs.mapr.trace", "debug"); } else if (args[i].equals("-valuesize")) { i++; if (i >= args.length) usage(args[i]); valueSize = Long.parseLong(args[i]); } else if (args[i].equals("-threads")) { i++; if (i >= args.length) usage(args[i]); numThreads = Integer.parseInt(args[i]); } else if (args[i].equals("-p")) { i++; if (i >= args.length) usage(args[i]); printPerNum = Long.parseLong(args[i]); } else if (args[i].equals("-hbase")) { i++; if (i >= args.length) usage(args[i]); conf.addResource(new Path(args[i])); } else if (args[i].equals("-numCF")) { i++; if (i >= args.length) usage(args[i]); numCF = Integer.parseInt(args[i]); } else if (args[i].equals("-numC")) { i++; if (i >= args.length) usage(args[i]); numC = Integer.parseInt(args[i]); } else if (args[i].equals("-batchSize")) { i++; if (i >= args.length) usage(args[i]); batchSize = Integer.parseInt(args[i]); } else if (args[i].equals("-preSplit")) { i++; if (i >= args.length) usage(args[i]); preSplit = Integer.parseInt(args[i]); } else if (args[i].equals("-zookeeperNodes")) { i++; if (i >= args.length) usage(args[i]); ZOOKEEPER_NODES = args[i]; } else if (args[i].equals("-AutoFlush")) { autoFlush = true; } else if (args[i].equals("-flush")) { flush = true; } else { usage(args[i]); } } if (tableName == null) { System.out.println("Must specify path"); usage("path"); } LoadTableMTBatch lt = new LoadTableMTBatch(); try { LoadTableMTBatch.init(conf, tableName, numRows, numCF, numC, valueSize, augment); lt.loadTable(conf, tableName, numRows, numCF, numC, valueSize, numThreads); } catch (Exception e) { e.printStackTrace(); System.exit(-1); } } public void generateKeySizes() { Random rand = new Random(uniqueSeed); keySizes = new int[10]; keySizes[0] = rand.nextInt(5) + 5; keySizes[1] = rand.nextInt(40) + 10; keySizes[2] = rand.nextInt(50) + 50; keySizes[3] = rand.nextInt(400) + 100; keySizes[4] = rand.nextInt(500) + 500; keySizes[5] = rand.nextInt(4000) + 1000; keySizes[6] = rand.nextInt(5000) + 5000; keySizes[7] = rand.nextInt(10000) + 10000; keySizes[8] = rand.nextInt(12000) + 20000; keySizes[9] = rand.nextInt(32 * 1024 - 1) + 1; } public void loadTable(Configuration conf, String tableName, long numRows, long numCF, long numC, long valueSize, int numThreads) throws Exception { Thread[] loadThreads = new Thread[numThreads]; count = new long[numThreads]; latency = new long[numThreads]; if (keySize &#x3C; 1) { generateKeySizes(); } long offset = (endKey - startKey) / numThreads; for (int i = 0; i &#x3C; loadThreads.length; i++) { latency[i] = 0; if (preSplit &#x3C;= 1000="" 1)="" {="" loadthreads[i]="new" thread(new="" loadtablerunnable(conf,="" tablename,="" numrows,="" numcf,="" numc,="" valuesize,="" i,="" numthreads,="" batchsize));="" }="" else="" batchsize,="" startkey="" +="" i="" *="" offset,="" ((i="" offset)="" -="" 1));="" for="" (int="" &#x3C;="" loadthreads.length;="" i++)="" loadthreads[i].start();="" long="" inserts="0," insertsold="0," rate="0," overallrate="0," ta="0," tb="0," t0="0," elapsedtime="0;" averagelatency="0;" minlatency="0;" maxlatency="0;" boolean="" alive="true;" 1;="" while="" (true)="" if="" (loadthreads[i].isalive())="" insertsold)="" (ta="" tb);="" t0);="" elapsedtime;="" min="" max="" average="" latency="" synchronized="" (latency)="" arrays.sort(latency);="" 1];="" latency.length);="" system.out.println("elapsed="" time:="" "="" ";="" inserts:="" current="" sec;="" overall="" batchsize="" 1000000l="" "ms;"="" "ms");="" (!alive)="" break;="" print="" out="" interval="" thread.sleep(1000);="" loadthreads[i].join();="" public="" static="" getsum(long[]="" array)="" sum="0;" (long="" l="" :="" return="" sum;="" void="" createtable(configuration="" conf,="" string="" numcf)="" throws="" exception="" hbaseadmin="" admin="new" hbaseadmin(conf);="" system.out.println("created="" object");="" htabledescriptor="" des="new" htabledescriptor(tablename.getbytes());="" numcf;="" des.addfamily(new="" hcolumndescriptor("f"="" i));="" try="" (presplit="" admin.createtable(des);="" byte[]="" startkeybyte="Bytes.toBytes(KEY_PREFIX+startKey);" endkeybyte="Bytes.toBytes(KEY_PREFIX+endKey);" admin.createtable(des,="" startkeybyte,="" endkeybyte,="" presplit);="" catch="" (tableexistsexception="" te)="" te.printstacktrace();="" (ioexception="" ie)="" ie.printstacktrace();="" init(configuration="" augment)="" ioexception,="" (augment)="" htable="" intable="new" htable(conf,="" tablename);="" result="" infores="inTable.get(new" get("homerow".getbytes()));="" startrow="inTable.incrementColumnValue("homeRow".getBytes()," "f0".getbytes(),="" "c0".getbytes(),="" numrows)="" numrows;="" numcf="Bytes.toLong(infoRes.getValue("f0".getBytes()," "c1".getbytes()));="" numc="Bytes.toLong(infoRes.getValue("f0".getBytes()," "c2".getbytes()));="" uniqueseed="Bytes.toLong(infoRes.getValue("f0".getBytes()," "c3".getbytes()));="" keysize="Bytes.toLong(infoRes.getValue("f0".getBytes()," "c4".getbytes()));="" createtable(conf,="" numcf);="" put="" info="new" put("homerow".getbytes());="" info.add("f0".getbytes(),="" bytes.tobytes(numrows));="" "c1".getbytes(),="" bytes.tobytes(numcf));="" "c2".getbytes(),="" bytes.tobytes(numc));="" "c3".getbytes(),="" bytes.tobytes(uniqueseed));="" "c4".getbytes(),="" bytes.tobytes(keysize));="" intable.put(info);="" intable.flushcommits();="" load(configuration="" int="" threadnum,="" startkey,="" endkey)="" ioexception="" system.out.println("starting="" load="" thread="" threadnum);="" threadnum="" start="" key="" (key_prefix="" startkey)="" end="" :"="" endkey));="" family;="" column;="" p="null;" counter="0;" table="null;" random="" rand="new" random(uniqueseed);="" incrementrandom(rand,="" (int)="" startrow);="" endrow="startRow" htable(createhbaseconfiguration(),="" tablename.getbytes());="" table.setautoflush(autoflush);="" startrow;="" endrow;="" byte[][]="" rowkeys="new" byte[batchsize][];="" families="new" columns="new" values="new" batch="0;" batchsize;="" batch++)="" rowkey="new" byte[(int)="" keysize];="" (keysize="" 0)="" randsize="keySizes[rand.nextInt(Integer.MAX_VALUE)" %="" 10];="" numthreads="" 1);="" byte[randsize="" stringbuilder="" keybuilder="new" stringbuilder();="" keybuilder.append(i);="" keybuilder.append(batch);="" createkey(rowkey,="" long.valueof(keybuilder.tostring())="" ^="" uniqueseed);="" rowkeys[batch]="rowKey;" generate="" endkey);="" value="" valuesize];="" fillbuffer(valuesize,="" value,="" batch);="" values[batch]="value;" cf="" c="" family="f" (numcf="" families[batch]="family.getBytes();" column="c" (numc="" columns[batch]="column.getBytes();" list="" puts="new" arraylist();="" starttime="System.nanoTime();" put(rowkeys[batch]);="" p.add(families[batch],="" columns[batch],="" values[batch]);="" puts.add(p);="" table.put(puts);="" (flush)="" table.flushcommits();="" (exception="" e)="" e.printstacktrace();="" endtime="System.nanoTime();" latency[threadnum]="(endTime" starttime);="" count[threadnum]="counter;" finally="" (table="" !="null)" table.close();="" incrementrandom(random="" rand,="" num)="" num;="" rand.nextint();="" createkey(byte[]="" buffer,="" seed)="" random(seed);="" crc32="" chksum="new" crc32();="" rand.nextbytes(buffer);="" chksum.update(buffer);="" return;="" createkeyforregion(byte[]="" longrandom().nextlong(endkey="" startkey);="" buffer="Bytes.toBytes(KEY_PREFIX" key);="" buffer;="" fillbuffernocrc(long="" newseed="seed" system.currenttimemillis();="" random(newseed);="" fillbuffer(long="" chksum.getvalue();="" configuration="" createhbaseconfiguration()="" conf="HBaseConfiguration.create();" conf.addresource(new="" path(hbase_resource_name));="" conf.set((string)zookeeper_settings.getfirst(),="" (string)="" zookeeper_settings.getsecond());="" conf;="" class="" loadtablerunnable="" implements="" runnable="" private="" tablename;="" valuesize;="" threadnum;="" endkey="-1;" loadtablerunnable(configuration="" batchsize)="" this.conf="conf;" this.tablename="tableName;" this.numrows="numRows;" this.numcf="numCF;" this.numc="numC;" this.valuesize="valueSize;" this.threadnum="threadNum;" this.numthreads="numThreads;" this.batchsize="batchSize;" this.startkey="startKey;" this.endkey="endKey;" run()="" (endkey="=" -1)="" loadtablemtbatch.load(conf,="" 0,="" 0);="" system.exit(-1);="" longrandom="" extends="" final="" serialversionuid="1L;" **="" generating="" a="" in="" the="" range="" of="" 0&#x3C;="value&#x3C;=n" @param="" n="" @return="" nextlong(long="" n)="" (n="" throw="" new="" illegalargumentexception();="" small="" use="" nextint="" and="" cast="" (long)="" nextint((int)="" n);="" large="" both="" high="" low="" ints="" highlimit="(int)">> 32); long high = (long) nextInt(highLimit) &#x3C;&#x3C; 32; long low = ((long) nextInt()) &#x26; 0xffffffffL; return (high | low); } } } </code></pre> <hr> <a id="product-names" /> ## MapR to HPE Product Naming | Historical MapR Product Name | New Product Name | | --- | --- | | MapR (Document) Database Enterprise Premier | HPE Ezmeral Data Fabric Document Database |https://developer.hpe.com/hpe-oneview-54-ecosystem-sdks-introduce-new-methods-for-ilo-configuratio/https://developer.hpe.com/hpe-oneview-54-ecosystem-sdks-introduce-new-methods-for-ilo-configuratio/Tue, 13 Oct 2020 18:55:34 GMT<p>HPE OneView Ecosystem SDKs (<a href="https://github.com/HewlettPackard/oneview-ansible">Ansible</a>, <a href="https://github.com/HewlettPackard/oneview-python">Python</a>, <a href="https://github.com/HewlettPackard/oneview-golang">Golang</a>, <a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.3.0">Terraform</a>, <a href="https://github.com/HewlettPackard/oneview-chef">Chef</a>, <a href="https://github.com/HewlettPackard/oneview-puppet">Puppet</a>, <a href="https://github.com/HewlettPackard/POSH-HPOneView">PowerShell</a> and <a href="https://github.com/HewlettPackard/oneview-sdk-ruby">Ruby</a>) now support HPE OneView 5.4 (REST API version 2000). Each release introduces time savings and error reduction enhancements.</p> <p>With the latest release of the HPE OneView PowerShell library, you can now configure iLO settings through HPE OneView server profiles and server profile templates. This release adds ILO helper Cmdlets New-OVServerProfileIloPolicy, New-OVIloLocalUserAccount and New-OVIloDirectoryGroup for server profiles and server profile templates, which enable the configuration of ILO settings and eliminate the need for a separate login into the ILO to apply the necessary settings.</p> <p>The newest releases of Ansible, Python, Golang, Terraform, Chef, Puppet and Ruby SDKs introduce an enhanced method to set the default API version to the appliance’s max API version. This ensures that valid API settings are used as default settings, eliminating a potential source of error.</p> <p>HPE OneView Python v5.4.0 and HPE OneView Ansible Module v5.8.0 introduce a breaking change from the previous SDK version. From this version onwards, the previous hpOneView module name has been renamed to hpeOneView. All HPE OneView libraries and examples will import the hpeOneView module as a parent for both SDKs.</p> <p>HPE OneView SDKs enable the automation of provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView. This enables active and reactive monitoring and automated deployments, in addition to the provisioning of networks, storage and server infrastructure. You can also use these SDKs to develop a resource topology similar to that of a public cloud on physical infrastructure. This provides public cloud-like “node management”, with the extra flexibility to directly configure underlying infrastructure when needed.</p> <p>To simplify installation and maintenance for container environments, HPE OneView 5.4 SDKs are available as Docker images. <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">Ansible</a> , <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">Terraform</a> , <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">Chef</a> , <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">Puppet</a> , <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">Python</a> , <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">Golang</a> and <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">Ruby</a> SDKs are now all available on Docker Hub. All prerequisite materials are incorporated into the container images to enable streamlined deployment to simplify maintenance, improve infrastructure agility, and reduce costs.</p> <h2>For more information:</h2> <p><a href="https://developer.hpe.com/platform/hpe-oneview/home">HPE OneView on the HPE DEV platform site</a></p> <table> <thead> <tr> <th>SDK</th> <th>GitHub</th> <th>Docker Hub</th> </tr> </thead> <tbody> <tr> <td>Ansible</td> <td><a href="https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.8.0">HPE OneView Ansible Module v5.8.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">HPE OneView SDK Docker Image for Ansible</a></td> </tr> <tr> <td>Terraform</td> <td><a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.5.0">HPE OneView Terraform Provider v1.5.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">HPE OneView SDK Docker Image for Terraform</a></td> </tr> <tr> <td>Chef</td> <td><a href="https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.6.0">HPE OneView Chef Cookbook v3.6.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">HPE OneView SDK Docker Image for Chef</a></td> </tr> <tr> <td>Puppet</td> <td><a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.8.0">HPE OneView Puppet Module v2.8.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">HPE OneView SDK Docker Image for Puppet</a></td> </tr> <tr> <td>Python</td> <td><a href="https://github.com/HewlettPackard/oneview-python/releases/tag/v5.4.0">HPE OneView Python SDK v5.4.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">HPE OneView SDK Docker Image for Python</a></td> </tr> <tr> <td>Golang</td> <td><a href="https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.6.0">HPE OneView Golang SDK v1.6.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">HPE OneView SDK Docker Image for Golang</a></td> </tr> <tr> <td>Ruby</td> <td><a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.16.0">HPE OneView Ruby SDK v5.16.0</a></td> <td><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">HPE OneView SDK Docker Image for Ruby</a></td> </tr> <tr> <td>PowerShell</td> <td><a href="https://github.com/HewlettPackard/POSH-HPEOneView/releases/tag/v5.40.2551.2353">HPE OneView PowerShell module v5.40.2551.2353</a></td> <td></td> </tr> </tbody> </table>https://developer.hpe.com/new-grommet-release-offers-new-components-and-improves-accessibility/https://developer.hpe.com/new-grommet-release-offers-new-components-and-improves-accessibility/Thu, 08 Oct 2020 16:15:08 GMT<p>With the help of our rock star community, the Grommet team has been able to continuously introduce new components and enhancements. As a matter of fact, at least 10 new components have been added over the last several months. The most recent release, Grommet 2.15.0, features two new components – Card and DateInput – along with enhancements to others, like DataChart and SkipLinks.</p> <p>The new <a href="https://v2.grommet.io/card">Card</a> component works as a container of information that delivers a wide variety of content, contextual background colors, and powerful display options. The new Grommet Card component provides options to customize a header, body, and footer using CardHeader, CardBody, and CardFooter components. The new <a href="https://v2.grommet.io/dateinput">DateInput</a> component offers developers a high level of control for date range input, from a single date to a range of dates, through numerous specified properties.</p> <p>One of the more interesting enhancements came from work we’ve been doing with the <a href="https://www.hpe.com/us/en/cloud-services.html">HPE GreenLake Cloud Services</a> organization to help visualize AI data charting through the <a href="https://v2.grommet.io/datachart">DataChart</a> component. By extending the DataChart API, we were able to better visually represent 4-dimensional data and complex chart combinations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/4-dimensions-1602173691496.jpg" alt="4 dimensions"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/complex-charts-1602173698168.png" alt="complex charts"></p> <p>Accessibility received a lot of attention in this release. <a href="https://v2.grommet.io/skiplinks">SkipLinks</a>, in particular, has been reworked significantly to align with Web Content Accessibility Guidelines (WCAG). The behavior of the SkipLinks component has been improved to allow the layer to open with a keyboard tab, edit SkipLinks messages, and enable access to interactive elements when the SkipLinks layer is shown. Other accessibility enhancements include the addition of <strong>messages</strong> prop to Video and <strong>a11yTitle</strong> support to TextArea.</p> <p>We had an impressive list of contributors to this release. We’re so grateful for your collaboration. We’d like to also extend our thanks to our <a href="https://github.com/grommet/grommet/wiki/What-is-grommet-stable-and-how-to-use-it%3F">stable branch</a> (beta test) users. Without your help in testing all the different elements before they were fully baked, we could never have gotten so far so fast. Thank you for all the feedback you gave us to make Grommet 2.15.0 such an exciting release. The information you provided on the new DateInput component was particularly valuable. Your input will help us ensure that we meet the needs of real world use cases such as yours, while offering flexibility for future implementations.</p> <p>You can find details of all the enhancements in this newest version in the <a href="https://github.com/grommet/grommet/releases/tag/v2.15.0">release notes</a>. For more examples on how to use these components, check out <a href="https://storybook.grommet.io/">Storybook here</a> or go directly to:</p> <ul> <li><a href="https://storybook.grommet.io/?path=/story/card--clickable">Card</a></li> <li><a href="https://storybook.grommet.io/?path=/story/dateinput--form">DateInput</a></li> <li><a href="https://storybook.grommet.io/?path=/story/datachart--everything">DataChart</a></li> <li><a href="https://storybook.grommet.io/?path=/story/skiplinks--simple">SkipLinks</a></li> </ul>https://developer.hpe.com/2020-October-01/https://developer.hpe.com/2020-October-01/Thu, 01 Oct 2020 05:00:00 GMThttps://developer.hpe.com/get-started-with-the-hpe-nimble-storage-content-collection-for-ansible/https://developer.hpe.com/get-started-with-the-hpe-nimble-storage-content-collection-for-ansible/Tue, 29 Sep 2020 18:33:30 GMT<p>With the initial release of the <a href="https://community.hpe.com/t5/around-the-storage-block/introducing-hpe-nimble-storage-content-collection-for-ansible/ba-p/7103452">HPE Nimble Storage Content Collection for Ansible</a>, it’s now possible to manage many aspects of a Nimble array using either Red Hat Ansible Tower or open source Ansible. Ansible is an IT automation platform that embraces an idempotent resource model, which is essential in declarative infrastructure management paradigms. In this blog post, we’ll go through a few examples on how to kickstart your Ansible automation projects with the newly released HPE Nimble Storage modules.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/screen-shot-2020-10-01-at-15454-pm-1601594378015.png" alt="picture1"></p> <p>All the functionality is embedded in the content collection as modules. For the initial release, these are the available modules.</p> <table> <thead> <tr> <th>Module</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>hpe_nimble_access_control_record</code></td> <td>Manage access control records</td> </tr> <tr> <td><code>hpe_nimble_array</code></td> <td>Manage array</td> </tr> <tr> <td><code>hpe_nimble_chap_user</code></td> <td>Manage iSCSI CHAP users</td> </tr> <tr> <td><code>hpe_nimble_disk</code></td> <td>Manage disks and media</td> </tr> <tr> <td><code>hpe_nimble_encryption</code></td> <td>Manage encryption settings</td> </tr> <tr> <td><code>hpe_nimble_fc</code></td> <td>Manage fibre channel configuration</td> </tr> <tr> <td><code>hpe_nimble_group</code></td> <td>Manage the array group</td> </tr> <tr> <td><code>hpe_nimble_info</code></td> <td>Query and collect information of any resource from an array</td> </tr> <tr> <td><code>hpe_nimble_initiator_group</code></td> <td>Manage initiator groups</td> </tr> <tr> <td><code>hpe_nimble_network</code></td> <td>Manage network configuration</td> </tr> <tr> <td><code>hpe_nimble_partner</code></td> <td>Manage replication partners</td> </tr> <tr> <td><code>hpe_nimble_performance_policy</code></td> <td>Manage performance policies</td> </tr> <tr> <td><code>hpe_nimble_pool</code></td> <td>Manage storage pools</td> </tr> <tr> <td><code>hpe_nimble_protection_schedule</code></td> <td>Manage protection schedules</td> </tr> <tr> <td><code>hpe_nimble_protection_template</code></td> <td>Manage protection templates</td> </tr> <tr> <td><code>hpe_nimble_shelf</code></td> <td>Manage shelves</td> </tr> <tr> <td><code>hpe_nimble_snapshot_collection</code></td> <td>Manage snapshot collections</td> </tr> <tr> <td><code>hpe_nimble_snapshot</code></td> <td>Manage volume snapshots</td> </tr> <tr> <td><code>hpe_nimble_user</code></td> <td>Manage array users</td> </tr> <tr> <td><code>hpe_nimble_user_policy</code></td> <td>Manage array user policies</td> </tr> <tr> <td><code>hpe_nimble_volume</code></td> <td>Manage volumes</td> </tr> <tr> <td><code>hpe_nimble_volume_collection</code></td> <td>Manage volume collections</td> </tr> </tbody> </table> <p>All modules are documented per Ansible community standards. The modules have not yet been merged into the official Collection Index and documentation is provided from <a href="https://hpe-storage.github.io/nimble-ansible-modules">the GitHub repo</a> as a temporary solution.</p> <h1>Preface</h1> <p>In the following examples there is one node acting as the Ansible management host (node21) and there are two nodes (node22 and node23) acting as iSCSI SAN hosts. There’s one HPE Nimble Storage array (nva) in the environment. Since an Ansible collection is a fairly new construct in the Ansible universe, version 2.9 is required, along with Python 3.6 or newer for the HPE Nimble Storage SDK for Python (which the Ansible modules rely on). We’ll also assume that iSCSI, multipathing and SAN connectivity is established between the SAN hosts and the HPE Nimble Storage array. NimbleOS 5.0 or newer is required on the array. These requirements are also listed with <a href="https://hpe-storage.github.io/nimble-ansible-modules/">the modules' documentation</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/hpedev-alster-for-blog-reva-1601597390641.png" alt="picture2"></p> <p>Since the collection contains more than a whopping twenty two modules, one blog post won’t be able to cover the entire suite. Expect a series of blog posts over the coming months to cover more use cases. In this first installment, we’ll cover basic volume provisioning, snapshotting, cloning, inspecting, mutations and ultimately decommissioning volumes. Host attachment included!</p> <blockquote> <p>Note: All examples listed below are available in <a href="https://github.com/NimbleStorage/automation-examples">this GitHub repo</a> (change directory to <code>ansible/introduction</code>).</p> </blockquote> <h1>Installation</h1> <p>No special privileges are needed on the Ansible host. Privileges are needed on managed hosts where we want to attach storage. Let’s begin with installing Ansible and the required HPE Nimble Storage SDK for Python using <code>pip</code>.</p> <pre><code class="language-markdown">$ pip3 install ansible nimble-sdk --user </code></pre> <p>Next, use <code>ansible-galaxy</code> to install the HPE Nimble Storage Content Collection for Ansible.</p> <pre><code class="language-markdown">$ ansible-galaxy collection install hpe.nimble </code></pre> <p>We’re now ready to create an inventory and start writing playbooks to manage storage resources.</p> <h1>Inventory</h1> <p>There are many different ways to write inventory files (classic .ini or YAML) and store variables. In this series of playbooks the variables will be stored with the nodes in the inventory groups. Playbooks will use the <code>-e</code> flag to provide "extra" variables to make the playbooks reusable for different nodes and storage resources. In a scenario where the infrastructure is managed via source code management (SCM), resources and nodes would be stored in separate variable files.</p> <pre><code class="language-ini">[nodes] node22 ansible_host=192.168.159.22 node23 ansible_host=192.168.159.23 [nodes:vars] ansible_user=vagrant ansible_password=vagrant [arrays] nva nimble_host=192.168.59.130 nimble_discovery_ip=192.168.59.130 [arrays:vars] ansible_connection=local nimble_user=admin nimble_pass=admin [all:vars] ansible_python_interpreter=/usr/bin/python3 </code></pre> <p>Let’s break these sections down:</p> <ul> <li><strong>[nodes]</strong> A group of my SAN connected nodes, name resolution is not working in my sandbox so I need to address nodes by IP address</li> <li><strong>[nodes:vars]</strong> All nodes will share these variables</li> <li><strong>[arrays]</strong> A group of arrays I have available in my environment, each with my custom variables to access the REST API and iSCSI discovery IP address</li> <li><strong>[arrays:vars]</strong> All arrays will share these variables, <strong>ansible_connection</strong> is set to local as Ansible is not logging in to the array itself, only indirect via the REST API</li> <li><strong>[all:vars]</strong> Ansible will run on any available Python version (near enough) the Nimble SDK require Python 3.6 or newer</li> </ul> <blockquote> <p>Note: Don’t put passwords in your inventory file. Use encrypted variable files with <code>ansible-vault</code> for that purpose.</p> </blockquote> <p>Also, an <code>ansible.cfg</code> is put in place to pickup the <code>inventory</code> file in the current directory.</p> <pre><code class="language-ini">[defaults] inventory = inventory </code></pre> <h1>Hello world</h1> <p>A pattern I usually employ before doing anything is to check if my inventory is reachable.</p> <pre><code class="language-markdown">$ ansible -m ping all nva | SUCCESS => { "changed": false, "ping": "pong" } node22 | SUCCESS => { "changed": false, "ping": "pong" } node23 | SUCCESS => { "changed": false, "ping": "pong" } </code></pre> <p>I created a playbook that would simplify future playbooks by creating the initiator groups for all nodes on all arrays. Initiator groups are required for assigning access control records (ACRs) to volumes so they could also be incorporated in a task just before assigning the ACR.</p> <pre><code class="language-yaml">--- - name: Retrieve node facts hosts: nodes - name: Create iSCSI initators for hosts in nodes group gather_facts: false hosts: arrays collections: - hpe.nimble tasks: - name: Add host hpe_nimble_initiator_group: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" access_protocol: iscsi name: "{{ item }}" iscsi_initiators: [ { iqn: "{{ hostvars[item]['ansible_iscsi_iqn'] }}", label: "{{ item }}" } ] description: "Initiator Group for {{ item }}" state: present with_items: "{{ groups.nodes }}" </code></pre> <p>Pay attention to the <code>collections</code> stanza. That allows the modules to be discovered properly. It’s also possible to omit the collections stanza but then modules need to be called out with their fully qualified names, <code>hpe.nimble.hpe_nimble_initiator_group</code>, which isn’t very pretty to look at.</p> <p>That said, the fully qualified module name is good to know about. If you need to look up any of the module options or examples, it’s available right at your fingertips. For example, to look up the documentation for the <code>hpe_nimble_initiator_group</code> module, simply use the <code>ansible-doc</code> command.</p> <pre><code class="language-markdown">$ ansible-doc hpe.nimble.hpe_nimble_initiator_group </code></pre> <p>Next, run the playbook.</p> <pre><code class="language-markdown">$ ansible-playbook connect.yaml PLAY [Retrieve node facts] ***************************************************** TASK [Gathering Facts] ********************************************************* ok: [node23] ok: [node22] PLAY [Create iSCSI initators for hosts in nodes group] ************************* TASK [Add host] **************************************************************** ok: [nva] => (item=node22) ok: [nva] => (item=node23) PLAY RECAP ********************************************************************* node22 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 node23 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 nva : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>The output will vary depending on if the initiator groups already exists or not. This playbook could be re-run for each inventory update to ensure all initiator groups exist as expected.</p> <h1>Volume provisioning</h1> <p>The next example provide a whole lot more content. The purpose of <code>provision.yaml</code> is to:</p> <ul> <li>Create a new Nimble volume, <strong>volume_name</strong>, on <strong>nimble_array</strong></li> <li>Create an ACR for the <strong>volume_igroup</strong></li> <li>Discover and attach the volume from the host</li> <li>Format the volume and mount it on the host</li> <li>Make the filesystem writeable by an application</li> </ul> <p>There’s also an optional parameter, <code>volume_size</code>. It falls back to <code>1000</code> (MiB) if not specified.</p> <p>By parameterizing the playbook, it becomes very flexible to reuse. It could also quite easily be turned into an Ansible role to make it even more reusable between projects.</p> <pre><code class="language-yaml">--- - name: Provision a Nimble Volume to a SAN host gather_facts: false connection: local hosts: "{{ nimble_array }}" collections: - hpe.nimble tasks: - block: - name: Create Volume hpe_nimble_volume: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" state: present name: "{{ volume_name }}" size: "{{ volume_size | default('1000') }}" description: "Volume for {{ volume_igroup }}" register: volume - name: Set facts to pass on to node play set_fact: volume_target_name: "{{ volume.attrs.target_name }}" volume_serial_number: "{{ volume.attrs.serial_number }}" when: volume_clone is not defined - block: - name: Create Volume from Snapshot hpe_nimble_volume: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" state: present name: "{{ volume_name }}" size: "{{ volume_size | default('1000') }}" description: "Volume for {{ volume_igroup }}" snapshot: "{{ snapshot_name | default(False) }}" clone: "{{ volume_clone | default(False) }}" parent: "{{ clone_from | default(False) }}" register: volume - name: Set facts to pass on to node play set_fact: volume_target_name: "{{ volume.attrs.target_name }}" volume_serial_number: "{{ volume.attrs.serial_number }}" when: volume_clone is defined - name: Create ACR hpe_nimble_access_control_record: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" state: present initiator_group: "{{ volume_igroup }}" volume: "{{ volume_name }}" - name: Attach a volume, format and mount it on a host hosts: "{{ volume_igroup }}" tasks: - name: Discover Target become: yes open_iscsi: portal: "{{ hostvars[nimble_array]['nimble_discovery_ip'] }}" discover: yes show_nodes: yes - name: Attach Target become: yes open_iscsi: target: "{{ hostvars[nimble_array]['volume_target_name'] }}" login: yes automatic: yes show_nodes: yes - name: Set volume device fact set_fact: volume_device_path: /dev/disk/by-id/dm-uuid-mpath-2{{ hostvars[nimble_array]['volume_serial_number'] }} - name: Create Filesystem become: yes filesystem: fstype: xfs dev: "{{ volume_device_path }}" - name: Mount the Volume become: yes mount: fstype: xfs state: mounted path: /mnt/{{ volume_name }} src: "{{ volume_device_path }}" - name: Set Permissions become: yes file: owner: "{{ ansible_env.LOGNAME }}" path: /mnt/{{ volume_name }} </code></pre> <p>There are a few opinionated decisions made here, like mounting the filesystem under <code>/mnt/volume_name</code>, but other than that, the example is quite comprehensive. Let’s run it!</p> <pre><code class="language-markdown">$ ansible-playbook provision.yaml -e nimble_array=nva \ -e volume_name=myvol1 -e volume_igroup=node22 PLAY [Provision a Nimble Volume to a SAN host] ********************************************************************************* TASK [Create Volume] *********************************************************************************************************** changed: [nva] TASK [Set facts to pass on to node play] *************************************************************************************** ok: [nva] TASK [Create Volume from Snapshot] ********************************************************************************************* skipping: [nva] TASK [Set facts to pass on to node play] *************************************************************************************** skipping: [nva] TASK [Create ACR] ************************************************************************************************************** changed: [nva] PLAY [Attach a volume, format and mount it on a host] ************************************************************************** TASK [Gathering Facts] ********************************************************************************************************* ok: [node22] TASK [Discover Target] ********************************************************************************************************* ok: [node22] TASK [Attach Target] *********************************************************************************************************** changed: [node22] TASK [Set volume device fact] ************************************************************************************************** ok: [node22] TASK [Create Filesystem] ******************************************************************************************************* changed: [node22] TASK [Mount the Volume] ******************************************************************************************************** changed: [node22] TASK [Set Permissions] ********************************************************************************************************* changed: [node22] PLAY RECAP ********************************************************************************************************************* node22 : ok=7 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 nva : ok=3 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 </code></pre> <p>Bliss! A new volume was created and correctly mapped to the node. On the node, the LUN was discovered, attached, formatted and mounted (including adding an entry to <code>/etc/fstab</code>) and is ready for use. This can be inspected by calling the <code>command</code> or <code>shell</code> module ad hoc.</p> <pre><code class="language-markdown">$ ansible -m command node22 -a 'df -h /mnt/myvol1' node22 | CHANGED | rc=0 >> Filesystem Size Used Avail Use% Mounted on /dev/mapper/223119d4f2e94525f6c9ce900e7f81e0a 997M 33M 965M 4% /mnt/myvol1 </code></pre> <h1>Inspecting and mutating resources</h1> <p>Among the list of modules there’s a module not specifically mapped to a resource — the <code>hpe_nimble_info</code> module. This module is a very powerful query tool that allows you to extract any piece of metadata from every API resource available on the array. The result set can also be filtered and limited in the query, hence preserving resources as the array only has to return the objects and metadata being requested. This functionality is also available in the Python SDK and the REST API directly and it made perfect sense to make it available natively in Ansible.</p> <p>Let’s grab some attributes from the volume we just created.</p> <pre><code class="language-yaml">--- - name: Query a Nimble array volume gather_facts: false connection: local hosts: "{{ nimble_array }}" collections: - hpe.nimble tasks: - name: Inspect volume hpe_nimble_info: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" gather_subset: - volumes: query: name: "{{ volume_name }}" fields: - limit_iops - description - iscsi_sessions register: query - name: Results debug: var: query </code></pre> <p>Here we’re gathering the IOPS limit of the volume along with the iSCSI sessions and description. Let’s see what we get.</p> <pre><code class="language-markdown">$ ansible-playbook query.yaml -e nimble_array=nva -e volume_name=myvol1 PLAY [Query a Nimble array volume] ********************************************************************************************* TASK [Inspect volume] ********************************************************************************************************** ok: [nva] TASK [Results] ***************************************************************************************************************** ok: [nva] => { "query": { "changed": false, "failed": false, "message": "Fetched the subset details.", "nimble_info": { "volumes": [ { "description": "Volume for node22", "iscsi_sessions": [ { "data_digest_enabled": false, "header_digest_enabled": false, "id": "346c943e6f668ac68800000000000000000000000f", "initiator_ip_addr": "192.168.59.190", "initiator_name": "iqn.1994-05.com.redhat:bcbe2e27f0", "num_connections": 1, "pr_key": 0, "session_id": "346c943e6f668ac68800000000000000000000000f", "target_ip_addr": "192.168.59.132" } ], "limit_iops": -1 } ] }, "return_status": true } } PLAY RECAP ********************************************************************************************************************* nva : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>The registered fact returned exactly what we asked for, the IOPS limit, description and details about the iSCSI sessions.</p> <p>Many attributes of a resource can be mutated during runtime. In this example, I want to set a new volume description and limit the IOPS to 1000 for a particular volume. Since Ansible is idempotent, you can add the <code>limit_iops</code> and <code>description</code> to the “Create Volume” stanza in the <code>provision.yaml</code> playbook. But then all subsequent volumes created from the playbook will get the same attributes.</p> <p>Let’s create a playbook to mutate these fields.</p> <pre><code class="language-yaml">--- - name: Mutate a Nimble array volume gather_facts: false connection: local hosts: "{{ nimble_array }}" collections: - hpe.nimble tasks: - name: Mutate volume hpe_nimble_volume: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" limit_iops: "{{ volume_iops | default('-1') }}" description: "{{ volume_description | default('') }}" name: "{{ volume_name }}" state: present </code></pre> <p>Run the playbook with a few extra variables.</p> <pre><code class="language-markdown">$ ansible-playbook tune.yaml -e nimble_array=nva \ -e volume_name=myvol1 -e volume_iops=1000 \ -e "volume_description='Mutated by Ansible'" PLAY [Mutate a Nimble array volume] ******************************************************************************************** TASK [Mutate volume] *********************************************************************************************************** changed: [nva] PLAY RECAP ********************************************************************************************************************* nva : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>If we re-run the query, we should be able to see the IOPS has been capped and the description updated.</p> <pre><code class="language-markdown">$ ansible-playbook query.yaml -e nimble_array=nva -e volume_name=myvol1 PLAY [Query a Nimble array volume] ********************************************************************************************* TASK [Inspect volume] ********************************************************************************************************** ok: [nva] TASK [Results] ***************************************************************************************************************** ok: [nva] => { "query": { "changed": false, "failed": false, "message": "Fetched the subset details.", "nimble_info": { "volumes": [ { "description": "Mutated by Ansible", "iscsi_sessions": [ { "data_digest_enabled": false, "header_digest_enabled": false, "id": "346c943e6f668ac68800000000000000000000000f", "initiator_ip_addr": "192.168.59.190", "initiator_name": "iqn.1994-05.com.redhat:bcbe2e27f0", "num_connections": 1, "pr_key": 0, "session_id": "346c943e6f668ac68800000000000000000000000f", "target_ip_addr": "192.168.59.132" } ], "limit_iops": 1000 } ] }, "return_status": true } } PLAY RECAP ********************************************************************************************************************* nva : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>This is a very brief demonstration on how to accomplish a storage management task in a simple and reusable manner. This playbook could be put on a crontab to throttle IOPS at certain times of day to accommodate a certain business need. It can also be made available to application owners who want to be able to tweak their application storage resources. It’s then practical to use something like Ansible Tower to enable role-based access control (RBAC) for inventories and playbooks. We’ll cover Ansible Tower in a future blog. Stay tuned!</p> <h1>Snapshotting and cloning</h1> <p>Very advanced workflows can be put together using the snapshot capabilities of the HPE Nimble Storage array. In this very brief example, we’ll create a snapshot of the volume that just got created and attach it to our second host in the inventory.</p> <p>First, create some content and flush the buffers. This way we’ll have something to look at once our clone comes up on the target host.</p> <pre><code class="language-markdown">$ ansible -m shell node22 -a 'date > /mnt/myvol1/date.txt &#x26;&#x26; sync' node22 | CHANGED | rc=0 >> </code></pre> <p>Next, create a snapshot with the <code>snapshot.yaml</code> playbook.</p> <pre><code class="language-yaml">--- - name: Create a Snapshot of a Nimble Volume gather_facts: false connection: local hosts: "{{ nimble_array }}" collections: - hpe.nimble tasks: - name: Snapshot operation hpe_nimble_snapshot: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" volume: "{{ volume_name }}" name: "{{ snapshot_name }}" description: "Snapshot created by Ansible" state: "{{ snapshot_state | default('present') }}” </code></pre> <p>Snapshot!</p> <pre><code class="language-markdown">$ ansible-playbook snapshot.yaml -e nimble_array=nva \ -e volume_name=myvol1 -e snapshot_name=mysnap1 PLAY [Create a Snapshot of a Nimble Volume] ************************************************************************************ TASK [Snapshot operation] ****************************************************************************************************** changed: [nva] PLAY RECAP ********************************************************************************************************************* nva : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>There’s now a snapshot named “mysnap1” on “myvol1”. We want to create a clone from that snapshot for our new volume that we’re attaching to the second host in the inventory. We’ll reuse the <code>provision.yaml</code> playbook by simply adding a few more variables: <code>snapshot_name</code>, <code>volume_clone</code> and <code>clone_from</code>.</p> <pre><code class="language-markdown">$ ansible-playbook provision.yaml -e nimble_array=nva \ -e volume_name=myvol1-clone -e snapshot_name=mysnap1 \ -e volume_clone=yes -e clone_from=myvol1 \ -e volume_igroup=node23 PLAY [Provision a Nimble Volume to a SAN host] ********************************************************************************* TASK [Create Volume] *********************************************************************************************************** skipping: [nva] TASK [Set facts to pass on to node play] *************************************************************************************** skipping: [nva] TASK [Create Volume from Snapshot] ********************************************************************************************* changed: [nva] TASK [Set facts to pass on to node play] *************************************************************************************** ok: [nva] TASK [Create ACR] ************************************************************************************************************** changed: [nva] PLAY [Attach a volume, format and mount it on a host] ************************************************************************** TASK [Gathering Facts] ********************************************************************************************************* ok: [node23] TASK [Discover Target] ********************************************************************************************************* ok: [node23] TASK [Attach Target] *********************************************************************************************************** changed: [node23] TASK [Set volume device fact] ************************************************************************************************** ok: [node23] TASK [Create Filesystem] ******************************************************************************************************* ok: [node23] TASK [Mount the Volume] ******************************************************************************************************** changed: [node23] TASK [Set Permissions] ********************************************************************************************************* ok: [node23] PLAY RECAP ********************************************************************************************************************* node23 : ok=7 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 nva : ok=3 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 </code></pre> <p>We should now be able to do a visual inspect of the content from our two nodes. Let’s check the file we just created before the snapshot got created.</p> <pre><code class="language-markdown">$ ansible -m command node22 -a 'cat /mnt/myvol1/date.txt' node22 | CHANGED | rc=0 >> Tue Sep 29 02:17:11 UTC 2020 $ ansible -m command node23 -a 'cat /mnt/myvol1-clone/date.txt' node23 | CHANGED | rc=0 >> Tue Sep 29 02:17:11 UTC 2020 </code></pre> <p>Presto! The clone may now be manipulated safely without impacting the parent.</p> <p>In future blogs, we’ll explore how to structure playbooks to idempotent “refresh” a clone from a parent in a safe and secure manner. That, in turn, empowers users to access volumes to allow dev and test on production-like data without running the risk of mistakingly disrupting mission-critical applications.</p> <h1>Volume decommissioning</h1> <p>Housekeeping is important. What’s being provisioned should also be able to get decommissioned in the same manner. Since provisioning a storage resource and attaching it to a host is a multi-task playbook, one can’t simply declare it “absent”. Tasks needs to be stacked in the reverse order with different verbs to effectively dismantle the data access and volumes.</p> <p>For completeness, here’s how to unmount and detach a volume from a host and subsequently offline and delete the volume from the Nimble array.</p> <pre><code class="language-yaml">--- - name: Prepare to delete Nimble Volume gather_facts: false connection: local hosts: "{{ nimble_array }}" collections: - hpe.nimble tasks: - name: Fetch Volume info hpe_nimble_info: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" gather_subset: - volumes: fields: - target_name query: name: "{{ volume_name }}" detail: True register: volumes failed_when: volumes.nimble_info.volumes.0.target_name is not defined - name: Set facts to pass on to node play set_fact: volume_target_name: "{{ volumes.nimble_info.volumes.0.target_name }}" - name: Unmount and detach a Nimble Volume from a host hosts: "{{ volume_igroup }}" tasks: - name: Unmount the Volume become: yes mount: fstype: xfs state: absent path: /mnt/{{ volume_name }} - name: Detach Target become: yes open_iscsi: target: "{{ hostvars[nimble_array]['volume_target_name'] }}" login: no - name: Delete Target become: yes command: iscsiadm -m node -o delete -T {{ hostvars[nimble_array]['volume_target_name'] }} - name: Delete a Nimble Volume gather_facts: false connection: local hosts: "{{ nimble_array }}" tags: - array_only collections: - hpe.nimble tasks: - name: Offline Volume hpe_nimble_volume: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" state: present online: False force: True name: "{{ volume_name }}" register: offline until: offline is not failed - name: Delete Volume hpe_nimble_volume: host: "{{ nimble_host }}" username: "{{ nimble_user }}" password: "{{ nimble_pass }}" state: absent name: "{{ volume_name }}" </code></pre> <p>Let’s run it! Beware — this operation is irreversible. Make sure volume names and nodes are correct. We'll start with the clone, as the parent can't be removed unless all dependent volumes have been destroyed.</p> <pre><code class="language-markdown">$ ansible-playbook decommission.yaml -e nimble_array=nva \ -e volume_name=myvol1-clone -e volume_igroup=node23 PLAY [Prepare to delete Nimble Volume] ***************************************** TASK [Fetch Volume info] ******************************************************* ok: [nva] TASK [Set facts to pass on to node play] *************************************** ok: [nva] PLAY [Unmount and detach a Nimble Volume from a host] ************************** TASK [Gathering Facts] ********************************************************* ok: [node22] TASK [Unmount the Volume] ****************************************************** changed: [node22] TASK [Detach Target] *********************************************************** changed: [node22] TASK [Delete Target] *********************************************************** changed: [node22] PLAY [Delete a Nimble Volume] ************************************************** TASK [Offline Volume] ********************************************************** changed: [nva] TASK [Delete Volume] *********************************************************** changed: [nva] PLAY RECAP ********************************************************************* node22 : ok=4 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 nva : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <p>Go ahead and destroy the parent volume.</p> <pre><code class="language-markdown">$ ansible-playbook decommission.yaml -e nimble_array=nva \ -e volume_name=myvol1 -e volume_igroup=node22 ... PLAY RECAP ********************************************************************************************************************* node22 : ok=4 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 nva : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre> <h1>Next steps</h1> <p>Expect more content on HPE DEV to frame more use cases relevant for automating storage management with the Ansible modules. This will include more advanced snapshot and clone management, scheduling and resource life-cycles. We also plan on going through user management, advanced reporting and network configuration. Every module will be touched on to describe its use and options to derive value from infrastructure automation.</p> <p>The HPE Nimble Storage Content Collection for Ansible is available immediately and installable from <a href="https://galaxy.ansible.com/hpe/nimble/">Ansible Galaxy</a> and the certified collection is also available on <a href="https://cloud.redhat.com/ansible/automation-hub/hpe/nimble">Red Hat Automation Hub</a>.</p> <ul> <li>Check out <a href="https://hpe-storage.github.io/nimble-ansible-modules/">the documentation</a> on the hosted GitHub Pages</li> <li>The content collection is hosted in the <a href="https://galaxy.ansible.com/hpe/nimble">hpe.nimble</a> namespace on Ansible Galaxy</li> <li>Source Code <a href="https://github.com/hpe-storage/nimble-ansible-modules">is available</a> on GitHub</li> <li>Read the product announcement on <a href="https://community.hpe.com/t5/around-the-storage-block/introducing-hpe-nimble-storage-content-collection-for-ansible/ba-p/7103452">Around The Storage Block</a></li> </ul> <p>We are open to feedback and collaboration. Join us on the HPE DEV Slack community and hang out with the team in #NimbleStorage. Signup <a href="https://slack.hpedev.io">here</a> and join us at <a href="https://hpedev.slack.com/archives/C7TTAHRUN">hpedev.slack.com</a>.</p>https://developer.hpe.com/how-to-persist-kafka-data-as-json-in-nosql-storage-using-mapr-event-stor/https://developer.hpe.com/how-to-persist-kafka-data-as-json-in-nosql-storage-using-mapr-event-stor/Fri, 25 Sep 2020 06:11:31 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Ian Downard", "publish": "2016-10-31T07:00:00.000Z", "tags": "nosql" </code></pre> <hr> <h2>Streaming data is like the phrase, “Now you see it. Now you don’t!”</h2> <p>One of the challenges when working with streams, especially streams of fast data, is the transitory nature of the data. Kafka streams are characterized by a retention period that defines the point at which messages will be permanently deleted. For many applications, such as those fed by streams of rapidly generated sensor data, the retention period is a desirable and convenient way to purge stale data, but in other cases, such as with insurance or banking applications, record-retention laws may require data to be persisted far beyond the point at which that data has any practical value to streaming analytics. This is challenging in situations where rapidly ingested data creates pressure on stream consumers designed to write streaming records to a database. Even if we can ensure these consumers keep up, we still need to guarantee zero data loss in the unlikely event that they do fail.</p> <p><strong>HPE Ezmeral Data Fabric Event Data Streams (Formerly MapR Event Store) and HPE Ezmeral Data Fabric Document Database (Formerly MapR Database) work together to provide a scalable and fault tolerant way to save streaming data in long-term storage.</strong> They both have a distributed, scale-out design based on the HPE Ezmeral Data Fabric (Formerly MapR Data Platform). Furthermore, as a NoSQL data store, HPE Ezmeral Data Fabric Document Database makes it easy to persist data with the same schema encoded into streams. This is not only convenient for the developer but also minimizes the work required to transform streaming records into persistable objects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/persist-kafka-json-streams-mapr-02_0-1601014236654.png" alt=""></p> <p>Let's illustrate these concepts with an example that persists streaming data in 5 simple steps:</p> <ol> <li>Setup stream and database connections.</li> <li>Consume records from a MapR stream using the standard Kafka API.</li> <li>Convert each consumed record to a JSON object.</li> <li>Persist that JSON object in HPE Ezmeral Data Fabric Document Database.</li> <li>Update the stream cursor to ensure graceful recovery should a stream consumer fail.</li> </ol> <h2>Step 1: Setup stream and database connections</h2> <p>Before we can do anything interesting we first have to setup our stream and database connections. We'll use the following two options that relate to fault tolerance:</p> <ul> <li>We disable the <code>enable.auto.commit</code> consumer option in order to commit stream cursors only after their corresponding records have been writing to the database.</li> <li>We disable the <code>BUFFERWRITE</code> table option in order to ensure database writes are not buffered on the client.</li> </ul> <p>With these options we're sacrificing speed for higher fault tolerance but we can compensate for that tradeoff by creating more topic partitions and running more concurrent consumers in parallel.</p> <p>So, here is what our database and consumer configurations look like:</p> <pre><code class="language-markdown">Table table; String tableName = "/user/mapr/ticktable"; if (MapRDB.tableExists(tableName)) { table = MapRDB.getTable(tableName); } else { table = MapRDB.createTable(tableName); } table.setOption(Table.TableOption.BUFFERWRITE, false); Properties props = new Properties(); props.put("enable.auto.commit","false"); props.put("group.id", “mygroup”); props.put("auto.offset.reset", "earliest"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumer = new KafkaConsumer&#x3C;String, String>(props); List&#x3C;String> topics = new ArrayList&#x3C;>(); topics.add(topic); consumer.subscribe(topics); </code></pre> <h2>Step 2: Consume records from the stream</h2> <p>To consume records from a stream, you first poll the stream. This gives you a collection of <code>ConsumerRecords</code> which you then iterate through in order to access each individual stream record. This is standard Kafka API stuff, and it looks like this:</p> <pre><code class="language-markdown">ConsumerRecords&#x3C;String, byte[]> records = consumer.poll(TIMEOUT); Iterator&#x3C;ConsumerRecord&#x3C;String, byte[]>> iter = msg.iterator(); while (iter.hasNext()) { ConsumerRecord&#x3C;String, byte[]> record = iter.next(); } </code></pre> <h2>Step 3: Convert streamed records to JSON</h2> <p>Before we write consumer records to the database, we need to put each record in a format that has columns. In our example we’re streaming byte arrays, which by themselves have no field related attributes, so we need to convert these byte arrays into a type containing attributes that will correspond to columns in our database. We’ll do this with a Java object, defined in <a target='\_blank' href='https://gist.github.com/iandow/92d3276e50a7e77f41e69f5c69c8563b'>Tick.java</a>, which uses the @JsonProperty annotation to conveniently convert Tick objects encoded as byte arrays into a JSON document, like this:</p> <pre><code class="language-markdown">Tick tick = new Tick(record.value()); Document document = MapRDB.newDocument((Object)tick); </code></pre> <h2>Step 4: Persist the JSON document to HPE Ezmeral Data Fabric Document Database</h2> <p>This part is easy. We can insert each JSON document as a new row to a table in MapR Database with one line of code, like this:</p> <pre><code class="language-markdown">table.insertOrReplace(tick.getTradeSequenceNumber(), document); </code></pre> <p>The first parameter in the insertOrReplace method is Document ID (or rowkey). It’s a property of our dataset that the value returned by <code>tick.getTradeSequenceNumber()</code> is unique for each record, so we’re referencing that as the Document ID for our table insert in order to avoid persisting duplicate records even if duplicate messages are consumed from the stream. This guarantees idempotency in our stream consumer.</p> <h2>Step 5: Update the stream cursor</h2> <p>Finally, we’ll update the cursor in our stream topic. In the unlikely event that our stream consumer fails, this ensures that a new consumer will be able to continue working from where the last consumer left off.</p> <pre><code class="language-markdown">consumer.commitSync(); </code></pre> <h2>Summary</h2> <p>The design we just outlined provides a scalable approach to persisting stream data. It ensures thread-safety by processing immutable stream data with idempotent stream consumers and achieves fault tolerance by updating stream cursors only after records have been persisted in HPE Ezmeral Data Fabric Document Database. This represents an elastic, responsive, resilient, and message-driven design consistent with the characteristics of reactive microservices. This is a reliable approach to persisting Kafka streams in long-term NoSQL storage.</p> <p>If you'd like to see a complete example application that uses the techniques described in this post, check out the <a target='\_blank' href='https://github.com/mapr-demos/finserv-application-blueprint/blob/master/src/main/java/com/mapr/demo/finserv/Persister.java'>Persister.java</a> in the <a target='\_blank' href='https://github.com/mapr-demos/finserv-application-blueprint'>Application blueprint</a> on GitHub.</p> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p>To view more articles on this topic, be sure to check back regularly on the <a href="/blog">HPE DEV blog site</a>.</p>https://developer.hpe.com/how-to-build-stanzas-using-mapr-installer-for-easy-and-efficient-provisi/https://developer.hpe.com/how-to-build-stanzas-using-mapr-installer-for-easy-and-efficient-provisi/Sat, 19 Sep 2020 05:31:19 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Prashant Rathi", "publish": "2016-12-09T08:00:00.000Z", "tags": "streaming" </code></pre> <hr> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p>The MapR Installer provides cluster operators an intuitive way to set up a cluster using a step-by-step wizard. The wizard guides you through:</p> <ul> <li>Selecting a Core Version and ecosystem services</li> <li>Using Auto-Provisioning templates</li> <li>Specifying a list of Nodes and Disks</li> <li>Grouping Services and laying them out across nodes</li> <li>Verification of all dependencies before cluster installation</li> </ul> <p>We have set up multiple clusters for several of our enterprise customers and learned quite a bit in the process. Increasingly, these deployments have not only grown in number, but have also evolved based on the type, purpose, and lifetime of these clusters. If you add to the mix the rapid innovation in the community and the complexity that it brings, there is a clear demand for a higher level of automated and consistent cluster provisioning.</p> <p>Installer Stanzas enable API-driven installation for the industry’s only converged data platform. With this capability, operators can build a Stanza that contains layout and settings for the cluster to be installed and passes it programmatically to the installer to execute the set of instructions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/stanza-1600932620174.png" alt=""></p> <p>This new capability is very useful when you need a script-based tool to install the software and you do not want to click through the menus and options provided by the installer wizard. While this method provides less visual feedback than the GUI version, it can be faster and more efficient at installing software on clusters with many nodes. Not only that, but once a Stanza gets defined, you can automate the cluster setup process for each successive cluster creation with a minimum set of changes.</p> <p>Read the detailed <a href="https://docs.datafabric.hpe.com/61/AdvancedInstallation/Stanzas/SilentInstaller.html">“how-to” guide here</a>. At the heart of these Stanzas is a YAML file. You must configure a YAML file before using this method to install or upgrade a cluster. Sample YAML files (basic and advanced) can be found in the installer package, but here are the top-level sections:</p> <ul> <li><em><strong>Environment</strong></em> – specifies the mapr_core_version</li> <li><em><strong>Config</strong></em> – specifies the list of nodes (with login information), disks, and other configuration info. Also includes the list of services chosen from pre-existing templates or custom-defined from from MapR Ecosystem Pack (MEP) versions.</li> <li><em><strong>Groups (optional)</strong></em> – selection of services grouped across nodes for advanced layout option</li> </ul> <p>Here’s an example structure for a 3-node cluster:</p> <pre><code class="language-markdown">environment: mapr_core_version: 5.2.0 config: hosts: - demonode[1-3].example.com ssh_id: root license_type: enterprise mep_version: 2.0 disks: - /dev/sdb - /dev/sdc services: template-05-converged: </code></pre> <p>MapR Installer Stanzas come with the following set of commands that can be executed on the command line:</p> <ul> <li><u>Install</u> – use to fresh install, incremental install, and upgrade a cluster</li> <li><u>Uninstall</u> – use to uninstall a cluster</li> <li><u>Export</u> – use to generate a YAML file to capture state of the cluster</li> <li><u>List</u> – list nodes, services, and groups in a cluster</li> </ul> <p><strong>Quick set of steps to get started:</strong></p> <ol> <li><a href="https://docs.datafabric.hpe.com/61/MapRInstaller.html">Download the Installer</a>.</li> <li><a href="https://docs.datafabric.hpe.com/61/AdvancedInstallation/Stanzas/SilentInstaller.html">Review the detailed documentation here</a>.</li> <li>Start building new clusters!</li> </ol> <p>To view more articles on this topic, be sure to check back regularly on the <a href="/blog">HPE DEV blog site</a>.</p>https://developer.hpe.com/crud-with-the-new-golang-client-for-mapr-database/https://developer.hpe.com/crud-with-the-new-golang-client-for-mapr-database/Fri, 18 Sep 2020 20:27:10 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": ["Magnus Pierre"], "publish": "2019-04-24T07:00:00.000Z", "tags": "hpe-ezmeral-data-fabric","hpe-ezmeral" </code></pre> <hr> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image2-1600461098259.png" alt=""></p> <h2>Introduction</h2> <p>HPE Ezmeral Data Fabric Document Database (formerly MapR Database) has come a long way since its inception. Today, the database can not only be used as a replacement of HBase, implementing the HBase API, but also as a document store, implementing an open source API called OJAI (Open JSON Application Interface).</p> <p>Document stores have increased in popularity over the last few years, since they make it easy to store meaningful entities (documents) in a database and then do operations on them without having to retrieve the complete document. A typical example is to ask for a certain subtree of a document identified with a key or a set of documents that fulfill a certain condition. However, documents can also be modified in place, replaced, and deleted – basically, what is traditionally called CRUD (Create, Read, Update, Delete) in the database world. OJAI is an API to provide CRUD and flexible query capabilities to clients in a simple, consistent manner.</p> <h2>New to Document Stores?</h2> <p>For those that have no experience with document stores, here’s a brief explanation, based on a comparison with traditional RDBMS technology.</p> <p>Let’s consider a traditional Third Normal form (3NF) RDBMS database. One reason to divide data into tables/entities is to make data changes easie. Another is to simplify access. The normalization rules all database developers are educated in are there to guide the developers in how to organize/store the data, so it can be accessed and processed in a consistent manner by the consumers. Albeit a very useful and valuable evolution for data storage, the step of splitting the data into separate entities/tables aims to solve historical technical limitations that today are more easily handled. Normalizing and splitting data does make data access simple. It forces you to look at the data and make decisions on type and how it relates to other data, but it also comes with a lot of problems:</p> <ul> <li>The original format is not kept, which creates lineage issues.</li> <li>The data often needs to be transformed in order to fit into the defined entities, increasing the lineage issues as well as introducing new points where data can be wrongly represented.</li> <li>Depending on relationships (i.e. the table structure populated), it can be very complex and time-consuming to ingest new data.</li> <li>Data types (domains) need to be decided, which can be very hard for dynamic data and so on.</li> <li>And above all, it is very work-intensive and error-prone, meaning it takes a lot of time to develop a good database.</li> </ul> <p>Let’s now consider another approach, where we throw all the rules out the window. Instead, we’ll create a table for customers and then store everything related to customers – orders, transactions, address information, products purchased, products offered, along with the complete history of the customer – in the same table. Please ignore the fact that it would not make sense in an RDBMS to do this. Consider it more from a conceptual perspective. Let’s also ignore types and so on. What would be the positive things this could give you?</p> <ul> <li>It would be easy to get a comprehensive overview of a single customer.</li> <li>Joins would be minimized, since most relevant data is already stored together.</li> <li>Conditions and filters could cover the whole scope of the customer engagement easily.</li> <li>Projections (what to return) would be central to extract a portion of the table to work with at a given time.</li> <li>Filtering would basically replace joins as the primary means of deciding what to retrieve.</li> <li>Locking would be simpler, since all information related to a customer would be in a single entity.</li> <li>Less initial work.</li> </ul> <p>What would be the drawbacks?</p> <ul> <li>Harder to change the structure.</li> <li>Since cardinality would be different for different types of data, one would have to deal with this in a more programmatic manner.</li> <li>Duplication of data embedded in the structure.</li> <li>Hard to optimize access for a certain part of the table.</li> <li>Structure would dictate access path.</li> <li>Structure would dictate how you could process the data.</li> </ul> <p>However, provided well-designed tools and APIs are available to operate on these <em>complex</em> structures, then the model of grouping all data together can make sense. This is really what a document database system such as the HPE Ezmeral Data Fabric Document Database provides – i.e., the tools necessary to process complex documents easily. There’s nothing in a document store that defines what the structure needs to look like and how each <em>row</em> can be different. So, in the example of the customer table, it could be one table with everything in it, or N documents with parts in it. The division is decided by the developers and what makes sense from the use case of the data.</p> <h2>HPE Ezmeral Data Fabric Document Database</h2> <p>HPE Ezmeral Data Fabric Document Database supports JSON documents. A JSON document is hierarchical in nature, which I prefer to see as a small table of its own. Each document can have its unique structure in the database. The only requirement on the document, other than it has to be valid JSON, is that it needs to be identified by an ID (_id). This ID can be used to find a unique document, or if the ID is generated using some type of logic, it can be used by filters.</p> <p>In the HPE Ezmeral Data Fabric Document Database, trees in a document can be stored separately in so-called column families. Learn how to parallelize access <a href="https://docs.datafabric.hpe.com/61/MapR-DB/introduction-to-column-families.html">here</a>.</p> <p>You can also improve scans by placing secondary indexes on attributes. Learn about that and more <a href="https://docs.datafabric.hpe.com/61/MapR-DB/Indexes/Indexes.html">here</a>.</p> <p>None of this really matters for our Golang discussion, but you get the concept of a complete database environment supporting the document concept in full.</p> <h2>Enter Language Bindings for OJAI</h2> <p>OJAI was originally a pure Java API, but since the API is really well designed, it fits well in other programming languages without significant changes to the conventions of the API. In order to support new languages, MapR introduced the Data Access Gateway 2.0 (DAG) in MapR Core 6.1. DAG provides the service back end for the language bindings, and in MapR Ecosystem Pack (MEP) 6.0 we shipped NodeJS and Python language bindings for HPE Ezmeral Data Fabric Document Database. Now it’s time to apply the same support for Golang (Go) and C#.</p> <p>If comparing our language bindings with the original API, the API and the functions defined are basically the same in each language. If you understand OJAI in one language, you will also understand it in another language implementation. Of course, there are slight differences, and these are mostly dependent on language specifics. A dynamic language, such as JavaScript, is not the same as Golang, which is strictly typed and where it makes sense that, due to language conventions or capabilities of the language, there might be deviations. But, in general, the language implementations are very close to the original API.</p> <p>Learn about supporting HPE Ezmeral Data Fabric Architecture for the Language Bindings <a href="https://docs.datafabric.hpe.com/61/MapROverview/MapRDataAccessGateway.html#MapRDataAccessGateway">here</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/image1-1600461364270.png" alt=""></p> <p>The client library connects to the Data Access Gateway (DAG) over gRPC, and the Data Access Gateway talks directly with the HPE Ezmeral Data Fabric Document Database using MapR RPC.</p> <p>Multiple Data Access Gateways can run on the same cluster, and you can integrate an external load balancer for high availability and failover of the DAG service. We have found gRPC DAG 2.0 is as fast as Java OJAI client performance. Always scale test your application and have at least two DAG services behind load balances for HA.</p> <h2>The Golang Client</h2> <p>Golang, or “Go,” hardly needs an introduction. It is a fairly new and upcoming programming language out of Google, which is as fast as C but much easier to write. It is used in real production environments at Google. Good open source tools, such as Kubernetes and Docker, have been written in it, so it is quite battle-tested already.</p> <h2>Installation of the HPE Ezmeral Data Fabric Document Database Golang Client</h2> <p>Installation is very simple. In the project where you are to use the client, install it using: <code>go get github.com/mapr/maprdb-go-client</code></p> <blockquote> <p>Please observe that you need to have Golang installed on the system prior to running the command.</p> </blockquote> <p>Once you have done this step, you can start using the API in the go files of your choice. A good convention, when using this API, is to provide an alias to the package in the import to get a simpler naming convention in the code itself:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-1-1600461373867.png" alt=""></p> <p>This will define client as the alias for the go client, meaning that everything you use from the API is referenced using client and then the function call you want to make.</p> <h2>Make a Connection</h2> <p>Connections in the Golang client are established using a connection string. A description of the parts included in a connection string can be found <a href="https://docs.datafabric.hpe.com/61/MapR-DB/JSON_DB/GettingStartedGoOJAI.html">here</a>.</p> <p>Some examples of connection strings:</p> <ul> <li>dag-somesystem.com:5678?auth=basic;user=mapr;password=somepasswd</li> <li>ojai:mapr:thin:v1@localhost:5768?auth=basic;user=fred;password=george;sslCA=/opt/app/conf/rootca.pem</li> </ul> <p>In the example code, we use a very simple convenience function to create the connection string, based on some values that reside in a <code>Struct</code>:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-2-1600461384388.png" alt=""></p> <h2>Create / Get a Table</h2> <p>Once you have the connection string, you can connect to the HPE Ezmeral Data Fabric Document Database and connect to a database. In this example code, it will create a database if the database asked for does not exist. This is probably not what you would like to do in production, but for simple loads and test purposes, this is very convenient:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-3-1600461391951.png" alt=""></p> <h2>Insert Data</h2> <p>OJAI contains several convenience functions to, for instance, make sure you create valid documents, assign fields, and much more.</p> <p>Here is a simple function that does the following:</p> <ul> <li>It receives a simple message of the type string that is used to create a Document by calling the function <code>CreateDocumentFromString</code>.</li> <li>It assigns the ID to <code>string(time.Now())</code>, using the OJAI <code>Document.SetIdString</code> function (not the best key but fits this simple example).</li> <li>It assigns a simple attribute called test the value 0 of the type Float64 in the Document, using <code>Document.SetFloat64</code>.</li> <li>It prints out the Document as a JSON, using the <code>Document.AsJsonString()</code> function to stdout.</li> <li>It adds the Document to the database, using <code>DocumentStore.InsertDocument()</code>.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-4-1600461399780.png" alt=""></p> <h2>Query data</h2> <p>Querying data in OJAI can be done in a few different ways:</p> <ul> <li><code>Find by ID</code></li> <li><code>Find by Query</code></li> <li><code>Find All</code></li> <li><code>FindQueryString</code></li> <li><code>FindQueryMap</code></li> </ul> <p><code>Find by ID</code> searches the system for a specific ID. As you remember, the ID is the row key identifying the document, so this is blazingly fast. <code>Find All</code> is the equivalent of a scan in HBase.</p> <h2>Making a Query</h2> <p>Queries are made using the convenience function, <code>MakeQuery</code>. <code>MakeQuery</code> receives a variant list of <code>QueryOptions</code>, which represents <code>SELECT</code>, <code>WHERE</code>, <code>ORDER BY</code>, <code>OFFSET</code>, and <code>LIMIT</code> and produces a <code>Query Object</code>. It has the same basic structure as Mutations and Conditions, and you work with them in a similar fashion. You also have the option to define an OJAI query directly in a JSON string and use it directly.</p> <p>A simple query looks like the following:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-5-1600461407820.png" alt=""></p> <h2>How to Change/Update Data</h2> <p>Data is changed through the concept of mutations. Mutations are executed on the server and are applied using a <code>DocumentMutation</code> struct. You create a <code>DocumentMutation</code> struct and provide the wanted <code>MutationOperations</code> to the function <code>MakeDocumentMutation</code>.</p> <p>The following mutations exist in OJAI and the respective convenience functions in the Go-client:</p> <ul> <li>Append <ul> <li>Read-modify-write operation. Use it to append specified values to existing binary, string, or array type fields.</li> <li>Go convenience functions: <ul> <li><code>AppendSlice</code></li> <li><code>AppendString</code></li> </ul> </li> </ul> </li> <li>Decrement <ul> <li>Decrements the value in the <code>fieldpath</code>. To decrement multiple field paths, use an array notation to list the field paths.</li> <li>Go convenience functions: <ul> <li><code>DecrementFloat64</code></li> <li><code>DecrementFloat64ByOne</code></li> <li><code>DecrementInt</code></li> <li><code>DecrementIntByOne</code></li> </ul> </li> </ul> </li> <li>Delete <ul> <li>Removes either a single field or a list of fields from a document. If the field does not exist, the delete ignores that field.</li> <li>Go convenience functions: <ul> <li><code>Delete</code></li> </ul> </li> </ul> </li> <li>Increment <ul> <li>Increments the value in the <code>fieldpath</code>. To increment multiple field paths, use an array notation to list the field paths.</li> <li>Go convenience functions: <ul> <li><code>IncrementFloat64</code></li> <li><code>IncrementFloat64ByOne</code></li> <li><code>IncrementInt</code></li> <li><code>IncrementIntByOne</code></li> </ul> </li> </ul> </li> <li>Merge <ul> <li>Combines a nested document with an existing document at a specified fieldpath</li> <li>Go convenience functions: <ul> <li><code>MergeDocument</code></li> <li><code>MergeMap</code></li> </ul> </li> </ul> </li> <li>Put <ul> <li>Replace operation</li> <li>Go convenience functions: <ul> <li><code>SetOrReplace</code></li> </ul> </li> </ul> </li> <li>Set <ul> <li>Updates one or more fields in a document.</li> <li>Go convenience functions: <ul> <li><code>Set</code></li> <li><code>Set&#x3C;Go Type> example SetInt</code></li> </ul> </li> </ul> </li> </ul> <p>It is also possible to provide the mutation as a map of the form: <code>map[string] interface{}</code>, where each entry is of the same form – i.e., <code>map[string] interface{}</code>.</p> <p>It is quite easy to create a static map, but if you need to generate maps and add them dynamically, a convenience function is probably needed that merges maps into a map and returns that map. I have an example in the GitHub repository of such a convenience function, if of interest, called <code>[MergeMaps](https://github.com/mapr/maprdb-go-client/search?q=MergeMaps&#x26;type=code)</code>.</p> <p>Since the two different forms of creating mutations exists, a set of functions exists to convert the mutation type to a form that can be sent to the database:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-6-1600461418004.png" alt=""></p> <p>These functions are:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/code-7-1600461425614.png" alt=""></p> <p>Now we can send the <code>MapOrStructMutation</code> to the database using:</p> <ul> <li><code>Update</code>, <code>CheckAndUpdate</code></li> </ul> <p><code>CheckAndUpdate</code> is interesting, since it allows you to only do the update if a certain condition is validated to be true. This is where the OJAI API shows its true powers. By doing proper checking this way, you can avoid having to return the data to the client, do the manipulation, and then write it back to the database. All steps can happen in the database, where they belong, local to the data. This is a fundamental feature, important for everyone that deals with mutable structures with big data sizes.</p> <h2>Conditions</h2> <p>Conditions share the same structure with mutations, i.e., the same underlying type structure, so you work with it in a similar fashion. A good way of understanding conditions comes from the HPE Ezmeral Data Fabric Document Database official documentation about writing conditions directly in JSON. With the Go client, it is not strictly necessary to write the conditions this way, however, since there’s convenience functions for each defined. Examples of conditions exist in the <a href="https://github.com/mapr/maprdb-go-client">referenced GitHub</a> for more information.</p> <h2>How to Delete Documents</h2> <p>Delete is very straight-forward and easily conducted, using the convenience functions: <a href="https://github.com/mapr/maprdb-go-client/search?q=DeleteByIdBinary"><code>DeleteByIdBinary</code></a>, <a href="https://github.com/mapr/maprdb-go-client/search?q=DeleteByIdString"><code>DeleteByIdString</code></a>, and <a href="https://github.com/mapr/maprdb-go-client/search?q=DeleteDoc"><code>DeleteDoc</code></a>.</p> <p><code>DeleteByIdBinary</code> and <code>DeleteByIdString</code> deletes a document based on identity _id, whilst <code>DeleteDoc</code> bases the delete on a Document on the client. Whether the <code>DeleteDoc</code> matches the complete document before deleting or if it just matches the identity of the document and then deletes is unknown by me. My guess is that it matches ID, but I will validate this.</p> <h2>Behind the Scenes of OJAI</h2> <p>OJAI clients use the Data Access Gateway 2.0 introduced in MapR 6. However, in order to support good things, such as secondary indexes and queries on massive volumes of data, OJAI uses an internal service called OJAI Distributed QueryService. This QueryService is based on Apache Drill and utilizes the good capabilities of Drill to query massive volumes of data in an efficient manner. Drill is a distributed query engine that can query almost anything, but in this case, it is geared towards only asking questions against the HPE Ezmeral Data Fabric Document Database. By utilizing Drill, OJAI gets access to advanced SQL behind the scenes, a good cost-based query planner, and, of course, the parallel execution engine for processing the intermediary steps that may be the result of a complex query.</p> <p>HPE Ezmeral Data Fabric has the ability to run multiple Drill clusters on one of the same clusters. In production, it may be important to isolate the load from OJAI clients using the Drill QueryService from regular use of Apache Drill, and this is easily doable by assigning multiple Drill clusters. Please see the official documentation regarding this <a href="https://docs.datafabric.hpe.com/61/Drill/config-multiple-drill-clusters.html">here</a>.</p> <h2>Summary</h2> <p>This blog post has focused on describing how to do CRUD using the Golang client for the HPE Ezmeral Data Fabric Document Database. The client provides all the tools necessary to do real applications with great database support on massive amounts of data. Mutations with conditions are key to being able to mutate data in a big data setting, since:</p> <ol> <li>You cannot return the data to the client due to volume.</li> <li>Even if you could, the round trip to fetch the data to the client, do the change, and then back again would be very costly.</li> <li>The database architecture also makes it possible to do proper applications in, for instance, simple languages such as JavaScript that you would normally not use when processing millions upon millions of rows of data.</li> </ol> <p>The Go client for the HPE Ezmeral Data Fabric Document Database supports really complex OJAI statements and is backed up with an architecture that is scalable and reliable.</p> <p>First, we have the gateway service, which can be individually scaled across the cluster and load balanced to support massive amounts of clients. Then, we have the database with its always-on and distributed architecture, where a document can be distributed across column families for parallelization with support for in-database operations, such as mutations and support for secondary indexes. Finally, we have the query service for complex queries built on Apache Drill with its scalable architecture, leading to basically an infinitely scalable solution, where each component is individually scalable based on need.</p> <p>Make sure to take a deep look at the client that fits your needs best – Node.js, Python, Golang, C#, and of course Java. You will see that they share the same wonderful architecture, the same ease of use, and the same wonderful API, making it a very good base for future applications.</p> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p>To view more articles on this topic, be sure to check back regularly on the <a href="/blog">HPE DEV blog site</a>.</p>https://developer.hpe.com/containers-best-practices-for-running-in-production/https://developer.hpe.com/containers-best-practices-for-running-in-production/Wed, 16 Sep 2020 16:06:50 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suzy Visvanathan", "publish": "2018-05-17T10:45:00.000", "tags": "hpe-ezmeral-data-fabric","hpe-ezmeral" </code></pre> <hr> <p>Market shifts are propelling organizations to become digital. Companies feel pressure to find better ways to offer services, respond faster and proactively to their customers, and manage growth. In their quest for digital maturity, customers are looking for solutions that not only address the aspect of capacity growth, but also other aspects such as performance, handling of different types of data through different protocols, and ability to adopt newer technology trends that makes them more efficient.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/9/containers-wide-1600290064568.png" alt=""></p> <p>Containers is one technology trend that is propelling a transition for IT. With the advent of Docker, containers became a popular method for DevOps and are quickly being adopted in production as well. But as I talk to customers, I find that many are still quite undecided about adopting containers, especially in production environments. There are several nuances to keep in mind when considering deploying applications in containers in production:</p> <ol> <li> <p><strong>Start small.</strong> Unless you have already made an organization-wide decision to deploy containers, start with a handful of containers and measure the time it takes to build and deploy an application on containers versus, say, in VMs or bare metal. This will give you an idea of whether it is worthwhile moving to containers or not. Since containers consume CPU resources, such an exercise will help you plan your cluster configuration as well.</p> </li> <li> <p><strong>Orchestration tool?</strong> Google search on this topic will produce a litany of articles that compares Kubernetes with Mesos or a Docker swarm and proclaims how Kubernetes has won the war. While that is great, most of you don’t need to start right away with any of these tools. Don’t get me wrong: I am a big proponent of these orchestration tools, but if you envision having just a few dozen containers in the next year or two, you are better off managing the environment manually, since it will give you good hands-on insight into how your environment behaves. If you are thinking instead of deploying hundreds or thousands of containers, by all means choose an orchestration tool.</p> </li> <li> <p><strong>Persistent storage for containers.</strong> When experimenting with containers, you can probably make do with local storage. If, on the other hand, you are contemplating running containers in production, then you will need persistent storage. Choose a data platform that is versatile in what it offers and not just storage for containers. A platform that has been built from the ground up for scale, reliability, and security with the ability to expand to newer technology trends is the ideal choice.</p> </li> <li> <p><strong>Network considerations.</strong> There are several CNI (Container Network Interface) solutions that are now available. Docker itself offers a pluggable interface and third-party vendors like Calico and Flannel offer solutions. Existing network frameworks in your data center should suffice for most deployments but evaluating CNI solutions to assess fit and needs in your data center is worth contemplating.</p> </li> <li> <p><strong>Standardizing on containers.</strong> When building new applications, take into account design considerations to run these applications in containers, even if they won’t be deployed as containers in the short term. This will save a lot of time and cost in reformatting them later.</p> </li> <li> <p><strong>Containers and microservices.</strong> Containers enable applications to be broken down into smaller modules, paving the way for a microservice architecture. Not all applications, especially legacy ones, can be easily broken down and may end up being more expensive to refactor, especially if you are moving to the cloud.</p> </li> </ol> <p>HPE offers several capabilities to address many of the above highlighted best practices. The <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a> brings together a solid foundation with a distributed, scalable file system and an upper layer that caters to diverse types of data – files, tables, containers, and more. The Ezmeral Data Fabric for Kubernetes extends its capabilities to offer persistent storage for containers.</p> <p>Armed with the right amount of education and access to community, one can easily make the transition to deploying containers in production. Taking a page out of DevOps, having operations teams understand, deploy, and educate other stakeholders will speed up understanding of the benefits and therefore adoption.</p>https://developer.hpe.com/boost-skills-with-free-on-demand-software-technology-workshops/https://developer.hpe.com/boost-skills-with-free-on-demand-software-technology-workshops/Wed, 16 Sep 2020 12:46:52 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/gettyimages-860310256_1600_0_72_rgb-1600260582799.JPG" alt="gettyimages 860310256_1600_0_72_rgb"></p> <p>Given the physical restrictions placed on us by the COVID-19 pandemic today, everyone appears to be finding new ways to connect and learn.</p> <p>It is in this spirit that the HPE DEV team is making their highly popular Jupyter Notebook-based workshops available for free. These on-demand workshops, which premiered at the HPE Discover Virtual Event, have met with rave reviews worldwide. They provide developers, data scientists, and IT professionals the opportunity to get more hands-on experience working with HPE and open source technologies.</p> <p>These on-demand workshops offer students an understanding of specific technologies through a hands-on experience guided by a subject matter expert (SME) and detailed instructions provided in a Jupyter Notebook format. Students will have access to other SMEs through Slack should any questions arise during their session. In this blog post, I’ll cover the details of this learning-on-demand program.</p> <p>Unlike the video <a href="/hackshack/replays/0">replays</a> found on the <a href="/hackshack/">HPE DEV Hack Shack</a>, these on-demand workshops will allow students to interact with the Jupyter Notebook and get hands-on experience with coding practices. Students will have 4 hours to go through the course, which includes time to review the video replay, follow the Jupyter Notebook instructions, and save their work to their local laptop should they want to do more work in their own environment or retake the workshop. They’ll also be able to connect with SMEs through off-line support. For those who are new to the topic being covered, it is recommended you watch the corresponding <a href="/hackshack/replays/0">video replay</a> first and then sign up for the course.</p> <h2>How it works</h2> <p>Those who wish to enroll in the workshop should go to the <a href="/hackshack/">Hack Shack</a> and navigate to the <strong>Workshops</strong> page. From there, select which on-demand workshop you want to take and click on the <strong>Register</strong> button:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/navigate-register-button-1600260600088.png" alt="navigate register button"></p> <p>At this point, the registration panel pops up. Enter the details requested and click on the <strong>Take the Workshop</strong> button. In a matter of just a few minutes, you’ll start your workshop.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/take-the-workshop-pramod-creds-highlighted-1600260627823.png" alt="take the workshop pramod creds highlighted"></p> <p>By pressing the <strong>Take the Workshop button</strong>, you initiate a back-end automated registration process. This process spawns a dedicated notebook environment for you and sends you a welcome email indicating that you have been registered in the database.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/welcome-email-modified-1600260633309.png" alt="welcome email modified"></p> <p>Not long after, it sends you a second email providing a link to your workshop, along with your StudentID and password.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/highlighted-clock-1600260594291.jpg" alt="highlighted clock"></p> <blockquote> <p><strong>IMPORTANT: Receipt of this email indicates that the workshop environment is ready for you to begin. You will have just 4 hours from the receipt of this second email to complete the workshop. It is recommended that you only register for a workshop when you know you will have the next 4 hours to work on it. We advise you to regularly save your work and download the Jupyter Notebook to refer to later should you not be able to finish the course within the given 4-hour time slot. If you cannot finish the workshop in that time, you will need to run the course again from the beginning.</strong></p> </blockquote> <h2>The Jupyter Notebook-based workshops</h2> <p>A Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. The Jupyter Project was an important step forward for sharing and interactive development. <a href="https://jupyter.org/index.html">Project Jupyter’s</a> <a href="https://jupyterhub.readthedocs.io/en/stable/">JupyterHub</a> was created to support many users. The Hub can offer notebook servers to an entire class of students, a corporate data science workgroup, a scientific research project team, or a high-performance computing group. With <a href="https://github.com/jupyterhub/jupyterhub">JupyterHub</a>, you can create a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user <a href="https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb">Jupyter Notebook</a> server.</p> <p>As explained in Fred Passeron’s earlier post, <a href="/blog/jupyter-saved-my-day">Jupyter saved my day</a>, for our on-demand workshops the notebooks contain simple Python or Powershell pieces of codes to interact with the different APIs available in the HPE portfolio. All instructions are provided in a markdown format. We centralize the different notebooks on a single JupyterHub server.</p> <p>When you click on the <strong>Start Workshop</strong> button found in your second email, it will bring you to a <strong>Sign In</strong> page where you will log into the workshop with your StudentID and the password provided in your second email.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/workshop-screen-1-3-1600260638100.png" alt="workshop screen 1 (3)"></p> <p>Once you log in, open the workshop folder on the left by double-clicking on it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/workshop-screen-2-1600260642513.png" alt="workshop screen 2"></p> <p>Each notebook generally has several sections. Start with the <strong>Read Me First</strong> and follow the instructions from there.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/workshop-screen-3-1600260646340.png" alt="workshop screen 3"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/read-me-first-page-1600260610550.png" alt="read me first page"></p> <h2>Don’t forget to save your work</h2> <p>One hour prior to the end of the 4-hour period, you will receive an email reminding you that your session is coming to a close and that you should download the workshop notebook if you anticipate using it in the future. At the end of every session, the environment is cleaned up automatically, so be sure to save your work.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/one-hr-warning-email-1600260622217.png" alt="one hr warning email"></p> <p>At the end of the workshop, you will receive a final email indicating that the workshop is over. In the email, you will also be asked to take a short survey. The results of this survey will help us improve how we offer the workshops-on-demand in the future. Your feedback is very important to our being able to meet your needs, so we encourage you to take just a few minutes to fill it out.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/end-workshop-survey-1600260589980.png" alt="end workshop survey"></p> <h2>How to get help</h2> <p>To receive SME assistance through our <a href="https://hpedev.slack.com/archives/C01B60X8SSD">workshop Slack channel</a>, make sure you join the <a href="https://slack.hpedev.io/">workspace</a>. We staff the channel to answer questions between 4 am and 4 pm EST Monday through Friday. You may wish to ensure you schedule the timing of your course between those hours if you think you’ll have questions or need additional help.</p> <p>There will be a limited number of seats available. These seats will be filled on a first-come/first-serve basis. We look forward to the opportunity of offering these workshops to you. Remember to <a href="/hackshack/workshops">check the Hack Shack workshops library</a> regularly for any further updates.</p>https://developer.hpe.com/open-source-elasticsearch-helped-us-globally-support-virtual-labs/https://developer.hpe.com/open-source-elasticsearch-helped-us-globally-support-virtual-labs/Thu, 10 Sep 2020 14:39:09 GMT<img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-0-1600073593195.png" alt="ElasticSearch + Kibana"> <p>Even if you are not a hard-core developer, you may have heard about the open source search and analytics engine ELK. The acronym ELK stands for <a href="https://www.elastic.co/home">Elasticsearch + Logstash + Kibana</a>. Though many have heard the term ELK, some have never really had any hands-on experience with it. This was the case with me until it was decided the HPE DEV team would offer software code challenges during our recent HPE Discover Virtual Experience.</p> <p>The team was faced with the difficult task of having to monitor all the registrations that came in for these challenges over the course of the multi-week event, as well as identify those who finished them and when. Since the HPE Discover Virtual Experience was a worldwide event, registrations for any of our four code challenges could come in at anytime from anywhere on Earth. I decided to put ELK to the test to see if it could help us figure this out.</p> <h2>What is ELK?</h2> <p>ELK is a highly-scalable distributed, open source search and analytics engine used for all types of structured and unstructured data, including textual, numerical, and geospatial. Kibana is an open source data visualization dashboard created for Elasticsearch. Logstash, also part of the stack, is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch.</p> <p>Elasticsearch started as an open source project and, like many open source projects, grew exponentially from a few downloads in 2010 to 250 million downloads in 2018. Out of this open source maelstrom came additional software components, such as Logstash and Kibana, which now complete this very popular software stack. The Elastic business model adds commercial software on top of the open source ELK Stack.</p> <p>To support those attempting our code challenges, we initially set up a Slack Channel on our Slack workspace monitored by enough team members to cover all time zones. The registration process was designed to automatically spawn a dedicated notebook environment on the JupyterHub server in our datacenter and send mails to customers to get them connected to start working on their challenge.</p> <p>While all data relative to the registration process was stored in a database, we didn’t have time to build a front-end application to manage this information, so not all support staff had access to it. We were left with the question: how would each of us know if there were one or more challenges going on at any point in time? We needed some sort of dashboard to display this information that all team members could quickly look at, from anywhere, at any time.</p> <h2>Keep it simple</h2> <p>Being tasked with building a dashboard meant I could finally get my hands dirty with ELK. Within a few minutes, I had both <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/install-elasticsearch.html">Elasticsearch</a> and <a href="https://www.elastic.co/guide/en/kibana/7.9/install.html">Kibana</a> installed on the Linux virtual machine on which our registration process was hosted. It was easy to do, as I used the Elastic web site, which has extremely precise installation instructions for both Elasticsearch and Kibana. Though I had the Elasticsearch instance, its store was empty and I still needed to feed data to it. I needed to get the right data store there before I could build a Kibana dashboard.</p> <h2>There is an API? Then yes, we can do it</h2> <p>There is only one way to interact with Elasticsearch and it’s using its API which by default, installs on port 9200. If this is a problem for you, please take a look at my article on <a href="/blog/understanding-api-basics-and-the-value-they-provide">understanding the basic principles of REST API</a>.</p> <p>My colleague Pramod, who had built the registration process application, was smart enough to provide a REST API to its backend application. This enabled me to retrieve from the database:</p> <ul> <li>Customer details</li> <li>Challenge details</li> <li>Booking details</li> </ul> <p>The API returned JSON. Elasticsearch also uses JSON, so it all looked too easy, giving the impression that I only needed:</p> <ol> <li>One REST API call to retrieve customer data from the registration application</li> <li>Another API call to upload this data into Elasticsearch</li> </ol> <h2>NDJSON?</h2> <p>It turns out that it wasn’t quite as easy as that. In order to bulk upload data in Elasticsearch, you have to feed it with not JSON but NDJSON, which I had never heard of. NDJSON stands for Newline Delimited JSON. So, it looked like a little <strong>data massaging</strong> exercise was needed.</p> <p>So, my new algorithm was:</p> <ol> <li>GET JSON customer data from registration App customer API</li> <li>Prepare JSON data for bulk load</li> <li>POST NDJSON data to Elasticsearch customer index in bulk mode</li> </ol> <h2>Trial and error</h2> <p>I experimented with the Elasticsearch REST API for a few hours in order to create a customer index and populate data for this index. I exclusively used the following single call to get this done:</p> <pre><code class="language-bash">curl -H "Content-Type:application/x-ndjson" -X POST "http://localhost:9200/customers/_bulk" --data-binary @customers.json </code></pre> <p>When things went wrong with the format of my NDJSON file, I deleted my index and restarted fresh using:</p> <pre><code class="language-bash">curl -H "Content-Type:application/x-ndjson" -X DELETE "http://localhost:9200/customers” </code></pre> <p>There are a lot of great articles available to help with formatting the data for bulk import, but <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/docs-bulk.html">this particular one</a> was of great help to me.</p> <p>I found that calling our own application to retrieve customer details was easy enough. I finally had all the individual pieces nailed down. All I needed next was to write a script to orchestrate all of this.</p> <h2>My worst nightmare: Regular Expressions</h2> <p>I know there are lots of programming languages that can perform powerful string manipulations. Specifically, I wanted something quick (like bash?), which didn’t require anything else installed on the system (like bash?), so I decided to manipulate the JSON file using a simple sed command in a bash (yes bash!) script. My regular expressions were a little rusty, so I struggled a bit, but I got help from a hard-core Linux colleague (who prefers to remain anonymous, but writes lots of great Redfish blog posts for us). His motto is “I can do anything with a RegEx!”. So finally, I managed to turn the input JSON data into ready-to-be-imported NDJSON.</p> <blockquote> <p>Note: It did require five successive sed commands, not just a single one as anticipated.</p> </blockquote> <p>That was it! I now had a script that would load the latest customer configuration into my Elasticsearch. I added this script in a cron table on our server, so it ran every 15 minutes, and I was now ready to move on to building the dashboard.</p> <h2>Simple yet secure</h2> <p>Kibana has a web interface that, by default, starts on port 5601. However, for a more secure approach, we used nginx (which was already installed on our server) as a reverse proxy for Kibana. I found a great article that helped me handle this configuration.</p> <p>The steps involve manipulating nginx configuration files under /etc/nginx/sites-available to:</p> <ul> <li>Create a kibanaadmin user using basic authentication</li> <li>Create a reverse proxy from our system port 8080 to Kibana 5601</li> <li>Enable SSL</li> </ul> <p>Our config file looked like this:</p> <pre><code class="language-markdown">server { listen [::]:8080 ssl ipv6only=on; # managed by Certbot listen 8080 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/ourservernamegoeshere/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/ourservernamegoeshere /privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot server_name ourservernamegoeshere.hpedev.io; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } </code></pre> <p>After restarting nginx, the team could log into our Kibana web interface from anywhere and start building shiny dashboards.## It’s dashboard time!</p> <p>Kibana dashboards are made of individual tiles called virtualizations and Kibana offers lots of different virtualization widgets.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-1-1600073613043.png" alt="Kibana virtualization options"> <p>As you can see, there are many types of visualization tools you can use and configure using the fields found in your index. In our case, we had the following data available in our customers’ index:</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-2-1600073620575.png" alt="ElasticSearch fields showing up in Kibana"> <p>I decided to keep it simple and visualise the following data points:</p> <ul> <li>Total number of active challenges</li> <li>Active challenges by type</li> <li>Total customers who registered</li> <li>Total challenges that were taken</li> </ul> <p>In no time, I was able to get my first version of a challenge dashboard.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-3-1600114029770.png" alt="Kibana dashboard"> <p>It was a good start, but I also wanted to know when the active challenges had started. So, I decided to add another visualisation to show the start time in a Date Histogram, as shown below.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-4-1600114050511.png" alt="Kibana dashboard"> <p>This looked really nice because, not only could we immediately see if there were challenges in flight, but we could also check what happened during the previous night, for example.</p> <h2>There is an API? Then yes, we can do it… again</h2> <p>Because our challenge participants had to use Github to collect the source material for the challenges and open pull requests (PR) to submit their responses, the next thing that we wanted to visualise was activity on the GitHub repos. But wait, GitHub has an API right?</p> <p>Absolutely! So, I went back to the data collection script, and now that I knew about NDJSON and regex, I was able to collect data for each of our Github repos in no time using their APIs:</p> <ul> <li>Number of Forks</li> <li>Number of Pull Requests</li> </ul> <p>I created a new index in Elasticsearch for Github data and updated my script to load the Github data like I had done previously (more sed and regex). Finally, I created a new tile to visualize the GitHub data.</p> <p>Et voilà: <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-5-1600114070320.png" alt="Kibana dashboard"></p> <h2>Ready to drill down?</h2> <p>There were so many things we could have added to our dashboard; it seemed limitless. But I discovered that you could actually drill down by right-clicking on certain tiles. So, I decided to implement drill down to display the emails of our participants. That was a nice enhancement as it allowed us to immediately know who was participating and be in a position to provide support. Again, even that was pretty easy and smooth, even for a newbie in Kibana.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/elk-6-1600073644276.png" alt="Kibana drill down"> <h2>Conclusion</h2> <p>I’ve only scratched the surface of what you can do with the ELK Stack in this post. And this was a result of spending just a few hours on it only using the community part of the stack. There is a lot more to the ELK Stack, both in terms of data that we could feed in (complete system monitoring using Beats/Logstash for example) and Kibana widgets we can use to display data in various forms. Give it a try and, like me, you may get instantaneously hooked. Don’t forget to check back on our <a href="/blog">HPE DEV blog</a> for more tutorials on interesting open source topics.</p>https://developer.hpe.com/all-hpe-oneview-ecosystem-sdks-now-support-oneview-53-automation/https://developer.hpe.com/all-hpe-oneview-ecosystem-sdks-now-support-oneview-53-automation/Fri, 04 Sep 2020 06:50:36 GMT<p>With our newest update, all of the HPE OneView Ecosystem SDKs now support <a href="https://www.hpe.com/us/en/integrated-systems/software.html?chatsrc=ot-en&#x26;jumpid=ps_ixkqsmug5a_aid-520023673&#x26;gclid=EAIaIQobChMIpPL8tqD16QIVUfDACh0i3g6WEAAYASAAEgLJ-_D_BwE&#x26;gclsrc=aw.ds">HPE OneView 5.3</a> (REST API version 1800) automation features. The HPE OneView Ecosystem actively supports <a href="https://github.com/HewlettPackard/oneview-ansible">Ansible</a>, <a href="https://github.com/HewlettPackard/oneview-python">Python</a>, <a href="https://github.com/HewlettPackard/oneview-golang">Golang</a>, <a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.3.0">Terraform</a>, <a href="https://github.com/HewlettPackard/oneview-chef">Chef</a>, <a href="https://github.com/HewlettPackard/oneview-puppet">Puppet</a>, <a href="https://github.com/HewlettPackard/POSH-HPOneView">PowerShell</a>, and <a href="https://github.com/HewlettPackard/oneview-sdk-ruby">Ruby</a> SDKs.  The updated SDKs allow IT organizations to leverage the latest HPE OneView features and provide the ability to automate software-defined infrastructure from core to cloud. With such a diverse partner ecosystem, IT organizations benefit from being able to integrate HPE OneView within their preferred platform's existing management frameworks.</p> <p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> uses software-defined intelligence via a template-driven approach to automate the deployment, provisioning, updating, and integration of resources, such as compute, storage, and networking infrastructure. Designed with a modern, standards-based API, HPE OneView enables IT administrators to deploy and update using only a single line of code. This makes composing new infrastructure not only faster and more agile, but also more predictable.</p> <p>HPE offers SDKs for industry-leading software deployment, provisioning, and configuration management tools, including Ansible, Terraform, Chef, and Puppet. HPE also provides API language support for Python, Ruby, Golang, and PowerShell languages. Developers can therefore easily build integrations, custom automations and scalable solutions. The SDKs allow for integration with cloud-based platforms to automatically provision and configure new machines, enabling administrators to create a resource topology similar to that of a public cloud on their own physical infrastructure.</p> <p>To simplify installation and maintenance for customers with container environments.  HPE OneView 5.3 SDKs are available as Docker images. <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">Ansible</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">Terraform</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">Chef</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">Puppet</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">Python</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">Golang</a>, and <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">Ruby</a> SDKs are now all available on Docker Hub. All prerequisite materials are incorporated into the container images to enable streamlined deployment, which will simplify maintenance, improve infrastructure agility, and reduce costs.</p> <h2>For more information:</h2> <p>HPE DEV OneView: <a href="https://developer.hpe.com/platform/hpe-oneview/home">https://developer.hpe.com/platform/hpe-oneview/home</a></p> <p>Ansible GitHub release: <a href="https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.7.0">https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.7.0</a></p> <p>Ansible Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">HPE OneView SDK Docker Image for Ansible</a></p> <p>Terraform GitHub Release: <a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.4.0">https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.4.0</a></p> <p>Terraform Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">HPE OneView SDK Docker Image for Terraform</a></p> <p>Chef GitHub Release: <a href="https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.5.0">https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.5.0</a></p> <p>Chef Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">HPE OneView SDK Docker Image for Chef</a></p> <p>Puppet GitHub Release: <a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.7.0">https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.7.0</a></p> <p>Puppet Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">HPE OneView SDK Docker Image for Puppet</a></p> <p>Python GitHub Release:  <a href="https://github.com/HewlettPackard/oneview-python/releases/tag/v5.3.0">https://github.com/HewlettPackard/oneview-python/releases/tag/v5.3.0</a></p> <p>Python Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">HPE OneView SDK Docker Image for Python</a></p> <p>Golang GitHub Release: <a href="https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.4.0">https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.5.0</a></p> <p>Golang Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">HPE OneView SDK Docker Image for Golang</a></p> <p>Ruby GitHub release: <a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.15.0">https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.15.0</a></p> <p>Ruby Docker Hub: <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">HPE OneView SDK Docker Image for Ruby</a></p> <p>PowerShell GitHub release: <a href="https://github.com/HewlettPackard/POSH-HPEOneView/releases/tag/v5.30.2515.1313">https://github.com/HewlettPackard/POSH-HPEOneView/releases/tag/v5.30.2515.1313</a></p>https://developer.hpe.com/2020-September-01/https://developer.hpe.com/2020-September-01/Tue, 01 Sep 2020 05:00:00 GMThttps://developer.hpe.com/hpe-dev-technology-workshops-to-infinity-and-beyond/https://developer.hpe.com/hpe-dev-technology-workshops-to-infinity-and-beyond/Fri, 28 Aug 2020 14:36:25 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/didier1-1598625790616.jpg" alt="didier1"></p> <p>In an earlier <a href="/blog/jupyter-saved-my-day">blog post</a>, one of our team members, <a href="https://twitter.com/FredPasseron">Frederic Passeron</a>, discussed how Jupyter Notebooks saved the day as we transitioned from traditional face-to-face events to virtual events. As a matter of fact, with the adoption of this open source technology, our team was able to deliver over 60 live, hands-on workshops to our developer audience since having to go virtual during this pandemic.</p> <p>We first offered this hands-on experience at technology events sponsored by Hewlett Packard Enterprise (HPE), such as the Technology and Solutions Summit (TSS), ASPIRE, and finally the HPE Discover Virtual Experience event this summer. Based on the feedback we’ve received from participants, these workshops were highly successful and proved to work well for remote, virtual environments.<br> <br/> Their success indicates that Jupyter Notebook-based workshops should have a life that extends beyond the individual events for which they were created. As such, we have made a number of decisions relative to these workshops and how they’ll continue to be made available to the HPE Developer Community going forward. Here's what we have decided to do:</p> <ul> <li>Keep the virtual, online <a href="/hackshack">Hack Shack</a> open between events</li> <li>Create workshops that can be deployed on-demand</li> <li>Maintain a library of Jupyter Notebooks that have been developed for these workshops</li> </ul> <br/> <p>Because people were only just starting to learn about our virtual workshops, we saw that there was an important need to keep <a href="/hackshack">the HPE Hack Shack</a> open between events we support. We will continue to customise and roll it out at upcoming venues, like we just did for KubeCon EMEA and we plan to do for KubeCon NA, but between events the experience will remain in place online, offering workshop replays, code challenges, an arcade section, and more.</p> <p>We also determined that it would be important to make our workshops available on-demand. On-demand versions will start to be rolled out this fall. Stay tuned for an announcement of the pilot program.</p> <p>Finally, we decided to create a home for the Jupyter Notebooks we developed for these recent events and those we plan on developing in the future. We created a <a href="https://github.com/HewlettPackard/hpe-notebooks">GitHub repo</a> to house these Jupyter Notebooks and a process by which one can submit them (through Pull Requests and following <a href="https://github.com/HewlettPackard/hpe-notebooks/blob/master/CONTRIBUTING.md">these instructions</a>), so the entire community can contribute their own notebooks.</p> <p>Our team will be responsible for maintaining this repo, which we hope will grow rapidly (though maybe not exactly to infinity). Feel free to <a href="https://github.com/HewlettPackard/hpe-notebooks">take a look</a>. Don’t hesitate to use these with your own teams. We welcome any feedback you receive regarding them so we can work to improve them for the future. Finally, please share some of your Jupyter Notebooks with us. It’s a great way to contribute to the community. After all, <a href="https://developer.hpe.com/community">HPE DEV</a> is all about <strong>sharing</strong>, <strong>communicating</strong> and <strong>collaborating</strong>.</p>https://developer.hpe.com/software-development-reset-a-remote-devops-epiphany/https://developer.hpe.com/software-development-reset-a-remote-devops-epiphany/Wed, 26 Aug 2020 19:55:02 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/e1-1598484223320.jpg" alt="e1"></p> <p>No aspect of society seems able to escape being altered by the tragic outbreak of COVID-19. The now well-known slogan “Stay-at-Home” has given rise to a future where working from home has become the <a href="https://www.gartner.com/en/newsroom/press-releases/2020-04-03-gartner-cfo-surey-reveals-74-percent-of-organizations-to-shift-some-employees-to-remote-work-permanently2">norm rather than the exception</a>. Even the IT sector has felt its impact. New ways of working and collaborating are being quickly deployed and implemented. Many organizations are learning they need to re-engineer business and IT processes and how they are consumed in order to adapt.</p> <p>Within many software development environments, software design, creation and testing is done on-premises; behind the corporate firewall, with operations and development working within a physical office environment. As organisations move towards adopting DevOps work practices, these principles of DevOps are being applied within the four walls of an office environment. Now, with the new reality of working from home, <strong>remote</strong> DevOps practices need to be looked at to reduce the development cycle through continuous integration and delivery.</p> <h2>The pandemic’s influence on DevOps</h2> <p>I believe, as a consequence of this dreadful crisis, the need to produce more web-enabled products and services at speed will become a priority for many more organisations. Hence, embracing DevOps processes, strategies, and a cultural ethos in the remote working environment can ultimately safeguard companies going forward by harmonizing remote agile collaboration and automation tools and practices throughout its development environment.</p> <p>This end-to-end process can bring together users, developers, and IT operations through the application of continuous integration, delivery and testing processes within automation pipelines. A well-defined structure for remote work, paired with good remote tools, can ensure that development teams’ productivity remains high, speed is embraced, and the quality of code is maintained through static and dynamic code analysis tools.</p> <h2>What remote working means for developers</h2> <p>Essentially, working from home, whether you are a developer or not, means that you do not need to travel to a traditional office space and you can do your work at home. How you do it depends on your preference and your organisational culture. Some organisations may insist on some sort of oversight and may employ some sort of tracking software, so they can make sure the developer is “punching in and out” at corporate times. Working from home requires developers to have some personal traits to guarantee effectiveness, such as:</p> <ul> <li><strong>Initiative</strong> - Most developers I have come across don’t enjoy constant supervision, hence those that can start and work through a project on their own fare well remotely.</li> <li><strong>Discipline</strong> - This is often a key failing in</li> </ul> <p>developers, where coding is never complete as there is always a better way of doing it, improving it, etc. The willpower to resist this inclination and keep your eyes on the ball is the key to working remotely.</p> <ul> <li><strong>A dedicated space (not your bedroom)</strong> - Developers, like all remote workers, need a place to work in that is separate from social home activities. Such a place provides the mindset that, when you are there, you know you should be working and not procrastinating.</li> <li><strong>A willingness to improve communications skills</strong> - Continuous communications with colleagues, bosses, customers, etc. will ensure that developers become trusted partners despite not working in the same room together. This inevitably involves into having clear and constructive conversations with users that underpins seamless software delivery.</li> </ul> <h2>Remote DevOps and user stories</h2> <p>Most developers know <em>user stories</em> (epics) are used as the basis for estimating, planning, and understanding whether value is/was delivered to customers. This is a key component to an effective DevOps strategy. It breaks large tasks down into smaller chunks that deliver value independently to make the work easier to visualize and understand. The potential benefits are obvious, as meaningful segments of work can be accomplished simultaneously within an inherent customer focus.</p> <p>Remote virtual work should not decrease the efficiency of good user stories (even though a physical story card is not on constant display) since digital tickets are virtually available. For instance, use a tool like Miro<sup>©</sup> online, a collaborative whiteboard used to create virtual Post-it notes with the basic 3-C concept of card, conversation and confirmation – writing on the virtual Post-it <code>as a ‘user type’, I want / need ‘goal’ so that I can accomplish ‘justification/business value’</code> . (Of course, don’t forget the context of this Post-it.)</p> <p>The brevity will ensure that greater emphasis is put on the conversations between the various stakeholders. It’s only once the conversation is complete that formal confirmation happens and there is mutual agreement on what will be delivered and when.</p> <h2>Developer virtual myths</h2> <p>The new normal of virtual working may pressure some developers into a quarantine-style existence and personal traits will push individual developers differently. It is therefore important to dispel any remote working folklore myths and legends that some associate with virtual working, namely:</p> <ul> <li><strong>Working remotely means I am isolated and on my own.</strong> To some developers, this is actually bliss (i.e. the much renowned hacker in his bedroom), as it may allow them to focus without office distractions. However, an effective developer is not an island and hence needs to collaborate continuously with his peers, customers and partners. Developers do not necessarily have to physically see people to avoid the feeling of loneliness or isolation. There are a number of virtual spaces (like Miro or Skype for example) that provide that real collaborative feeling of catching-up with colleagues.</li> <li><strong>As a junior member of the development team, you can’t work alone.</strong> This is so untrue that it is unbelievable that some developers feel this way. This is not an issue for junior developers, but of the organisation that hired them, as the organisation does not have the correct processes in place to support the junior developers. This can include, for example, a lack of effective feedback, inadequate on-boarding, and senior developer mentoring, to name a few.</li> <li><strong>Developers lose or misunderstand company culture.</strong> Office culture evolves over time. Culture can include ethics, values, integrity, principles, and reliability. The key to remote culture assimilation is to focus on how the organization uses its remote channels to convey what is expected.</li> <li><strong>Developers cannot build personal relationships if they don’t meet.</strong> Millennials have come to terms with the nature of friendship over social media. This should be no different for developers when working remotely. Using virtual tools to communicate can assist developers in having more direct technical conversations online when needed, as a developer is able to produce a considered response to an issue and not respond just because someone is standing in front of you.</li> <li><strong>Developer collaboration is reduced through working remotely.</strong> For developer teams that are not accustomed to remote work, shifting to a virtual world just requires revisiting approaches to collaboration, communication, security, quality control, individual and team performance, work assignment, compliance monitoring, and governance.</li> </ul> <h2>Three steps to get to remote DevOps</h2> <p>I suggest that the following three basic remote DevOps principles need to be considered when approaching DevOps remotely:</p> <ol> <li><strong>Remote teamwork is essential.</strong></li> </ol> <p>Sitting down with a colleague at their desk is not possible. You will need to use remote communications tools to chat and collaborate. But, in this remote world, it is essential to explain the problem comprehensively with well thought-out descriptions and documentation. Eye contact and body language will not be the key to understanding – documentation will be. At last, developers will be forced to document their software.<br> Obviously, remote working provides better work-life balance in terms of the flexibility of work and when to work in a 24hr period. However, it will mean that someone working from home cannot be an island and needs to be integrated fully into the corporate ethos and standards through effective communications and contact. Additionally, corporate security policies need to be maintained end-2-end through effective security tools, encryption and policies. For example, through HPE’s GitHub repository, teams can securely share code safely with appropriate fine-grained access controls and effective two-factor authentication.</p> <ol start="2"> <li><strong>Corporate continuous integration within a remote working environment is mandatory.</strong></li> </ol> <p>Developers writing code from home, like the young hackers in their bedrooms, require secure access to the corporate environment so that they can spin up a container, run their code, and start testing. Within a CI pipeline, automated services will take that code, spin up a container and run all the needed tests and standards (formats). This pipeline will provide the results back to the users in seconds, provide a repository (like HPE’s GitHub) for all code, all in an automated way.<br> Effective CI/CD pipelines are outcomes of the DevOps world and, as such, are critical in the virtual work as well. There are many CI/CD tools (i.e. Jenkins, etc.) that can assist in this endeavour. This brings us to integration needs which are even more fundamental in the virtual world. To help, DevOps apps such as GitHub can be the best place for virtual working to share code with remote co-workers. However, there will also be a need to use cloud resources within the concept of a cloud software containers across clusters. This is where Kubernetes provides building block entities within PaaS services.</p> <ol start="3"> <li><strong>Corporate continuous deployment within the remote working environment is also mandatory.</strong></li> </ol> <p>This is so code can be automatically merged safely into the master production environment by wrapping the code in a bundle, creating a new container, and deploying it. It also ensures that IT production and system administration staff (who are also working remotely) can instantly view it and support it.<br> Today’s new normal still requires continuous software delivery, which is demanded by the ever increasing virtual world of apps. Gone are the days when software deployment was at the behest of long-winded and time-consuming IT processes. Users demand new app releases in real-time. Virtual DevOps requires the tools, automation, and resources to deploy at will. This is where hybrid “as-a-Service” services are essential to continuous deployment, as they provide this ability. Fundamental to these three steps for remote DevOps is the need to have an effective cloud working experience that developers can trust and use/consume. This cloud service must be designed with DevOps in mind in terms of the required performance for a development environment; in other words, a true Platform-as –a-Service (PaaS) provision.</p> <p>What I mean by this is that it must go over and above the basic well-known marketing terminology of cloud services, i.e. elasticity (which we need to provision the right cloud services on request), agility (for rapid testing and experimentation environments), self-provisioning (for dynamic service utilization within an auto-scaling environment) and security (for appropriate access, firewalls, VPNs, Secure Endpoints, etc.).</p> <p>Through such a cloud service, DevOps developers will get the level of redundancy needed to continue their work in today’s environment. In addition, their work will be automatically backed up and they will encounter minimal process interruptions.</p> <h2>Conclusion</h2> <p>This is just a start as to how to deploy a remote DevOps working world. You could go even further and automate many of the administration jobs by setting up measurements and metrics that will auto-manage the space and performance requirements of the containers, servers, etc., which, in turn, could lead to a NoOps nirvana.</p> <p>As DevOps comes of age, it will undoubtedly be affected by this frightful crisis that makes a remote environment a mandatory condition. We have always known DevOps adoption would take time, and time has a way of introducing change. It is always been said that transitioning to DevOps is, first and foremost, a cultural shift, and then a process and organisational shift. The need for remote working has highlighted the need to work differently – with open collaboration, trust, working to an output not a time, reduction in the human touch, and many other things which have always been part of the DevOps Culture.</p> <p>Get started with DevOps and your transformation today by visiting our <a href="https://www.hpe.com/us/en/services.html">HPE Pointnext website</a>. And be sure to come back and check out the <a href="/blog">HPE DEV blog</a> for more articles like this.</p>https://developer.hpe.com/tutorial-enabling-remote-copy-using-the-hpe-csi-driver-for-kubernetes-on/https://developer.hpe.com/tutorial-enabling-remote-copy-using-the-hpe-csi-driver-for-kubernetes-on/Wed, 26 Aug 2020 02:02:34 GMT<p>The addition of new features to the HPE CSI Driver for Kubernetes never stops and, with the newly released 1.3.0 version of the CSI Driver, comes the much requested support for HPE Primera and 3PAR Remote Copy Peer Persistence. Remote Copy support within Kubernetes provides enhanced availability and transparent failover for disaster recovery protection with Kubernetes. As more and more applications migrate into Kubernetes, HPE recommends customers deploy mission-critical applications with replicated persistent volumes to ensure that these applications are highly available and resistant to failure. HPE Primera and 3PAR Remote Copy can serve as the foundation for a disaster recovery solution.</p> <h1>Configuring Remote Copy in the HPE CSI Driver</h1> <p>In the example I show here, I will start with an existing single zone Kubernetes cluster. For the most up-to-date information and examples on HPE Storage and containers, please refer to <a href="https://scod.hpedev.io">HPE Storage Container Orchestrator Documentation</a> (SCOD). Currently, the HPE CSI Driver for Kubernetes only supports HPE Primera and 3PAR Remote Copy in 2DC Peer Persistence mode. Remote Copy Periodic (async) mode is not currently supported but will be available in a future release.</p> <p>For information on creating a Peer Persistence configuration, review the <a href="https://techhub.hpe.com/eginfolib/storage/docs/Primera/RemoteCopy/RCconfig/GUID-1F726F48-A372-4ED8-B1D7-9545D091AE98.html#GUID-1F726F48-A372-4ED8-B1D7-9545D091AE98">HPE Primera Peer Persistence Host OS Support Matrix</a> for the supported host OSs and host persona requirements. Refer to <a href="https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&#x26;docId=emr_na-a00088914en_us">HPE Primera OS: Configuring data replication using Remote Copy over IP</a> for more information.</p> <h2>Requirements:</h2> <ul> <li>Single zone Kubernetes cluster</li> <li>Deployment of HPE CSI Driver for Kubernetes</li> <li>Create Secrets for Primary and Target arrays</li> <li>Create CustomResourceDefinition (CRD) for Peer Persistence</li> <li>Create StorageClass for replicated volumes</li> </ul> <h2>Deploy the HPE CSI Driver for Kubernetes</h2> <p>I will start by installing the latest version of the HPE CSI Driver for Kubernetes, which as of this writing is version 1.3.0. Here are two methods you can use do this:</p> <h4>Fresh installation of the HPE CSI Driver for Kubernetes</h4> <pre><code class="language-markdown">helm repo add hpe https://hpe-storage.github.io/co-deployments/ helm repo update helm install hpe-csi hpe/hpe-csi-driver --namespace hpe-csi --version 1.3.0 </code></pre> <h4>Upgrading an existing deployment of the HPE CSI Driver for Kubernetes</h4> <pre><code class="language-markdown">helm repo update helm search repo hpe-csi-driver -l helm upgrade hpe-csi hpe/hpe-csi-driver --namespace &#x3C;namespace> --version 1.3.0 </code></pre> <p>I can check the status of the deployment by running the following command.</p> <pre><code class="language-markdown">kubectl get all -n hpe-csi </code></pre> <p>If I used a different namespace during the deployment, I can use this command instead.</p> <pre><code class="language-markdown">kubectl get pods --all-namespaces -l 'app in (nimble-csp, primera3par-csp, hpe-csi-node, hpe-csi-controller)' </code></pre> <p>You should see something like this:</p> <pre><code class="language-markdown">NAMESPACE NAME READY STATUS RESTARTS AGE hpe-csi hpe-csi-controller-6d9bb97cd-njnj9 7/7 Running 0 1m hpe-csi hpe-csi-node-dlcz5 2/2 Running 0 1m ... hpe-csi nimble-csp-745cb4d948-6449z 1/1 Running 0 1m hpe-csi primera3par-csp-867984bf86-dkf2d 1/1 Running 0 1m </code></pre> <h2>Create Remote Copy link Secrets</h2> <p>Here's a tip for creating Kubernetes objects at the command line.</p> <pre><code class="language-markdown">kubectl create -f- &#x3C; paste the YAML > ^D (CTRL + D) </code></pre> <p>With the HPE CSI Driver deployed, you will need to create 2 secrets. One for each Primera array (i.e. default-primera-secret and secondary-primera-secret) that are part of the Remote Copy links.</p> <h4>Primary Array</h4> <pre><code class="language-markdown">apiVersion: v1 kind: Secret metadata: name: default-primera-secret namespace: hpe-csi stringData: serviceName: primera3par-csp-svc servicePort: "8080" backend: 10.0.0.2 username: 3paradm password: 3pardata </code></pre> <h4>Secondary Array</h4> <pre><code class="language-markdown">apiVersion: v1 kind: Secret metadata: name: secondary-primera-secret namespace: hpe-csi stringData: serviceName: primera3par-csp-svc servicePort: "8080" backend: 10.0.0.3 username: 3paradm password: 3pardata </code></pre> <h2>Create Peer Persistence CustomResourceDefinition</h2> <p>Next, you will need to create a <code>CustomResourceDefinition</code> that holds the target array information that will be used when creating the volume pairs.</p> <pre><code class="language-markdown">apiVersion: storage.hpe.com/v1 kind: HPEReplicationDeviceInfo metadata: name: replication-crd spec: target_array_details: - targetCpg: SSD_r6 targetName: primera-c670 targetSecret: secondary-primera-secret #targetSnapCpg: SSD_r6 (optional) targetSecretNamespace: hpe-csi </code></pre> <h2>Create Peer Persistence StorageClass</h2> <p>Next, define the <strong>remoteCopyGroup: &#x3C;remote_copy_group_name></strong> and the <strong>replicationDevices: &#x3C;replication_crd_name></strong> parameters. The HPE CSI Driver can use an existing Remote Copy Group or, if it doesn't exist, it will create a new one. The CSI Driver will also use the information from the <code>CRD</code> to create the replicated volume on the target array.</p> <pre><code class="language-markdown">apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "false" name: rep-sc provisioner: csi.hpe.com reclaimPolicy: Delete allowVolumeExpansion: true parameters: csi.storage.k8s.io/fstype: xfs csi.storage.k8s.io/controller-expand-secret-name: default-primera-secret csi.storage.k8s.io/controller-expand-secret-namespace: hpe-csi csi.storage.k8s.io/controller-publish-secret-name: default-primera-secret csi.storage.k8s.io/controller-publish-secret-namespace: hpe-csi csi.storage.k8s.io/node-publish-secret-name: default-primera-secret csi.storage.k8s.io/node-publish-secret-namespace: hpe-csi csi.storage.k8s.io/node-stage-secret-name: default-primera-secret csi.storage.k8s.io/node-stage-secret-namespace: hpe-csi csi.storage.k8s.io/provisioner-secret-name: default-primera-secret csi.storage.k8s.io/provisioner-secret-namespace: hpe-csi description: "Volume created using Peer Persistence with the HPE CSI Driver for Kubernetes" accessProtocol: fc # Primera customizations cpg: SSD_r6 remoteCopyGroup: new-rcg replicationDevices: replication-crd provisioning_type: tpvv allowOverrides: description,provisioning_type,cpg,remoteCopyGroup </code></pre> <p>Once you have created the <code>StorageClass</code> within the cluster, you can request Persistent Volumes as normal.</p> <h2>Create Peer Persistence PVC</h2> <pre><code class="language-markdown">apiVersion: v1 kind: PersistentVolumeClaim metadata: name: replicated-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 200Gi storageClassName: rep-sc </code></pre> <p>As volumes are created using the Remote Copy, the replication between the <strong>default</strong> and <strong>secondary</strong> Primeras will be transparent to Kubernetes. There are <a href="https://youtu.be/Eet92dOra24">multiple videos</a> out on YouTube demonstrating how automatic transparent failover works with various workloads and within Kubernetes it is no different. In the case of an array failure, automatic transparent failover will manage the pathing between the Primary and Secondary arrays on the Worker nodes so the application IO is not interrupted.</p> <pre><code class="language-markdown">kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE replicated-pvc Bound pvc-ca03a916-a6fb-434c-bc00-6b8 200Gi RWO rep-sc 1m </code></pre> <p>Finally, run <strong>showrcopy</strong> on both Primeras to see the sync status of the Remote Copy Group.</p> <pre><code class="language-markdown">$ showrcopy Remote Copy System Information Status: Started, Normal Target Information Name ID Type Status Options Policy virt-primera-c670 4 IP ready - mirror_config Link Information Target Node Address Status Options virt-primera-c670 0:3:1 172.17.20.5 Up - virt-primera-c670 1:3:1 172.17.20.6 Up - receive 0:3:1 receive Up - receive 1:3:1 receive Up - Group Information Name Target Status Role Mode Options new-rcg virt-primera-c670 Started Primary Sync auto_failover,path_management LocalVV ID RemoteVV ID SyncStatus LastSyncTime pvc-ca03a916-a6fb-434c-bc00-6b8 168 pvc-ca03a916-a6fb-434c-bc00-6b8 83 Synced NA </code></pre> <p>This verifies the replication status of the volumes created within your Kubernetes cluster.</p> <p>By adding Remote Copy support within the HPE CSI Driver along with the ability to perform volume snapshots and cloning capabilities, HPE gives many options within your Disaster Recovery strategy and gives you the peace of mind that your mission-critical application data is protected.</p> <h1>Next Steps</h1> <p>Stay tuned to the <a href="/blog">HPE DEV blog</a> for future posts regarding the HPE CSI Driver for Kubernetes. In the meantime, check out the blog about the new <a href="/blog/8nlLVWP1RKFROlvZJDo9/introducing-kubernetes-csi-sidecar-containers-from-hpe">Volume Mutator capabilities of the HPE CSI Driver</a>. Also, if you want to learn more about Kubernetes, CSI, and the integration with HPE storage products, you can find a ton of resources out on <a href="https://scod.hpedev.io">SCOD</a>! If you are already on Slack or an HPE employee, connect with us on Slack. If you are a new user, signup at <a href="https://slack.hpedev.io">slack.hpedev.io</a>. We hang out in #kubernetes, #nimblestorage and #3par-primera.</p>https://developer.hpe.com/introducing-kubernetes-csi-sidecar-containers-from-hpe/https://developer.hpe.com/introducing-kubernetes-csi-sidecar-containers-from-hpe/Tue, 25 Aug 2020 01:45:01 GMT<p>With the release of the upcoming HPE CSI Driver for Kubernetes version 1.3.0, Hewlett Packard Enterprise (HPE) introduces the concept of Container Storage Interface (CSI) extensions to the CSI driver using Kubernetes CSI sidecar containers. This concept is not foreign to anyone familiar with the CSI architecture as most new major features get implemented as a sidecar in a true microservice architecture. Services are tightly coupled and communicate over a UNIX socket using a high-speed Remote Procedure Call (RPC) interface, gRPC, for secure and reliable communication.</p> <p>The interface allows third parties to write extensions to their drivers to expose a particular storage platform’s differentiating feature where it’s difficult to conceive a broad stroke feature in a vendor neutral manner. It’s also possible to leapfrog SIG Storage (the Kubernetes working group for storage) for features currently in the discovery or design phase if customer demand is being prioritized over standardization.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/csi-130-slate-1598320004312.png" alt="picture1"></p> <p>The first (yes, there’s quite a few in the works) CSI sidecar is a volume mutator. It will allow end-users to alter their <code>PersistentVolumeClaims</code> (PVCs) during runtime, even while the <code>PersistentVolume</code> (PV) is mounted and serving a workload. What attributes are mutable depends on the backend Container Storage Provider (CSP) being used. Also, what attributes are allowed to be altered by an end-user is controlled by the Kubernetes cluster administrator through the <code>StorageClass</code>.</p> <p>Let’s go through an example on how you could put the volume mutator to work using the HPE Nimble Storage CSP.</p> <h1>Mutating persistent volume claims</h1> <p>With the CSI driver deployed and a HPE Nimble Storage backend configured, it’s good to understand what attributes are mutable. On the <a href="https://scod.hpedev.io/container_storage_provider/hpe_nimble_storage/index.html#common_parameters_for_provisioning_and_cloning">HPE Storage Container Orchestration Documentation</a> (SCOD) portal for the respective CSP, you'll find the supported parameters. For reference, this table represents the current mutable attributes.</p> <table> <thead> <tr> <th>Attribute</th> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>destroyOnDelete</td> <td>Boolean</td> <td>Used to control deletion of volume in the backend after PV removal</td> </tr> <tr> <td>description</td> <td>Text</td> <td>Volume description</td> </tr> <tr> <td>folder</td> <td>Text</td> <td>Place volume into an existing folder</td> </tr> <tr> <td>limitIops</td> <td>Integer</td> <td>Change IOPS limits on volume</td> </tr> <tr> <td>limitMbps</td> <td>Integer</td> <td>Change Throughput limits on volume</td> </tr> <tr> <td>performancePolicy</td> <td>Text</td> <td>Change performance policy for volume (within the same block size)</td> </tr> <tr> <td>dedupeEnabled</td> <td>Boolean</td> <td>Enable/Disable deduplication on volume</td> </tr> <tr> <td>thick</td> <td>Boolean</td> <td>Thick/thin provisioning of volume</td> </tr> <tr> <td>syncOnDetach</td> <td>Boolean</td> <td>Control that a snapshot of the volume should be synced to the replication partner each time it is detached from a node.</td> </tr> </tbody> </table> <p>For the purposes of this example, let’s assume we want to allow users to be in control of a few storage attributes. We will also allow them to override the parameters during creation of the <code>PVC</code>. <a href="https://scod.hpedev.io/csi_driver/using.html#using_pvc_overrides">Overriding parameters</a> during creation is a cornerstone feature that has been part of the HPE primary storage solution since the FlexVolume days.</p> <p>Create a default <code>StorageClass</code> with the <code>allowOverrides</code> and <code>allowMutations</code> set to allow certain performance tuning.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: hpe-standard provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: xfs csi.storage.k8s.io/controller-publish-secret-name: hpe-backend csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: hpe-backend csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: hpe-backend csi.storage.k8s.io/node-stage-secret-namespace: kube-system csi.storage.k8s.io/provisioner-secret-name: hpe-backend csi.storage.k8s.io/provisioner-secret-namespace: kube-system description: "Volume created by the HPE CSI Driver for Kubernetes" allowOverrides: description,limitIops,limitMbps,performancePolicy allowMutations: description,limitIops,limitMbps,performancePolicy </code></pre> <blockquote> <p><strong>Note:</strong> The volume mutator sidecar is dependent on the <code>"csi.storage.k8s.io/provisioner-secret-name"</code> and <code>"csi.storage.k8s.io/provisioner-secret-namespace"</code> to mutate volumes.</p> </blockquote> <p>Next, create a <code>PVC</code> with the following <code>.metadata.annotations</code>:</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-data annotations: csi.hpe.com/description: This is my volume description csi.hpe.com/limitIops: "10000" csi.hpe.com/limitMbps: "200" spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi </code></pre> <p>Switching over to the backend array, you can see that the volume was created with the desired overrides.</p> <pre><code class="language-markdown">Nimble OS $ vol --info pvc-2d1795ec-7bce-4af8-b841-437a435f29e1 | egrep -iw 'description|iops|throughput|performance' Description: This is my volume description Performance policy: default IOPS Limit: 10000 Throughput Limit (MiB/s): 200 </code></pre> <blockquote> <p><strong>Note:</strong> The volume name may be retrieved with <code>kubectl get pvc/my-data</code>.</p> </blockquote> <p>Let’s edit the object definition. This can be done with <code>kubectl edit</code> or you can create a YAML file and subsequently patch the <code>PVC</code>.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-data annotations: csi.hpe.com/description: Need more oomph! csi.hpe.com/performancePolicy: double-down csi.hpe.com/limitIops: "50000" csi.hpe.com/limitMbps: "1000" spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi </code></pre> <p>Patch the <code>PVC</code>.</p> <pre><code class="language-bash">kubectl patch pvc/my-data --patch "$(cat my-data-boost.yaml)" </code></pre> <p>Back on the array, you can see that the attributes have changed.</p> <pre><code class="language-markdown">Nimble OS $ vol --info pvc-2d1795ec-7bce-4af8-b841-437a435f29e1 | egrep -iw 'description|iops|throughput|performance' Description: Need more oomph! Performance policy: double-down IOPS Limit: 50000 Throughput Limit (MiB/s): 1000 </code></pre> <p>Since the <code>.spec.csi.volumeAttributes</code> of the <code>PV</code> that the backend volume was created with are immutable, the latest successful changes are annotated on the <code>PV</code>.</p> <pre><code class="language-yaml">apiVersion: v1 kind: PersistentVolume metadata: annotations: csi.hpe.com/description: Need more oomph! csi.hpe.com/limitIops: "50000" csi.hpe.com/limitMbps: "1000" csi.hpe.com/performancePolicy: double-down pv.kubernetes.io/provisioned-by: csi.hpe.com ... </code></pre> <p>Further adjustments may be performed anytime at any stage of the lifecycle of the <code>PV</code>.</p> <h1>Use cases</h1> <p>Given the gamut of options for the HPE Nimble Storage CSP, there are a number of creative ways to accelerate certain use cases that require runtime tuning of storage characteristics.</p> <h2>Performance management</h2> <p>Like in the example above, throttling volumes to adhere to a certain performance characteristic is by far the most prolific use case, especially if there's cost associated with the performance limits. The use case can be further extended by allowing users to move volumes between folders on the Nimble array, such as Gold, Silver and Bronze, all with different performance caps. Certain restrictions apply. See the <a href="https://scod.hpedev.io/container_storage_provider/hpe_nimble_storage/index.html#common_parameters_for_provisioning_and_cloning">documentation</a> for more information.</p> <h2>Data reduction changes</h2> <p>Using compression and deduplication may be desirable for the initial ingest of a dataset. Just note that future churn might cause issues on the workload requirement and data reduction capabilities may be toggled at will. The need might arise during runtime to prioritize space reserve. Toggling thin-provisioning with the <code>thick</code> parameter may be used to control the reservations.</p> <h2>Data migration control</h2> <p>In the event where you need to perform a workload transition between clusters, it’s practical to apply <code>destroyOnDelete: "false"</code> and <code>syncOnDetach: "true"</code> on the backend volume. This is to ensure the replica destination gets updated with the latest data from the source when destaging the workload. Also, retaining the volume on the array when the Kubernetes objects are being cleaned out from the source namespace is neccesary in the event of the replica destination is being configured to reverse the replication after the transition.</p> <p>It will be exciting to see what other use cases will surface from the installed base with this new capability!</p> <h1>Next steps</h1> <p>The HPE CSI Driver for Kubernetes version 1.3.0 will become available in the next few weeks. <code>StorageClasses</code> may then be created with the <code>allowMutations</code> parameter and the CSI volume mutator may be used without any further tweaks.</p> <ul> <li><a href="https://scod.hpedev.io/csi_driver/using.html#using_volume_mutations">Using volume mutations</a> on SCOD</li> <li>Overview of the <a href="https://scod.hpedev.io/csi_driver/index.html">HPE CSI Driver for Kubernetes</a></li> <li>Source code <a href="https://github.com/hpe-storage/csi-driver">available on GitHub</a></li> <li>Check out the HPE primary storage platform pages: <a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">HPE Nimble Storage</a> and <a href="https://developer.hpe.com/platform/hpe-3par-and-primera/home">HPE Primera</a></li> </ul> <p>Watch the <a href="https://hpedev.io">HPE Developer Community</a> for future exciting updates to the HPE CSI Driver for Kubernetes!</p>https://developer.hpe.com/hpe-firmware-updates-part-1-file-types-and-smart-components/https://developer.hpe.com/hpe-firmware-updates-part-1-file-types-and-smart-components/Fri, 21 Aug 2020 15:07:31 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/firmware_updates/part1/firmware_update_part1">Server Management portal</a>.</big></p> <br>https://developer.hpe.com/hpe-firmware-updates-part-2-interaction-in-operating-modes/https://developer.hpe.com/hpe-firmware-updates-part-2-interaction-in-operating-modes/Thu, 20 Aug 2020 11:38:44 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/firmware_updates/part2/firmware_update_part2">Server Management Portal</a></big></p> <br>https://developer.hpe.com/hpe-dev-hack-shack-attracts-over-10k-visitors-at-hpe-discover-event/https://developer.hpe.com/hpe-dev-hack-shack-attracts-over-10k-visitors-at-hpe-discover-event/Wed, 19 Aug 2020 07:42:44 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/hack-shack-image1-1597850961298.png" alt="hack shack image1"></p> <p>March 2020 marked a turning point for physical tradeshows and events. Faced with travel and gathering restrictions imposed by the pandemic, many companies scrambled to find new ways to connect with their customers. Virtual events quickly rose to the forefront as a preferred venue. But how to make a virtual event as engaging, interactive, and valuable as a physical event had yet to be determined. For Hewlett Packard Enterprise (HPE), this was an important consideration as it planned its premier customer technology event, HPE Discover.</p> <p>Since the inception of the Discover events, HPE senior executives, technologists, partners, and IT thought leaders found that it was a great way to connect with customers. One of the highlights on the show floor in recent years has been the HPE DEV Hack Shack. There, developers would come and chat with other developers, participate in hands-on workshops, and take coding challenges; all in a relaxed, fun-filled environment.</p> <p>As the venue changed from being a physical event, the immediate challenge for the HPE DEV team was how to replicate this experience virtually. The Dev team rose to the challenge and successfully attracted over 10,000 visitors during the HPE Discover 2020 Virtual Event. Here’s how they did it.</p> <h2>Bringing the Hack Shack online</h2> <p>While most of the HPE Discover Virtual Experience was hosted on a virtual event platform from Intrado Digital Media, the <a href="/hackshack/">Hack Shack</a> was linked to as an off-platform site designed by HPE developers, which used <a href="https://developer.hpe.com/platform/grommet/home">Grommet</a> for its underlying web application structure. From the HPE Discover Virtual Event Intrado platform, attendees would click on a link to “enter” the Hack Shack lobby. In the lobby, a quick video tour explained how to navigate the site, pointing out all the different types of activities that were offered during the event.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/hack-shack-image2-1597850953889.png" alt="hack shack image2"></p> <h2>Workshops</h2> <p>As one of the major attractions of a physical Hack Shack is the opportunity for attendees to meet up with other developers and subject matter experts (SMEs) in hands-on workshops to learn and transfer knowledge, the virtual Hack Shack needed a way to accomplish something similar. To do this, Dev team members created a set of live workshops where students could interact with the presenter online using Jupyter Notebooks. SMEs were on hand to help facilitate the workshops, answer questions, and ensure no attendee got left behind during the session. This approach proved to be very successful, judging by the high scores given by participants in the post-workshop surveys.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/hack-shack-image3-1597850947424.png" alt="hack shack image3"></p> <h2>Coding Challenges</h2> <p>Another popular physical Hack Shack feature the team wanted to incorporate into the virtual experience was its coding challenges. These challenges are timed events where developers are asked to create their own code to achieve a defined function while taking specific parameters into account. They challenge an attendee’s understanding of concepts delivered through the virtual workshops and offer an opportunity to apply what was learned into a real world example. The team is pleased to report that three challenge winners were selected and some cool prizes were awarded.</p> <h2>Fun and games</h2> <p>The Hack Shack is also well-known for its fun Hack Shack Attack! retro arcade style video game. Given the fact that the game is already an app, it was fairly easy to integrate it into the new, online Hack Shack Arcade. This attracted a lot of attendees who competed to achieve the highest score each week, with the weekly winner being awarded a prize along with some serious bragging rights! Stickers and dev art were also offered in the arcade for attendees to download and use for wallpaper or social posts.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/hack-shack-image4-1597853418103.png" alt="hack shack image4"></p> <h2>A virtual gathering place</h2> <p>The Hack Shack has always been a place for the community to gather. As one might imagine, this is not easily done virtually. Live workshops provided one excellent touch point. The Hack Shack Community page provided another, offering attendees social connection opportunities, like Slack and Twitter, to use to connect with peers. It also highlighted the HPE DEV Newsletter, which not only delivers the latest news directly to subscribers' email box, but also offers an opportunity for community members to submit their own articles to be placed on the HPE DEV blog site. This proved interesting to many, as 146 new members signed up over the course of the event. The Learn On-Demand videos also provided another touch point.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/hack-shack-image5-1597850929851.png" alt="hack shack image5"></p> <p>The HPE DEV team worked long and hard to bring this experience online where it can be shared with a much broader audience. We are now participating at KubeCon + CloudNativeCon EU and are planning to offer the <a href="/hackshack/">Hack Shack</a> at other future virtual events. It’s no surprise that virtual events will continue on into the foreseeable future. As a matter of fact, on June 9, 2020 <a href="https://builtin.com/marketing/virtual-event-engagement-strategies">Brian Nordli posted a blog article</a> on the increasing popularity of virtual events. In it, he pointed out that many people were already planning virtual events for 2021 and 2022. He emphasized why this was by quoting Ben Chodor, president of the virtual event platform company Intrado Digital Media, who remarked <em>“Even when everything gets back to normal, there’s going to be a lot of people who aren’t going to feel comfortable going to Vegas or Orlando with 10,000 other people.”</em></p> <p>If you happen to be at any of these virtual events, look up the <a href="/hackshack/">HPE DEV Hack Shack</a>. If you need hints on how to create something similar for your event, check in with one of our HPE DEV team members on our <a href="https://slack.hpedev.io/">Slack channel</a>. There’s a good chance they’ll be able to offer some excellent advice.</p>https://developer.hpe.com/datasets-dataframes-and-spark-sql-for-processing-of-tabular-data/https://developer.hpe.com/datasets-dataframes-and-spark-sql-for-processing-of-tabular-data/Wed, 19 Aug 2020 06:24:20 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-10-24T07:00:00.000Z", "tags": "spark" </code></pre> <hr> <p>With Apache Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster. The Spark SQL and the Dataset/DataFrame APIs provide ease of use, space efficiency, and performance gains with Spark SQL's optimized execution engine. In this blog post we will give an introduction to Spark Datasets, DataFrames and Spark SQL. This is part 2 of a multi-blog series. You can read <a href="/blog/4jqBP6MO3rc1Yy0QjMOq/spark-101-what-is-it-what-it-does-and-why-it-matters">part 1 here</a>.</p> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p>A Spark Dataset is a distributed collection of typed objects, which are partitioned across multiple nodes in a cluster and can be operated on in parallel. Datasets can be created from MapR XD files, MapR Database tables, or MapR Event Store topics, and can be cached, allowing reuse across parallel operations. A Dataset can be manipulated using functional transformations (map, flatMap, filter, etc.) and/or Spark SQL. A DataFrame is a Dataset of Row objects and represents a table of data with rows and columns. A DataFrame consists of partitions, each of which is a range of rows in cache on a data node.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image12-1597819064184.png" alt=""></p> <h2>The SparkSession Object</h2> <p>As discussed before, a Spark application runs as independent processes, coordinated by the SparkSession object in the driver program. The entry point to programming in Spark is the org.apache.spark.sql.SparkSession class, which you use to create a SparkSession object as shown below:</p> <pre><code class="language-scala">val spark = SparkSession.builder().appName("example").master("local[*]".getOrCreate() </code></pre> <p>If you are using the spark-shell or a notebook, the SparkSession object is already created and available as the variable Spark.</p> <h2>Interactive Analysis with the Spark Shell</h2> <p>The Spark shell provides an easy way to learn Spark interactively. You can start the shell with the following command:</p> <pre><code class="language-bash">$ /[installation path]/bin/spark-shell --master local[2] </code></pre> <p>You can enter the code from the rest of this chapter into the Spark shell. Outputs from the shell are prefaced with ---result---.</p> <h2>Exploring U.S. Flight Data with Spark Datasets and DataFrames</h2> <p>To go over some core concepts of Spark Datasets, we will be using some flight information from the <a href="https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&#x26;DB_Short_Name=On-Time">United States Department of Transportation</a>. Later, we will use this same data to predict flight delays, so we want to explore the flight attributes that contribute most to flight delays. Using Spark Datasets, we will explore the data to answer questions like: which airline carriers, days of the week, originating airport, and hours of the day have the highest number of flight delays, when a delay is greater than 40 minutes?</p> <p>The flight data is in JSON files, with each flight having the following information:</p> <ul> <li><strong>id</strong>: ID composed of carrier, date, origin, destination, flight number</li> <li><strong>dofW</strong>: day of week (1=Monday, 7=Sunday)        </li> <li><strong>carrier</strong>: carrier code        </li> <li><strong>origin</strong>: origin airport code        </li> <li><strong>dest</strong>: destination airport code</li> <li><strong>crsdephour</strong>: scheduled departure hour        </li> <li><strong>crsdeptime</strong>: scheduled departure time        </li> <li><strong>depdelay</strong>: departure delay in minutes        </li> <li><strong>crsarrtime</strong>: scheduled arrival time        </li> <li><strong>arrdelay</strong>: arrival delay minutes        </li> <li><strong>crselapsedtime</strong>: elapsed time        </li> <li><strong>dist</strong>: distance        </li> </ul> <p>It appears in the following format:</p> <pre><code class="language-markdown">{     "_id": "AA_2017-01-01_ATL_LGA_1678", "dofW": 7, "carrier": "AA", "origin": "ATL", "dest": "LGA", "crsdephour": 17, "crsdeptime": 1700, "depdelay": 0.0, "crsarrtime": 1912, "arrdelay": 0.0, "crselapsedtime": 132.0, "dist": 762.0 } </code></pre> <p>(<a href="https://github.com/mapr-demos/mapr-spark2-ebook">The complete data and code for all examples are available here.</a>)</p> <h2>Loading Data from a File into a Dataset</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image10-1597819073355.png" alt=""></p> <p>With the SparkSession <code>read</code> method, we can read data from a file into a DataFrame, specifying the file type, file path, and input options for the schema. The schema can optionally be inferred from the contents of the JSON file, but you will get better performance and accuracy by specifying the schema.</p> <pre><code class="language-scala">import org.apache.spark.sql.types._ import org.apache.spark.sql._ import org.apache.spark.sql.functions._ val schema = StructType(Array(     StructField("_id", StringType, true),     StructField("dofW", IntegerType, true),     StructField("carrier", StringType, true),     StructField("origin", StringType, true),     StructField("dest", StringType, true),     StructField("crsdephour", IntegerType, true),     StructField("crsdeptime", DoubleType, true),     StructField("crsarrtime", DoubleType, true),     StructField("crselapsedtime", DoubleType, true),     StructField("label", DoubleType, true),     StructField("pred_dtree", DoubleType, true)   )) var file = "maprfs:///data/flights.json" val df = spark.read.format("json").option("inferSchema", "false").schema(schema).load(file) ---result:--- df: org.apache.spark.sql.DataFrame = [_id: string, dofW: int ... 10 more fields] </code></pre> <p>The take method returns an array with objects from this Dataset, which we see is of type Row.</p> <pre><code class="language-markdown">df.take(1) ---result:--- Array[org.apache.spark.sql.Row] = Array([ATL_LGA_2017-01-01_17_AA_1678, 7, AA, ATL, LGA, 17, 1700.0, 0.0, 1912.0, 0.0, 132.0, 762.0]) </code></pre> <p>If we supply a case class with the <code>as</code> method when loading the data, then the data is read into a Dataset of typed objects corresponding to the case class.</p> <pre><code class="language-scala">case class Flight(_id: String, dofW: Integer, carrier: String, origin: String, dest: String, crsdephour: Integer, crsdeptime: Double, depdelay: Double,crsarrtime: Double, arrdelay: Double, crselapsedtime: Double, dist: Double) extends Serializable val df = spark.read.format("json").option("inferSchema", "false").schema(schema).load(file).as[Flight] ---result---: df: org.apache.spark.sql.Dataset[Flight] = [_id: string, dofW: int ... 10 more fields] </code></pre> <p>Now the take method returns an array of Flight objects.</p> <pre><code class="language-scala">df.take(1) ---result:--- Array[Flight] = Array(Flight(ATL_LGA_2017-01-01_17_AA_1678, 7,AA,ATL,LGA,17,1700.0,0.0,1912.0,0.0,132.0,762.0)) </code></pre> <h2>Transformations and Actions</h2> <p>There are two types of operations you can perform on a Dataset:</p> <ul> <li><strong>transformations</strong>: create a new Dataset from the current Dataset</li> <li><strong>actions</strong>: trigger computation and return a result to the driver program</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image1-1597819082131.png" alt=""></p> <p>Transformations are lazily evaluated, which means they are not computed immediately. A transformation is executed only when it is triggered by an action. Once an action has run and the value is returned, the Dataset is no longer in memory, unless you call the <code>cache</code> method on the Dataset. If you will reuse a Dataset for more than one action, you should cache it.</p> <p><strong>Datasets and Type Safety</strong></p> <p>Datasets are composed of typed objects, which means that transformation syntax errors (like a typo in the method name) and analysis errors (like an incorrect input variable type) can be caught at compile time. DataFrames are composed of untyped Row objects, which means that only syntax errors can be caught at compile time. Spark SQL is composed of a string, which means that syntax errors and analysis errors are only caught at runtime. Datasets save a developer’s time by catching errors sooner, even while typing when using an IDE.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image2-1597819089388.png" alt=""></p> <p><strong>Dataset Transformations</strong></p> <p>Here is a list of some commonly used typed transformations, which can be used on Datasets of typed objects (Dataset[T]).</p> <p>| Transformation | Description | | --- | --- | --- | | map | Returns new Dataset with result of applying input function to each element | | filter | Returns new Dataset containing elements where input function is true | | groupByKey | Returns a KeyValueGroupedDataset where the data is grouped by the given key function |</p> <p>This example filter transformation on the flight Dataset returns a Dataset with flights that departed at 10 AM. The take action returns an array of flight objects to the driver program.</p> <pre><code class="language-scala">df.filter(flight => flight.crsdephour == 10).take(3) ---result:--- Array[Flight] = Array(Flight(ORD_DEN_2017-01-01_AA_2300, 7,AA,ORD,DEN,10,1005.0,5.0,1145.0,3.0,160.0,888.0), Flight(MIA_ORD_2017-01-01_AA_2439,7,AA,MIA,ORD,10, 1005.0,4.0,1231.0,0.0,206.0,1197.0)) </code></pre> <p><strong>DataFrame Transformations</strong></p> <p>Here is a list of some commonly used untyped transformations, which can be used on Dataframes (Dataset[Row]).</p> <p>| Transformation | Description | | --- | --- | --- | | select | Selects a set of columns | | join | Join with another DataFrame, using the given join expression | | groupBy | Groups the DataFrame, using the specified columns |</p> <p>This <code>groupBy</code> transformation example groups the flight Dataset by carrier, then the count action counts the number of flights for each carrier. The show action prints out the resulting DataFrame rows in tabular format.</p> <pre><code class="language-scala">df.groupBy("carrier").count().show() ---result:--- +-------+-----+ |carrier|count| +-------+-----+ |     UA|18873| |     AA|10031| |     DL|10055| |     WN| 2389| +-------+-----+ </code></pre> <p>Here is a list of some commonly used Dataset actions.</p> <p>| Action | Description | | --- | --- | --- | | show(n) | Displays the first n rows in a tabular form | | take(n) | Returns the first n objects in the Dataset in an array | | count | Returns the number of rows in the Dataset |</p> <p>Here is an example using typed and untyped transformations and actions to get the destinations with the highest number of departure delays, where a delay is greater than 40 minutes. We count the departure delays greater than 40 minutes by destination and sort them with the highest first.</p> <pre><code class="language-scala">df.filter($"depdelay" > 40).groupBy("dest").count() .orderBy(desc("count")).show(3) ---result:--- +----+-----+ |dest|count| +----+-----+ | SFO|  711| | EWR|  620| | ORD|  593| +----+-----+ </code></pre> <h2>Exploring the Flight Dataset with Spark SQL</h2> <p>Now let’s explore the flight Dataset using Spark SQL and DataFrame transformations. After we register the DataFrame as a SQL temporary view, we can use SQL functions on the SparkSession to run SQL queries, which will return the results as a DataFrame. We cache the DataFrame, since we will reuse it and because Spark can cache DataFrames or Tables in columnar format in memory, which can improve memory usage and performance.</p> <pre><code class="language-scala">// cache DataFrame in columnar format in memory df.cache // create Table view of DataFrame for Spark SQL df.createOrReplaceTempView("flights") // cache flights table in columnar format in memory spark.catalog.cacheTable("flights") </code></pre> <p>Below, we display information for the top five longest departure delays with Spark SQL and with DataFrame transformations (where a delay is considered greater than 40 minutes):</p> <pre><code class="language-scala">// Spark SQL spark.sql("select carrier,origin, dest, depdelay,crsdephour, dist, dofW from flights where depdelay > 40 order by depdelay desc limit 5").show // same query using DataFrame transformations df.select($"carrier",$"origin",$"dest",$"depdelay", $"crsdephour").filter($"depdelay" > 40).orderBy(desc( "depdelay" )).show(5) ---result:--- +-------+------+----+--------+----------+ |carrier|origin|dest|depdelay|crsdephour| +-------+------+----+--------+----------+ |     AA|   SFO| ORD|  1440.0|         8| |     DL|   BOS| ATL|  1185.0|        17| |     UA|   DEN| EWR|  1138.0|        12| |     DL|   ORD| ATL|  1087.0|        19| |     UA|   MIA| EWR|  1072.0|        20| +-------+------+----+--------+----------+ </code></pre> <p>Below, we display the average departure delay by carrier:</p> <pre><code class="language-scala">// DataFrame transformations df.groupBy("carrier").agg(avg("depdelay")).show ---result:--- +-------+------------------+ |carrier|     avg(depdelay)| +-------+------------------+ |     UA|17.477878450696764| |     AA| 10.45768118831622| |     DL|15.316061660865241| |     WN|13.491000418585182| +-------+------------------+ </code></pre> <p>With a notebook like Zeppelin or Jupyter, you can also display the SQL results in graph formats.</p> <pre><code class="language-scala">// Spark SQL %sql select carrier, avg(depdelay)  from flights  group by carrier </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1597819097878.png" alt=""></p> <p>Let’s explore this data for flight delays, when the departure delay is greater than 40 minutes. Below, we see that United Airlines and Delta have the highest count of flight delays for January and February 2017 (the training set).</p> <pre><code class="language-scala">// __Count of Departure Delays by Carrier (where delay=40 minutes)__ df.filter($"depdelay" > 40) .groupBy("carrier").count.orderBy(desc( "count")).show(5) ---result:--- +-------+-----+ |carrier|count| +-------+-----+ |     UA| 2420| |     DL| 1043| |     AA|  757| |     WN|  244| +-------+-----+ </code></pre> <pre><code class="language-sql">// Count of Departure Delays by Carrier (where delay=40 minutes) %sql select carrier, count(depdelay) from flights where depdelay > 40 group by carrier </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image11-1597819106731.png" alt=""></p> <p>In the query below, we see that Monday (1), Tuesday (2), and Sunday (7) have the highest count of flight delays.</p> <pre><code class="language-sql">// Count of Departure Delays by Day of the Week %sql select dofW, count(depdelay) from flights where depdelay > 40 group by dofW </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1597819122799.png" alt=""></p> <p>In the query below, we see that the hours between 13:00-19:00 have the highest count of flight delays.</p> <pre><code class="language-sql">%sql select crsdephour, count(depdelay) from flights where depdelay > 40 group by crsdephour order by crsdephour </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1597819140727.png" alt=""></p> <p>In the query below, we see that the originating airports, Chicago and Atlanta, have the highest count of flight delays.</p> <pre><code class="language-sql">%sql select origin, count(depdelay) from flights where depdelay > 40 group by origin ORDER BY count(depdelay) desc </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1597819149739.png" alt=""></p> <p>In the query below, we see the count of departure delays by origin and destination. The routes ORD->SFO and DEN->SFO have the highest delays, maybe because of weather in January and February. Adding weather to this Dataset would give better results.</p> <pre><code class="language-sql">%sql select origin, dest, count(depdelay) from flights where depdelay > 40 group by origin, dest ORDER BY count(depdelay) desc </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1597819157738.png" alt=""></p> <h2>Summary</h2> <p>You have now learned how to load data into Spark Datasets and DataFrames and how to explore tabular data with Spark SQL. These code examples can be reused as the foundation to solve many types of business problems.</p>https://developer.hpe.com/resource-allocation-configuration-for-spark-on-yarn/https://developer.hpe.com/resource-allocation-configuration-for-spark-on-yarn/Wed, 19 Aug 2020 06:14:55 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Hao Zhu", "publish": "2015-09-11T07:00:00.000Z", "tags": "spark" </code></pre> <hr> <p>In this blog post, I will explain the resource allocation configurations for Spark on YARN, describe the yarn-client and yarn-cluster modes, and include examples.</p> <p>Spark can request two resources in YARN; CPU and memory. Note that Spark configurations for resource allocation are set in spark-defaults.conf, with a name like spark.xx.xx. Some of them have a corresponding flag for client tools such as spark-submit/spark-shell/pyspark, with a name like --xx-xx. If the configuration has a corresponding flag for client tools, you need to put the flag after the configurations in parenthesis"()". For example:</p> <pre><code class="language-scala">spark.driver.cores (--driver-cores) </code></pre> <h2>1. yarn-client vs. yarn-cluster mode</h2> <p>There are two deploy modes that can be used to launch Spark applications on YARN per <a target='\_blank' href='https://spark.apache.org/docs/latest/running-on-yarn.html'>Spark documentation</a>:</p> <ul> <li>In <strong>yarn-client</strong> mode, the driver runs in the client process and the application master is only used for requesting resources from YARN.</li> <li>In <strong>yarn-cluster</strong> mode, the Spark driver runs inside an application master process that is managed by YARN on the cluster, and the client can go away after initiating the application.</li> </ul> <h2>2. Application Master (AM)</h2> <p><strong>a. yarn-client</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/reallocation-blog-img1-1597817848006.png" alt=""></p> <p>Let’s look at the settings below as an example:</p> <pre><code class="language-scala">[root@h1 conf]# cat spark-defaults.conf |grep am **spark.yarn.am.cores 4 spark.yarn.am.memory 777m** </code></pre> <p>By default, <code>spark.yarn.am.memoryOverhead</code> is AM memory * 0.07, with a minimum of 384. This means that if we set spark.yarn.am.memory to 777M, the actual AM container size would be 2G. This is because 777+Max(384, 777 * 0.07) = 777+384 = 1161, and the default yarn.scheduler.minimum-allocation-mb=1024, so 2GB container will be allocated to AM. As a result, a (2G, 4 Cores) AM container with Java heap size -Xmx777M is allocated:</p> <p>Assigned container container_1432752481069_0129_01_000001 of capacity</p> <pre><code class="language-scala">&#x3C;memory:2048, vCores:4, disks:0.0> </code></pre> <p><strong>b. yarn-cluster</strong></p> <p>In yarn-cluster mode, the Spark driver is inside the YARN AM. The driver-related configurations listed below also control the resource allocation for AM.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/reallocation-blog-img2-1597817860836.png" alt=""></p> <p>Take a look at the settings below as an example:</p> <pre><code class="language-scala">MASTER=yarn-cluster /opt/mapr/spark/spark-1.3.1/bin/spark-submit --class org.apache.spark.examples.SparkPi \ --driver-memory 1665m \ --driver-cores 2 \ /opt/mapr/spark/spark-1.3.1/lib/spark-examples*.jar 1000 </code></pre> <p>Since 1665+Max(384,1665*0.07)=1665+384=2049 > 2048(2G), a 3G container will be allocated to AM. As a result, a (3G, 2 Cores) AM container with Java heap size -Xmx1665M is allocated:<br> Assigned container container_1432752481069_0135_02_000001 of capacity</p> <pre><code class="language-scala">&#x3C;**memory:3072, vCores:2**, disks:0.0> </code></pre> <h2>3. Containers for Spark executors</h2> <p>For Spark executor resources, yarn-client and yarn-cluster modes use the same configurations:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/reallocation-blog-img3-1597817872761.png" alt=""></p> <p>In <code>spark-defaults.conf</code>, <code>spark.executor.memory</code> is set to 2g.</p> <p>Spark will start 2 (3G, 1 core) executor containers with Java heap size -Xmx2048M: Assigned container container_1432752481069_0140_01_000002 of capacity</p> <pre><code class="language-scala">&#x3C;**memory:3072, vCores:1**, disks:0.0> </code></pre> <p>Assigned container container_1432752481069_0140_01_000003 of capacity</p> <pre><code class="language-scala">&#x3C;**memory:3072, vCores:1**, disks:0.0> </code></pre> <p>However, one core per executor means only one task can be running at any time for one executor. In the case of a broadcast join, the memory can be shared by multiple running tasks in the same executor if we increase the number of cores per executor.</p> <p>Note that if <a target='\_blank' href='https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation'>dynamic resource allocation</a> is enabled by setting <code>spark.dynamicAllocation.enabled</code> to true, Spark can scale the number of executors registered with this application up and down based on the workload. In this case, you do not need to specify <code>spark.executor.instances</code> manually.</p> <h2>Key takeaways:</h2> <ul> <li>Spark driver resource related configurations also control the YARN application master resource in yarn-cluster mode.</li> <li>Be aware of the max (7%, 384m) overhead off-heap memory when calculating the memory for executors.</li> <li>The number of CPU cores per executor controls the number of concurrent tasks per executor.</li> </ul> <p>In this blog post, you’ve learned about resource allocation configurations for Spark on YARN. If you have any further questions, please <a href="https://slack.hpedev.io/">reach out to us via Slack</a>.</p> <p>Make sure you check the <a href="/blog">HPE DEV blog</a> regularly to view more articles on this subject.</p>https://developer.hpe.com/how-to-log-in-apache-spark/https://developer.hpe.com/how-to-log-in-apache-spark/Wed, 19 Aug 2020 06:01:02 GMT<p><strong>Editor’s Note:</strong> MapR products and solutions sold prior to the acquisition of such assets by Hewlett Packard Enterprise Company in 2019, may have older product names and model numbers that differ from current solutions. For information about current offerings, which are now part of HPE Ezmeral Data Fabric, please visit <a href="https://www.hpe.com/us/en/software/data-fabric.html">https://www.hpe.com/us/en/software/data-fabric.html</a></p> <h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Nicolas A Perez", "publish": "2016-03-01T08:00:00.000Z", "tags": "spark" </code></pre> <hr> <p>An important part of any application is the underlying log system we incorporate into it. Logs are not only for debugging and traceability, but also for business intelligence. Building a robust logging system within our apps can provide significant insights into the business problems we are trying to solve.</p> <h2>Log4j in Apache Spark</h2> <p>Spark uses <code>log4j</code> as the standard library for its own logging. Everything that happens inside Spark gets logged to the shell console and to the configured underlying storage. Spark also provides a template for app writers so we could use the same <code>log4j</code> libraries to add whatever <code>messages</code> we want to the existing login method for Spark.</p> <h2>Configuring Log4j</h2> <p>Under the <code>SPARK_HOME/conf</code> folder, there is <code>log4j.properties.template</code> file which serves as an starting point for our own <code>logging</code> system.</p> <p>Based on this file, we created the <code>log4j.properties</code> file and put it under the same directory.</p> <p><code>log4j.properties</code> looks like follows:</p> <pre><code class="language-markdown">log4j.appender.myConsoleAppender=org.apache.log4j.ConsoleAppender log4j.appender.myConsoleAppender.layout=org.apache.log4j.PatternLayout log4j.appender.myConsoleAppender.layout.ConversionPattern=%d [%t] %-5p %c - %m%n log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.RollingAppender.File=/var/log/spark.log log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender log4j.appender.RollingAppenderU.File=/var/log/sparkU.log log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - %m%n </code></pre> <p><strong>By default, everything goes to console and file</strong></p> <pre><code class="language-markdown">log4j.rootLogger=INFO, RollingAppender, myConsoleAppender </code></pre> <p><strong>My custom logging goes to another file</strong></p> <pre><code class="language-markdown">log4j.logger.myLogger=INFO, RollingAppenderU </code></pre> <p><strong>The noisier spark logs go to file only</strong></p> <pre><code class="language-markdown">log4j.logger.spark.storage=INFO, RollingAppender log4j.additivity.spark.storage=false log4j.logger.spark.scheduler=INFO, RollingAppender log4j.additivity.spark.scheduler=false log4j.logger.spark.CacheTracker=INFO, RollingAppender log4j.additivity.spark.CacheTracker=false log4j.logger.spark.CacheTrackerActor=INFO, RollingAppender log4j.additivity.spark.CacheTrackerActor=false log4j.logger.spark.MapOutputTrackerActor=INFO, RollingAppender log4j.additivity.spark.MapOutputTrackerActor=false log4j.logger.spark.MapOutputTracker=INFO, RollingAppender log4j.additivty.spark.MapOutputTracker=false </code></pre> <p>Basically, we want to hide all logs Spark generates so we don’t have to deal with them in the shell. We redirect them to be logged in the file system. On the other hand, we want our own logs to be logged in the shell and in a separate file so they don’t get mixed up with the ones from Spark. From here, we will point Splunk to the files where our own logs are which in this particular case is <code>/var/log/sparkU.log.</code></p> <p>This (<code>log4j.properties</code>) file is picked up by Spark when the application starts so we don’t have to do anything aside of placing it in the designated location.</p> <h2>Writing Our Own Logs</h2> <p>Now that we have configured the components that Spark requires in order to manage our logs, we just need to start writing logs within our apps.</p> <p>In order to show how this is done, let’s write a small app that helps us in the demonstration.</p> <p>Our app:</p> <pre><code class="language-scala">object app {  def main(args: Array[String]) {    val log = LogManager.getRootLogger    log.setLevel(Level.WARN)    val conf = new SparkConf().setAppName("demo-app")    val sc = new SparkContext(conf)    log.warn("Hello demo")    val data = sc.parallelize(1 to 100000)    log.warn("I am done")  } } </code></pre> <p>Running this Spark app will demonstrate that our log system works. We will be able to see <code>Hello demo</code> and <code>I am done</code> messages being logged in the shell and in the file system, while the Spark logs will only go to the file system.</p> <p>So far, everything seems fine, yet there is a problem we haven’t mentioned.</p> <p>The class <code>org.apache.log4j.Logger</code> is not <code>serializable</code>, which implies we cannot use it inside a <code>closure</code> while doing operations on some parts of the Spark API.</p> <p>For example, if we do the following in our app:</p> <pre><code class="language-scala">val log = LogManager.getRootLogger val data = sc.parallelize(1 to 100000) data.map { value =>     log.info(value)    value.toString } </code></pre> <p>This will fail when running on Spark. Spark complains that the <code>log</code> object is not <code>Serializable</code>, so it cannot be sent over the network to the Spark workers.</p> <p>This problem is actually easy to solve. Let’s create a class that does something to our data set while doing a lot of logging.</p> <pre><code class="language-scala">class Mapper(n: Int) extends Serializable{  @transient lazy val log = org.apache.log4j.LogManager.getLogger("myLogger")  def doSomeMappingOnDataSetAndLogIt(rdd: RDD[Int]): RDD[String] =    rdd.map{ i =>      log.warn("mapping: " + i)      (i + n).toString    } } </code></pre> <p><code>Mapper</code> receives a <code>RDD[Int]</code> and returns a <code>RDD[String]</code> and it also logs what value is being mapped. In this case, note how the <code>log</code> object has been marked as <code>@transient</code>, which allows the serialization system to ignore the <code>log</code> object. Now, <code>Mapper</code> is being serialized and sent to each worker but the log object is being resolved when it is needed in the worker, solving our problem.</p> <p>Another solution is to wrap the <code>log</code> object into a <code>object</code> construct and use it all over the place. We would rather have <code>log</code> within the class we are going to use it, but the alternative is also valid.</p> <p>At this point, our entire app looks like the following:</p> <pre><code class="language-scala">import org.apache.log4j.{Level, LogManager, PropertyConfigurator} import org.apache.spark.` import org.apache.spark.rdd.RDD class Mapper(n: Int) extends Serializable{  @transient lazy val log = org.apache.log4j.LogManager.getLogger("myLogger")  def doSomeMappingOnDataSetAndLogIt(rdd: RDD[Int]): RDD[String] =    rdd.map{ i =>      log.warn("mapping: " + i)      (i + n).toString    } } object Mapper {  def apply(n: Int): Mapper = new Mapper(n) } object app {  def main(args: Array[String]) {    val log = LogManager.getRootLogger    log.setLevel(Level.WARN)    val conf = new SparkConf().setAppName("demo-app")    val sc = new SparkContext(conf)    log.warn("Hello demo")    val data = sc.parallelize(1 to 100000)    val mapper = Mapper(1)    val other = mapper.doSomeMappingOnDataSetAndLogIt(data)    other.collect()    log.warn("I am done")  } } </code></pre> <h2>Conclusions</h2> <p>Our logs are now being shown in the shell and also stored in their own files. Spark logs are being hidden from the shell and being logged into their own file. We also solved the serialization problem that appears when trying to log in different workers.</p> <p>We now can build more robust BI systems based on our own Spark logs as we do with other non-distributed systems and applications we have today. Having the right insights is an important aspect of Business Intelligence, and this can help you achieve that.</p> <p>This post was originally published <a target='\_blank' href='https://medium.com/@anicolaspp/how-to-log-in-apache-spark-f4204fad78a#.xo31z5vrd'>here</a>.</p>https://developer.hpe.com/building-dynamic-machine-learning-pipelines-with-kubedirector/https://developer.hpe.com/building-dynamic-machine-learning-pipelines-with-kubedirector/Fri, 14 Aug 2020 10:58:54 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Imagine that you’re a data scientist who’s been asked to create an application or service that can predict travel time for a proposed taxi ride. The application needs to have the ability to update the service with new data as it becomes available, so its predictions take recent patterns into account. In this tutorial, we’ll show you how to set up a Machine Learning (ML) pipeline using KubeDirector to train, register, and query your model.</p> <p><a href="/blog/running-non-cloud-native-apps-on-kubernetes-with-kubedirector">KubeDirector</a> was introduced to the open source community to address stateful application deployment in standard Kubernetes clusters. In the latest release (<a href="https://github.com/bluek8s/kubedirector/releases/tag/v0.5.0">version 0.5</a>), KubeDirector now allows multiple clusters to share data very easily using a new feature called <strong><em>Connections</em></strong>. The new feature helps users create large-scale dynamic, stateful containerized applications such as are found in Machine Learning (ML) pipelines and allow them to constantly evolve as models are improved and the data changes.</p> <p>A basic ML pipeline consists of three stages:</p> <ul> <li>Training, where the input datasets are processed to create ML models</li> <li>Model registration, where the models to be used are identified and characterized</li> <li>Inferencing, where selected models are made available for answering queries</li> </ul> <p>After a brief description of the KubeDirector architecture used for this solution, we’ll show you how to train, register, and finally query your model for answers. The sample dataset used, and available <a href="https://s3.amazonaws.com/nyc-tlc/">here under trip data</a>, contains taxi ride data that will be used to infer total time per ride for given taxi pickup and drop-off locations. And yes, this is pre-2020 data. Things look much different now due to the pandemic. That’s why it’s important that your ML model is flexible and can take real-time data into account.</p> <p>Our solution uses three KubeDirector Applications (kdapps): A training deployment kdapp named <strong><em>training-engine</em></strong>, a Jupyter Notebook kdapp named <strong><em>jupyter-notebook</em></strong>, and an inferencing deployment kdapp named <strong><em>deployment-engine</em></strong>.</p> <p>A kdapp is a custom resource (CR) that can be created in any Kubernetes cluster. The kdapp instructs KubeDirector on how a particular kind of virtual application cluster should be deployed and managed. The three kdapps used in this solution are examples that can be found online in the <a href="https://github.com/bluek8s/kubedirector/tree/master/deploy/example_catalog">KubeDirector github example catalog</a>.</p> <h3>Register Your KubeDirector Applications with Kubernetes</h3> <p>Assuming that KubeDirector is already deployed and running in the Kubernetes cluster, these kdapp CRs can be created to register apps with KubeDirector, e.g. by using kubectl:</p> <pre><code class="language-markdown">kubectl create –f cr-app-training-engine.json kubectl create –f cr-app-jupyter-notebook.json kubectl create –f cr-app-deployment-engine.json </code></pre> <p>This is a one-time step that builds a catalog of applications that data scientists can now instantiate as needed.</p> <p>Once a kdapp is created, an instance of that app can be deployed by creating a KubeDirector virtual cluster (kdcluster) CR. A kdcluster identifies the desired kdapp and specifies runtime configuration parameters, such as the size and resource requirements of the virtual cluster.</p> <h3>KDApp Connections</h3> <p>For the purposes of this discussion, one of the most interesting parts of the kdcluster spec is the Connections stanza, which identifies other resources of interest to that kdcluster. This is detailed in the GitHub readme link <a href="https://github.com/bluedatainc/solutions/tree/master/MLOps/examples/KDApp%20connections">here</a>.</p> <p>The KubeDirector GitHub repo includes <a href="https://github.com/bluek8s/kubedirector/tree/master/deploy/example_clusters">examples of kdcluster CRs</a> that instantiate these kdapps. This tutorial will show you how to create kdclusters much like those examples, except that yours will add information about the Connections that each kdcluster will use.</p> <p>The input data and trained ML models for this example pipeline will live in a <strong><em>project repository</em></strong> of shared persistent storage. The kdapps used will expect to be able to access that repository through directories mounted within the app containers. There are multiple ways to implement this arrangement, but here you’ll use the <strong><em>FS Mounts</em></strong> feature of the <a href="https://www.hpe.com/info/container-platform">HPE Ezmeral Container Platform</a> to access our pre-integrated <a href="https://www.hpe.com/info/data-fabric">HPE Ezmeral Data Fabric</a> (formerly known as MapR Data Platform) for persistent container storage. The final piece of the puzzle will be the use of <em>ConfigMap</em> resources to describe and locate the ML models used for drawing inferences.</p> <p>These components come together to build the pipeline like so:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/mlops-figure-1-1597402607009.png" alt="mlops figure 1"></p> <p>A project repository is used to store the key data components needed in any ML pipeline, such as source data, the model itself and the scoring script.</p> <p>In this example, the project repository is maintained within the HPE Ezmeral Data Fabric, since KubeDirector was itself deployed using the HPE Ezmeral Container Platform (which includes our data fabric by default); KubeDirector is a key component of the HPE Ezmeral Container Platform. However, KubeDirector is an open-source project and it can be used with any Kubernetes cluster and corresponding project repository.</p> <h2>Training</h2> <p>Once you have registered the ML training kdapp and the Jupyter Notebook kdapp for Model Building, you’ll need to launch instances of those applications as KubeDirector clusters (kdclusters) to put the ML pipeline to work. First, launch an instance of the ML training kdapp:</p> <pre><code class="language-markdown">kubectl create –f cr-cluster-training-engine.yaml </code></pre> <h3>KubeDirector Connection creation: Jupyter Notebook kdcluster -> Training engine kdcluster</h3> <p>Next you’ll need to launch an instance of your Jupyter Notebook kdcluster. Before you do that, you need to modify the example kdcluster yaml file and include a new Connection stanza into the top level “spec” section. This modification would look similar to the following code:</p> <pre><code class="language-yaml">spec: app: “jupyter-notebook” appCatalog: “local” connections: clusters: - “training-engine-instance” </code></pre> <p>Now, use the modified yaml file to launch an instance of your Jupyter Notebook kdcluster:</p> <pre><code class="language-markdown">kubectl create –f cr-cluster-jupyter-notebook.yaml </code></pre> <p>The runtime configuration of the connected training-engine kdcluster, including its service endpoints, is therefore injected by KubeDirector into the jupyter-notebook app. This info is used by the notebook app when KubeDirector triggers configuration scripts to run within the app containers.</p> <blockquote> <p>Note that these Connections can also be edited in the kdcluster spec later, for example if the notebook should be redirected to work with a different training engine.</p> </blockquote> <h3>Training the model</h3> <p>Now that the notebook kdcluster is running, a user accesses the Jupyter Notebook web UI via the network service port. Then, the usual process of directing the connected training deployment to generate models from the input data can be performed.</p> <h3>Using a little training magic</h3> <p>Those familiar with Jupyter Notebooks know that they include support for predefined <em>magic</em> functions that can be called within a notebook using a command line style syntax. These magic functions expand the capabilities of the notebook adding support for things like: <em>%history</em>, <em>%edit</em>, <em>%rerun</em>, <em>%recall</em>, <em>%macro</em>, <em>%save</em>, and <em>%pastebin</em>.</p> <p>Jupyter Notebook also includes hooks that allow defining custom magics. The kdapps in our solution utilize these custom commands (magics) to seamlessly integrate with KubeDirector Connections. This includes custom magics to handle remotely submitting training code and retrieving results and logs. These magic functions make REST API calls to the API server that runs as part of a training environment.</p> <p>For example the following magic functions are included:</p> <ul> <li><em>%attachments</em>: Returns a list of connected training environments</li> <li><em>%logs --url</em>: URL of the training server load balancer</li> <li><em>%%&#x3C;&#x3C;training_cluster_name>></em>: Submits training code to the training environment</li> </ul> <p>The below screenshot from the <a href="https://github.com/bluedatainc/solutions/blob/master/MLOps/examples/NYCTaxi/TensorFlow/TensorflowPipelineFullTaxiDataSet.ipynb">GitHub site</a> shows you an example of how a Jupyter Notebook can use these magic commands:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/mlops-figure-2-1597402615159.png" alt="mlops figure 2"></p> <p>Note at the top of this screenshot of the Jupyter Notebook that the <em>%attachments</em> magic command is used to retrieve the Training Cluster name - <em>traningengineinstance</em>. Then, that name is used in the following magic command - <em>%%trainingengineinstance</em>. This single command then submits training code to the training environment – truly magic!</p> <p>As the training engine generates models, it will store the model data into the project repository. From the training engine’s point of view, it is simply writing to a designated subdirectory of its filesystem.</p> <p>Below is a snapshot from the HPE Ezmeral Container Platform Web User Interface. This illustrates the use of the integrated HPE Ezmeral Data Fabric. Specifically, the FsMounts feature is used here for organizing the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/mlops-figure-3-1597402623656.png" alt="mlops figure 3"></p> <h2>Model registration</h2> <p>Next, you’ll need to create a <em>ConfigMap</em> resource to store metadata about the model to be used in deployments. Here's an example ConfigMap for the model data generated above:</p> <pre><code class="language-yaml">apiVersion: v1 kind: ConfigMap metadata: name: tensorflowmodel data: name: tensorflowmodel description: "example model" model-version: "1" path: /bd-fs-mnt/TenantShare/models/10yrdatasetchecknames/0_tf scoring-path: /bd-fs-mnt/TenantShare/code/TF_Scoring.py </code></pre> <p>The two crucial parameters that will be used by the deployment-engine kdcluster are the <strong><em>path</em></strong> and the <strong><em>scoring-path</em></strong>, each of which identifies a location within the project repository as seen within an app container. The path is the location of the serialized model data. This path can be retrieved from the job log when the model was trained. The scoring-path locates a script that will be used by the deployment engine to deserialize and process the model.</p> <h2>Create inference deployment</h2> <p>This is what you’ve been waiting for! How long will my taxi ride take? You’ve trained a model from a data set, and now it’s time to extract value from your model. To do this, you need to create a deployment-engine.</p> <p>A deployment-engine kdcluster is used to stand up services that will allow clients to draw ride-time inferences from the models you have created and registered.</p> <p>A deployment engine will serve inferences for one or more models. To specify models to a deployment-engine kdcluster, the Connections feature once again comes in handy. Instead of naming other kdclusters of interest, this time it will be used to name ConfigMap resources, one for each model that the deployment should use. In this case, just use the one example model that was given.</p> <h3>KubeDirector Connection creation: Inference server kdcluster -> Model ConfigMap</h3> <p>For this deployment, the example <strong><em>cr-cluster-endpoint-wrapper.yaml</em></strong> file can be used. Similar to how the Jupyter Notebook kdcluster yaml file was modified, this kdcluster yaml file will be edited to include the Connection stanza. A new property in the top-level spec section is added, similar to the following:</p> <pre><code class="language-yaml"> spec: app: deployment-engine appCatalog: “local” connections: configmaps: - “tensorflowmodel” </code></pre> <p>When these ConfigMap Connections are named in a kdcluster spec, KubeDirector will inject the contents of those ConfigMaps into a JSON file within the kdcluster's app containers where they can be used by the deployment app. If the contents of the ConfigMaps change, KubeDirector will immediately update this file. The deployment app can therefore reference this file to find the models it should use, which it can then load from the FS Mount directories exposed within its containers.</p> <p>This key use of KubeDirector Applications, Clusters and Connections is what makes an ML pipeline created with KubeDirector very dynamic – it’s able to easily accept constantly updating and changing data sets and models of that data. The <a href="https://www.nbcnewyork.com/news/local/nyc-taxis-for-hire-cars-took-sharp-hit-with-pandemic-and-are-only-slowly-coming-back/2541796/">taxi ride data for New York City in 2020</a> will likely look very different than the sample pre-pandemic dataset. ML pipelines need to be constantly monitored and often retrained to accommodate this model drift, as the ML model lifecycle is a highly iterative and dynamic process.</p> <p>A similar process could be implemented using ConfigMaps mounted into application pods using native Kubernetes mechanisms. An important benefit of the Connections feature, however, is that the connected set of ConfigMaps can grow, shrink, and change while the pod remains running. In terms of the use case here, that means you could add another model to be handled by the deployment engine without interrupting any current requests that the engine is processing.</p> <h3>Serving Queries</h3> <p>The “haproxy” service port on the inference deployment can now be used to service REST API queries, using the model created earlier to make inferences about how long a proposed taxi ride will take. An example script for making queries to this service can be found <a href="https://github.com/bluedatainc/solutions/blob/master/MLOps/examples/NYCTaxi/TensorFlow/query_api_script_tf.py">here</a>.</p> <p>Here’s that script in action, sending a query to the inferencing deployment previously created:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/mlops-figure-4-1597402633083.png" alt="mlops figure 4"></p> <p>From this output, our example script queries the model via REST API calls, providing a few parameters, and the model returns a prediction.</p> <h2>Conclusion</h2> <p>While we’ve presented the steps from input dataset to query result sequentially here, in “steady state” operation all of these operations can be happening in different order and/or at the same time.</p> <p>Building our pipeline using Kubernetes resources and the open source KubeDirector project lets us deal flexibly with a wide range of configurations and changes, with minimal disruption. The possibilities include:</p> <ul> <li>Multiple notebooks per training deployment, with each notebook’s unique password given to a group or a single user for finer-grained access control.</li> <li>Multiple training deployments to implement different training methods or access different datasets.</li> <li>Changing existing notebooks to connect to different training deployments, without losing any of the working environment of the notebook.</li> <li>Multiple inferencing deployments, for access control or load-balancing.</li> <li>Multiple models served per inferencing deployment, at different service ports or different URLs.</li> <li>Changing the set of models served by an inferencing deployment, without interrupting its operation.</li> </ul> <p>Find out more about KubeDirector applications and KubeDirector clusters at the GitHub site: <a href="https://github.com/bluek8s/kubedirector/wiki">GitHub</a>. From there you can download open-source KubeDirector and use it in your own Kubernetes clusters!</p> <p>And to see how we use KubeDirector in our HPE Ezmeral Container Platform, check out the interactive demo environment <a href="http://www.hpe.com/engage/containerplatform">here</a>.</p> <p>Look for more KubeDirector posts coming soon on the <a href="/blog">HPE DEV blog site</a>.</p> <blockquote> <p>This article was co-authored by:</p> </blockquote> <ul> <li>Joel Baxter, Distinguished Technologist</li> <li>Kartik Mathur, Master Technologist</li> <li>Don Wake, Technical Marketing Engineer</li> </ul>https://developer.hpe.com/how-to-set-and-enforce-tagging-policies-in-hpe-greenlake-for-private-clo/https://developer.hpe.com/how-to-set-and-enforce-tagging-policies-in-hpe-greenlake-for-private-clo/Mon, 03 Aug 2020 13:15:13 GMT<p><strong>Editor’s Note – NAME CHANGE: HPE GreenLake for Private Cloud is now part of HPE GreenLake for Private Cloud Enterprise.</strong></p> <hr> <h2>Introduction</h2> <p>HPE GreenLake for private cloud allows customers to assign metadata to their instances in the form of tags. An instance is a set of virtual machines that compose a horizontally scalable entity or a service suite, like a database. Tags help customers manage, report and filter their instances, providing a business context to resource consumption and cost. The goal of this article is to discuss tags and the process of creating and enforcing tagging policies.</p> <h2>Tags</h2> <p>A tag is simply a label that consists of a customer-defined key and value that makes it easier to manage and filter instances. Tags allow you to categorize your instances in different ways. For example, you could define a set of tags for your instances that help you track each instances’ owner, department, and purpose.</p> <p>The following diagram illustrates how tagging works. In this example, I assigned two tags to each instance. One tag defines the key <strong>Owner</strong> and the other delineates the key <strong>Purpose</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b1-1596460810780.png" alt="b1"></p> <h2>Tag Policies</h2> <p>With HPE GreenLake for private cloud tag policies, you can enforce a tag with a specific key and value or you can enforce the value coming from a specific list of values when an instance is provisioned. The list of acceptable values you designate can be any Option List that already exists in HPE GreenLake for private cloud.</p> <p>Option Lists can be populated in a number of ways including manually within HPE GreenLake for private cloud or from a REST API. In this article, I will illustrate the process of creating Option Lists designated as type "Manual".</p> <blockquote> <p><strong>NOTE:</strong> Refer to <a href="https://cmpdocs.privatecloud.greenlake.hpe.com/en/latest/provisioning/library/library.html">HPE GreenLake for private cloud documentation</a> to obtain more information on Option Lists</p> </blockquote> <p>Other Tag Policy capabilities include:</p> <ul> <li>Multiple tag policies can be combined to enforce a comprehensive tag compliance program</li> <li>An Instance details page will warn when tags are not in compliance</li> <li>Administrators have the option to enable strict enforcement, stopping instance provisioning that would violate the tag policies</li> </ul> <h2>Creating an Option List of Type "Manual"</h2> <ul> <li>Navigate to Provisioning - Library - OPTIONS LISTS and select +ADD key.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b2-1596460824531.png" alt="b2"></p> <ul> <li>Enter a NAME for the Option List, select <strong>"MANUAL"</strong> type, provide DATASET and click on SAVE CHANGES.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b3-1596460832147.png" alt="b3"></p> <blockquote> <p><strong>NOTE:</strong> JSON entries must be formatted as shown in the following example.</p> </blockquote> <p><code>[{"name": "Purpose1","value": "Development"},{"name": "Purpose2","value": "Production"}]</code></p> <h2>Creating a Tag Policy</h2> <ul> <li>Navigate to Administration - Policies and select +ADD POLICY key.</li> <li>In the TYPE drop-down, select <strong>Tags</strong>. The new policy modal will show options pertaining specifically to tagging, as shown below:</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b4-1596460839745.png" alt="b4"></p> <ul> <li>Set the options as required. In this example, the following options are set: <ul> <li><strong>TYPE:</strong> Tags</li> <li><strong>NAME:</strong> A name given to this specific policy</li> <li><strong>ENABLED:</strong> When checked, the policy will be put into effect upon Save</li> <li><strong>STRICT ENFORCEMENT:</strong> When checked, new instances will not be provisioned if they violate the tag policy. When unchecked, users are only warned for non-compliance.</li> <li><strong>KEY:</strong> HPE GreenLake for private cloud requires a tag be added with the entered Key</li> <li><strong>VALUE LIST:</strong> If set, a tag with the key indicated above must have a value contained in the Option List selected</li> <li><strong>SCOPE:</strong> In this example case, it is set to Global, but tag policies can also be targeted to Groups, Clouds, Users and Roles</li> </ul> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b5-1596460846266.png" alt="b5"></p> <p>Once you have clicked "SAVE CHANGES", the policy is in effect so long as the "ENABLED" box is checked (which is true by default for new policies).</p> <h2>Checking the Tag Policy enforcement</h2> <p>With the Tag Policy currently in place, you can see it in action when trying to provision an instance. Since I am enforcing a tag with key "Purpose" to exist with STRICT ENFORCEMENT, the provisioning wizard will not allow me to progress past the configuration tab if the tag is not set as shown below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b6-1596460852753.png" alt="b6"></p> <p>If an Option List is associated with the Tag Policy, you will see a similar validation error if you enter a tag key with no value or with a value that is not in the Option List.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b7-1596460859332.png" alt="b7"></p> <p>If you return to the policy and deselect the "STRICT ENFORCEMENT" option, you will no longer be prevented from provisioning it, even when the tags violate the policy. You will, however see a message on the instance details page with information on which tag policy is being violated by the given instance, as shown below:<img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/b8-1596470828071.png" height="100" width="700" align="left"></p> <p>In a future blog entry, I’ll show how you can use tags to provide showback reporting and budgeting with business context in Consumption Analytics. Make sure you check the <a href="/blog">HPE DEV blog</a> site often to find future posts related to this topic. If you have questions, please feel free to connect with me on the <a href="https://slack.hpedev.io/">HPE DEV Slack channel</a>.</p>https://developer.hpe.com/2020-August-03/https://developer.hpe.com/2020-August-03/Mon, 03 Aug 2020 05:00:00 GMThttps://developer.hpe.com/the-hpe-dev-community-hack-shack-a-place-to-gather-learn-and-play/https://developer.hpe.com/the-hpe-dev-community-hack-shack-a-place-to-gather-learn-and-play/Fri, 31 Jul 2020 07:39:28 GMT<p>From August 17-20, the <a href="http://cncf.io">Cloud Native Computing Foundation (CNCF)</a> will welcome technologists from all over the world to virtually connect at their flagship event, <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/"> KubeCon | CloudNativeCon EU</a> . There, Platinum Sponsor Hewlett Packard Enterprise (HPE), will host the HPE DEV Community Hack Shack virtual experience. HPE will also highlight the new HPE Ezmeral software portfolio (which includes the HPE Ezmeral Container Platform) and showcase its strategic offerings in the areas of containers and Kubernetes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/kubblog2-1596183367896.png" alt="kubblog2"></p> <h2>Meet you in the Hack Shack!</h2> <p>The HPE DEV Community Hack Shack is a virtual experience designed to help developers, designers, and data scientists connect with the HPE DEV team and collaborate with them to build innovative solutions. It provides a place where you can engage with experts to learn about HPE and open source solutions and work with them to guide product development. At physical events, the Hack Shack was known for its unique, informal and fun atmosphere. Our online virtual experience aims to bring this same feeling to this virtual event!</p> <h2>Navigating the Hack Shack</h2> <p>Within the HPE DEV Hack Shack, you’ll find the following tabs where you’ll be able to collaborate, learn, interact with experts, and compete for fun prizes:</p> <ul> <li><a href="/hackshack/challenges">CHALLENGES:</a> Compete with others for prizes in a number of gaming and coding challenges.</li> <li><a href="/hackshack/arcade">ARCADE:</a> Access our well-known Hack Shack Attack! retro video game to compete for the highest score.</li> <li><a href="/hackshack/replays">REPLAYS:</a> Watch some of the in-depth Technical Workshops we’ve offered in the past. Topics include the HPE Ezmeral Container Platform, SPIFFE and SPIRE fundamentals, and the HPE Container Storage Interface for Kubernetes.</li> <li><a href="/hackshack/ezmeral">HPE EZMERAL:</a> Get detailed information regarding the HPE Ezmeral software portfolio.</li> <li><a href="/hackshack/community">COMMUNITY:</a> Connect with HPE experts and participate in on-demand training. While there, sign-up for our HPE DEV Newsletter to stay up-to-date on the newest blog posts and tutorials. As an incentive to become a part of the community, those who sign up for the <a href="https://developer.hpe.com/event/kubecon-europe-2020?listid=10605211">HPE DEV Newsletter</a> between August 17 and August 20, will be entered into a drawing to win a prize.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/kubblog3-1596183377034.png" alt="kubblog3"></p> <h2>Looking for more developer info?</h2> <p>Visit the <a href="https://developer.hpe.com">HPE DEV Community Portal</a> via the <a href="/hackshack">Hack Shack Lobby</a>. There, you can access a treasure trove of information, including:</p> <ul> <li><a href="https://developer.hpe.com/platforms">PLATFORMS</a> – Access GitHub resources and software development kits (SDKs)</li> <li><a href="/blog">BLOG</a> – Read blogs and tutorials</li> <li><a href="https://developer.hpe.com/projects">OPEN SOURCE</a> – Discover platforms, apps and contributions</li> <li><a href="https://developer.hpe.com/events">EVENTS</a> – Plan on attending upcoming events</li> <li><a href="https://developer.hpe.com/community">COMMUNITY</a> – Participate and contribute to the community</li> </ul> <p>At the <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon | CloudNativeCon EU</a> HPE virtual booth, you’ll get to explore HPE’s different software-enabling and application development technologies, including:</p> <ul> <li><a href="https://www.hpe.com/us/en/solutions/container-platform.html">The HPE Ezmeral Container Platform</a>, the industry’s first Kubernetes container platform software designed to run both cloud-native and non cloud-native applications, enabling true hybrid cloud operations across any location.</li> <li><a href="https://www.hpe.com/us/en/storage/intelligent-storage.html?chatsrc=ot-en&#x26;jumpid=ps_8r5mdg32xs_aid-520023673&#x26;gclid=Cj0KCQiAs67yBRC7ARIsAF49CdU6O6Hbaj1lwT8tcrU702BzRnZboWNQILTShb0cCk-eEk7nUjQ-yhMaAv4fEALw_wcB&#x26;gclsrc=aw.ds">The intelligent data platform</a> from <a href="https://www.hpe.com/us/en/storage.html">HPE storage</a>, focused on persistent storage use cases for Kubernetes in private, public, and hybrid clouds and how to enable CI/CD pipelines.</li> <li><a href="https://spiffe.io/">The SPIFFE and SPIRE Projects</a>. With the recent acquisition of Scytale, HPE is the leading contributor for CNCF’s SPIFFE and SPIRE projects. These projects help organizations build a foundation for zero trust through the use of platform agnostic, cryptographic service identity.</li> </ul> <h2>Sessions outside the booth at KubeCon | CloudNativeCon EU</h2> <p><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon | CloudNativeCon EU</a> will also offer two sessions highlighting HPE offerings:</p> <ul> <li><strong>Predictable Performance Through Prometheus and Performance Aware Scheduling</strong>, co-sponsored by HPE and Intel and delivered by Chief Technologist, Tom Golway (HPE), and Software Engineer, Killian Muldoon (Intel)</li> <li><strong>Taming Data/State Challenges for ML Applications and Kubeflow</strong>, presented by HPE Distinguished Technologist, Skyler Thomas</li> </ul> <p>Refer to the <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/schedule/">program schedule</a> for specific times and locations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/grommet-1596183385085.png" alt="grommet"></p> <p>Virtual events can be challenging. Many who attend physical conferences come specifically to engage personally with industry and subject matter experts. That’s why the <a href="https://developer.hpe.com/community">HPE DEV community</a> has worked hard to deliver its <a href="/hackshack">HPE DEV Community Hack Shack</a> virtual experience. We want to ensure you have a place where you can do just that. For questions or help, be sure to connect with us on our <a href="https://slack.hpedev.io/">HPE DEV Slack Channel</a>.</p>https://developer.hpe.com/master-the-automation-universe-the-easy-way-part-2-the-art-of-packing/https://developer.hpe.com/master-the-automation-universe-the-easy-way-part-2-the-art-of-packing/Tue, 28 Jul 2020 13:46:21 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/7/stackstorm-part2-1-1595944683785.png" alt="StackStorm integration pack"></p> <p>Recently, <a href="/blog/master-the-automation-universe-the-easy-way-part-1-introduction-to-stack">I introduced you to StackStorm</a>. In this post, I'll present StackStorm integration packs! StackStorm is one of the most innovative automation platforms I have had the opportunity of becoming familiar with. When you first start working with StackStorm, it can appear to be a bit overwhelming. But fear not. I will share with you all the secrets I have discovered in my unceasing googling of the interwebs. Naturally, I can't take you on a super deep dive in this post, so I will leave that to the documents page. But you can check out the full documentation over <a href="https://docs.stackstorm.com/packs.html">here</a>. Now, let's dive in!</p> <p>A StackStorm integration pack is a predefined set of actions, sensors, rules, Python or shell scripts, and other miscellaneous items. A StackStorm pack has a specific structure that looks like what's shown in the picture below. Nested under the actions directory is the workflow directory.</p> <pre><code># Contents of a pack actions/ # rules/ # sensors/ # aliases/ # policies/ # tests/ # etc/ # any additional things (e.g. training, scripts) config.schema.yaml # Configuration schema packname.yaml.example # example of config, used in CI pack.yaml # pack definition file requirements.txt # requirements for python packs requirements-tests.txt # requirements for python tests icon.png # 64x64 /png icon </code></pre> <p>The good news is, if you have some existing Python or shell scripts you are using for automating things, you will be able to recycle them with StackStorm. I will be working with Python scripts in this example.</p> <p>The first thing to do is pair the Python file <strong>myPython.py</strong> with a <strong>myPython.yaml</strong> file. When this is done, the resulting pair of files represents a StackStorm <strong>action</strong>. It can get a little confusing here. You will need a YAML file and a Python script in the <strong>action directory</strong> and both of them should have <strong>the same name</strong>. You can think of the YAML file as something that introduces the Python script.</p> <p><strong>Actions</strong></p> <p>Since the best place to start a journey is at the beginning, let me show you how easy it is to start packing by taking a look at the action YAML file.</p> <pre><code class="language-yaml">--- name: get_switches pack: hpecfm description: Get an array of switches from the hpecfm controller runner_type: python-script entry_point: get_switches.py enabled: true </code></pre> <p>As you can see, defining the action is fairly straight forward. <strong>YAML</strong> files start with the action name and the name of the pack where the action is located. The obligatory description is always recommended. The runner type is a Python script but there are many others you can use as well (e.g. shell scripts, webhooks). Our entry point will be a Python script called <strong>get_switches</strong>.</p> <p>Here's a closer look at the get_switches Python script:</p> <pre><code class="language-python"># A python script for getting a dictionary of switches from pyhpecfm import fabric from lib.actions import HpecfmBaseAction class switchLookup(HpecfmBaseAction): def run(self): # Get switches from hpecfm controller. switches = fabric.get_switches(self.client) if isinstance(switches, list): # Setup a list for holding dictionaries switch_data = [] # Iterate through switch data from CFM API for i in switches: # Build dictionary for return out = { 'u_health': i['health'], 'u_ip_address': i['ip_address'], 'u_mac_address': i['mac_address'], 'u_name': i['name'], 'u_sw_version': i['sw_version'] } switch_data.append(out) return (True, switch_data) return (False, switches) </code></pre> <p>You can take just about any Python script you have written in the past and wrap it up in a Python class where you reference the base action. <strong>HpecfmBaseAction</strong> is the script that performs the <strong>API</strong> authentication (other actions would have a different base action). It uses credentials in the <strong>/opt/stackstorm/configs</strong> directory to authenticate.</p> <p>The script runs the <strong>fabric.get_switches()</strong> function and assigns the output to a variable called <strong>switches</strong>. As the script runs, a little checking is performed and verifies the returned items are a Python list. If verified, the script proceeds. You'll receive a lot of information back in the returned results, but you only need a few fields. So, setup a loop and grab the five variables that you need and return the <strong>switch_data</strong> array. By creating this action, you can now reference it in workflows.</p> <p><strong>Workflows</strong></p> <p>Workflows are YAML files that can run one or many actions. <strong>Orquesta</strong>, StackStorm's new workflow application, allows you to do some fairly complex workflows. If you're interested in learning more, check out <a href="https://docs.stackstorm.com/orquesta/index.html">Orquesta</a>.</p> <p>Typically, a <strong>sensor</strong> would identify an event, load a <strong>trigger,</strong> and that trigger would cause a <strong>workflow</strong> to spring into action. Getting to know StackStorm is like eating an elephant, you'll have to take one bite at a time. Because of this, I will discuss sensors, triggers, and rules in another blog.</p> <p>Earlier, I talked about how actions need the same name, but different file types. Workflows will appear odd at first because they will have the exact same name <strong>and</strong> file type. A workflow needs a YAML file, stored in the actions directory, just like the actions. It will also require <strong>another YAML</strong> file with the exact same name stored in the <strong>/actions/workflows</strong> directory. Don't worry. After you make several hundred StackStorm integration packs, you will get used to this. My little "trick" to help keep things straight in my head is to use an underscore in the title. For example, for an action, I would use <strong>do_this.yaml</strong> and for a workflow I would use <strong>dothis.yaml</strong>.</p> <p>OK, I'm getting to the end. I am going to wrap it up now with this example. Here, I have an action, <strong>get_switches</strong>, but it's just an action. If I want to really automate, I need a workflow to run this action and then do something with the results. Below you will find the <strong>getswitches</strong> workflow. If we start with the introductory YAML, it will get us to the entry_point, another YAML called getswitches.</p> <pre><code class="language-yaml">--- name: getswitches description: A workflow for getting plexxi switches into stackstorm. runner_type: orquesta entry_point: workflows/getswitches.yaml enabled: true </code></pre> <p>Here is a screenshot of the <strong>getswitches</strong> workflow YAML file stored in the actions/workflow directory.</p> <pre><code class="language-yaml">version: 1.0 description: A workflow to copy switch inventory from hpecfm to ServiceNow. tasks: getswitches: action: hpecfm.get_switches next: - when: &#x3C;% succeeded() %> publish: - switches: &#x3C;% result().result %> do: sendsnow sendsnow: action: hpecfm.sendsnow switches=&#x3C;% ctx().switches %> </code></pre> <p>When the workflow runs, it calls the action to <strong>get_switches</strong>. We just have to prefix the action with the name of the pack that contains it. Once it has the Python list from the action, it can <strong>publish</strong> it in the <strong>context,</strong> in an array called switches. The information is stored in the context, which is a strange concept at first, but it's just a place for saving things so other actions can read from it. The workflow goes on to run a final action called <strong>sendsnow</strong>. You can see that the <strong>sendsnow</strong> (no underscore) is another <strong>workflow</strong> in the hpecfm StackStorm pack. Yes, workflows can call other workflows. I'll just pause here a second and let that sink in…</p> <p>This is the introductory YAML file for <strong>sendsnow</strong>. It informs StackStorm to run the workflow sendsnow, but here it tells the workflow to expect an array (Python list) called <strong>switches</strong> with the <strong>parameters</strong> tag.</p> <pre><code class="language-yaml">--- name: sendsnow description: A workflow for sending plexxi switches to servicenow. runner_type: orquesta entry_point: workflows/sendsnow.yaml enabled: true parameters: switches: required: true type: array </code></pre> <p>I use <strong>with</strong> and <strong>item</strong> to iterate through the switches array. Remember, StackStorm saved the array in the context referenced by ctx().</p> <pre><code class="language-yaml">version: 1.0 description: "Send hpecfm switches to snow" input: - switches tasks: snowswitches: with: &#x3C;% ctx().switches %> action: servicenow.create_record table="u_cfm_asset" payload='&#x3C;% item() %>' </code></pre> <p>What is interesting to note is the final action being called is in another StackStorm integration pack called <strong>servicenow</strong>. This means I have the ServiceNow integration pack installed, and it has an action called create_record. StackStorm integration packs are freely available on the StackStorm Exchange <a href="https://exchange.stackstorm.org/">here</a>.</p> <p>Note that, in this example, I have workflows that can call any series of actions (or workflows) from a variety of integration packs, which are just waiting for you to use them. If you are not using StackStorm, guess what? You are going to have to write all those Python scripts yourself. Why reinvent the wheel? Using StackStorm, you can save yourself from having to write thousands of lines of Python and instead just use a handful of simple YAML files to stitch things together.</p> <p>After this workflow runs, all of the switches in the network are added to an asset inventory tracking database in Service Now. You could easily modify this for asset management of servers or storage devices as well. Keep an eye out on the <a href="/blog">HPE DEV blog site</a> for my next post where I will be talking about sensors, triggers, and rules. Our journey into StackStorm is just getting started, so stay tuned, you seekers of knowledge!</p>https://developer.hpe.com/container-images-for-hpe-oneview-sdks-are-now-available/https://developer.hpe.com/container-images-for-hpe-oneview-sdks-are-now-available/Mon, 13 Jul 2020 18:18:16 GMT<p>In keeping with Hewlett Packard Enterprise’s strategic vision on container use for hybrid IT, I am pleased to report that container images of HPE OneView 5.2 SDKs are now available. Docker images of <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">Ansible</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">Terraform</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">Chef</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">Puppet</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">Python</a>, <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">Golang</a> and <a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">Ruby</a> SDKs are now all available on Docker Hub. All prerequisite materials are incorporated into the container images to enable streamlined deployment, which will allow you to simplify maintenance, improve infrastructure agility, and reduce costs. In addition, you can expect all SDK releases in the future to incorporate an updated Docker image.</p> <p>Why this focus on containers? The advantage of using containers is that containers offer a virtual runtime environment that runs on top of a single operating system (OS) kernel. Containers virtualize at the operating system level, with multiple containers running atop the OS kernel directly, instead of virtualizing the hardware stack, as with the virtual machine approach. This means that containers are far more lightweight: they share the OS kernel, start much faster, and use a fraction of the memory when compared to booting an entire OS.</p> <p>Container images include everything required to enable a streamlined deployment process. They eliminate the need to sort through complex support matrices and package dependencies, offering one succinct manifest that can be version controlled and that allows for easy replication across machines in a cluster. Container images also include the software dependencies needed by the SDK, such as specific versions of programming language, runtimes and other software libraries.</p> <p>Containerized SDKs simplify maintenance as the images contain only the necessary package dependences. In addition, they do not require a dedicated node or OS, which guarantees deployment consistency. Combined with a service-based architecture, the entire unit that developers are asked to reason about becomes much smaller, leading to greater agility and productivity. Using containers eases development, testing, and overall management.</p> <p>All this translates to reduced cost through increased productivity. Not only do containers help developers. When containers are used, IT operations teams can focus on application deployment and management without bothering with details such as specific software versions and configurations. Teams spend less time debugging and diagnosing differences in environments, and more time shipping new functionality for users. Container use also means fewer bugs overall, since developers can now make assumptions in dev and test environments they can be sure will hold true in production.</p> <p>SDK Docker images are available in Docker Hub and GitHub SDK repositories. For more information, please refer to the following:</p> <ul> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-terraform">HPE OneView SDK for Terraform</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ruby">HPE OneView SDK for Ruby</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-python">HPE OneView SDK for Python</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-puppet">HPE OneView SDK for Puppet</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-chef">HPE OneView SDK for Chef</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-golang">HPE OneView SDK for Golang</a></li> <li><a href="https://hub.docker.com/repository/docker/hewlettpackardenterprise/hpe-oneview-sdk-for-ansible">HPE OneView SDK for Ansible</a></li> </ul>https://developer.hpe.com/kubernetes-application-containers-managing-containers-and-cluster-resour/https://developer.hpe.com/kubernetes-application-containers-managing-containers-and-cluster-resour/Fri, 10 Jul 2020 02:52:44 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Suzanne Ferry", "publish": "2018-10-12T07:00:00.000Z", "tags": "containers, kubernetes" </code></pre> <hr> <h2>Application Containers – What They Are, What They Do, And Why They Matter</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image10-1594350208331.png" alt=""></p> <p>We’ll start with an overview of what application containers are and how they are used in enterprises to help improve deployment time, consistency, and the efficiency and availability of applications. Along the way, we will cover key characteristics of what containers can and cannot do, and how they compare to virtual machines.</p> <p>We will also cover how Kubernetes is used to orchestrate containers and associated resources. We’ll discuss how Kubernetes schedules the deployment of containers, scales container resources, manages communication between applications in containers, and is used to monitor the health and availability of containers. Combining the HPE Ezmeral Data Fabric with application containers and Kubernetes makes applications fully capable of consuming and processing live, enterprise-level data.</p> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <h2>What’s an Application Container?</h2> <p>An application container is a stand-alone, all-in-one package for a software application. Containers include the application binaries, plus the software dependencies and the hardware requirements needed to run, all wrapped up into an independent, self-contained unit.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image13-1594350297951.png" alt=""></p> <h2>Deploy Containers</h2> <p>The application container is dropped into a system, then runs using the local hardware and operating system. Since it includes all of the necessary dependencies, the container functions exactly the same when deployed on a laptop, on a server, on a virtual machine, in the cloud, or on any other compatible system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1594350308304.png" alt=""></p> <h2>Share Containers</h2> <p>As a self-contained package, an application container can easily be moved to a different system, or even be uploaded and downloaded using a software hub, to be shared with others.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image22-1594350318108.png" alt=""></p> <h2>Containerized Applications in the Enterprise Environment</h2> <p>Let’s take a look at an example of how and why you might containerize an application in an enterprise environment.</p> <p>This example tracks how containers are used in the development cycle of an app designed to monitor the health sensors on wearable devices. The sensor data is streamed to the customer’s health care provider network, which performs machine learning on the data, to look for warning signs of health issues.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image18-1594350327111.jpg" alt=""></p> <p>Development of the health monitoring sensor tracking app is done by a team of developers, working on laptops. They commit their work to a software hub every day. For reasons that will become clear in a moment, let’s assume that the development laptops are all 3 - 4 years old, use spinning hard drives, have 16 GB of RAM, and each has its own, specialized version of some flavor of Linux and an IDE. The application works flawlessly in their development environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image21-1594350336265.jpg" alt=""></p> <h2>Containerized Microservices</h2> <p>Because the health monitoring application is large and complex, it gets broken into microservices, in order to run more efficiently. Each service is then packaged into a separate application container.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1594350345325.png" alt=""></p> <h2>Complete Containers</h2> <p>Each container includes the final code binaries for the individual services.</p> <p>Included are the dependencies for each service, such as the libraries and APIs needed to communicate with the sensors on any device the app supports, geolocation to know where the data is coming from, data streaming, web protocols, authentication with the health care system, and anything else needed for the app to run.</p> <p>Also included is a YAML (YAML Ain't Markup Language) file, which defines the CPU, RAM, network, and storage needs for each service.</p> <p>Each container includes only the dependencies it needs for the single service in it. The microservices approach allows each container to specialize for its service. Two services can even use different versions of the same library if needed, because the container environment allows them to function independently of each other.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image17-1594350356545.png" alt=""></p> <h2>Application Testing Environment</h2> <p>Once application development is complete, the containers are tested by the QA team.</p> <p>The QA environment uses a dozen nodes with CentOS, running a stripped-down version of the live environment. The servers are a few years old; they vary greatly with CPU speed and cores and the amount of RAM and network cards on each server. Some have spinning disks while others use SSDs.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image15-1594350365360.png" alt=""></p> <p>Because containers include all the software dependencies each service needs, the application runs exactly the same on the QA servers as it did on the development laptops cited earlier.</p> <p>Containers define what hardware resources are required for each service to run. The server carves these resources out when the service is needed and provides them for the container. The service runs inside the containerized environment, which is spared from having its resources cannibalized by other applications on the server. When the service is no longer needed, the container is shut down and the resources are released back to the system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image12-1594350391823.png" alt=""></p> <h2>Application Live Environment</h2> <p>When the application clears QA, it is ready to be released on the live environment, which, in this example, is made up of 100 nodes in a distributed cluster. These nodes use CentOS on the very latest hardware with 12 cores, 256 GB of RAM, SSDs, and gigabit network cards.</p> <p>Again, because the services are containerized with all of the dependencies they need, they run just as they did on the development laptops and QA servers, albeit quite a bit faster. The application services run happily in their containers, safe from disturbance by any other applications on the system, pulling resources when needed and releasing them when not.</p> <h2>High Availability with Containers</h2> <p>To create high availability for the application, each service is scaled by spinning up multiple instances of the container image and distributing them across the cluster. Each container only includes a single service and its dependencies, which makes them very lightweight. Cloning and launching new copies of a container takes just seconds. Quickly, several copies of the services are spread out across the cluster, each performing the work sent to it and returning results, independent of the other copies in the environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image2-1594350677060.png" alt=""></p> <h2>Share Containers</h2> <p>To share containers, you could, for example, just upload the containers to the cloud where they can be downloaded. Since the containers include everything the app needs to run, it will run functions the same as it does on your own clusters.</p> <h2>How Containers and Virtual Machines Differ</h2> <p>You might be wondering how using containers to develop an application differs from running the application in a virtual machine.</p> <p>An application container is similar to, but not the same as, a virtual machine. A container is a package of the application and its dependencies. It will run on any compatible OS and hardware. Containers are designed to be streamlined application packages and nothing else. In the case of the example, the containers were streamlined down to microservices, so that each container is as lean and efficient as possible.</p> <p>A virtual machine, on the other hand, includes a complete copy of an operating system, along with any applications and software dependencies running on it. In addition, a virtual machine requires a hypervisor layer to talk to the host server. A VM can be capable of performing more complex tasks, but requires a much larger package, plus more overhead in time and resources from a server.</p> <p>An application container is a clean and effective package for running a single application as efficiently as possible.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1594350685473.png" alt=""></p> <h2>Application Container Characteristics - Lightweight</h2> <p>Application containers consist of just an application, its software dependencies, and a small YAML file with its hardware requirements. They use the OS and infrastructure native to the system they are deployed onto. Therefore, a container is very lightweight when compared to other virtualization techniques, like VMs. They can be ported quickly to other environments and take just seconds to deploy, clone, or even relaunch in the case of a problem.</p> <p>While a container can include more than one application, that is not generally considered a best practice. An application container is a streamlined environment, designed to run an application efficiently. Adding more applications adds complexity and potential conflict to the package. In some cases, tightly coupled applications may share a container. In the vast majority of cases, however, an application will have its own specialized container.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1594350700977.png" alt=""></p> <h2>Containers Are Scalable</h2> <p>Since a container only includes an application and its dependencies, it has a very small footprint on the server. Because of this small size, several copies of a container can be launched on each server.</p> <p>A higher application density leads to more efficient processing, as there are more copies of the application to distribute the workload.</p> <p>Availability of the app is also greatly improved. The loss of a single container will not affect the overall function of the service, when there are several others to pick up the lost production.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image11-1594350709356.png" alt=""></p> <h2>Containers Are Portable</h2> <p>The small size of an application container makes it quick to move between servers, up to a cloud, or to mirror to another cluster. In addition, a container is completely self-sufficient and has all of the resources that the application needs to run.</p> <p>When moving a container, it can just be dropped onto its new home and launched in a plug-and-play fashion. You do not need to test for compatibility or load additional software.</p> <p>The containerized app can even run directly off of an instance in the cloud, giving you absolute flexibility over availability and the run-time environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594350718828.png" alt=""></p> <h2>Containers Have an Ecosystem</h2> <p>The application container ecosystem is very large and diverse. Thousands of containerized applications are available on hubs to use as templates or foundations for proprietary apps. A simple container with Apache Web Server can be downloaded, pre-built, saving the development time and resources needed to create common services over and over.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1594350727418.png" alt=""></p> <h2>Containers Help with Secure Multi-Tenancy</h2> <p>Because each application container creates an isolated environment for its application, the resources allocated to it are the entire machine. Other copies of the same container are "unaware" of each other.</p> <p>As a result, it’s easy to have multiple different applications on the same server, all running simultaneously. Each application uses the resources assigned to it. There are no concerns about an application taking resources from another application and causing it to crash. When an application completes its work, resources are released back to the system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image16-1594350736509.png" alt=""></p> <h2>Containers Offer Flexible Deployment</h2> <p>You know, now, that application containers are completely self-contained. Any application in a container runs in the same exact manner on any hardware. Because the container contains all dependencies, it doesn’t require the OS to supply any code. As long as the hardware can match the needs of the application, it will happily chug along no matter where you put it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image14-1594350746049.png" alt=""></p> <h2>Container Are Easy to Maintain and Specialized for Microservices</h2> <p>When an application is containerized, it is saved as an image. As the application grows, and as changes are made, these changes are saved as a new image on top of the original one. Therefore, any time you need to push or pull an upgrade of a containerized application, all you need to move and install is the new image. You do not need the entire container each time.</p> <p>Containers are a highly specialized runtime environment, designed to run a single application as efficiently as possible. Because of this specialization, they are perfectly designed to break up large applications into microservices.</p> <p>Each microservice can run in its container when called upon, then release the resources back to the system when it is done. You can also define a different number of clones for each container, based on the use of each service, allowing for more copies of services that are used more frequently in your app.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image24-1594350756822.png" alt=""></p> <h2>What Containers Are NOT – Persistent or Stateful</h2> <p>When a container is shut down, either intentionally or due to some failure, everything in the container goes down with it. You can save out results as part of your application, but other progress and data, like logs, are lost. Each time you need to use the application, or spin up a new copy, it starts from a new beginning state. The state of the application cannot be saved and built upon with a new iteration.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image3-1594350768170.png" alt=""></p> <h2>What Containers Are NOT – Data-Aware</h2> <p>A container does not know where its data comes from, nor where it is going. The container is completely enclosed in its environment, and cannot see data sources for work or storage. External data can be accessed, and results from work can be saved out of the container environment, but the container has no idea what or where that data is.</p> <p>Therefore, the app in the container cannot take advantage of data locality to make sure the work and data are on the same node. This may result in extra movement of data and reduce the efficiency of the service in the container.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image20-1594350784627.png" alt=""></p> <h2>What Containers Are NOT – Environment-Aware</h2> <p>Containers are not aware that the hardware that is provisioned to it is the entire server. This is helpful in that the containerized application is protected, but this also means that the application is not aware of, nor can it take advantage of, common services for enterprise applications, like networking with other apps, scheduling, distribution, and load balancing.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image19-1594350794873.png" alt=""></p> <h2>Container Summary</h2> <p>You have seen how application containers speed up the pipeline from production to delivery. They solve the problem of having an app working reliably in different environments, making development processes much more efficient and flexible.</p> <p>Breaking up large applications into microservices and putting them into containers provides fast and flexible services that can quickly be deployed and moved between systems.</p> <p>Containers are not a stand-alone solution, however, and have a few inherent limitations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image1-1594350805136.png" alt=""></p> <hr> <h2>Using Kubernetes to Manage Containers and Cluster Resources</h2> <p>This section discusses how Kubernetes manages containers and resources.</p> <p>Kubernetes is an environment that automates the orchestration of application containers. What does "Kubernetes automated orchestration" cover? It covers deployment, scaling, management, monitoring, and upgrades of individual, containerized microservices. Kubernetes takes care of all of the maintenance and tools around running application containers, so you can focus on application functionality.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image3-1594350972863.jpg" alt=""></p> <h2>Application Deployment</h2> <p>Deploying an application with Kubernetes requires just a single command. In the background, Kubernetes creates the runtime environment, requests the needed resources, handles the launch of the services, and provides each with an IP address. It also scales the containers across the cluster until each service is deployed to the level requested and maintains these levels 24/7.</p> <h2>Application Scaling</h2> <p>You decide how many clones of each service are needed. Because the services are containerized, you can set different levels for different parts of the app. When you first deploy, you calculate some starting numbers for each service. Kubernetes makes sure each service is running the correct number of copies. If there are too few, it will launch more. If there are too many, it will kill a few until the correct number are running.</p> <h2>Application Scale</h2> <p>Suppose you determine that there are too many copies of a service running and they are sitting dormant, or that application usage has increased and you need more copies to handle the load. You can change the settings on the deployment file, redeploy, and Kubernetes will update the number of each running service to meet the new requirements.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1594350995532.jpg" alt=""></p> <h2>High Availability</h2> <p>Kubernetes watches how many copies of each service are up. If a container has a failure and goes down, Kubernetes launches a new copy. Kubernetes continually verifies that the number of each service on the system matches what was requested.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1594351003832.jpg" alt=""></p> <p>If an entire server goes down, Kubernetes redeploys the missing containers on other nodes, again until the number of services running matches the defined limits. You can rest assured that your app will achieve the required six nines of availability, as long as your data center is active.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1594351011817.jpg" alt=""></p> <h2>Load Balancing</h2> <p>Kubernetes continuously monitors the usage of containers across nodes, verifying that the work is evenly distributed. If it finds an underused container or resource, it moves work to that resource, and may even move copies of a container to underused hardware.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image2-1594351020037.jpg" alt=""></p> <h2>Communication</h2> <p>When applications are broken into microservices, the individual services need to talk to each other, in order to pass along client information. Kubernetes creates a service within itself to enable the different microservices to communicate. This communication service determines which containers can use it, based on labels on the container, and then defines a port that can be used by any container with that label.</p> <p>As a service reads data from a wearable device on a customer, it will pass that data to the other services in the app that will stream the data, authenticate it with the health-care provider, and so on. Each instance of any service can use the same port to communicate with the other microservices in the app or any other services on the cluster that it needs.</p> <p>The communication service in Kubernetes is persistent, independent of the services that use it. If a container goes down or a new container is spun up, the service will continue to be available at its port to any application with the correct label.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1594351028278.jpg" alt=""></p> <h2>Multi-Tenancy</h2> <p>Let's consider the example of a health-monitoring application, serving thousands of users, sending data to a variety of health-care providers. With Kubernetes, the services could be divided up by health-care provider. Each provider could offer a differing number of services, based on usage, or could even provide variations on a service to a client, based on that client's particular needs.</p> <p>For example, say that this application spins up three copies of the app for users of Mega-Health, but provides four copies to Health R Us because they have a larger customer base. In addition, Health R Us uses a communication protocol different from Mega-Health – so, a separate microservice is used to connect to their system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594351052333.jpg" alt=""></p> <h2>Rolling Upgrades</h2> <p>When an application update is ready to roll out, the Kubernetes deployment file needs to be updated with the new information.</p> <p>Kubernetes will gradually kill existing containers with the current version of the app and spin up new containers with the updated version, until all containers for that service are running the new version.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1594351071753.jpg" alt=""></p> <h2>Rolling Downgrades</h2> <p>If there is a problem along the way, you can roll back the upgrade with a single command. Kubernetes will gradually kill containers with the new 2.0 version of the app and replace them with new instances of the older 1.0 version.</p> <hr> <p>This third section provides an overview of Kubernetes architecture.</p> <h2>Kubernetes Architecture</h2> <p>Here you can see the standard architecture for running application containers with Kubernetes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image15-1594351169191.jpg" alt=""></p> <h2>Kubernetes Object</h2> <p>An object in Kubernetes is anything that persists in the system, such as Pods, deployment records like ReplicationControllers, ReplicaSets, and Deployment controllers. Objects are used to define the desired state of the Kubernetes system.</p> <p>Pods define what is to be on the system; deployments define parameters of the Pods, such as how many copies; and services define their connections. When objects are created, Kubernetes will ensure that the objects persist, and the system matches what is defined by the objects.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image10-1594351181271.jpg" alt=""></p> <h2>Kubernetes Pod</h2> <p>A Pod is the basic unit of organization in Kubernetes. A Pod will contain one or more containers, all of which share storage, network, and deployment specifications. Everything in a Pod will be deployed together, at the same time, in the same location, and will share a schedule along with the deployment parameters.</p> <p>A Pod can contain more than one container, but in most practical applications, each Pod contains just a single container. Similar to the container itself, a Pod is a specialized environment. Each Pod has its own unique specifications, and those are usually optimized for the container in the Pod. Tightly coupled service containers that all perform a similar function, such as file locators or controllers, may share a Pod, but in most cases a single container keeps its environment optimized for the service it runs.</p> <p>If you do have multiple containers within a single Pod, they share an IP address that they can use to communicate outside of the Pod, and they can see each other through localhost.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1594351189416.jpg" alt=""></p> <h2>Kubernetes Name</h2> <p>Each object in Kubernetes is given a Name. The name of each object is provided to Kubernetes in the deployment record. Object names need to be unique within a namespace, but can be reused across separate namespaces in the system. If an object is removed from the system, the name can freely be used by another object.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image16-1594351201487.jpg" alt=""></p> <h2>Kubernetes Name Example</h2> <p>For example, let’s say you have a Pod that contains AwesomeApp. You want 3 instances of the Pod, so you name the Pod objects AwesomeApp1, AwesomeApp2, and AwesomeApp3. You decide that you want another copy, so you spin up a new instance of the Pod. Since it is in the same namespace as the other instances, you name it AwesomeApp4. Something happens along the line, and service AwesomeApp2 fails. Kubernetes kills the Pod that contains AwesomeApp2 and spins up a new copy. The name AwesomeApp2 is free to use now, since there is no object in the namespace with that name.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image11-1594351210989.jpg" alt=""></p> <p>If you create another namespace in the system, you are free to use the names AwesomeApp1, AwesomeApp2, and AwesomeApp3 again in this space.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1594351227395.jpg" alt=""></p> <h2>Kubernetes UID</h2> <p>The UID is a unique, internal identifier for each object in the system. The UID is defined by Kubernetes when the object is created and is used by the system to differentiate between clones of the same object.</p> <p>For example, say you have a Pod that contains AwesomeApp. You want 3 instances of the Pod, so let’s say Kubernetes assigns them the UIDs 001, 002, and 003. If you make a new namespace and deploy 3 new Pods into that space, each new Pod will be given its own unique UID. The UID of each object must be unique on the system, even across namespaces.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image14-1594351236821.jpg" alt=""></p> <h2>Kubernetes Label</h2> <p>Labels are key-value pairs used to identify and describe objects, such as "version:1.1" or "cluster:sf". An object can have as many labels as you want, but cannot have more than one of the same key.</p> <p>Labels are not used by the Kubernetes system, but provide a way for users to organize and map the objects in the system. For example, you can use them to perform an action on all Pods with the label "version:1.1" or a different action on all Pods with the label "cluster:tokyo".</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image12-1594351250937.jpg" alt=""></p> <h2>Kubernetes ReplicaSet (ReplicationController)</h2> <p>ReplicaSets and ReplicationControllers are used to ensure that the correct number of Pods are running. The ReplicaSet defines what Pods are in it and how many copies of the Pods are to exist across the system. The desired number of Pods in the system can easily be scaled up or down by updating the ReplicaSet. Kubernetes then updates the number of Pods on the system to match the new specifications.</p> <p>ReplicaSets define the Pods in it by the container image and one or more labels. Therefore, a different ReplicaSet can be made for different uses of the same container. For example, you can decide to have 3 copies of a container on your cluster in San Francisco, but only two copies of the same container on the Tokyo cluster. Alternately, you could have a ReplicaSet for our San Francisco cluster running version 1.1 of a service, while the Tokyo copies run version 1.2 of the same service.</p> <p>ReplicaSets allow you to customize the number and types of services for your app, running across the environment.</p> <p>ReplicaSet recently replaced the ReplicationController. Both perform the same function in a Kubernetes system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1594351262324.jpg" alt=""></p> <h2>Kubernetes Deployment Controller</h2> <p>The Deployment Controller defines the state of Deployment Objects, like Pods and ReplicaSets. In the example of deploying the health care app in a Kubernetes environment, we commonly referred to the state of the system being updated to match a defined state. Deployments are the object used to define and maintain the state of the system. Deployments are used to create and deploy, scale, monitor, roll back, and otherwise manage the state of Pods and ReplicaSets on the system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1594351271206.jpg" alt=""></p> <h2>Kubernetes Namespace</h2> <p>Namespaces are useful in multi-tenant systems, to divide resources among different users of the system. Similar Pods can be deployed in different namespaces with access restrictions to different user groups of the system, each providing unique specifications for that group.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1594351280707.jpg" alt=""></p> <h2>Kubernetes Service</h2> <p>When you launch a container, it is given an IP address to communicate in and out. This is fine for the life of the container, but each container has its own IP address, and the life cycle of any given container is not consistent.</p> <p>Kubernetes opens a port for application containers to see all other apps in the system. This port will remain open and consistent, even as instances of the containers are started and stopped. Therefore, communication between your application and other services on the system remains as long as your system is running.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image3-1594351299565.jpg" alt=""></p> <h2>Kubernetes Volume</h2> <p>Kubernetes Volumes solve the problem containers have with persistent data storage, at least to a point. A volume in Kubernetes is associated with the Pod, not the container within the Pod. Therefore, a container can fail and be relaunched, and the Volume will persist. The relaunched container can pick up where the failed one left off. If the Pod is killed, however, the Volume will be removed with it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image13-1594351309586.jpg" alt=""></p> <h2>Kubernetes Architecture Summary</h2> <p>We’ve seen how application containers make the development pipeline faster and provide a specialized environment for apps to run more efficiently and consistently across systems.</p> <p>We’ve also seen how Kubernetes orchestrates the use of containers at the enterprise level. With just a simple deployment file, and an occasional command, Kubernetes manages the deployment, scale, partitioning, distribution, and availability of containerized applications.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594351346680.jpg" alt=""></p> <hr> <p>This section discusses how Kubernetes and Containers work on MapR.</p> <p>You’re almost done with your introduction to using application containers and Kubernetes. The last part of your journey is to see how to put this all together in a real-world, enterprise environment.</p> <h2>Application Container Example</h2> <p>Throughout this Kubernetes blog series, we’ve used a fictional health care app to demonstrate how application containers and Kubernetes can be used in an enterprise environment. Let’s now take a look at a couple of different ways you can use the tools in the MapR Data Platform to deploy and manage such an app on a single platform and take advantage of unique features available with MapR.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image15-1594351490580.png" alt=""></p> <h2>MapR Solution: On-Premises</h2> <p>The scenario in this blog series involves streaming data to a health provider cluster for processing. In this example, the solution means processing the IoT data in an on-premises cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1594351501203.jpg" alt=""></p> <h2>Stream Sensor Data to MapR Cluster</h2> <p>The data is created by a wearable device. An app on that device uses the MapR Event Store service to stream the data to on-premises storage at the customer’s health care system.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image16-1594351510942.jpg" alt=""></p> <h2>Store Data in POSIX-Compliant Distributed File System</h2> <p>The IoT data is streamed directly into the MapR XD POSIX-compliant distributed file and object store and saved in their native JSON, CSV, or Parquet format.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image13-1594351527979.jpg" alt=""></p> <h2>Process Data Natively on MapR</h2> <p>Using both files saved in MapR XD and new live streaming, we can process the data with Spark to compare live information to legacy data saved in the system.</p> <p>In addition, MapR XD allows for the different sources of data to be tiered, based on level of access, giving faster access to data that is used more frequently.</p> <p>Kubernetes can spin up containerized machine learning apps on MapR to analyze the data natively as it streams in, all on the same cluster, saving the time of transforming the data before it is processed. In addition, MapR can scale the compute and storage independently. If more data is coming in, you can add more application containers on MapR to support the increased demand.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image3-1594351538907.jpg" alt=""></p> <h2>MapR Solution: Cloud</h2> <p>Alternatively, MapR provides an all cloud-based solution to the wearable health care app. In this solution, we containerize the MapR Data Platform and move it to the data, rather than bringing the data to the MapR cluster. This is vastly more efficient, as data can be processed where it is created, saving time and resources needed to move the data to the cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1594351546333.jpg" alt=""></p> <h2>Stream Data to the Cloud</h2> <p>Wearable IoT devices create data and send it to the cloud, using the MapR Event Store for Apache Kafka services. A cloud provider close to the device reduces data transfer time.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1594351553676.jpg" alt=""></p> <h2>Deploy Containerized MapR on Cloud</h2> <p>The MapR Data Platform is spun up in a container in the same cloud environment. The MapR Data Platform is broken into microservices, and the cluster can be spun up in just seconds. The MapR Event Store ingests the data into MapR XD on the cloud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image14-1594351561392.jpg" alt=""></p> <p>As clients in other areas create data on their wearable devices, MapR Event Store sends that data to cloud platforms hosted nearby. Containerized MapR clusters are spun up in those cloud environments as well, ingesting the data into MapR XD.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1594351569349.jpg" alt=""></p> <h2>Process Natively in the Cloud</h2> <p>Spark processes the data close to where it was created, greatly reducing time and resources needed to move the data.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image2-1594351576898.jpg" alt=""></p> <h2>Global Namespace Views Content as a Single Source</h2> <p>The MapR Global Namespace allows all of this data to be processed at its local center, but viewed together as though they are coming from a single cluster, without the need to move any data between clouds or to on-premises storage.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1594351584377.jpg" alt=""></p> <h2>MapR Data Platform Components</h2> <p>The following components of the MapR Data Platform can be used to make our health care app example functional, using application containers and Kubernetes.</p> <ul> <li>MapR Data Platform</li> <li>MapR Event Store for Apache Kafka</li> <li>MapR Distributed File and Object Store (MapR XD)</li> <li>Cloud Integration</li> <li>Global Namespace</li> <li>Live Data Processing</li> </ul> <h2>MapR Data Platform</h2> <p>The MapR Data Platform is a single, complete platform that supports enterprise-level data storage and processing needs.</p> <p>Just as containers provide a self-contained environment for an app to run as efficiently as possible, the MapR Data Platform provides a single environment for streaming, ingesting, processing, and storing data from the edge, IoT devices, the cloud, on-premises, or any combination of data sources and types.</p> <p>Just as Kubernetes handles all of the orchestration and maintenance of application containers, the MapR Data Platform handles all of the orchestration, distribution, scale, connectivity, replication, security, and high availability of your data and processing. MapR will take care of the maintenance, and your team can focus on the results.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594351592863.png" alt=""></p> <h2>MapR Event Store for Apache Kafka</h2> <p>MapR Event Store for Apache Kafka provides a platform for streaming data live from IoT devices. With MapR, you can stream data at an enterprise level, easily handling data from all of your customer's wearable devices.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image17-1594351600815.jpg" alt=""></p> <h2>MapR XD</h2> <p>MapR XD is a POSIX-compliant distributed file and object store. This allows you to directly ingest data from IoT devices in their native JSON, CSV, or Parquet formats, then directly query this data with Apache Drill, without spending any time or resources processing the data. You can also include large binary files like images, audio, or video for something like a security app that monitors a streaming CCTV feed. All of your data can be processed as it is streaming and gains the high availability, security, and replication advantages of a MapR cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image11-1594351610742.jpg" alt=""></p> <h2>MapR Data Storage Solutions: Cloud Integration</h2> <p>MapR natively supports data storage and application containers in all major cloud providers.</p> <p>The IoT data from your customer devices can be streamed to a cloud storage environment. From there, it can be accessed for processing from an on-premises cluster or even processed directly, using containers that are deployed in the same, or a different, cloud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image18-1594351618408.jpg" alt=""></p> <h2>MapR Data Storage Solutions: Global Namespace</h2> <p>MapR provides a global namespace for all these different data sources used on the platform. This global namespace allows all data sources that are used by the application containers on your cluster to be seen as coming from a single source. Therefore, data does not have to be moved or copied, saving valuable time and resources. In addition, live streaming data and data saved in the cloud or on-premises can all be processed together, without the need for any preprocessing or consolidation.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image1-1594351626084.jpg" alt=""></p> <h2>Live Data Processing</h2> <p>In our fictional application, we stream the data to on-premises storage at the customer’s health care system. MapR can spin up machine learning applications to analyze the data as it streams in, all on the same cluster, saving the time of copying or transferring the data. In addition, MapR can scale the compute and storage independently. If more data is coming in, just add more application containers to support the increased demand.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image12-1594351634768.jpg" alt=""></p> <h2>MapR Data Platform</h2> <p>All of these tools in the MapR Data Platform share the same high availability, security, and replication technology that is consistent across MapR, and with the global namespace available with MapR, containerized apps finally have a persistent data source that will remain throughout the lifetime of the cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594351645052.png" alt=""></p> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p>https://developer.hpe.com/tips-and-best-practices-to-take-advantage-of-spark-2x/https://developer.hpe.com/tips-and-best-practices-to-take-advantage-of-spark-2x/Wed, 08 Jul 2020 05:54:32 GMT<h2>Original post information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2019-02-05T07:00:00.000Z", "tags": "spark" </code></pre> <hr> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image1-1594188264230.png" alt=""></p> <p>With Apache Spark 2.0 and later versions, big improvements were implemented to enable Spark to execute faster, making lot of earlier tips and best practices obsolete. This blog post will first give a quick overview of what changes were made and then some tips to take advantage of these changes.</p> <h2>Project Tungsten</h2> <p><a href="https://databricks.com/glossary/tungsten">Tungsten</a> is the code name for the Spark project that makes changes to Apache Spark’s execution engine, focusing on improvements to the efficiency of memory and CPU usage. Tungsten builds upon ideas from modern compilers and massively parallel processing (MPP) technologies, such as Apache Drill, <a href="https://prestodb.io/docs/current/overview/concepts.html%23query-execution-model">Presto</a>, and <a href="https://arrow.apache.org/">Apache Arrow</a>. Spark 2.x improvements include:</p> <ul> <li>To reduce JVM object memory size, creation, and garbage collection processing, Spark explicitly manages memory and converts most operations to operate directly against binary data.</li> <li><a href="https://arrow.apache.org/">Columnar layout for memory data</a> avoids unnecessary I/O and accelerates analytical processing performance on modern CPUs and GPUs.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image10-1594188764938.png" alt=""></p> <ul> <li>Vectorization allows the CPU to operate on vectors, which are arrays of column values from multiple records. This takes advantage of modern CPU designs, by keeping all pipelines full to achieve efficiency.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image9-1594188775215.png" alt=""></p> <ul> <li> <p>To improve the speed of data processing through more effective use of L1/ L2/L3 CPU caches, Spark algorithms and data structures exploit memory hierarchy with cache-aware computation.</p> </li> <li> <p>Spark SQL’s Catalyst Optimizer underpins all the major new APIs in Spark 2.0 and later versions, from <a href="https://databricks.com/glossary/what-are-dataframes">DataFrames</a> and <a href="https://databricks.com/glossary/what-are-datasets">Datasets</a> to Structured Streaming. The Catalyst optimizer handles: analysis, logical optimization, physical planning, and code generation to compile parts of queries to Java bytecode. Catalyst now supports both rule-based and cost-based optimization.</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image12-1594188785209.png" alt=""></p> <ul> <li>Spark SQL “Whole-Stage Java Code Generation” optimizes CPU usage by generating a single optimized function in bytecode for the set of operators in a SQL query (when possible), instead of generating iterator code for each operator.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image11-1594188795498.png" alt=""></p> <h2>Tips for Taking Advantage of Spark 2.x Improvements</h2> <h3>Use Dataset, DataFrames, Spark SQL</h3> <p>In order to take advantage of Spark 2.x, you should be using Datasets, DataFrames, and Spark SQL, instead of RDDs. Datasets, DataFrames, and Spark SQL provide the following advantages:</p> <ul> <li>Compact columnar memory format</li> <li>Direct memory access</li> <li>Reduced garbage collection processing overhead</li> <li>Catalyst query optimization</li> <li>Whole-stage code generation</li> </ul> <p>When possible, use Spark SQL functions – for example, <code>to_date()</code>, <code>hour()</code> – instead of custom UDFs in order to benefit from the advantages above.</p> <p>Datasets provide the advantage of compile time type safety over DataFrames. However, Dataset functional transformations (like map) will not take advantage of query optimization, whole-stage code generation, and reduced GC.</p> <h3>Use the Best Data Store for Your Use Case</h3> <p>Spark supports several data formats, including CSV, JSON, ORC, and Parquet, and several data sources or connectors, popular NoSQL databases, and distributed messaging stores.</p> <p>But just because Spark supports a given data storage or format doesn’t mean you’ll get the same performance with all of them. Typically, data pipelines will involve multiple data sources and sinks and multiple formats to support different use cases and different read/write latency requirements. Here are some guidelines:</p> <ul> <li>File data stores are good for write once (append only), read many use cases. CSV and JSON data formats give excellent write path performance but are slower for reading; these formats are good candidates for collecting raw data for example logs, which require high throughput writes. Parquet is slower for writing but gives the best performance for reading;this format is good for BI and analytics, which require low latency reads.</li> <li>Apache HBase and MapR Database (now part of <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>) are good for random read/write use cases. MapR Database supports consistent, predictable, high throughput, fast reads and writes with efficient updates, automatic partitioning, and sorting.</li> <li>Apache Kafka and MapR Event Store for Kafka (also now part of <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>) are good for scalable reading and writing of real-time streaming data. MapR Event Store is good for data pipelines with stream-first architecture patterns and kappa or lambda architectures.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image13-1594188803710.png" alt=""></p> <h3>CSV and JSON Tips and Best Practices</h3> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image8-1594188812308.png" alt=""></p> <p>When persisting and compressing CSV and JSON files, make sure they are splittable, give high speeds, and yield reasonable compression. ZIP compression is not splittable, whereas Snappy is splittable; Snappy also gives reasonable compression with high speed. When reading CSV and JSON files, you will get better performance by specifying the schema, instead of using inference; specifying the schema reduces errors for data types and is recommended for production code.</p> <h3>Parquet Tips and Best Practices</h3> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image16-1594188820503.png" alt=""></p> <p>Apache Parquet gives the fastest read performance with Spark. Parquet arranges data in columns, putting related values in close proximity to each other to optimize query performance, minimize I/O, and facilitate compression. Parquet detects and encodes the same or similar data, using a technique that conserves resources. Parquet also stores column metadata and statistics, which can be pushed down to filter columns (discussed below). Spark 2.x has a vectorized Parquet reader that does decompression and decoding in column batches, providing ~ 10x faster read performance.</p> <p>Parquet files are immutable; modifications require a rewrite of the dataset. For streaming data, you can stream to a fast read/write data store, then extract data to Parquet files for specific analytic use cases or stream new datasets to a new partition (see partitioning below).</p> <h3>Parquet Partitioning</h3> <p>Spark table partitioning optimizes reads by storing files in a hierarchy of directories based on partitioning columns. For example, a directory structure could be organized by location, such as state/city, or by date, such as year/month, shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1594188830995.png" alt=""></p> <p>DataFrames can be saved as persistent tables into a Hive metastore, using the <a href="https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html">saveAsTable</a> command. If you do not have Hive setup, Spark will create a default local Hive metastore (using Derby). Persistent tables have several optimization benefits: partition and statistic metadata, and they can be bucketed (discussed later).</p> <p>As an example with the flight dataset, a lot of queries about departure delays are organized around the originating airport (the src column), so this could make a good partitioning column. Here is a JSON row from this Dataset:</p> <pre><code class="language-json">{ "id": "ATL_LGA_2017-01-01_AA_1678", "dofW": 7, "carrier": "AA", "src": "ATL", "dst": "LGA", "crsdephour": 17, "crsdeptime": 1700, "depdelay": 0.0, "crsarrtime": 1912, "arrdelay": 0.0, "crselapsedtime": 132.0, "dist": 762.0 } </code></pre> <p>Here is the code to persist a flights DataFrame as a table consisting of Parquet files partitioned by the src column:</p> <pre><code class="language-markdown">df.write.format("parquet") .partitionBy("src") .option("path", "/user/mapr/data/flights") .saveAsTable("flights") </code></pre> <p>Below is the resulting directory structure as shown by a Hadoop list files command:</p> <pre><code class="language-markdown">hadoop fs -ls /user/mapr/data/flights /user/mapr/data/flights/src=ATL /user/mapr/data/flights/src=BOS /user/mapr/data/flights/src=CLT /user/mapr/data/flights/src=DEN /user/mapr/data/flights/src=DFW /user/mapr/data/flights/src=EWR /user/mapr/data/flights/src=IAH /user/mapr/data/flights/src=LAX /user/mapr/data/flights/src=LGA /user/mapr/data/flights/src=MIA /user/mapr/data/flights/src=ORD /user/mapr/data/flights/src=SEA /user/mapr/data/flights/src=SFO </code></pre> <p>Below, we see that the src=DEN subdirectory contains two Parquet files:</p> <pre><code class="language-markdown">hadoop fs -ls /user/mapr/data/flights/src=DEN /user/mapr/data/flights/src=DEN/part-00000-deb4a3d4-d8c3-4983-8756-ad7e0b29e780.c000.snappy.parquet /user/mapr/data/flights/src=DEN/part-00001-deb4a3d4-d8c3-4983-8756-ad7e0b29e780.c000.snappy.parquet </code></pre> <h3>Partition Pruning and Predicate Pushdown</h3> <p>Partition pruning is a performance optimization that limits the number of files and partitions that Spark reads when querying. After partitioning the data, queries that match certain partition filter criteria improve performance by allowing Spark to only read a subset of the directories and files. When partition filters are present, the catalyst optimizer pushes down the partition filters. The scan reads only the directories that match the partition filters, thus reducing disk I/O. For example, the following query reads only the files in the src=DEN partition directory in order to query the average departure delay for flights originating from Denver.</p> <pre><code class="language-markdown">df.filter("src = 'DEN' and depdelay > 1") .groupBy("src", "dst").avg("depdelay") .sort(desc("avg(depdelay)")).show() result: +---+---+------------------+ |src|dst| avg(depdelay)| +---+---+------------------+ |DEN|EWR|54.352020860495436| |DEN|MIA| 48.95263157894737| |DEN|SFO|47.189473684210526| |DEN|ORD| 46.47721518987342| |DEN|DFW|44.473118279569896| |DEN|CLT|37.097744360902254| |DEN|LAX|36.398936170212764| |DEN|LGA| 34.59444444444444| |DEN|BOS|33.633187772925766| |DEN|IAH| 32.10775862068966| |DEN|SEA|30.532345013477087| |DEN|ATL| 29.29113924050633| +---+---+------------------+ </code></pre> <p>Or in SQL:</p> <pre><code class="language-sql">%sql select src, dst, avg(depdelay) from flights where src='DEN' and depdelay > 1 group by src, dst ORDER BY src </code></pre> <p>You can see the physical plan for a DataFrame query in the Spark web UI SQL tab (discussed in chapter 3) or by calling the explain method shown below. Here in red, we see partition filter push down, which means that the src=DEN filter is pushed down into the Parquet file scan. This minimizes the files and data scanned and reduces the amount of data passed back to the Spark engine for the aggregation average on the departure delay.</p> <pre><code class="language-markdown">df.filter("src = 'DEN' and depdelay > 1") .groupBy("src", "dst").avg("depdelay") .sort(desc("avg(depdelay)")).explain == Physical Plan == TakeOrderedAndProject(limit=1001, orderBy=[avg(depdelay)#304 DESC NULLS LAST], output=[src#157,dst#149,avg(depdelay)#314]) +- \*(2) HashAggregate(keys=[src#157, dst#149], functions=[avg(depdelay#152)], output=[src#157, dst#149, avg(depdelay)#304]) +- Exchange hashpartitioning(src#157, dst#149, 200) +- \*(1) HashAggregate(keys=[src#157, dst#149], functions=[partial_avg(depdelay#152)], output=[src#157, dst#149, sum#321, count#322L]) +- \*(1) Project[dst#149, depdelay#152, src#157] +- \*(1)Filter (isnotnull(depdelay#152) &#x26;&#x26; (depdelay#152 > 1.0)) +- \*(1) FileScan parquet default.flights[dst#149,depdelay#152,src#157] Batched: true, Format: Parquet, Location: PrunedInMemoryFileIndex[maprfs:/user/mapr/data/flights/src=DEN], PartitionCount: 1, PartitionFilters: [isnotnull(src#157), (src#157 = DEN)], PushedFilters: [IsNotNull(depdelay), GreaterThan(depdelay,1.0)], ReadSchema: struct&#x3C;dst:string,depdelay:double&#x26;gt; </code></pre> <p>The physical plan is read from the bottom up, whereas the DAG is read from the top down. Note: the Exchange means a shuffle occurred between stages.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image2-1594188838987.png" alt=""></p> <h3>Partitioning Tips</h3> <p>The partition columns should be used frequently in queries for filtering and should have a small range of values with enough corresponding data to distribute the files in the directories. You want to avoid too many small files, which make scans less efficient with excessive parallelism. You also want to avoid having too few large files, which can hurt parallelism.</p> <h3>Coalesce and Repartition</h3> <p>Before or when writing a DataFrame, you can use dataframe.coalesce(N) to reduce the number of partitions in a DataFrame, without shuffling, or df.repartition(N) to reorder and either increase or decrease the number of partitions with shuffling data across the network to achieve even load balancing.</p> <pre><code class="language-markdown">df.write.format("parquet") .repartition(13) .partitionBy("src") .option("path", "/user/mapr/data/flights") .saveAsTable("flights") </code></pre> <h3>Bucketing</h3> <p>Bucketing is another data organization technique that groups data with the same bucket value across a fixed number of “buckets.” This can improve performance in wide transformations and joins by avoiding “shuffles.” With wide transformation shuffles, data is sent across the network to other nodes and written to disk, causing network and disk I/O and making the shuffle a costly operation. Below is a shuffle caused by a df.groupBy("carrier").count; if this dataset were bucketed by “carrier,” then the shuffle could be avoided.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image14-1594188846233.png" alt=""></p> <p>Bucketing is similar to partitioning, but partitioning creates a directory for each partition, whereas bucketing distributes data across a fixed number of buckets by a hash on the bucket value. Tables can be bucketed on more than one value and bucketing can be used with or without partitioning.</p> <p>As an example with the flight dataset, here is the code to persist a flights DataFrame as a table, consisting of Parquet files partitioned by the src column and bucketed by the dst and carrier columns (sorting by the id will sort by the src, dst, flightdate, and carrier, since that is what the id is made up of):</p> <pre><code class="language-markdown">df.write.format("parquet") .sortBy("id") .partitionBy("src") .bucketBy(4,"dst","carrier") .option("path", "/user/mapr/data/flightsbkdc") .saveAsTable("flightsbkdc") </code></pre> <p>The resulting directory structure is the same as before, with the files in the src directories bucketed by dst and carrier. The code below computes statistics on the table, which can then be used by the Catalyst optimizer. Next, the partitioned and bucketed table is read into a new DataFrame df2.</p> <pre><code class="language-markdown">spark.sql("ANALYZE TABLE flightsbkdc COMPUTE STATISTICS") val df2 = spark.table("flightsbkdc") </code></pre> <p>Next, let’s look at the optimizations for the following query:</p> <pre><code class="language-markdown">df2.filter("src = 'DEN' and depdelay > 1") .groupBy("src", "dst","carrier") .avg("depdelay") .sort(desc("avg(depdelay)")).show() result: +---+---+-------+------------------+ |src|dst|carrier| avg(depdelay)| +---+---+-------+------------------+ |DEN|EWR| UA| 60.95841209829867| |DEN|LAX| DL|59.849624060150376| |DEN|SFO| UA|59.058282208588956| . . . </code></pre> <p>Here again, we see partition filter and filter pushdown, but we also see that there is no “Exchange” like there was before bucketing, which means there was no shuffle to aggregate by src, dst, and carrier.</p> <pre><code class="language-markdown">== Physical Plan == TakeOrderedAndProject(limit=1001, orderBy=[avg(depdelay)#491 DESC NULLS LAST], output=[src#460,dst#452,carrier#451,avg(depdelay)#504]) +- \*(1) HashAggregate(keys=[src#460, dst#452, carrier#451], functions=[avg(depdelay#455)], output=[src#460, dst#452, carrier#451, avg(depdelay)#491]) +- \*(1) HashAggregate(keys=[src#460, dst#452, carrier#451], functions=[partial_avg(depdelay#455)], output=[src#460, dst#452, carrier#451, sum#512, count#513L]) +- \*(1)Project [carrier#451, dst#452, depdelay#455, src#460] +- \*(1) Filter(isnotnull(depdelay#455) &#x26;&#x26; (depdelay#455 > 1.0)) +- \*(1) FileScan parquet default.flightsbkdc [carrier#451,dst#452,depdelay#455,src#460] Batched: true, Format: Parquet, Location: PrunedInMemoryFileIndex [maprfs:/user/mapr/data/flightsbkdc/src=DEN], PartitionCount: 1, PartitionFilters: [isnotnull(src#460), (src#460 = DEN)], PushedFilters: [IsNotNull(depdelay), GreaterThan(depdelay,1.0)], ReadSchema: struct&#x3C;carrier:string,dst:string,depdelay:double&#x26;gt; </code></pre> <p>In the DAG below, we see that there is no exchange shuffle, and we see “Whole-Stage Java Code Generation,” which optimizes CPU usage by generating a single optimized function in bytecode.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image15-1594188853521.png" alt=""></p> <h3>Bucketing Tips</h3> <p>Partitioning should only be used with columns that have a limited number of values; bucketing works well when the number of unique values is large. Columns which are used often in queries and provide high selectivity are good choices for bucketing. Spark tables that are bucketed store metadata about how they are bucketed and sorted, which optimizes:</p> <ul> <li>Queries on bucketed values (Spark 2.4 supports bucket pruning)</li> <li>Aggregations on bucketed values (wide transformations)</li> <li>Joins on bucketed values</li> </ul> <h2>Data Modeling, Partitioning, and Filter Pushdown</h2> <h3>Data Modeling: Partitioning and Row Key Design</h3> <p>With MapR Database (now part of <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>), a table is automatically partitioned into tablets across a cluster by key range, providing for scalable and fast reads and writes by row key.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1594188884373.png" alt=""></p> <p>In this use case, the row key (the id) starts with the origin (destination airport codes), followed by the flightdate and carrier, so the table is automatically partitioned and sorted by the src, dst, date, and carrier.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1594188895684.png" alt=""></p> <h3>Data Modeling: Avoiding JOINS with Nested Entities</h3> <p>If your tables exist in a one-to-many relationship, it’s possible to model it as a single document; this can avoid expensive JOINS. In the one-to-many relationship example below, we have an order table, which has a one-to-many relationship with an order items table.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1594188904918.png" alt=""></p> <p>Here is a nested entity example of this one-to-many relationship in a document database. In this example, the order and related line items are stored together and can be read together with a find on the row key (_id). This makes the reads a lot faster than joining tables together.</p> <pre><code class="language-markdown">{ “id”: “123”, “date”: “10/10/2017”, “ship_status”:”backordered” “orderitems”: \[ { “itemid”: “4348”, “price”: 10.00 }, { “itemid”: “5648”, “price”: 15.00 }] } </code></pre> <h3>Projection and Filter Pushdown into MapR Database (now part of <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>)</h3> <p>Below, we see the physical plan for a DataFrame query, with projection and filter pushdown highlighted in red. This means that the scanning of the src, dst, and depdelay columns and the filter on the depdelay column are pushed down into MapR Database, meaning that the scanning and filtering will take place in MapR Database before returning the data to Spark. Projection pushdown minimizes data transfer between MapR Database and the Spark engine by omitting unnecessary fields from table scans. It is especially beneficial when a table contains many columns. Filter pushdown improves performance by reducing the amount of data passed between MapR Database and the Spark engine when filtering data.</p> <pre><code class="language-markdown">df.filter("src = 'ATL' and depdelay > 1") .groupBy("src", "dst") .avg("depdelay").sort(desc("avg(depdelay)")).explain == Physical Plan == \*(3) Sort [avg(depdelay)#273 DESC NULLS LAST], true, 0 +- Exchange rangepartitioning(avg(depdelay)#273 DESC NULLS LAST, 200) +- \*(2) HashAggregate(keys=[src#5, dst#6], functions=[avg(depdelay#9)]) +- Exchange hashpartitioning(src#5, dst#6, 200) +- \*(1) HashAggregate(keys=[src#5, dst#6], functions=[partial_avg(depdelay#9)]) +- \*(1) Filter (((isnotnull(src#5) &#x26;&#x26; isnotnull(depdelay#9)) &#x26;&#x26; (src#5 = ATL)) &#x26;&#x26; (depdelay#9 > 1.0)) +- \*(1) Scan MapRDBRelation(/user/mapr/flighttable [src#5,dst#6,depdelay#9] PushedFilters: [IsNotNull(src), IsNotNull(depdelay), EqualTo(src,ATL), GreaterThan(depdelay,1.0)] </code></pre> <h2>Spark Web UI and SQL Tips</h2> <p>Here is a summary of tips and what to look for:</p> <h3>SQL Tab</h3> <p>You can see details about the query plan produced by Catalyst on the web UI SQL tab. In the query plan details, you can check and see:</p> <ul> <li>The amount of time for each stage.</li> <li>If partition filters, projection, and filter pushdown are occurring.</li> <li>Shuffles between stages (Exchange) and the amount of data shuffled. If joins or aggregations are shuffling a lot of data, consider bucketing. You can set the number of partitions to use when shuffling with the spark.sql.shuffle.partitions option.</li> <li>The join algorithm being used. Broadcast join should be used when one table is small; sort-merge join should be used for large tables. You can use broadcast hint to guide Spark to broadcast a table in a join. For faster joins with large tables using the sort-merge join algorithm, you can use bucketing to pre-sort and group tables; this will avoid shuffling in the sort merge.</li> </ul> <p>Use the Spark SQL <code>ANALYZE TABLE tablename COMPUTE STATISTICS</code> to take advantage of cost-based optimization in the Catalyst Planner.</p> <h3>Stages Tab</h3> <p>You can use the stage detail metrics to identify problems with an executor or task distribution. Things to look for:</p> <ul> <li>Tasks that are taking longer and/or killed tasks. If your task process time is not balanced, resources could be wasted.</li> <li>Shuffle read size that is not balanced.</li> <li>If your partitions/tasks are not balanced, then consider repartition as described under partitioning.</li> </ul> <h3>Storage Tab</h3> <p>Caching Datasets can make execution faster if the data will be reused. You can use the storage tab to see if important Datasets are fitting into memory.</p> <h3>Executors Tab</h3> <p>You can use the executors tab to confirm that your application has the amount of resources needed.</p> <ul> <li>Shuffle Read Write Columns: shows size of data transferred between stages</li> <li>Storage Memory Column: shows the current used/available memory</li> <li>Task Time Column: shows task time/garbage collection time</li> </ul> <h2>References and More Information</h2> <ul> <li><a href="https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html">Project Tungsten: Bringing Apache Spark Closer to Bare Metal</a></li> <li><a href="https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html">Apache Spark as a Compiler</a></li> <li><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">Apache Spark SQL Performance Tuning</a></li> <li><a href="https://databricks.com/session/optimizing-apache-spark-sql-joins">Spark Summit Session: Optimizing Apache Spark SQL Joins</a></li> <li><a href="https://db-blog.web.cern.ch/blog/luca-canali/2017-06-diving-spark-and-parquet-workloads-example">Diving into Spark and Parquet Workloads, by Example</a></li> <li><a href="https://databricks.com/session/hive-bucketing-in-apache-spark">Spark Summit Hive Bucketing in Apache Spark</a></li> <li><a href="https://databricks.com/session/lessons-from-the-field-episode-ii-applying-best-practices-to-your-apache-spark-applications">Lessons from the Field, Episode II: Applying Best Practices to Your Apache Spark Applications</a></li> <li><a href="https://databricks.com/session/why-you-should-care-about-data-layout-in-the-filesystem">Why You Should Care about Data Layout in the Filesystem</a></li> <li><a href="https://databricks.com/session/spark-parquet-in-depth">Spark + Parquet In Depth</a></li> <li><a href="http://c2fo.io/c2fo/spark/aws/emr/2016/07/06/apache-spark-config-cheatsheet/">Apache Spark: Config Cheatsheet</a></li> <li><a href="https://spark.apache.org/docs/latest/configuration.html">Spark Configuration</a></li> <li><a href="https://dzone.com/articles/talend-and-apache-spark-debugging-and-logging-best">Apache Spark: Debugging and Logging Best Practices</a></li> </ul> <p><strong>Editor’s Note:</strong> MapR products referenced are now part of the <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>.</p>https://developer.hpe.com/data-modeling-guidelines-for-nosql-json-document-databases/https://developer.hpe.com/data-modeling-guidelines-for-nosql-json-document-databases/Wed, 08 Jul 2020 05:22:33 GMT<h2>Original Post Information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2017-10-26T12:00:00.000", "tags": "data modeling" </code></pre> <hr> <p>In this blog post, I’ll discuss how NoSQL data modeling is different from traditional relational schema data modeling, and I’ll also provide you with some guidelines for document database data modeling.<br> Document databases, such as MapR Database (now part of <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>), are sometimes called "schema-less", but this is a misnomer. Document databases don't require the same predefined structure as a relational database, but you do have to define the facets of how you plan to organize your data. Typically with a NoSQL data store you want to aggregate your data so that the data can quickly be read together, instead of using joins. A properly designed data model can make all the difference in how your application performs. One of our solution architects worked with a customer, and in a one-hour conversation about schema design, was able to improve access performance by a factor of 1,000x. These concepts matter.</p> <h2><strong>Why NoSQL?</strong></h2> <p>Simply put the motivation behind NoSQL is data volume, velocity, and/or variety. MapR Database (now part of HPE Ezmeral Data Fabric) provides for data variety with two different data models:</p> <ul> <li> <p>HPE Ezmeral Data Fabric as a Wide column database with an Apache HBase API</p> </li> <li> <p>HPE Ezmeral Data Fabric as a Document database with an Open JSON API</p> </li> </ul> <img src="/uploads/media/2020/6/mapr-db-json-1594186549663.png" alt="HPE Ezmeral Data Fabric" width="900"> <p>HPE Ezmeral Data Fabric JSON is different than other Document data stores in that the row key design is the same for both models, and both can store data (columns or documents) with different access patterns in a different column family with the same row key.</p> <h2><strong>Relational vs. NoSQL Data Modeling</strong></h2> <p>In relational design, the focus and effort is around describing the entity and its relation with other entities; the queries and indexes are designed later. With a relational database you normalize your schema, which eliminates redundant data and makes storage efficient. Then queries with joins bring the data back together again. However joins cause bottlenecks on read, with data distributed across a cluster, this model does not scale horizontally. With HPE Ezmeral Data Fabric, a table is automatically partitioned across a cluster by key range, and each server is the source for a subset of a table (called a tablet). HPE Ezmeral Data Fabric has a “query-first” schema design, queries should be identified first, then the row key should be designed to distribute the data evenly and also to give a meaningful primary index to query by. The row document (JSON) or columns (HBase) should be designed to group data together that will be read together. With HPE Ezmeral Data Fabric you de-normalize your schema to store in one row or document what would be multiple tables with indexes in a relational world. Grouping the data by key range provides for fast reads and writes by row key.</p> <img src="/uploads/media/2020/6/mapr-db-read-writes-1594186238593.png" alt="HPE Ezmeral Data Fabric faster read and writes" width="900"> <h2><strong>NoSQL Data Modeling Process</strong></h2> <p>It is useful to start off with <a href="https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model">Entity Relationship</a> modeling in order to define the entities, relationships, and attributes in your application:</p> <ul> <li> <p>Entities: Main objects in your application</p> </li> <li> <p>Attributes: properties of the objects in your application</p> </li> <li> <p>Relationships: connections between entities - 1-1, 1-many, many-many</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/artist-performs-song-1594186280849.png" alt="Artist - Performs - Song"></p> <p>The E-R model can be used with your query and data access patterns to define the physical model so that the data that is read together is stored together. As a modeling example we will use a social application similar to reddit (Note: I do not know how reddit is really implemented). Here are the use cases:</p> <ul> <li> <p>Users can post URLs to articles by category (like news, sports…).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/reddit-use-case-1-1594186305947.png" alt="Reddit use case"></p> </li> <li> <p>Users can then make comments on posts</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/reddit-use-case-2-1594186327388.png" alt="Reddit use case"></p> </li> </ul> <p>Some of the query requirements are:</p> <ul> <li> <p>Display the posts by category and date (most recent first)</p> </li> <li> <p>Display the comments by post</p> </li> <li> <p>Display the posts by userid</p> </li> </ul> <h2><strong>Logical Model Example</strong></h2> <p>This is an E-R Diagram for our example social application: <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/e-r-diagram-1594186359002.png" alt="E-R Diagram"></p> <p>The Entities are:</p> <ul> <li>User, Post, Comment, Category</li> </ul> <p>The relations are:</p> <ul> <li> <p>A User makes a post</p> </li> <li> <p>A Post has comments</p> </li> <li> <p>A Post belongs to a category</p> </li> </ul> <h2><strong>Relational Model Example</strong></h2> <p>This is the relational model for the example social application: <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/relation-model-social-app-1594186378259.png" alt="Relation Model for Social Application"></p> <ul> <li> <p>Users are stored in the user table</p> </li> <li> <p>The posted URL is stored in the Post table with a foreign key to the user that posted it, and a foreign key to the category for the post.</p> </li> <li> <p>Comments about a post are stored in the comments table with a foreign key to the post and a foreign key to the user that commented.</p> </li> </ul> <p></p> <h2><strong>Normalization</strong></h2> <p>In a relational database, you normalize the schema to eliminate redundancy by putting repeating information into a table of its own. In this example below, we have an order table, which has a one-to-many relationship with an order items table. The order items table has a foreign key with the id of the corresponding order.</p> <img src="/uploads/media/2020/6/order-items-table-1594186396974.png" alt="Order Items Table" width="900"> <h2><strong>Denormalization</strong></h2> <p>In a denormalized datastore, you store in one table what would be multiple indexes in a relational world. <a href="https://en.wikipedia.org/wiki/Denormalization">Denormalization</a> can be thought of as a replacement for joins. Often with NoSQL, you de-normalize or duplicate data so that data is accessed and stored together.</p> <h2><strong>Parent-Child Relationship–Embedded Entity</strong></h2> <p>Here is an example of denormalization of the SALES_ITEM schema in a Document database:</p> <pre><code class="language-json">{ "_id": "123", "date": "10/10/2017", “ship_status”:”backordered” "orderitems": [ { "itemid": "4348", "price": 10.00 }, { "itemid": "5648", "price": 15.00 }] } </code></pre> <p>If your tables exist in a one-to-many relationship, it’s possible to model it as a single document. In this example, the order and related line items are stored together and can be read together with a find on the row key (_id). This makes the reads a lot faster than joining tables together. Note: that the maximum default row size is 32MB, and optimal size is between 50-100KB. If the embedded entities are really long then they could be bucketed by row key, or you could just store the id to the embedded entity table (which would require your application to query that table also).</p> <h2><strong>Document Model Example</strong></h2> <p>This is the document model for the example social application:</p> <img src="/uploads/media/2020/6/document-model-social-app-1594186415200.png" alt="Document Model for Social Application" width="900"> <p>There are 2 tables in the document model compared to 4 in the relational:</p> <ul> <li> <p>User details are stored in the user table</p> </li> <li> <p>Posted URLs are stored in the Post table</p> </li> <li> <p>The row key is composed of the category and a reverse timestamp so that posts will be grouped by category with the most recent first.</p> </li> <li> <p>There is a secondary index on the posted by attribute, to query by who submitted the URL.</p> </li> <li> <p>Comments are embedded in the post table</p> </li> </ul> <h2><strong>Composite Row Key Design</strong></h2> <p>Row keys are the primary index for MapR Database (now part of HPE Ezmeral Data Fabric) (HPE Ezmeral Data Fabric JSON 6.0 also has secondary indexes). Data is automatically distributed as it is written by sorted row key range. You can include multiple data elements in a “composite” row key, which can be useful for grouping rows together for finding by key range. For example if you wanted to group posts by category and date, you could use a row key like <code>“SPORTS_20131012”</code> (if you want the most recent first use a reverse timestamp). If you wanted to group restaurants by location you could use a row key like <code>“TN_NASHVL_PANCAKEPANTRY”</code>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/prefix-1594186429138.png" alt="Prefix"></p> <p>Another option is to add a hash prefix to the row key in order to get good distribution, and still have a secondary grouping.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/prefix-row-key-1594186446213.png" alt="Prefix the row key"></p> <h2><strong>Generic Data, Event Data, and Entity-Attribute-Value</strong></h2> <p>Generic data is often expressed as name value or entity attribute value. In a relational database, this is complicated to represent because every row represents an instance of a <strong>similar object</strong>. JSON allows easy variation across records. Here is an example of clinical patient event data:</p> <pre><code class="language-json">patientid-timestamp, Temperature , "102" patientid-timestamp, Coughing, "True" patientid-timestamp, Heart Rate, "98" </code></pre> <p>This is the document model for the clinical patient event data:</p> <img src="/uploads/media/2020/6/document-model-event-data-1594186470650.png" alt="Document Model Event Data" width="900"> <p>The Row Key is the patient ID plus a time stamp. The variable event type and measurement is put in name value pairs.</p> <h2><strong>Tree, Adjacency List, Graph Data</strong></h2> <p>Here is an example of a tree, or <a href="https://en.wikipedia.org/wiki/Adjacency_list">adjacency list</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/document-model-tree-1594186489204.png" alt="Document Model Tree"></p> <p>Here is a document model for the tree shown above (there are multiple ways to represent trees):</p> <pre><code class="language-json">{ "_id": "USA", “type”:”state”, "children": ["TN",”FL] "parent": null } { "_id": "TN", “type”:”state”, "children": ["Nashville”,”Memphis”] "parent": "USA” } { "_id": "FL", “type”:”state”, "children": ["Miami”,”Jacksonville”] "parent": "USA” } { "_id": "Nashville", “type”:”city”, "children": [] "parent": "TN” } </code></pre> <p>Each document is a tree node, with the row key equal to the node id. The parent field stores the parent node id. The children field stores an array of children node ids. A secondary index on the parent and children fields allows to quickly find the parent or children nodes.</p> <h2><strong>Inheritance Mapping</strong></h2> <p>In modern object-oriented programming models, different object types can be related, for instance, by extending the same base type. In object-oriented design, these objects are considered instances of the same base type, as well as instances of their respective subtypes. It is useful to store objects in a single database table to simplify comparisons and calculations over multiple objects. But we also need to allow objects of each subtype to store their respective attributes, which may not apply to the base type or to other subtypes. This does not match a relational model but is very easy to do with a document model. Here is an example of object inheritance for store products, bike, pedal, and jersey are all types of store products: <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/inheritance-mapping-1594186510565.png" alt="Inheritance Mapping Online Store Example"></p> <p>In this online store example, the type of product is a prefix in the row key. Some of the name value pairs are different, and may be missing depending on the type of product. This allows to model different product types in the same table and to find a group of products easily by product type.</p> <img src="/uploads/media/2020/6/product-type-1594186531426.png" alt="Product Types" width="900"> <p>In this blog post, you learned how document database data modeling is different from traditional relational schema modeling, and you also got some guidelines for document database data modeling.</p> <h2><strong>For More Information:</strong></h2> <ul> <li> <p><a href="https://github.com/ojai/ojai">Open JSON Application Interface</a></p> </li> <li> <p><a href="https://github.com/ojai/ojai/wiki">Open JSON Application Interface wiki</a></p> </li> <li> <p><a href="https://www.infoq.com/articles/unified-data-modeling-for-relational-and-nosql-databases">Unified Data Modeling for Relational and NoSQL Databases</a></p> </li> <li> <p><a href="https://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/">NOSQL DATA MODELING TECHNIQUES</a></p> </li> </ul>https://developer.hpe.com/spark-101-what-is-it-what-it-does-and-why-it-matters/https://developer.hpe.com/spark-101-what-is-it-what-it-does-and-why-it-matters/Fri, 03 Jul 2020 06:19:21 GMT<h2>Original post information:</h2> <pre><code>"authorDisplayName": "Carol McDonald", "publish": "2018-10-17T08:00:00.000Z", "tags": "spark" </code></pre> <hr> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image5-1593756989434.jpg" alt=""></p> <p>In this blog post, we will give an introduction to Apache Spark and its history and explore some of the areas in which its particular set of capabilities show the most promise. We will discuss the relationship to other key technologies and provide some helpful pointers.</p> <p>With Spark 2.0 and later versions, big improvements were implemented to make Spark easier to program and execute faster.</p> <h2>What Is Apache Spark?</h2> <p>Spark is a general-purpose distributed data processing engine that is suitable for use in a wide range of circumstances. On top of the Spark core data processing engine, there are libraries for SQL, machine learning, graph computation, and stream processing, which can be used together in an application. Programming languages supported by Spark include: Java, Python, Scala, and R. Application developers and data scientists incorporate Spark into their applications to rapidly query, analyze, and transform data at scale. Tasks most frequently associated with Spark include ETL and SQL batch jobs across large data sets, processing of streaming data from sensors, IoT, or financial systems, and machine learning tasks.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image7-1593817446675.jpg" alt=""></p> <h3>History</h3> <p>In order to understand Spark, it helps to understand its history. Before Spark, there was MapReduce, a resilient distributed processing framework, which enabled Google to index the exploding volume of content on the web, across large clusters of commodity servers. <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image6-1593757021829.png" alt=""></p> <p>There were 3 core concepts to the Google strategy:</p> <ol> <li> <p><strong>Distribute data:</strong> when a data file is uploaded into the cluster, it is split into chunks, called data blocks, and distributed amongst the data nodes and replicated across the cluster.</p> </li> <li> <p><strong>Distribute computation:</strong> users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs and a reduce function that merges all intermediate values associated with the same intermediate key. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines in the following way:</p> <ul> <li>The mapping process runs on each assigned data node, working only on its block of data from a distributed file.</li> <li>The results from the mapping processes are sent to the reducers in a process called "shuffle and sort": key/value pairs from the mappers are sorted by key, partitioned by the number of reducers, and then sent across the network and written to key sorted "sequence files" on the reducer nodes.</li> <li>The reducer process executes on its assigned node and works only on its subset of the data (its sequence file). The output from the reducer process is written to an output file.</li> </ul> </li> <li> <p><strong>Tolerate faults:</strong> both data and computation can tolerate failures by failing over to another node for data or processing.</p> </li> </ol> <h3>MapReduce word count execution example:</h3> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image4-1593757039111.png" alt=""></p> <p>Some iterative algorithms, like PageRank, which Google used to rank websites in their search engine results, require chaining multiple MapReduce jobs together, which causes a lot of reading and writing to disk. When multiple MapReduce jobs are chained together, for each MapReduce job, data is read from a distributed file block into a map process, written to and read from a SequenceFile in between, and then written to an output file from a reducer process.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image3-1593817522915.jpg" alt=""></p> <p>A year after Google published a <a href="http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf">white paper describing the MapReduce</a> framework (2004), Doug Cutting and Mike Cafarella created Apache Hadoop™.</p> <p>Apache Spark™ began life in 2009 as a project within the AMPLab at the University of California, Berkeley. Spark became an incubated project of the Apache Software Foundation in 2013, and it was promoted early in 2014 to become one of the Foundation’s top-level projects. Spark is currently one of the most active projects managed by the Foundation, and the community that has grown up around the project includes both prolific individual contributors and well-funded corporate backers, such as Databricks, IBM, and China’s Huawei.</p> <p>The goal of the Spark project was to keep the benefits of MapReduce’s scalable, distributed, fault-tolerant processing framework, while making it more efficient and easier to use. The advantages of Spark over MapReduce are:</p> <ul> <li>Spark executes much faster by caching data in memory across multiple parallel operations, whereas MapReduce involves more reading and writing from disk.</li> <li>Spark runs multi-threaded tasks inside of JVM processes, whereas MapReduce runs as heavier weight JVM processes. This gives Spark faster startup, better parallelism, and better CPU utilization.</li> <li>Spark provides a richer functional programming model than MapReduce.</li> <li>Spark is especially useful for parallel processing of distributed data with <strong>iterative</strong> algorithms.</li> </ul> <h2>How a Spark Application Runs on a Cluster</h2> <p>The diagram below shows a Spark application running on a cluster.</p> <ul> <li>A Spark application runs as independent processes, coordinated by the SparkSession object in the driver program.</li> <li>The resource or cluster manager assigns tasks to workers, one task per partition.</li> <li>A task applies its unit of work to the dataset in its partition and outputs a new partition dataset. Because iterative algorithms apply operations repeatedly to data, they benefit from caching datasets across iterations.</li> <li>Results are sent back to the driver application or can be saved to disk.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/image1-1593757064272.png" alt=""></p> <p>Spark supports the following resource/cluster managers:</p> <ul> <li><strong>Spark Standalone</strong> – a simple cluster manager included with Spark</li> <li><strong>Apache Mesos</strong> – a general cluster manager that can also run Hadoop applications</li> <li><strong>Apache Hadoop YARN</strong> – the resource manager in Hadoop 2</li> <li><strong>Kubernetes</strong> – an open source system for automating deployment, scaling, and management of containerized applications</li> </ul> <p>Spark also has a local mode, where the driver and executors run as threads on your computer instead of a cluster, which is useful for developing your applications from a personal computer.</p> <h2>What Does Spark Do?</h2> <p>Spark is capable of handling several petabytes of data at a time, distributed across a cluster of thousands of cooperating physical or virtual servers. It has an extensive set of developer libraries and APIs and supports languages such as Java, Python, R, and Scala; its flexibility makes it well-suited for a range of use cases. Spark is often used with distributed data stores such as <a href="https://www.hpe.com/us/en/software/data-fabric.html">HPE Ezmeral Data Fabric</a>, Hadoop’s HDFS, and Amazon’s S3, with popular NoSQL databases such as HPE Ezmeral Data Fabric, Apache HBase, Apache Cassandra, and MongoDB, and with distributed messaging stores such as HPE Ezmeral Data Fabric and Apache Kafka.</p> <p>Typical use cases include:</p> <p><strong>Stream processing:</strong> From log files to sensor data, application developers are increasingly having to cope with "streams" of data. This data arrives in a steady stream, often from multiple sources simultaneously. While it is certainly feasible to store these data streams on disk and analyze them retrospectively, it can sometimes be sensible or important to process and act upon the data as it arrives. Streams of data related to financial transactions, for example, can be processed in real time to identify– and refuse– potentially fraudulent transactions.</p> <p><strong>Machine learning:</strong> As data volumes grow, machine learning approaches become more feasible and increasingly accurate. Software can be trained to identify and act upon triggers within well-understood data sets before applying the same solutions to new and unknown data. Spark’s ability to store data in memory and rapidly run repeated queries makes it a good choice for training machine learning algorithms. Running broadly similar queries again and again, at scale, significantly reduces the time required to go through a set of possible solutions in order to find the most efficient algorithms.</p> <p><strong>Interactive analytics:</strong> Rather than running pre-defined queries to create static dashboards of sales or production line productivity or stock prices, business analysts and data scientists want to explore their data by asking a question, viewing the result, and then either altering the initial question slightly or drilling deeper into results. This interactive query process requires systems such as Spark that are able to respond and adapt quickly.</p> <p><strong>Data integration:</strong> Data produced by different systems across a business is rarely clean or consistent enough to simply and easily be combined for reporting or analysis. Extract, transform, and load (ETL) processes are often used to pull data from different systems, clean and standardize it, and then load it into a separate system for analysis. Spark (and Hadoop) are increasingly being used to reduce the cost and time required for this ETL process.</p> <h2>Who Uses Spark?</h2> <p>A wide range of technology vendors have been quick to support Spark, recognizing the opportunity to extend their existing big data products into areas where Spark delivers real value, such as interactive querying and machine learning. Well-known companies such as IBM and Huawei have invested significant sums in the technology, and a growing number of startups are building businesses that depend in whole or in part upon Spark. For example, in 2013 the Berkeley team responsible for creating Spark founded Databricks, which provides a hosted end-to-end data platform powered by Spark. The company is well-funded, having received $247 million across four rounds of investment in 2013, 2014, 2016 and 2017, and Databricks employees continue to play a prominent role in improving and extending the open source code of the Apache Spark project.</p> <p>The major Hadoop vendors, including MapR, Cloudera, and Hortonworks, have all moved to support YARN-based Spark alongside their existing products, and each vendor is working to add value for its customers. Elsewhere, IBM, Huawei, and others have all made significant investments in Apache Spark, integrating it into their own products and contributing enhancements and extensions back to the Apache project. Web-based companies, like Chinese search engine Baidu, e-commerce operation Taobao, and social networking company Tencent, all run Spark-based operations at scale, with Tencent’s 800 million active users reportedly generating over 700 TB of data per day for processing on a cluster of more than 8,000 compute nodes.</p> <p>In addition to those web-based giants, pharmaceutical company Novartis depends upon Spark to reduce the time required to get modeling data into the hands of researchers, while ensuring that ethical and contractual safeguards are maintained.</p> <h2>What Sets Spark Apart?</h2> <p>There are many reasons to choose Spark, but the following three are key:</p> <p><strong>Simplicity:</strong> Spark’s capabilities are accessible via a set of rich APIs, all designed specifically for interacting quickly and easily with data at scale. These APIs are well-documented and structured in a way that makes it straightforward for data scientists and application developers to quickly put Spark to work.</p> <p><strong>Speed:</strong> Spark is designed for speed, operating both in memory and on disk. Using Spark, a team from Databricks <a href="https://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html">tied for first place</a> with a team from the University of California, San Diego, in the 2014 Daytona GraySort benchmarking challenge (<a href="https://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html">https://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html</a>). The challenge involves processing a static data set; the Databricks team was able to process 100 terabytes of data stored on solid-state drives in just 23 minutes, and the previous winner took 72 minutes by using Hadoop and a different cluster configuration. Spark can perform even better when supporting interactive queries of data stored in memory. In those situations, there are claims that Spark can be 100 times faster than Hadoop’s MapReduce.</p> <p><strong>Support:</strong> Spark supports a range of programming languages, including Java, Python, R, and Scala. Spark includes support for tight integration with a number of leading storage solutions in the Hadoop ecosystem and beyond, including HPE Ezmeral Data Fabric (file system, database, and event store), Apache Hadoop (HDFS), Apache HBase, and Apache Cassandra. Furthermore, the Apache Spark community is large, active, and international. A growing set of commercial providers, including Databricks, IBM, and all of the main Hadoop vendors, deliver comprehensive support for Spark-based solutions.</p> <h3>The Power of Data Pipelines</h3> <p>Much of Spark's power lies in its ability to combine very different techniques and processes together into a single, coherent whole. Outside Spark, the discrete tasks of selecting data, transforming that data in various ways, and analyzing the transformed results might easily require a series of separate processing frameworks, such as Apache Oozie. Spark, on the other hand, offers the ability to combine these together, crossing boundaries between batch, streaming, and interactive workflows in ways that make the user more productive.</p> <p>Spark jobs perform multiple operations consecutively, in memory, and only spilling to disk when required by memory limitations. Spark simplifies the management of these disparate processes, offering an integrated whole – a data pipeline that is easier to configure, easier to run, and easier to maintain. In use cases such as ETL, these pipelines can become extremely rich and complex, combining large numbers of inputs and a wide range of processing steps into a unified whole that consistently delivers the desired result.</p>https://developer.hpe.com/2020-July-02/https://developer.hpe.com/2020-July-02/Thu, 02 Jul 2020 05:00:00 GMThttps://developer.hpe.com/meet-you-in-the-hack-shack/https://developer.hpe.com/meet-you-in-the-hack-shack/Tue, 30 Jun 2020 16:48:15 GMT<p>Prior to the pandemic, developers looked forward to connecting with one another at the HPE DEV Hack Shack at events like HPE Technology and Solutions Summit (TSS) and HPE Discover. Out of necessity for the health and safety of everyone, these events have gone virtual. While there’s some concern that this detracts from the personal touch offered at physical events, the option of going virtual has significant benefits. One major advantage is being able to reach a broader audience than just those folks who can afford to fly into town for an event.</p> <p>The Hack Shack, by design, is a place where people meet, collaborate, and learn from one another. It’s meant to be a place where innovative thinkers can share their ideas and engage in a bit of camaraderie. It’s also a place where event attendees come to relax and have a bit of fun, participating in competitive coding challenges and games to achieve high scores and win prizes. How does one make all this virtual?</p> <p>This was the challenge posed to our group of talented developers and designers. And though it’s impossible to say whether we’ve been successful or not with only a week of the virtual event under way, what I can say is that what we have been able to accomplish has been very well received. In addition, the work that we’ve done also puts us in a great position to offer a permanent online Hack Shack, complete with the ability to offer on-demand workshops and challenges, continuing the education of devs, data scientists, and designers around the world.</p> <p>During the first week of the HPE Discover Virtual Event, we hosted 10 sessions/workshops at the Hack Shack:</p> <ul> <li>Hack Shack Workshop W479: Introduction to the HPE Container Platform REST API</li> <li>Hack Shack Workshop W480: API 101 - API basics and the value they provide</li> <li>Hack Shack Workshop W481: Aruba API yourself!</li> <li>Hack Shack Workshop W482: Redfish API use with PowerShell, Python, &#x26; Bash/cURL</li> <li>Hack Shack Workshop W485: AI 101 - Convolutional neural network (CNN) for MNIST</li> <li>Hack Shack Workshop W486: Automate apps with the HPE Container Platform</li> <li>Hack Shack Session T491: What data scientists can learn from software developers</li> <li>Hack Shack Session T492: Accelerate innovation with DevOps for machine learning</li> <li>Making AI Real B329: A View From a Practitioner</li> <li>Hack Shack Session T490: Demystifying AI Technology Choices</li> </ul> <p>Each of the hands-on workshops included a presenter who ran the students through Jupyter notebooks covering the material and a couple of subject matter experts (SMEs) who helped facilitate the sessions and answer questions. The Jupyter Notebook format worked out really well. It provided the students with a fool-proof method to follow along with the instructor, step-by-step, as well as the opportunity to download material for later use.</p> <p>The workshops were capped for attendance so as to preserve a high-quality interactive experience for the students. The API 101 session proved to be very popular, as well as the one covering how to automate apps with the HPE Container Platform, where a lot of the participants remained thoroughly engaged to the very end and asked a lot of questions.</p> <p>Because the workshops are done virtually and you’re not physically in the room with the students, we knew it would be difficult to gauge how people felt during the sessions, i.e. if someone was confused or getting frustrated about something. We used a format where students could ask questions and receive help from the SMEs online to try and mitigate this. We also included a survey at the end of each workshop. Answers provided in the surveys will be used to help improve the workshops as we move forward.</p> <p>It was heartening to hear some of the feedback that the students provided:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure1-1593537773963.jpg" alt="hs blog figure1"></p> <p>While the workshops are great, there’s a whole lot more you can do at the <a href="/hackshack/">HPE Discover Virtual Event Hack Shack</a>. Starting Week 2, we will be offering coding challenges where you can show off your coding chops and compete with others for prizes. And we’ll have the Arcade open for those of you who just want a chance to go after the IT Monster in the Hack Shack Attack! game. You can even download stickers! One of the best ways to find out what’s happening in the Hack Shack is to scroll down on our new page and check out our little video tour <strong>This Week in the Hack Shack</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure2-1593537838340.jpg" alt="hs blog figure2"></p> <p>For those of you who are looking for the Hack Shack through the HPE Discover Virtual Event catalog, the way one views what sessions and workshops are available at the HPE Discover Virtual Event has changed since the days leading up to the event. In order to see the catalog and determine what sessions you wish to attend, you must first register. Don’t worry… it doesn’t cost anything. It’s just that, now that the event is running, things other than the catalog take center stage.</p> <p>When you <a href="https://www.hpe.com/us/en/discover.html">register</a>, make sure that the password you choose is <strong>ALPHA-NUMBERIC ONLY!</strong> Don’t try putting in any fancy or special characters. If you run into problems with the registration, there is online chat and the opportunity to email folks who can help. If you haven’t already signed up for them, there are a couple of ways to find the Hack Shack Workshops and Challenges. I’ve found two easy methods you can use to navigate to them.</p> <p><strong>Option 1</strong></p> <ol> <li>Click on this <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1043&#x26;locale=en_US">link</a></li> <li>This should bring you to a screen that looks like what’s shown below. This is where you log in:</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure3-1593537844172.jpg" alt="hs blog figure3"></p> <ol start="3"> <li>Once logged in, you will be brought to a page where you can navigate to the session catalog (Content Catalog). Click on this link.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure4-1593537855197.jpg" alt="hs blog figure4"></p> <ol start="4"> <li>Once you are in the catalog, search for <strong>Hack Shack</strong> to find Workshops and Challenges.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure5-1593537864210.jpg" alt="hs blog figure5"></p> <ol start="5"> <li>You can also use filters looking for Content Type to find the Hack Shack Workshops and Challenges.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure6-1593537873734.jpg" alt="hs blog figure6"></p> <p><strong>Option2</strong></p> <p>You can also find what Hack Shack Workshops and Challenges are available by navigating through the Hack Shack Lobby. Each week, the lobby schedule will be updated to show what sessions are available for registration. Simply click on the <a href="/hackshack">Hack Shack Lobby link</a> to follow the highlighted Steps 1, 2, and 3.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hs-blog-figure7-1593537881565.jpg" alt="hs blog figure7"></p> <p>We hope you’ll take advantage of attending the <a href="https://www.hpe.com/us/en/discover.html">HPE Discover Virtual Event</a>. It’s free, informative, and chock full of material you just can’t get anywhere else. And we hope you’ll continue to follow us at the <a href="https://developer.hpe.com/">HPE DEV site</a> where we’ll be opening up new platform pages, offering more tutorials and blog posts, and work to provide you with a continuing Hack Shack experience!</p>https://developer.hpe.com/get-involved-in-the-open-source-community-part-3-contributing-back-to-th/https://developer.hpe.com/get-involved-in-the-open-source-community-part-3-contributing-back-to-th/Wed, 24 Jun 2020 08:06:31 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/git-icon-1788c-1590702885345.png" alt="git101-part1-git icon 1788c"></p> <p>In the <a href="/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-gi">Part 1</a> of my blog series, I discussed how to get started with Git and leverage some of the content provided by the open source community. In <a href="/blog/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun">Part 2</a> of the series, I covered how to create and populate your first repo on GitHub. So far, we have covered use cases 1, 2 and 3 from the list below:</p> <ul> <li>Use case 1: I’d like to use something from the community</li> <li>Use case 2: I'd like to report an issue on a repository</li> <li>Use case 3: I'd like to share something with the community</li> <li>Use case 4: I'd like to contribute code to a repository</li> </ul> <p>In this article, I’m covering the last use case, where we will modify code from an existing repo and submit it back to its owner.</p> <h1>Use case 4: I'd like to contribute code to a repo</h1> <p>In use case 1, we saw that we could use code from the community using <code>git clone</code>. In use case 2, we reported an issue on a repo that we cloned and used. But what if you would like to contribute back to this project? For example, you might have discovered a bug that you have already fixed in your copy of the code and would like to share this fix. Or maybe you have enhanced a section of the code or fixed the documentation. Basically, anything that belongs to the project can be contributed back. In cases like this, the best approach is to <code>fork</code> the original repository into your own GitHub account (which we have created in use case 3) and work on this private copy. Let's see how this works.</p> <h2>Step 1: Forking a repo into your own GitHub account</h2> <p>In this lab, we would like to fork the <a href="https://github.com/Didier-Lalli/WelcomeGitDidier">https://github.com/Didier-Lalli/WelcomeGitDidier</a> repo in order to improve it. This step is done from the GUI of GitHub and requires that you own a GitHub account yourself and are logged into it.</p> <ol> <li>Make sure you are connected to your GitHub account</li> <li>Then, open <a href="https://github.com/Didier-Lalli/WelcomeGitDidier">https://github.com/Didier-Lalli/WelcomeGitDidier</a></li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/welcomegit-1592986241565.png" alt="git-part3-welcomegit"></p> <ol start="3"> <li>Click on the Fork button in the upper right corner</li> <li>After a while (depending on the size of the repo), you’ll be redirected to your own GitHub account. There, you will find your own copy of the WelcomeGitDidier repo, mentioning that it was forked from another source.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/forkedrepo-1592986346589.png" alt="git-part3-forkedrepo"></p> <h2>Step 2: Cloning the forked repo</h2> <p>In order to start making changes to this repo, we need to clone it locally like we did in <a href="/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-gi">Part1</a>.</p> <p>Open a terminal session (terminal on Mac, PowerShell on Windows).</p> <p>1/ Clone your copy of the WelcomeGitDidier</p> <pre><code class="language-bash">$ git clone https://github.com/&#x3C;YourGitHubUsername>/WelcomeGitDidier </code></pre> <p>For example, for me it’s:</p> <pre><code class="language-bash">$ git clone https://github.com/didou06/WelcomeGitDidier </code></pre> <p>2/ Check files part of the repo</p> <pre><code class="language-bash">$ cd WelcomeGitDidier $ ls -l total 16 -rw-r--r-- 1 lalli staff 20 May 25 18:11 README.md -rw-r--r-- 1 lalli staff 79 May 25 18:11 helloworld.py </code></pre> <p>3/ Check content of Python file</p> <pre><code class="language-bash">$ cat helloworld.py # This is part of lab 4 of our Git101 Jupyter Notebook print("Hello world !") </code></pre> <p>4/ Check status of repo</p> <pre><code class="language-bash">$ git status On branch master Your branch is up to date with 'origin/master'. nothing to commit, working tree clean </code></pre> <h2>Step 3: Making changes to the cloned repo</h2> <p>Let's say that we would like to make a contribution to the Python script. For example, we would like to change that "Hello world!" into "Hello World!" or, even better, add another line that prints your name or “hello world” in a different language. You decide, but please make sure you keep what was already there in the file. Just keep adding to it.</p> <p>1/ Edit helloworld.py with your favorite editor and add a line at the end of the file with something like:</p> <pre><code class="language-bash">print("Hello Didou06, many thanks for the contribution") </code></pre> <p>Save the file</p> <p>2/ Check status of repo</p> <pre><code class="language-bash">$ git status On branch master Your branch is up to date with 'origin/master'. Changes not staged for commit: (use "git add &#x3C;file>..." to update what will be committed) (use "git checkout -- &#x3C;file>..." to discard changes in working directory) modified: helloworld.py no changes added to commit (use "git add" and/or "git commit -a") </code></pre> <p>You can see that Git is telling us in the <code>git status</code> output that changes have been made to a file (helloworld.py) that have not been staged (nor committed) yet. We are going to commit those changes, but if you would like to cancel the changes you made and revert back to the original content of the clone repo (from master branch at origin), you can use <code>git checkout -f</code>.</p> <p>3/ Before you commit those changes, make sure the new code works fine. This is very important. You should not commit non-working code. We can run the Python code with:</p> <pre><code class="language-bash">$ python helloworld.py Hello world ! Hello Didou06, many thanks for the contribution </code></pre> <p>Now that we are happy about this new version of the code, let's add the file and commit the changes like we did in <a href="/blog/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun">Part 2</a> blog:</p> <p>4/ Commit changes locally using <code>git add</code> and <code>git commit</code> and verify that Git has picked up the change using <code>git status</code></p> <pre><code class="language-bash">$ git add helloworld.py $ git commit -m "Updated helloworld for Part 3 blog3" [master 0a7dbb0] Updated helloworld for Part 3 blog 1 file changed, 1 insertion(+) $ git status On branch master Your branch is ahead of 'origin/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working tree clean </code></pre> <h2>Step 4: Working in your own branch</h2> <p>At this point we are ready to <code>git push</code> our changes. But before we do this, let's discuss branches. Branches are a key concept that are at the heart of Git.</p> <blockquote> <p>Note: Mastering branches is beyond the scope of this lab, but you can read more about it <a href="https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell">here</a>.</p> </blockquote> <p>It is not considered good practice to push changes to the master branch, which is the default when first cloning a repo (note all the "On branch master" messages in the output from the git commands). Instead, developers would typically create a branch (in most cases corresponding to a theme that they have worked on, such as a new feature or a fix) and push the changes to a new branch. Git allows as many branches as you'd like. Git also provides the tooling to merge branches with the master branch when all validation checks have been successful.</p> <p>So, let's create a branch for this update called &#x3C;yourname>/AddedMyName.</p> <p>This next script will do the following:</p> <p>1/ Create a branch called &#x3C;yourname>/AddedMyName using <code>git branch</code></p> <pre><code class="language-bash">$ git branch didou06/AddedMyName </code></pre> <p>2/ Switch to using that new branch instead of the master using <code>git checkout</code></p> <pre><code class="language-bash">$ git checkout didou06/AddedMyName Switched to branch 'didou06/AddedMyName' </code></pre> <p>You can see that you are now operating from this new branch, but still within your local repo. It's now time to push your changes back to our remote repo using a <code>git push</code> command.</p> <p>3/ Push changes to your own remote repo using <code>git push</code>. You will be prompted for your GitHub username and password.</p> <pre><code class="language-bash">$ git push origin didou06/AddedMyName Username for 'https://github.com': didou06 Password for 'https://didou06@github.com': Enumerating objects: 5, done. Counting objects: 100% (5/5), done. Delta compression using up to 8 threads Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 413 bytes | 413.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: remote: Create a pull request for 'didou06/AddedMyName' on GitHub by visiting: remote: https://github.com/Didou06/WelcomeGitDidier/pull/new/didou06/AddedMyName remote: To https://github.com/Didou06/WelcomeGitDidier.git * [new branch] didou06/AddedMyName -> didou06/AddMyName </code></pre> <p>4/ Verify that Git has picked up the change using <code>git status</code></p> <pre><code class="language-bash">$ git status On branch didou06/AddedMyName nothing to commit, working tree clean </code></pre> <p>5/ Now, open GitHub and verify that you have a new branch (in addition to master) in the branches section of the repo page</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/branches-1592986802278.png" alt="git-part3-branches"></p> <p>You can drill down to the content of <strong>helloworld.py</strong> to verify that your changes are there.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/updatedpython-1592986856289.png" alt="git-part3-updatedpython"></p> <p>6/ At this stage, your only remote repo is WelcomeGitDidier on your own GitHub account. You can verify this using the command <code>git remote -v</code></p> <pre><code class="language-bash">$ git remote -v origin https://github.com/Didou06/WelcomeGitDidier.git (fetch) origin https://github.com/Didou06/WelcomeGitDidier.git (push) </code></pre> <p>But what would happen if the original repo that you forked was to change? It is possible that, if you forked something a while ago, things have evolved on the original master branch, and your branch is now out of synch. One way to fix this is to add the original repo as an additional remote repo to your local repo using git remote. This is a best practice and we usually name that remote repo: <strong>upstream</strong>.</p> <p>7/ Let's do this now using <code>git remote add</code> and check again using <code>git remote -v</code></p> <pre><code class="language-bash">$ git remote add upstream https://github.com/Didier-Lalli/WelcomeGitDidier $ git remote -v origin https://github.com/Didou06/WelcomeGitDidier.git (fetch) origin https://github.com/Didou06/WelcomeGitDidier.git (push) upstream https://github.com/Didier-Lalli/WelcomeGitDidier (fetch) upstream https://github.com/Didier-Lalli/WelcomeGitDidier (push) </code></pre> <p>8/ If you need to integrate some recent changes made upstream, you can do it using the <code>git merge</code> command.</p> <p>This git merge might generate merge conflicts as multiple changes may have been made on the same section of one of the files. Git will do its best to automatically merge the changes, but in impossible cases, it will tell you and you would have to fix those by hand (in a text editor) before committing changes to your repo. You can use <code>git status</code> to check if this is the case.</p> <pre><code class="language-bash">$ git merge upstream/master Already up to date. </code></pre> <p>This is not the case in our example.</p> <h2>Step 5: Opening a Pull Request</h2> <p>It's now time to contribute back these changes to the original DidierWelcomeGit repo. This is done by opening a so-called <strong>Pull Request</strong> (often abbreviated PR by developers). This action tells the owner of the repo that you forked (that's me) that you are proposing some changes and you are asking the owner to review and pull those changes (thus the Pull Request term) into the shared repo master copy. These pull requests are created from the GitHub web page.</p> <p>1/ In GitHub, list the branches available to identify the one that you have just created (&#x3C;yourname>/AddedMyName)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/branches-1592986802278.png" alt="git-part3-branches"></p> <p>2/ Use the <code>New pull request</code> button to open a new PR. Notice that this is opening a pull request on the original repo we forked earlier in the lab. Put a simple comment with your email, check that the changes you made are part of the pull request at the bottom of the page, and hit the Create pull request button.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/openingpr-1592986965058.png" alt="git-part3-openingpr"></p> <p>3/ You can now see your <strong>Pull Request</strong> (there might be other ones already there). This now requires approval from the owner and might lead to an exchange between the owner and the contributor, until he/she accepts the PR (or not).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/viewingpr-1592987054957.png" alt="git-part3-viewingpr"></p> <blockquote> <p>Note: You can continue to make updates to this branch (and by default this Pull Request) until it is accepted by the owner of the repo.</p> </blockquote> <p>This terminates use case 4: You should now be able to open a Pull Request to enhance a public repo. Congratulations. You are now a real open source contributor. Welcome to the community!</p> <h1>Where do I go from here?</h1> <p>We only touched the surface of Git in these three blog posts. We showed you some typical use cases in order to illustate the most important Git actions:</p> <ol> <li>Cloning an existing public repo (covered in <a href="/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-gi">Part 1</a>)</li> <li>Opening an issue on a public repo (covered in <a href="/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-gi">Part 1</a>)</li> <li>Creating your GitHub account and populating a first public repo (covered in <a href="/blog/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun">Part 2</a>)</li> <li>Forking a public repo and opening a pull request (covered here)</li> </ol> <p>If you want to discover more about Git, I recommend the following resources:</p> <ul> <li><a href="https://www.git-scm.com/book/en/v2">The Pro Git Book</a></li> <li><a href="https://guides.github.com/introduction/git-handbook/">The Git Handbook</a></li> <li><a href="https://www.youtube.com/watch?v=SWYqp7iY_Tc&#x26;feature=youtu.be">Video Git Crash Course</a></li> </ul> <p>Stay tuned to <a href="/blog">HPE DEV</a> for the last article where I will cover use case 4. In this use case, you will start contributing some code back to an existing project.</p>https://developer.hpe.com/master-the-automation-universe-the-easy-way-part-1-introduction-to-stack/https://developer.hpe.com/master-the-automation-universe-the-easy-way-part-1-introduction-to-stack/Mon, 22 Jun 2020 07:20:59 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/stackstorm-1592810807425.png" alt="stackstorm"></p> <p>In case you haven’t noticed, the world is obsessed with automation. You hear it every day in meetings with sales people, team members and in the latest blog entries about data centers. Inevitably, someone will mention the words “Rest API” and, to prove they really know what they are talking about, they quickly mention Salt, Chef, Puppet, and Ansible, just like a trusted advisor would. Talk about name-dropping!</p> <p>But not everyone is reading off the same script. The automation space is filled with a lot of choices when it comes to picking a solution and there are plenty of ways one can <em>automate</em> tasks. When I hear the word <em>automate</em>, I naturally think <em>remove humans from the process</em>. This means choosing a solution that can be <em>aware</em> of the environment, something that can listen and watch. Then, when a predefined event happens, automation can spring into action and do the heavy lifting for us humans.</p> <p>When we look at automation tools like Salt, Chef, Puppet, and Ansible, they all have a place in the automation world. These are tried and true industry solutions but, for the most part, need to be initiated by some sort of process, even if it’s simply logging in and manually kicking it off. StackStorm takes a different approach. StackStorm (st2) is an event-based automation framework that is often described as <em>If this, then that</em> automation. StackStorm can use sensors to monitor systems and listen for specific events. If an event happens, then a rule can be applied to run a single action or a complex set of actions called a <em>workflow</em>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/stackstorm-view-1592810817247.png" alt="stackstorm view"></p> <p>StackStorm has quite a few moving parts. The good news is you can start small, automating with a couple of <strong>actions</strong>, and later begin using <strong>sensors</strong> and <strong>rules</strong> as your understanding grows. Let’s take a closer look at actions. <strong>Actions</strong> are just scripts, pieces of code that can perform automation or remediation tasks. They are the workhorse of StackStorm. Actions can restart a service on a server, spin up a VM, or send an alert or notification, just to name a few possibilities. These examples are really not all that impressive, but hold on, ‘cause we’re just getting started.</p> <p><strong>Sensors</strong> are deployed to watch event/alarm queues. An alarm sensor can listen for specific alarms and recognize when the alarm is present. Next, the sensor will load a <strong>trigger</strong> that is assigned to a <strong>rule</strong>. When the trigger fires, the rule that is tied to that trigger runs the actions or workflows that are assigned to the rule. This could be something like sending a notification to pager duty, spin up or down a VM, or open an incident report in <strong>Service Now</strong>.</p> <p>My point here is that when an event happens, several things are put into motion with no human intervention. Sure, tools like Ansible, Chef, and Puppet have capabilities to take action and make changes, but what they are missing is the ability to automatically <em>know</em> when something happens.</p> <p>StackStorm sensors, actions, trigger, rules, and workflows are all provided together in StackStorm packs. If you have a st2 server, you can own all the pre-written automation required to integrate into something like <strong>Service Now</strong> simply by typing <code>st2 pack install servicenow</code>. With one command, you have installed all the automation software you need to fully integrate with <strong>Service Now</strong>.</p> <p>If you point your web browser to <a href="https://exchange.stackstorm.com">exchange.stackstorm.com</a>, you will discover over 170 StackStorm automation packs just waiting for you to consume. Azure, AWS, VMware… hundreds of actions just waiting for you to install and use to automate just about anything.</p> <p>I have been deep into StackStorm for over a year and a half now. I’ve developed st2 packs for HPE OneView, iLoAmplifier, HPE Composable Fabric, Aruba CX and Qumulo and I am just getting started. Want to learn more? Head on over to my <a href="https://github.com/xod442/stackstorm-tutorial">StackStorm tutorial</a> and you, too, can master the automation universe, if you’re into that sort of thing. Keep an eye out on the <a href="/blog">HPE DEV blog</a> site for more interesting articles and tutorials on automation.</p>https://developer.hpe.com/introducing-an-nfs-server-provisioner-for-the-hpe-csi-driver-for-kuberne/https://developer.hpe.com/introducing-an-nfs-server-provisioner-for-the-hpe-csi-driver-for-kuberne/Sat, 20 Jun 2020 21:04:47 GMT<p>With the release of HPE CSI Driver for Kubernetes 1.2.0, a new set of features has been made available as a technology preview, including an NFS Server Provisioner and a Pod Monitor. The motivation behind including these new features is explored in the official announcement – <a href="https://community.hpe.com/t5/around-the-storage-block/tech-preview-network-file-system-server-provisioner-for-hpe-csi/ba-p/7092948">Tech preview: Network File System Server Provisioner for HPE CSI Driver for Kubernetes</a>. In this blog post, I’ll demonstrate how to put these features to good use!</p> <blockquote> <p><strong>Note:</strong> The features and capabilities showcased within this blog are considered beta and subject to change. Do not use for production workloads until the official general availability release. Currently on target for version 1.3.0 of the HPE CSI Driver for Kubernetes.</p> </blockquote> <p>This tutorial assumes version 1.2.0 or later of the HPE CSI Driver for Kubernetes has been installed with a functioning backend, such as HPE Nimble Storage, HPE Primera or HPE 3PAR. The CSI driver is available as a <a href="https://hub.helm.sh/charts/hpe-storage/hpe-csi-driver">Helm chart</a> or <a href="https://operatorhub.io/operator/hpe-csi-operator">Operator</a>.</p> <h1>Background</h1> <p>The Container Storage Providers supported by the HPE CSI Driver are block storage solutions that serve volumes over either iSCSI or Fibre Channel. Inherently, traditional filesystems on these volumes are either XFS, ext3/4 or Btrfs. In other words, non-clustered filesystems that only allow a single host at time to access the volumes. With that limitation in mind, the HPE CSI Driver will only support <code>ReadWriteOnce</code> (RWO) <code>PersistentVolumeClaims</code> (PVCs) natively. In an effort to serve multiple <code>Pods</code> across multiple Kubernetes worker nodes, the user would have to be creative by either running Rook or the upstream NFS server provisioner to provide what is called <code>ReadWriteMany</code> (RWX) and <code>ReadOnlyMany</code> (ROX) access modes.</p> <p>In an effort to simplify deployment, Hewlett Packard Enterprise opted to create a solution that is seamless for the HPE CSI Driver users and administrators. More information about the design and the motivation behind it can be found in the <a href="https://community.hpe.com/t5/around-the-storage-block/tech-preview-network-file-system-server-provisioner-for-hpe-csi/ba-p/7092948">tech preview blog post</a>.</p> <h1>Enable the NFS Server Provisioner</h1> <p>Enabling the NFS Server Provisioner for PVCs is straightforward, as it’s controlled by the <code>StorageClass</code> option <code>nfsResources</code> parameter. What’s important to understand is that, once <code>nfsResources</code> have been enabled on a <code>StorageClass</code>, all PVCs are subject to be served by a NFS server setup by the NFS Server Provisioner, including RWO.</p> <blockquote> <p><strong>Note:</strong> All API objects declarations below assumes <code>kubectl create -f-</code>, paste the YAML stanza and hit CTRL-D on a new line after pasting.</p> </blockquote> <p>This is the shipping default <code>StorageClass</code> with <code>nfsResources</code> enabled.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: hpe-standard provisioner: csi.hpe.com parameters: csi.storage.k8s.io/controller-expand-secret-name: nimble-secret csi.storage.k8s.io/controller-expand-secret-namespace: kube-system csi.storage.k8s.io/controller-publish-secret-name: nimble-secret csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: nimble-secret csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: nimble-secret csi.storage.k8s.io/node-stage-secret-namespace: kube-system csi.storage.k8s.io/provisioner-secret-name: nimble-secret csi.storage.k8s.io/provisioner-secret-namespace: kube-system description: Volume created by the HPE CSI Driver for Kubernetes accessProtocol: iscsi csi.storage.k8s.io/fstype: xfs nfsResources: "true" reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>Creating an RWX claim is as simple as it can be.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-rwx-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 32Gi </code></pre> <p>It’s now possible to create a <code>Deployment</code> with multiple replicas to access the claim.</p> <pre><code class="language-yaml">--- apiVersion: apps/v1 kind: Deployment metadata: name: my-app labels: app: my-app spec: replicas: 5 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - image: datamattsson/my-app name: my-app volumeMounts: - name: my-app mountPath: /data volumes: - name: my-app persistentVolumeClaim: claimName: my-rwx-pvc </code></pre> <p>Once deployed and scaled, it should look like what you see below.</p> <pre><code class="language-bash">$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE my-app-6bfbb6f87f-5gr8k 1/1 Running 0 37s 10.45.0.2 tme-lnx-worker4 my-app-6bfbb6f87f-rzn64 1/1 Running 0 37s 10.36.0.2 tme-lnx-worker1 my-app-6bfbb6f87f-tk4vx 1/1 Running 0 37s 10.44.0.3 tme-lnx-worker2 my-app-6bfbb6f87f-vkqrc 1/1 Running 0 37s 10.44.0.2 tme-lnx-worker2 my-app-6bfbb6f87f-z87p2 1/1 Running 0 37s 10.44.0.1 tme-lnx-worker2 </code></pre> <blockquote> <p><strong>Pro tip:</strong> Creating a ROX claim requires the <code>Pod</code> to mount the claim read-only. Please check the documentation on the <a href="https://scod.hpedev.io/csi_driver/using.html#using_the_nfs_server_provisioner">HPE Storage Container Orchestrator Documentation</a> (SCOD) for more details.</p> </blockquote> <h1>Inspecting the actual NFS Deployment</h1> <p>While all parts of both the NFS client and server are being deployed transparently for the user, it is important to understand what actually end up running on the Kubernetes cluster. By default, each PVC creates a single replica <code>Deployment</code>, a <code>Service</code>, and a RWO PVC that maps to a supported backend. The NFS mount for the Pods accessing the claim is being taken care of by the HPE CSI Driver.</p> <p>Where the NFS servers gets deployed is controlled by the <code>nfsNamespace</code> parameter. The default is "hpe-nfs".</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/csi-120-rev1b-1592687082644.png" alt=""></p> <p>Object <a href="https://scod.hpedev.io/csi_driver/diagnostics.html#nfs_server_provisioner_resources">naming conventions and other diagnostics</a> are available on SCOD.</p> <h1>User control of the NFS server</h1> <p>Sometimes it’s desired to have a single default <code>StorageClass</code> on the cluster for all access mode needs. This is possible but requires non-portable PVCs and a tweak to the <code>StorageClass</code> using <code>allowOverrides</code> and omitting <code>nfsResources</code>.</p> <p>Another important detail is that the NFS server needs to be deployed in the same Namespace as the requesting claim to allow users to create CSI snapshots and clones.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: hpe-standard provisioner: csi.hpe.com parameters: csi.storage.k8s.io/controller-expand-secret-name: nimble-secret csi.storage.k8s.io/controller-expand-secret-namespace: kube-system csi.storage.k8s.io/controller-publish-secret-name: nimble-secret csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: nimble-secret csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: nimble-secret csi.storage.k8s.io/node-stage-secret-namespace: kube-system csi.storage.k8s.io/provisioner-secret-name: nimble-secret csi.storage.k8s.io/provisioner-secret-namespace: kube-system description: Volume created by the HPE CSI Driver for Kubernetes accessProtocol: iscsi csi.storage.k8s.io/fstype: xfs allowOverrides: nfsResources,nfsNamespace reclaimPolicy: Delete allowVolumeExpansion: true </code></pre> <p>By annotating the PVC, a user can request the NFS Server Provisioner to serve the claim. The user can also deploy the server in the <code>Namespace</code> requesting the claim.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-other-rwx-pvc annotations: csi.hpe.com/nfsResources: "true" csi.hpe.com/nfsNamespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 32Gi </code></pre> <p>Inspecting the <code>Namespace</code> where the claim was created, we may observe the API objects that were created.</p> <pre><code class="language-bash">$ kubectl get configmap,deploy,pvc,service -o name configmap/hpe-nfs-config deployment.apps/hpe-nfs-053b6374-db9c-46c9-94d9-d3c3e59a55e4 persistentvolumeclaim/hpe-nfs-053b6374-db9c-46c9-94d9-d3c3e59a55e4 persistentvolumeclaim/my-other-rwx-pvc service/hpe-nfs-053b6374-db9c-46c9-94d9-d3c3e59a55e4 </code></pre> <p>The user may now use the requesting claim as a <code>dataSource</code> in a new claim to clone it. For a comprehensive tutorial on how to use CSI snapshots and clones, check out this previous blog post: <a href="/blog/PklOy39w8NtX6M2RvAxW/hpe-csi-driver-for-kubernetes-snapshots-clones-and-volume-expansion">HPE CSI Driver for Kubernetes: Snapshots, Clones and Volume Expansion</a></p> <h1>Advanced configuration</h1> <p>The tech preview is currently hardcoded to allow twenty running NFS servers per worker node in the cluster. Request limits are also unrestricted. It’s encouraged to tinker with the request limits during the tech preview as the engineering team is looking for real world guidance. In the GA release, the request limits will be not be unrestricted. We know that the NFS server has around 150MiB memory foot print coming up cold and it all comes around to how much buffer cache you want to put aside for servicing cached read requests. We also know the NFS server will consume quite a lot of CPU cycles during load tests.</p> <p>These are the <code>StorageClass</code> parameters that control the request limits:</p> <ul> <li><strong>nfsResourceLimitsCpuM</strong>: Specify CPU limits for the server <code>Deployment</code> in milli CPU. Default: no limits applied. Example: "500m"</li> <li><strong>nfsResourceLimitsMemoryMi</strong>: Specify memory limits (in megabytes) for the server <code>Deployment</code>. Default: no limits applied. Example: "500Mi"</li> </ul> <p>It’s also possible to fine tune the NFS server itself. The <code>ConfigMap</code> "hpe-nfs-config" in the <code>Namespace</code> where the server is deployed represents the running server configuration. Samples can be found in the <a href="https://github.com/nfs-ganesha/nfs-ganesha/tree/master/src/config_samples">NFS-Ganesha GitHub</a> repo.</p> <p>The NFS client mount options are also tunable, this could be useful for making tweaks to fit a certain best practice for running a particular application over NFS. Do note that NFSv4 is the only version supported at this time.</p> <ul> <li><strong>nfsMountOptions</strong>: Customize NFS mount options for the <code>Pods</code> to the server <code>Deployment</code>. Default: "nolock, hard,vers=4"</li> </ul> <h1>Automatic recovery with the Pod Monitor</h1> <p>In an effort to ensure the NFS servers are kept alive, HPE introduced a customer facing feature called Pod Monitor. It monitors <code>Pods</code> on the cluster with the label <code>monitored-by: hpe-csi</code>. It checks the status of the <code>Pod</code> on 30 second intervals (tunable) and watches for the <code>NodeLost</code> transition. It also verifies that the node is indeed unreachable and effectively deletes the <code>Pod</code> and <code>VolumeAttachments</code> to let Kubernetes reschedule the <code>Pod</code> on a healthy node.</p> <p>The Pod Monitor is necessary because the defaults in Kubernetes used to perform automatic recovery for node outages are too extreme and would stall workloads running over 10 minutes to recover. And, there would still be problems with the <code>VolumeAttachment</code>, as CSI won’t forcefully remove it because, for all it knows, the volume is still mounted on the node that became isolated. As the CSPs supported by the HPE CSI Driver strip the initiator groups from a volume before applying the new ones, split brain would never happen. There might be dirty buffers, but modern filesystems recover gracefully and if there’s an application level corruption, there should be backups or at least snapshots of the volume to recover from.</p> <p>Read more about the <a href="https://scod.hpedev.io/csi_driver/monitor.html">Pod Monitor</a> on SCOD as it’s possible to apply it to any workload backed by a HPE CSI Driver volume.</p> <h1>Summary</h1> <p>Please take the NFS Server Provisioner and Pod Monitor for a spin if you get the chance. We value your feedback and we keep all channels of communication open for this purpose. If you’re an HPE employee, join our Slack community at <a href="https://hpedev.slack.com">hpedev.slack.com</a>. If you’re not an employee, sign up at <a href="https://slack.hpedev.io">slack.hpedev.io</a> first. It’s also possible to <a href="https://github.com/hpe-storage/csi-driver/issues/new?title=Feedback%20on%20RWX%20functionality%20in%201.2.0">report issues through GitHub</a>. Beta feedback, questions and concerns may also be raised through a regular support ticket. Please check with your HPE representative of the respective storage backend on how to log a support case.</p> <ul> <li>HPE CSI Driver for Kubernetes <a href="https://hub.helm.sh/charts/hpe-storage/hpe-csi-driver">Helm Chart</a></li> <li>HPE CSI Operator on <a href="https://operatorhub.io/operator/hpe-csi-operator">OperatorHub.io</a></li> <li>Read the Tech Preview announcement blog COMING SOON!</li> <li>Visit <a href="https://scod.hpedev.io">HPE Storage Container Orchestrator Documentation</a> (SCOD)</li> <li>HPE CSI Driver <a href="https://github.com/hpe-storage/csi-driver">source code</a> on GitHub</li> <li>Browse the <a href="https://developer.hpe.com/api/hpe-nimble-csp/">CSP specification</a> if you want to include your platform with the HPE CSI Driver</li> </ul> <p>Until my next blog post, happy containerizing!</p>https://developer.hpe.com/all-hpe-composable-ecosystem-sdks-now-support-oneview-52-automation/https://developer.hpe.com/all-hpe-composable-ecosystem-sdks-now-support-oneview-52-automation/Wed, 17 Jun 2020 18:50:49 GMT<p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView 5.2</a> (REST API version 1600) automation features are now supported by all of the HPE OneView SDKs. HPE’s Composable Ecosystem actively supports <a href="https://github.com/HewlettPackard/oneview-ansible">Ansible</a>, <a href="https://github.com/HewlettPackard/oneview-python">Python</a>, <a href="https://github.com/HewlettPackard/oneview-golang">Golang</a>, <a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.3.0">Terraform</a>, <a href="https://github.com/HewlettPackard/oneview-chef">Chef</a>, <a href="https://github.com/HewlettPackard/oneview-puppet">Puppet</a>, <a href="https://github.com/HewlettPackard/POSH-HPOneView">PowerShell</a> and <a href="https://github.com/HewlettPackard/oneview-sdk-ruby">Ruby</a> SDKs. Using the unified HPE OneView API and these popular tools, IT administrators can deploy and update servers, storage, and networking simultaneously, using only a single line of code. Composing new infrastructure is now not only faster and more agile, but also more predictable.</p> <p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> uses software-defined intelligence in a template-driven approach to automate the deployment, provisioning, updating, and integration of resources, such as compute, storage, and networking infrastructure. Designed with a modern, standards-based API, HPE OneView gives IT organizations the ability to connect their software-defined infrastructure from core to cloud within a diverse partner ecosystem. IT organizations can leverage this partner ecosystem to integrate HPE OneView within their existing management frameworks for their preferred platforms.</p> <p>HPE offers SDKs for industry-leading software deployment, provisioning, and configuration management tools, including Ansible, Terraform, Chef and Puppet. These SDKs can be used to configure HPE OneView managed infrastructure resources. IT organizations can use these SDKs to streamline the task of configuring and maintaining a company's servers. The SDKs allow for integration with cloud-based platforms to automatically provision and configure new machines, enabling administrators to create a resource topology similar to that of a public cloud on their own physical infrastructure.</p> <p>Developers can programmatically control HPE OneView managed resources using infrastructure-as-code for physical compute, storage, and fabric resources. Infrastructure-as-code enables complete datacenter automation, consistent reproducibility, versioning, and roll back.</p> <p>The HPE OneView SDKs also provide API language support for Python, Ruby, Golang and PowerShell languages, enabling developers to easily build integrations and scalable solutions with HPE OneView. With language support, you can integrate popular automation tools based on these languages with HPE OneView.</p> <p>For more information:</p> <ul> <li>[HPE DEV OneView] (<a href="https://developer.hpe.com/platform/hpe-oneview/home">https://developer.hpe.com/platform/hpe-oneview/home</a>)</li> <li><a href="https://github.com/HewlettPackard/oneview-ansible">Ansible GitHub release</a></li> <li>[Terraform GitHub Release] (<a href="https://github.com/HewlettPackard/terraform-provider-oneview">https://github.com/HewlettPackard/terraform-provider-oneview</a>)</li> <li>[Chef GitHub Release] (<a href="https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.4.0">https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.4.0</a>)</li> <li>[Puppet GitHub Release] (<a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.6.0">https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.6.0</a>)</li> <li>[Python GitHub Release] (<a href="https://github.com/HewlettPackard/oneview-python/releases/tag/v5.2.0">https://github.com/HewlettPackard/oneview-python/releases/tag/v5.2.0</a>)</li> <li>[Golang GitHub Release] (<a href="https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.4.0">https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.4.0</a>)</li> <li>[Ruby GitHub release] (<a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.13.0">https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.13.0</a>)</li> <li>[PowerShell] (<a href="https://github.com/HewlettPackard/POSH-HPOneView">https://github.com/HewlettPackard/POSH-HPOneView</a>)</li> </ul>https://developer.hpe.com/hpe-achieves-gold-for-large-scale-enterprise-kubernetes-deployments/https://developer.hpe.com/hpe-achieves-gold-for-large-scale-enterprise-kubernetes-deployments/Wed, 17 Jun 2020 15:29:22 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/cloudnativelogo-1593006822299.png" alt="cloudnativelogo"></p> <p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Hewlett Packard Enterprise (HPE) is now a Gold member of the <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a> (CNCF). Over the past several months, HPE has made significant contributions to the open source community. These contributions are instrumental in helping enterprises modernize their applications across hybrid cloud, multi-cloud, and edge environments. As a Gold member, HPE emphasizes its continued dedication to promote open source projects.</p> <p>CNCF hosts critical components of the global technology infrastructure, serving as the vendor-neutral home for many of the fastest-growing open source projects, including Kubernetes, Prometheus, and Envoy. Its goal is to build sustainable ecosystems for cloud-native software. HPE recognizes the crucial role CNCF plays in bringing together the world’s top developers, end users, and vendors and has been an active member of CNCF since 2017.</p> <p>At HPE, our focus is on more than just next generation cloud-native applications. We are dedicated to delivering breakthrough innovation and a new approach to helping enterprises deploy containerized environments at scale for both cloud-native and non-cloud-native applications. Our contributions range from delivering products and solutions for cloud-native storage and Kubernetes to providing advisory and professional support and training services focused on designing and implementing Kubernetes and cloud-native technologies. CNCF recently awarded HPE the Kubernetes <a href="https://landscape.cncf.io/format=card-mode&#x26;organization=hewlett-packard-enterprise&#x26;selected=hpe-kcsp">Certified Service Provider</a> (KCSP) designation, as a pre-qualified service provider with deep experience in helping enterprises successfully adopt and deploy Kubernetes.</p> <p>HPE has actively collaborated with the open source community for many years, providing expertise and deep contributions to numerous open source projects. Our commitment to fostering innovation with open source Kubernetes and the container ecosystem has been accelerated through our acquisitions of <a href="https://www.hpe.com/us/en/newsroom/press-release/2019/05/hewlett-packard-enterprise-integrates-bluedata-to-accelerate-ai-and-data-driven-innovation-in-the-enterprise.html">BlueData</a> and <a href="https://www.hpe.com/us/en/newsroom/press-release/2019/08/hpe-advances-its-intelligent-data-platform-with-acquisition-of-mapr-business-assets.html">MapR</a>. Building on these recent software acquisitions, HPE recently introduced our ground-breaking new container management solution: <a href="https://www.hpe.com/us/en/solutions/container-platform.html">the HPE Ezmeral Container Platform</a>.</p> <img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hpe-kubernetescertified-1592924398917.png" style="float: left; margin-right: 10px;"> <p>The HPE Ezmeral Container Platform is an integrated turnkey solution with BlueData software as the multi-cluster container management control plane and the MapR distributed file system as the unified <a href="https://www.hpe.com/info/data-fabric">data fabric</a> for persistent storage. HPE Ezmeral Container Platform provides the ability to deploy and manage multiple versions of 100% open source Kubernetes and was recently certified by CNCF as a <a href="https://landscape.cncf.io/selected=hpe-container-platform">Kubernetes Certified Distribution</a> — ensuring conformance, interoperability, consistency, and access to the latest open source Kubernetes features.</p> <p>This new platform uniquely addresses the requirements for large-scale enterprise Kubernetes deployments across a wide range of use cases, from machine learning and edge analytics to CI/CD pipelines and application modernization. IT teams can use it to manage multiple Kubernetes clusters with multi-tenant container isolation and pre-integrated persistent storage. It includes innovations such as <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1043&#x26;sid=20267_0&#x26;locale=en_US">KubeDirector</a>, an open source project from HPE that enables organizations to run non-cloud-native stateful applications on Kubernetes without modifying code. The HPE Ezmeral Container Platform also provides developers with secure, on-demand access to their environments. This access lets them develop apps and release code faster, while providing the container portability benefits of build once and deploy anywhere.</p> <p>More recently, <a href="https://www.hpe.com/us/en/newsroom/blog-post/2020/02/hpe-acquires-scytale-to-advance-open-secure-edge-to-cloud-strategy.html">HPE acquired Scytale</a>, a founding contributor to CNCF’s <a href="https://github.com/spiffe/spiffe">SPIFFE</a> (the Secure Production Identity Framework for Everyone) and SPIRE (the SPIFFE Runtime Environment) open source projects. The SPIFFE and SPIRE projects deliver a foundational capability—service identity authentication—for cloud- and container-deployed micro-services. The projects have rapidly grown in popularity, receiving contributions from Google, Pinterest, Square, Uber, and others.</p> <p>The open source SPIFFE and SPIRE projects enable organizations to deliver secure, high performance, multi-cloud, and multi-tenant IT infrastructure more quickly to employees, customers, and partners. In fact, the CNCF Technical Oversight Committee (TOC) just <a href="https://www.cncf.io/blog/2020/06/22/toc-approves-spiffe-and-spire-to-incubation/">promoted them from sandbox to incubation-level hosted projects</a>. Learn more in this new blog post <a href="https://community.hpe.com/t5/shifting-to-software-defined/bringing-trusted-computing-to-the-cloud/ba-p/7092622#.XvDMTJpKiM8">Bringing Trusted Computing to the Cloud</a> from my colleague Sunil James, founder and CEO of Scytale, now at HPE.</p> <p>HPE looks forward to taking on new responsibilities and doubling down on our contributions within the CNCF as a Gold member. Working with CNCF, our goal is to accelerate the creation and adoption of open, standards-based data access; advance new capabilities for cloud-native applications and environments; and promote new technologies and solutions that support them.</p> <p><a href="https://www.cncf.io/announcement/2020/06/22/cloud-native-computing-foundation-announces-hewlett-packard-enterprise-as-gold-member/">Read the CNFC press release announcing HPE as a Gold member</a>. You can also check out more information on <a href="https://www.hpe.com/info/container-platform">HPE Container Platform</a> and HPE’s <a href="https://developer.hpe.com/projects">open source</a> initiatives. We also recommend you keep an eye out on the <a href="/blog">HPE DEV blog</a> site for more interesting articles on containers, Kubernetes, and other open source offerings.</p>https://developer.hpe.com/hpe-greenlake-for-private-cloud/https://developer.hpe.com/hpe-greenlake-for-private-cloud/Tue, 16 Jun 2020 10:51:11 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/blog-greenlake-intro-1200p-1592501708480.jpg" alt="blog greenlake intro 1200p"></p> <p>The complexity of today’s IT brings many challenges. IT organizations find themselves dealing with clustered applications, multiple data centers, public, private, and hybrid clouds, and applications with complex dependencies. Deploying and operating enterprise workloads across these varied environments is often a time-consuming, manual process, which is both inefficient and often fraught with errors.</p> <p>You need a solution and approach that can orchestrate and automate your processes and ensure that all tasks happen efficiently and in the proper order. HPE GreenLake provides a solution for private clouds that enables savings in both time and money, elimination of errors, and construction of more predictable and reliable services.</p> <h3>The all-encompassing cloud</h3> <p>In the early days of cloud computing, private clouds promised the scalability, elasticity, and manageability of public clouds, combined with the security and control within on-premises datacenter environments. Providing a private cloud turned out to be harder than everyone expected. Some of the earliest private cloud implementations didn’t offer the same scalability, elasticity, and resilience characteristic of cloud environments. Fortunately, over time, vendor offerings improved.</p> <p>While private clouds still retain their importance, they are now necessarily part of a broader discussion. The role of a private cloud is looked at more in terms of how it is implemented within a hybrid cloud strategy, taking into account the application it serves and the desired outcome. For instance, depending on the application, it could be cheaper to run a workload in a private cloud versus a public cloud (or vice versa).</p> <p>Managing private cloud environments can be quite arduous, involving many repetitive tasks. These tasks might include sizing, provisioning and configuring resources like virtual machines (VMs), establishing VM clusters and load balancing, creating storage logical unit numbers (LUNs), invoking virtual networks, making the actual deployment and then monitoring and managing availability and performance. Although each of these processes is effective, they are inefficient and can lead to errors. These errors then require troubleshooting, which delays the workload's availability. They may also expose security vulnerabilities that can put the enterprise at risk.</p> <p>With complete automation across the cloud environment, an organization eliminates these repetitive and manual processes for workload deployment and management. To achieve cloud automation, an IT team needs to use orchestration and automation tools that run on top of their virtualized environment. Orchestration enables an administrator to codify the various steps and processes involved with workload deployment and management, while automation invokes those steps without human intervention.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/greenlake-central-blueprint-image-1592499731966.png" alt="greenlake central blueprint image"></p> <p>The <a href="https://www.hpe.com/psnow/doc/a50003040enw?jumpid=in_lit-psnow-red">HPE GreenLake for private cloud technical whitepaper</a> describes the application blueprint capabilities inherent with HPE GreenLake that provides this automation for private clouds. The solution enables savings in both time and money, elimination of errors, and construction of more predictable and reliable services beyond bare virtual machines.</p>https://developer.hpe.com/create-agile-infrastructure-with-hpe-oneview-api-based-automation/https://developer.hpe.com/create-agile-infrastructure-with-hpe-oneview-api-based-automation/Mon, 15 Jun 2020 19:23:25 GMT<p>The HPE OneView Python SDK v5.2 is now available, providing support for HPE OneView 5.2 (REST API version 1600) and Image Streamer 5.2 (API 1600). This release leverages new Python standards, including refactored base classes and the introduction of mixed classes, and now supports 30 HPE OneView managed resources.</p> <p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> takes a software-defined, programmatic approach to enable agile management of infrastructure with efficient workflow automation, a modern RESTful API, and a comprehensive partner ecosystem. By automating the provisioning of physical infrastructure on-demand, using software-defined templates from HPE OneView, this integration allows administrators to create a resource topology and user experience similar to that of a public cloud on their own physical infrastructure.</p> <p>The HPE OneView Python SDK allows developers who use the Python language to programmatically control HPE OneView managed resources using infrastructure-as-code for physical compute, storage, and fabric resources. Infrastructure-as-code enables complete datacenter automation, significantly increasing reliability, compliance, and deployment flexibility. HPE has provided a wide range of <a href="https://github.com/HewlettPackard/oneview-python/blob/master/examples/README.md">examples</a> to help you use the OneView API and test functionality.</p> <p>For more information:</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-python">GitHub Release</a></li> <li><a href="https://github.com/HewlettPackard/oneview-python/releases/tag/v5.2.0">Release notes</a></li> </ul>https://developer.hpe.com/hpe-oneview-ansible-module-v560-now-available/https://developer.hpe.com/hpe-oneview-ansible-module-v560-now-available/Mon, 15 Jun 2020 19:09:06 GMT<p>HPE is pleased to announce the availability of the HPE OneView Ansible Module v5.6.0. This module provides integration of HPE OneView with Ansible by Red Hat®, an industry-leading software deployment, provisioning, and configuration management tool. This new module supports HPE OneView 5.2 (REST API version 1600). It also provides support for Image Streamer 5.2 (API 1600).</p> <p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> uses software-defined intelligence in a template-driven approach to deploy, provision, update, and integrate resources, such as compute, storage, and networking infrastructure. Designed with a modern, standards-based API, HPE OneView gives IT organizations the ability to connect their software-defined infrastructure from core to cloud by provisioning turnkey private cloud infrastructure within a diverse partner ecosystem. IT organizations can leverage the partner ecosystem to integrate HPE OneView within their existing management frameworks for their preferred platforms. These capabilities allow teams to deliver projects consistently while meeting desired outcomes for key stakeholders.</p> <p>The HPE OneView Ansible module enables the transformation of on-premises infrastructure through the software-defined infrastructure automated provisioning of bare-metal resources, including servers, storage, and networking. It does so as part of an application deployment process. Using Ansible with HPE OneView allows customers to create a flexible and adaptive infrastructure that is essential to addressing the need for organizational agility and supporting initiatives that accelerate the delivery of customer and business value.</p> <p>For more information:</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.6.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-ansible/blob/master/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/oneview-ansible-samples/blob/master/infrastructure-as-code/infrastructure-as-code.md">Whitepaper: Infrastructure as code with HPE OneView and Ansible by Red Hat</a></li> <li><a href="https://www.hpe.com/h22228/video-gallery/us/en/700000796/EN/US/7f333bd5-49f8-4a91-9891-a66554ea402c/demo-video-ansible-integration-with-hpe-oneview/video?lang=en-US">Demonstration Video</a></li> </ul>https://developer.hpe.com/dockerizing-your-nodejs-based-backend-applications/https://developer.hpe.com/dockerizing-your-nodejs-based-backend-applications/Mon, 15 Jun 2020 14:51:00 GMT<p>Hello developers! In this blog post, I will take you through a step-by-step procedure that uses best practices to dockerize your NodeJS backend applications. To better understand this tutorial, the reader should have some basic familiarity with Docker as-a-container technology and NodeJS-based backend applications. Briefly, NodeJS is the most popular JavaScript-based server side ecosystem. It is mostly used to build highly scalable, fault tolerant microservices-based backend enterprise applications. Many enterprise application developers prefer NodeJS over other programming languages, such as JAVA, because of its light memory footprint, event driven, non-blocking model. NodeJS is also very popular in building end-to-end applications in JavaScript, typically using MEAN (Mongo, Express, Angular, Node) and MERN (Mongo, Express, React, Node) stacks. In this post, I do not intend to focus on NodeJS-based application development, but rather on how to dockerize already developed NodeJS based applications.</p> <p>Let’s first try to build a sample web server using express, which is a NodeJS package. Express is a node package typically used for building REST APIs in server side JavaScript. Below are the steps we are going to perform in this article:</p> <ul> <li>Create and Initialize a sample NodeJS based backend application using express package</li> <li>Dockerize the NodeJS application( i.e creating Dockerfile for application)</li> <li>Build a docker image out of Dockerfile</li> <li>Run a built image as a docker container and access the NodeJS backend application in web client typically web browser</li> <li>Push the finalized docker image on docker hub or any private docker registry using docker CLI.</li> </ul> <blockquote> <p>Pre-requisite: you must have latest version of Node and Docker installed on your machine . ( Supported platform are Linux/windows/MAC). Throughout this article , i will be using docker for windows 10 using Hyper-V hypervisor . Follow the official documentation to install Node and Docker on your platform.</p> </blockquote> <p><strong>STEP 1: Create and initialize a sample NodeJS based application</strong></p> <p>Create sample project directory as below:</p> <pre><code class="language-bash">mkdir sample_node_app </code></pre> <p>Initialize the NodeJS project using node package manager (npm). It will generate a project skeleton with a package.json file (also called project descriptor) which carries all the meta data and dependencies information for your application.</p> <pre><code class="language-bash">cd sample_node_app &#x26;&#x26; npm init </code></pre> <p>Once the project skeleton is ready, edit the "scripts" section of the package.json file as shown below:</p> <pre><code class="language-bash">cat package.json { "name": "sampleapp", "version": "1.0.0", "description": "This is sample Node Project", "main": "index.js", "scripts": { "start": "node index.js" }, "author": "rahul kumar", "license": "ISC" } </code></pre> <p>In order to build a web server using NodeJS, install the express package using npm as below:</p> <pre><code class="language-bash">npm install express --save </code></pre> <p>By this time, your package.json should look as what’s shown below:</p> <pre><code class="language-bash">cat package.json { "name": "sampleapp", "version": "1.0.0", "description": "This is sample Node Project", "main": "index.js", "scripts": { "start": "node index.js" }, "author": "rahul kumar", "license": "ISC", "dependencies": { "express": "^4.17.1" } } </code></pre> <p>Once we have the project skeleton ready, let’s add express code to build a simple web server that listens on a certain port, for example, port 4000.</p> <pre><code class="language-bash">cat index.js const express = require('express'); const application=new express(); const PORT=4000; application.get('/',(req,resp)=>{ resp.send(`Congrats ! Your Node Express server is running on PORT ${PORT}`); }); application.listen(PORT,()=>{ console.log(`Node express server is running on ${PORT}. Enjoy NodeJS`) }); </code></pre> <p>Now that we have the Node express server ready, let’s run it!</p> <pre><code class="language-bash"> npm start </code></pre> <p>Once you have started the web server using the above command, you will see the following on the console:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/node-1592239169346.PNG" alt=""></p> <p>Then, you can access the application on <a href="http://localhost:4000">http://localhost:4000</a> in a browser.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/browser-1592239218657.PNG" alt=""></p> <p>Okay! Now we have the simplest possible NodeJS backend application running on port 4000.</p> <p><strong>STEP 2: Dockerize the application</strong></p> <p>Now, let’s create a Dockerfile in the project directory.</p> <pre><code class="language-bash">cat Dockerfile # Base image used FROM ALPINE # Installing project dependencies RUN npm install # Running default command CMD ["npm", "start"] </code></pre> <p>Simple enough. So, now we have the smallest possible Dockerfile. Let’s move on to the next step .</p> <p><strong>STEP 3: Building the docker image</strong></p> <p>Navigate to the directory where your Dockerfile is present, also known as the “build context” for your Dockerfile. Note that you can change the name of the docker file to what you desire. If you do, you need to use the –f option in the docker CLI while building the docker image. By default, the docker CLI looks for a file named Dockerfile in the build context.</p> <pre><code class="language-bash">docker build . Sending build context to Docker daemon 2.01MB Step 1/3 : FROM alpine latest: Pulling from library/alpine df20fa9351a1: Pull complete Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 Status: Downloaded newer image for alpine:latest ---> a24bb4013296 Step 2/3 : RUN npm install ---> Running in 0b3e3ae93e9c /bin/sh: npm: not found The command '/bin/sh -c npm install' returned a non-zero code: 127 </code></pre> <p>We have found a problem: We need a Docker base image that has node and npm already installed.</p> <p>Problem resolution: Why we are getting this error? To better understand this, you need to understand on what basis to select the base images in your custom (or project specific) Dockerfile. The rule of thumb is that you should choose your base image based on what programs you need to build your custom image. In this case, npm is the package manager for node and it is required for installing dependencies that your project needs inside the container. In Docker world, alpine images are very small in size (just a few megabytes) and the alpine image we selected does not seem to have npm in it.</p> <p>One way to overcome this problem is to use some other base image that has node and npm installed. The best option in this case would be to use the official base image found on the Docker Hub public registry. On Docker Hub, if we see a full-fledged Node official image tag, its size is probably on the order of 300-400 MB. But we don't need that large a sized Node image. Instead, what we need is just a tiny version of Node image that has at least npm in it. That is why Node also comes with its own alpine tag just like other official images.</p> <p>Let’s try to modify our Dockerfile to fix this.</p> <pre><code class="language-bash">cat Dockerfile # Base image used FROM node:alpine # Install project dependencies RUN npm install # Run default command CMD ["npm", "start"] </code></pre> <p>Now, build it again with the alpine tag of the official node base image:</p> <pre><code class="language-bash">docker build . Sending build context to Docker daemon 2.01MB Step 1/3 : FROM node:alpine alpine: Pulling from library/node cbdbe7a5bc2a: Pull complete fb0e3739aee1: Pull complete 738de7869598: Pull complete ffd68be3d86c: Pull complete Digest: sha256:7d11fea6d901bfe59999dda0fa3514438628a134d43c27c2eaec43cc8f4b98d5 Status: Downloaded newer image for node:alpine ---> 3bf5a7d41d77 Step 2/3 : RUN npm install ---> Running in 091b75730aa4 npm WARN saveError ENOENT: no such file or directory, open '/package.json' npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN enoent ENOENT: no such file or directory, open '/package.json' npm WARN !invalid#2 No description npm WARN !invalid#2 No repository field. npm WARN !invalid#2 No README data npm WARN !invalid#2 No license field. </code></pre> <p>We have found a problem: We need to ensure the docker container is aware of our package.json file.</p> <p>Problem resolution: When you see this warning or error, it is because the node package manager always relies on package.json file (which is generated as part of npm init during project initialization) for the installation of project dependencies. It shouldn’t be any surprise that containers don't have this package.json on its file system snapshot because the whole of our project source code is on a local hard disk. So the File system snapshot found in the container is merely the one which comes from node:alpine base image. One solution to this problem is to make your project source code somehow available in the container. Hence, before we install project dependencies using npm install in Dockerfile, package.json must be made available inside your container file system. The way we do it is to use the COPY instruction of Dockerfile.</p> <p>Here is the syntax:</p> <pre><code class="language-bash"># COPY docker instruction syntax COPY &#x3C;PATH to folder to copy from> &#x3C;destination inside container> </code></pre> <p>Notice that there are various ways in which you can make your source code available to the container. The preferred ways are docker volumes and bind mounts. But in this example, we will directly copy the whole source code into the container file system. Here is the updated docker file :</p> <pre><code class="language-bash">cat Dockerfile # Base image used FROM node:alpine COPY ./ ./ # Install project dependencies RUN npm install # Running default command CMD ["npm", "start"] </code></pre> <p>Let’s build and tag it in order to push it to Docker Hub.</p> <pre><code class="language-bash">docker build -t rajput/sample_node_app . Sending build context to Docker daemon 2.01MB Step 1/4 : FROM node:alpine ---> 3bf5a7d41d77 Step 2/4 : COPY ./ ./ ---> 586eada1b908 Step 3/4 : RUN npm install ---> Running in 90bcbf81b81c npm WARN sampleapp@1.0.0 No repository field. audited 50 packages in 2.925s found 0 vulnerabilities Removing intermediate container 90bcbf81b81c ---> bab8c88e351b Step 4/4 : CMD ["npm","start"] ---> Running in bcb475eb3b19 Removing intermediate container bcb475eb3b19 ---> 14119783c338 Successfully built 14119783c338 Successfully tagged rajput/sample_node_app:latest </code></pre> <p>We have now successfully built a docker image for our project. Let’s verify it using the following:</p> <pre><code class="language-bash">docker images REPOSITORY TAG IMAGE ID CREATED SIZE rajput/sample_node_app latest 14119783c338 About a minute ago 119MB </code></pre> <p>As you can see, we have built the docker image and tagged it. Let’s run it now.</p> <p><strong>STEP 4: Running the Docker container</strong></p> <p>Run the docker image that we built in the previous step and see what happens.</p> <pre><code class="language-bash">docker run rajput/sample_node_app > sampleapp@1.0.0 start / > node index.js Node express server is running on 4000. Enjoy NodeJS </code></pre> <p>Great! Our NodeJS express server is running. Let’s access our application on <a href="http://localhost:4000">http://localhost:4000</a>. Uh oh! Something’s wrong. We are not able to connect to our application.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/access-1592307009901.PNG" alt=""></p> <p>Let’s see if the NodeJS container is running to find the root cause.</p> <pre><code class="language-bash">docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c2408a62350e rajput/sample_node_app "docker-entrypoint.s…" 14 seconds ago Up 13 seconds practical_tharp </code></pre> <p>Although we have successfully built our docker image and the container is running as well, the application still can't be accessed. As you can see , the PORT column in the screen above is empty. To understand why this is, you need to understand Docker Networking concepts. The port 4000 is inside the docker container, but we are accessing this 4000 on our host where the docker engine is running. One way to solve this problem is to route the traffic that is coming from your local host to the container where our application resides. Each docker container is considered to be an isolated sandbox with its own namespace, port, mount, etc. So, let’s make sure that whenever any request comes from the local host it is routed or redirected to the application in the container. This concept is called Port Mapping at run time in the docker networking world.</p> <p>Here’s how it looks:</p> <pre><code class="language-bash"># syntax for port mapping docker run -p &#x3C;PORT on HOST>: &#x3C;PORT in container> &#x3C;Image name or ID> docker run -p 8080:4000 rajput/sample_node_app </code></pre> <p>And now we can access our Node application which is running in container!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/sucaccess-1592307705946.PNG" alt=""></p> <p>Let’s visualize the NodeJS container file system snapshot by running a shell inside it</p> <pre><code class="language-bash">docker run -it rajput/sample_node_app sh / # ls Dockerfile etc lib node_modules package.json run sys var bin home media opt proc sbin tmp dev index.js mnt package-lock.json root srv usr </code></pre> <p>Let’s see if we can improve on this: Our application source code is mixed with the root file system of the container.</p> <p>Let’s do it! You can see the application source code, including package.json, was copied inside the root file system of the container. Notice that, in order to avoid any undesired state of your container root file system once it is mixed with the project source code, it is recommended that you choose your own custom application directory inside the container file system. This will help in avoiding any undesired overriding of the file system by your own project source code. We can achieve this by using the WORKDIR instruction in our Docker file. This instruction ensures that any further commands (followed by WORKDIR) in the Dockerfile will be executed relative to WORKDIR in the container</p> <p>We will use this to optimize our Dockerfile:</p> <pre><code class="language-bash">cat Dockerfile # Base image used FROM node:alpine WORKDIR /usr/mynodeapp COPY ./ ./ # Install project dependencies RUN npm install # Running default command CMD ["npm", "start"] </code></pre> <p>Build and tag the docker file</p> <pre><code class="language-bash">docker build -t rajput/sample_node_app . </code></pre> <p>Run the container out of the successfully built image:</p> <pre><code class="language-bash">docker run -p 8080:4000 rajput/sample_node_app </code></pre> <p>Notice that you might get an error here saying:</p> <pre><code class="language-bash">Error response from daemon: driver failed programming external connectivity on endpoint bold_germain (22e41c9ded62fe9f7347d0bbe116e4b23ad7c33890f5ec7401c0566441210616): Bind for 0.0.0.0:8080 failed: port is already allocated. </code></pre> <p>In order to resolve this, you need to ensure that the port on your host machine is not already allocated before running the NodeJS container. One common way to solve this is by stopping the running container on that conflicted port or choose a different port on your host machine. You can stop running the container using the below command:</p> <pre><code class="language-bash">docker container stop &#x3C;container ID> </code></pre> <p>Let's list the running containers</p> <pre><code class="language-bash">docker ps </code></pre> <p>Examine the file system for NodeJS container</p> <pre><code class="language-bash">docker exec -it 13940468222f sh /usr/mynodeapp # ls Dockerfile index.js node_modules package-lock.json package.json </code></pre> <p>You can see that all of our project source code is not in the root file system anymore and instead, it is in /usr/mynodeapp/.</p> <pre><code class="language-bash">/usr/mynodeapp # ls Dockerfile index.js node_modules package-lock.json package.json </code></pre> <p>There is still an opportunity to improve this docker file. The Node package manager (npm) just uses package.json to install any project specific dependencies and it doesn’t care about the rest of the source code of your application. To further optimize the code, you need to first copy the package.json and then run npm install. Later, you will copy the remaining source code. This will ensure that any source code change in your application does not install project dependencies again and again, and that it does so only when new dependencies are added or an existing dependency version is changed in the package.json file in your project. It is always a best practice to keep the things that more frequently change, such as project or application code, at the bottom of the docker file.</p> <p>If you have done all this, the new Dockerfile should look like what’s shown below:</p> <pre><code class="language-bash">cat Dockerfile MAINTAINER geeks@hpe.com # Base image FROM node:alpine # Working directory WORKDIR /usr/mynodeapp COPY ./package.json ./ RUN npm install COPY ./ ./ </code></pre> <p>Let’s build and tag the image again . Ensure tag name for your image is unique at any point of time.</p> <pre><code class="language-bash">docker build -t rajput/sample_node_app . </code></pre> <p>Once the finalized image is successfully built, we can run the NodeJS container out of it:</p> <pre><code class="language-bash">docker run -p 8080:4000 rajput/sample_node_app </code></pre> <p>Good, so we have the final docker image for our NodeJS based backend application. Let’s push it to Docker Hub.</p> <p><strong>Step 5: Push the image on docker hub</strong></p> <p>Congratulations ! It's time to push the docker image onto the docker hub. Docker hub is a public registry for your docker images ! You need to have docker hub account in order to push your image onto it.</p> <pre><code class="language-bash">docker login Username: rajput Password: Login Succeeded </code></pre> <pre><code class="language-bash">docker push rajput/sample_node_app The push refers to repository [docker.io/rajput/sample_node_app] 312072b77e32: Pushed 5a3885fb97b9: Pushed latest: digest: sha256:707eae883285a5209283aea94950fee5c9f9357a36b1d6f53c60cb659fd950ec size: 1782 </code></pre> <p>Once you have pushed your image, this is what it will look like on the Docker Hub public registry.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/6/hub-1592318453804.PNG" alt=""></p> <p>Even though we have made great progress, we are still left with the problem that for any change in the application source code still requires rebuilding the docker image. Any time we want to modify project source code, we need to rebuild the image again. This is because the source code is still on our local hard disk and it is nowhere referenced in the container and it is copied into the container only when the docker image is built. One way to solve this problem is to use docker volumes or bind mounts, which let you change your source code from local hard disk and have the change reflected in the running container without even rebuilding the image. I plan on writing a follow-on article that will address this.</p> <p>Along with addressing this issue, I plan on showing you the same application built with multiple containers. We will add some No-SQL backend DB, like Redis or Mongo to the NodeJS application service layer. It will be a multi-tier app with a proper service and database layer. We will leverage docker compose to run both containers; one for database and one for express server and see how they interact with each other. As a full-stack developer with a sincere passion for the latest software technologies, I like to help others by writing tutorials like this. I hope you enjoyed my post. You might want to check out another post I wrote <a href="https://developers.redhat.com/blog/author/rkumar/">here</a>. Keep checking back on the <a href="/blog">HPE DEV blog site</a> for new and interesting articles on containers!</p>https://developer.hpe.com/using-raw-block-and-ephemeral-inline-volumes-on-kubernetes/https://developer.hpe.com/using-raw-block-and-ephemeral-inline-volumes-on-kubernetes/Thu, 11 Jun 2020 21:18:38 GMT<p>With the release of the <a href="https://community.hpe.com/t5/around-the-storage-block/hpe-csi-driver-for-kubernetes-1-2-0-available-now/ba-p/7091977">HPE CSI Driver for Kubernetes 1.2.0</a> quite a few new Container Storage Interface (CSI) concepts were introduced as fully supported features. As always, new capabilities introduce new YAML stanzas that needs to be understood to take full advantage of these capabilities. In this blog post, we’ll explore how to expose raw block volumes and the different ways to declare an ephemeral inline volume for Kubernetes Pods.</p> <p>Many CSI drivers support these capabilities. For the examples below, we’ll use a recent version of the HPE CSI Driver with the default <code>StorageClass</code> installed on Kubernetes 1.18. Do note that ephemeral inline volumes are still considered beta in Kubernetes.</p> <h1>Raw block volumes</h1> <p>Kubernetes supports running a diverse set of applications with various needs when it comes to infrastructure requirements, such as compute, networking and storage. Historically, a “volume” on Kubernetes translates to a POSIX-like filesystem to store persistent data at a given path inside a <code>Pod</code>. With the introduction of raw block volumes, there’s now a way to present the underlying block device that the filesystem normally is created on. This is beneficial for applications that are capable of addressing the device directly to store data. It effectively removes the double-buffering effects that filesystems introduces along with the POSIX semantics and filesystem internals. Applications that truly can take advantage of raw block volumes on Kubernetes are few and far between.</p> <p>The concept of presenting a raw block volume to a <code>Pod</code> on Kubernetes is very similar to how Raw Device Mappings (RDMs) are presented on VMware vSphere, where a virtual machine get unfettered direct access to a LUN on a storage fabric exposed to the VMware ESX host.</p> <p>Let’s compare the Kubernetes minutia needed to declare a regular volume versus a raw block volume.</p> <pre><code class="language-yaml">apiVersion: v1 kind: PersistentVolumeClaim metadata: name: block-device spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi volumeMode: Block </code></pre> <p>This is a conventional Persistent Volume Claim (PVC). The only thing that stands out is the <code>.spec.volumeMode</code>. By default, <code>volumeMode</code> is set to <code>Filesystem</code> and is usually never called out explicitly. Setting the <code>volumeMode</code> attribute to <code>Block</code> will change this, presenting the device itself, once it is exposed to a <code>Pod</code>.</p> <p>To be able to address the block device, there’s additional details that needs to be declared in the <code>Pod</code> specification. Let’s bring up a <code>Pod</code> as an example:</p> <pre><code class="language-bash">--- apiVersion: v1 kind: Pod metadata: name: ioping spec: containers: - name: ioping image: hpestorage/ioping command: [ "ioping" ] args: [ "/dev/xvda" ] volumeDevices: - name: raw devicePath: /dev/xvda volumes: - name: raw persistentVolumeClaim: claimName: block-device </code></pre> <p>The <code>.spec.volumes</code> stanza is exactly the same as it would be for using a filesystem. It’s the <code>.spec.containers.volumeDevices</code> and <code>.spec.containers.volumeDevices.devicePath</code> that just got introduced. Creating the above PVC and <code>Pod</code> would result in the following log output:</p> <pre><code class="language-bash">$ kubectl logs -f pod/ioping 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=1 time=1.10 ms (warmup) 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=2 time=1.01 ms 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=3 time=862.1 us 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=4 time=1.11 ms 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=5 time=895.1 us 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=6 time=1.11 ms 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=7 time=976.4 us 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=8 time=853.5 us (fast) 4 KiB &#x3C;&#x3C;&#x3C; /dev/xvda (block device 32 GiB): request=9 time=912.7 us ^C </code></pre> <p>It’s evident that we are indeed accessing a raw block device from inside the <code>Pod</code>.</p> <h1>Real world example for raw block volumes: Rook</h1> <p><a href="https://github.com/rook/rook">Rook</a> is a Cloud Native Computing Foundation (CNCF) incubator project (a graduation proposal in the works at this time) to provide open source cloud-native storage for Kubernetes. Rook provides object, file and block storage to Kubernetes using <a href="https://ceph.io/">CEPH</a>. Rook is complementary to the HPE CSI Driver, which only provide block (Nimble, Primera, 3PAR) versus Rook giving the option to deploy a distributed filesystem on Kubernetes backed by Enterprise storage to present additional data access protocols.</p> <p>Let’s assume we have deployed the Rook Operator on the Kubernetes cluster. Creating a new <code>CephCluster</code> is done as follows:</p> <pre><code class="language-bash">--- apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:v14.2.9 dataDirHostPath: /var/lib/rook mon: count: 3 volumeClaimTemplate: spec: storageClassName: hpe-standard resources: requests: storage: 10Gi storage: storageClassDeviceSets: - name: set1 count: 3 portable: true tuneSlowDeviceClass: false volumeClaimTemplates: - metadata: name: data spec: resources: requests: storage: 32Gi storageClassName: hpe-standard volumeMode: Block accessModes: - ReadWriteOnce </code></pre> <p>Pay attention to the <code>volumeMode: Block</code> attribute in the specification. We can further inspect the PVC created by the <code>StatefulSet</code> that has been declared:</p> <pre><code class="language-bash">kubectl get pvc -n rook-ceph -l ceph.rook.io/DeviceSetPVCId=set1-data-0 -o json | json items.0.spec { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "32Gi" } }, "storageClassName": "hpe-standard", "volumeMode": "Block", "volumeName": "pvc-26e4e5d5-0e08-46c6-9a2e-679e0bde6264" } </code></pre> <p>It’s now possible to use the CEPH cluster to create <a href="https://rook.io/docs/rook/v1.3/ceph-filesystem.html">filesystems</a> and <a href="https://rook.io/docs/rook/v1.3/ceph-object.html">object stores</a>.</p> <blockquote> <p><strong>Note!</strong> Use Rook at your own risk. This is an example, not an endorsement.</p> </blockquote> <h1>Ephemeral inline volumes</h1> <p>The term ephemeral inline volume is quite a mouthful for what it is – a temporary placement of data which you don’t really care about long-term, most commonly talked about as a "scratch disk". However, this is a very important construct for data intensive applications where Kubernetes administrators now have the ability to dictate placement of IO intensive applications that require temporary storage. Up until the introduction of ephemeral inline volumes, applications have simply used the container runtime provided union filesystem inside the container for scratch space or used other shared mechanisms like <code>hostPath</code> or <code>emptyDir</code>. Sharing resources on the host has its challenges. First off, the Kubernetes admin has no means to put any sort of boundaries in place for an individual container. That, in turn, could lead to potentially having a single container consume the entire host filesystem and starve other containers on the host for resources.</p> <p>The term “inline” means the volume declaration resides inside the <code>Pod</code> specification. Each <code>Pod</code>, regardless of replica count, will be given a dedicated <code>ReadWriteOnce</code> volume as per the declaration. Let’s see what it looks like.</p> <pre><code class="language-bash">--- apiVersion: v1 kind: Pod metadata: name: my-pod-inline-mount-2 spec: containers: - name: pod-datelog-1 image: nginx command: ["bin/sh"] args: ["-c", "while true; do date >> /data/mydata.txt; sleep 1; done"] volumeMounts: - name: my-volume-1 mountPath: /data volumes: - name: my-volume-1 csi: driver: csi.hpe.com fsType: ext3 volumeAttributes: csi.storage.k8s.io/ephemeral: "true" inline-volume-secret-name: nimble-secret inline-volume-secret-namespace: kube-system accessProtocol: "iscsi" size: "7Gi" </code></pre> <p>The interesting part here is the <code>.spec.volumes.csi</code> stanza. This is the bare minimum amount of parameters required to provision an inline volume. Any additional parameters supported by the Container Storage Provider (CSP) may be used here. Note that there’s no <code>StorageClass</code> at play here. All parameters, including the <code>Secret</code> needs to be part of the declaration. This is where a word of caution is warranted. Handing out the <code>Secret</code> to a user is the same as handing over credentials to the CSP backend!</p> <p>CSI ephemeral inline volumes provide a means for the Kubernetes admin to make cluster users aware of how and where temporary storage resources may be provisioned. This is not a particularly good idea with the HPE Nimble Storage CSP at this time as there’s no mechanism to create the necessary separation. With the HPE 3PAR/Primera CSP, it’s possible to create a separate Virtual Domain for inline volumes and the user is essentially a tenant on the backend storage array.</p> <h1>Next steps</h1> <p>It’s always exciting to talk about new features and capabilities. Take the new CSI driver for a spin and let us know what you think. We hang out on the HPE DEV Slack community. Sign up on <a href="https://slack.hpedev.io">slack.hpedev.io</a> if you’re an external HPE user or login directly at <a href="https://hpedev.slack.com">hpedev.slack.com</a> if you’re an HPE employee.</p> <ul> <li>HPE CSI Driver for Kubernetes <a href="https://hub.helm.sh/charts/hpe-storage/hpe-csi-driver">Helm Chart</a></li> <li>HPE CSI Operator for Kubernetes on <a href="https://operatorhub.io/operator/hpe-csi-operator">OperatorHub.io</a></li> <li>Learn about the CSI driver on <a href="https://scod.hpedev.io">HPE Storage Container Orchestrator Documentation</a> (SCOD)</li> <li>Visit the <a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">HPE Nimble Storage</a> or <a href="https://developer.hpe.com/platform/hpe-3par-and-primera/home">HPE Primera</a> platform pages</li> <li>Read the HPE CSI Driver for Kubernetes release blog on <a href="https://community.hpe.com/t5/around-the-storage-block/hpe-csi-driver-for-kubernetes-1-2-0-available-now/ba-p/7091977">Around The Storage Block</a></li> </ul>https://developer.hpe.com/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun/https://developer.hpe.com/get-involved-in-the-open-source-community-part-2-sharing-with-the-commun/Fri, 05 Jun 2020 12:47:25 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/git-icon-1788c-1590702885345.png" alt="git101-part1-git icon 1788c"></p> <p>In the <a href="/blog/get-involved-in-the-open-source-community-part-1-getting-started-with-gi">Part 1</a> of my blog series, I discussed how to get started with Git and leverage some of the content provided by the open source community. We already covered use case 1 and use case 2 from the list below:</p> <ul> <li>Use case 1: I’d like to use something from the community</li> <li>Use case 2: I'd like to report an issue on a repository</li> <li>Use case 3: I'd like to share something with the community</li> <li>Use case 4: I'd like to contribute code to a repository</li> </ul> <p>In this article I am covering use case 3, where we will start contributing to the community.</p> <h1>Use case 3: I'd like to share something with the community</h1> <p>Let's try sharing something with the open source community. In this use case, you will create your first repository on GitHub. And if you don't have a GitHub account yet, it will also be a good time to create one in order to start sharing things with others. GitHub started as a place for sharing and versioning source code, but many other things can be shared on GitHub, including PDF files (tutorials, for example), config files, markdown files, and Jupyter Notebooks. So, don't be shy and start to contribute.</p> <blockquote> <p>Note: Anything can be shared, but Git works best on non-binary content.</p> </blockquote> <h2>Step 1: Creating a personal account on Github</h2> <p>For those of you who do not have a GitHub account (or equivalent), now is a good time to create one. To do so:</p> <ol> <li>Connect to <a href="https://github.com/join">https://github.com/join</a></li> <li>Choose a username, enter your email, and select a password</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/signup-1591362322035.png" alt="git101-part2-signup"></p> <p>It’s as simple as that!</p> <h2>Step 2: Create an empty repo called WelcomeGit</h2> <p>Let’s create our first repository (a.k.a. repo) using the following instructions:</p> <ol> <li>Use the + sign next to your profile to Add a repository</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/newrepo-1591362355840.png" alt="git101-part2-newrepo"></p> <ol start="2"> <li>Provide WelcomeGit for a repo name</li> <li>Provide a description</li> <li>Make it Public</li> <li>Do not initialize a README file</li> <li>Do not setup an ignore file</li> <li>Select Apache License 2.0</li> </ol> <blockquote> <p>Note: It is good practice to select a licensing scheme as soon as you create a repository. There are many choices, and the specificities of each scheme is beyond the scope of this article. We selected Apache License 2.0, which is a very permissive license for your code (basically, anyone can use it for any purpose and there is no requirement to share changes). You can find out more regarding open source licensing schemes [here] (<a href="https://opensource.org/licenses">https://opensource.org/licenses</a>).</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/createrepo-1591362387371.png" alt="git101-part2-createrepo"></p> <h2>Step 3: Working securely with GitHub</h2> <p>You could use your GitHub password from the CLI, but it's not considered good practice to use passwords in clear text in a terminal session. Our recommendation is to use a specifically generated token instead (which we can revoke when you don’t need it anymore). For this, we will need to follow the instructions described [here] (<a href="https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line">https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line</a>).</p> <blockquote> <p>Note: Another popular option is to use an SSH key to access GitHub and avoid providing any password or token. Creating a public/private keypair is beyond the scope of this lab, so we will stick to HTTPS and use a token. But feel free to check how to import your SSH key <a href="https://help.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account">here</a>.</p> </blockquote> <p>Let's go:</p> <ol> <li><a href="https://github.com/settings/profile">Open Personal Settings</a></li> <li><a href="https://github.com/settings/apps">Select the Developer settings</a></li> <li><a href="https://github.com/settings/tokens">Select Personal access tokens</a></li> <li><a href="https://github.com/settings/tokens/new">Click Generate new token</a></li> <li>Set up a note such as "Git101 Token", select "repo" for the scope and click Generate token (this will prompt you for your GitHub account password)</li> <li>Copy the generated token to the clipboard and use it when prompted for you git password</li> </ol> <blockquote> <p>Note: You cannot display the value of a token once you close the window. You will need to regenerate a new token if you forgot its value.</p> </blockquote> <h2>Step 4: Build content into your WelcomeGit repo from Git CLI</h2> <p>Open a terminal session (terminal on Mac, PowerShell on Windows).</p> <p>1/ Create a folder called <strong>welcomegit</strong></p> <pre><code class="language-bash">$ mkdir welcomegit $ cd welcomegit </code></pre> <p>2/ Initialize Git on this empty folder using <code>git init</code>. After Git was initialize all changes in this folder will be monitored by Git. Your folder has become a local repo(sitory)</p> <pre><code class="language-bash">$ git init Initialized empty Git repository in /Users/lalli/welcomegit/.git/ </code></pre> <p>3/ Create a <strong>README.md</strong> file with just one line in it. For example: “This is my README file”</p> <pre><code class="language-bash">$ cat >> README.md This is my README file ˆD </code></pre> <p>4/ Let’s query the status of our local repo using <code>git status</code></p> <pre><code class="language-markdown">$ git status On branch master No commits yet Untracked files: (use "git add &#x3C;file>..." to include in what will be committed) README.md nothing added to commit but untracked files present (use "git add" to track) </code></pre> <p>Git tells you that a file was discovered and it needs to be added to Git to make it part of the repo.</p> <p>5/ Add file to the repo with git add (<code>git add .</code> will add all files in folder)</p> <pre><code class="language-bash">$ git add README.md </code></pre> <p>6/ Let’s query the status of our local repo using <code>git status</code> again</p> <pre><code class="language-markdown">$ git status On branch master No commits yet Changes to be committed: (use "git rm --cached &#x3C;file>..." to unstage. new file: README.md </code></pre> <p>We see that the new file is now part of the repo, but the changes have not yet been committed.</p> <p>7/ Before we commit, let’s set a few Git environment variables</p> <pre><code class="language-markdown">$ git config --global user.email &#x3C;your-git-email> $ git config --global user.name &#x3C;your-git-username> </code></pre> <p>8/ Commit changes to repo with <code>git commit</code></p> <pre><code class="language-markdown">$ git commit -m “Added README.md to repo” [master (root-commit) ab3adf5] Added README.md to repo 1 file changed, 1 insertion(+) create mode 100644 README.md </code></pre> <p>9/ Let’s now connect this local repo to our WelcomeGit repo (created in Step 2) using <code>git remote</code></p> <blockquote> <p>Note: Best practice is to call this remote: origin</p> </blockquote> <pre><code class="language-bash">$ git remote add origin &#x3C;your-repo-URL-goes-here> -m master </code></pre> <p>For example:</p> <pre><code class="language-bash">$ git remote add origin https://github.com/didou06/WelcomeGit -m master </code></pre> <p>10/ Verify status of our remote repo with <code>git remote -v</code></p> <pre><code class="language-markdown">$ git remote -v origin https://github.com/Didou06/WelcomeGit.git (fetch) origin https://github.com/Didou06/WelcomeGit.git (push) </code></pre> <p>11/ It’s now time to push our changes to your remote repo using git push. You should be prompted for GitHub username and password.</p> <pre><code class="language-markdown">$ git push --set-upstream origin master Username for 'https://github.com': didou06 Password for 'https://didou06@github.com': Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), 247 bytes | 247.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To https://github.com/Didou06/WelcomeGit.git * [new branch] master -> master Branch 'master' set up to track remote branch 'master' from 'origin'. </code></pre> <p>12/ Query status of repo one last time</p> <pre><code class="language-markdown">$ git status On branch master Your branch is up to date with 'origin/master'. nothing to commit, working tree clean </code></pre> <p>That’s already quite a lot of commands. There are a lot more, but these are the most important ones to learn to get started.</p> <h2>Step 5: Verify WelcomeGit repo in web GUI</h2> <p>On your GitHub repo, you can verify that you now have a <strong>README.md</strong> file with some content (refresh page, if needed). Well done!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/readmeadded-1591365686744.png" alt="git101-part2-readmeadded"></p> <p>That's it. From now on, every change made in the WelcomeGit folder on your machine (add file with <code>git add</code>, delete file with <code>git rm</code>, rename file with <code>git mv</code>, modify content of files) will be tracked by Git. It is your choice to commit changes locally like we did in Step 4 using <code>git commit</code>. It is also your choice to push changes from your local repo to your remote repo using <code>git push</code>. You can add contributors to your project, and each will work the same way, using a local repo, then pushing back centrally. Git makes it possible to scale to hundreds of contributors as shown in the <a href="https://github.com/grommet/grommet">Grommet GitHub</a>, for example.</p> <p>This terminates use case 3: Congratulations! You have created your GitHub account and populated your first repo from the command line using Git. You have just become an active member of the open source community :-).</p> <p>If you want to discover more about Git, I recommend the following resources:</p> <ul> <li><a href="https://www.git-scm.com/book/en/v2">The Pro Git Book</a></li> <li><a href="https://guides.github.com/introduction/git-handbook/">The Git Handbook</a></li> <li><a href="https://www.youtube.com/watch?v=SWYqp7iY_Tc&#x26;feature=youtu.be">Video Git Crash Course</a></li> </ul> <p>Stay tuned to <a href="/blog">HPE DEV</a> for the last article where I will cover use case 4. In this use case, you will start contributing some code back to an existing project.</p>https://developer.hpe.com/wow-part-3-conveying-the-business-value-in-a-purposefully-designed-ux/https://developer.hpe.com/wow-part-3-conveying-the-business-value-in-a-purposefully-designed-ux/Thu, 04 Jun 2020 16:59:57 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture1-1591290304651.png" alt="picture1"></p> <h2>What is WOW?</h2> <p>Previously in my last two posts (<a href="/blog/wow-a-practiced-and-perfected-design-process-part-1-uncovering-the-merit">Part 1</a>, <a href="/blog/wow-part-2-communicating-with-customers-and-constructors">Part 2</a>), I covered the first three stages of a methodology our group developed called WOW (Why On What with customers and constructors). This workflow demonstrates how to quantify the value of the design process and provides enterprise UX designers with a practiced and perfected path to achieve success.</p> <p>The WOW methodology helps creative teams focus on four important stages of a project:</p> <ol> <li>Uncover why the business (and this particular project) needs a UX</li> <li>Involve the customer early in the design phase</li> <li>Ensure the constructor (developer) has the right information during design implementation</li> <li>Convey the business value of the design</li> </ol> <p>Now that we’ve helped uncover why a project needs UX design (by illustrating the benefits of the tools it offers and shown how a designer’s use of empathy can smooth the path to achieving a successful design), I’d like to cover the last part of WOW. In this article, I’ll show you how to articulate the business value of the design process.</p> <h2>UX deliverables with data, comparisons, and designs</h2> <p>The tools that designers use to create designs, such as wireframes or screen designs, are not enough to demonstrate the value of design. Audiences understand the screens but, at the same time, they also like to debate the screens. Using data and comparison studies can help avoid vague and lengthy debates that rarely result in any specific outcome.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture3-1591290322580.png" alt="picture3"></p> <p>Our data shows that UX-reviews were among the lengthiest meetings for our team over the past few years. I agree these meetings are very engaging!! But just a mention of a small number of data points can bring the meeting duration down by 10-12 minutes.</p> <h2>Wireframes versus the voice of a customer?</h2> <p>Wireframes have a notorious history of being treated as a designer’s opinion. Wireframes, as primary deliverable, are not enough to help stakeholders realize the value the UX design process brings to the project. Facts, testimonials, and comparative studies should be used as the primary deliverables to be given to stakeholders. Wireframes should only be used as a secondary method that supports the primary deliverables.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture5-1591290355435.png" alt="picture5"></p> <p>As part of the WOW process, designers will have gone through stages that helps them accumulate the facts required to sway opinion. They will be able to show how the UX design process added value from the very beginning, identifying potential gaps in the project and adding competitive value. They will be able to express the value of the UX in customer terms, relaying specifically what customers found helpful and necessary for the success of the product experience.</p> <h2>Key takeaways</h2> <p>As I’ve shown in these past three blog posts, there is a lot more to UX design than what is often considered. UX design isn’t just used to make an application look pretty. UX design brings many benefits to the product development process, in that it:</p> <ol> <li>Offers specific UX tools and techniques that can help identify the existing gaps in the product.</li> <li>Promotes empathy, not just with customers but also with UI developers (the constructors).</li> <li>Provides tools to demonstrate the business value achieved through UX.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture6-1591290392865.png" alt="picture6"></p> <p>I hope that, by using this WOW methodology, you will find it easier to convince others of the value of the UX design process and achieve greater success in your next project. I’m always interested in hearing about your experience. Feel free to reach out to me at <a href="https://twitter.com/uxwithparul">@uxwithparul</a> if you have any questions. Don’t forget to check back at <a href="/blog">HPE DEV</a> and see what other tutorials we have that can make your life easier.</p>https://developer.hpe.com/hpe-container-platform-rest-api-part-2-deploying-containerized-applicati/https://developer.hpe.com/hpe-container-platform-rest-api-part-2-deploying-containerized-applicati/Thu, 04 Jun 2020 16:41:01 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>In my previous blog post, <a href="/blog/hpe-container-platform-rest-api-part-1-authenticating">HPE Container platform REST API – Part 1: Authenticating</a>, I introduced the HPE Container Platform (HPE CP) REST API. I showed you how to authenticate to the HPE Container Platform API endpoint and retrieve data from objects in a secure way using the command line cURL. Continuing with this series, my second article will walk you through the steps you need to take to deploy containerized applications programmatically on Kubernetes clusters that are managed by the HPE Container Platform. It will show you how to take the REST API authentication call you established while going through the first blog and apply it to a real life scenario focused on the following areas:</p> <ul> <li>Deployment of cloud native microservices based applications</li> <li>Deployment of non-cloud native, stateful, distributed analytics workloads using pre-configured <a href="/blog/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern">KubeDirector applications</a></li> </ul> <h2>Deploying stateless and stateful containerized applications using a programmatic approach</h2> <p>This tutorial assumes you have established a login session with the HPE Container Platform as explained in the first part of this series. The next step in deploying a containerized application in Kubernetes clusters managed by the HPE Container Platform is to get the kubeconfig file for your tenant working context.</p> <blockquote> <p>Note: A working context establishes the user identity, its tenant name and role (member or admin). Based on this context, tenant users are granted privileges and permissions to create and manage resources for their tenant on Kubernetes clusters managed by HPE CP.</p> </blockquote> <p>The HPE CP REST API call below allows you to obtain the kubeconfig file used to access the Kubernetes cluster for your tenant user account based on your assigned role (tenant member or tenant admin), just as if you had downloaded it from the HPE CP UI.</p> <p>The REST API call is a <strong>GET</strong> request for the target URL <strong>/api/v2/k8skubeconfig</strong> authenticated for your working tenant context (X-BDS-SESSION). Here, the kubeconfig file is saved as <em>config</em> in your $HOME/.kube directory. The call retrieves a configuration file suitable for use by K8s API client such as <em>kubectl</em>, which includes a valid session location (token) for your current session.</p> <pre><code class="language-markdown"> curl -k -s --request GET "https://&#x3C;Gateway-IP-Address-or-fqdn>:8080/api/v2/k8skubeconfig" \ --header "X-BDS-SESSION: $sessionlocation" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' > $HOME/.kube/config </code></pre> <p>You can now send Kubernetes API requests using kubectl to deploy enterprise workloads to the kubernetes cluster using the privileges assigned to your tenant role.</p> <blockquote> <p>Note: Once the session location expires (after 24 hours by default), any attempt to execute kubectl commands will prompt you for your password and will require you to log in again before continuing.</p> </blockquote> <p>Let's see how this works by deploying a simple hello-world stateless, microservices application. We’ll also try it with a complex, distributed, stateful application, using big data analytics-processing, such as Spark.</p> <h3>Hello-World stateless application deployment</h3> <p>The hello-world application is a <strong>stateless</strong> application because it does not require persistence of data nor an application state. The hello-world application is a very simple application that will return <code>Hello Kubernetes!</code> when accessed. The YAML file below describes the application resources involved, such as the deployment, the Pod, the Docker container image and port, and the NodePort service used to expose the application outside of the Kubernetes cluster.</p> <pre><code class="language-yaml">apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-demouser spec: selector: matchLabels: run: hello-demouser replicas: 2 template: metadata: labels: run: hello-demouser spec: containers: - name: hello-world-demouser image: gcr.io/google-samples/node-hello:1.0 ports: - containerPort: 8080 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: hello-world-service-demouser spec: selector: run: hello-demouser ports: - name: http-hello protocol: TCP port: 8080 targetPort: 8080 type: NodePort </code></pre> <p>The next step is to save the file, for example as <em>hello-world-app.yaml</em>, and deploy the application using the K8s API call <code>kubectl apply -f hello-world-app.yaml</code>. As shown by the command <code>kubectl get</code> below, this simple hello-world application will be represented by standard Kubernetes resource elements (Deployment, Pods and Service) that compose your containerized application.</p> <pre><code class="language-markdown">kubectl get deploy,pod,service NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hello-world-demouser 2/2 2 2 3m6s NAME READY STATUS RESTARTS AGE pod/hello-world-demouser-54b6fcd974-mt8sc 1/1 Running 0 3m6s pod/hello-world-demouser-54b6fcd974-wnkld 1/1 Running 0 3m6s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) service/hello-world-service-demouser NodePort 10.96.113.175 &#x3C;none> 8080:32295/TCP </code></pre> <p>For this tutorial, the HPE Container Platform has been configured to automatically map the NodePort service endpoint to the HPE Container Platform gateway host. In this setup, access to application services running in containers in the HPE Container Platform is proxied via the gateway host and a port number greater than 10000. The following <code>kubectl</code> command can be used to obtain the application service endpoint from the annotations:</p> <pre><code class="language-markdown">kubectl describe service hello-world-service-demouser Name: hello-world-service-demouser Namespace: k8shacktenant Labels: hpecp.hpe.com/hpecp-internal-gateway=true Annotations: hpecp-internal-gateway/8080: gateway1.hpedevlab.net:10012 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-world-service-demouser","namespace":"k8shacktenant"},"spec"... Selector: run=hello-demouser Type: NodePort IP: 10.96.113.175 Port: http-hello 8080/TCP TargetPort: 8080/TCP NodePort: http-hello 32295/TCP Endpoints: 10.192.0.217:8080,10.192.1.102:8080 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HpeCp 16s hpecp-agent Created HPECP K8S service </code></pre> <p>In this example, the application service endpoint is <em>gateway1.hpedevlab.net:10012</em>. You can connect to the application using the cURL command below or connect through your favorite browser:</p> <pre><code class="language-markdown">curl –k –s --request GET https://gateway1.hpedevlab.net:10012 Hello Kubernetes! </code></pre> <h3>Spark stateful application deployment using KubeDirector</h3> <p>HPE has been working within the open source Kubernetes community to add capabilities that enable the running of <strong>stateful</strong> analytics workloads (e.g: data-intensive, AI/ML and analytics-processing distributed applications) on Kubernetes. The open source project is known as <strong>Kubernetes Director</strong> or <strong>KubeDirector</strong> for short.</p> <p>KubeDirector is a key component of the HPE Container Platform, implemented as a Kubernetes custom controller (also known as operator) on every Kubernetes cluster managed by the HPE Container Platform. You can find more information about KubeDirector by visiting the HPE DEV portal <a href="https://developer.hpe.com/platform/hpe-container-platform/home">here.</a></p> <p>A <strong>stateful</strong> application may require persistence of network identity (i.e.: hostname) and persistence of certain mount points across application cluster nodes rescheduling, restarts, upgrades and rollbacks. In the HPE Container Platform, these applications generally refer to a distributed, single-node or multi-node application <strong>virtual cluster</strong>. Each application virtual cluster node runs as a <strong>container</strong> in a Pod in the Kubernetes cluster managed by the HPE Container Platform.</p> <p>In our HPE CP deployment, three pre-configured KubeDirector Application types have been installed on the Kubernetes cluster managed by HPE Controller Platform. As tenant user, you can get the list of KubeDirector applications that are visible to your tenant using the <code>kubectl</code> command below:</p> <pre><code class="language-markdown">kubectl get kubedirectorapp NAME AGE centos7x 37d ml-jupyter-notebook 37d spark221e2 37d </code></pre> <p>Now, let’s inspect the definition of the Spark application type using kubectl command <code>kubectl describe kubedirectorapp spark221e2</code></p> <p>Spark is a non-cloud native multi-tier application with tightly coupled and interdependent services. As shown in the output of command below, the Spark221e2 KubeDirector Application describes the application metadata: the service roles, the service endpoints port and port name prefix (that comes from the URL Scheme), the Docker images, the configuration packages, the cardinality (minimum number of members in a role), and the root file system directories (e.g.: /etc, /bin, /opt, /var, /usr) of the containers to persist beyond the life span of the containers. This means stateful applications that require writing data to their root file systems can now successfully run on Kubernetes.</p> <pre><code class="language-markdown">kubectl describe kubedirectorapp spark221e2 Name: spark221e2 Namespace: k8shacktenant Labels: &#x3C;none> Annotations: &#x3C;none> API Version: kubedirector.hpe.com/v1beta1 Kind: KubeDirectorApp Metadata: Creation Timestamp: 2020-03-24T08:25:15Z Generation: 1 Resource Version: 981021 Self Link: /apis/kubedirector.hpe.com/v1beta1/namespaces/k8shacktenant/kubedirectorapps/spark221e2 UID: 753618be-1edf-470c-bff5-385d0e76fafe Spec: Config: Role Services: Role ID: controller Service I Ds: ssh spark spark-master spark-worker Role ID: worker Service I Ds: ssh spark-worker Role ID: jupyter Service I Ds: ssh jupyter-nb Selected Roles: controller worker jupyter Config Schema Version: 7 Distro ID: bluedata/spark221e2 Label: Description: Spark 2.2.1 with Jupyter notebook Name: Spark 2.2.1 + Jupyter Roles: Cardinality: 1 Config Package: Package URL: file:///opt/configscripts/appconfig-2.6.tgz Id: controller Image Repo Tag: docker.io/bluedata/sparkbase:2.2 Persist Dirs: /usr /opt /var /data Cardinality: 0+ Config Package: Package URL: file:///opt/configscripts/appconfig-2.6.tgz Id: worker Image Repo Tag: docker.io/bluedata/sparkbase:2.2 Persist Dirs: /usr /opt /var /data Cardinality: 0+ Config Package: Package URL: file:///opt/configscripts/appconfig-2.6.tgz Id: jupyter Image Repo Tag: docker.io/bluedata/jupyter:2.3 Persist Dirs: /usr /opt /var /data Services: Endpoint: Is Dashboard: false Port: 22 Id: ssh Label: Name: SSH Endpoint: Is Dashboard: true Path: / Port: 8080 URL Scheme: http Id: spark Label: Name: Spark master Endpoint: Is Dashboard: false Port: 7077 URL Scheme: spark Id: spark-master Label: Name: Spark master Endpoint: Is Dashboard: true Path: / Port: 8081 URL Scheme: http Id: spark-worker Label: Name: Spark worker Endpoint: Is Dashboard: true Path: / Port: 8888 URL Scheme: http Id: jupyter-nb Label: Name: Jupyter Notebook Systemd Required: true Version: 2.7 Events: &#x3C;none> </code></pre> <p>A configuration manifest YAML file is then used to create an application virtual cluster that <strong>instantiates</strong> a defined KubeDirector Application type. The configuration file is used to describe the attributes of a given KubeDirector Application type instance, such as the application instance name, KubeDirector App type, the service roles and their number of nodes and compute size, as well as the persistent storage size.</p> <blockquote> <p>Note: The Spark KubeDirector Application variant used in this tutorial is a distributed implementation of the data-processing Spark cluster where the master (Spark driver) and worker (Spark executors) services run on different cluster nodes (1 controller node and 2 worker nodes). A separate Jupyter node is used as an interactive client to execute programs on the Spark cluster.</p> </blockquote> <pre><code class="language-yaml">apiVersion: "kubedirector.hpe.com/v1beta1" kind: "KubeDirectorCluster" metadata: name: "spark221e2-demouser" spec: app: "spark221e2" appCatalog: "local" roles: - id: "controller" members: 1 resources: requests: memory: "4Gi" cpu: "2" limits: memory: "4Gi" cpu: "2" storage: size: "10Gi" - id: "worker" members: 2 resources: requests: memory: "4Gi" cpu: "2" limits: memory: "4Gi" cpu: "2" storage: size: "10Gi" - id: "jupyter" members: 1 resources: requests: memory: "4Gi" cpu: "2" limits: memory: "4Gi" cpu: "2" storage: size: "10Gi" </code></pre> <p>Now, you can save the file, for example as <em>spark-config.yaml</em>, and deploy the application using the K8s API call <code>kubectl apply –f spark-config.yaml</code>.</p> <p>Your Spark application virtual cluster will be represented in the Kubernetes cluster by a resource of type <strong>KubeDirectorCluster</strong>, with the name that was indicated inside the YAML file used to create it. You can use the <code>kubectl describe kubedirectorcluster &#x3C;clustername></code> command below to observe the status of all the resources that compose the virtual cluster, the state of the virtual cluster, and any events logged against it. Upon successful creation of the virtual cluster, the state of the cluster should have a value of <strong>"configured"</strong>.</p> <p>As shown in the output of the kubectl commands below, the instance of the KubeDirector Application virtual cluster is made up of a <strong>StatefulSet</strong> per role (Spark controller, Spark workers, and jupyter), a <strong>Pod</strong> (a cluster node) per service role member, a <strong>NodePort Service</strong> per service role member, a <strong>headless service</strong> for the application cluster, and a <strong>Persistent Volume Claim (pvc)</strong> per Pod that requested persistent storage.</p> <p>Some lines have been deleted from the output of the kubectl command to display the essential information.</p> <pre><code class="language-markdown">kubectl describe kubedirectorcluster spark221e2-demouser Name: spark221e2-demouser Namespace: k8shacktenant … … Kind: KubeDirectorCluster … … Status: Cluster Service: kdhs-lbjmt … … Roles: Id: controller Members: Node ID: 1 Pod: kdss-j2v6x-0 Pvc: p-kdss-j2v6x-0 Service: s-kdss-j2v6x-0 State: configured Stateful Set: kdss-j2v6x … … Id: worker Members: Node ID: 2 Pod: kdss-7dtqk-0 Pvc: p-kdss-7dtqk-0 Service: s-kdss-7dtqk-0 State: configured … … Node ID: 3 Pod: kdss-7dtqk-1 Pvc: p-kdss-7dtqk-1 Service: s-kdss-7dtqk-1 State: configured Stateful Set: kdss-7dtqk … … Id: jupyter Members: Node ID: 4 Pod: kdss-mwkh4-0 Pvc: p-kdss-mwkh4-0 Service: s-kdss-mwkh4-0 State: configured Stateful Set: kdss-mwkh4 … … State: configured Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Cluster 31m kubedirector new Normal Role 31m kubedirector creating role … … </code></pre> <pre><code class="language-markdown">kubectl get all -l kubedirector.hpe.com/kdcluster= spark221e2-demouser NAME READY STATUS RESTARTS AGE pod/kdss-7dtqk-0 1/1 Running 0 31m pod/kdss-7dtqk-1 1/1 Running 0 31m pod/kdss-j2v6x-0 1/1 Running 0 31m pod/kdss-mwkh4-0 1/1 Running 0 31m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) service/kdhs-lbjmt ClusterIP None &#x3C;none> 8888/TCP service/s-kdss-7dtqk-0 NodePort 10.96.141.31 &#x3C;none> 22:32677/TCP,8081:31223/TCP service/s-kdss-7dtqk-1 NodePort 10.96.44.156 &#x3C;none> 22:32143/TCP,8081:30019/TCP service/s-kdss-j2v6x-0 NodePort 10.96.215.65 &#x3C;none> 22:30358/TCP,8080:31430/TCP,7077:31358/TCP,8081:31160/TCP service/s-kdss-mwkh4-0 NodePort 10.96.5.87 &#x3C;none> 22:30390/TCP,8888:30227/TCP NAME READY AGE statefulset.apps/kdss-7dtqk 2/2 31m statefulset.apps/kdss-j2v6x 1/1 31m statefulset.apps/kdss-mwkh4 1/1 31m </code></pre> <pre><code class="language-markdown">kubectl get pvc -l kubedirector.hpe.com/kdcluster= spark221e2-demouser NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE p-kdss-7dtqk-0 Bound mapr-pv-84ebfd58-e33e-4535-8856-6ef3b54f84a5 10Gi RWO default 31m p-kdss-7dtqk-1 Bound mapr-pv-898d2f58-f516-40ef-bc37-7bd418888f78 10Gi RWO default 31m p-kdss-j2v6x-0 Bound mapr-pv-c635bfa2-b773-430b-aabf-ecc3e0a8bfb5 10Gi RWO default 31m p-kdss-mwkh4-0 Bound mapr-pv-49b9ffb5-b14a-4703-95a3-95b4695eb4f7 10Gi RWO default 31m </code></pre> <p>In the output above, the HPE Container Platform defines HPE Data Fabric (MapR) as default Kubernetes StorageClass for the Kubernetes Clusters managed by the HPE Container Platform. HPE CP uses MapR Container Storage Interface (CSI) storage plugin to expose the HPE Data Fabric as storage provider to the Kubernetes containerized workloads (Pods) that request persistent storage.</p> <p>The ClusterIP service is the headless service required by a Kubernetes StatefulSet to work. It maintains a stable Pod network identity (that is, persistence of the hostname of the Pods across Pods rescheduling).</p> <p>And, the NodePort services expose the application services outside the Kubernetes cluster. The HPE Container Platform automatically maps the NodePort Service endpoints to the HPE Container Platform gateway host to port greater than 10000. To get a report on all the services related to a specific virtual cluster, you can use the command <code>kubectl describe service -l kubedirector.hpe.com/kdcluster=kubedirectorclusterName</code></p> <p>Now, you can connect to the Spark dashboard and the Jupyter Notebook from your browser and start your real time data processing.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture1-1591289916989.png" alt="picture1"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture2-1591289929686.png" alt="picture2"></p> <p>To increase or decrease the number of members in a role, you would just have to edit the configuration YAML file for your application and use the <code>kubectl apply -f file.yaml</code> command to implement the changes. The KubeDirector operator will manage the application cluster expansion or shrinkage for you.</p> <p>Hopefully, this blog has helped you learn how to programmatically interact with the HPE Container Platform to deploy both cloud native stateless, microservices based applications and non-cloud native distributed stateful <a href="/blog/running-non-cloud-native-apps-on-kubernetes-with-kubedirector">KubeDirector</a> applications for various use cases.</p> <p>You can stay up to date with the latest news from HPE DEV by <a href="https://developer.hpe.com/newsletter-signup">signing up for our monthly newsletter.</a> In it, you will find more awesome developers and data scientists focused posts about the HPE Container Platform. You can also follow our community on <a href="https://twitter.com/HPE_Developer">Twitter</a> and join the conversation on our <a href="https://slack.hpedev.io/">HPE DEV Slack Channel.</a></p>https://developer.hpe.com/2020-June-02/https://developer.hpe.com/2020-June-02/Tue, 02 Jun 2020 05:00:00 GMThttps://developer.hpe.com/get-involved-in-the-open-source-community-part-1-getting-started-with-git/https://developer.hpe.com/get-involved-in-the-open-source-community-part-1-getting-started-with-git/Thu, 28 May 2020 18:03:56 GMT<h2>Git</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/git-icon-1788c-1590702885345.png" alt="git101-part1-git icon 1788c"></p> <p>Wikipedia cites: "Git is a distributed version-control system (VCS) for tracking changes in source code during software development. It is designed for coordinating work among programmers, but it can be used to track changes in any set of files. Its goals include speed, data integrity, and support for distributed, non-linear workflows. Git was created by Linus Torvalds in 2005 for development of the Linux kernel, with other kernel developers contributing to its initial development. Its current maintainer since 2005 is Junio Hamano. As with most other distributed version-control systems, and unlike most client–server systems, every Git directory on every computer is a full-fledged repository with complete history and full version-tracking abilities, independent of network access or a central server. Git is free and open-source software distributed under the terms of the GNU General Public License version 2".</p> <blockquote> <p>Note: The origin of the name Git is unclear and subject to many interpretations :-).</p> </blockquote> <p>Latest version is 2.25 from the 13-JAN-2020.</p> <p>Competitor products include Subversion, Microsoft Team Foundation Server, Mercurial, CVS, Perforce, Microsoft Visual SourceSafe, Rational ClearCase. According to a survey from StackOverflow, Git was used by 87.2% of developers in 2018 and is still growing.</p> <h2>GitHub</h2> <p><img src="/img/github-mark.png" alt="git101-part1-github mark light 120px plus"></p> <p>Some commercial companies are providing Source Code Management (SCM) solutions based on Git. One of the most popular is GitHub (<a href="http://github.com">http://github.com</a>). It was bought by Microsoft in</p> <ol start="2018"> <li>From Wikipedia, “as of January 2020, GitHub reports having over 40 million users and more than 100 million repositories (including at least 28 million public repositories), making it the largest host of source code in the world.”</li> </ol> <p>Competitors to GitHub include: Bitbucket, Microsoft Team Foundation Server, Gitlab, Phabricator, Assembla, Beanstalk, Helix Core, Gerrit, SourceForge. We can safely claim that Git is a key component of the open source community and has contributed to its success.</p> <h2>Git Command Line Interface (CLI)</h2> <p>Most developers interact with Git using its command line interface (CLI) in a terminal/command window. But for those of you not comfortable with CLI, lots of Git graphical user interfaces (GUIs) exist and most of them are open source. Choosing one is generally a matter of personal preference. This <a href="https://en.wikipedia.org/wiki/Comparison_of_Git_GUIs">site</a> might help you choose one. Even if you choose to use a Git GUI, it’s best to learn the basics of Git using the CLI. We will be using the CLI in this blog series.</p> <h2>Installing Git on your machine</h2> <p>There are multiple ways to install Git on your machine. For Windows platforms, one option is to install from <a href="https://gitforwindows.org/">https://gitforwindows.org/</a>. For Mac, the easiest is to use brew: brew install git. Both will install the command line interface (CLI) to Git. If you prefer graphical user interfaces, you have plenty of options, too. A recommended one, for both Windows and Mac, is GitHub Desktop . This being said, you might not need any of these as there is a very good integration of Git in most code editors. For example, Visual Studio Code, which has now become a very popular open source Integrated Development Environment (IDE), has very good support for Git.</p> <h2>Getting involved in the open source community</h2> <p>To get you involved in the open source community, we will take you through four typical use cases involving Git:</p> <ul> <li>Use case 1: I’d like to use something from the community</li> <li>Use case 2: I'd like to report an issue on a repository</li> <li>Use case 3: I'd like to share something with the community</li> <li>Use case 4: I'd like to contribute code to a repository</li> </ul> <p>To help you get started using Git, I will cover use cases 1 and 2 in this blog post. In my next post, Part 2, I will cover how to create your own GitHub account to share something with the community. And in my final post, Part 3, I will cover how to contribute code to an existing repository.</p> <h2>Use case 1: I’d like to use something from the community</h2> <p>This is probably the most frequent use case. While looking for a solution to a problem, you discover that someone has already provided a great open source solution with a GitHub repository pointer. How can you take advantage of this? There are actually 2 options: you can clone this repo or you can fork it. You would most likely clone the repository locally if all you want is to take a look at the content and try it. You should fork a repository to your own GitHub account in the case where you would like to modify the content and contribute it back to the original project. I will show you how to fork in use case 4. In this use case, we will use git clone.</p> <h3>Step 1: Cloning a repo locally</h3> <p>Let's imagine that you have found a great repo at <a href="https://github.com/Didier-Lalli/WelcomeGitDidier">https://github.com/Didier-Lalli/WelcomeGitDidier</a> and you'd like to use the Python program shared by its author.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/welcomegit-1590699942819.png" alt="git101-part1-welcomegit"></p> <p>The <strong>Clone or download</strong> button shows the URL of the repo. Copy it to the clipboard. Open a terminal session (terminal on Mac, PowerShell on Windows).</p> <p>1/ Clone repo in that folder with <code>git clone</code></p> <pre><code>$ git clone &#x3C;PasteClipboardContentHere> </code></pre> <p>It should look like:</p> <pre><code>$ git clone https://github.com/Didier-Lalli/WelcomeGitDidier.git Cloning into 'WelcomeGitDidier'... remote: Enumerating objects: 11, done. remote: Counting objects: 100% (11/11), done. remote: Compressing objects: 100% (7/7), done. remote: Total 11 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (11/11), done. </code></pre> <p>2/ Change to that folder and list the content</p> <pre><code>$ cd WelcomeGitDidier $ ls -l -rw-r--r-- 1 lalli staff 20 May 25 18:11 README.md -rw-r--r-- 1 lalli staff 79 May 25 18:11 helloworld.py </code></pre> <p>You can now see that there are two files and one of them is <strong>helloworld.py</strong>. That’s the code we’d like to execute.</p> <h3>Step 2: Running shared code</h3> <p>You can now use the Python program with: <code>python helloworld.py</code></p> <p>Note: you need to have a Python environment running on your machine. If this is not the case, check <a href="https://www.python.org/">https://www.python.org/</a> to get started.</p> <p>1/ Let’s take a quick look at the code</p> <pre><code>$ cat helloworld.py # This is part of labs of our Git101 Jupyter Notebook print("Hello world!") </code></pre> <p>2/ Execute code</p> <pre><code>$ python helloworld.py Hello world! </code></pre> <p>3/ Check the status of your copy of the repo with <code>git status</code></p> <pre><code>$ git status On branch master Your branch is up to date with 'origin/master'. nothing to commit, working tree clean </code></pre> <p>This terminates use case 1: You should now be able to clone a public repo and execute a code provided by the author. But there will be cases where you will want to modify some of the code that was provided by the author. When this happens, you have two options:</p> <p>1/ Contact the author and ask him/her to make the change and <code>git clone</code> it again</p> <p>2/ Make the change yourself in your private copy of the repo and then tell the author about it</p> <p>I’ll tackle the second option in another post, where we’ll look at contributing code to a project (use case 4). But for now, let’s cover the first option in our next use case.</p> <h2>Use case 2: I'd like to report an issue on a repo</h2> <p>There are many ways to get involved in the open source community. Most of them involve writing code, such as when you’re designing new features or fixing bugs, which we will cover in use case 4 in another post. There is another way to contribute, which is to open an issue (think of it as a opening a ticket) to either signal a problem you have found in the code or propose ideas for new features or enhancements. GitHub offers this capability in each project/repo web page.</p> <p>Let's imagine that, in the <strong>WelcomeGitDidier</strong> repo from use case 1 we would like to ask the owner to change the Python code, for example, to display "Hello World!" instead of "Hello world!".</p> <h2>Opening an issue on WelcomeGitDidier</h2> <p>Go back to <strong><a href="https://github.com/Didier-Lalli/WelcomeGitDidier">https://github.com/Didier-Lalli/WelcomeGitDidier</a></strong> and locate the <strong>Issues</strong> tab. Create a new issue, set a title and a description for it, and click <strong>Submit new issue</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/newissuecreated-1590699914277.png" alt="git101-part1-newissuecreated"></p> <p>That's it! The owner(s) of the project will use this issue list to track feedback from the community and, hopefully, take action.</p> <blockquote> <p>Note: Taking a look at the activity in the Issues section of a project is always a good way to sense its level of activity and its level of openness.</p> </blockquote> <p>This terminates use case 2: You should now be able to open an issue on a public repo.</p> <p>If you want to discover more about Git, I recommend the following resources:</p> <ul> <li><a href="https://www.git-scm.com/book/en/v2">The Pro Git Book</a></li> <li><a href="https://guides.github.com/introduction/git-handbook">The Git Handbook</a></li> <li><a href="https://youtu.be/SWYqp7iY_Tc">Video Git Crash Course</a></li> </ul> <p>Stay tuned to <a href="https://developer.hpe.com/blog">HPE DEV</a> for the next articles where I will cover use case 3 and use case 4 where you will start contributing to the community.</p>https://developer.hpe.com/hpe-discovers-brand-new-virtual-hpe-dev-hack-shack/https://developer.hpe.com/hpe-discovers-brand-new-virtual-hpe-dev-hack-shack/Wed, 27 May 2020 14:17:00 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/hack-shack-larger-1590677030652.png" alt="hack shack larger"></p> <p>The world has changed, shifting everyone’s priorities. Digital transformation is no longer something that’s being planned. It’s something that is <em>imperative</em>, <strong>now</strong>. Hewlett Packard Enterprise (HPE) is here to help. HPE Discover offers you the opportunity to connect with subject matter experts and explore solutions through instructor-led workshops to help you realize how HPE can help move your business forward.</p> <p>This year, HPE is bringing <a href="https://www.hpe.com/us/en/discover.html">HPE Discover</a> to you as an on-going virtual experience, offering more opportunities for customers to attend from all over the world – free of charge. We will be kicking off the event on June 23rd and producing it across three different time zones. As part of the virtual experience, you’ll be given the opportunity to view important keynotes, check out what’s new, and hear from key industry analysts. Best of all, you’ll get to experience the all-new, virtual HPE DEV Hack Shack! In the Hack Shack, you’ll interact with experts to learn and try out new solutions for building, designing, and using software. If you’ve ever been to the HPE DEV Hack Shack at HPE Discover, you’ll know how valuable and fun it can be. In addition to the Jupyter Notebooks-based workshops, we’ll be offering competitive coding challenges and fun games to play.</p> <h2>What you’ll find in the Hack Shack</h2> <p>In the HPE DEV Hack Shack you’ll find experts giving talks on a wide variety of topics, from AI and machine learning to containers and DevOps. Workshops are open to all levels with some aimed specifically at beginners, like API 101 or Introduction to the HPE Container Platform. Here’s a quick list of some of our currently planned sessions and workshops. For more complete information, please visit the HPE Discover <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1043&#x26;locale=en_US">session catalog</a>.</p> <p><strong>Hack Shack T490: Demystifying AI Technology Choices</strong></p> <ul> <li>Speaker: <a href="mailto:arti.garg@hpe.com">Arti Garg</a></li> </ul> <p><strong>Hack Shack T491: What Data Scientists Can Learn From Software Developers</strong></p> <ul> <li>Speaker: <a href="mailto:glyn.bowden@hpe.com">Glyn Bowden</a></li> </ul> <p><strong>Hack Shack T492: Accelerate Innovation With DevOps for Machine Learning</strong></p> <ul> <li>Speaker: <a href="mailto:nanda.vijaydev@hpe.com">Nanda Vijaydev</a></li> </ul> <p><strong>Hack Shack W479: Introduction to the HPE Container Platform REST API</strong></p> <ul> <li>Speaker: <a href="mailto:denis.Choukroun@hpe.com">Denis Choukroun</a></li> </ul> <p><strong>Hack Shack W480: API 101 - API basics and the value they provide</strong></p> <ul> <li>Speaker: <a href="mailto:Didier.Lalli@hpe.com">Didier Lalli</a></li> </ul> <p><strong>Hack Shack W481: Aruba API yourself!</strong></p> <ul> <li>Speaker: <a href="mailto:joe.neville@hpe.com">Joe Neville</a></li> </ul> <p><strong>Hack Shack W482: Redfish API use with PowerShell, Python, &#x26; Bash/cURL</strong></p> <ul> <li>Speaker: <a href="mailto:francois.donze@hpe.com">Francois Donze</a></li> </ul> <p><strong>Hack Shack W485: AI 101 - Convolutional neural network (CNN) for MNIST</strong></p> <ul> <li>Speaker: <a href="mailto:hana.malha@hpe.com">Hana Malha</a></li> </ul> <p><strong>Hack Shack W486: Automate apps with the HPE Container Platform</strong></p> <ul> <li>Speaker: <a href="mailto:Chris.Crawford@hpe.com">Chris Crawford</a></li> </ul> <p>During progressive weeks we’ll add in a number of different Hack Shack Challenges that test your hacking prowess. Perhaps you’d like to try deploying your app in a Kubernetes cluster. Or, maybe you’d like to try streamlining queries using Redfish. You can even check out the new Grommet Designer by using it to design your own app!</p> <p>As you may know, HPE is hyper focused on delivering everything as-a-service, all in a hybrid cloud environment. We embrace whatever form of cloud our customers adopt, providing products and services that are singularly focused on delivery through hybrid IT. So, where does the Hack Shack fit in? HPE DEV provides IT folks, whether they’re developers, IT admins, data scientists, or machine learning engineers, the resources needed to take advantage of hybrid cloud. We’ll help you accelerate your recovery by showing you easy ways to become more agile in your business. And we’ll help you accelerate your digital transformation through innovative solutions to complex problems, using open source software, and implementing DevOps methodologies and ITOps tools.</p> <p>For more details on available courses and times, please visit the HPE Discover site <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=1043&#x26;locale=en_US">session catalog</a>. Register now! It’s free, informative, and fun!</p> <p>Note: Starting June 23rd, please log in to the <a href="https://content.attend.hpe.com/go/virtualplatform.landing/?l=1043&#x26;local=en_US">HPE Discover Virtual Experience</a> and search by Session ID or by area of interest (i.e. Hack Shack) to view a session.</p>https://developer.hpe.com/pharmlbind-covid-19-compound-affinity-prediction/https://developer.hpe.com/pharmlbind-covid-19-compound-affinity-prediction/Tue, 26 May 2020 14:47:37 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture1-1590504636961.png" alt="picture1"></p> <p>Hewlett Packard Enterprise (HPE), in collaboration with the Medical University of South Carolina (MUSC), recently announced the open-sourcing of PharML.Bind, a powerful plug-and-play framework for drug discovery. In addition to the codebase for training, inference, data pre-processing and visualization, we are releasing an ensemble of six Molecular-Highway Graph Neural Networks (MH-GNNs) that have been pre-trained for hundreds of hours on state-of-the-art HPE Cray Supercomputers. This announcement is expected to have far reaching implications for COVID-19 research and will ultimately affect how research for other diseases is handled. For all you data scientists out there, I thought I would take a moment to talk a little more about this initiative, offer some examples of how the research can be applied, and direct you to where you can get started with your own research.</p> <p>This document will present some examples that produce encouraging results relative to SARS-CoV-2 and point users to additional resources. It should be considered an overview of the Quick-Start Guide, which is provided on <a href="https://github.com/jbalma/pharml/tree/master/docs">GitHub.</a></p> <p>In the Quick-Start Guide, you’ll learn how to use the ensemble of pre-trained MH-GNNx5 models to make affinity predictions relative to a given structure file (PDB). The guide outlines, in (far) more detail, how users can setup PharML.Bind at home to test various compound sets against relevant COVID-19 structures and even fine-tune the pre-trained models for other interesting mappings. For this post, we’ll just discuss some examples of the framework and introduce some of its capabilities.</p> <blockquote> <p>Note: Any compounds or discussion around treatments for COVID-19 should be considered just that – discussions.</p> </blockquote> <h2>Example Experiment</h2> <p>PharML.Bind is a framework for predicting compound affinity for full protein structures. It differs from other approaches concerning affinity prediction in two ways:</p> <ol> <li><strong>It learns the physical rules governing binding interactions from high-resolution data</strong>. This means that rather than approximating the rules of quantum mechanics (docking) or simulating the kinetics of the interaction through Molecular Dynamics (MD) or Quantum Chemistry (QC) based numerical methods, PharML.Bind can use whatever information is embedded in real-world data to make decisions. And it does so orders of magnitude faster than alternative approaches.</li> <li><strong>It is active-site and compound-pose agnostic.</strong> This means that it uses full-protein structure when making predictions and it doesn’t require users to specify the region of interest along the protein target for a given compound. In addition, it does not need to determine how that compound binds to a given target, just whether it does or not.</li> </ol> <p>Fundamental to the capability and efficiency of PharMLbined is its use of graph neural networks and their associated graph representations used to test candidate molecules against a given protein’s conformation. The MH-GNNx5 models that were released have been trained to represent the binding behaviour across tens of thousands of proteins paired with hundreds of thousands of compounds currently approved under the FDA. These models can be put to immediate use in generating potential compounds rank-ordered by their potential affinity for COVID-19 molecular structures.</p> <p>PharML.Bind utilizes PDB files to define the spatial conformations of a protein of interest (targets), and SDF files (and CSV files) to define sets of compounds (ligands) to be tested against those proteins. You can learn about PDB files and how they are organized <a href="https://pdb101.rcsb.org/learn/guide-to-understanding-pdb-data/introduction">here.</a></p> <p>For our purposes, PDB files only need to contain the biomolecular structure information for how the protein is arranged in a 3D space. The hierarchical definition of a protein in PDB format is defined from the bottom up:</p> <p>atoms -> residues -> chains -> assemblies</p> <p>The basement reality for a protein is the atom. Atoms are defined by the their type and location in X,Y,Z space – no other information is needed. The atomic resolution varies depending on the method used to gather the data (x-ray crystallography, NMR, or Cryo-EM), but typically is on the order of a few Angstroms (~ 0.1 nanometers).</p> <p>For example, using one of these pre-trained models for inference can generate affinity predictions for the well-known “spike” glycoprotein (6VSB). For this particular protein, composed of over 20,000 atoms, we can predict rank-ordered affinity across a set of more than 300,000 compounds in under 25 minutes (1314.1s) using only a single Cray CS-Storm Server containing 8 Nvidia V100 GPUs. Traditional docking methods would require weeks to produce similar results.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture2-1590504725019.png" alt="picture2"></p> <p>With that rate in mind, we laid out an experiment to refine the set of drugs predicted to bind to the primary infection mechanism of the SARS-CoV-2 virus.</p> <p>The virus targets human cells in the lungs, heart, kidneys, and intestines. It does so by evolving a structure (made of protein) that fits spatially into the “active site” defined by a distinct set of proteins on the cell surface. Specifically, the shape of some parts of this spike are highly likely to “stick” to the ACE2-receptor, so long as the ACE2-receptor is also bonded to (or in “complex” with) an amino acid transporter. The conformation of this combined structure on the surface defines a high affinity “lock” for the virus to fit into via its novel “key” at the tip of these spikes. The key in this case is referred to as a Receptor Binding Domain (RBD) and serves as the primary mechanism by which the SARS-CoV-2 assembly infects its host cells. For a great overview of how SARS-CoV-2 is suspected to interact with ACE2, check out Dr Scott Klioze’s video <a href="https://www.youtube.com/watch?v=W1k1sUoLPlA&#x26;feature=youtu.be">here.</a></p> <p>Using what we learned above about how PharML.Bind works, can we use this to predict interactions between the human cell’s receptor site and the RBD of the virus’s spikes?</p> <p>Yes, we can. However, because we don’t know much about the active site that the ACE2 receptor SARs-CoV-2 uses to attach to human cells, we’ll need two structures in total to make reasonable assertions about what might potentially interfere with the mechanism of action for the RBD.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/image-1590529010362.png" alt="image"></p> <p>Notice that items 1 &#x26; 2 listed above are of the same spike structure but were obtained from two different experiments with varied resolutions. Item 2 shows the same glycoprotein structure but with the RBD structure bound to ACE-2. This data was collected via X-Ray Crystallography and has the highest resolution.</p> <p>In order to maximize the likelihood that a subset of the compounds we test against these structures are useful for treating infection by SARS-CoV2-19, we need to carefully think through the question we’re asking PharML.Bind to solve:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture4-1590504846111.png" alt="picture4"></p> <p>One of the things that makes the SARS-CoV-19 virus so dangerous is that it seems to exploit a never before observed active site of the ACE2 receptor at the surface of the cell. Even with the advent of Cryo-EM technologies in the last few years, producing crystallized proteins appropriate for X-Ray Crystallography or atomic resolution microscopy is an incredibly complex and time-consuming process. This complexity greatly limits the size, type and interactions we can currently observe for many proteins.</p> <p>The ACE2 receptor is found on cells which make up the lungs, GI tract, kidneys and blood. It appears that there are two primary RBDs along these spike structures. Because PharML.Bind is active-site agnostic (meaning it doesn’t need information about where the active site is along a structure in order to make predictions), the spikes are a great place to start searching for potential active regions which might interact with existing drugs. We can carefully tune the test cases we run so that we end up with a short list of drugs that potentially bind with high-affinity to the RBD at the tip of the spike. This potentially could limit an infection or prevent the virus from spreading once inside a host.</p> <p>Let’s look at how the conformation of the RBD changes when it is in complex with ACE2 versus the open state. I’ve highlighted a chunk of the amino-acid sequence for a particular part of the protein (PLQSYGF) in green for reference.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture5-1590504935947.png" alt="picture5"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture6-1590504955042.png" alt="picture6"></p> <p>Now, we can clearly see that there is definitely a region along the spike’s RBD which strongly interacts with the ACE2 receptor. The change to the RBD’s shape is noticeable between the two structures. The next step is to run the entire set of FDA-approved compounds against these two structures, and attempt to make some inferences about compounds that might strongly interact with the RBD before it binds to ACE2, therebye protecting cells from infection.</p> <p>The next steps are a bit more involved and require some preprocessing of the BindingDB FDA-Approved Compounds dataset, running the actual inference phase with Tensorflow in a Conda environment, and finally doing some additional post-processing. For a detailed walk-through, please visit the <a href="https://github.com/jbalma/pharml/tree/master/docs">Quick-Start Guide,</a> provided on GitHub.</p> <p>After running the experiment proposed above, we can immediately see that the distribution of compounds PharML.Bind predicts are not random, and, in fact, have some interesting features.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture5-1590505329500.png" alt="picture5"></p> <p>The histogram to the right shows compounds from the FDA-approved dataset arranged by their molecular similarity (where similarity comes from their pair-wise Tanimoto score). If we selected compounds at random from this dataset, we would expect to see a gaussian distribution (orange). However, we see that PharML.Bind has selected non-gaussian sets of compounds as potential binders against the target protein (in this case the main protease, or Mpro). It has selected some particular compounds that are highly similar (>0.7 fingerprint similarity). This implies that PharML is detecting features present only in specific subsets or classes of compounds available in the dataset.</p> <p>We can look at the summary CSV produced after many COVID-19 structures were run against this same database. Each of the ensemble models has casted their vote for which compounds it thinks it should bind. The post-processing phase aggregates these votes, and then rank orders them by their probability to bind. The image below shows some of the interesting output for the two variations on the structure we’ve been looking at.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/screenshot-bindingdb-fda-1590529064031.jpg" alt="screenshot bindingdb fda"></p> <p>When rank-ordering compounds that have CHEMBL IDs, PharML.Bind will also produce a convenient web-based summary of the results. Let’s take a look at what we see for <strong>CHEMBL20:</strong></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/screenshot-bindingdb-compound-1590529046012.jpg" alt="screenshot bindingdb compound"></p> <p>Here, we can see that this is an FDA-approved drug, which goes by various trade names. It is predicted by PharML.Bind to interact with spike glycoprotein of SARS-CoV-2. Because it appears in the “open” conformation of the spike, but not in the ACE2-Bound conformation, we have some confidence that the compound will actually bind somewhere that potentially interferes with the spike’s interaction with its target receptor. Obviously, there are plenty of other structures we can now investigate just as quickly and across much larger datasets. To dig deeper into the results generated for several other structures comprising SARS-CoV-2 against the BindingDB FDA-approved dataset, visit the subdirectory on GitHub <a href="https://github.com/jbalma/pharml/tree/master/examples">here.</a></p> <p>In general, we observe PharML.Bind making overlapping predictions of compounds either in clinical trials for treatment of COVID-19 or which align with docking and MD simulations, suggesting further study is warranted for many more compounds than currently in clinical trials. For more information on how these compounds were generated, and how you can use PharML.Bind in your own research, please consult the Quick-Start Guide and GitHub examples page linked above.</p> <p>In this article, we showed how you can get started with plug-and-play drug discovery with the newly released Open-Therapeutics framework PharML.Bind. We reviewed how to use a pre-trained model on a new target structure (the spike protrusion of COVID-19) against a large database of FDA-approved drugs. By using an ensemble of models, we were able to assign confidence scores to each drug predicted to bind and use the results to zoom in on a few interesting FDA-approved drugs that might be viable for repurposing in the fight against COVID-19.</p> <p>Hopefully, sharing this information with you will help spark ideas on how you might apply this sort of machine learning to your activities. We will be building out more material on the HPE DEV site specific to AI and machine learning. Make sure you check back at <a href="/blog">HPE DEV</a> for more interesting blogs as we move forward.</p>https://developer.hpe.com/hpe-container-platform-rest-api-part-1-authenticating/https://developer.hpe.com/hpe-container-platform-rest-api-part-1-authenticating/Tue, 26 May 2020 03:08:40 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/image001-1590504102737.png" alt="image001"></p> <p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Businesses are challenged today with being able to run their existing monolithic applications alongside their cloud-native apps in hybrid cloud environments. The <a href="https://developer.hpe.com/platform/hpe-container-platform/home">HPE Container Platform</a> (HPE CP) uses container technology to make it simpler and more cost-effective to deploy, run and manage both cloud native microservices enterprise workloads and non-cloud native monolithic applications with containers. Businesses can take advantage of the HPE Container Platform for a variety of use cases, including machine learning (ML), data analytics, and DevOps workloads. Its abilities make the HPE Container Platform ideal for helping IT accelerate their application development and deployment on containers on-demand through a self-service portal.</p> <p>In this two-part blog series, I am going to discuss how the HPE Container Platform exposes a RESTful API that provides programmable access to capabilities via a self-service portal. I will then share some recent learnings and experience doing this.</p> <blockquote> <p>Note: This series does not cover how to perform IT administrative tasks through the HPE CP REST API.</p> </blockquote> <p>These blog posts are targeted at developers who want to get started with the REST API so they can interact with the HPE Container Platform programmatically. This series is also designed for solution architects who simply want to understand the product from a developer and data scientist’s perspective so they can discuss its capabilities with their customers’ developers and data analysts.</p> <p>In the first part of the series, you will interact with the HPE CP REST API using a handy graphical tool called <a href="https://www.postman.com/">Postman.</a> You will also have the opportunity to use cURL <a href="https://curl.haxx.se/">(Command-line URL),</a> the universal and well-appreciated command-line utility from the Linux community. In the second part, I will go a step further to explain how you can use this REST API to deploy containerized applications using a programmatic approach.</p> <p>If you are not already familiar with REST API calls and Postman, I encourage you to check out the <a href="/blog/understanding-api-basics-and-the-value-they-provide">Understanding API basics and the value they provide</a> article. It explains REST API concepts such as HTTP verbs you call against a REST API service, the headers and payloads, and how to use Postman to make REST API calls.</p> <h2>The HPE Container Platform High-Level Architecture</h2> <p>The diagram below depicts a simplified view of the physical architecture within the HPE Container Platform being deployed for this tutorial. It illustrates how you can interact programmatically with the HPE Container Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture2-1590462775163.png" alt="picture2"></p> <p>The HPE Controller Platform deployment includes the following key components:</p> <ul> <li>The <strong>Controller host</strong> manages all the hosts that comprise the HPE Container Platform deployment.</li> <li>The <strong>Kubernetes (K8s) hosts</strong> are under the direct control of the Controller host. These hosts can be grouped into one or more distinct Kubernetes clusters that run containerized applications.</li> <li>The <strong>Gateway host</strong> acts as a proxy server that carries client requests, i.e. HPE CP UI, REST API, K8s API (kubectl commands) to the HPE Container Platform controller, to one of the Kubernetes clusters, or to one of the containerized application services running in one of the Kubernetes clusters. Containerized application service endpoints are exposed outside the Kubernetes cluster to users via the gateway re-mapped ports.</li> <li>The <strong>Authentication Proxy</strong> handles user authentication and forwards authenticated K8s API traffic (kubectl commands) to the Kubernetes cluster master and returns any responses to the request back to the user.</li> <li>The <strong>HPE Data Fabric</strong> (a MapR File System) is a storage provider for persistent volumes for the containerized applications that require persistence of data. The default StorageClass is available out of the box from the HPE Data Fabric (MapR) using the HPE Container Storage Interface (CSI) driver for MapR.</li> </ul> <h2>The HPE Container Platform REST API Reference</h2> <p>The HPE CP REST API allows you to execute multiple actions programmatically, from performing administrative tasks like creating Kubernetes clusters to deploying applications for various use cases in a shared multi-tenant environment.</p> <p>Before you can call the HPE CP REST API, you need to know what calls can be placed. The REST API reference documentation describes each object type, along with the supported operations, request input parameters, response model, and response codes.</p> <p>To access the HPE CP REST API reference documentation, obtain the IP address or hostname of the HPE Container Platform <strong>Gateway</strong> host or <strong>Controller</strong> host from the platform administrator. Then, in a web browser, navigate to the following URL: <strong>http(s)://Gateway-or-Controller-IP-address-or-fqdn/apidocs</strong>. Access protocols for the HPE Container Platform REST API reference documentation varies depending on whether your platform has been configured to use HTTPS secure protocol (recommended) or non-secure HTTP protocol.</p> <h2>Session Authentication</h2> <p>With the exception of some REST API calls, most of the API calls you can do against the HPE CP REST API must be authenticated with a sort of token that is retrieved by sending a username, password, and tenant name to the HPE CP API server. The HPE Container Platform uses a <em>session location</em> to identify the <strong>working tenant context</strong> of a given REST API operation. The session location is then stored in the HTTP header of subsequent requests to perform Create, Read, Update, and Delete (CRUD) operations using HTTP verbs, such as POST, GET, PUT, PATCH, and DELETE.</p> <p>A tenant is a group of users created by the platform administrator. A tenant is allocated a quota of resources such as CPU, GPU, memory, storage, and Kubernetes clusters resources. A tenant can represent, for example, an office location, a business unit, a department, or a project. Tenant users can then deploy applications within the context of their tenant. Once a resource is actively in use by one tenant, it will not be shared with other tenants. The platform administrator assigns users roles (tenant Member or Tenant Admin) and tenant membership through either LDAP/AD authentication groups or local directory user accounts.</p> <p>A working context establishes the user identity, its tenant name, and role (member or admin). Based on this context, tenant users are granted privileges and permissions to create and manage resources for their tenant on Kubernetes clusters managed by HPE CP.</p> <p>You request an authentication <em>session location</em> by issuing an authentication request for a new login session, providing your username/password credentials and your tenant name in the JSON body.</p> <p>This is confirmed by looking at the HPE CP REST API reference documentation for session object: <strong>POST /api/v2/session</strong>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture3-1590462845389.png" alt="picture3"></p> <p>You then get the session location (in the form <em>/api/v2/session/sessionId</em>) from the JSON response header with a status <em>201 (created)</em>, which means that the session object has been created successfully as the result of the HTTP POST request. This is also confirmed by looking at the REST API reference guide, as you can see below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture4-1590462859828.png" alt="picture4"></p> <p>You then extract and save the session location value from the JSON response header. For each subsequent call, you set a new HTTP header with its key set to <strong>X-BDS-SESSION</strong> and its value set to the session location value. This will set the <strong>working tenant context</strong> for your request.</p> <p><strong>Now, let’s put it into action with Postman:</strong></p> <p>The REST API calls shown below explicitly specify JSON as the exchange format between the API client (Postman) and the API server (HPE Container Platform).</p> <p>The communication with REST API is HTTP or HTTP over HTTPS on port 8080. The communication protocol varies depending on whether your platform has been configured to use HTTPS secure protocol (recommended) or non-secure HTTP protocol.</p> <p>All the API calls are in the form:</p> <ul> <li>An HTTP verb such as GET, POST, DELETE, PATCH, UPDATE, PUT</li> <li>The target REST API object URL: <strong>http(s)://Gateway-IP-or-fqdn:8080/api/v2/object</strong></li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture5-1590462875550.png" alt="picture5"></p> <p>In the example here, I am acting as a tenant user and I request an authentication session location through a <strong>POST /api/v2/session</strong> API call. The user credentials and tenant name are provided in the request body using Postman Environment variables:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture6-1590464352997.png" alt="picture6"></p> <p>When you click the <strong>Send</strong> button, you get the response, <em>201 created</em>, in the response body, which means the session resource was successfully created.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture7-1590464474273.png" alt="picture7"></p> <p>The response header provides the resource path for the created session object in the form <code>/api/v2/session/SessionId</code>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture8-1590464521826.png" alt="picture8"></p> <p>All subsequent REST API calls will need to specify the <em>session location</em> resource path in their headers to use as <strong>working tenant context</strong>. The REST API call must include an <strong>X-BDS-SESSION</strong> header with the value of the session object’s resource path <code>/api/v2/session/SessionId</code> previously created. In the example below, the GET REST API call request will fetch information about the session you have just established with the HPE Container Platform as a tenant user:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture9-1590464546066.png" alt="picture9"></p> <p>As tenant user, the response body lists information about your session, such as your username, tenant name, and expiration time. Notice that the session location token will remain valid for 1440 minutes (or 24 hours), after which time you will have to establish a new login session.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture10-1590464591318.png" alt="picture10"></p> <h1>From Postman to code</h1> <p>You may ask yourself, how do these calls translate into code you can use in your own application? Postman can help with that thanks to its embedded code-generators. Imagine you have been experimenting with a POST session request in Postman and you would like to run the same POST call from <em>cURL</em> in a script or from other code languages. As shown in the picture below, Postman lets you generate snippets of code in various languages and frameworks that will help you do this. You can use the <strong>Code</strong> link under the blue <strong>Send</strong> button to open the <em>GENERATE CODE SNIPPETS</em> and select your preferred language or framework. In the rest of this blog, cURL is used as the preferred script language.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture11-1590464615471.png" alt="picture11"></p> <pre><code class="language-bash">curl –k –i –s --location --request POST 'http(s)://&#x3C;Gateway-IP-Address-or-fqdn>:8080/api/v2/session' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "&#x3C;YourUSername>", "password": "&#x3C;YourPassword>", "tenant_name": "&#x3C;YourTenantName>" }' </code></pre> <p>An example of the response header received is shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture12-1590464628659.png" alt="picture12"></p> <p>Extract the session location path value <code>/api/v2/session/&#x3C;sessionId></code> from the header response and use it in any subsequent REST API calls. As shown by the example below, you can use a <em>GET</em> REST API call for object <strong>/api/v2/session</strong> to fetch information about the session you have just established with the HPE Container Platform as a tenant user. You can combine <em>cURL</em> command with <strong>jq</strong> as the command line JSON processor to parse the JSON body responses and obtain a structured print of the output as shown here:</p> <pre><code class="language-bash">curl -k -s --location --request GET 'http(s)://&#x3C;Gateway-IP-Address-or-fqdn>:8080/api/v2/session' \ --header 'X-BDS-SESSION: /api/v2/session/&#x3C;SessionId>' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' | jq </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture13-1590464642369.png" alt="picture13"></p> <p>Although sessions have a time to live (TTL) of 24 hours, it is a best practice in REST API programming to cleanup and delete those sessions when done with your REST API calls. You can use a <em>DELETE</em> call to the target object <strong>/api/v2/session/SessionId</strong> to achieve this:</p> <pre><code class="language-bash">curl -k -i -s --request DELETE 'http(s)://&#x3C;Gateway-IP-Address-or-fqdn>:8080/api/v2/&#x3C;SessionId>' \ --header 'X-BDS-SESSION: /api/v2/session/&#x3C;SessionId>' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' </code></pre> <p>You have now learned the basics of programmatic access to the HPE Container Platform through its REST API. In <a href="/blog/hpe-container-platform-rest-api-part-2-deploying-containerized-applicati">the next article</a> in this series, I will discuss how you can deploy programmatically cloud native stateless, microservices-based applications and non-cloud native distributed, stateful applications within the context of Kubernetes clusters managed by the HPE Container Platform. You can stay up to date with the latest news from HPE DEV by <a href="https://developer.hpe.com/newsletter-signup">signing up for our monthly newsletter.</a> In it, you will receive more awesome developer and data scientist focused posts about the HPE Container Platform. You can also follow our community on <a href="https://twitter.com/HPE_Developer">Twitter</a> and join the conversation on our <a href="https://slack.hpedev.io/">HPE DEV Slack Channel.</a></p>https://developer.hpe.com/the-neuroscience-behind-a-design-system/https://developer.hpe.com/the-neuroscience-behind-a-design-system/Wed, 13 May 2020 14:53:52 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/taylor-cropped-3-1589386197151.png" alt="taylor cropped (3)"></p> <p>After joining the Hewlett Packard Enterprise (HPE) Experience Studio as a UI Developer this past July, my first main project was to work on the HPE Design System. The HPE Design System is a design, development, and research effort aimed at creating a consistent user experience across the breadth of software offered by (HPE). The HPE Design System will put in place a prescribed model for all HPE software developers to adhere to, optimizing the way HPE teams create user experiences for their customers.</p> <p>I graduated last May with a B.S. in Computational Neuroscience, so, out of interest – or maybe habit – I’m constantly thinking about the way the brain interprets and perceives our experiences. Since I started work on the HPE Design System, I’ve begun to think more broadly about <em>why</em> we invest time in building one. Because, there’s a lot more to a design system than the slick, tightly-wrapped feeling that accompanies the term. A design system is more than just a library of components and design standards; it’s an entire visual and interactive language that engages with a user’s emotion, attention, and decision-making abilities. The way I see it, the benefits of creating a design system are grounded in three important aspects of cognition: emotion, attention, and decision making.</p> <h2>Emotion and memory</h2> <p>Our memories are incredibly emotionally driven. The saying, “Someone might not remember what you say to them, but they’ll remember how you made them feel” holds more weight than a simple push for thoughtful interactions. In a similar way, in regards to applications, a user may not always remember the words on the page or the exact images used, but they will likely remember the feeling of that experience. Like a go-to friend whose demeanor I know I can rely on in times of stress or joy, a design system allows various applications and products to form a united experience that establishes a relationship of trust with the user regardless of if this is their first time using an application or their hundredth.</p> <h2>Attention is limited</h2> <p>Humans are incredibly smart, but our attentional capacity is, in fact, quite limited. Working memory, which refers to the information that we are currently attending to or “working with”, has a capacity defined roughly at 7 ± 2 items and a duration of about 20 seconds. However, our attention can be aided by consistency and the ability to group information.</p> <p>A well thought out design system helps a user focus their attention and break information into digestible pieces. By developing consistency in how information is presented and how a user is able to engage with that information, a design system liberates the user in a way that allows them to focus on what they’re trying to do as opposed to how to do it.</p> <h2>Decision-making</h2> <p>Ultimately, a user engages with an application to complete a task. Maybe that task is to check the status of a server or to manage user permissions, or maybe it’s just to explore a new topic. Regardless of the user’s goal, a design system enables and empowers the user in regards to this activity. By honing in on the ways emotion and attention drive our perception, a design system allows a user to make quicker, more accurate decisions about how to quickly achieve what the user set out to do.</p> <h2>Helping teams help customers</h2> <p>Beyond the end user’s experience, a design system helps expedite the design and development cycle for teams using it. In a sense, a design system is like bumpers on a bowling lane that gently redirect an application towards the ideal end deliverable. It provides templates and patterns to point them in the right direction, as well as resources and a community to engage with, amongst other things. A design system should be seen as being just as valuable to the internal teams designing/developing it as it is to the end user. It should be an aid to internal development teams in creating the best possible experience for the end user.</p> <p>By establishing a consistent visual and interaction language, the HPE Design System enables experiences to be crafted with uncompromising integrity. These benefits are not found just at the surface level but grounded in the ways our brains perceive the world.</p> <p>In conclusion, I’m really excited about the work I am doing on the HPE Design System and the improved experiences it will create for HPE customers as well as the internal teams using it. A design system leverages the knowledge we have about emotion, attention, and decision making to optimize the experience users have with an application. Check out the work that has started on the <a href="https://design-system.hpe.design/">design system.</a> For other updates from the HPE DEV team, be sure to keep visiting <a href="/blog">HPE DEV blog.</a></p>https://developer.hpe.com/introducing-the-hpe-nimble-storage-sdk-for-python/https://developer.hpe.com/introducing-the-hpe-nimble-storage-sdk-for-python/Fri, 08 May 2020 22:17:59 GMT<p>Python is by far one of the most approachable programming languages available today. Easy to read semantics, a humongous library for integrating with anything under the sun and best of all, it’s all open source, available on every popular operating system platform. Today, Hewlett Packard Enterprise (HPE) releases HPE Nimble Storage SDK for Python 1.0.0. It allows customers to extract even more value from their HPE Nimble Storage array by abstracting functionality into larger frameworks, whether it’s for custom automation, resource reporting, conformance adherence or any data the business might need to extract from the storage system in a programmatic fashion.</p> <p>The software development kit (SDK) includes all public application programming interfaces (API) available via the Nimble OS REST API (REST stands for REpresentational State Transfer). It is now trivial to write both simple or intricate applications interfacing with Nimble OS. Historically, our Python users have used the popular Requests library to manipulate API resources on the array and we’ve now abstracted away that interaction to ultimately make it simpler and more structured to interface with Nimble OS.</p> <p>In this blog post, we’ll get you started with the SDK and show you how to write a simple asset reporting program to copy and paste into a spreadsheet. In a more realistic scenario, this data would’ve been inserted into an inventory and asset tracking system.</p> <h1>Getting started</h1> <p>Like all Python libraries, the HPE Nimble Storage SDK for Python is hosted on PyPi, the Python Package Index. Most Python distributions come with the command line (CLI) tool, <code>pip</code>, to manage installations from PyPi. This tutorial assumes <code>pip</code> is already installed.</p> <p>From the CLI, install the SDK.</p> <pre><code class="language-bash">pip install nimble-sdk </code></pre> <p><strong>Note</strong>: Python 3.6 or newer is required. If multiple Python distributions are installed on your system, make sure Python 3 is installed or suffix Python commands with a ‘3’, like <code>pip3</code> and <code>python3</code>.</p> <p>Let’s run an interactive example against a Nimble array and pull out a list of drives in the array.</p> <pre><code class="language-python">python3 Python 3.7.7 (default, Mar 10 2020, 15:43:03) [Clang 11.0.0 (clang-1100.0.33.17)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from nimbleclient import NimOSClient >>> import pprint >>> client = NimOSClient("192.168.1.1", "admin", "admin”) >>> pprint.pprint(client.disks.list()) [&#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000100)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000200)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000300)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000400)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000500)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000600)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000700)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000800)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000900)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000a00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000b00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000c00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000d00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000e00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000000f00)>, &#x3C;Disk(id=2c49686580b78e0b160001000000000a0000001000)>] </code></pre> <p>Let’s pull out more info about a particular drive, in the same session.</p> <pre><code class="language-python">>>> pprint.pprint(client.disks.get("2c49686580b78e0b160001000000000a0000000c00").attrs) {'array_id': '0949686580b78e0b16000000000000000000000001', 'array_name': 'nva-test', 'bank': 0, 'block_type': 'block_unknown', 'disk_internal_stat_1': '00e29a628e', 'firmware_version': '1.0', 'hba': 2, 'id': '2c49686580b78e0b160001000000000a0000000c00', 'is_dfc': False, 'model': 'VMware Virtual S', 'path': '/dev/sdm', 'port': 8, 'raid_id': 10, 'raid_resync_average_speed': 0, 'raid_resync_current_speed': 0, 'raid_resync_percent': 100, 'raid_state': 'okay', 'serial': '/dev/sdm', 'shelf_id': '2d49686580b78e0b16000000010000637300000013', 'shelf_location': 'A.0', 'shelf_location_id': 0, 'shelf_serial': 'cs-19d266', 'size': 39728447488, 'slot': 12, 'smart_attribute_list': [], 'state': 'in use', 'type': 'hdd', 'vendor': 'Nimble', 'vshelf_id': 0} </code></pre> <p>To get an idea of what APIs are available, hitting <code>&#x3C;TAB></code> in an interactive session, just before entering which resource to manipulate will list the available APIs.</p> <pre><code class="language-python">>>> client.&#x3C;TAB> client.access_control_records client.folders client.snapshot_collections client.active_directory_memberships client.groups client.snapshots client.alarms client.initiator_groups client.software_versions client.application_categories client.initiators client.space_domains client.application_servers client.jobs client.subnets client.arrays client.key_managers client.support client.audit_log client.master_key client.tokens client.chap_users client.network_configs client.user_groups client.controllers client.network_interfaces client.user_policies client.disks client.performance_policies client.users client.events client.pools client.versions client.fibre_channel_configs client.protection_schedules client.volume_collections client.fibre_channel_initiator_aliases client.protection_templates client.volumes client.fibre_channel_interfaces client.protocol_endpoints client.witnesses client.fibre_channel_ports client.replication_partners client.fibre_channel_sessions client.shelves </code></pre> <p>How to interact with each of these resources is available in the <a href="https://hpe-storage.github.io/nimble-python-sdk">documentation</a> hosted on GitHub.</p> <h1>Hello World</h1> <p>Running Python interactively is practical to use as a sidekick when developing a new script to find attributes and other syntax minutia. Let’s write a real “Hello World” Python program that extracts some array meta data that is used to track inventory.</p> <pre><code class="language-bash">#!/usr/bin/env python3 # Import the required modules import argparse from nimbleclient import NimOSClient # The NimbleInventory class class NimbleInventory: def __init__(self, username, password, arrays): self.inventory = {} self.username = username self.password = password self.arrays = arrays self.fetch() # Loop over requested arrays and gather the data def fetch(self): for array in self.arrays: try: api = NimOSClient(array, self.username, self.password) entry = api.arrays.get() self.inventory[entry.attrs.get('name')] = [entry.attrs.get('version'), entry.attrs.get('extended_model'), entry.attrs.get('serial')] except: self.inventory[array] = ['n/a', 'n/a', 'n/a'] # Print the list of arrays in a spreadsheet friendly manner def report(self): for xlsout in self.inventory: print ("{name}\t{version}\t{model}\t{serial}".format(name=xlsout, version=self.inventory[xlsout][0], model=self.inventory[xlsout][1], serial=self.inventory[xlsout][2])) # If called directly if __name__ == '__main__': # Parse CLI arguments parser = argparse.ArgumentParser(description='Nimble array inspector') parser.add_argument('--username', required=True, type=str, help='Nimble OS username') parser.add_argument('--password', required=True, type=str, help='Nimble OS password') parser.add_argument('--array', required=True, type=str, help='Nimble array hostname or IP address', nargs='+') args = parser.parse_args() # New inventory data = NimbleInventory(args.username, args.password, args.array) # Report data.report() </code></pre> <p>Running the program against a set of arrays will produce something like the following (includes the syntax):</p> <pre><code class="language-bash">./nimble_inventory.py --username admin --password admin --array nimble-array1 nimble-array2 nimble-array3 nimble-array4 nimble-array1 5.1.3.100-668356-opt Virtual-6G-12T-320F cs-XXXXX nimble-array2 5.1.4.0-683149-opt AF3000-2P-11T AF-XXXXX nimble-array3 5.1.3.0-663613-opt CS3000-2P-21T-1440F AF-XXXXX nimble-array4 5.0.7.300-644174-opt AF5000-2F-46T AF-XXXXX </code></pre> <blockquote> <p><strong>Note:</strong> If this would’ve been a real-world use case, a read-only user account would’ve been appropriate to set up on the targeted systems.</p> </blockquote> <h1>Next steps</h1> <p>The HPE Nimble Storage SDK for Python source code and documentation links have been added to the HPE DEV portal <a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">platform page for HPE Nimble Storage</a>, which makes it easy to find in the future.</p> <ul> <li><a href="https://github.com/hpe-storage/nimble-python-sdk">Source code</a> on GitHub</li> <li><a href="https://hpe-storage.github.io/nimble-python-sdk">Documentation</a> hosted on GitHub Pages</li> <li><a href="https://pypi.org/project/nimble-sdk/">Python package</a> hosted on PyPi</li> </ul> <p>Come hang out with us on our Slack community. HPE employees may sign up directly at <a href="https://hpedev.slack.com">hpedev.slack.com</a>, partners and customers need to signup at <a href="https://slack.hpedev.io">slack.hpedev.io</a> first. Also, watch this space for future releases around software development kits for HPE Nimble Storage. There’s plenty of really exciting projects in the works. Stay tuned!</p>https://developer.hpe.com/wow-part-2-communicating-with-customers-and-constructors/https://developer.hpe.com/wow-part-2-communicating-with-customers-and-constructors/Fri, 08 May 2020 16:40:10 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/title-slide-1588956219016.jpg" alt="title slide"></p> <h2>What is WOW?</h2> <p>In my <a href="/blog/wow-a-practiced-and-perfected-design-process-part-1-uncovering-the-merit">previous post,</a> I explained that many businesses struggle to understand the true value designers bring to a project. To help UX designers overcome this issue, I’m sharing a methodology our group developed called WOW (Why On What with customers and constructors). This workflow demonstrates how to quantify the value of the design process and provides enterprise UX designers with a practiced and perfected path to achieve success. To briefly summarize, the WOW methodology helps creative teams focus on four important stages of a project:</p> <ol> <li>Uncover why the business (and this particular project) needs a UX</li> <li>Involve the customer early in the design phase</li> <li>Ensure the constructor (developer) has the right information during design implementation</li> <li>Convey the business value of the design</li> </ol> <p>In part one of this blog series, I discussed the first part of the workflow – addressing why businesses should embrace UX design. Here, in part two, I will cover the next two steps of the process – focusing on communications with the customer (user) of the UX design and the constructor (developer). Keeping close communications with these two personas is key to any successful UX design.</p> <h2>Customer - Right brain of product</h2> <p>An essential ingredient in the creative UX design process is customer communications. You can think of this as flexing the right half of the brain during the creative process. This is where empathy really starts to come into play. Working with the customer directly helps bring the right needs, insights, and feel to a product. Customer interactions can help remove opinioned workflows and bring clarity to the early phases of design.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-2-slide-2-_-after-title-slide-1588956251003.jpg" alt="part 2 slide 2 _ after title slide"></p> <p>Customer interactions can be done through face-to-face or remote meetings. With the right set of questions, a designer can quickly gather enough information to make designs move ahead without ambiguity. During your interactions, help customers keep a “feel as if you are using the product” mindset and make sure you ask the right questions to ensure clarity. Your work here is to help them understand that you truly care about how they feel about using the product. Remember, it’s not so important at this point to consider “how” things are implemented. You’ll address the “how” later.</p> <p>Because customers use a UI (user interface) or real screens to interact with products, you’ll use wireframes as the intermediary steps to help create the prescribed vision of the UX and achieve the final UI. You can then take advantage of these wireframes and use them to help communicate with the constructor.</p> <h2>Constructor - Left brain of product</h2> <p>Another essential ingredient in the UX design process is ensuring clear and accurate communications with the constructors of the product, aka the developers. This is like working the left half of the brain, where you need to be more concerned about technical feasibility and how things work from the developer’s point of view.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-2-slide-3-1588956289625.jpg" alt="part 2 slide 3"></p> <p>Spending the time to ensure a developer understands the interaction, component usage, and other subtle aspects of a design is very important for the success of the UX. This step in the process is called the designer-developer handoff. Oftentimes this important phase is missed in the design process because of time constraints, resource dependencies, etc. This phase is so important that it should truly be a non-dismissible requirement. This is why we have integrated it as a distinct step in the WOW UX-flow.</p> <p>A strong designer–developer relationship based on mutual understanding is key for the overall success of a UX. Navigating this step carefully will help the designer identify any design shortcoming far before production release. It will also help the developer better understand the requirements and develop faster with less iterations. Design system approaches, like being three sprints ahead (3-SA), employing iterative development, and using UI templates, are all particularly helpful in these designer-developer handoffs.</p> <h2>Summary</h2> <p>By this point in using a UX design process, the designer has already added value to the product. Potential functionality gaps have been identified and competitive features have been added. Relying on empathy, the customer has been heard and the developer has all the information required to achieve the desired result. What’s left is using the tools available to demonstrate the business value achieved through the UX. In Part 3, I’ll discuss more on wireframes and their use in addressing the final stage of WOW, conveying the overall business value of the design.</p> <p>Again, please feel free to reach out to me <a href="https://twitter.com/uxwithparul">@uxwithparul</a> if you have any questions. You’ll be able to find subsequent posts in this series and view other informative blogs at <a href="/blog">HPE DEV.</a></p>https://developer.hpe.com/2020-May-01/https://developer.hpe.com/2020-May-01/Fri, 01 May 2020 05:00:00 GMThttps://developer.hpe.com/tutorial-how-to-get-started-with-the-hpe-csi-driver-and-hpe-primera-and-/https://developer.hpe.com/tutorial-how-to-get-started-with-the-hpe-csi-driver-and-hpe-primera-and-/Thu, 30 Apr 2020 16:12:11 GMT<h1>Tutorial: How to get started with the HPE CSI Driver and HPE Primera and 3PAR</h1> <p>With the release of the HPE Container Storage Interface (CSI) driver for Kubernetes back in January, HPE has been hard at work on integrating additional platforms into the CSI driver framework. Initially the HPE CSI Driver for Kubernetes only supported Nimble Storage now with the latest v1.1.1 comes <a href="https://community.hpe.com/t5/hpe-storage-tech-insiders/hpe-csi-driver-for-kubernetes-1-1-1-and-hpe-3par-and-hpe-primera/ba-p/7086675">support for HPE Primera and 3PAR arrays</a>. In this tutorial, I will walk you through the steps of deploying the CSI driver with HPE Primera and then we will deploy a Wordpress site using persistent storage. With that, let's get going!</p> <h2>Assumptions</h2> <p>I will be starting with an existing Kubernetes cluster. This can be a fresh install with <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/">kubeadm</a> or <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubespray/">kubespray</a>. The deployment of Kubernetes is outside of the scope of this document. If you don't have a cluster up and running, I recommend that you get started there.</p> <p>Also I am assuming <code>kubectl</code> is installed and configured to communicate with the cluster and Helm 3 to deploy the HPE CSI Driver for Kubernetes. If not, here are some good resources to check out for assistance <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">Install and Set Up kubectl</a> and <a href="https://helm.sh/docs/">https://helm.sh/docs/</a> to get setup.</p> <h2>Deploying the HPE CSI Driver and HPE 3PAR and Primera Container Storage Provider with Helm</h2> <p>To get started with the deployment of the HPE CSI Driver, check out the <a href="https://scod.hpedev.io/">HPE Storage Container Orchestrator Documentation</a> (SCOD for short) site. SCOD is an umbrella documentation project for all Kubernetes and Docker integrations for HPE primary storage tailored for IT Ops, developers and partners. It includes HPE 3PAR, HPE Primera, HPE Cloud Volumes and HPE Nimble Storage.</p> <p>The HPE CSI Driver is deployed by using industry standard means, either a Helm chart or an Operator. For this tutorial, I will be using Helm to the deploy the CSI driver.</p> <p>The official Helm chart for the HPE CSI Driver for Kubernetes is hosted on <a href="https://hub.helm.sh/charts/hpe-storage/hpe-csi-driver">hub.helm.sh</a>. There, you will find the configuration and installation instructions for the chart.</p> <p>The first step of installing the HPE CSI Driver is creating the <strong>values.yaml</strong> file.</p> <p>Please refer to this sample <a href="https://github.com/hpe-storage/co-deployments/tree/master/helm/values/csi-driver">values.yaml</a> file.</p> <pre><code class="language-bash">vi primera-values.yaml </code></pre> <p>Copy the following into the file. Make sure to set the <strong>backendType: primera3par</strong> and the <strong>backend</strong> to the array IP along with the array username and password.</p> <pre><code class="language-bash"># HPE backend storage type (nimble, primera3par) backendType: primera3par secret: # parameters for specified backendType (nimble, primera3par) create: true backend: 192.168.1.10 username: 3paradm password: 3pardata servicePort: "8080" ## For creating the StorageClass automatically: storageClass: create: false </code></pre> <blockquote> <p><strong>NOTE:</strong> <br /> The user specified will need at a minimum the <strong>edit</strong> role on the array.</p> </blockquote> <p>Save and exit.</p> <blockquote> <p><strong>IMPORTANT</strong><br />Deploying the HPE CSI Driver with the HPE 3PAR and Primera CSP currently doesn't support the creation of the <strong>default</strong> <code>StorageClass</code> in the Helm chart. Make sure to set <strong>create: false</strong> or omit the <code>StorageClass</code> section.</p> </blockquote> <h3>Installing the chart</h3> <p>To install the chart with the name hpe-csi:</p> <p>Add the HPE CSI Driver for Kubernetes helm repo:</p> <pre><code class="language-bash">helm repo add hpe https://hpe-storage.github.io/co-deployments helm repo update </code></pre> <p>Install the latest chart:</p> <pre><code class="language-bash">helm install hpe-csi hpe/hpe-csi-driver --namespace kube-system -f primera-values.yaml </code></pre> <p>Wait a few minutes as the deployment finishes.</p> <p>Verify that everything is up and running correctly with the listing out the pods.</p> <pre><code class="language-bash">kubectl get pods --all-namespaces -l 'app in (primera3par-csp, hpe-csi-node, hpe-csi-controller)' NAMESPACE NAME READY STATUS RESTARTS AGE kube-system hpe-csi-controller-84d8569476-vt7xg 5/5 Running 0 13m kube-system hpe-csi-node-s4c8z 2/2 Running 0 13m kube-system primera3par-csp-66f775b555-2qclg 1/1 Running 0 13m </code></pre> <pre><code class="language-bash">kubectl get secret -n kube-system | grep primera3par primera3par-secret Opaque 5 13m </code></pre> <p>If all of the components show in <code>Running</code> state, then the HPE CSI driver and the HPE 3PAR and Primera Container Storage Provider has been successfully deployed.</p> <h2>Using the HPE CSI Driver and HPE 3PAR and Primera Container Storage Provider</h2> <p>Now, let's validate the deployment by creating a <code>StorageClass</code>, <code>PersistentVolumeClaim</code> and deploy a Wordpress site.</p> <p>We need to create a <code>StorageClass</code> API object using the HPE CSI driver, along with the parameters specific to HPE 3PAR and Primera CSP, as well as the <code>Secret</code> used for the <strong>primera3par</strong> backend. For a full list of supported parameters, please refer to the <a href="https://scod.hpedev.io/container_storage_provider/hpe_3par_primera/index.html">CSP specific documentation</a>.</p> <p>The below YAML declarations are meant to be created with <code>kubectl create</code>. Either copy the content to a file on the host where <code>kubectl</code> is being executed, or copy &#x26; paste into the terminal, like this:</p> <pre><code class="language-bash">kubectl create -f- &#x3C; paste the YAML > ^D (CTRL + D) </code></pre> <p>Create a <code>StorageClass</code> API object for a Primera Data Reduction volume.</p> <pre><code class="language-yaml">--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: primera-reduce-sc provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/provisioner-secret-name: primera3par-secret csi.storage.k8s.io/provisioner-secret-namespace: kube-system csi.storage.k8s.io/controller-publish-secret-name: primera3par-secret csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: primera3par-secret csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: primera3par-secret csi.storage.k8s.io/node-stage-secret-namespace: kube-system # Uncomment for k8s 1.15 for resize support csi.storage.k8s.io/controller-expand-secret-name: primera3par-secret csi.storage.k8s.io/controller-expand-secret-namespace: kube-system cpg: SSD_r6 provisioning_type: reduce accessProtocol: fc </code></pre> <p>Create a <code>PersistentVolumeClaim</code> for MariaDB for use by Wordpress. This object creates a <code>PersistentVolume</code> as defined. Make sure to reference the correct <code>.spec.storageClassName</code>.</p> <pre><code class="language-yaml">--- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: data-my-wordpress-mariadb-0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: primera-reduce-sc </code></pre> <p>Next, let's make another for the Wordpress application.</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-wordpress spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: primera-reduce-sc </code></pre> <p>Let's again verify the <code>PersistentVolume</code> were created successfully.</p> <pre><code class="language-markdown">kubectl get pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-my-wordpress-mariadb-0 Bound pvc-1abdb7d7-374e-45b3-8fa1-534131ec7ec6 50Gi RWO primera-reduce-sc 1m my-wordpress Bound pvc-ff6dc8fd-2b14-4726-b608-be8b27485603 20Gi RWO primera-reduce-sc 1m </code></pre> <p>The above output means that the HPE CSI Driver successfully provisioned a new volume based upon the <strong>primera-reduce-sc</strong> <code>StorageClass</code>. The volume is not attached to any node yet. It will only be attached to a node once a scheduled workload requests the <code>PersistentVolumeClaim</code> <strong>pvc-nginx</strong>.</p> <p>Now, let's use Helm to deploy Wordpress using the <code>PVC</code> created previously. When Wordpress is deployed, the volumes will be attached, formatted and mounted.</p> <p>The first step is to add the Wordpress chart.</p> <pre><code class="language-bash">helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update helm search repo bitnami/wordpress NAME CHART VERSION APP VERSION DESCRIPTION bitnami/wordpress 9.2.1 5.4.0 Web publishing platform for building blogs and ... </code></pre> <p>Deploy Wordpress by setting <code>persistence.existingClaim=&#x3C;existing_PVC></code> to the <code>PVC</code> created in the previous step.</p> <pre><code class="language-bash">helm install my-wordpress bitnami/wordpress --version 9.2.1 --set service.type=ClusterIP,wordpressUsername=admin,wordpressPassword=adminpassword,mariadb.mariadbRootPassword=secretpassword,persistence.existingClaim=my-wordpress,allowEmptyPassword=false </code></pre> <p>Check to verify that Wordpress and MariaDB were deployed and are in the <strong>Running</strong> state. This may take a few minutes.</p> <pre><code class="language-bash">kubectl get pods NAME READY STATUS RESTARTS AGE my-wordpress-69b7976c85-9mfjv 1/1 Running 0 2m my-wordpress-mariadb-0 1/1 Running 0 2m </code></pre> <p>Finally lets take a look at the Wordpress site. You can use <code>kubectl port-forward</code> to access the Wordpress application from within the Kubernetes cluster to verify everything is working correctly.</p> <pre><code class="language-bash">kubectl port-forward svc/my-wordpress 80:80 Forwarding from 127.0.0.1:80 -> 8080 Forwarding from [::1]:80 -> 8080 </code></pre> <blockquote> <p><strong>NOTE:</strong><br />If you have something already running locally on port 80, modify the port-forward to an unused port (i.e. 5000:80).</p> </blockquote> <p>Open a browser on your workstation to <strong><a href="http://127.0.0.1">http://127.0.0.1</a></strong> and you should see, <strong>"Hello World!"</strong>. Access the admin console at: <strong><a href="http://127.0.0.1/admin">http://127.0.0.1/admin</a></strong> using the user/password used to deploy the Helm Chart. Happy Blogging!</p> <h2>Next steps</h2> <p>Stay tuned to HPE DEV for future blogs regarding the HPE CSI Driver for Kubernetes. In the meantime, if you want to learn more about Kubernetes, CSI and the integration with HPE storage products, you can find a ton of Resources out on <a href="https://scod.hpedev.io/">SCOD</a>! If you are already on Slack or an HPE employee, connect with us on <a href="https://hpedev.slack.com/">Slack</a>. If you are a new user, signup at <a href="https://slack.hpedev.io/">slack.hpedev.io</a>. We hang out in #kubernetes, #nimblestorage and #3par-primera.</p>https://developer.hpe.com/multiple-hpe-efforts-help-fight-covid-19/https://developer.hpe.com/multiple-hpe-efforts-help-fight-covid-19/Tue, 28 Apr 2020 15:22:08 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/title-slide-for-covid-blog-1588088234878.jpg" alt="title slide for covid blog"></p> <p>Scientists around the world are racing to determine how to defeat COVID-19. Technology companies, like Hewlett Packard Enterprise (HPE), are in a unique position to assist. As a company whose purpose is to advance the way people live and work, HPE is proud that its technology and its employees’ expertise are being called into action to help organizations address the COVID-19 crisis. Here are four ways that HPE’s technology and talent are on the front lines in the fight against COVID-19, and how you can help.</p> <h2>Key member of the COVID-19 High Performance Computing Consortium</h2> <p>Given enough compute power, scientists can accelerate finding a resolution to the COVID-19 pandemic. Computing resources help researchers collect, process, and analyze the massive amounts of data required to understand and model the virus’ genetic coding. Epidemiological data scientists rely on supercomputing processing to help them understand disease conditions and distribution patterns within the population. This data is essential to identify risk factors and determine health-related policies.</p> <p>As part of the <a href="https://covid19-hpc-consortium.org/">COVID-19 High Performance Computing Consortium,</a> HPE joins other key industry, government, and academic partners in providing COVID-19 researchers with access to the world’s most powerful high-performance computing resources. One way consortium members are helping is by offering those with approved COVID-19 related research proposals free access to technology resources required to find a cure for the virus. More specifics can be found in the <a href="https://www.whitehouse.gov/briefings-statements/white-house-announces-new-partnership-unleash-u-s-supercomputing-resources-fight-covid-19/">White House Announces New Partnership to Unleash U.S. Supercomputing Resources to Fight COVID-19</a> article.</p> <h2>HPE AI/ML projects could lead to quicker cures overall</h2> <p>With the acquisition of supercomputing leader, Cray Inc., HPE added powerful computing resources to its arsenal. HPE DEV community member, Sreenivas Rangan Sukumar, explains that the HPE artificial intelligence (AI) / machine learning (ML) team currently has several projects in the works helping scientists find treatments. Three projects, in particular, may set the basis for an end-to-end pipeline by which cures for numerous diseases can be found more quickly than ever before.</p> <p>The first project, <a href="https://github.com/jbalma/pharml">PharML.Bind,</a> is an AI-driven approach that screens databases of potential drug compounds. It works by feeding supercomputer-based simulations of molecular dynamics (docking) to validate chemicals as potential targets for a vaccine or drug. Current bio-physical molecular docking is expensive and takes several hours in simulation testing for each potential chemical, which slows results. In contrast, PharML.Bind is computationally affordable and significantly faster in predicting the binding potential for a chemical to a target virus protein.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/screen-shot-2020-04-21-at-24215-pm-1588088160374.PNG" alt="screen shot 2020 04 21 at 2.42.15 pm"></p> <p>Another project addresses the treatment phase of the COVID-19 pandemic. This project focuses on using AI to augment human researchers in validating a screen by connecting-the-dots across literature and curated-databases (such as Biosamples, Uniprot, Clinical Trials, DrugBank, PubChem etc.). The project uses the Cray Graph Engine, <a href="https://www.cray.com/blog/cray-graph-engine-takes-trillion-triples/">the world’s fastest graph database,</a> to host, query, and reason with a knowledgebase of 100+ billion facts in a few seconds – something that otherwise would have taken months. Now using a supercomputer, researchers have access to this integrated body of knowledge – all in one place.</p> <p>Finally, several HPE engineers and developers are working on the <a href="https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/data">White House COVID-19 “Kaggle” Challenge.</a> This urgent call-to-action is asking the world’s AI experts for assistance in answering key scientific questions related to COVID-19. These questions include: “What is known about transmission, incubation, and environmental stability?”, “What do we know about COVID-19 risk factors?”, and “What worked and what didn’t?”</p> <p>While each of these COVID-19 specific AI/ML efforts will provide significant contributions to the community individually, the combined use of these projects can assist healthcare efforts far beyond this immediate crisis. As Sreenivas Rangan Sukumar points out, accelerated chemical-structure discovery and literature-based evidence discovery means faster cures for many other illnesses.</p> <h2>The Folding@Home effort</h2> <p>In addition to HPE offering corporate resources, HPE employees are personally doing what they can to help. Numerous engineers and developers are volunteering CPU time from their home computers as part of a distributed computing project called <a href="https://foldingathome.org/">Folding@Home.</a> An open source project initially focused on fighting different types of cancer and neurological diseases, Folding@Home now has a major focus on COVID-19.</p> <p>Greg Bowman, PhD, an assistant professor of biochemistry and molecular biophysics at Washington University School of Medicine in St. Louis, leads this open source activity. The goal is to simulate the dynamics of COVID-19 proteins in the hunt for new therapeutic opportunities. <a href="https://foldingathome.org/2020/03/15/coronavirus-what-were-doing-and-how-you-can-help-in-simple-terms/">By unravelling the mysteries of protein dynamics, including the folding process,</a> Dr. Bowman hopes to better understand how viral proteins work and design therapies to stop the coronavirus.</p> <p>As pointed out in HPE DEV member Michael Mattsson’s <a href="https://datamattsson.tumblr.com/post/613349069061046272/lets-hack-covid-19">blog,</a> a couple of HPE community teams are already actively supporting Folding@Home within HPE: HPE DEV (246814) and WeAreHPE (247332). The worldwide response to this effort has been tremendous. Folding@Home recently clocked 470 PFLOPs of compute power, which is more than two times higher than the peak performance of ORNL's Summit, the world's most powerful supercomputer. In just over a month, the WeAreHPE team placed in the top 500 in terms of work unit count, ranking 470 out of 251,480.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/folding-at-home-1-1588088181250.png" alt="folding at home 1"></p> <h2>Networking with HPE Aruba and Airheads volunteers</h2> <p>In addition to compute power, network infrastructure and the skills required to set it up are also needed to address this crisis. Many hospitals are struggling to handle the patient load and are building temporary COVID-related healthcare facilities.</p> <p>To help healthcare organizations set up quickly, Aruba created the Airheads Volunteer Corps, an opt-in registry of volunteer network engineers ready to assist in the build out of networks for medical facilities battling this pandemic. In creating this registry, Aruba connects those in need of IT assistance, with those who have the ability to help. HPE is also supporting this effort by donating thousands of secure connectivity kits. Learn more <a href="https://community.arubanetworks.com/t5/Community-Matters-Blog/Airheads-Volunteer-Corps-and-Healthcare-Connectivity-Bundles/ba-p/645495">here.</a></p> <p>From Genoa, Italy, Aruba’s Stefano Brioschi shared how he and Chiara Cipollini, HPE account manager for the Italian cruise market, volunteered time and equipment to build a Wi-Fi network on a passenger vessel. The Grandi Navi Veloci (GNV) <a href="https://www.seatrade-cruise.com/news/splendid-case-study-how-first-passenger-ship-was-transformed-coronavirus-relief">Splendid, was being turned into a fully-equipped hospital ship,</a> but the ship’s iron construction prevented adequate mobile connectivity. The two connected with local partner, Mantero Sistemi, to deploy a wireless infrastructure to support communications between ambulance crews, doctors, medical devices, the ship’s crew, and patients.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/aruba-3-1588088198966.jpg" alt="aruba 3"></p> <p>In only 5 days and working with a team of 10 people, they were able to create a network composed of 70 access points, 10 switches, and over 4 kilometers of UTP cable. Through this effort, they provided Wi-Fi coverage across the various decks of the ship, helping to form a controlled facility where patients could recover before returning to their homes.</p> <h2>Join the effort</h2> <p>The HPE DEV team supports many of these efforts. Interested in volunteering your expertise to help find a cure? Register to our <a href="https://slack.hpedev.io/">Slack</a> workspace and join us on the <a href="https://hpedev.slack.com/archives/CSA0R2T7B">#hpedev-volunteers</a> channel to learn more about how you can join the HPE DEV Folding@home team (TeamID: 246814). If you’re interested in participating in the Aruba Airheads opt-in registry, you can <a href="https://connect.arubanetworks.com/Airheads_Volunteer_Corps">volunteer here.</a></p>https://developer.hpe.com/covid-19-limits-accessibility-free-offer-for-hpe-ilo-advanced-opens-it-u/https://developer.hpe.com/covid-19-limits-accessibility-free-offer-for-hpe-ilo-advanced-opens-it-u/Thu, 23 Apr 2020 15:56:14 GMT<p>COVID-19 has introduced a variety of new challenges for many HPE customers and partners, such as closed offices, inaccessible server rooms, and/or work-from-home mandates. Like many businesses across the globe, you may be facing the need to quickly adapt and transform, in order to continue to serve your customers’ needs. If you are an HPE customer or partner (or plan to be), know that we are here to support you, so you can focus your efforts on your customers.</p> <p>HPE is here to help. As of now, HPE iLO Advanced is free for all HPE customers and partners through 2020. Configure, monitor, and update HPE servers seamlessly from anywhere, TODAY. <a href="https://community.hpe.com/t5/Servers-The-Right-Compute/HPE-iLO-Advanced-is-now-FREE-for-all-HPE-customers-and-partners/ba-p/7084121#.Xp3R9MhKg2w">Read more here</a> and learn how to claim your free license.</p>https://developer.hpe.com/top-13-capabilities-within-spiffe-and-spire-released-in-2019/https://developer.hpe.com/top-13-capabilities-within-spiffe-and-spire-released-in-2019/Tue, 21 Apr 2020 00:42:07 GMT<p>With the acquisition of Scytale this past February, Hewlett Packard Enterprise (HPE) acquired a seasoned team of experts in cloud-native security and zero-trust networking. The Scytale team is recognized as founding contributors to Cloud Native Foundation’s (CNCF’s) SPIFFE (the Secure Production Identity Framework for Everyone) and SPIRE (the SPIFFE Runtime Environment) open source projects.</p> <p>HPE is fully-committed to continuing Scytale’s stewardship and contributions to SPIFFE and SPIRE, as these projects will play a fundamental role in HPE’s plans to deliver a dynamic, open, and secure edge-to-cloud platform</p> <p>The <em>following</em> content was originally posted on Scytale’s blog:</p> <p>Over the course of 2019, the Scytale team continued its work shepherding the SPIFFE and SPIRE communities towards building standards and tooling that support the service identity concept. Part of the Cloud Native Computing Foundation (CNCF), these projects have grown in popularity and seen an increasing number of contributions from engineering teams at Amazon, Bloomberg, Google, Pinterest, Square, TransferWise, Uber, Yahoo Japan, and more.</p> <p>These community-led innovations and contributions have enabled SPIFFE to authenticate to more types of software systems and improved the deployment, operability, and performance of SPIRE in large-scale environments.</p> <h2>Authenticate to Service Mesh, Cloud Platforms, Databases, and more</h2> <p>Until recently, SPIFFE has primarily been used to secure communications between services identified by a shared SPIFFE identity provider (IdP). While numerous contributions were made throughout this past year, thirteen key capabilities were released in 2019, which allow you to use SPIFFE and SPIRE to enable strong trust between cloud-native services and other shared services—including databases, service meshes, and public cloud providers. Some enhance ease of deployment and operability, while others offer performance and scalability improvements. These key features include:</p> <ol> <li> <p><strong>SPIFFE federation:</strong> This feature allows services in disparate domains identified by independent SPIFFE identity providers, such as SPIRE, to securely authenticate and communicate with each other. Key use cases for SPIFFE federation include:</p> <ul> <li>Federation between multiple mesh implementations</li> <li>Federating trust across different domains within an organization</li> </ul> </li> </ol> <p>      A number of emerging projects and platforms have adopted SPIFFE. This includes Istio, SPIRE, Scytale Enterprise, and several service meshes, including NGINX (F5 Networks) and Grey Matter (Decipher Technology Studios). SPIFFE federation will enable service interoperability across all these distributed environments. View this recent session from KubeCon to see how you can <a href="https://www.youtube.com/watch?v=cx_NnvbsCP4">secure communications between meshes and beyond with SPIFFE federation.</a></p> <ol start="2"> <li> <p><strong>OIDC federation:</strong> SPIRE now supports OIDC (OpenID Connect). With federation primitives built into SPIFFE and SPIRE, you can directly authenticate to OIDC-compatible validators without having to generate or manage secrets. For example, a system running within an on-premise data center managed by SPIRE can now directly authenticate to cloud platforms like AWS without sharing secrets or private keys. <a href="https://www.youtube.com/watch?v=db_3LefoG9k&#x26;feature=youtu.be">Here is a video that demonstrates this capability.</a></p> </li> <li> <p><strong>Database authentication:</strong> You can now use SPIRE-issued identities to directly authenticate to databases and other systems that support legacy x509 authentication. This approach negates the need for a secret store and instead relies on short-lived asymmetric keys. This approach can also provide traffic encryption. To learn more, <a href="https://www.youtube.com/watch?v=YFll-3jgFrU&#x26;feature=youtu.be">view this demo.</a></p> </li> </ol> <h3>Ease of Deployment and Operability</h3> <ol start="4"> <li> <p><strong>Plugin infrastructure refactoring:</strong> This feature enables plugins to provide auxiliary services, such as health check, to the SPIRE core. You can also request services from the SPIRE core or use server-like telemetry. The Notifier Plugin was added as a result of this refactoring. This plugin detects certain events and then performs an action in response. An example of this is the Kubernetes Bundle notifier plugin, which resolves a bootstrapping problem within Kubernetes. It publishes the latest trust bundle to a Kubernetes ConfigMap when the SPIRE server is started and the bundle is updated.</p> </li> <li> <p><strong>Registration API remote access:</strong> This enhancement allows remote software services to authenticate against the registration API via SVIDs. It enables greater deployment flexibility by allowing software services to manage registrations, even if these services are not running next to a SPIRE server.</p> </li> <li> <p><strong>Kubernetes workload registrar:</strong> This registrar automates the registration of Kubernetes workloads on SPIRE when they are registered in Kubernetes. Using this capability, you can interact exclusively with the Kubernetes API server to get SPIRE fully deployed and operational. <a href="https://github.com/spiffe/spire/tree/master/support/k8s/k8s-workload-registrar">View this link for more details.</a></p> </li> <li> <p><strong>Telemetry and logging:</strong> These capabilities have been significantly improved, helping engineers better track and troubleshoot performance issues in production environments. Several telemetry points were added, including a SQL datastore plugin. In addition, existing points were audited, augmented, and labeled with meaningful descriptions. Finally, key health and performance metrics from the SPIRE server can now be exported directly to Prometheus, Statsd, and DataDog.</p> </li> <li> <p><strong>Upstream plugin for AWS Secrets Manager and AWS Certificate Manager:</strong> This plugin enables you to use AWS Secrets Manager or AWS Certificate Manager as an upstream signing authority for SPIRE-issued identities. It can improve your security posture since keys do not need to be stored on disk.</p> </li> <li> <p><strong>MySql support:</strong> Now, MySql can be used to store configuration data for the SPIRE server.</p> </li> <li> <p><strong>Docker-based workload attestor:</strong> This capability enables you to use selectors based on Docker labels and the container's image ID.</p> </li> <li> <p><strong>Improved documentation:</strong> New documentation enhancements make it easier to get started on SPIFFE and SPIRE. Documentation now includes a guide for getting started with Kubernetes, as well as use cases and case studies. You can find these on SPIFFE.io.</p> </li> </ol> <h3>Performance and Scalability Improvements</h3> <ol start="12"> <li> <p><strong>Nested SPIRE:</strong> This new capability enables you to organize your SPIRE deployment into a hierarchy in which one SPIRE server can deliver CA identities to a downstream SPIRE server. This enables you to segregate your SPIRE deployment so you can improve fault management and resiliency.</p> </li> <li> <p><strong>Performance improvements:</strong> The team made several performance improvements, particularly within the datastore plugin. These enhancements optimize database performance and speed up scale-related hot spots.</p> </li> </ol> <p>HPE and the Scytale team are grateful for all the contributors that help keep the momentum going with SPIFFE and SPIRE. The Scytale team is now working along with the HPE DEV team to expand contribution opportunities. We’re looking forward to even more contributions in 2020.</p>https://developer.hpe.com/introducing-hpe-storage-container-orchestrator-documentation/https://developer.hpe.com/introducing-hpe-storage-container-orchestrator-documentation/Mon, 20 Apr 2020 19:56:01 GMT<p>Writing intuitive, end-user facing documentation for complex systems that is easy to navigate and consume is not a simple task. The Hewlett Packard Enterprise (HPE) storage team picked up the responsibility of consolidating all documentation pertaining to Kubernetes and Docker integration with HPE storage solutions under one umbrella. <a href="https://scod.hpedev.io">HPE Storage Container Orchestrator Documentation</a>, or SCOD for short, is a documentation portal that is now officially available to customers and partners interested in HPE persistent storage solutions that they can integrate with their Kubernetes or Docker projects. Let's explore the toolchain that makes writing documentation effortless and beautiful!</p> <h1>Lean and modern tooling</h1> <p>SCOD stores all the documentation source files in markdown on GitHub. When a pull request is merged to the master branch (after a careful review), the documentation portal is automatically rebuilt using a GitHub Action. The site itself is hosted by GitHub and uses GitHub Pages to serve the content. MkDocs is the rendering engine used for SCOD. Its theme is based on the widely popular <a href="https://docs.readthedocs.io">readthedocs.io</a> look and feel, augmented with some HPE DEV flair.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/screen-shot-2020-04-17-at-100820-am-1587412198284.png" alt=""></p> <p>The unique advantage of using MkDocs is that a tech writer may render the entire project locally and immediately observe what the content would look like in the final product. The tech writer may also use their favorite text editing tool as long as it doesn’t add any hidden formatting. This gives both writers and reviewers a very lean and efficient way to publish documentation without obstacles and impediments posed by dinosaur tools and error prone review processes. It’s single source, single output and provides a shared view in GitHub to review each other’s branches and carefully track changes.</p> <h1>Full steam ahead!</h1> <p>It’s difficult to estimate how far a system built using the SCOD toolchain would scale, but for single project documentation it’s a very efficient process with minimal overhead and gets everything out of the way for tech writers to simply do what they’re supposed to do – write and review documentation! Expect a flurry up of updates in the coming months. HPE looks forward to helping you learn more about the HPE storage integrations with Kubernetes and Docker through SCOD.</p> <p>Would you like to know more?</p> <ul> <li>Learn about <a href="https://pages.github.com/">GitHub Pages</a></li> <li>Deploy MkDocs <a href="https://github.com/marketplace/actions/deploy-mkdocs">GitHub Action</a></li> <li>Build your own site with <a href="https://www.mkdocs.org">MkDocs</a></li> <li>Checkout the <a href="https://datamattsson.tumblr.com/post/612351271067893760/the-perfect-documentation-storm">complete cookbook</a> for SCOD</li> <li>Visit <a href="https://scod.hpedev.io">SCOD</a>!</li> </ul>https://developer.hpe.com/using-jitsi-meet-to-reach-out-to-others/https://developer.hpe.com/using-jitsi-meet-to-reach-out-to-others/Wed, 15 Apr 2020 21:37:50 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture11-1586988196990.png" alt="picture11"></p> <p>Along with half of the world’s population, I find myself working from home due to sheltering requirements imposed on us by the COVID-19 pandemic. In these troubled times, communication with my friends and family have become critical. I found that I needed ways to stay in touch with them easily. Of course, mobile apps offer a great diversity of options to stay in touch with one another. But conducting a multi-user video conference on a mobile phone is not very easy nor is it convenient.</p> <p>There are many people today who are looking for ways to virtually connect through video conferencing. From a more professional standpoint, teachers need to be able to instruct and stay in touch with their students. Small companies could use it for meetings. Doctor/patient interactions could also benefit from such a tool. But it would need to be very inexpensive (i.e. free) and easy to use.</p> <p>As a Hewlett Packard Enterprise (HPE) employee, the company offers us many ways to video conference one another, i.e. through Teams, Skype, and Zoom. Even though these technologies try to make it easy for everyone to use and affordable, I thought I might check on the open source side of the world to see what was available there as well.</p> <p>Zoom is free for 40 minutes. That’s nice, but is it enough? Looking a little deeper, I found <a href="https://jitsi.org/jitsi-meet/">Jitsi Meet.</a> Jitsi is free and ready to go whenever you want. Just open the <a href="https://meet.jit.si/">link</a> and start your meeting. That’s all it takes.</p> <p>For those who are a bit more technical, you may want consider setting up your own Jitsi Meet server at home. This could be a service you could deliver to your family and friends that would offer you a bit more control and privacy, as you wouldn’t have to rely on an external provider. To help those of you who may be interested in setting up your own Jitsi Meet instance, I thought I’d share my experience. In this post, I’ll run you through the installation process and show you how to use it.</p> <h2>What is Jitsi?</h2> <p>Jitsi is a set of open-source projects that allows you to easily build and deploy secure videoconferencing solutions. At the heart of Jitsi are two projects, Jitsi Videobridge and Jitsi Meet. These projects allow you to have conferences on the internet, while other projects in the community enable features such as audio, dial-in, recording, and simulcasting.</p> <h2>Why Jitsi?</h2> <p>Why was I interested in finding an open source-based solution? I wanted a living project with a community behind it in case I ran into issues or required support. I also needed a solution that I could set up myself and would not require too many resources.</p> <p>Official minimum hardware requirements for the Jitsi Meet server are very low. One gigabyte of memory and one CPU core plus 25 gigabytes of storage should be enough when used on a Linux platform. It can be set up directly on your own hardware or on a virtual machine hosted by your favorite Cloud provider.</p> <p>I own a ProLiant Microserver Gen8 at home. It runs a hypervisor and hosts several virtual machines (OPNsense Firewall and Nextcloud server). I use this server to perform tests on new operating systems and software stacks. Since I had some room left on the system, I decided to create a new VM for this new project.</p> <p>My home network looks like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture2-1586988219313.png" alt="picture2"></p> <p>I followed several guides on how to perform the installation:</p> <p>Jitsi-Meet official install guide: <a href="https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md">https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md</a></p> <p><a href="https://vitux.com/how-to-install-jitsi-meet-video-conference-platform-on-ubuntu/">https://vitux.com/how-to-install-jitsi-meet-video-conference-platform-on-ubuntu/</a></p> <p>HA Proxy – Let’s Encrypt: <a href="https://forum.opnsense.org/index.php?topic=12126.0">https://forum.opnsense.org/index.php?topic=12126.0</a></p> <p>Advanced Configuration: Behind NAT configuration <a href="https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md">https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md</a></p> <p>Secure Domain: First four steps only. <a href="https://github.com/jitsi/jicofo#secure-domain">https://github.com/jitsi/jicofo#secure-domain</a></p> <h2>Installing Jitsi Meet</h2> <p>My VM requirements: CPU 1 RAM 2 GB HDD 25 GB Nic 1</p> <h2>Initial Setup</h2> <p>Jitsi Meet requirements: A server running Ubuntu 18.04 LTS A non-root user with sudo privileges Before starting, update your operating system with the latest version with the following command:</p> <pre><code class="language-bash"> sudo apt-get update -y sudo apt-get upgrade -y </code></pre> <p>Once your system is up-to-date, restart your system to apply the changes. Next, you will need to set up a relevant hostname (node1 in this example) and FQDN to your system. You can do this by running the following commands:</p> <p><code>sudo hostnamectl set-hostname node1</code></p> <p>Next, open /etc/hosts file and add FQDN.</p> <p><code>sudo nano /etc/hosts</code></p> <p>Add the following line and adapt to your host and domain names:</p> <p><code>127.0.1.1 node1.example.com node1</code></p> <p>Add the public IP and the relevant domain name, too.</p> <p><code>X.X.X.X node1.example.com node1</code></p> <p>Save and close the file. Then, verify the hostname with the following command:</p> <p><code>hostname -f</code></p> <h2>Swap Adjustment</h2> <p>With a limited set of resources, I needed to adapt and adjust the swap file.</p> <pre><code class="language-bash"> sudo dd if=/dev/zero of=/swapfile count=2048 bs=1M sudo chmod600 /swapfile sudo makeswap /swapfile sudo swapon /swapfile echo ‘/swapfile none swap sw 0 0’ | sudo tee –a /etc/fstab </code></pre> <h2>Install Java</h2> <p>Next, you will need to install Java to your system. You can install OpenJDK JRE 8 by running the following command:</p> <p><code>sudo apt-get install -y openjdk-8-jre-headless -y</code></p> <p>Once Java is installed, verify the Java version with the following command:</p> <p><code>java -version</code></p> <p>Output:</p> <pre><code class="language-markdown"> openjdk version "10.0.2" 2018-07-17 OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3) OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode) </code></pre> <h2>Install Nginx</h2> <p>Jitsi Meet uses Nginx as a reverse proxy. So you will need to install it to your system. You can install it with the following command:</p> <p><code>sudo apt-get install nginx -y</code></p> <p>Once Nginx is installed, you can check the Nginx service with the following command:</p> <p><code>sudo systemctl status nginx</code></p> <p>Output:</p> <pre><code class="language-markdown"> ? nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-11-14 10:06:01 UTC; 1h 4min ago Docs: man:nginx(8) Main PID: 16734 (nginx) Tasks: 2 (limit: 1114) CGroup: /system.slice/nginx.service ??16734 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ??18709 nginx: worker process Nov 14 10:06:01 node1 systemd[1]: Starting A high performance web server and a reverse proxy server... </code></pre> <h2>Install Jitsi Meet</h2> <p>By default, Jitsi Meet is not available in the Ubuntu 18.04 default program repository. So, you will need to add the Jitsi Meet download server to the Ubuntu program repository.</p> <p>You can do this by running the following command:</p> <pre><code class="language-bash">wget -qO - https://download.jitsi.org/jitsi-key.gpg.key | sudo apt-key add - sudo sh -c "echo 'deb https://download.jitsi.org stable/' > /etc/apt/sources.list.d/jitsi.list" </code></pre> <p>Next, update the repository and install Jitsi Meet with the following command:</p> <pre><code class="language-bash"> sudo apt-get update -y sudo apt-get install jitsi-meet -y </code></pre> <p>During the installation process, you will need to provide your hostname as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture4-1586988248790.png" alt="picture4"></p> <p>Provide your hostname and click on the <strong>OK</strong> button. You will be asked to select the SSL certificate as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture6-1586988261662.png" alt="picture6"></p> <h2>Local Firewall Update</h2> <p>Jitsi Meet requires ports 80 and 433 (TCP) and a range from 10000 to 20000 (UDP) to be opened locally and on your network to work properly. Consider also opening SSH for remote access You update the firewall rules by running the following commands:</p> <p>Ubuntu Firewall rules:</p> <p>Allow SSH for remote access:</p> <p><code>sudo ufw allow ssh</code></p> <p>Allow Ports 80, 443, and range from 10000 to 20000 for Jitsi Meet.</p> <pre><code class="language-bash"> sudo ufw allow http sudo ufw allow https sudo ufw allow in 10000:20000/udp </code></pre> <p>Enable firewall rules.</p> <p><code>sudo ufw enable</code></p> <h2>Local Area Network Firewall Rules (OPNsense)</h2> <p>As I had already set up several servers behind my firewall, I had to use HA Proxy to distribute the different HTTP / HTTPS requests to the different servers (Nextcloud, Jitsi Meet).</p> <p>I followed the thread below to set everything up. <a href="https://forum.opnsense.org/index.php?topic=12126.0">https://forum.opnsense.org/index.php?topic=12126.0</a></p> <p>The thread actually covered the HA Proxy and the Let’s Encrypt topics. This proved to be very helpful in the end. Certificates are now handled by the HA Proxy layer in front of the different servers.</p> <h2>Secure Meeting Room Access</h2> <p>By default, anyone who has access to your Jitsi instance will be able to start a conference. If your server is open to the world, anyone can have a chat with anyone else. (Unfortunately, I don’t have unlimited resources. Therefore, I needed to set some limits.)</p> <p>If you want to limit the ability to start a conference to registered users only, set up a "secure domain". Follow the instructions at <a href="https://github.com/jitsi/jicofo#secure-domain">https://github.com/jitsi/jicofo#secure-domain</a>.</p> <p>Follow the first 4 steps and you’ll be ready to go! This will allow you to set up a user with credentials with the necessary rights to create a room.</p> <h2>Access Jitsi Meet</h2> <p>If you have followed my instructions, Jitsi Meet is now up and listening on port 443. Open your web browser and type the URL <code>https://node1.example.com</code> or <code>https://your-server-ip.</code> You will be redirected to the following page:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture7-1586988283022.png" alt="picture7"></p> <p>Here, provide the room name you desire and click on the <strong>GO</strong> button. You should see the following on your screen:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture8-1586988303937.png" alt="picture8"></p> <p>Click on the <strong>Allow</strong> button to start the live video conference.</p> <h2>Some last comments</h2> <p>After a few hours of work, I had the server ready to go and I could open my first room. If you’ve followed my instructions, you should now be all set to easily meet with your friends, family, and colleagues!</p> <p>Keeping a social link is very important during these times. That’s why simple solutions like Jitsi Meet can really be helpful. Jitsi Meet offers an easy way for people to connect.</p> <p>There are so many possible applications for this technology. Educators can use this solution with the relevant hardware configuration so that a complete class could easily attend a course. Small companies meetings could be hosted via a dedicated server, too.</p> <p>For those who might find technology a little more daunting or less accessible, Jitsi Meet still can be easily used with a simple browser. Think how this could help the elderly keep in touch and interact with their families.</p> <p>Stay tuned to the <a href="/blog">HPE DEV blog site</a> for more helpful tutorials.</p>https://developer.hpe.com/wow-a-practiced-and-perfected-design-process-part-1-uncovering-the-merit/https://developer.hpe.com/wow-a-practiced-and-perfected-design-process-part-1-uncovering-the-merit/Mon, 13 Apr 2020 17:45:00 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/title-slide-1587659126865.jpg" alt="title slide"></p> <h1>What is WOW?</h1> <p>The HPE Experience Studio works with design teams throughout Hewlett Packard Enterprise (HPE) to deliver a consistent user experience (UX) for customers. We are constantly learning, iterating, and innovating the UX design process across the HPE portfolio. As the manager for experience design, I’ve found that many businesses struggle with understanding the true value UX designers bring to a project. In this series of blog posts, I want to share with you a method our group developed that demonstrates this value and provides enterprise UX designers with a practiced and perfected path to achieving success. This is a workflow we call WOW (Why On What with customers and constructors). You’ll note that, as with any good UX design, we used a little creativity in the spelling, with two reflected “C’s” (representing the customers and the constructors) used to create the “O”.</p> <p>WOW is a domain and persona agnostic workflow that has been perfected over a period of time. The flow doesn’t provide any governance on tools, activities, or timelines, but instead sets the right framework in place to bring about a level of certainty and success to a project while moving from one part of the process to the next. It is an easy, engaging methodology that helps creative teams focus on four important stages of a project:</p> <ol> <li>Uncover why the business (and this particular project) needs a UX</li> <li>Involve the customer early in the design phase</li> <li>Ensure the constructor (developer) has the right information during design implementation</li> <li>Convey the business value of the design</li> </ol> <br/> <h2>Uncovering the project value of the UX using a contextual presentation</h2> <p>Traditionally, designers tend to focus on what UX design tools and methods they plan to use for a project when they present to stakeholders (such as software developers, program and product managers, etc.) As they attempt to convey the value of a purposefully designed UX, they also cover the revenue and efficiency impact the design will have on customers. Though this approach is good for on-stage presentations, in the HPE Experience Studio, we’ve found that this method is not effective in meetings where stakeholders are looking to understand the value of the UX in the context of the overall project.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-1-slide-2-1587659158892.jpg" alt="part 1 slide 2"></p> <p>Presentations like this tend to generalize outcomes from design efforts, which can undermine the importance of the UX for a project. Stakeholders can become overwhelmed or confused and may not be able to relate the benefits presented directly to the UX within the context of their projects. Because of this, it is important to avoid generalizing UX outcomes and to instead talk about the impact of the UX in terms the project at hand.</p> <p>This point became very clear as we worked with several groups and practiced a concept called contextual inquiry, which is part of a <a href="https://en.wikipedia.org/wiki/User-centered_design">user-centered design</a> (UCD) research method. Experience in this area helped us to devise a contextual UX presentation, which focuses on business KPIs that can be used to easily articulate the design-value for a specific project.</p> <h2>Shift mindsets from pretty to pragmatic by focusing on tools that address KPIs</h2> <p>Many times a UX project is perceived as an attempt of beautifying the existing look and feel of a product. Though this is one important aspect of UX design, there is far more to it. More importantly, enterprise stakeholders should understand how the UX design relates to the overall value of the product.</p> <p>Almost every enterprise product is developed with four KPIs (key performance indicators) in mind:</p> <ol> <li>New customer acquisition</li> <li>Customer retention</li> <li>Increased yield</li> <li>Reduced costs</li> </ol> <p>Often, the reduced cost KPI is the biggest impediment for considering UX proposals and investing in a focus on UX within a product.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-1-slide-3-1587659291004.jpg" alt="part 1 slide 3"></p> <p>Using specific tools, the UX design process can actually play a key role in helping developers address KPIs and easily justify “Why” a product team should invest in UX design. Tools like <a href="https://en.wikipedia.org/wiki/System_usability_scale">SUS (System Usability Scale)</a> and heuristic evaluations can uncover design debts present in the product that could lead to issues down the road. Heuristic analysis and evaluations can illustrate the scope of improvements a properly designed UX can bring in terms of product usability, encouraging the <em>acquisition of new customers</em>. Using a SWOT analysis, designers can pinpoint current product functionality debts and highlight competitive advantages, which will help <em>increase the yield</em> of the product.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-1-slide-4-1587659222302.jpg" alt="part 1 slide 4"></p> <p>By measuring the product’s system usability with SUS, the UX design process addresses the requirement to retain customers by making sure customers feel they have been heard and that their requirements are addressed on a timely basis. Performing a UX-competitive analysis on a product helps product managers identify and prioritize investment opportunities. This can lower expenses by reducing the number of wrong investments. As you can see, the design process not only uncovers issues but also very elegantly uncovers gaps and opportunities, a combination that is often hard to find!!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/part-1-slide-5-1587659254292.jpg" alt="part 1 slide 5"></p> <h2>Summary</h2> <p>There is a lot more to UX design than what is often considered. UX design processes aren’t intended to just make an application look pretty. The benefits of using a thoughtfully developed UX design process extend way beyond the resulting user interface (UI). Benefits can be obtained immediately by involving UX designers at the very beginning of a project. As I’ve pointed out, in the first part of our WOW workflow, designers use tools that really help uncover the business value of UX design. In Part 2, I will cover steps 2 and 3 of WOW; the importance of involving the customer early in the design phase, and ensuring that the constructor (developer) has the right information during design implementation.</p> <p>Feel free to reach out to me <a href="https://twitter.com/uxwithparul">@uxwithparul</a> if you have any questions. Follow subsequent posts in this series and view other informative blogs at <a href="/blog">HPE DEV.</a></p>https://developer.hpe.com/app-dev-and-the-hpe-container-platform/https://developer.hpe.com/app-dev-and-the-hpe-container-platform/Thu, 09 Apr 2020 15:41:06 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/hpecpimage2-1586446910822.jpg" alt="hpecpimage2"></p> <p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>The HPE Container Platform offers enterprise IT and software development teams an agile application development and delivery platform. It can be used for both cloud-native microservices-based applications, as well as non-cloud-native monolithic applications. In addition to providing IT Operations teams with a unified control plane based on 100% open source Kubernetes, the HPE Container Platform gives data scientists and application developers a broad set of tools. They can use these tools to build a variety of use cases – including application modernization, CI/CD pipelines, machine learning, and edge analytics.</p> <p>The HPE Container Platform combines several innovations from HPE. One such innovation is KubeDirector, an open source project integrated into the HPE Container Platform. KubeDirector manages an application’s dependency on access to the root file system. This dependency on root file systems usually makes it impossible for non-cloud-native applications to run in containers. With KubeDirector, enterprises can deploy all their enterprise applications on a common Kubernetes framework. The ability to deploy and manage these as containers allows them to take advantage of portability and cloud economics.</p> <p>In addition to KubeDirector, the HPE Container Platform offers a rich set of additional functionality. For example, it provides a high level of security with integrations into enterprise security and authentication services, such as SAML (Security Assertion Markup Language), AD (Active Directory Security), Kerberos, and more. HPE is working on additional extensions in this area, including integrating HPE’s “silicon root of trust”, the link between HPE silicon and its Integrated Lights Out (iLo) management system, in support of the concept of Zero Trust. The SPIRE and SPIFFE open source initiatives, contributed by HPE’s recently acquired Scytale, is expected to play an important role in this.</p> <p>Using the HPE Container Platform also provides unprecedented resiliency and scalability, supporting standard CSI Storage drivers and several different types of storage, including the HPE Data Fabric (previously the MapR Data Fabric). These are important elements that developers or data scientists can leverage in their applications without writing a single line of code. These capabilities empower data scientists and data analysts to quickly stitch together machine learning and analytics pipelines that can then be deployed on-premises, in multiple public clouds, or at the edge.</p> <p>Today’s enterprises look to benefit from the many advantages provided through a cloud-native architecture. They desire the capabilities it offers to rapidly process data and glean insights that help them better service their customers. They want to be able to pay for only the services they need. And they appreciate being able to develop and run applications in an agile environment that allows them to deploy applications on any infrastructure – on-premises, cloud, or at the edge.</p> <p>Learn more about how HPE Container Platform enables enterprises to achieve their transformation goals by reading Matheen Raza’s blog post, <a href="https://www.hpe.com/us/en/insights/articles/the-cloud-is-an-experience-not-a-destination-2002.html?sdfsdf">The cloud is an experience, not a destination.</a> In his blog, you’ll also find a link to a podcast by Chris Gardner, principal analyst serving infrastructure and operations professionals, Forrester and Tom Phelan, Fellow, hybrid IT infrastructure, HPE on how containers and open source Kubernetes accelerate innovation. Stay up-to-date on what’s happening in this area by going to the HPE DEV <a href="https://developer.hpe.com/platform/hpe-container-platform/home">HPE Container Platform page.</a></p>https://developer.hpe.com/hpe-oneview-chef-cookbook-now-supports-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-chef-cookbook-now-supports-hpe-oneview-50/Mon, 06 Apr 2020 16:38:30 GMT<p>The HPE OneView Chef Cookbook 3.2.0 now supports HPE OneView 5.0 (REST API version 1200). The Cookbook provides Chef recipes to interact with HPE OneView and HPE Synergy Image Streamer APIs, enabling IT automation teams to easily build integrations and scalable solutions. By automating the provisioning of physical infrastructure on-demand, using software-defined templates from HPE OneView, this integration allows administrators to create a resource topology and user experience similar to that of a public cloud on their own physical infrastructure. In addition, it significantly increases reliability, compliance, and deployment flexibility.</p> <p>Chef recipes configure a series of HPE OneView and HPE Synergy Composable Infrastructure resources to a particular state. They define how the resources are to be configured, which specific versions of software to run, and ensure that software is installed in the correct order, based on dependencies. Chef checks that each resource is properly configured and corrects the configuration of any resources that are not in the desired state. This lets IT organizations streamline the task of configuring and maintaining a company's servers, in addition to enabling integration with cloud-based platforms to automatically provision and configure new machines.</p> <p>A Chef Cookbook is where a collection of recipes are stored. Each cookbook should relate to a single task, but can have multiple recipes or server configurations stored together. The Chef server stores each of these cookbooks. When a new Chef client node checks in with the server, recipes are sent to tell the node how to configure itself. The client will then check in every now and again to make sure that no changes have occurred and no modifications need to be made. If there is a change, the client will take action to manage the change. Patches and updates can be rolled out over an entire infrastructure simply by changing the recipe, eliminating the need to interact with each machine individually.</p> <p>Chef is a one of the industry's most popular Infrastructure-as-Code tools. It is based on the Ruby programming language, which can be leveraged to model solutions for complex or custom scenarios. Additional support can be found in Chef’s open-source development community.</p> <p>The list of supported resources and changes is available at:</p> <p><a href="https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.2.0">https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.2.0</a></p> <p>The repository with code and examples is available on GitHub at:</p> <p><a href="https://github.com/HewlettPackard/oneview-chef">https://github.com/HewlettPackard/oneview-chef</a></p> <p>Chef Supermarket details are available at:</p> <p><a href="https://supermarket.chef.io/cookbooks/oneview">https://supermarket.chef.io/cookbooks/oneview</a></p>https://developer.hpe.com/2020-April-01/https://developer.hpe.com/2020-April-01/Wed, 01 Apr 2020 05:00:00 GMThttps://developer.hpe.com/project-golang-sdk-a-valuable-lesson-in-doing/https://developer.hpe.com/project-golang-sdk-a-valuable-lesson-in-doing/Mon, 30 Mar 2020 15:54:22 GMT<p>This year, I decided to learn the Golang programming language. I have always preferred learning any language or technology by working on a project, rather than just implementing some simple example programs from tutorials. So, I carefully considered what sort of project I could do that would not only provide me with a valuable learning experience, but also make a positive contribution. In this blog post, I’d like to share with you how the project I chose helped me achieve my goal in learning Golang.</p> <p>In considering the project I wanted to take on, I had a strong urge to work on developing an SDK (software development toolkit) for the Hewlett Packard Enterprise’s (HPE) Composable Fabric Manager API (application programming interface). Why an SDK? Glad you asked! SDKs are like wrappers to APIs, giving them extra capabilities. For instance:</p> <p>Developers don’t have to worry about creating a JSON payload in the right format and taking care of the endpoints with parameters. All they need to do is to send the required parameters and the SDK will take care of REST operations. More importantly, the developer can implement his software suite employing the programming language he is most comfortable with. In this case, it would be Golang.</p> <ul> <li>An SDK adds another layer of security layer to the application.</li> <li>An SDK also isolates the developer from the changes in the API which significantly reduces the time and effort to maintain the code base.</li> <li>SDKs simplify development of applications that need to scale to multiple devices.</li> </ul> <p>Given all this, I was pretty much convinced that my idea to implement the Golang SDK was a good one. I started off by reading the Golang documentation to understand the syntax, see how similar or how different it is from other programming languages, and learn about features that are exclusive to Golang.</p> <p>Here are a few key things I quickly learned:</p> <ul> <li>Golang is a compiled language. It also supports cross compilation. For those who may be unfamiliar with this term, cross compilation is the mechanism for creating executable code for a platform other than the one on which the compiler is running.</li> <li>It was mostly built for those who want to write less code and achieve more.</li> <li>It is predominantly a functional programming language. However, Golang does incorporate a touch of an object-oriented approach.</li> <li>It supports the development of web applications, APIs, etc. pretty well.</li> <li>Golang doesn’t have exceptions or assertions, which might feel a bit odd at first if one is used to coding with programming languages like Python, Java or C#.</li> <li>Golang also has functional testing module built in, which makes it a great programming language to use in CI/CD pipelines.</li> <li>Golang is also web aware. That is, it can pull the dependencies directly from the web, version control system like GitHub, SVN, etc.</li> </ul> <p>Truly, I would not be surprised if a Python lover discovers a similar love for Golang!</p> <p>After learning the basics of Golang, and few advanced concepts, I started working on the design of my SDK. I must say, it involved a lot of thought and iteration. Finally, I came up with a nice design and started the implementation. Here are some of the things I learned while designing the SDK for this API:</p> <ol> <li>The SDK must mimic the API. This means that the resource files in the SDK must implement functions corresponding to the endpoints which are defined by the API. For example, if you have an API endpoint for CRUD operations (CRUD — Create, Read, Update and Delete, which is very common in REST API terminologies) that is /rest/v1/users, then the SDK must have a method like AddUser(newUser), GetUsers(), UpdateUser(userUUID, updatedUserInfo) and DeleteUser(userUUID). The naming convention must be very intuitive.</li> <li>A standard must be followed to map the API resources and their attributes to the resources and attributes in the SDK. This means that, in some sense, the resources are datatypes in the SDK. This is an advantage, since the developer can map the API resources to variables instead of parsing the JSON payloads all by himself.</li> <li>The SDK should have documentation. Reading the documentation for either the API or the SDK must help you understand how the API works. Any limitations must be explicitly stated.</li> <li>The SDK must implement a good error handling mechanism, with error logging and error messages. Logging must not log any sensitive information</li> </ol> <p>I began building the unit test suite simultaneously along with the implementation system test suite. Most of the time, it is easier to implement functional testing only. But sometimes, it is not. With all this done, I am now confident that I can demo my Golang SDK.</p> <p>As Dave Packard once pointed out, it’s important to always set an extraordinary goal by which all of your trivial goals can be achieved. Going through the process that I have shared above, I not only learned a new programming language, but also produced something valuable. I am looking forward to getting this project reviewed by Open Source Review Board at HPE and possibly making it an open source project. If this project becomes open source, then I’ll probably have to work on deciding on the contribution guidelines, test frameworks, how to build validations, and even set up a CI/CD pipeline using Travis CI or Jenkins or GitHub actions or whatever. A lot of work to do, to be sure!</p> <p>Interested in learning more about SDKs and APIs? You’ll find a lot of good material on the <a href="/blog">HPE DEV blog</a> site. Make sure you check it out. Or, sign up for the <a href="https://developer.hpe.com/newsletter-signup">HPE DEV newsletter</a> to read the latest articles.</p>https://developer.hpe.com/hpe-oneview-ruby-sdk-now-available-with-support-for-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-ruby-sdk-now-available-with-support-for-hpe-oneview-50/Thu, 19 Mar 2020 19:59:13 GMT<p>With a modern, RESTful API and a comprehensive partner ecosystem, HPE OneView delivers a software-defined, programmatic approach to managing infrastructure for efficient workflow automation. The <a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.10.0">HPE OneView Ruby SDK v5.10.0</a> is now available and supports HPE OneView 5.0 (REST API version 1200). The HPE OneView RUBY SDK extends the HPE OneView API language support for the Ruby language. This Ruby SDK provides a Ruby Library to easily interact with the HPE OneView API, enabling developers to easily build integrations and scalable solutions with HPE OneView.</p> <p>This SDK allows Ruby developers to programmatically control HPE OneView managed resources using an infrastructure-as-code approach for physical compute, storage, and fabric resources. Using infrastructure-as-code enables complete datacenter automation, consistent reproducibility, versioning, and roll back.</p> <p>Ruby is a popular software language that supports multiple programming paradigms, including procedural, object-oriented, and functional programming. A number of widely used tools and frameworks, such as Puppet, are written in Ruby. In addition, it is a popular choice for building web and data analysis applications.</p> <p>For more information:</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.10.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-sdk-ruby">Code repository and examples</a></li> <li><a href="https://rubygems.org/gems/oneview-sdk/versions/5.10.0">Ruby Gem</a></li> </ul>https://developer.hpe.com/hpe-oneview-puppet-module-v240-now-supports-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-puppet-module-v240-now-supports-hpe-oneview-50/Thu, 19 Mar 2020 19:56:04 GMT<p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">Hewlett Packard Enterprise is pleased to introduce the HPE OneView</a> Puppet Module v2.4.0 that now supports HPE OneView 5.0 (REST API version 1200). The <a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.4.0">HPE OneView Puppet Module</a> automates the configuring and provisioning of bare-metal infrastructure. This enables faster and easier app and service delivery, while increasing reliability, maintaining compliance, and offering deployment flexibility. By automating the provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView, administrators can create a resource topology similar to that of a public cloud on their own physical infrastructure.</p> <p>The Puppet Module for HPE OneView allows for management of HPE OneView appliances through the use of Puppet manifests and resource style declarations. The manifests use the HPE OneView Ruby SDK to make calls to the HPE OneView REST API. The module adds several resource types to Puppet and custom methods to allow users to easily create, update, query, and delete hardware resources.</p> <p>Puppet offers customers a mature configuration management solution. It is easy to install and features an excellent graphical user interface (GUI). It runs on all major operating systems, such as Linux, Windows, Unix, and even Mac OS X.</p> <p>For more information:</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.4.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-puppet">Code repository and examples</a></li> <li><a href="https://forge.puppet.com/hewlettpackard/oneview">Puppet Forge</a></li> </ul>https://developer.hpe.com/hpe-csi-driver-for-kubernetes-snapshots-clones-and-volume-expansion/https://developer.hpe.com/hpe-csi-driver-for-kubernetes-snapshots-clones-and-volume-expansion/Thu, 19 Mar 2020 03:36:56 GMT<p>The Container Storage Interface (CSI) introduces enterprise data management, such as volume snapshots and volume clones as native Kubernetes objects. In Kubernetes 1.17, these interfaces have matured to a beta state. Recently, Hewlett Packard Enterprise (HPE) <a href="https://community.hpe.com/t5/HPE-Storage-Tech-Insiders/HPE-CSI-Driver-for-Kubernetes-1-1-0-Generally-Available/ba-p/7082995">released version 1.1.0</a> of the HPE CSI Driver for Kubernetes with full support for these features. Let’s walk through how a Kubernetes user can take advantage of these constructs to become more agile by deploying, testing and running stateful applications on Kubernetes.</p> <h1>Deploy the CSI driver with Helm</h1> <p>In this tutorial, upstream Kubernetes 1.17.4 is being used along with Helm 3. Let’s add in the HPE storage container orchestrator deployments Helm repository.</p> <pre><code class="language-bash">helm repo add hpe-storage https://hpe-storage.github.io/co-deployments/ "hpe-storage" has been added to your repositories </code></pre> <p>Before installing the Helm chart, a values file needs to be created to instruct the driver on how to find and authenticate to the backend storage system. For this exercise, the <code>StorageClass</code> that will be installed is also marked as the default <code>StorageClass</code>.</p> <pre><code class="language-bash"># Contents of a file named values.yaml secret: backend: 192.168.1.1 username: admin password: admin storageClass: defaultClass: true </code></pre> <p>Install the CSI driver into the "kube-system" namespace.</p> <pre><code class="language-bash">helm install hpe-csi-driver hpe-storage/hpe-csi-driver --version 1.1.0 --namespace kube-system -f values.yaml NAME: hpe-csi-driver LAST DEPLOYED: Tue Mar 17 09:21:12 2020 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None </code></pre> <p>The required components should come online fairly quickly. The following <code>kubectl</code> command may be used to monitor the driver workloads.</p> <pre><code class="language-bash">kubectl get pods --all-namespaces -l 'app in (nimble-csp, hpe-csi-node, hpe-csi-controller)' NAMESPACE NAME READY STATUS RESTARTS AGE kube-system hpe-csi-controller-7d9cd6b855-zzmd9 5/5 Running 0 15s kube-system hpe-csi-node-dk5t4 2/2 Running 0 15s kube-system hpe-csi-node-pwq2d 2/2 Running 0 15s kube-system nimble-csp-546c9c4dd4-5lsdt 1/1 Running 0 15s </code></pre> <p>As described above, a default <code>StorageClass</code> is also deployed on the cluster.</p> <pre><code class="language-bash">kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE hpe-standard (default) csi.hpe.com Delete Immediate true 20s </code></pre> <h1>Kubernetes distribution specific details</h1> <p>As per the Kubernetes Special Interest Group (SIG) Storage, the snapshot controllers, custom resource definitions and RBAC resources should be deployed on the cluster by the vendor of the Kubernetes distribution, not the CSI driver vendor. These resources are not deployed on upstream Kubernetes 1.17.4, which is being used in this tutorial. Now, let’s deploy the necessary resources.</p> <pre><code class="language-bash">kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml </code></pre> <p><strong>Note:</strong> The resources will be deployed in the "default" namespace.</p> <p>For each CSI driver that supports snapshots, at least one <code>VolumeSnapshotClass</code> object needs to be created. There’s only one backend that supports snapshots on this cluster and the <code>VolumeSnapshotClass</code> is therefore marked as default, which makes it easy for users to not care about implementation details.</p> <pre><code class="language-bash">apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshotClass metadata: name: hpe-snapshot annotations: snapshot.storage.kubernetes.io/is-default-class: "true" driver: csi.hpe.com deletionPolicy: Delete parameters: description: "Snapshot created by the HPE CSI Driver" csi.storage.k8s.io/snapshotter-secret-name: nimble-secret csi.storage.k8s.io/snapshotter-secret-namespace: kube-system </code></pre> <p><strong>Note:</strong> All YAML presented in this blog post should be created with <code>kubectl create -f- &#x3C;hit ENTER, then paste the content and hit CTRL-D on a new line></code> unless otherwise specified.</p> <h1>Get started!</h1> <p>The next set of tasks are completely agnostic to which particular storage vendor CSI driver being used. This is the ideal behavior for users interacting with Kubernetes so as to not worry about implementation details.</p> <p>For this tutorial, a dual replica Redis deployment is being used as an example application. The default is three replicas, but the cluster only has two worker nodes and affinity rules can’t be fulfilled for the Redis chart with three replicas.</p> <p>Ensure the upstream Helm stable repo is accessible.</p> <pre><code class="language-bash">helm repo add stable https://kubernetes-charts.storage.googleapis.com "stable" has been added to your repositories helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "stable" chart repository ...Successfully got an update from the "hpe-storage" chart repository Update Complete. ⎈ Happy Helming!⎈ </code></pre> <p>In subsequent examples, some output has been truncated to enhance readability. Now, let’s install Redis and insert some data.</p> <pre><code class="language-bash">helm install prod stable/redis-ha --version 4.4.1 --set-string replicas=2 kubectl exec -it prod-redis-ha-server-0 sh -n default Defaulting container name to redis. /data $ redis-cli set hpedev testing OK /data $ exit </code></pre> <p>There’s a key in the Redis database named "hpedev" with the value "testing". Imagine this as state of the application that needs to be preserved. Now, let’s create a snapshot of the Persistent Volume Claims (PVC) that the Redis application requested.</p> <pre><code class="language-yaml"> --- apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: snapshot-0 spec: source: persistentVolumeClaimName: data-prod-redis-ha-server-0 --- apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: snapshot-1 spec: source: persistentVolumeClaimName: data-prod-redis-ha-server-1 </code></pre> <p>If everything went well, the snapshots may be enumerated.</p> <pre><code class="language-bash">kubectl get volumesnapshots NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE snapshot-0 true data-prod-redis-ha-server-0 10Gi hpe-snapshot snapcontent-abc0c69c-a22e-499e-8353-b6a6611cd283 16s 17s snapshot-1 true data-prod-redis-ha-server-1 10Gi hpe-snapshot snapcontent-b7f81e8c-661d-42df-b452-13cdab878505 16s 17s </code></pre> <p>We now have the opportunity to instantiate another Redis instance using these snapshots as the source for a new deployment. The key here is that the PVCs need to be created before we bring the new deployment online. Since naming is deterministic for Helm charts, this is quite simple. Create the following PVCs.</p> <pre><code class="language-bash"> --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-test-redis-ha-server-0 spec: dataSource: name: snapshot-0 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-test-redis-ha-server-1 spec: dataSource: name: snapshot-1 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 10Gi </code></pre> <p>Next, bring up a "test" deployment and read the key we inserted prior and insert another key we want to use for a subsequent test.</p> <pre><code class="language-bash">helm install test stable/redis-ha --version 4.4.1 --set-string replicas=2 kubectl exec -it test-redis-ha-server-0 sh -n default Defaulting container name to redis. /data $ redis-cli get hpedev “testing" /data $ redis-cli set upgrade anothertest OK /data $ exit </code></pre> <p>It’s now possible to transform the data of the "test" deployment without disturbing the data of the "prod" deployment. This opens up the possibility to create advanced testing and development workflows that uses an exact representation of production data. Whether this dataset is a few bytes or a handful of terabytes, the operation will only take a few seconds to execute as the snapshots and clones are not making any copies of the source data.</p> <p>The "upgrade" key inserted above will now be used in the next workflow. Clone directly from an existing PVC without creating a snapshot. Let’s now create a set of new PVCs.</p> <pre><code class="language-yaml"> --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-clone-redis-ha-server-0 spec: dataSource: name: data-test-redis-ha-server-0 kind: PersistentVolumeClaim accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-clone-redis-ha-server-1 spec: dataSource: name: data-test-redis-ha-server-1 kind: PersistentVolumeClaim accessModes: - ReadWriteOnce resources: requests: storage: 10Gi </code></pre> <p>Create a new "clone" Redis instance and retrieve the key from the previous workflow.</p> <pre><code class="language-bash">helm install clone stable/redis-ha --version 4.4.1 --set-string replicas=2 $ kubectl exec -it clone-redis-ha-server-0 sh -n default Defaulting container name to redis. /data $ redis-cli get upgrade "anothertest" /data $ exit </code></pre> <p>Not only does this demonstrate the ability to clone directly from a PVC as declared in the <code>dataSource</code>, but it also demonstrates the ability to perform a cloning operation on an already existing clone. It’s also possible to create forks of the Redis database to create even more sophisticated workflows. A practical example would be to create a snapshot of a production instance, clone from that instance, obfuscate some data for the end user (could be a potentially IO intensive operation when working with terabyte datasets) and then use the obfuscated clone as a source for subsequent workflows presented to end-users. The idea is to obfuscate the data once and stamp out many new permutations quickly from that one source.</p> <h1>Volume expansion</h1> <p>One of the most common “Day 2” operations in storage and data management is to expand volume capacity. This feature has been in beta since Kubernetes 1.16 and is now available in the HPE CSI Driver for Kubernetes as a supported feature. In true Kubernetes simplistic fashion the end-user that created the PVC may simply increase the capacity of the PVC specification and the CSI resizer will pick it up and perform all the necessary operations. These operations include increasing the backend storage system volume size, rescanning the multipath device on the host and finally growing the filesystem. This used to be a tedious operation that required a storage admin and Kubernetes admin to satisfy a user requirement, which is very counter-productive.</p> <p>Let’s expand the storage requests for the Redis production instance (this can be done with <code>kubectl edit</code> or <code>kubectl patch</code> as well). Run <code>kubectl apply</code> with the following specification.</p> <pre><code class="language-yaml"> --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-prod-redis-ha-server-0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-prod-redis-ha-server-1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 32Gi </code></pre> <p>The original size of the PVC was 10Gi. In a few moments, the new size should be been picked up by the <code>Pod</code> running Redis.</p> <pre><code class="language-bash">kubectl exec -it prod-redis-ha-server-0 sh -n default Defaulting container name to redis. /data $ df -h . Filesystem Size Used Available Use% Mounted on /dev/mapper/mpathi 32.0G 32.7M 32.0G 0% /data </code></pre> <h1>Next steps</h1> <p>Stay tuned to HPE DEV for future blogs regarding the HPE CSI Driver for Kubernetes. In the meantime, connect with us on <a href="https://hpedev.slack.com">Slack</a>. We hang out in #kubernetes and #nimblestorage</p> <p>If you want to learn more about Kubernetes, CSI and the integration with HPE storage products, here are a few pointers that would get you started.</p> <ul> <li>Learn about Kubernetes on <a href="https://kubernetes.io">kubernetes.io</a></li> <li>Explore the <a href="https://kubernetes-csi.github.io/">Container Storage Interface</a></li> <li>Check out the <a href="https://developer.hpe.com/platform/hpe-container-platform/home">HPE Container Platform</a></li> <li>GitHub repository for the <a href="https://github.com/hpe-storage/csi-driver">HPE CSI Driver for Kubernetes</a></li> </ul>https://developer.hpe.com/why-devsecops-approach-is-key-to-mainstream-container-use/https://developer.hpe.com/why-devsecops-approach-is-key-to-mainstream-container-use/Tue, 17 Mar 2020 11:46:32 GMT<p><strong>Editor’s note: This article was originally posted on HPE Enterprise.nxt on March 17, 2020</strong></p> <hr> <br /> <p>Container-based deployments are gaining in popularity, but to maintain this momentum, security must start early in project development.</p> <p>Containers offer a range of benefits across enterprise computing environments―from corporate data centers to the cloud to the edge―and IT organizations are looking to increase use cases. But for container-based deployments to continue on this upward trajectory, security is a must, and it needs to be addressed at the start of development using DevSecOps best practices.</p> <p>Join this discussion with HPE Pointnext Services' Simon Leech and host Dana Gardner to learn about the advantages containers bring, what it takes to move your projects from proof of concept to mainstream production, and why a DevSecOps approach to security is key to that.</p> <p><a href="https://www.youtube.com/watch?v=ayoJY1EGsAw"><img src="https://img.youtube.com/vi/ayoJY1EGsAw/hqdefault.jpg" alt="Containers go mainstream"></a></p> <br /> <br /> <p><strong>Dana Gardner:</strong> Hello, and welcome to the next edition of the BriefingsDirect Voice of Innovation podcast series. I'm <a href="https://www.linkedin.com/in/danagardner/">Dana Gardner</a>, principal analyst at Interarbor Solutions, your host and moderator for this ongoing discussion on the latest insights into modern IT deployment architecture strategies.</p> <p>Container-based deployment models have rapidly gained popularity, from cloud models to corporate data centers. IT operators are now looking to extend the benefits of containers to more use cases, including the computing edge.</p> <p>Yet, in order to push containers further into the mainstream, security concerns need to be addressed across this new end-to-end container deployment spectrum―and that means security addressed during development and employment under the rubric of <a href="https://www.devsecops.org/blog/tag/DevSecOps+Explained">DevSecOps</a> best practices.</p> <p>Stay with us now as we examine the escalating benefits that come from secure and robust container use with our guest, <a href="https://www.linkedin.com/in/simonleech/">Simon Leech</a>, worldwide security and risk management practice at Hewlett Packard Enterprise (HPE) Pointnext Services. Welcome, Simon.</p> <p><strong>Simon Leech:</strong> Hey, Dana. Good afternoon.</p> <p><strong>Gardner:</strong> Simon, are we at an inflection point where we're going to see containers take off in the mainstream? Why is this the next level of virtualization?</p> <h3>Mainstream containers coming</h3> <p><strong>Leech:</strong> We are certainly seeing a lot of interest from our customers when we speak to them about the best practices they want to following in terms of rapid application development.</p> <p>One of the things that always held people back a little bit with virtualization was that you are always reliant on an operating system managing the applications that sit on top of that OS in managing the application code that you would deploy to that environment.</p> <p>But what we have seen with containers is that, as everything starts to follow a cloud-native approach, we start to deal with our applications as lots of individual microservices that all communicate integrally to provide the application experience to the user. It makes a lot more sense from a development perspective to be able to address the development in these small, microservice-based or module-based development approaches.</p> <p>So, while we are not seeing a massive influx of container-based projects going into mainstream production at the moment, there are certainly a lot of customers testing their toes in the water to identify the best possibilities to adopt and address container use within their own application development environments.</p> <p><strong>Gardner:</strong> And because we saw developers grok the benefits of containers early and often, we have also seen them operate within a closed environment, not necessarily thinking about deployment. Is now the time to get developers thinking differently about containers, as not just perhaps a proof of concept (POC) or test environment but as ready for the production mainstream?</p> <p><strong>Leech:</strong> Yes. One of the challenges I have seen with what you just described is a lot of container projects start as a developer's project behind his laptop. So the developer is going out there, identifying a container-based technology as something interesting to play around with, and as time has gone by, has realized he can actually make a lot of progress by developing his applications using a container-based architecture.</p> <p>What that means from an organizational perspective is that this is often done under the radar of management. One of the things we are discussing with our customers as we go and talk about addressing DevSecOps and DevOps initiatives is to make sure that you do get that buy-in from the executive team and so you can start to enable some top-down integration.</p> <p>Don't just see containers as a developer's laptop project, but look at it broadly and understand how you can integrate that into the overall IT processes that your organization is operating with. And that does require a good level of buy-in from the top.</p> <p><strong>Gardner:</strong> I imagine this requires a lifecycle approach to containers thinking―not just about the development but in how they are going to be used over time and in different places.</p> <p>Now, 451 Research recently predicted that the <a href="https://451research.com/images/Marketing/press_releases/Application-container-market-will-reach-2-7bn-in-2020_final_graphic.pdf">market for containers will hit $2.7 billion this year</a>. Why do you think that the IT operators―the people who will be inheriting these applications and microservices―will also take advantage of containers? What does it bring to their needs and requirements beyond what the developers get out of it?</p> <h3>Quick-change code artists</h3> <p><strong>Leech:</strong> One of the biggest advantages from an operational perspective is the ability to make fast changes to the code you are using. So, whereas in the traditional application development environment, a developer would need to make a change to some code and it would involve requesting a downtime to be able to update the complete application, with a container-based architecture, you only have to update parts of the architecture.</p> <p>So it allows you to make many more changes than you previously would have been able to deliver to the organization, and it allows you to address those changes very rapidly.</p> <p><strong>Gardner:</strong> How does this allow for a more common environment to extend across hybrid IT, from on premises to cloud to hybrid cloud and then ultimately to the edge?</p> <p><strong>Leech:</strong> Well, applications developed in containers and developed within a cloud-native approach typically are very portable. So you don't need to be restricted to a particular version or limits, for example. The container itself runs on top of any OS of the same genre. Obviously, you can't run a Windows container on top of a Linux OS or vice versa.</p> <p>But within the general Linux space, that pretty much has compatibility. So it makes it very easy for the containers to be developed in one environment and then released into different environments.</p> <p><strong>Gardner:</strong> And that portability extends to the hyperscale cloud environments, the public cloud, so is there a multicloud extensibility benefit?</p> <p><strong>Leech:</strong> Yes, definitely. You see a lot of developers developing their applications in an on-premises environment with the intention that they are going to be provisioned into a cloud. If they are done properly, it shouldn't matter if that's a <a href="https://cloud.google.com/">Google Cloud Platform</a> instance, a <a href="https://azure.microsoft.com/en-us/">Microsoft Azure</a> instance, or <a href="https://aws.amazon.com/">Amazon Web Services (AWS)</a>.</p> <p><strong>Gardner:</strong> We have quite an opportunity in front of us with containers across the spectrum of continuous development and deployment and for multiple deployment scenarios. What challenges do we need to think about to embrace this as a lifecycle approach?</p> <p>What are the challenges to providing security specifically, making sure that the containers are not going to add risk and, in fact, improve the deployment productivity of organizations?</p> <h3>Make security a business priority</h3> <p><strong>Leech:</strong> When I address the security challenges with customers, I always focus on two areas. The first is the business challenge of adopting containers and the security concerns and constrains that come along with that. And the second one is much more around the technology or technical challenges.</p> <p>If you begin by looking at the business challenges, of how to adopt containers securely, this requires a cultural shift, as I already mentioned. If we are going to adopt containers, we need to make sure we get the appropriate executive support and move past the concept that the developer is doing everything on his laptop. We train our coders on the needs for secure coding.</p> <p>A lot of developers have as their main goal to produce high-quality software fast, and they are not trained as security specialists. It makes a lot of sense to put an education program into place that allows you to train those internal coders so that they understand the need to think a little bit more about security, especially in a container environment where you have fast release cycles and sometimes the security checks get missed or don't get properly instigated. It's good to start with a very secure baseline.</p> <p>And once you have addressed the cultural shift, the next thing is to think about the role of the security team in your container development team, your DevOps development teams. And I always like to try and discuss with my customers the value of getting a security guy into the product development team from day one.</p> <p>Often, we see in a traditional IT space that the application gets built, the infrastructure gets designed, and then the day before it's all going to go into production someone calls security. Security comes along and says, "Hey, have you done risk assessments on this?" And that ends up delaying the project.</p> <p>If you introduce the security person into the small, agile team as you build it to deliver your container development strategy, then they can think together with the developers. They can start doing risk assessments and threat modeling right from the very beginning of the project. It allows us to reduce delays that you might have with security testing.</p> <p>At the same time, it also allows us to shift our testing model left in a traditional waterfall model, where testing happens right before the product goes live. But in a DevOps or DevSecOps model, it's much better to embed the security, best practices, and proper tooling right into the continuous integration/continuous delivery (CI/CD) pipeline.</p> <p>The last point around the business view is that, again, going back to the comment I made earlier, developers often are not aware of secure coding and how to make things secure. Providing a secure-by-default approach―or even a security self-service approach―allows developers to gain a security registry, for example. That provides known good instances of container images or provides infrastructure and compliance code so that they can follow a much more template-based approach to security. That also pays a lot of dividends in the quality of the software as it goes out the door.</p> <p><strong>Gardner:</strong> Are we talking about the same security precautions that traditional IT people might be accustomed to but now extending to containers? Or is there something different about how containers need to be secured?</p> <h3>Updates, the container way</h3> <p><strong>Leech:</strong> A lot of the principles are the same. So, there's obviously still a need for network security tools. There's still a need to do vulnerability assessments. There is still a need for encryption capabilities. But the difference with the way you would go about using technical controls to protect a container environment is all around this concept of the <a href="https://thenewstack.io/how-to-lock-down-the-kernel-to-secure-the-container/">shared kernel</a>.</p> <p>An interesting white paper has been released by the <a href="https://www.nist.gov/">National Institute of Standards and Technology</a> in the U.S., <a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf">SP 800-190</a>, which is their Application Container Security Guide. And this paper identifies five container security challenges around risks with the images, registry, orchestrator, the containers themselves, and the host OS.</p> <p>So, when we're looking at defining a security architecture for our customers, we always look at the risks within those five areas and try to define a security model that protects those best of all.</p> <p>One of the important things to understand when we're talking about securing containers is that we have a different approach to the way we do updates. In a traditional environment, we take a gold image for a virtual machine. We deploy it to the hypervisor. Then we realize that if there is a missing patch, or a required update, that we roll that update out using whatever patch management tools we use.</p> <p>In a container environment, we take a completely different approach. We never update running containers. The source of your known good image is your registry. The registry is where we update containers, have updated versions of those containers, and use the container orchestration platform to make sure that next time somebody calls a new container that it's launched from the new container image.</p> <p>It's important to remember we don't update things in the running environment. We always use the container lifecycle and involve the orchestration platform to make those updates. And that's really a change in the mindset for a lot of security professionals, because they think, "OK, I need to do a vulnerability assessment or risk assessment. Let me get out my Qualys and my Rapid7," or whatever, and, "I'm going to scan the environment. I'm going to find out what's missing, and then I'm going to deploy patches to plug in the risk."</p> <p>So we need to make sure that our vulnerability assessment process gets built right into the CI/CD pipeline and into the container orchestration tools we use to address that needed change in behavior.</p> <p><strong>Gardner:</strong> It certainly sounds like the orchestration tools are playing a larger role in container security management. Do those in charge of the container orchestration need to be thinking more about security and risk?</p> <h3>Simplify app separation</h3> <p><strong>Leech:</strong> Yes and no. I think the orchestration platform definitely plays a role and the individuals that use it will need to be controlled in terms of making sure there is good privileged account management and integration into the enterprise authentication services. But there are a lot of capabilities built into the orchestration platforms today that make the job easier.</p> <p>One of the challenges we've seen for a long time in software development, for example, is that developers take shortcuts by hard coding cleartext passwords into the text, because it's easier. And, yeah, that's understandable. You don't need to worry about managing or remembering passwords.</p> <p>But what you see a lot of orchestration platforms offering is the capability to deliver sequence management. So rather than storing the passcode within the code, you can now request the secret from the secrets management platform that the orchestration platform offers to you.</p> <p>These orchestration tools also give you the capability to separate container workloads for differing sensitivity levels within your organization.</p> <p>For example, you would not want to run containers that operate your web applications on the same physical host as containers that operate your financial applications. Why? Because although you have the capability with the container environment using separate namespaces to separate the individual container architectures from one another, it's still a good security best practice to run those on completely different physical hosts or in a virtualized container environment on top of different VMs.</p> <p>This provides physical separation between the applications. Very often the orchestrators will allow you to provide that functionality within the environment without having to think too much about it.</p> <p><strong>Gardner:</strong> There is another burgeoning new area where containers are being used. Not just in applications and runtime environments but also for data and persistent data. HPE has been leading the charge on making containers appropriate for use with data in addition to applications.</p> <p>How should the all-important security around data caches and different data sources enter into our thinking?</p> <h3>Save a slice for security</h3> <p><strong>Leech:</strong> Because containers are temporary instances, it's important that you're not actually storing any data within the container itself. At the same time, as importantly, you're not storing any of that data on the host OS either.</p> <p>It's important to provide persistent storage on an external storage array. So looking at storage arrays, things like from HPE, we have Nimble Storage or Primera. They have the capability through plug-ins to interact with the container environment and provide you with that persistent storage that remains even as the containers are being provisioned and deprovisioned.</p> <p>So your container itself, as I said, doesn't store any of the data, but a well-architected application infrastructure will allow you to store that on a third-party storage array.</p> <p><strong>Gardner:</strong> Simon, I've had an opportunity to <a href="https://community.hpe.com/t5/Transforming-IT/5-cultural-changes-to-make-container-security-and-DevSecOps-real/ba-p/7070077#.XlPrGC3MzUI">read some of your blogs</a>, and one of your statements jumped out: "The organizational culture still lags behind when it comes to security." What did you mean by that? And how does that organizational culture need to be examined, particularly with an increased use of containers?</p> <p><strong>Leech:</strong> It's about getting the security guys involved in the DevSecOps projects early on in the lifecycle of that project. Don't bring them to the table toward the end of the project. Make them a valuable member of that team. There was a comment made about the idea of a <a href="http://blog.idonethis.com/two-pizza-team/">two-pizza team</a>.</p> <p>A two-pizza team means a meeting should never have more people in it than can be fed by two pizzas, and I think that that applies equally to development teams when you're working on container projects. They don't need to be big; they don't need to be massive.</p> <p>It's important to make sure there's enough pizza saved for the security guy! You need to have that security guy in the room from the beginning to understand what the risks are. That's a lot of where this cultural shift needs to change. And as I said, executive support plays a strong role in making sure that that happens.</p> <p><strong>Gardner:</strong> We've talked about people and process. There is also, of course, that third leg of the stool: the technology. Are the people building container platforms, like HPE, thinking along these lines as well? What does the technology and the way it's being designed bring to the table to help organizations be DevSecOps-oriented?</p> <h3>Select specific, secure solutions</h3> <p><strong>Leech:</strong> There are a couple of ways that <a href="https://www.csoonline.com/article/3398485/28-devsecops-tools-for-baking-security-into-the-development-process.html">technology solutions</a> are going to help. The first are the pre-production commercial solutions. These are the things that tend to get integrated into the orchestration platform itself, like image scanning, secure registry services, and secrets management.</p> <p>A lot of those are going to be built into any container orchestration platform that you choose to adopt. There are also commercial solutions that support similar functions. It's always up to an organization to do a thorough assessment of whether their needs can be met by the standard functions in the orchestration platform or if they need to look at some of the third-party vendors in that space, like <a href="https://www.aquasec.com/">Aqua Security</a> or <a href="https://www.twistlock.com/">Twistlock</a>, which was recently acquired by <a href="https://www.paloaltonetworks.com/">Palo Alto Networks</a>, I believe.</p> <p>And then there are the solutions that I would gather up as post-production commercial solutions. These are for things such as runtime protection of the container environment, container forensic capabilities, and network overlay products that allow you to separate your workloads at the network level and provide container-based firewalls and that sort of stuff.</p> <p>Very few of these capabilities are actually built into the orchestration platforms. They tend to be third parties such as <a href="https://sysdig.com/">Sysdig</a>, <a href="https://www.guardicore.com/">Guardicore</a>, and <a href="https://neuvector.com/">NeuVector</a>. And then there's another bucket of solutions, which are more open source solutions. These typically focus on a single function in a very cost-effective way and are typically open source community-led. And these are solutions such as <a href="https://www.sonarqube.org/">SonarQube</a>, Platform as a Service (PaaS), and <a href="https://sysdig.com/opensource/falco/">Falco</a>, which is the open source project that Sysdig runs. You also have <a href="https://dockerbench.com/">Docker Bench</a> and <a href="https://www.projectcalico.org/">Calico</a>, a networking security tool.</p> <p>But no single solution covers all of an enterprise customer's requirements. It remains a bit of a task to assess where you have security shortcomings, what products you need, and who's going to be the best partner to deliver those products with those technology solutions for you.</p> <p><strong>Gardner:</strong> And how are you designing HPE Pointnext Services to fill that need to provide guidance across this still dynamic ecosystem of different solutions? How does the services part of the equation shake out?</p> <p><strong>Leech:</strong> We obviously have the technology solutions that we have built. For example, the HPE Container Platform, which is based around technology that we acquired as part of <a href="https://www.hpe.com/us/en/solutions/bluedata.html">the BlueData acquisition</a>. But at the end of the day, these are products. Companies need to understand how they can best use those products within their own specific enterprise environments.</p> <p>I'm part of Pointnext Services, within the advisory and professional services team. A lot of the work that we do is around advising customers on the best approaches they can take. On one hand, we'd like them to purchase our HPE technology solutions, but on the other hand, a container-based engagement needs to be a services-led engagement, especially in the early phases where a lot of customers aren't necessarily aware of all of the changes they're going to have to make to their IT model.</p> <p>At Pointnext, we deliver a number of container-oriented services, both in the general container implementation area as well as more specifically around container security. For example, I have developed and delivered transformation workshops around DevSecOps.</p> <p>We also have container security planning workshops where we can help customers to understand the security requirements of containers in the context of their specific environments. A lot of this work is based around some discovery we've done to build our own container security solution reference architecture.</p> <p><strong>Gardner:</strong> Do you have any examples of organizations that have worked toward a DevSecOps perspective on continuous delivery and cloud native development? How are people putting this to work on the ground?</p> <h3>Edge elevates container benefits</h3> <p><strong>Leech:</strong> A lot of the customers we deal with today are still in the early phases of adopting containers. We see a lot of POC engagement where a particular customer may be wanting to understand how they could take traditional applications and modernize or architect those into cloud-native or container-based applications.</p> <p>There's a lot of experimentation going on. A lot of the implementations we see start off small, so the customer may buy a single technology stack for the purpose of testing and playing around with containers in their environment. But they have intentions within 12 to 18 months of being able to take that into a production setting and reaping the benefits of container-based deployments.</p> <p><strong>Gardner:</strong> And over the past few years, we've heard an awful lot of the benefits for moving closer to the computing edge, bringing more compute and even data and analytics processing to the edge. This could be in a number of vertical industries, from autonomous vehicles to manufacturing and healthcare.</p> <p>But one of the concerns, if we move more compute to the edge, is will security risks go up? Is there something about doing container security properly that will make that edge more robust and more secure?</p> <p><strong>Leech:</strong> Yes, a container project done properly can actually be more secure than a traditional VM environment. This begins from the way you manage the code in the environment. And when you're talking about edge deployments, that rings very true.</p> <p>From the perspective of the amount of resources it has to use, it's going to be a lot lighter, when you're talking about something like autonomous driving, to have a shared kernel rather than lots of instances of a VM running, for example.</p> <p>From a strictly security perspective, if you deal with container lifecycle management effectively; involve the security guys early; have a process around releasing, updating, and retiring container images into your registry; and have a process around introducing security controls and code scanning in your software development lifecycle―making sure that every container that gets released is signed with an appropriate enterprise signing key―then you have something that is very repeatable, compared with a traditional virtualized approach to application and delivery.</p> <p>That's one of the big benefits of containers. It's very much a declarative environment. It's something that you prescribe. This is how it's going to look. And it's going to be repeatable every time you deploy that. Whereas with a VM environment, you have a lot of VM sprawl. And there are a lot of changes across the different platforms as different people have connected and changed things along the way for their own purposes.</p> <p>There are many benefits with the tighter control in a container environment. That can give you some very good security benefits.</p> <p><strong>Gardner:</strong> What comes next? How do organizations get started? How should they set themselves up to take advantage of containers in the right way, a secure way?</p> <h3>Begin with risk evaluation</h3> <p><strong>Leech:</strong> The first step is to do the appropriate due diligence. Containers are not going to be for every application. There are going to be certain things that you just can't modernize, and they're going to remain in your traditional data center for a number of years.</p> <p>I suggest looking for the projects that are going to give you the quickest wins and use those POCs to demonstrate the value that containers can deliver for your organization. Make sure that you do appropriate risk awareness, work with the services organizations that can help you. The advantage of a services organization is they've probably been there with another customer previously so they can use the best practices and experiences that they have already gained to help your organization adopt containers.</p> <p>Just make sure that you approach it using a DevSecOps model. There is a lot of discussion in the market at the moment about it. Should we be calling it DevOps or should we call it SecDevOps or DevOpsSec? My personal opinion is call it DevSecOps because security in a DevSecOps module sits right in the middle of development and operations, and that's really where it belongs.</p> <p>In terms of assets, there is plenty of information out there in a Google search; it finds you a lot of assets. But as I mentioned earlier, the <a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf">NIST white paper SP 800-190</a> is a great starting point to understand not only container security challenges but also to get a good understanding of what containers can deliver for you.</p> <p>At the same time, at <a href="https://www.hpe.com/us/en/home.html">HPE</a>, we are also committed to delivering relevant information to our customers. If you look on our website, you will see a lot of articles about best practices on container deployments, case studies, and architectures for running container orchestration platforms on our hardware. All of this is available for people to download and to consume.</p> <p><strong>Gardner:</strong> I'm afraid we will have to leave it there. We have been exploring how container-based deployment models have gained popularity, from cloud models to corporate data centers. And we have learned how, in order to push containers further into the mainstream, security concerns need to be addressed across this new end-to-end container deployment spectrum.</p> <p>So please join me in thanking our guest, Simon Leech, worldwide security and risk management practice at HPE Pointnext Services. Thank you so much, Simon.</p> <p><strong>Leech:</strong> Thanks for having me.</p> <p><strong>Gardner:</strong> I learned a lot. And thanks as well to our audience for joining this sponsored BriefingsDirect Voice of Innovation discussion. I'm Dana Gardner, principal analyst at Interarbor Solutions, your host for this ongoing series of HPE-supported discussions.</p> <p>Thanks again for listening. Please pass this along to your IT community, and do come back next time.</p> <p>Container deployment best practices: Lessons for leader</p> <ul> <li> <p>Begin with a risk assessment―containers aren't appropriate for every application.</p> </li> <li> <p>Look for projects that will give quick wins and show value.</p> </li> <li> <p>Approach container projects using a DevSecOps model, with security integrated from the start.</p> </li> </ul> <p>This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.</p> <p><u><strong>About the author:</strong></u></p> <p>Since 1999, Dana Gardner has emerged as a leading identifier of enterprise software solutions, strategies, partnerships, and markets and new IT business value opportunities. He is frequently quoted as a thought leader in top news and IT industry publications such as The New York Times, The Wall Street Journal, The Boston Globe, The Washington Post, Businessweek, San Francisco Chronicle, Reuters, Associated Press, MSNBC.com, CNN.com, and more. As a skilled multimedia communicator and evangelist, he has written dozens of industry reports on the business benefits of IT and Internet innovation for advancing general productivity, improving employee efficiency, and reducing total IT costs. As founder and president of Interarbor Solutions, Dana has taken a strong record in consulting services for IT vendors, carriers, and enterprises to yet another level: the exciting new communications capabilities around Internet social media. Businesses of all kinds are quickly exploiting blogs, podcasts, and video podcasts for education, communications, and viral outreach. Dana practices what he preaches as a frequent blogger, on ZDNet and his personal blog, as well as a podcaster. He began podcasting as a founding member of the Gillmor Gang in 2005.</p>https://developer.hpe.com/jupyter-saved-my-day/https://developer.hpe.com/jupyter-saved-my-day/Tue, 10 Mar 2020 20:48:19 GMT<p>Prepping for an industry event always requires a lot of planning, and sometimes a lot of last minute juggling. Preparing for the Hack Shack activities for HPE’s Technology and Solutions Summit (TSS) event in Paris was no different. With the event only a few weeks away, everything appeared to be ready to go. We had lined up the subjects matter experts, reserved our space, attracted a registered audience, and arranged for a dedicated infrastructure to host the technical activities. Unfortunately, the Coronavirus situation escalated and, for the safety of everyone, HPE has decided to adjust the onsite event and turn parts of it into a virtual event.</p> <h2>How do you switch to a virtual event quickly?</h2> <p>Before the announcement of the onsite event cancellation, event organizers had already contacted the different teams who were participating to figure out how the event could be delivered virtually. They proposed that a subset of the sessions be offered. Those selected would need to be eligible for remote delivery.</p> <p>Because our team had already learned how to deal with remote delivery, most of our sessions already fit that requirement. This made it fairly easy to propose a new agenda that included nearly all of the originally planned workshops. It has been a few years since we last shipped material to an event location to support our labs or workshops. Due to budget constraints, and also simple noise issues (it is not easy to run a lab with racks full of servers running next to you), most of our lab activities became remote based-labs anyway. A simple and reliable internet connection is now all that one needs to run them.</p> <h2>Jupyter saves the day</h2> <p>HPE DEV Hack Shack sessions are a little more complicated than simple slide presentations. We always do our best to provide attendees with a hands-on experience that they can’t get elsewhere. So, how would we achieve similar results in a virtual situation?</p> <p>During a TSS event in 2018, a participant provided his solution to a Hack Shack challenge in a Jupyter Notebook format. Didier Lalli, our team leader, found both the solution and the format interesting. As a consequence, he proposed that our team adopt this format for any future labs. I volunteered to take a look at Jupyter Notebooks and see how we could best use it to deliver our labs. In the end, I proposed we use Jupyterhub, as I felt it was the most appropriate tool to address our needs. As a happy happenstance, this decision put us in a great position to turn TSS 2020 into a virtual event.</p> <h2>What are Jupyter and Jupyterhub?</h2> <p>To give you a bit of background: The Jupyter project (for Julia / Python / R) is the result of a project called IPython, which is an advanced Python interpreter, that improves the productivity of your code in Python. IPython has gradually evolved, notably with the creation of Notebooks, and now offers a web interface in JSON format that allows you to run a Python kernel and to code directly in a browser with a display of intermediate results. The Jupyter Project was an important step forward for sharing and "interactive" development. <a href="https://jupyter.org/index.html">Project Jupyter’s</a> <a href="https://jupyterhub.readthedocs.io/en/stable/">JupyterHub</a> was created to support many users. The Hub can offer notebook servers to an entire class of students, a corporate data science workgroup, a scientific research project team, or a high-performance computing group. With <a href="https://github.com/jupyterhub/jupyterhub">JupyterHub,</a> you can create a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user <a href="https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb">Jupyter Notebook</a> server.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1583962183422.png" alt="picture1"></p> <p>The setup we plan on using for the TSS event consists of one HPE ProLiant DL360 Gen 10 with 2x Intel Xeon Silver 4114 CPUs running at 2.2GHz with 10 Cores in each (20 in Hyperthreading mode). The system has 160 GB of ram. It is running an Ubuntu 18.04 distribution and installation was performed following the Jupyter guidelines provided <a href="https://jupyterhub.readthedocs.io/en/stable/installation-guide-hard.html">here.</a> We are leveraging Python and Powershell kernels. I installed <a href="https://github.com/timkpaine/jupyterlab_iframe/">Iframe</a> as well, but have not made any use of it so far.</p> <h2>What is a Jupyter Notebook?</h2> <p>The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. In our case, Notebooks contain simple Python or Powershell pieces of codes to interact with the different APIs available in the HPE portfolio. Instructions are provided in a markdown format. We centralize the different notebooks on a single Jupyterhub server. This allows us to replicate them across the 40 students currently configured on the server fairly easily through Ansible playbooks. When changes are necessary (fixing a typo for instance in a notebook), updates can be performed quickly by just launching the playbook.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture2-1583962189410.png" alt="picture2"></p> <h2>Advantages of using Jupyter Notebooks</h2> <p>From a course standpoint, the notebook format is a really simple and flexible solution. The HPE DEV team used to deliver labs in a very academic fashion providing Adobe Acrobat Reader files to the lab attendees. Lab attendees would follow the instructions of the lab, and copy and paste commands from the pdf file to the Windows terminal server session or to a simple PuTTY session. In many cases, errors would occur because some hidden characters were copied from the pdf document to the session (credentials for instance).</p> <p>In addition, the format was not the most flexible when any changes needed to be made to the master document (a doc file to be converted in pdf). One would have to redistribute the new version of the file to all the students, making things cumbersome. Even though we always managed to overcome these issues, Didier’s idea was quite appealing since most of our labs and workshops use coding exercises based on Python and Powershell.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture3-1583962195154.png" alt="picture3"></p> <p>Taking Didier’s idea, from a logistical standpoint, we architected a solution that could be accessible on the internet. Using Jupyter Notebooks allows us to deliver labs from simply anywhere. The students only needs an internet access to our Jupyterhub server et voila! They can even download the notebooks and run them locally on their laptops if they want to do them again later on or reuse some of the code for their own projects.</p> <p>As I worked on developing this solution, I realized that there were so many new possibilities for its use, like Jupyter Notebooks-as-a-Service. This could be a service that would allow you to learn about the latest and greatest updates from our different APIs. At a minimum, these Jupyter Notebooks will be used in the Hack Shack for future events, like HPE Discover 2020. We are also investigating a simple sharing possibility leveraging our GitHub repository in order to provide people with some of our notebooks.</p> <p>We are just starting off using this Jupyter technology. The future holds great promise as the forthcoming HPE Container Platform will allow us to deploy and use Jupyter Notebooks beyond simple API calls. For instance, we will have the opportunity to dive into the Artificial Intelligence world through notebooks that make use of TensorFlow technology. Please be sure to check back at <a href="https://developer.hpe.com/">HPE DEV</a> for a follow up on this.</p>https://developer.hpe.com/second-internal-hpe-hackathon-around-iot-and-ai/https://developer.hpe.com/second-internal-hpe-hackathon-around-iot-and-ai/Thu, 05 Mar 2020 18:11:36 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1583431980969.png" alt="picture1"></p> <p>Under the banner of *"We burn for IoT and AI", the HPE presales IoT &#x26; Data Analytics team recently hosted our second Hackathon in Boeblingen, Germany. As more and more concrete IoT and AI projects come our way, we find hackathons helpful in developing our knowledge base. Hackathons help us build proficiency by going beyond slide presentations and offering hands-on experience with the technologies required to address the needs of IoT and AI projects.</p> <p>As a Hybrid IT presales consultant who was responsible for the planning of the first and the second hackathon, I am proud to say that we not only doubled the number of participants but also the number of use cases we developed. About 20 highly motivated people from the HPE AI, Data Analytics &#x26; IoT Competence Center, HPE Pointnext Advisory and Professional Services, and HPE Pointnext Delivery groups pursued the goal of educating themselves and creating tangible outcomes. Besides colleagues from more traditional HPE departments, colleagues from new HPE acquisitions, BlueData and MapR, participated and made a noticeable impact on the development of the use cases. The HPE Customer Technology Center (CTC) in Boeblingen provided us with the perfect location for planning, developing and implementing our visions.</p> <p>Use case topics ranged from IoT to AI, with the goal of building Edge-to-Cloud Data Pipelines powered by HPE infrastructure and software that address today’s customer challenges. To give you an idea of what we were able to achieve, here’s a summary of the use cases we covered:## Industrial Data Fabric from Edge-to-Cloud with OT Link &#x26; MapR Edge Cluster</p> <p>In our first use case, we aimed at setting up an Edge-to-Cloud Data Pipeline with an industrial focus. To achieve this, we built a MapR Edge cluster on an HPE Edgeline server and connected it to an already installed MapR datalake on HPE Apollo servers in the data center. Using features like the MapR global namespace and the event store for Apache Kafka, we were able to collect data at the edge and process it in the core/cloud. The data collection performed by an HPE OT Link instance brought the industrial focus to the use case. In fact, HPE OT Link was used to combine operation technology capabilities with enterprise-class IT capabilities in a single system. In the end, we were able to route Industrial IoT data acquired by the OT Link through a MapR Data Pipeline to the core/cloud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1583268352426.png" alt="picture1"></p> <h2>Image Classification using HPE ML Ops, powered by NVIDIA GPU acceleration</h2> <p>The goal of this use case was to recognize server parts through an Android app to assist users in differentiating components. A corresponding model for the classification of the components was trained using HPE ML Ops. The training and inference took place on HPE Apollo servers, equipped with Nvidia GPUs. In particular, HPE ML Ops along with GPU acceleration noticeably shortened the training process. The final result was a deep-learning model that was deployed with the feature set of HPE ML Ops.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture2-1583268373183.png" alt="picture2"></p> <h2>Industrial Data Pipeline and visualization for OEE calculation with Edgeline, OT Link and Azure IoT Hub</h2> <p>In the industrial Data Pipeline use case, we wanted to calculate the key performance indicator for Overall Equipment Effectiveness (OEE) in HPE OT Link and visualize it utilizing OT Links dashboard capabilities and Grafana. For this purpose, we gathered a variety of machine data and built different flows in OT Link in order to calculate the OEE. We sent the telemetry data to Microsoft Azure and added the Azure IoT Edge Device integration in OT Link. As a final result, data was sent to Azure and written into a time-series database. The real-time data is visualized in Grafana and in OT Link.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture3-1583268389028.png" alt="picture3"></p> <h2>Twitter Sentiment Analysis using MapR Event Store, Kafka, and R on HPE ML Ops</h2> <p>In the last use case, our goal was to perform social media sentiment analysis using HPE ML Ops. For this purpose, a Data Pipeline was built for batch analytics. Using the statistical programming language R, various statistical analyses about the popularity of the keyword were conducted. In a second step, a streaming component utilizing the MapR Event Store for Apache Kafka was added. The resulting real-time data was visualized using a dashboard displaying various reports. The result of this use case is an in-depth batch analysis and a real-time dashboard evaluation to the popularity of the given keyword.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture4-1583268402834.png" alt="picture4"></p> <p>Some impressions of a week full of hacking and new developments around HPE products and IoT/AI can be seen below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture5-1583268416136.png" alt="picture5"></p> <p>Apart from topic-related synergy effects, the team spirit between colleagues in different positions was strengthened. We were excited that our organizational leaders were also interested in our challenges and came around to watch us tackling these challenges. Just like they did for our first hackathon, <a href="https://developer.hpe.com/community">HPE DEV</a> supported us by supplying us not only equipment but also with helpful tips for planning the event. Additionally, we benefited from Nvidia’s support, since the Nvidia GPUs made a significant difference in training and inference for our second use case.</p> <p>I want to thank the participants for a great week and fantastic achievements. A special thank you goes to our leaders for making this possible.</p> <p>In the end, all I can say is that AI, Data Analytics &#x26; IoT Competence Center is already looking forward to the next Hackathon!</p>https://developer.hpe.com/2020-March-03/https://developer.hpe.com/2020-March-03/Tue, 03 Mar 2020 06:00:00 GMThttps://developer.hpe.com/hpe-brings-a-little-magic-to-kubecon-cloudnativecon-europe/https://developer.hpe.com/hpe-brings-a-little-magic-to-kubecon-cloudnativecon-europe/Wed, 19 Feb 2020 17:32:59 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/kubecon-eu-new-date-image3-1585572572202.png" alt="kubecon eu new date image3"></p> <p>Will you be among the 10,000 technologists heading to Amsterdam from August 13 – 16, 2020 to attend <a href="https://www.cncf.io/">Cloud Native Computing Foundation’s</a> flagship conference, <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon | CloudNativeCon EU</a>? We will and we’re so excited for opportunity to demonstrate a little magic while we show off what Hewlett Packard Enterprise (HPE) offers in the areas of containers and Kubernetes (k8s)!</p> <p>HPE will be featured in two KubeCon breakout sessions. Both will take place on Wednesday, April 1st. The first session, <strong>Topology Aware Scheduling using Prometheus and Telemetry Aware Scheduler</strong>, is co-sponsored by HPE and Intel and given by Chief Technologist, Tom Golway (HPE), and Cloud Software Engineer, Swati Sehgal (Intel). The second breakout session, <strong>Taming Data/State Challenges for ML Applications and Kubeflow</strong>, will be presented by HPE Distinguished Technologist, Skyler Thomas. Refer to the <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/schedule/">program schedule</a> for specific times and locations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture10-1582133922632.png" alt="picture10"></p> <p>On the event floor at booth P17, our magician will greet you and introduce you to HPE representatives from our different software-enabling and application development groups. You’ll get to learn about some pretty magical technologies and programs, including:</p> <ul> <li>The <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Container Platform</a>, the industry’s first Kubernetes container platform software designed to run both cloud-native and non cloud-native applications, enabling true hybrid cloud operations across <strong>any</strong> location. Now if that’s not magical, we don’t know what is!</li> <li>The <a href="https://www.hpe.com/us/en/storage/intelligent-storage.html?chatsrc=ot-en&#x26;jumpid=ps_8r5mdg32xs_aid-520023673&#x26;gclid=Cj0KCQiAs67yBRC7ARIsAF49CdU6O6Hbaj1lwT8tcrU702BzRnZboWNQILTShb0cCk-eEk7nUjQ-yhMaAv4fEALw_wcB&#x26;gclsrc=aw.ds">intelligent data platform</a> from <a href="https://www.hpe.com/us/en/storage.html">HPE storage</a>, focused on persistent storage use cases for Kubernetes in private, public, and hybrid clouds and how to enable CI/CD pipelines. Another bit of magic we know you’ve been waiting to see.</li> <li><a href="https://developer.hpe.com/">HPE DEV</a>, a growing community that helps developers and designers focus on creating the best possible software experiences using open source and HPE products. The magic here is how much easier you can make your life by tapping into these resources.</li> </ul> <p>You’ll want to catch at least one of our theater sessions that will run throughout the day. There will be a random drawing for an Argon Kit after each presentation. Presenters will cover topics such as:</p> <ul> <li>An introduction to the HPE Container Platform</li> <li>Exploring the benefits of joining the HPE DEV community</li> <li>Persistent Storage for Kubernetes</li> <li>Hybrid Cloud CI/CD with Nimble/CV/HCP</li> <li>Taming Data and State Challenges with Kubeflow</li> <li>App centric MongoDB 3PAR/Primera</li> <li>SPIRE and SPIFFE</li> </ul> <p>HPE’s newest acquisition, Scytale, a cloud native security startup that is built on the open-source Secure Production Identity Framework for Everyone (SPIFFE) protocol, will also be in attendance in a separate booth (S48) on the tradeshow floor. Some Scytale presentations will also be given in booth P17.</p> <p>As always, the <a href="https://developer.hpe.com/community">HPE DEV community</a> is excited to have an opportunity to connect with developers and designers at an event such as this. If you’re going, make sure you stop on by and see us at HPE booth P17. Hope to see you in Amsterdam!</p>https://developer.hpe.com/understanding-concurrency-in-python-part-3-asyncio/https://developer.hpe.com/understanding-concurrency-in-python-part-3-asyncio/Wed, 19 Feb 2020 17:32:20 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1582323160514.png" alt="picture1"></p> <p>If you have been following my blog posts on concurrency in Python, you’ll remember that I first covered the <a href="/blog/understanding-concurrency-in-python-part-1-threading">threading library in Python</a> and found that it helped significantly with multi-threaded execution of I/O bound tasks. However, it did not improve efficiencies for CPU bound tasks. In <a href="/blog/understanding-concurrency-in-python-part-2-multiprocessing">Part 2</a>, I pointed out how multiprocessing can help you get around this. In this post, I will show you how the Python asyncio library can help achieve concurrency of processes by letting the application have control over context switching. This library can come in handy when you are dealing with APIs that are implemented with a polling mechanism to handle asynchronous tasks.</p> <p><em>Asyncio</em> became associated with Python in version 3.4. It is a single thread/single process cooperative multitasking library that uses something called co-routines, event loops, and awaitable objects to achieve concurrency. An asyncio task has exclusive use of the CPU until it wishes to give it up to the event loop. Asyncio is very beneficial for I/O bound operations, but not very helpful for CPU bound operations.</p> <p>Let’s use the same I/O bound example we used in Part 1 and Part 2 of this series to fetch responses from websites and see how to implement concurrency of this I/O bound operation using asyncio.</p> <p>Step 1: Import the necessary libraries and modules.</p> <pre><code class="language-python"> import asyncio import requests import time </code></pre> <p>Step 2: Define the co-routines. A co-routine can be defined by prefixing the keyword async before the function definition. An asyncio co-routine is a function that can pause and resume during execution. This means that it acts, more or less, like a <a href="https://wiki.python.org/moin/Generators">generator</a> in Python.</p> <pre><code class="language-python"> async def get_response(site): return requests.get(site) async def main(): # define some sites to query tasks = [] sites = ["http://www.google.com", "http://www.linkedin.com", "http://www.quora.com", "http://www.facebook.com"] for site in sites: tasks.append(asyncio.ensure_future(get_response(site))) await asyncio.gather(*tasks) </code></pre> <p><em>ensure_future</em> takes in a co-routine and returns the future version of it. Basically, it assumes that a result will eventually be given to it at some point in time. And while a co-routine is waiting, its execution is temporarily suspended. Once the future of that co-routine is set to some result, the co-routine is resumed.</p> <p>Now, create the event loop and call the co-routines to fetch responses. Capture the time taken for execution to see how asyncio performs.</p> <pre><code class="language-python"> start_time = time.time() loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() print("Time taken for asyncio", time.time()-start_time) </code></pre> <p>Using asyncio in this example, I got the following results:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture3-1582323145443.png" alt="picture3"></p> <p>You can see that the asyncio execution took relatively less time than a regular iterative execution of an I/O bound operation.</p> <p>The complete code we used to illustrate how the asyncio library helps would look like this:</p> <pre><code class="language-python"> import asyncio import requests import time async def get_response(site): return requests.get(site) async def main(): # define some sites to query tasks = [] sites = ["http://www.google.com", "http://www.linkedin.com", "http://www.quora.com", “http://www.facebook.com"] for site in sites: tasks.append(asyncio.ensure_future(get_response(site))) await asyncio.gather(*tasks) start_time = time.time() loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() print("Time taken for asyncio", time.time()-start_time) </code></pre> <p>Note - You might come across an error that says ‘event loop is already running’, while executing the above code in Jupyter Notebooks. In that case, either run the code from your terminal as a regular Python program or fix the issue by installing nest_asyncio from <a href="https://pypi.org/project/nest-asyncio/">https://pypi.org/project/nest-asyncio/</a> and importing it in the code.</p> <h2>Conclusion</h2> <p>As you can see from the example above, the Python asyncio library can be very advantageous when you want the application to have more control over context switching than the operating system, especially when dealing with APIs that have a polling mechanism or any other I/O bound operation.</p> <p>Hopefully, after reading through all three of my articles, you now have a basic understanding of the libraries that are available in Python for concurrency, including:</p> <ul> <li>Threading</li> <li>Multiprocessing</li> <li>Asyncio</li> </ul> <p>By reading these posts, I hope you also learned to determine which library to employ based on whether the operation is I/O bound or CPU bound. My three articles just covered the basics of concurrency. Is there more to it? Definitely! You may want to explore more on locks in threading, pools in multiprocessing, and a bunch of other cool features found in <a href="https://docs.python.org/3/library/asyncio.html">asyncio.</a></p> <p>You’ll find more articles and tutorials like this on the <a href="/blog">HPE blog site.</a> Remember to check back of-ten to see what’s new! Don’t forget, you can follow me on Twitter <a href="https://twitter.com/deyagondsamarth">@deyagondsamarth</a> or connect with me on <a href="https://hpedev.slack.com/?redir=%2Fteam%2FUQM0ZTE1F">Slack.</a> Happy coding!</p>https://developer.hpe.com/understanding-concurrency-in-python-part-2-multiprocessing/https://developer.hpe.com/understanding-concurrency-in-python-part-2-multiprocessing/Wed, 19 Feb 2020 17:23:08 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture2-1582133413956.png" alt="picture2"></p> <p>In Part 1 of this series, <a href="/blog/understanding-concurrency-in-python-part-1-threading">Understanding Concurrency in Python</a>, I covered the timing of multi-threaded executions. Through specific examples, I showed you that, no matter how many cores you have on your computer, the Python threading library does not really help you fully exploit the abilities of multi-threading. But a resolution is available, which I promised to show you. I am covering that resolution here in Part 2. Thanks to the Python multiprocessing library, you can take complete advantage of all the cores in your computer and more efficiently handle CPU bound functions.</p> <p>Multiprocessing allows you to run functions as independent Python processes on different cores. Even though these processes run independently, they can still communicate with each other when needed.</p> <p>Let’s look at an example where multiprocessing helps us achieve concurrency and the speed required to handle a CPU bound function. To compare and contrast our results, we will be using the same example we looked at in our previous post to determine how threading affected CPU bound functions.</p> <p>Step 1: Import the necessary libraries and modules.</p> <pre><code class="language-python"> import multiprocessing import time </code></pre> <p>Step 2: Define a CPU intensive function cpu_bound() that accepts a number, multiplies it by a million, and calculates the sum of all the numbers in a range of 0 to that product. For ease of reference, we’ll use the same example we used in Understanding Concurrency in Python Part 1 – Threading. As we did previously, remember to create an additional list of random numbers. The example we used before is shown below:</p> <pre><code class="language-python"> def cpu_bound(num): return sum([i for i in range(num*1000000)]) numbers = [11, 23, 53, 34] </code></pre> <p>When we previously used this example as we looked at threading, we determined the time taken for regular execution of this function iterated over the list of numbers. In that instance, the results were as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture4-1582133425400.png" alt="picture4"></p> <p>Step 3: This time, let’s capture the time taken to execute this CPU intensive function using the Python Multiprocessing library to invoke multiple processes. Here, we use the Process method of the multiprocessing library, which takes two parameters. One parameter is target, which is set to <em>cpu_bound</em> function, and another is <em>args,</em> which is the arguments for the function <em>cpu_bound</em> function.</p> <pre><code class="language-python"> start_time = time.time() for number in numbers: p = multiprocessing.Process(target=cpu_bound, args=(number,)) p.start() print("Time taken for multi-processing execution", time.time()-start_time) </code></pre> <p>When I re-executed the regular and multiprocessing-fashioned code, the results were as follows:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture5-1582133442778.png" alt="picture5"></p> <p>This is so awesome! You can see the drastic reduction in the time taken to execute the same <em>cpu_bound()</em> function using the multiprocessing library where <em>cpu_bound</em> function is executed as an independent process for every number in the list.</p> <p>The complete code that we used to illustrate how the multiprocessing library helps would look like this:</p> <pre><code class="language-python"> import multiprocessing import time def cpu_bound(num): return sum([i for i in range(num*1000000)]) numbers = [11, 23, 53, 34] start_time = time.time() for number in numbers: cpu_bound(number) print("Time taken for regular execution", time.time()-start_time) start_time = time.time() for number in numbers: p = multiprocessing.Process(target=cpu_bound, args=(number,)) p.start() p.join() print("Time taken for multi-processing execution", time.time()-start_time) </code></pre> <p>If you are working on a Microsoft Windows-based system, you can open your Task Manager while the above Python code is running to see the number of Python processes being spawned.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture6-1582133457943.png" alt="picture6"></p> <h2>Summary</h2> <p>As promised, I’ve shown you how to achieve efficient concurrency in Python for CPU bound functions using the Multiprocessing library. This method executes the target method as independent processes for every input, thus utilizing the CPU resources to the maximum extent. However, the control over context switching between the processes is still with the operating system. This might be a concern at times. Is there an alternative for this? Yes, there is! So, make sure you check out my next post, <em>Understanding Concurrency in Python Part 3 – Asyncio</em>, to learn more about how this library can let an application have control over context switching and execute multiple functions simultaneously. You can read all my blog posts on <a href="/blog">HPE DEV.</a> Feel free to reach out to me with any questions on <a href="https://hpedev.slack.com/?redir=%2Fteam%2FUQM0ZTE1F">Slack</a> or connect with me on Twitter <a href="https://twitter.com/deyagondsamarth">@deyagondsamarth.</a></p>https://developer.hpe.com/automation-is-not-testing/https://developer.hpe.com/automation-is-not-testing/Tue, 18 Feb 2020 17:55:26 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1582049527283.png" alt="picture1"></p> <p>I recently attended a webinar where the speaker commented that "Automation is not testing". It started me thinking about testing in general and some of the challenges it can present. We all know that we should be testing scripts, but you can fall into a lot of potential pit falls when starting this journey. In this article, I’m going to jump straight into the deep end and deal with one of the struggles that took me a while to deal with.</p> <p>Imagine the following:</p> <ul> <li>You have a local device with a REST API.</li> <li>You write a library that accesses that REST API.</li> <li>You write tests for that library so it runs against the local devices REST API.</li> <li>You push the library to GITHUB so other people can leverage your work. (You are a good person, right? )</li> <li>You configure TravisCI or CircleCI for integration testing.</li> <li>You realize all your tests fail because Travis/Circle doesn’t have access to your internal device.</li> </ul> <p>And now, your GITHUB badges all show red, and no one trusts your code. Which brings us to <em>vcrpy.</em></p> <h2>What’s vcrpy?</h2> <p>Wow! So glad you asked! <em>Vcrpy</em> is a REST API library that helps to automatically record responses from a REST API and captures them on a local file you can reuse later. According to the <em>vcrpy</em> docs, (available <a href="https://vcrpy.readthedocs.io/en/latest/">here</a>), the three main benefits of the library are:</p> <ul> <li>The ability to work offline</li> <li>Completely deterministic tests</li> </ul> <p>*Increased test execution speed</p> <p>Let’s dig in by writing a quick piece of code that is going to access a REST API. In this case, I’m going to use a public API, but let’s imagine it’s behind your firewall on a device where you don’t want anyone fiddling with anything.</p> <p>For this example, I’m going to be using the public API at <a href="https://api.kanye.rest">https://api.kanye.rest</a>. This is a public API that responds to every GET request with a quote from Kanye West. Let’s use python to create a small library that will access the API, print a message, and return the quote as JSON. If you want to take a look at the library, feel free to check out the GITHUB repository <a href="https://github.com/netmanchris/pykanyerest">pykanyerest.</a> If you don’t want to leave this page, I’ve included the library function below.</p> <pre><code class="language-python"> import requests import json def get_new_quote(): url = 'https://api.kanye.rest' r = requests.get(url) quote = json.loads(r.text) print ("New Kanye Quote coming up!") return quote </code></pre> <p>Now that we’ve got the new function built, let’s take a quick look at what we get. We will run the function and capture it in a python variable called x.</p> <pre><code class="language-python"> x = get_new_quote() New Kanye Quote coming up! </code></pre> <p>Now, let’s use the json library to take a look at what was returned.</p> <pre><code class="language-python"> json.dumps(x) '{"quote": "I’m nice at ping pong"}' </code></pre> <p>Now that we’ve got some working code, let’s run it again and capture this as the python variable y.</p> <pre><code class="language-python"> y = get_new_quote() New Kanye Quote coming up! json.dumps(y) '{"quote": "The world is our office"}' </code></pre> <p><em>Hmmmmm…</em></p> <p>As you can see, each time you perform a GET against the API, another variable is returned. You can imagine this probably isn’t the best thing for testing because we really don’t have a clear indication as to what <em>exactly</em> we’re going to expect from this API.</p> <p>We do know a couple of things that can be used for testing:</p> <ul> <li>the returned object is a python dictionary</li> <li>the returned object has a single key/value pair</li> <li>the returned object’s first key is “quote”</li> <li>the returned object’s first value is going to be an object of type STR</li> </ul> <p>Imagine code that could test for all these things.</p> <p><em>Take your time… I’ll wait.</em></p> <p>While you were imagining it, I wrote a quick test to help make sure that the API returns what we expect.</p> <p><strong>If you don’t expect anything, you deserve what you get, right?</strong></p> <pre><code class="language-python"> from unittest import TestCase from pykanyerest.quotes import * class TestGetNewQuote(TestCase): """ Test Case for get_new_quote function from kanye.rest """ def test_GetNewQuote(self): """ """ quote = get_new_quote() self.assertEqual(type(quote), dict) self.assertEqual(len(quote), 1) keys = quote.keys() self.assertIn('quote', keys) self.assertEqual(type(quote['quote']), str) </code></pre> <p>If you were to run this test, you would find that it passes. <em>You can trust me on this, right?</em></p> <p>Now, we’ve already got a lot we can hang our tests on, but imagine we want to also test the exact contents of the value. The code to test would now look like this:</p> <pre><code class="language-python"> class TestGetNewQuote(TestCase): """ Test Case for get_new_quote function from kanye.rest """ def test_GetNewQuote(self): """ """ quote = get_new_quote() self.assertEqual(type(quote), dict) self.assertEqual(len(quote), 1) keys = quote.keys() self.assertIn('quote', keys) self.assertEqual(type(quote['quote']), str) self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death") </code></pre> <p>If you ran THIS code, you would now find it fails.</p> <pre><code class="language-python"> Traceback (most recent call last): File "/Users/christopheryoung/PycharmProjects/OpenTestingBlog/tests/test_quotes.py", line 18, in test_GetNewQuote self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death") File "/Applications/PyCharm.app/Contents/helpers/pycharm/teamcity/diff_tools.py", line 38, in _patched_equals raise error teamcity.diff_tools.EqualsAssertionError: :: People only get jealous when they care. != If I got any cooler I would freeze to death </code></pre> <h2>Why did it fail?</h2> <p>Well, of course it failed, because the API is supposed to return a <em>NEW</em> quote every time we hit it.</p> <p>This is where the <em>vcrpy</em> library comes in SUPER handy, as it can record and freeze the API response so we can make sure the last test passes every time.</p> <p>So, the first thing we’re going to do is to import the VCR library into our test file and configure the new test to record the API response into a file on the local filesystem. As you can see below, we’ve really only added two lines. The first is the “import vcr” line at the top that makes the vcrpy library available to the test script. The second is the decorator on top of the “test_GetNewQuote” function, which does two things:</p> <ol> <li>Tells the script where to copy the results of the API call the first time you run it</li> <li>Tells the script where to look for the results every subsequent time you run it.</li> </ol> <pre><code class="language-python"> import vcr from unittest import TestCase from pykanyerest.quotes import * class TestGetNewQuote(TestCase): """ Test Case for get_new_quote function from kanye.rest """ @vcr.use_cassette(cassette_library_dir='./test_pykanyerest/fixtures/cassettes') def test_GetNewQuote(self): """ """ quote = get_new_quote() self.assertEqual(type(quote), dict) self.assertEqual(len(quote), 1) keys = quote.keys() self.assertIn('quote', keys) self.assertEqual(type(quote['quote']), str) self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death") </code></pre> <p>Once you’ve run the test once, a new file will appear in the ./test_pykanyerest/fixtures/cassettes/test_GetNewQuote folder with the following contents:</p> <pre><code class="language-markdown"> interactions: - request: body: null headers: Accept: - '*/*' Accept-Encoding: - gzip, deflate Connection: - keep-alive User-Agent: - python-requests/2.22.0 method: GET uri: https://api.kanye.rest/ response: body: string: '{"quote":"I''m a creative genius"}' headers: Access-Control-Allow-Headers: - Content-Type Access-Control-Allow-Methods: - GET Access-Control-Allow-Origin: - '*' CF-RAY: - 55a5b75cac3aecee-YUL Connection: - keep-alive Content-Length: - '33' Content-Type: - application/json Date: - Fri, 24 Jan 2020 23:16:38 GMT Expect-CT: - max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Server: - cloudflare Set-Cookie: - __cfduid=d1c38c52e7d73039e047e768fdea93b5d1579907798; expires=Sun, 23-Feb-20 23:16:38 GMT; path=/; domain=.kanye.rest; HttpOnly; SameSite=Lax Vary: - Accept-Encoding status: code: 200 message: OK version: 1 </code></pre> <p>If you look closely, you can see a lot of information in there, including the contents of the response body string, which is now ‘{“quote”:”I’m a creative genius”}’.</p> <p><em>Note: For those paying attention, you will probably have guessed that the test we created above will still fail, as the Kanye quote we were looking for has changed again.</em></p> <p>Let’s change the test to look for the new quote that’s captured in the test_GetNewQuote file above.</p> <pre><code class="language-python"> import vcr from unittest import TestCase from pykanyerest.quotes import * class TestGetNewQuote(TestCase): """ Test Case for get_new_quote function from kanye.rest """ @vcr.use_cassette(cassette_library_dir='./test_pykanyerest/fixtures/cassettes') def test_GetNewQuote(self): """ """ quote = get_new_quote() self.assertEqual(type(quote), dict) self.assertEqual(len(quote), 1) keys = quote.keys() self.assertIn('quote', keys) self.assertEqual(type(quote['quote']), str) self.assertEqual(quote['quote'], "I'm a creative genius") </code></pre> <p>Now when we run the tests, they all pass.</p> <p>And when we run them again? … They still pass.</p> <p>And when we run them again? … <strong>They still pass.</strong> We can now run the same test a million times and get the same answer, which is pretty cool, right?</p> <p><strong>One More Thing</strong></p> <p>One of the other major benefits of using something like <em>vcrpy</em> is we can now continue to refactor our code even when we don’t have access to the resource we’re testing.</p> <p>Stuck in a plane with no internet? No problem, you’ve captured the <em>actual</em> response from the original server. Your code has no clue you’re not connected.</p> <p>No access to the corporate network? No problem, you’ve captured the <em>actual</em> response from the original server. Your code has no clue you’re not connected.</p> <p>TravisCI has no access to your internal resources? No problem, you’ve captured the actual response from the original server and posted to GITHUB. TravisCI has no clue you’re not connected.</p> <h2>VCRPY</h2> <p>I’ve only begun to scratch the surface of this library, but hopefully it will spark your curiosity to investigate it a bit more and see how you can write some new tests for that fancy REST API on your new infrastructure.</p> <p>Just remember, you don’t want to write your secret usernames and passwords to the public GITHUB. The nice thing is that the vcrpy also has the ability to hide credentials. I’ll leave it up to you to see if you can figure that part out.</p> <p><a href="https://kontrolissues.net/mentions/netmanchris/">@netmanchris</a></p>https://developer.hpe.com/understanding-concurrency-in-python-part-1-threading/https://developer.hpe.com/understanding-concurrency-in-python-part-1-threading/Mon, 10 Feb 2020 22:17:20 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture3-1581373479403.png" alt="picture3"></p> <p>Although most developers understand the basic concepts of concurrency and parallelism, the nuances can be pretty tricky to understand. At a high level, these are techniques/mechanisms employed to execute multiple processes or threads simultaneously while, at the same time, ensure the CPU is used to its maximum extent. To provide you with a more complete understanding of concurrency in Python, I’ve written a three-part tutorial. I will start off by covering the topic of threading, and then delve into multiprocessing, and asyncio.</p> <p><strong>Concurrency</strong> is when processes are executed on a single processor by context switching, and they appear to be running simultaneously. <strong>Parallelism</strong> is when processes are executed on multiple processors or cores and are actually running simultaneously.</p> <p>Python provides multiple libraries to achieve concurrency, namely threading, multiprocessing, and asyncio. These libraries will be better employed if we understand a few aspects about concurrency in Python.</p> <ul> <li> <p>CPython enforces GIL (Global Interpreter Lock), which mandates one thread execution at a time. The thread needs to acquire this exclusive lock every time before the execution of any bytecode.</p> </li> <li> <p>Concurrency is preferred when the process is either I/O bound or CPU bound. I/O bound processes are those that communicate with the devices that are slower than the processor.</p> </li> </ul> <p>For example, a process talking to a poor network connection, printer/scanner, etc. is an I/O bound process. CPU bound processes are those that do significant CPU intensive computations. Here, the resource that limits the speed of execution is the CPU, unlike those in I/O bound processes.</p> <ul> <li> <p>In I/O bound or CPU bound processes, the threads don’t need to struggle/race to acquire the GIL.</p> </li> <li> <p>The threading and asyncio libraries are best used when the process is I/O bound, and the multiprocessing library is good to use when the process is CPU bound.</p> </li> </ul> <p>Here are some examples of how to employ the threading library in detail:</p> <p>Remember, this library is best for dealing with I/O bound functions.</p> <p>Start by looking at an I/O bound function for fetching responses from several websites. (I’m going to refer to this example in Parts 2 and 3 of this series, so make sure you take notes!) If you execute this task in both a regular and multi-threaded fashion and capture the time taken to fetch responses from all the sites, you can see it’s faster with a multi-threaded execution and slower with a regular execution.</p> <p>Step 1: First, import the necessary libraries and modules.</p> <pre><code class="language-python"> import threading import time import requests </code></pre> <p>Step 2: Define a function get_response() that accepts site as an input and fetches the response data from that site using requests.get() method.</p> <pre><code class="language-python"> def get_response(site): return requests.get(site) </code></pre> <p>Step 3: Create a list of several websites. Append any site and as many sites as you want to the list.</p> <pre><code class="language-python"> sites = ["http://www.google.com", "http://www.linkedin.com", "http://www.quora.com", "http://www.facebook.com"] </code></pre> <p>Step 4: Iterate through this list of sites and invoke the function get_response() for each site. Capture and print the time taken for this complete iteration using a time.time() method.</p> <pre><code class="language-python"> start_time = time.time() for site in sites: get_response(site) print("Time taken for regular execution", time.time()-start_time) </code></pre> <p>Step 5: Now, define threads using the threading library with target to get_response() function and arguments set to sites in the list.</p> <pre><code class="language-python"> threads = [threading.Thread(target=get_response, args=(site,)) for site in sites] </code></pre> <p>Step 6: Iterate over these threads and start these threads using the thread.start() method. Use the thread.join() method to wait till the thread execution completes. Also, capture the time using the time.time() method to see the time taken to complete the execution.</p> <pre><code class="language-python"> start_time = time.time() for thread in threads: thread.start() for thread in threads: thread.join() print("Time taken for multi-threaded execution", time.time()-start_time) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1581373455996.png" alt="picture1"></p> <p>You can see that the multi-threaded execution of this I/O bound task is way faster than the regular execution. The efficiency of a multi-threaded execution is so significant in a scenario like this with only four sites to fetch responses from. Imagine how much more of an advantage we would see when the list of sites grows longer! Did you just think about working on a mind-blowing web scraping project?</p> <p>The consolidated code would look like what’s shown below:</p> <pre><code class="language-python">import threading import time import requests def get_response(site): return requests.get(site) sites = ["http://www.google.com", "http://www.linkedin.com", "http://www.quora.com", "http://www.facebook.com"] start_time = time.time() for site in sites: get_response(site) print("Time taken for regular execution", time.time()-start_time) threads = [threading.Thread(target=get_response, args=(site,)) for site in sites] start_time = time.time() for thread in threads: thread.start() for thread in threads: thread.join() print("Time taken for multi-threaded execution", time.time()-start_time) </code></pre> <p>Now, let’s consider a CPU bound function and observe how a threading library isn’t of much help in achieving any further efficiency.</p> <p>Step 1: Again, be sure to first import the necessary libraries and modules.</p> <pre><code class="language-python"> import threading import time import requests </code></pre> <p>Step 2: This time, define a CPU intensive function cpu_bound() that accepts a number, multiplies it by 10^6 and calculates the sum of all numbers in a range of 0 to that product.</p> <pre><code class="language-python">def cpu_bound(num): return sum([i for i in range(num*1000000)]) </code></pre> <p>Step 3: Create a list of random numbers.</p> <pre><code class="language-python"> numbers = [11, 23, 53, 34] </code></pre> <p>Step 4: Just like in the last example, iterate over these numbers and invoke the cpu intensive function cpu_bound(). Capture the time taken to complete the execution. Print out the time taken for regular execution.</p> <pre><code class="language-python"> start_time = time.time() for number in numbers: cpu_bound(number) print("Time taken for regular execution", time.time()-start_time) </code></pre> <p>Step 5: As shown previously, define the variable threads using threading.Thread() method with target function set to cpu_bound and arguments set to the numbers in the list.</p> <pre><code class="language-python"> threads = [threading.Thread(target=cpu_bound, args=(number,)) for number in numbers] </code></pre> <p>Step 6: Iterate over these threads and start the execution of these threads using the thread.start() method. Use the thread.join() method to wait till the thread execution completes. Also, capture the time using the time.time() method to see the time taken to complete the execution and print it out.</p> <pre><code class="language-python"> start_time = time.time() for thread in threads: thread.start() for thread in threads: thread.join() print("Time taken for multi-threaded execution", time.time() - start_time) </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture2-1581373442995.png" alt="picture2"></p> <p>You can see that employing the threading module did not help us much in achieving any further efficiency while executing a CPU bound function. This is probably because line-by-line execution is done faster than waiting for a thread to complete and for another thread to acquire the GIL and proceed.</p> <p>The consolidated code would look like this:</p> <pre><code class="language-python">import threading import time import requests def cpu_bound(num): return sum([i for i in range(num*1000000)]) numbers = [11, 23, 53, 34] start_time = time.time() for number in numbers: cpu_bound(number) print("Time taken for regular execution", time.time()-start_time) threads = [threading.Thread(target=cpu_bound, args=(number,)) for number in numbers] start_time = time.time() for thread in threads: thread.start() for thread in threads: thread.join() print("Time taken for multi-threaded execution", time.time()-start_time) </code></pre> <p>Through these two examples, it’s apparent that multi-threaded execution takes more than, or almost the same amount of time as, that of regular execution while handling CPU bound functions. What you need to understand is that, <em>no matter how many cores you have on your computer, the threading library of Python will not help you to completely exploit the abilities of multi-threading</em>. Because of this, any CPU intensive functions won’t benefit from multi-threaded execution.</p> <p>Are there any instances wherein we can take complete advantage of all the cores that our computer has and experience an efficient handling of especially CPU bound functions?</p> <p>Yes! Python supplies a Multiprocessing library that helps this exact situation. In my next post that covers more of the nuances of concurrency, Understanding Concurrency in Python Part 2 – Multiprocessing, I will explain more about this. Remember to check back on the HPE DEV blog site often to keep up with the many different tutorials we post. If you want to, you can also follow me on Twitter <a href="https://twitter.com/deyagondsamarth">@deyagondsamarth.</a> or connect with me on <a href="https://hpedev.slack.com/?redir=%2Fteam%2FUQM0ZTE1F">slack.</a></p>https://developer.hpe.com/meet-eric-soderberg-and-shimrit-yacobi-hpe-dev-team-members-working-on-g/https://developer.hpe.com/meet-eric-soderberg-and-shimrit-yacobi-hpe-dev-team-members-working-on-g/Fri, 07 Feb 2020 18:33:36 GMT<p>In this Meet the HPE DEV Team blog, I’d like to introduce Eric Soderberg <a href="https://twitter.com/ericsoderberg?lang=en">@ericsoderberg</a> and Shimrit (aka “Shimi”) Yacobi, members of the <a href="https://developer.hpe.com/community">HPE Dev Community</a> who are focused on <a href="https://developer.hpe.com/platform/grommet/home">Grommet.</a> Grommet is a React-based library of reusable UI components that help developers and designers create web applications. This open source UI development and design framework simplifies the way web applications are built by providing a package of commonly used interface elements from which developers and designers can choose to use.</p> <h2>Eric Soderberg – Co-creator of Grommet</h2> <p>Awaking each morning with the hope of finding unexpected beauty is the way Eric best describes his approach to life. He is constantly seeking out the fresh, different, and creative – what he describes as the many reflections of God. He lives simply and with purpose. The one car he owns is driven mostly by his wife, while Eric bikes back and forth to work, rain or shine. An elder at his local church, Eric finds a great deal of fulfillment in his volunteer work.</p> <p>As co-creator of Grommet, Eric brings this sense of beauty, simplicity, and service to his work. When customers expressed frustration at the profusion of disparate user interfaces across HPE products (a result of HPE having acquired many different companies), Eric worked closely with Chris Carlozzi <a href="https://twitter.com/chriscarlozzi?lang=en">@chriscarlozzi</a> to design a user experience framework that was accessible, responsive, and simple. Seeking a neutral framework that could be broadly adopted, they started with ReactJS and developed the Grommet libraries for theming, accessibility, and modularity.</p> <p>Eric encouraged developers in the different groups to use Grommet by building sample applications and showing them how easily it could be done. As a result, applications across HPE began to look more unified, and Grommet itself improved as the learnings from each interaction found its way back into the code.</p> <h2>Shimi Yacobi – Grommet Core Developer and Community Manager</h2> <p>Finding herself as the only female in the advanced computer classes in her high school, Shimi gravitated naturally towards engineering, and wound up in a career she truly loves. An outgoing and personable developer, Shimi enjoys yoga, hiking, scuba diving, and spending time with her family. She is also very involved with the community, including a local Women’s Network focused on empowering women in technology.</p> <p>Shimi first encountered Grommet while she was working as a developer in the <a href="https://www.hpe.com/us/en/integrated-systems/hyper-converged.html">HPE Hyperconverged</a> systems group. She is now a core developer for Grommet, as well as the Grommet Community Manager. In this latter role, she connects with developers and designers to understand their requirements and extends the Grommet framework to address their current and future needs. She also provides consulting to Grommet users on best practices and how to get things done using the framework. Shimi values the feedback provided by each contributor, as being so tuned into the community ensures that Grommet is always on the cutting edge.<br> To keep up to date with everything that’s going on with Grommet, make sure you connect with them on <a href="https://grommet.slack.com/">Slack.</a> You can follow the <a href="https://twitter.com/grommet_io">Grommet</a> Twitter handle or follow <a href="https://twitter.com/ericsoderberg">Eric</a> directly on Twitter. And don’t forget to check out the HPE DEV portal to learn more about the <a href="https://developer.hpe.com/platform/grommet/home">Grommet platform.</a></p>https://developer.hpe.com/new-hpe-oneview-ansible-module-v540-includes-support-for-hpe-oneview-50/https://developer.hpe.com/new-hpe-oneview-ansible-module-v540-includes-support-for-hpe-oneview-50/Fri, 07 Feb 2020 18:29:39 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/picture1-1581100387470.png" alt="picture1"></p> <p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> offers a software-defined, programmatic approach to managing infrastructure with efficient workflow automation, a modern RESTful API, and a comprehensive partner ecosystem. HPE’s software development kits (SDKs) and modules enable HPE OneView to integrate with partner products and take advantage of the benefits they offer.</p> <p>HPE is pleased to announce the availability of the HPE OneView Ansible Module v5.4.0. This module provides integration of HPE OneView with Ansible by Red Hat®, an industry-leading software deployment, provisioning, and configuration management tool. This new module supports HPE OneView 5.0 (REST API version 1200). It also provides backwards compatibility, extending support to API 800 and API 1000.</p> <p>The HPE OneView Ansible module enables the automated provisioning of bare-metal resources, including servers, storage, and networking as part of the application deployment process. Using Ansible with HPE OneView allows customers to create a flexible and adaptive infrastructure, essential to addressing the need for organizational agility and supporting initiatives that accelerate the delivery of customer and business value.</p> <p>For more information</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.4.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-ansible/blob/master/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/oneview-ansible">Code repository and examples</a></li> <li>Whitepaper: <a href="https://github.com/HewlettPackard/oneview-ansible-samples/blob/master/infrastructure-as-code/infrastructure-as-code.md">Infrastructure as code with HPE OneView and Ansible by Red Hat</a></li> </ul>https://developer.hpe.com/2020-February-03/https://developer.hpe.com/2020-February-03/Mon, 03 Feb 2020 06:00:00 GMThttps://developer.hpe.com/refactoring-in-python/https://developer.hpe.com/refactoring-in-python/Thu, 30 Jan 2020 22:57:21 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/1/01-refactoring-1024-1581354751665.jpg" alt="01 refactoring 1024"></p> <h2>What is refactoring, and why do we need it?</h2> <p>Too often, developers don’t take the time required to refactor and refine their code, resulting in the accumulation of technical debt that needs to be addressed somewhere down the road. Refactoring code is a technique used to improve the non-functional requirements of the code by modifying the external structure without altering the default behavior of the code. By non-functional requirements, it refers to attributes like modularity, readability, testability, maintainability and other enhancements, the absence of which can add technical debt to the code. Refactoring code is like making final touches to a painting.</p> <h2>Refactoring techniques for Python:</h2> <p>While the same refactoring techniques can be employed across different programming languages, this article focuses on refactoring techniques that are more relevant for the Python programming language.</p> <p><strong>Flat is better than nested</strong></p> <p>When given a <em>while</em> <em>loop</em> driven by a condition whose iterations check the truth value of another ‘if condition’ inside it, then the <em>loop</em> <em>condition</em> and nested <em>if</em> <em>condition</em> can be combined with an <em>AND</em> logical operator and used as the new <em>loop</em> <em>condition</em> to yield the same result. The code still works perfectly. But when refactored as described, it becomes easier to read and debug.</p> <p>Below is an example code snippet before refactoring:</p> <pre><code class="language-python">def _upheap(self, index): parent_index = (index-1)//2 while index > 0: # condition driving the while loop if self._heap[parent_index] > self._heap[index]: # if conditional self._swap(parent_index, index) index = parent_index parent_index = (index-1)//2 </code></pre> <p>The refactored code snippet is shown below:</p> <pre><code class="language-python">def _upheap(self, index): parent_index = (index-1)//2 while index > 0 and self._heap[parent_index] > self._heap[index]: self._swap(parent_index, index) index = parent_index parent_index = (index-1)//2 </code></pre> <p>Doesn’t the code look much more refined now?</p> <p><strong>Global variables are always a bad idea</strong></p> <p>Every function in code has access to global variables. If the global variables are modified, it’s hard to find out which function made the change. Debugging becomes tougher as the code grows. When it comes to readability, strange variables will start popping up from out of nowhere. This is because the global variables in use might be declared somewhere very far into the code. (Don’t you think that declaring the variables close to their usage in the code is a good refactoring idea? This would be especially helpful in Python, which is a dynamically-typed programming language.)</p> <p>To avoid changing variables that shouldn’t be changed, it is always a good idea not to implement global variables. Global constants, however, are pretty handy.</p> <p>Consider the below snippet that uses a global variable.</p> <pre><code class="language-python">x = 10 def increment_x_by_one(): global x x += 1 return x def increment_x_by_two(): global x x += 2 return x print(increment_x_by_one()) print(increment_x_by_two()) </code></pre> <p>The output will be as shown below:</p> <p><code>11</code> <code>13</code></p> <p>We wanted the second output to be 12, but earlier, the <code>function increment_x_by_one</code> has already been modified with the initial/default value that the variable x is supposed to have. This causes us to have an error.</p> <p>Now, let’s refactor the code, avoiding the use of a global variable.</p> <pre><code class="language-python">def increment_x_by_one(x): x += 1 return x def increment_x_by_two(x): x += 2 return x x = 10 print(increment_x_by_one(x)) print(increment_x_by_two(x)) </code></pre> <p>The output now will be accurate, as shown below:</p> <p><code>11</code> <code>12</code></p> <p>The result is absolutely as expected, and the function that modified variable x can be easily tracked.</p> <h2>Don’t use magic literals</h2> <p>Many times, string literals and numeric values are used directly, as opposed to being assigned to a variable, even though they imply something significant in the code. It is good to store them as variables with proper names to improve the code’s readability. Those string literals and numeric values might be referred to in multiple places. If they are to be modified, then all their references need to be traced, which is, again, an effort that is mostly prone to error.</p> <p>Consider the below snippet before refactoring:</p> <pre><code class="language-python">if age > 21: return True </code></pre> <p>Now, consider the refactored code:</p> <pre><code class="language-python">age_limit = 21 if age > age_limit: return True </code></pre> <p>Comprehending the code block is much easier now, isn’t it?</p> <h2>Remove the commented code</h2> <p>Often working/non-working code sections are commented out when prompted to try an alternative logic, and then they are left “as is”. This makes the readability of the code much more difficult. Also, other comments might get overlooked. So, the commented code should be erased. The best way to keep track of your commented code is via a version control mechanism, like git, using proper commit messages.</p> <h2>Address the redundancy</h2> <p>If two blocks of code are doing the same logic or same set of instructions, then address the redundancy by either extracting a function out of the common code or by placing the redundant instruction on an appropriate line before or after (but only in one place).</p> <p>Consider the below example before refactoring:</p> <pre><code class="language-python">x = 10 if x % 2 == 0: print(“The number”, x, “is even”) print(“The squared of the number is”, x**2) else: print(“The number”, x, “is odd”) print(“The squared of the number is”, x**2) </code></pre> <p>Now, let’s refactor it:</p> <pre><code class="language-python">x = 10 if x % 2 == 0: print(“The number”, x, “is even”) else: print(“The number”, x, “is odd”) print(“The squared of the number is”, x**2) </code></pre> <p>Refactoring not only makes it more readable, but also saves on Python code interpretation time.</p> <h2>Conclusion</h2> <p>Many refactoring tools are available as software packages or extensions for almost every programming language. Some IDE’s (Integrated Development Environments) like PyCharm have built-in refactoring tools. Practice refactoring your code before committing it to the codebase. Besides meeting the non-functional requirements, a neatly refactored code eases debugging, too.</p> <p>And guess what?! This blog, too, is refactored a couple of times to improve its readability. You see, fundamentals work everywhere!</p> <p>I hope you found my post on refactoring useful. For more coding tips, keep checking the <a href="/blog">HPE DEV blog.</a> And if you have any questions, feel free to reach out to me <a href="https://twitter.com/deyagondsamarth">@deyagondsamarth</a> or connect with me on <a href="https://hpedev.slack.com/?redir=%2Fteam%2FUQM0ZTE1F">Slack</a>. Happy coding!</p>https://developer.hpe.com/using-grommet-with-gatsby/https://developer.hpe.com/using-grommet-with-gatsby/Tue, 21 Jan 2020 17:34:14 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1579628420945.png" alt="picture1"></p> <p><a href="https://www.gatsbyjs.org/">Gatsby</a> is a popular, free, and open source framework based on <a href="https://reactjs.org/">ReactJS</a> that helps developers quickly build websites and apps. Because Gatsby is a PWA (Progressive Web App) generator, it provides code and data splitting out-of-the-box. Gatsby loads only the critical HTML, CSS (cascading style sheets), data, and JavaScript required so websites load quickly. Gatsby uses a GraphQL query interface to easily get data from just about any source, making clicking around the website feel very fast. You can augment these capabilities and make great looking applications and websites mobile-friendly, accessible, and responsive by using <a href="https://v2.grommet.io/">Grommet.</a> In this tutorial, I’ll show you how to get started using Grommet components and styles with Gatsby to make websites and applications more inviting.</p> <h2>Pre-requisites</h2> <p>This tutorial assumes you have <a href="https://yarnpkg.com/lang/en/">Yarn</a> as well as <a href="https://nodejs.org/en/download/">Node.js and npm</a> installed.</p> <h2>Step 1: Create a basic Gatsby app</h2> <p>First, use npm to install the Gatsby commands and create a basic Gatsby app with the <code>gatsby new &#x3C;name></code> command. You can use an existing Gatsby starter (there are some that include Grommet), but we'll use the default Gatsby minimal app for this example.</p> <pre><code class="language-bash">npm install -g gatsby-cli gatsby new gatsby-with-grommet cd gatsby-with-grommet </code></pre> <h2>Step 2: Add in grommet dependencies</h2> <p>Add Grommet to the dependencies. We'll use Yarn, since Gatsby uses it when doing <code>gatsby new</code>.</p> <pre><code class="language-bash">yarn add grommet grommet-icons styled-components </code></pre> <h2>Step 3: Start up a development server</h2> <p>Start up a Gatsby development server to make changes and see the effect of those changes in real time.</p> <pre><code class="language-bash">gatsby develop </code></pre> <p>Now, you can view the result in your local browser at <code>http://localhost:8000/.</code></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1579628526610.png" alt="picture2"></p> <p>We aren't using Grommet components yet, so let's add some in, along with the Grommet themes.</p> <h2>Step 4: Start using Grommet components</h2> <p>First, replace the contents of the <code>src/components/layout.css</code> file with this simple bit of CSS to reset some browser defaults we don't want. We'll rely on Grommet's internal styles rather than use a separate CSS file to style the user interface (UI).</p> <pre><code class="language-css">html { box-sizing: border-box; } *, *:before, *:after { box-sizing: inherit; } body { margin: 0; padding: 0; font-weight: normal; } img { max-width: 100%; height: auto; } </code></pre> <p>Next, make the main Layout component using the Grommet wrapper and theme, as well as the other equivalent Grommet components, instead of the literal html elements. Change <code>src/components/layout.js</code> to look like this:</p> <pre><code class="language-jsx">import React from "react" import PropTypes from "prop-types" import { useStaticQuery, graphql } from "gatsby" import Header from "./header" import "./layout.css" import { Grommet, Anchor, Box, Footer, Text } from "grommet" import { grommet } from "grommet/themes" const Layout = ({ children }) => { const data = useStaticQuery(graphql` query SiteTitleQuery { site { siteMetadata { title } } } `) return ( &#x3C;Grommet theme={grommet} full style={{ display: "flex", flexDirection: "column", }} > &#x3C;Header siteTitle={data.site.siteMetadata.title} /> &#x3C;Box as="main" pad="medium" flex overflow="auto"> {children} &#x3C;/Box> &#x3C;Footer background="light-4" justify="center" pad="small"> &#x3C;Text textAlign="center" size="small"> © {new Date().getFullYear()}, Built with {` `} &#x3C;Anchor href="https://www.gatsbyjs.org">Gatsby&#x3C;/Anchor> {` and `} &#x3C;Anchor href="https://v2.grommet.io">Grommet&#x3C;/Anchor> &#x3C;/Text> &#x3C;/Footer> &#x3C;/Grommet> ) } Layout.propTypes = { children: PropTypes.node.isRequired, } export default Layout </code></pre> <p>You’ll also want to change <code>src/components/header.js</code> to use the Grommet equivalents.</p> <pre><code class="language-jsx">import { Link } from "gatsby" import PropTypes from "prop-types" import React from "react" import { Header as GrommetHeader, Heading } from "grommet" const Header = ({ siteTitle }) => ( &#x3C;GrommetHeader background="brand" justify="center"> &#x3C;Heading> &#x3C;Link to="/" style={{ color: `white`, textDecoration: `none`, }} > {siteTitle} &#x3C;/Link> &#x3C;/Heading> &#x3C;/GrommetHeader> ) Header.propTypes = { siteTitle: PropTypes.string, } Header.defaultProps = { siteTitle: ``, } export default Header </code></pre> <p>Finally, change <code>src/pages/index.js</code> to use Grommet equivalents.</p> <pre><code class="language-jsx">import React from "react" import { Link } from "gatsby" import { Box, Heading, Paragraph } from "grommet" import Layout from "../components/layout" import Image from "../components/image" import SEO from "../components/seo" const IndexPage = () => ( &#x3C;Layout> &#x3C;SEO title="Home" /> &#x3C;Heading>Hi people&#x3C;/Heading> &#x3C;Paragraph>Welcome to your new Gatsby site.&#x3C;/Paragraph> &#x3C;Paragraph>Now go build something great.&#x3C;/Paragraph> &#x3C;Box width={{ max: "300px" }} pad="small"> &#x3C;Image /> &#x3C;/Box> &#x3C;Link to="/page-2/">Go to page 2&#x3C;/Link> &#x3C;/Layout> ) export default IndexPage </code></pre> <p>Make similar changes to <code>src/pages/page-2.js</code>.</p> <blockquote> <p>Note that Gatsby has a way of optimizing and lazy-loading images that’s implemented in the example <code>src/components/image.js</code>. You can use something like this rather than Grommet's <code>&#x3C;Image></code> component to help optimize a Gatsby site.</p> </blockquote> <p>You should now see the Grommet styling in the browser.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1579638304412.png" alt="picture3"></p> <h2>Step 5: Routing</h2> <p>Gatsby defines routes automatically for pages in the <code>src/pages</code> folder so many Gatsby applications don't have to set up routes or even interact with the router directly. Gatsby uses <code>@reach/router</code> instead of the react router. So, for efficient internal links, Gatsby applications use Gatsby's <code>&#x3C;Link></code> component. This component uses <code>navigateTo</code> from <code>@reach/router</code> to change routes. One way to get Grommet's styling for these internal links is to make a version of Grommet's <code>&#x3C;Anchor></code> that uses <code>navigateTo</code>.</p> <p>Create a file in <code>src/components</code> called <code>link.js</code> that contains the following:</p> <pre><code class="language-jsx">import React from "react" import PropTypes from "prop-types" import { Anchor } from "grommet" import { navigate } from "gatsby" const Link = ({ to, ...rest }) => ( &#x3C;Anchor href={to} onClick={ev => { navigate(to) ev.preventDefault() }} {...rest} /> ) Link.propTypes = { to: PropTypes.string, } export default Link </code></pre> <p>Now, change all the places that use Gatsby's <code>&#x3C;Link></code> to use this Link component instead. For example, change <code>src/pages/index.js</code> to import this new component and not the Gatsby Link component.</p> <pre><code class="language-diff">import React from "react" // import { Link } from "gatsby" import Link from "../components/link" import { Box, Heading, Paragraph } from "grommet" import Layout from "../components/layout" import Image from "../components/image" import SEO from "../components/seo" const IndexPage = () => ( &#x3C;Layout> &#x3C;SEO title="Home" /> &#x3C;Heading>Hi people&#x3C;/Heading> &#x3C;Paragraph>Welcome to your new Gatsby site.&#x3C;/Paragraph> &#x3C;Paragraph>Now go build something great.&#x3C;/Paragraph> &#x3C;Box width={{ max: "300px" }} pad="small"> &#x3C;Image /> &#x3C;/Box> &#x3C;Link to="/page-2/">Go to page 2&#x3C;/Link> &#x3C;/Layout> ) export default IndexPage </code></pre> <p>Because you started using this other Link, the <strong>Go to page 2</strong> link on the main page is using the Grommet styling.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture4-1579628555064.png" alt="picture4"></p> <p>Make the same change to <code>src/pages/page2.js</code> and <code>src/components/header.js</code> to ensure those use the Grommet link styling as well.</p> <h2>Summary</h2> <p>You can see the complete result at <a href="https://github.com/MikeKingdom/gatsby-with-grommet-starter">https://github.com/MikeKingdom/gatsby-with-grommet-starter</a>.</p> <p>Gatsby is a great framework for making high-performing websites and web applications. Try adding Grommet to make these websites even better looking, responsive, accessible and mobile-friendly. If you have any questions, please join me on the <a href="https://app.slack.com/client/T04LMHMUT/C04LMHN59">Grommet Slack channel.</a> You’ll find a lot of support there. And don’t forget to check out the <a href="https://developer.hpe.com/">HPE DEV site</a> to learn more about all our platforms.</p>https://developer.hpe.com/the-tools-we-use-to-shape-user-experiences/https://developer.hpe.com/the-tools-we-use-to-shape-user-experiences/Wed, 15 Jan 2020 21:15:44 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture4-1579123006064.png" alt="picture4"></p> <p>As a designer myself, I’ve found that folks who don’t get the chance to work closely with designers are often curious to know what we use to shape the designs and experiences we create. How do we get the results we want to achieve?</p> <p>Just as developers rely on a handful of tools to code their projects, designers use a handful of tools as well. And, in the same vein, different tools are used to solve different design problems. For instance, it’s hard to edit a vector image in a raster program and vice versa.</p> <p>This article will explore some of the common tools used by experience designers at the <a href="https://hpe.design/">HPE Experience Studio</a> throughout the design process to work with stakeholders (i.e. product managers, developers, and design engineers) to develop a solution. What each of these tools provides in terms of details and functionality for prototyping ranges widely across a spectrum, from low-fidelity (sketches and notes for high-level brainstorming and collaboration) to high-fidelity (final, polished pixel-perfect mock ups).</p> <h2>Wireframes</h2> <p>Wireframes are low-fidelity designs used to communicate an idea. Just as the title infers, they are simple lines, boxes, and text (or dummy text) used to pull together a concept or layout well enough to give a clear understanding of the direction being taken. Wireframes are used early in the development process to establish the basic structure of a page before more of an investment of time is made in the visual design and content. Designers use this fidelity of work so that stakeholders don’t get caught up in the fine details of a design, like choice of colors and copy specifics.</p> <p>Many of the tools used to create wireframes are also used for other types of deliverables. Here are a couple popular ones.</p> <ol> <li> <p><a href="https://balsamiq.com/">Balsamiq</a> – This tool is almost specifically tailored for wireframes and is very easy for beginners to learn.</p> </li> <li> <p><a href="https://www.axure.com/">Axure</a> – This is another tool that has a specific wireframe “mode”. It is a really powerful tool that can be used for many deliverables; however, it has a steep learning curve. Axure can also be used for high-fidelity mockups and interactive prototypes.</p> </li> </ol> <h2>High-Fidelity Mockups</h2> <p>High-fidelity mockups are screens intended to reflect a finished product (or very close to it). They are often used as the final stage in design when visual details are being worked out. High-fidelity mockups are static screens that can also be pulled together to create an interactive prototype (I’ll cover more on this later.).</p> <ol> <li> <p><a href="https://www.sketch.com/">Sketch</a> – This is a very powerful desktop based tool, and the one our team uses the most. You can use it to go from low fidelity to high fidelity all in one place. While I am a big fan of Sketch, it does have some limitations – particularly in the cloud storage and prototype arena. Fortunately, Sketch developers are currently addressing these limitations.</p> </li> <li> <p><a href="https://www.figma.com/">Figma</a> – This product is available both as a cloud and desktop platform. You can use either Figma’s web-based tool or download their desktop version. Very similar to Sketch, Figma is a newer tool that offers some interactive and code-based design capabilities Sketch does not. I personally love Figma, as their web-based app is responsive, easy to use, and convenient. It also makes it easier to collaborate with others, since the file can be accessed by more than one person in the web app.</p> </li> <li> <p><a href="https://www.adobe.com/products/xd.html">Adobe XD</a> – Any list of high-fidelity mockup tools would be amiss if it didn’t include an Adobe product. Adobe XD is a high-fidelity mockup and prototyping tool that has many of the same functions as the tools listed above. It was a bit late to the game (it’s only been officially launched and out of beta for a year), and it just doesn’t feel as easy to learn and use as Sketch and Figma.</p> </li> </ol> <h2>Interactive Prototypes</h2> <p>Interactive prototypes are designs that supply a level of interactivity with a user. They might be screens linked together by buttons or displayed interactivity through a form. The intent of their use is to make the design and user experience simulate what you would use in the real app or website. These prototypes can range in fidelity from low to high, as they are stitched together views of the deliverables listed above: wireframes and high-fidelity mockups. Below are some of the tools used to create interactive prototypes.</p> <ol> <li> <p><a href="https://www.invisionapp.com/">Invision</a> – With this product, you can upload any images and connect them to create a prototype. In our group, we rely on the stack Sketch + Craft + Invision to deliver interactive prototypes. I use Sketch, as covered above, plus a Sketch plug-in called Craft, which allows users to create links and connect artboards (screens) in Sketch, and then export to Invision. I could skip the Craft plug-in, but the plug-in makes life a lot easier by removing a step for exporting and uploading images.</p> </li> <li> <p><a href="https://www.framer.com/">FramerX</a>– This really cool product allows you to both design and prototype using the same tool. It also allows some code editing of React components. This code editing ability is very helpful because you can use UI libraries based in React (like Grommet) to play around with components and understand what is behind them a little better.</p> </li> <li> <p><a href="https://zeplin.io/">Zeplin</a> – Our team doesn’t use Zeplin, but since it is one of the most popular and widely used prototype tools according to this [survey] (<a href="https://uxtools.co/survey-2019/#prototyping">https://uxtools.co/survey-2019/#prototyping</a>), I’ve included it in this list.</p> </li> <li> <p><a href="https://reactjs.org/">React</a> and <a href="https://v2.grommet.io/">Grommet</a> – Since HPE has its own UI library (Grommet) it makes sense that the team I work in builds some of our own prototypes as coded prototypes with React.</p> </li> </ol> <p>The funny thing about most of the tools listed above is that many of them can equally be used to create wireframes, mockups, and prototypes. It all depends on what you are looking to do and how you want to get there. Another consideration you might want to take into account when choosing which tools to work with is how you will be hosting the files, whether it’s via a company server or in the cloud. Most of these tools are easily affordable and have a low learning curve to get started. That means that even if you are not a designer, you can explore these tools to communicate design decisions with your colleagues and stakeholders.</p> <p>Check out <a href="https://hpe.design/resources">HPE design tools</a> you can use with many of the applications listed above. Also, be sure to visit <a href="https://hpe.design/">hpe.design</a> to learn more about our team and what we do.</p>https://developer.hpe.com/the-hack-shack-returns-to-hpe-tss-paris-2020/https://developer.hpe.com/the-hack-shack-returns-to-hpe-tss-paris-2020/Wed, 15 Jan 2020 20:16:51 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1579119466925.png" alt="picture3"></p> <p>HPE Technology and Solutions Summit (TSS) is the largest and most comprehensive technical knowledge transfer event held by Hewlett Packard Enterprise (HPE). Delivered annually since its inception in 2006, HPE TSS has established a reputation as a renowned training initiative. With separate events held in Europe, Asia-Pacific, and the US, presales consultants and solutions architects from HPE and partner communities from all over the world benefit from direct access to HPE technology experts, engineers, and chief technologists. In Europe this year, the city of lights (Paris, France) will once again host <a href="https://h41382.www4.hpe.com/tss/">this event March 9-13, 2020.</a></p> <p>If you plan on attending, you will receive up-to-the-minute training on product developments, roadmap updates, and strategic insights. You will even have opportunities to practice and apply your knowledge under non-disclosure. As evidenced by the schedule, many exciting sessions are planned.</p> <p>The HPE DEV team will again be present at this event. We even have our own dedicated area, called the Hack Shack. It’s made up of a garden and a lab room, providing a cozy place where you can learn and have some fun. Many different activities will take place in the Hack Shack throughout the week. To discover more about the HPE DEV program, head straight to the Hack Shack desk. It opens at the same time as the Presales Suite, which is where attendees learn how HPE tools and resources make daily business easier and more effective.</p> <p>In the garden, we will host different code and game challenges where players compete with other players and coders for prizes. On Monday, you’ll find breakout sessions that describe the HPE DEV program and its importance to you. These sessions will be followed by several technical workshops where you can learn about the different API ecosystems available in the pan HPE portfolio. The workshops will cover <a href="https://developer.hpe.com/platform/hpe-oneview/home">HPE OneView,</a> <a href="https://developer.hpe.com/platform/ilo-restful-api/home">ILO Redfish,</a> <a href="https://developer.hpe.com/platform/hpe-simplivity/home">HPE SimpliVity,</a> <a href="https://developer.hpe.com/platform/hpe-nimble-storage/home">HPE Nimble Storage,</a> <a href="https://developer.hpe.com/platform/grommet/home">Grommet</a> and <a href="https://www.hpe.com/us/en/integrated-systems/composable-fabric.html#portfolio">HPE Composable Fabric Manager.</a> Our open labs on Wednesday and Friday will cover the <a href="https://developer.hpe.com/platform/bluedata/home">HPE Container Platform,</a> <a href="https://www.arubanetworks.com/">Aruba</a> networking products, and a really cool facial recognition app.</p> <p>You’ll find so much to do at the Hack Shack in a fun and inviting atmosphere. Presales attendees and partners will have the opportunity to engage with HPE experts during the different sessions. The complete agenda will be posted soon on the <a href="https://h41382.www4.hpe.com/tss/#agenda">HPE TSS website.</a> You’ll find all the details you need when you search for Hack Shack as the session type. The Hack Shack will be located on floor 7.3 next to the generic lab rooms.</p> <p>Make sure you stop by the Hack Shack to learn more about the HPE DEV program, as well as other key DevOps contributions and value offered through the HPE DEV community. Hope to see you at the event!</p>https://developer.hpe.com/teaching-students-to-code-reaps-rewards/https://developer.hpe.com/teaching-students-to-code-reaps-rewards/Wed, 15 Jan 2020 20:01:10 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1579118573174.png" alt="picture1"></p> <p>In a world where software drives everything, from work and education to transportation and entertainment, developing an understanding of code and how it works at an early age is important. In addition to cultivating opportunities for future careers, teaching students to code helps them develop crucial skills that support math and processing concepts and encourages critical thinking.</p> <h2>HPE DEV supports teaching students to code</h2> <p>Under its charter, Build – Communicate – Collaborate, the HPE DEV community supports initiatives dedicated to teaching programming to young students, such as the <a href="https://code.org/learn">Hour of Code</a>™ project and the <a href="https://www.codeclubworld.org/">Code Club.</a> Volunteers, like Didier Lalli and Frederic Passeron, see it as a way to encourage an interest in STEM (Science, Technology, Engineering and Mathematics), develop future talent, and give back to the community by exposing how application programming works. I recently had the opportunity to speak with Didier and Frederic to discuss the programs and the differences between them.</p> <h2>How these programs work</h2> <p>Didier, who participated in the <a href="https://www.webtimemedias.com/article/sophia-initiative-code-club-pour-initier-les-collegiens-au-codage?old_id=65306">Code Club in Sophia Antipolis,</a> set the stage. “While the Hour of Code and the Code Club are two different initiatives, both expose students to programming skills at an early age, doing so in a fun and engaging manner. Both programs have us work within classroom settings alongside the student’s regular math or technology teacher.”</p> <p>“We introduce software programming concepts through the use of a program called <a href="https://scratch.mit.edu/">Scratch,</a>” Didier continued. “This is a graphical programming language developed by the Massachusetts Institute of Technology (MIT) that allows students to create their own interactive stories, games, and animations. After we present the software program, we describe the necessary steps used to connect to the website where they will interact with it. The students then select an activity they wish to perform.”</p> <p>Frederic pointed out how the use of Scratch energized the sessions. “Scratch is so graphical and simple to use. It makes it easy for the students to understand programming concepts like 'loops', 'if/then' conditions, and iterative processes. Even students who weren’t initially excited about participating were quickly hooked. I’d say that, within ten minutes, all were thoroughly engaged.”</p> <p>Frederic showed me the program, demonstrating how students could choose from different characters and make the character move across the screen using simple commands. Seeing Scratch in action, I immediately understood how this felt more like a game rather than a lesson.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1579118765885.png" alt="How Scratch presents code to students"></p> <p>“Learning this way sure is a far cry from how I learned to code years ago, what with reading all the documentation and writing so many lines of code, only to find that a small typo stopped my code from actually working.”</p> <h2>Comparing the different programs</h2> <p>While Didier worked with the Code Club, Frederic volunteered with the Hour of Code project. The main difference between the two programs is really about reach. The Code Club boasts <a href="https://codeclub.org/en/about">over 13,000 clubs around the world in over 160 different countries</a> and runs all year long. The Hour of Code touches <a href="https://code.org/learn">millions of students in over 180 countries</a> and is held during Computer Science Week, usually in December or January. Whether a volunteer works with one group or the other generally depends on local availability. However, there are some other points of differentiation between the programs.</p> <p>For instance, the Code Club is sponsored by <a href="https://www.raspberrypi.org/about/">Raspberry Pi Foundation</a> and mixes volunteers from different companies (such as HPE, ARM, Cadence, Orange Labs, and SAP) to deliver one-hour sessions to the same class over ten weeks. For the first 5 or 6 sessions, the students engage in pre-defined coursework. The next 4 weeks are dedicated to a project where students are given free rein to let their imagination run wild and design their own game.</p> <p>The Hour of Code, on the other hand, is backed by <a href="https://csfirst.withgoogle.com/c/cs-first/en/code-your-hero/overview.html?utm_source=google&#x26;utm_medium=cpc&#x26;utm_campaign=20191105-Firewood_HOC19--hsms-ins-&#x26;src=cpc-google-20191105-Firewood_HOC19--hsms-ins-&#x26;gclid=CjwKCAiAx_DwBRAfEiwA3vwZYpw3qtT6quajGSUh5HM4eHi-UbNxq1jqYjfwnhhOWTCX7G2ibRubfRoCFCcQAvD_BwE">Google</a> and tends to be taught by volunteers from a single company, such HPE. Its format is based on 1.5 hour sessions for groups of 30 students. A single volunteer cycle consists of four sessions, allowing the volunteers to teach a total of 120 students.</p> <p>“Another thing that sets the Hour of Code apart,” explains Frederic, “is that it actively promotes diversity. This program works at increasing gender, racial, and socioeconomic diversity in tech education and, by extension, in the tech workforce. I see that sometimes girls are shy about their math and technical skills. Hour of Code points out that female developers are rare, yet they are in high demand by companies. It encourages girls and others who may not see themselves as fitting into ‘geek’ roles to seek out these types of careers.”</p> <h2>Join HPE DEV in giving of your time and talent</h2> <p>Teaching kids to code doesn’t just benefit students. Volunteers like Didier and Frederic find it rewarding as well. “Understanding how a PC, tablet, or phone operates is a good thing for all kids, even those that don’t end up working in the tech field. Removing the veil of magic behind how things work helps them understand how their involvement can have an effect,” Didier explained. “Hopefully, they will apply this newfound self-awareness to the world we all live in and strive to make it better.”</p> <p>“For volunteers,” he continued, “it’s very interesting to see how kids react to problems; how some are quick to get it and some more interested in designing icons than spending time on the logic of their program. While each student is different, it’s obvious that a number of them already show a real attraction to software development.”</p> <p>Frederic expressed how participating in activities like this helped him realize another ambition. “When I was in college, I studied to become an English teacher. Life intervened, and I found myself working for HPE instead. Volunteering for Hours of Code helped to finally close that loop, offering me the opportunity to teach.” As Frederic continued, he pointed out other things that attracted him to the program. “I think that the Hour of Code program is particularly important in that it promotes diversity. It does so by telling young girls that they can play an important part in the future of code, since female coders are rare and therefore, very valuable.”</p> <p>For those interested in volunteering for <a href="https://www.codeclubworld.org/about/countries/">Code Club</a> teaching opportunities, connect with your local Code Club. Volunteers for <a href="https://hourofcode.com/us/how-to">Hour of Code</a> should connect with local schools to learn how to participate. As an additional benefit for HPE employees, the HPE Gives program provides credit for HPE volunteers for programs like these.</p> <p>Giving has long been a key component of the Hewlett Packard Enterprise company culture, tracing back to the values of the company founders, Bill Hewlett and David Packard. As David Packard once said, “The betterment of society is not a job to be left to a few. It's a responsibility to be shared by all.” That’s why HPE DEV supports initiatives like these. Check out our <a href="https://slack.hpedev.io/">HPE DEV Slack</a> #hpedev-volunteers channel to discuss how we all can make a difference by sharing our skills and knowledge with others. Connect on Twitter with <a href="https://twitter.com/DidierLalli">Didier</a> and <a href="https://twitter.com/FredPasseron">Frederic</a> to follow their adventures with HPE DEV.</p>https://developer.hpe.com/hpe-oneview-python-sdk-now-available-with-support-for-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-python-sdk-now-available-with-support-for-hpe-oneview-50/Wed, 08 Jan 2020 18:31:57 GMT<h2>Use HPE OneView software-defined intelligence to save time and effort</h2> <p>The HPE OneView Python SDK v5.0 is now available, providing support for HPE OneView 5.0 (REST API version 1200). The HPE OneView Python SDK allows developers who use the Python language to programmatically control HPE OneView managed resources. It also provides for the integration of popular automation tools based on Python, such as Ansible, with HPE OneView. These capabilities enable complete datacenter automation using infrastructure-as-code for physical compute, storage, and fabric resources.</p> <p>In addition, the HPE OneView Python SDK code was refactored. SDK code now resides in two different repositories; <a href="https://github.com/HewlettPackard/oneview-python">HPE OneView-Python</a> and <a href="https://github.com/HewlettPackard/python-hpOneView">python-hpOneView.</a> Going forward, the <a href="https://github.com/HewlettPackard/oneview-python">HPE OneView-Python</a> repository will be updated with the latest SDKs. The <a href="https://github.com/HewlettPackard/python-hpOneView">python-hpOneView</a> repository will support legacy SDKs for existing customers who do not wish to change their client code.</p> <p>For more information about the HPE OneView Python SDK v5.0 release, please go to: <a href="https://github.com/HewlettPackard/oneview-python/releases/tag/v5.0.0">https://github.com/HewlettPackard/oneview-python/releases/tag/v5.0.0</a></p>https://developer.hpe.com/2020-January-06/https://developer.hpe.com/2020-January-06/Mon, 06 Jan 2020 06:00:00 GMThttps://developer.hpe.com/hpe-oneview-golang-sdk-is-now-available-with-support-for-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-golang-sdk-is-now-available-with-support-for-hpe-oneview-50/Mon, 16 Dec 2019 17:01:31 GMT<p>The HPE OneView Golang SDK v1.2.0 now supports HPE OneView 5.0 (REST API version 1200). The HPE OneView Golang SDK extends the HPE OneView API language support for the <a href="https://golang.org/">go language,</a> which is popular for cloud-native applications. With language support, you can integrate popular tools based on Golang, such as Terraform with HPE OneView. <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> takes a software-defined, programmatic approach to managing infrastructure with efficient workflow automation, a modern RESTful API, and a comprehensive partner ecosystem.</p> <p>The HPE OneView Golang SDK allows developers who use the go language to programmatically control HPE OneView managed resources using infrastructure-as-code for physical compute, storage, and fabric resources. Infrastructure-as-code enables complete datacenter automation, consistent reproducibility, versioning, and roll back.</p> <p>FOR MORE INFORMATION:</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.2.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-golang/blob/master/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/oneview-golang">Code repository and examples</a></li> </ul>https://developer.hpe.com/hpe-oneview-terraform-provider-now-supports-hpe-oneview-50/https://developer.hpe.com/hpe-oneview-terraform-provider-now-supports-hpe-oneview-50/Mon, 16 Dec 2019 16:58:24 GMT<p><a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> Terraform provider v1.2.0 now supports HPE OneView 5.0 (REST API version 1200). <a href="https://www.terraform.io/">Terraform</a>, by <a href="https://www.hashicorp.com/">HashiCorp,</a> is a popular tool for infrastructure automation that facilitates a multicloud or hybrid cloud approach to infrastructure management. When integrated with HPE OneView, Terraform is a powerful orchestration tool that can be used to create, manage, and update infrastructure resources. These resources may be physical machines, VMs, network switches, containers, or others.</p> <p>HPE OneView Terraform provider automates the provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView. This enables administrators to create a resource topology similar to that of a public cloud on their own physical infrastructure. With this type of resource topology, administrators can easily migrate applications and workloads to an on-premises private cloud environment to realize their hybrid cloud strategy.</p> <p>FOR MORE INFORMATION:</p> <ul> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.2.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/blob/master/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview">Code repository and examples</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/blob/master/README.md">Details about the Terraform provider</a></li> </ul>https://developer.hpe.com/new-hpe-dev-leadership-team-discusses-program-focus-and-expansion/https://developer.hpe.com/new-hpe-dev-leadership-team-discusses-program-focus-and-expansion/Mon, 09 Dec 2019 16:10:49 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1575907933380.png" alt="picture1"></p> <p>As Hewlett Packard Enterprise (HPE) pivots to an as-a-Service and software-enabled infrastructure company, I was eager to find out more about how HPE DEV can assist developers and designers in modernizing their legacy applications, as well help them create new, cloud-native apps. And with the recent <a href="https://www.hpe.com/us/en/newsroom/press-release/2019/11/Hewlett-Packard-Enterprise-introduces-Kubernetes-based-platform-for-bare-metal-and-edge-to-cloud-deployments.html">HPE Container Platform announcement,</a> I wanted to know more about HPE DEV’s plans to attract developers, designers, data scientists, and DevOps teams to join the community – especially given the teams’ expertise in API (application programming interface) development.</p> <p>To discover answers, I reached out to the new HPE DEV community leads: Distinguished Technologist, Didier Lalli, and hybrid IT technology consultant, Denis Choukroun.</p> <p><strong>Dale:</strong> Didier and Denis, thank you for taking the time to talk about HPE DEV and how it can assist the community. I wonder if you could provide a high-level summary of what the HPE Developer Community Program is and what it offers to its members.</p> <p><strong>Didier:</strong> The HPE Developer Community Program was launched during KubeCon in Austin, Texas December 2017. The program is built around three main pillars: Build, Communicate, and Collaborate with the goal of helping developers and designers accelerate their innovation. We do this by assisting them with our APIs to integrate with products such as <a href="https://developer.hpe.com/platform/hpe-oneview/home">HPE OneView,</a> <a href="https://developer.hpe.com/platform/hpe-simplivity/home">HPE SimpliVity</a> and <a href="https://developer.hpe.com/platform/ilo-restful-api/home">iLO.</a> But this isn’t all we do. For example, we share information about open source projects we support or initiate, such as <a href="https://developer.hpe.com/platform/grommet/home">Grommet.</a> We reach out through several communication channels, so members of the community can select which channel they prefer to engage on. These include our <a href="https://developer.hpe.com/">web site</a>, a monthly newsletter, <a href="https://slack.hpedev.io/">Slack</a>, <a href="https://twitter.com/HPE_Developer?lang=en">Twitter,</a> and a Yammer group for folks within HPE.</p> <p><strong>Denis:</strong> Didier described what HPE DEV offers to the community very well. I would just like to add that the HPE DEV portal offers developers, designers, DevOps engineers, and data scientists a voice of their own. Here they can share and learn via blogs about HPE products and solutions, along with API integration and automation capabilities.</p> <p><strong>Dale:</strong> As HPE evolves into a company focused on software-defined infrastructures and delivering everything-as-a-Service, where does HPE DEV fit in? I understand the group assists developers in providing the best user experience in their applications through research and open source projects, like Grommet. I also know others in the group develop tutorials using APIs to develop cool applications. How does HPE DEV fit into the overall company strategy and what services does it provide?</p> <p><strong>Didier:</strong> The group we belong to, the HPE Experience Studio, has two components: a User Experience team (UX) and a Developer Experience team (DEVX). The HPE Developer Community Program is managed by the DEVX team; but we leverage research, learnings, and development done by the UX team. We encourage contributions from the UX team, as they contribute to the design and development of the as-a-Service model, which is central to HPE’s new delivery strategy. Our DEVX team also adds to these projects. We look at improving the developer’s experience by ensuring common ways to access services. As an example, there is research and prototyping going on right now for a unique command line (CLI) experience that spans across multiple HPE services.</p> <p><strong>Denis:</strong> Yes, this is exactly the case. HPE is pivoting to an as-a-Service edge to cloud company helping ITOps and developers consume infrastructure in different ways. For example, the recent HPE announcement about the HPE Container Platform gives the HPE Developer Community Program more opportunities to help developers release new code quicker. It also enables us to assist data scientists with the deployment of AI/ML and Big Data tools. All this can now be done faster and anywhere – on premises and in the cloud.</p> <p><strong>Dale:</strong> How do you, personally, fit into HPE DEV? What has been your role in the past and what new responsibilities have you recently taken on?</p> <p><strong>Didier:</strong> Over 30 years, I played many roles. I started as a Software Engineer at Digital Equipment Corporation (DEC), writing mostly in C, C++, and ADA. Then, I moved to pre-sales, mostly focused on PC networking. At the time, connecting a PC to the rest of the network was a real challenge. I spent almost 15 years in EMEA pre-sales for HPE ProLiant systems, but I was always considered the software guy in a hardware team. I was lucky enough to witness the birth of Docker containers and had the opportunity to travel throughout Europe to teach our folks how to master this new technology. More recently, my focus returned to development. I participated in the creation of the HPE OneView Ecosystem, working with partners and ISVs (i.e. Turbonomic, Mesosphere, SaltStack) to help them better use the HPE OneView API and develop joint solutions for our customers. While I was doing this, I discovered the real power of REST API. Since then, I’ve worked on several of our product APIs, providing some sort of SDK (software development kit) in PowerShell, Python and, more recently in Go. When the HPE Developer Community Program started, I saw it as the perfect fit for me, and I contributed a lot, especially through blog posts and tutorials. Today I’m proud to be the technical lead of the HPE Developer Community Program.</p> <p><strong>Denis:</strong> Like Didier, I have more than 30 years of IT experience in various technology roles, including work as an operations and support networking engineer, a technical consultant, and a solution architect. I started as Local Area Network engineer at DEC. I then got the chance to be part of the first IT outsourcing (Managed Service) team. We delivered Wide Area Network support to customers who outsourced their IT infrastructure to the company. When Hewlett Packard acquired Compaq, I worked as a technical consultant and solution architect focused on converged infrastructure and hybrid IT solutions for both internal and customer-facing engagements. During this period, Didier and I co-authored the “Programming CloudSystem Matrix for Dummies” book. Just recently, I took on the role of program manager of HPE DEV. This is an area where I feel I can add a lot of value due to the project management skills I’ve learned through other roles.</p> <p><strong>Dale:</strong> Do you have any specific goals associated with your new roles?</p> <p><strong>Didier:</strong> As the technical lead of the program, I have a number of specific goals for 2020. A major goal is to grow the community both in terms of engagement as well as representation. This not only means increasing visits to our web site, but also expanding participation via Slack conversations and the number of followers on our Twitter channel. It also means inviting more content contributors from HPE by reaching out to HPE teams who aren’t currently represented. By doing this, all community members will benefit, gaining access to a wider knowledge base and driving consistency, interoperability and ease of use throughout products. Ultimately we’d like to change how developers view HPE. Today, a lot of developers we meet at events wonder what we have to offer to them so we have to spend a lot of time explaining it. The goal of HPE DEV is to change this perception so developers start to recognize the value of developing on HPE platforms.</p> <p><strong>Denis:</strong> I have specific goals as well. As an ambassador of the HPE Developer Community Program, my goal is to help developers and front-end application designers accelerate innovation and development time. So I work hard to spread the message about the value HPE DEV brings, evangelizing HPE DEV initiatives and offers to developers at HPE events and open source conferences. In my role as program manager, my primary goal is to coordinate and ensure clarity among all the multiple projects related to HPE Developer Community Program.</p> <p><strong>Dale:</strong> Denis, I understand you recently attended the Linux Foundation’s premier event, KubeCon + CloudNativeCon in San Diego and heard about the HPE Container Platform announcement first-hand. It sounds pretty exciting. Can you give some details about the announcement, what you learned there, and your thoughts on how this might impact the work HPE DEV takes on?</p> <p><strong>Denis:</strong> It was great to be able to attend this conference in San Diego and hear the HPE announcement around containers and Kubernetes at a time when the containerization of applications is on the rise. The HPE Container Platform is of paramount importance to HPE. This announcement was the highlight of the event, focusing on the convergence of HPE innovations. It combines the BlueData container-based control plane with the <a href="/blog/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern">KubeDirector</a> open source project, together with the MapR data fabric for container persistent data storage, and integrates Kubernetes, the de facto open source standard for container orchestration. This emergent technology will enable coders to develop applications faster and deploy them anywhere. The HPE Container Platform will undoubtedly impact HPE DEV’s work, as this container platform gives us opportunities to expand the reach of the community to data scientists, DevOps, and ITOps engineers.</p> <p><strong>Dale:</strong> Modern apps, written using containers, certainly have an advantage in cloud environments. They’re more portable and scale much more easily. What do you think this announcement means for existing apps? How will this benefit enterprises overall?</p> <p><strong>Didier:</strong> This announcement is a real game changer. This gives us a great story to deliver to developers. The HPE Container Platform lets you not only run modern cloud-native applications, but also legacy applications that you have always wanted to containerize, but couldn't. The HPE Container Platform allows more traditional apps, the ones with mostly persistent data, to operate in a modern container environment, with no need of rewriting them.</p> <p><strong>Dale:</strong> Given all this, what platforms do you believe will be the most important to HPE and our customers as we move forward?</p> <p><strong>Didier:</strong> I think we will continue to provide a lot of assistance in the use of our existing APIs, but we will also explore several new areas. First, as already discussed, we will provide more content regarding design and research, thanks to the other half of the team, the UX group. Then, we will look to add more content from other products within the HPE portfolio that may not yet be covered within the HPE Developer Community. I’m thinking possibly about Cloud Volumes, Composable Fabric Manager, or even Aruba. Finally, we will provide more focus on our team around the HPE Container Platform, so that we can help developers and their ITOps teams easily adopt this promising platform.</p> <p><strong>Denis:</strong> The HPE Container Platform is certainly an important piece of HPE’s hybrid stack. It will drive our focus toward containers. This solution addresses the evolving needs of application developers and the capabilities enterprises demand. It should help cut down complexity and time for developers, data scientists, DevOps, and ITOps teams to migrate their applications (both cloud-native and monolithic traditional apps) to containers and deploy them anywhere.</p> <p><strong>Dale:</strong> This deeper focus on assisting developers with platforms to modernize their applications all sounds very exciting. How can folks connect and engage as a part of the HPE Developer Community?</p> <p><strong>Didier:</strong> The best way is to connect with us is to go to <a href="https://developer.hpe.com/">hpedev.io,</a> our web portal. From there you can find all the different links that will allow you to subscribe to the monthly newsletter, join us on <a href="https://slack.hpedev.io/">Slack</a> or follow us on <a href="https://twitter.com/HPE_Developer?lang=en">Twitter.</a> Give it a try!</p>https://developer.hpe.com/think-container-first/https://developer.hpe.com/think-container-first/Thu, 05 Dec 2019 17:13:30 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>Attendees at KubeCon + CloudNativeCon 2019 were on hand to see the new HPE Container Platform that Hewlett Packard Enterprise (HPE) announced at the event. The HPE Container Platform is the industry’s first enterprise-grade Kubernetes-based container platform designed for both cloud-native applications and monolithic applications with persistent storage. With the HPE Container Platform, enterprise customers can accelerate application development for new and existing apps – running on bare-metal or virtualized infrastructure, on any public cloud, and at the edge. You may be asking yourself, what does this have to do with HPE, a leading infrastructure company?</p> <p><strong>Everything.</strong></p> <p>As a developer, you know first-hand how IT depends on software to optimize resources and manage what otherwise would be unmanageable. Almost every business today operates as a software business, as vendors write code to connect their systems to the rest of the universe – whether it’s to ensure operation in the cloud, provide a user interface on a smart phone, collect data from edge devices, or simply check in on stock availability in the warehouse. Code, and its ability to run anywhere and everywhere, is key to today’s businesses.</p> <p>But it’s not always easy to get your code to run everywhere. Newly written apps? Yeah, that’s really no problem. Containers and Kubernetes orchestration offer DevOps agility and quick creation of cloud-native applications. But other legacy apps, the ones that have been around for a while and businesses rely on, just aren’t easy to move. IT operations (ITops) managers have been scratching their heads, trying to figure out how they can make the most of cloud-enabled economics and still work within the confines of what they’re stuck with. While many good reasons justify hybrid cloud implementations, such as data security, sometimes ITops groups feel as though they just have to accept a hybrid cloud environment, because it would be too heavy a lift and costly to cloud-enable these types of apps.</p> <p>HPE understands this. After all, as a worldwide enterprise, it has had to deal with these same issues. Phil Davis, HPE President of Hybrid IT, stated, “The cloud is not a destination. It’s an experience and operating model.” In regards to this recent announcement, Phil Davis <a href="https://www.hpe.com/us/en/newsroom/blog-post/2019/11/With-HPE-Container-Platform-enterprise-wide-containerization-is-a-reality.html">reiterated this</a> fact and added “… the cloud experience can offer considerable benefits to organizations, such as speed, ease of provisioning, scalability, and pay-per-use. But the vast majority of existing non-cloud-native apps have been left behind. They were written over the past decades and often are the core applications and data which run the business.”</p> <p>You and your fellow ITops teams, developers, and designers most likely appreciate how containerization for both new and existing apps is going to help your organization move forward. But others in your business may not. They may not understand why they should look toward HPE for the industry’s first Kubernetes platform designed to run legacy and cloud native apps that enables a true hybrid cloud environment across any location – on premises, in public clouds, and at the edge.</p> <p>You can help them. You can help business managers envision their company as one built on the accumulation of data they can actually put to use to provide a better service to their customers and as one that runs as economically as possible using public cloud services in ways they couldn’t previously imagine.</p> <p>Start by showing <a href="https://www.youtube.com/watch?v=1G0D7ZY0dvk&#x26;feature=youtu.be">this short video</a> to those around you who don't understand how containers can improve the business. It’s concise and gets right to the point. And it will hopefully open up some conversations you may have been meaning to have for a while.</p>https://developer.hpe.com/hpe-dev-and-the-new-hpe-container-platform-at-kubecon-2019/https://developer.hpe.com/hpe-dev-and-the-new-hpe-container-platform-at-kubecon-2019/Wed, 04 Dec 2019 20:27:38 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1575491346999.png" alt="picture1"></p> <p>HPE Developer Community teammate, Pramod Sareddy, and I just returned from KubeCon + CloudNativeCon North America 2019. During my long journey back home to France, I decided to write about what I learned at the event while the breath-taking view of the San Diego bay was still fresh in my mind.</p> <p>It had been a busy week for the HPE Developer Community team as we greeted dozens of event attendees and updated them on what’s new from Hewlett Packard Enterprise (HPE). The event offered a great opportunity to exchange thoughts and information with developers, front-end application designers, enterprise solution architects, and chief technology officers who were interested in containers and hybrid cloud solutions. The HPE Developer Community team also got to meet the HPE designers and architects of the newly announced HPE Container Platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1575491389586.png" alt="picture2"></p> <p>At the conference, HPE presented and demoed the newly announced HPE Container Platform, as well as a <a href="https://community.hpe.com/t5/HPE-Storage-Tech-Insiders/HPE-storage-supporting-the-HPE-Container-Platform-at-KubeCon/ba-p/7070094#.Xd8FCuhKh9A">true hybrid cloud CI/CD pipeline</a> enabled by the HPE Container Platform, HPE Nimble storage, and HPE Cloud Volumes. Multiple in-booth theater sessions focused on these topics. Attendees crowded around the HPE booth to interact with HPE experts and get a chance to look at these new innovations.</p> <p>Also in the booth, Pramod and I presented the benefits of joining the HPE Developer Community. Many attendees were interested in how the community can help developers, designers, and DevOps and ITOps engineers accelerate innovation and speed up their development time. They were also eager to sign up for our <a href="https://developer.hpe.com/newsletter-signup">HPE Developer community newsletter,</a> which earned them a chance to win one of our daily prizes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1575491496447.png" alt="picture3"></p> <p>We also showed off a pretty cool face recognition application Pramod wrote in <a href="https://developer.hpe.com/platform/grommet/home">Grommet,</a> which runs on containers in a Kubernetes cluster, hosted on an HPE EdgeLine system. Another attraction in our booth was our HPE DEV-developed <a href="https://github.com/HewlettPackard/hpe-hack-shack-attack">Hack Shack Attack</a> gaming competition where players try to tame the IT monster. The game proved very popular among the developer community, as it earned participants extra chances to win a prize. We selected winners through a raffle at the end of each day.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture4-1575491590488.png" alt="picture4"></p> <p>The highlight of the event for us was witnessing the convergence of HPE innovations, resulting in the newly announced <a href="https://www.hpe.com/us/en/solutions/container-platform.html">HPE Container Platform.</a> The technology combines the BlueData container-based control plane with the <a href="/blog/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern">KubeDirector</a> open source project, along with the MapR data fabric for container persistent data storage and integrates Kubernetes, the de facto open source standard for container orchestration. This announcement piqued everyone’s attention, as evidenced by a continuous stream of event attendees stopping by to learn more about the <a href="https://www.hpe.com/us/en/newsroom/press-release/2019/11/Hewlett-Packard-Enterprise-introduces-Kubernetes-based-platform-for-bare-metal-and-edge-to-cloud-deployments.html">announcement.</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture5-1575491696591.png" alt="picture5"></p> <p>In short, the HPE Container Platform (a hardware agnostic, enterprise-grade container management software product) extends the benefits of running cloud-native enterprise applications with CI/CD micro services architecture on containers to non-cloud-native applications with persistent storage. Distributed, stateful applications, such as AI, ML, and big data applications, as well as enterprise traditional legacy workloads (which are generally not cloud-native and require persistent storage), can now run at scale on containers without the need to re-architect them. The HPE Container Platform delivers a seamless, cloud-like experience whether on-premises, at the edge, in multiple public clouds, or in a hybrid model. This makes the HPE Container Platform ideal for helping application developers, DevOps/ITops/CloudOps engineers, and data scientists accelerate their application development and deployment on containers, on-demand through a self-service portal and a RESTful API that surfaces programmable access.</p> <p>The HPE Container Platform is now available for preview on any customers’ infrastructure of choice (bare metal and VM) and any of the major public clouds. To learn more about this unique HPE container software platform visit the <a href="https://developer.hpe.com/platform/bluedata/home">HPE DEV portal</a> and check out our blog article <a href="/blog/running-non-cloud-native-apps-on-kubernetes-with-kubedirector">here!</a></p> <p>For more opportunities to contribute your insights, knowledge and experience to building the best possible developer experience around containers and open source Kubernetes, as well as learn from your peers, join the conversation on our <a href="https://slack.hpedev.io/">HPE DEV Slack Channel</a> and <a href="https://developer.hpe.com/signup">join the community.</a> Stay up to date with the latest news from HPE DEV by <a href="https://developer.hpe.com/newsletter-signup">signing up for our monthly newsletter</a> and we will send you more awesome developer-focused posts about HPE projects and platforms every month. You can also follow our community on <a href="https://twitter.com/HPE_Developer">Twitter.</a></p> <p>We look forward to seeing you again at <a href="https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2020/">KubeCon + CloudNativeCon EMEA 2020,</a> in Amsterdam, from March 30 to April 2, 2020.</p>https://developer.hpe.com/understanding-api-basics-and-the-value-they-provide/https://developer.hpe.com/understanding-api-basics-and-the-value-they-provide/Tue, 03 Dec 2019 16:58:32 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1575393131999.png" alt="picture1"></p> <p>Over the course of my career, I’ve come across many technical colleagues who were experts on a given product or technology, but always from a user’s point of view. They knew how to design a solution, sell it to the customer, install it, and customize it to meet the customer’s requirements. And, sometimes, they could even debug rather difficult problems. Yet, when developers asked these technology experts about possible integration of their product with third-party software, or when they were pressed to provide scripts and automate processes, the technology experts were at a loss. They simply did not understand the product from a developer’s perspective.</p> <p>And for good reason. There’s a certain amount of magic that happens within an application done by the code itself. Many applications you work with today, on PCs, servers, or off in the cloud, include a hidden jewel called the Application Programming Interface (API). In a nutshell, an API makes it possible to interact with an application programmatically. This means that developers can integrate their application with your applications in a standardized way. This becomes very useful because it opens up a whole lot of new use cases and business opportunities. But in order for you to take advantage of these opportunities, you need to understand the product’s API well enough so that you can discuss its capabilities with developers.</p> <h2>REST and HTTP</h2> <p>Exploring how you interact programmatically with a product is no easy task. Developers are more familiar with this exercise, but as a non-developer, you might wonder how to get started. The good news is that a de facto standard for building APIs has emerged in the last few years. It’s called REST (REpresentation State Transfer). Let me first explain what REST is, and then I’ll define a few important concepts used by REST APIs, such as HTTP and JSON.</p> <p>Most of you are familiar with the terms HTTP (HyperText Transfer Protocol) and URL (Uniform Resource Locator) because you use a web browser every day. Yet you may not know how they relate to REST and APIs. REST is an architecture that uses a set of principles to describe how networked resources are defined and addressed, leveraging HTTP to build APIs.</p> <p>HTTP defines how messages are formatted and transmitted, as well as what actions Web servers should take in response to various commands. In HTTP, one uses verbs to describe an action that will apply to a given URL. When you launch a web browser, the implicit verb is GET. However, there are other verbs in the HTTP specifications such as POST, PUT, PATCH, and DELETE. REST APIs use these verbs to describe possible actions. The URL in a REST API call is, like in a browser, the target to which the verb must be applied. For example, <code>GET http://google.com</code> means "retrieve the content of the main index page at google.com, and render it in my browser window".</p> <p>Let’s review the most important HTTP verbs:</p> <ul> <li>GET – Retrieve object instance properties</li> <li>PUT/PATCH –Modify object instance properties (two different ways)</li> <li>POST – Create a new object instance</li> <li>DELETE - Remove an object instance</li> </ul> <h2>Passing data back and forth</h2> <p>When we know how to use an HTTP verb to describe an action, we can use a URL to point to something in the API, but we need a mechanism to exchange data with the application at the other end of the API. This mechanism is useful to create new items or to retrieve a list of items from the application. We use a couple of techniques to do this. The first technique uses something described by the HTTP specifications called HTTP headers. A header allows the attachment of information with the API call as a key/value pair of string.</p> <p>For example, you can add an <code>Accept </code> header with value <code>application/json</code> to tell the API to return the result in json. There is no standard for HTTP headers, but there are many well-known headers used by APIs such as Accept, Content-Type, X-API-Version, etc. Headers are useful for passing a simple text value.</p> <p>But what if you want to pass more data to your API? For this, you can use the HTTP message payload, the body of your HTTP call. This is typically used to specify the credentials to get access to an API.</p> <h2>JSON</h2> <p>To pass data back and forth with an API, you need to make sure the client (calling the API) and the API agree on a format for this data. The most popular format nowadays is called JSON (for JavaScript Object Notation). JSON is a fairly readable machine language which typically looks like a series of Key/Value pairs separated with colons, with string values in quotes and multiple key/values separated with commas. The following is a simple example of a valid JSON snippet:</p> <pre><code class="language-json"> { “userName”: “foo@hpe.com”, “password”: “super-secure-password” } </code></pre> <p>JSON is case sensitive, so make sure you have the key names right ("userName" is not the same as "username"). You can express complex values by nesting JSON values and by using square brackets to indicate a collection of similar items as shown below:</p> <pre><code class="language-json">{ “members”: [ { “lastname”: “max”, “firstname”: “mad”, “age”: 50}, {“lastname”: “james”, “firstname”: “bond”, “age”: 60}], “count”: 2 } </code></pre> <p>An API might accept more than one format, and it may also provide responses in more than one format (JSON, XML, TEXT). Therefore, the convention is to use HTTP headers to specify which format you are going to use to pass data to the API (in your payload) and what data format you expect for the response. These headers are:</p> <ul> <li>Accept: application/json (please provide me with a response in JSON)</li> <li>Content-Type: application/json (I’m handing out a payload in JSON)</li> </ul> <h2>Putting it all together</h2> <p>In summary, an API uses:</p> <ul> <li>A URL. It also uses the term URI (Uniform Resource Identifier) to specify an object ID or a class of objects</li> <li>A verb to specify what action to take on that URL/URI</li> <li>One or more HTTP headers to attach certain properties on the API call, such as what format to accept for the response</li> <li>A payload to pass data to the API</li> </ul> <p>Let’s look at a few examples:</p> <ul> <li>GET http(s)://myAPIendpoint/rest/servers with HEADER Accept: application/json: Retrieves the list of servers from API endpoint http(s)://myAPIendpoint in JSON format</li> <li>POST http(s)://myAPIendpoint/rest/servers with HEADER Content-Type: application/json and appropriate JSON payload: Creates a new object of type server</li> <li>DELETE http(s)://myAPIendpoint/rest/servers/1212138349873: Deletes server object given provided UUID</li> </ul> <h2>Postman to the rescue</h2> <p>Since a browser can only really handle GET, it is clearly not the appropriate tool for manipulating an API. The good news is that a fantastic tool called Postman exists, which comes with a free version you can use to explore APIs. Visit <a href="http://getpostman.com">http://getpostman.com</a> to find the version that works for you (Windows, Mac and Linux). Once installed, start Postman on your machine. You do not need to sign-in to use it for exploring APIs. But since Postman is more than just a client API tool, you’ll want to check out its full feature set.</p> <h2>Let’s find a sandbox to give it a try</h2> <p><strong>Note:</strong> Make sure you have access to the Internet for this next section.</p> <p>Millions of APIs are available on the Internet today. Most of them are fee-based and require proper authentication, but a few free APIs are still available, which you can use with or without authentication. I selected a very simple one about a weather report, which doesn’t require authentication to use. You can look up more in this list to find another API, if you prefer: <a href="https://github.com/public-apis/public-apis">https://github.com/public-apis/public-apis</a></p> <p>The API end-point is: <a href="https://www.metaweather.com/api/">https://www.metaweather.com/api/</a>. According to the documentation, we first issue a call to find a location on Earth:</p> <ol> <li>Querying API for WOEID of a city, such as Paris</li> </ol> <p>In Postman, make sure the URL is set to <a href="https://www.metaweather.com/api/location/search/?query=Paris">https://www.metaweather.com/api/location/search/?query=Paris</a>, and that GET is the selected verb before you hit Send to issue the API call. You can see in the response section that it is, indeed provided in JSON, which is somehow readable:</p> <pre><code class="language-json">[{"title":"Paris","location_type":"City","woeid":615702,"latt_long":"48.856930,2.341200"}] </code></pre> <p>From the JSON response, we can extract the woeid (Where On Earth ID) for Paris (615702) and use this value to query for a weather report in Paris:</p> <ol start="2"> <li>Query API for weather report in Paris using WOEID</li> </ol> <p>Still in Postman, make sure the URL is set to <a href="https://www.metaweather.com/api/location/615702/">https://www.metaweather.com/api/location/615702/</a> and hit Send again:</p> <pre><code class="language-json"> { "consolidated_weather": [ { "id": 6618427756118016, "weather_state_name": "Showers", "weather_state_abbr": "s", "wind_direction_compass": "W", "created": "2019-11-06T12:36:06.334166Z", "applicable_date": "2019-11-06", "min_temp": 8.605, "max_temp": 12.215, "the_temp": 11.875, "wind_speed": 4.475518589403219, "wind_direction": 277.3125549075835, "air_pressure": 1005.5, "humidity": 75, "visibility": 10.535348564383998, "predictability": 73 }, { "id": 5413015522377728, "weather_state_name": "Light Rain", "weather_state_abbr": "lr", "wind_direction_compass": "SW", "created": "2019-11-06T12:36:09.637230Z", "applicable_date": "2019-11-07", "min_temp": 7.0, "max_temp": 10.62, "the_temp": 10.41, "wind_speed": 8.787927365497495, "wind_direction": 230.868718440178, "air_pressure": 1000.0, "humidity": 68, "visibility": 15.52930883639545, "predictability": 75 }, { "id": 6503888796516352, "weather_state_name": "Showers", "weather_state_abbr": "s", "wind_direction_compass": "ESE", "created": "2019-11-06T12:36:12.628582Z", "applicable_date": "2019-11-08", "min_temp": 4.395, "max_temp": 9.465, "the_temp": 9.11, "wind_speed": 3.1676953831490766, "wind_direction": 119.31121355342638, "air_pressure": 1005.5, "humidity": 76, "visibility": 13.218118686868687, "predictability": 73 }, { "id": 6070183267401728, "weather_state_name": "Light Rain", "weather_state_abbr": "lr", "wind_direction_compass": "SW", "created": "2019-11-06T12:36:15.316626Z", "applicable_date": "2019-11-09", "min_temp": 4.02, "max_temp": 9.68, "the_temp": 9.61, "wind_speed": 4.867012487862881, "wind_direction": 224.66670428043918, "air_pressure": 1012.5, "humidity": 73, "visibility": 12.878850015907101, "predictability": 75 }, { "id": 5082031551676416, "weather_state_name": "Showers", "weather_state_abbr": "s", "wind_direction_compass": "WNW", "created": "2019-11-06T12:36:18.342176Z", "applicable_date": "2019-11-10", "min_temp": 4.32, "max_temp": 9.120000000000001, "the_temp": 9.379999999999999, "wind_speed": 5.455819375540178, "wind_direction": 286.2336385713633, "air_pressure": 1006.5, "humidity": 72, "visibility": 12.235109460749225, "predictability": 73 }, { "id": 6399005695148032, "weather_state_name": "Showers", "weather_state_abbr": "s", "wind_direction_compass": "W", "created": "2019-11-06T12:36:21.245629Z", "applicable_date": "2019-11-11", "min_temp": 4.1850000000000005, "max_temp": 8.07, "the_temp": 6.65, "wind_speed": 2.8226837270341205, "wind_direction": 267.5, "air_pressure": 1009.0, "humidity": 77, "visibility": 9.999726596675416, "predictability": 73 } ], "time": "2019-11-06T14:45:55.699444+01:00", "sun_rise": "2019-11-06T07:44:50.213694+01:00", "sun_set": "2019-11-06T17:23:01.239529+01:00", "timezone_name": "LMT", "parent": { "title": "France", "location_type": "Country", "woeid": 23424819, "latt_long": "46.71,1.72" }, "sources": [ { "title": "BBC", "slug": "bbc", "url": "http://www.bbc.co.uk/weather/", "crawl_rate": 360 }, { "title": "Forecast.io", "slug": "forecast-io", "url": "http://forecast.io/", "crawl_rate": 480 }, { "title": "HAMweather", "slug": "hamweather", "url": "http://www.hamweather.com/", "crawl_rate": 360 }, { "title": "Met Office", "slug": "met-office", "url": "http://www.metoffice.gov.uk/", "crawl_rate": 180 }, { "title": "OpenWeatherMap", "slug": "openweathermap", "url": "http://openweathermap.org/", "crawl_rate": 360 }, { "title": "Weather Underground", "slug": "wunderground", "url": "https://www.wunderground.com/?apiref=fc30dc3cd224e19b", "crawl_rate": 720 }, { "title": "World Weather Online", "slug": "world-weather-online", "url": "http://www.worldweatheronline.com/", "crawl_rate": 360 } ], "title": "Paris", "location_type": "City", "woeid": 615702, "latt_long": "48.856930,2.341200", "timezone": "Europe/Paris" } </code></pre> <p>As you can see, a fair amount of information is provided by the API, including latitude, longitude, temperature, and weather conditions in Paris (which is not great at the moment by the way). As a developer, you could parse this JSON response and use it in your own application.</p> <h2>Now what?</h2> <p>Congratulations! You have placed your first API calls using Postman. This simple API only accepted GET calls, so one might argue everything could have been handled by a standard browser. The advantage of using Postman is that you can save your calls and start building collections of API calls that are useful in your job. We used a very limited API to get the weather report in Paris, but the exact same technique (using Postman) could be applied with other REST APIs, such as the ones from HPE OneView, HPE SimpliVity or HPE iLO. I plan on discussing these in future articles.</p> <p>If you have questions or want to connect, you can reach me on Twitter <a href="https://twitter.com/DidierLalli">@DidierLalli.</a> In addition, don’t forget to keep an eye out on the <a href="/blog">HPE DEV blog site</a> for future articles on this topic.</p>https://developer.hpe.com/2019-December-02/https://developer.hpe.com/2019-December-02/Mon, 02 Dec 2019 06:00:00 GMThttps://developer.hpe.com/using-hpe-cloud-volumes-with-amazon-eks/https://developer.hpe.com/using-hpe-cloud-volumes-with-amazon-eks/Fri, 29 Nov 2019 17:05:55 GMT<p>Running applications on Kubernetes is becoming more mainstream by the minute. Kubernetes is fairly unambiguous to the developer but may pose challenges for the Ops teams. Private cloud, hybrid cloud or public cloud? Managed Kubernetes or BYO (build your own) Kubernetes? There’s an endless matrix of combinations. It becomes even more challenging when you start caring about stateful applications and persistent storage, as data has gravity in the sense it becomes sticky to where you deploy your application. This technical tutorial will explore the paradigm of managed Kubernetes and how Hewlett Packard Enterprise (HPE) bring value to the public cloud by providing advanced data services to Kubernetes on Amazon Web Services (AWS).</p> <p>At KubeCon this year, HPE put together <a href="https://community.hpe.com/t5/HPE-Storage-Tech-Insiders/HPE-storage-supporting-thech-HPE-Container-Platform-at-KubeCon/ba-p/7070094">a hybrid cloud CI/CD pipeline</a> using the new HPE Container Platform and Amazon EKS (Elastic Kubernetes Service). Central to the solution was the integration of HPE Nimble Storage and HPE Cloud Volumes, two key components that enable true hybrid cloud for containers. This blog post will focus on getting started with Amazon EKS, deploying the HPE Volume Driver for Kubernetes FlexVolume Plugin onto EKS, and using HPE Cloud Volumes instead of Amazon EBS (Elastic Block Storage) for persistent storage.</p> <h1>Managed Kubernetes in Public Cloud</h1> <p>There are plenty of ways to run Kubernetes in the public cloud. Many distributions of Kubernetes allow you to deploy directly to the hyper-scalers by using simple cloud native tooling to instantiate Kubernetes clusters. But doing so brings the inconvenience of managing your own control plane, the Kubernetes masters with their associated services, which can be a bit daunting and not highly valuable or impactful work. Managed Kubernetes also comes in different shapes and sizes. The most common model is centered around creating clusters, where the cloud provider manages the entire control plane for you, and only worker nodes are deployed as regular instances in your cloud infrastructure. This model brings the benefit of being able to choose instance type (CPU, GPU, and memory) and allows customization of the worker nodes, like installing custom packages and other tweaks necessary to run a certain workload.</p> <p>The managed Kubernetes service I’m going to focus on in this blog post is Amazon EKS. HPE recently released the <a href="https://github.com/hpe-storage/flexvolume-driver">HPE Volume Driver for Kubernetes FlexVolume Plugin</a> version 3.1, which supports HPE Cloud Volumes and includes support not only for EKS, but also for Azure AKS (Azure Kubernetes Service). As HPE just received connectivity with Google Cloud, expect full support for GKE (Google Kubernetes Engine) in the near future.</p> <h1>Prerequisites</h1> <p>This tutorial assumes a certain familiarity with AWS and HPE Cloud Volumes, as well as the following:</p> <ul> <li>HPE Cloud Volumes connectivity that is setup according to <a href="https://docs.cloudvolumes.hpe.com/help/">the documentation</a> in a region that provides Amazon EKS. The litmus test is if you can connect a cloud volume to an EC2 instance in the same region.</li> <li>A VPC ID and IPv4 CIDR of the existing VPC from where HPE Cloud Volumes can be accessed.</li> <li>HPE Cloud Volumes API Access Key and API Secret Key generated for use by the FlexVolume driver. Make sure you're logged in to <a href="https://cloudvolumes.hpe.com/login">cloudvolumes.hpe.com</a> and <a href="https://cloudvolumes.hpe.com/usersettings">copy from here</a>.</li> <li>The environment variables <code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code> and <code>AWS_DEFAULT_REGION</code> set.</li> <li>The utilities <code>aws</code>, <code>helm</code> (version 2), <code>kubectl</code> and <code>eksctl</code> installed on the machine you’re using to provision your EKS cluster (further info outlined where used below).</li> <li>An SSH public key that was generated from the provisioning host.</li> </ul> <p>If you have any questions on the prerequisites or need clarification, please feel free to reach out to me <a href="https://hpedev.slack.com/">on Slack</a>.</p> <h1>Configure your subnets</h1> <p>By default, <code>eksctl</code> creates new VPCs and subnets for the cluster it provisions. This is impractical when using HPE Cloud Volumes, as you want to use an existing VPC with subnets already accessible from HPE Cloud Volumes. In this tutorial, I’ll tag two subnets for the internal load-balancer and two subnets for the external load-balancer. These subnets will be referenced in the EKS setup later.</p> <pre><code class="language-markdown">aws ec2 create-tags --resources subnet-AA000000 subnet-BB000000 --tags Key=kubernetes.io/role/elb,Value=1 aws ec2 create-tags --resources subnet-CC000000 subnet-DD000000 --tags Key=kubernetes.io/role/internal-elb,Value=1 </code></pre> <p>The above command produces no output if it is successful. Ensure the environment variables are set as outlined in the prerequisites. Learn more about the <a href="https://aws.amazon.com/cli/">the AWS CLI here</a>.</p> <h1>Fire up the EKS cluster</h1> <p>The tool <code>eksctl</code> was initially developed outside of Amazon. It’s now a joint effort between Amazon and the original author, Weaveworks. Please see the <a href="https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html">the official documentation</a> for the <code>eksctl</code> tool to learn how to get started.</p> <p>Review and edit the <code>eksctl</code> command below to fit the target environment. Ensure there’s a public SSH key in place, environment variables set and the subnets have been tagged. </p> <pre><code class="language-markdown">eksctl create cluster \ --name HPEDEV \ --version 1.13 \ --region ${AWS_DEFAULT_REGION} \ --nodegroup-name standard-workers \ --node-type t3.medium \ --nodes 3 \ --nodes-min 1 \ --nodes-max 4 \ --ssh-public-key ~/.ssh/id_rsa.pub \ --node-ami auto \ --vpc-public-subnets subnet-AA000000,subnet-BB000000 \ --vpc-private-subnets subnet-CC000000,subnet-DD000000 </code></pre> <p><strong>Note:</strong> Kubernetes 1.12 or 1.13 is currently required for the FlexVolume driver when deployed on EKS. Stay tuned for future updates.</p> <p>This command takes a few minutes to execute. Here’s an example output from a successful run:</p> <pre><code class="language-markdown">[ℹ]  eksctl version 0.8.0 [ℹ]  using region us-west-2 [✔]  using existing VPC (vpc-00000000) and subnets (private:[subnet-CC000000 subnet-DD000000] public:[subnet-AA000000 subnet-AA000000]) [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets [ℹ]  nodegroup "standard-workers" will use "ami-04e247c4613de71fa" [AmazonLinux2/1.13] [ℹ]  using SSH public key "/home/ubuntu/.ssh/id_rsa.pub" as "eksctl-HPEDEV-nodegroup-standard-workers-d0:ac:7f:86:04:d6:1a:ec:71:cc:3d:75:cf:1b: f2:46" [ℹ]  using Kubernetes version 1.13 [ℹ]  creating EKS cluster "HPEDEV" in "us-west-2" region [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --cluster=HPEDEV' [ℹ]  CloudWatch logging will not be enabled for cluster "HPEDEV" in "us-west-2" [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-west-2 --cluster=HPEDEV' [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "HPEDEV" in "us-west-2" [ℹ]  2 sequential tasks: { create cluster control plane "HPEDEV", create nodegroup "standard-workers" } [ℹ]  building cluster stack "eksctl-HPEDEV-cluster" [ℹ]  deploying stack "eksctl-HPEDEV-cluster" [ℹ]  building nodegroup stack "eksctl-HPEDEV-nodegroup-standard-workers" [ℹ]  deploying stack "eksctl-HPEDEV-nodegroup-standard-workers" [✔]  all EKS cluster resources for "HPEDEV" have been created [✔]  saved kubeconfig as "/home/ubuntu/.kube/config" [ℹ]  adding identity "arn:aws:iam::000000000000:role/eksctl-HPEDEV-nodegroup-standard-NodeInstanceRole-000000000000" to auth ConfigMap [ℹ]  nodegroup "standard-workers" has 0 node(s) [ℹ]  waiting for at least 1 node(s) to become ready in "standard-workers" [ℹ]  nodegroup "standard-workers" has 3 node(s) [ℹ]  node "ip-172-31-1-186.us-west-2.compute.internal" is not ready [ℹ]  node "ip-172-31-24-30.us-west-2.compute.internal" is not ready [ℹ]  node "ip-172-31-28-199.us-west-2.compute.internal" is ready [ℹ]  kubectl command should work with "/home/ubuntu/.kube/config", try 'kubectl get nodes' [✔]  EKS cluster "HPEDEV" in "us-west-2" region is ready </code></pre> <p>At this point, you should have connectivity to the cluster using <code>kubectl</code>, as shown below:</p> <pre><code class="language-markdown">$ kubectl get nodes NAME                                          STATUS   ROLES    AGE   VERSION ip-172-31-1-186.us-west-2.compute.internal    Ready    &#x3C;none>   2m    v1.13.11-eks-5876d6 ip-172-31-24-30.us-west-2.compute.internal    Ready    &#x3C;none>   2m    v1.13.11-eks-5876d6 ip-172-31-28-199.us-west-2.compute.internal   Ready    &#x3C;none>   2m    v1.13.11-eks-5876d6 </code></pre> <p><img src="/uploads/media/2019/10/screen-shot-2019-11-26-at-22906-pm-1574814807294.png" alt=""></p> <p>You're now ready to deploy the FlexVolume driver!</p> <h1>Deploy the FlexVolume driver</h1> <p>The easiest way to install the HPE Volume Driver for Kubernetes FlexVolume Plugin is to use Helm. The FlexVolume driver Helm chart is available on <a href="https://hub.helm.sh/charts/hpe-storage/hpe-flexvolume-driver">hub.helm.sh</a>. But before you deploy the driver, you need to create a <code>values.yaml</code> file that corresponds to the HPE Cloud Volumes setup. This file is used as input to Helm.</p> <p>Use this <code>values.yaml</code> file as a starting point:</p> <pre><code class="language-yaml">--- backend: cloudvolumes.hpe.com username: &#x3C; HPE Cloud Volumes API Access Key > password: &#x3C; HPE Cloud Volumes API Secret Key > pluginType: cv fsType: xfs storageClass:   defaultClass: true cv:   config:     snapPrefix: BaseFor     automatedConnection: true     existingCloudSubnet: &#x3C; CIDR network in VPC, i.e 172.31.0.0/16 >     region: us-west-2     privateCloud: &#x3C; VPC ID, i.e vpc-00000000 >     cloudComputeProvider: "Amazon AWS"     perfPolicy: Other     volumeType: GPF     encryption: true     protectionTemplate: twicedaily:4     destroyOnRm: true     limitIOPS: "1000"     initiators:       - '"eth0"'       - '"eth1"' </code></pre> <p>Next you need to install Tiller on your cluster. Helm 3 does not require Tiller, however, the Helm chart has not been tested with Helm 3. So I'm using Helm 2 for this exercise.</p> <pre><code class="language-markdown">kubectl -n kube-system create serviceaccount tiller kubectl create clusterrolebinding tiller \   --clusterrole=cluster-admin \   --serviceaccount=kube-system:tiller helm init --service-account tiller kubectl -n kube-system  rollout status deploy/tiller-deploy </code></pre> <p>Tiller should now be installed on the cluster. Now install the chart:</p> <pre><code class="language-markdown">helm repo add hpe-storage https://hpe-storage.github.io/co-deployments/ helm install hpe-storage/hpe-flexvolume-driver --version 3.1.0 --namespace kube-system --name hpe-flexvolume-driver -f values.yaml </code></pre> <p>It should take less than a minute or so to install the required components. Run the below <code>kubectl</code> command to verify that all components are running:</p> <pre><code class="language-markdown">kubectl -n kube-system get deploy/hpe-dynamic-provisioner deploy/cv-cp deploy/hpe-flexvolume-driver </code></pre> <p>It should output something similar to this:</p> <pre><code class="language-markdown">NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE deployment.extensions/hpe-dynamic-provisioner   1/1     1            1           30s deployment.extensions/cv-cp                     1/1     1            1           30s NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE daemonset.extensions/hpe-flexvolume-driver   3         3         3       3            3           &#x3C;none>          30s </code></pre> <p><strong>Note:</strong> Incidentally, the cluster will end up with two default storage classes, which will block provisioning from a default storage class. Annotate the built-in “gp2” storage class to become non-default:</p> <pre><code class="language-markdown">kubectl annotate sc/gp2 storageclass.kubernetes.io/is-default-class=false --overwrite </code></pre> <p>The EKS cluster is now ready for users to declare PVCs (Persistent Volume Claims) served by HPE Cloud Volumes!</p> <h1>Hello World</h1> <p>In the current incarnation of HPE Cloud Volumes it may take some time for the first PV (Persistent Volume) to connect to the first Pod. Network resources are provisioned in the background as needed. The second volume to attach only takes a few seconds. Now create a PVC and attach a workload.</p> <p>Here’s a an example PVC that creates a 64GiB volume:</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: mypvc spec:   accessModes:   - ReadWriteOnce   resources:     requests:       storage: 64Gi </code></pre> <p>Either create a file with the PVC stanza above or paste to <code>stdin</code> with <code>kubectl create -f &#x3C;file></code>. Once the PVC is declared, a PV will be created. Inspect it with the following:</p> <pre><code class="language-yaml">kubectl get pvc NAME    STATUS   VOLUME                                              CAPACITY   ACCESS MODES   STORAGECLASS   AGE mypvc   Bound    hpe-standard-5b75aaa8-10a7-11ea-8232-0a210d2b9ace   64Gi       RWO            hpe-standard   18s </code></pre> <p>Volume creation does not attach the device to any host. The attachment is being done once a workload requests the claim. So, you must declare a simple Pod to attach the claim:</p> <pre><code class="language-yaml">--- apiVersion: v1 kind: Pod metadata:   name: ioping spec:   containers:   - name: ioping     image: hpestorage/ioping     command: [ "ioping" ]     args: [ "/data" ]     volumeMounts:     - name: data       mountPath: /data   volumes:   - name: data     persistentVolumeClaim:       claimName: mypvc </code></pre> <p>Depending on the environment, this initial deployment could take up to 10 minutes. I’ve been bouncing clusters up and down all day, hence my pod was up in less than a minute:</p> <pre><code class="language-markdown">kubectl get pod NAME     READY   STATUS    RESTARTS   AGE ioping   1/1     Running   0          53s </code></pre> <p>Logs from the Pod indicate a XFS filesystem mount on /data served by a multipath device:</p> <pre><code class="language-markdown">kubectl logs pod/ioping 4 KiB &#x3C;&#x3C;&#x3C; /data (xfs /dev/mapper/mpatha): request=1 time=8.26 ms (warmup) 4 KiB &#x3C;&#x3C;&#x3C; /data (xfs /dev/mapper/mpatha): request=2 time=7.05 ms 4 KiB &#x3C;&#x3C;&#x3C; /data (xfs /dev/mapper/mpatha): request=3 time=7.33 ms 4 KiB &#x3C;&#x3C;&#x3C; /data (xfs /dev/mapper/mpatha): request=4 time=8.13 ms 4 KiB &#x3C;&#x3C;&#x3C; /data (xfs /dev/mapper/mpatha): request=5 time=8.44 ms ^C </code></pre> <p>At this point, you can also login to the HPE Cloud Volumes portal to observe the attached volume.</p> <p><img src="/uploads/media/2019/10/screen-shot-2019-11-26-at-43542-pm-1574815092842.png" alt=""></p> <h1>Summary</h1> <p>Many infrastructure choices are available today that enable Kubernetes. Likewise, there are many options on the persistent storage side. HPE Cloud Volumes gives Ops teams much needed relief, as they may rely on a service to persist their applications data rather than becoming storage administrators. Not many of the cloud native software-defined storage (SDS) solutions for Kubernetes give you that luxury. Consuming storage services directly from the cloud provider creates an unnecessary lock-in situation, and the lack of enterprise-grade features is what drives users to alternative SDS solutions. HPE checks all the boxes by providing reliable and performant storage for Kubernetes. And it’s for private cloud, hybrid cloud, and public cloud as well as managed Kubernetes and BYO Kubernetes. Multicloud mobility and competitive pricing comes bundled!</p> <p>Start today by learning more about:</p> <ul> <li><a href="https://cloudvolumes.hpe.com">HPE Cloud Volumes</a></li> <li><a href="https://github.com/hpe-storage/flexvolume-driver/tree/master/examples/kubernetes/hpe-cloud-volumes">HPE Cloud Volumes StorageClass parameters</a></li> <li><a href="https://aws.amazon.com/eks/">Amazon EKS</a></li> </ul>https://developer.hpe.com/devops-d-day-speakers-focus-on-real-world-use-cases/https://developer.hpe.com/devops-d-day-speakers-focus-on-real-world-use-cases/Thu, 21 Nov 2019 16:00:12 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1574352241077.png" alt="picture3"></p> <p>A large crowd of DevOps fanatics gathered in Marseille, France on November 14th to participate in sessions and workshops at the 5th DevOps D-Day event. The event was held in the VIP lounges of the famous, French soccer club arena of the Olympique de Marseille, right in the heart of the city. Despite the bad weather, 1,200 people and 24 sponsors gathered for the event.</p> <p>The conference featured around fifty different sessions ranging from high-level talks about how to develop a DevOps mindset to detailed examples of how companies implemented DevOps within their organization. There were numerous sessions on specific technologies as well, with Kubernetes being a very popular topic. Hewlett Packard Enterprise (HPE) and longtime partner, AntemetA, co-sponsored a large booth and two keynote sessions. These were presented by Christian Schutz who spoke about the adoption of Kubernetes in French companies over the course of 2019 and how to build a CI/CD pipeline with Kubernetes and OpenStack.</p> <p>During the breaks between sessions, crowds gathered in our booth, eager to play the famous arcade game, Space Invaders. The one who achieved the highest score won a prize offered by AntemetA. Those of us on the HPE Dev team spent a lot of time conversing with interesting event attendees, such as La Poste and Airbus Helicopters, as well as many university students who came looking for swag (and jobs). We also had a drawing for a prize where we randomly chose a name from those who signed up for the <a href="https://hpe-developer.8ar.ms/newsletter-signup">HPE Dev Newsletter.</a> The winner of our drawing was from NeoPost. He came to pick up the gift and have his picture taken at the end of the day.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/small-version-didier-with-winner-3-1574454720874.jpg" alt="small version didier with winner 3"></p> <p>The few sessions I went to focused on customer experiences, which I always find interesting. In the sessions I attended, the speakers all delivered their talk in French in a laidback, interesting way. I found two sessions particularly engaging.</p> <p>The first was from the sport’s gear manufacturer, Decathlon, who discussed their DevOps implementation and the toolset they put in place for their developer teams. The speaker explained that, within Decathlon, they have a developer community that participates in regular face-to-face meetups on particular technical subjects and their technology choices are made in conjunction with suggestions from the community.</p> <p>The second talk I attended was given by an IT Manager from CMA CGM, one of the world’s top container transportation and shipping companies, who talked about their DevOps transformation. It was quite funny to hear a physical container shipping company talk about software containers, and how DevOps, when not done correctly, can quickly become DevOops!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1574352210141.png" alt="picture1"></p> <p>Don’t we know it! DevOps requires many enterprises to change a lot of deeply entrenched processes. These changes can be challenging for some, as evidenced in our recent blog <a href="/blog/devops-and-its-impact-on-project-management">DevOps and its impact on project management.</a> Don’t forget to check back and see what new blogs are up on the <a href="/blog">HPE Dev site</a> on this and other interesting topics.</p>https://developer.hpe.com/insights-from-hpe-dev-at-microsoft-ignite-2019/https://developer.hpe.com/insights-from-hpe-dev-at-microsoft-ignite-2019/Wed, 20 Nov 2019 17:04:41 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture6-1574269899492.png" alt="picture6"></p> <p>The light shone brightly on HPE DEV at Microsoft Ignite 2019, as we shared knowledge, collaborated, and connected with developers. Our objectives at this show were straightforward – show how to build the best possible experiences with Hewlett Packard Enterprise (HPE) solutions. We did that by:</p> <ol> <li>Demonstrating how the HPE development toolset/frameworks easily build out end-to-end applications</li> <li>Exhibiting real-world use cases with practical suggestions and ideas</li> <li>Promoting the HPE DEV Community and showcasing how we’re encouraging developer collaboration</li> <li>Spotlighting key HPE offerings around IoT and edge computing, Microsoft Azure and Azure Stack</li> </ol> <p>HPE DEV supported the HPE tagline <strong>Fear no Cloud</strong> by demonstrating how attendees could consume Microsoft Azure Cognitive Services at edge and by demystifying DevOps for use with Azure Stack as part of a hybrid IT infrastructure.</p> <p>A live demo showing Face Detection with Azure Cognitive Services was one of the main attractions at the event. The demo offered a great opportunity for attendees to visualize an application architecture built with lightweight containers running in a Kubernetes cluster on a ruggedized Edgeline300 Server receiving a live feed from a simple, everyday camera. Each attendee was able to interact with the application and find out more about how edge processing works in real world use cases combining artificial intelligence (AI) with the best HPE solutions for edge computing.</p> <p>The HPE DEV demonstration of DevOps workflows with an HPE Proliant for Microsoft Azure Stack Hub satisfied developer curiosity regarding how hybrid IT can support and extend an application’s capability from Edge to Cloud with a <strong>Fear No Cloud</strong> mindset.</p> <p>As HPE DEV has done in previous events, attendees had the opportunity to play the <a href="https://github.com/HewlettPackard/hpe-hack-shack-attack">Hack Shack Attack</a> game, an open source 8-bit arcade game. To add to the fun, as part of a raffle encouraging folks to sign up for the HPE DEV newsletter, we offered a prize each day to one lucky winner.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture4-1574269876302.png" alt="picture4"></p> <p><em>Everyone waiting for the results of the daily raffle</em></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1574269854835.png" alt="picture3"></p> <p><em>And the winner is…</em></p> <p>There were many announcements during Microsoft Ignite, but the two most significant to HPE were:</p> <p><strong>Azure Arc:</strong> Microsoft was among the first of the big cloud vendors to bet big on hybrid deployments. With Azure Arc, the company is taking this a step further. It will let enterprises use Azure to manage their resources across clouds -- including those of competitors, such as AWS and Google Cloud.</p> <p>HPE played a key role in the announcement, with the HPE Superdome Flex featured prominently on the announcement stage. HPE will participate in the preview of the product and, although it’s not yet publically available, customers can sign up for this now. For more information on <a href="https://azure.microsoft.com/en-us/blog/azure-arc-extending-azure-management-to-any-infrastructure/?ActivityID=NA&#x26;AssetID=NA&#x26;elq2=~~eloqua..type--emailfield..syntax--recipientid..encodeFor--url~~">Azure Arc, check out this link.</a></p> <p><strong>Azure Stack Portfolio:</strong> Microsoft also launched a full portfolio of hybrid solutions, spanning from the edge to Azure Stack Hub (previously referred to as Azure Stack).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1574269831398.png" alt="picture2"></p> <p><a href="https://azure.microsoft.com/en-us/blog/expanding-the-azure-stack-portfolio-to-run-hybrid-applications-across-the-cloud-datacenter-and-the-edge/?ActivityID=NA&#x26;AssetID=NA&#x26;elq2=~~eloqua..type--emailfield..syntax--recipientid..encodeFor--url~~">Click here for specifics on this new portfolio.</a></p> <p>The HPE DEV area was the talk of town within the HPE booth. It had a stronger than ever presence given the number of developers and Microsoft MVP’s in attendance who wanted to know more about real world use case examples and to see sample code. Many wanted to directly signup as part of the HPE DEV Community and keep plugged into HPE DEV collaborations.</p> <p>As the tradeshow floor activities wound down, Microsoft hosted an event at Universal Studios. Like everyone else, HPE DEV joined in the fun.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1574269799671.png" alt="picture1"></p> <p><em>Pramod, Vivek, and Emmy join in the celebration</em></p>https://developer.hpe.com/running-non-cloud-native-apps-on-kubernetes-with-kubedirector/https://developer.hpe.com/running-non-cloud-native-apps-on-kubernetes-with-kubedirector/Mon, 18 Nov 2019 17:16:18 GMT<p><strong>Editor’s Note – HPE Ezmeral Container Platform is now HPE Ezmeral Runtime Enterprise</strong>. For more information on why the name was changed, please <a href="https://community.hpe.com/t5/HPE-Ezmeral-Uncut/HPE-Ezmeral-Container-Platform-is-now-HPE-Ezmeral-Runtime/ba-p/7151720#.YW7nOxrMKM8">click here</a>.</p> <hr> <p>This week at <a href="https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/">KubeCon North America,</a> Hewlett Packard Enterprise (HPE) unveiled the new <a href="https://www.hpe.com/us/en/newsroom/press-release/2019/11/Hewlett-Packard-Enterprise-introduces-Kubernetes-based-platform-for-bare-metal-and-edge-to-cloud-deployments.html">HPE Container Platform.</a> It’s the industry’s first Kubernetes-based software platform designed to run both cloud-native and non-cloud-native applications in containers, enabling true hybrid cloud operations across any location: on-premises, public clouds, and the edge.</p> <p>It’s widely acknowledged that open source Kubernetes is the de facto standard for the orchestration of containerized cloud-native applications. However, it’s another thing entirely to use Kubernetes to orchestrate the containerized deployment and management of non-cloud-native monolithic applications as well. How can HPE make such a bold claim?</p> <p>Let’s take a peek under the hood at one of the technical innovations in the HPE Container Platform that permits HPE to back up such an aggressive claim. By way of background, I’m one of the co-founders of BlueData, a software company that HPE acquired just about a year ago. The HPE Container Platform represents the next major release of BlueData’s container-based software platform, combined with integrated persistent container storage from MapR (also recently acquired by HPE), and open source Kubernetes for container orchestration.</p> <p>Prior to the acquisition, the BlueData team initiated an Apache open source project called KubeDirector. The initiative was focused on running non-cloud-native, stateful applications on Kubernetes. You can refer to my earlier blog post <a href="https://kubernetes.io/blog/2018/10/03/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubernetes/">here,</a> where I described the KubeDirector project. Now, with the HPE Container Platform, KubeDirector has come into its own as a key component of the platform -- delivering on its objective to deploy and manage non-cloud-native applications on Kubernetes.</p> <h2>KubeDirector and Non-Cloud-Native Applications</h2> <p>Non-cloud-native, monolithic applications (e.g. stateful applications with persistent storage) have a number of specific requirements. For example, they typically require fixed network configuration (i.e. static IP addresses). Kubernetes provides a construct, known as <a href="/blog/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern">StatefulSets,</a> which permits an application to be deployed with stable, unique network identifiers. This means that the IP address of a container will be preserved across pod rescheduling.</p> <p>However, another attribute of non-cloud-native applications is that they may store data in /etc or other directories typically located on the root (“/”) file system of the container. When Kubernetes restarts a crashed container, the storage associated with the original instance of the container is lost. This means that any data stored by the stateful application in the /etc directory will be lost.</p> <p>Kubernetes uses the <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolume</a> construct to overcome this default behavior and allow the container to store data in a unit of storage that persists beyond the lifespan of a single container. But PersistentVolumes cannot easily be used for the root or “/” file system of a container. Any data written to the root file system will be lost when the container exits.</p> <p>By clever use of the Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">Init Container</a> concept, KubeDirector overcomes this limitation and allows the root file system of a container to persist beyond the life span of the container. In fact, KubeDirector permits the user to specify which of the directories typically located on the root file system (/etc, /bin, etc) need to be preserved. This means stateful applications that write data to their root file systems can now successfully run on Kubernetes.</p> <p>Another common attribute of non-cloud-native, stateful applications is that they require careful management to survive node loss, software upgrade, and application cluster expansion and contraction. Common Kubernetes application deployment tools such Helm or KubeFlow are “client side” only mechanisms. This means that once they are used to deploy the application, they are out of the picture. There is no “server side” component to help the application respond to events such as node (container) loss, software upgrade, or application cluster expansion. In order to support these sorts of operations, an application-specific <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator</a> needs to be written. But writing an application Operator is not an easy task – it requires deep application domain knowledge as well as familiarity with the Kubernetes custom resource APIs.</p> <p>KubeDirector is an application deployment tool as well as a Kubernetes Operator, and it is application-neutral. In other words, the KubeDirector Operator can be used for multiple applications: it can deploy and manage any application using Kubernetes without additional code being written. The term KubeDirectorApp identifies a given application type managed by KubeDirector. A simple YAML file is used to describe the attributes of a given KubeDirectorApp. This YAML file is given to the KubeDirector Operator, which can then deploy the application and manage the application cluster-specific expansion, contraction, upgrade, node loss, and other operations.</p> <p>The use of the single KubeDirector operator to manage multiple KubeDirectorApps also removes the complex “Day 2” operations required to add a new Operator to a running Kubernetes cluster. No need to grant escalated privileges in order to add a new Operator to support a new application. Just write the YAML file describing the new application and KubeDirector does the rest.</p> <p>In short, with KubeDirector, a lot less effort is required to add support for a new application to Kubernetes:</p> <ul> <li>No need to create or modify an application-specific Operator</li> <li>No need to rely on or certify an Operator from elsewhere</li> <li>No need to register a new Custom Resource Definition and/or update user/group ACLs</li> <li>No need to (possibly dramatically) change clients to deal with new schemas</li> <li>No need for complexity with an easy mechanism for configuring per user per KubeDirectorApp permissions</li> </ul> <p>The new HPE Container Platform will be generally available early next year, so stay tuned for more. Many innovative aspects of the HPE Container Platform ease the deployment and management of containerized applications for environments requiring enterprise-grade security, performance, and scalability – running on bare-metal or virtualized infrastructure, from edge to core to cloud. And one thing is for sure: with KubeDirector under the hood, HPE is delivering on its claim to run non-cloud-native, stateful applications on Kubernetes.</p> <p>If you’re at <a href="https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/">KubeCon + CloudNativeCon</a> this week, you can join my presentation on Thursday afternoon with my colleague Joel Baxter to learn more: <a href="https://kccncna19.sched.com/event/UaaF/kubedirector-deploying-complex-stateful-applications-on-kubernetes-joel-baxter-thomas-phelan-hewlett-packard-enterprise">KubeDirector - Deploying Complex Stateful Applications on Kubernetes.</a> You can also stop by the HPE booth (P28) at KubeCon to meet us and ask any questions, and check out the KubeDirector GitHub repo <a href="https://github.com/bluek8s/kubedirector/">here.</a></p>https://developer.hpe.com/10th-anniversary-of-devops-days-celebrated-in-ghent/https://developer.hpe.com/10th-anniversary-of-devops-days-celebrated-in-ghent/Thu, 14 Nov 2019 15:56:42 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1573748147341.png" alt="picture2"></p> <p>I recently enjoyed attending DevOps Days in Ghent, one of the prettiest cities in Europe. Ghent was especially beautiful during the two days I was at the event, given the gorgeous weather we experienced. Hewlett Packard Enterprise (HPE) was a Gold Sponsor for this 10th year anniversary celebration of DevOps Days, and Stefan De Schuyter (Belgium presales), Simon Leech, Mario Devargas (PointNext), and I were excited to be able to present the HPE Dev Community to the 500+ attendees who came to Ghent.</p> <p>To encourage conversations between attendees and the 25+ vendors at the event, the organizers offered a prize for those who went around to each booth to have a form stamped. Our local HPE host, Stefan, immediately hooked people and brought them into our booth by saying “What does HPE have for you? Stamps, stickers, and APIs.” It was a fun way to start the dialog as we gave each visitor a stamp. We also engaged in conversations about HPE Composable Infrastructure and our DevOps transformation services.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1573748200636.png" alt="picture3"></p> <p>The DevOps Days session format was somewhat unusual. There was a good balance of 15 minute breakout sessions, 5 minute "Ignite" sessions, and so-called OpenSpace sessions where attendees got to choose subjects for an open discussion in which they could both listen and contribute. The Ignite sessions were delivered through automatically advancing slides and most speakers were just incredible in their ability to convey their thoughts and still keep up with the timing. The event used very creative formats to communicate with the audience and it provided me with a lot of good ideas on how I could improve my own presentation skills.</p> <p>During our time at the event, we were also very lucky to have "the godfather of DevOps", Patrick Debois, visit our booth and shake hands with the gremlin, our mascot. I would say that the show was very successful, given everyone’s smiling faces and the whole engaged atmosphere at the event.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/shaking-hands-with-stack-4-1573752028593.png" alt="shaking hands with stack (4)"></p> <p>In the past decade, DevOps Days has grown from a single event to over 80 events that are now being hosted in different locations around the world. Each technical conference features session topics on software development, IT infrastructure operations, and the intersection between them. They are hosted by local volunteers and deliver a combination of curated talks combined with self-organized open space content focused on areas like automation, testing, security, and organizational culture. Make sure you check the <a href="https://devopsdays.org/">DevOps Days website</a> to find a DevOps Days event close to you…I highly recommend them.</p>https://developer.hpe.com/meet-the-hpe-dev-team-blaine-southam-and-michael-mattsson/https://developer.hpe.com/meet-the-hpe-dev-team-blaine-southam-and-michael-mattsson/Wed, 06 Nov 2019 18:16:14 GMT<p><a href="https://developer.hpe.com/community">HPE DEV community</a> members hail from a variety of backgrounds. In this edition of our <strong>Meet the HPE DEV Team</strong> blog, I’m going to introduce you to Blaine Southam and Michael Mattsson.</p> <h2>Blaine Southam – Distinguished Technologist by day / ham radio operator by night</h2> <p>As a Distinguished Technologist, Blaine is often pulled into numerous projects. Folks rely on him to gather the right people, current processes, and correct technologies required to ensure the success of a new project. He’s worked as the Composable API Ecosystem principle architect, as the designer for multiple HPE OneView plugins, and was also responsible for the HPE OneView Global Dashboard and the HPE HC380 hyperconverged offering. Blaine also led the HPE Synergy / VMware Cloud Foundation integration project, which was recently added as an HPE GreenLake service. As the current HPE Cloud Ecosystem Architect, Blaine is now involved in helping internal and external partners offer their content as-a-Service through the HPE portal.</p> <p>Blaine is the type of person who’s often involved in many projects while at work, and with his four kids, he’s kept pretty busy at home as well. When he’s not spending time with his family, Blaine volunteers with the youth group at his church, planning activities and summer camps. Yet, even with all this going on, he’s somehow found the time to get his Amateur Radio License (aka Ham radio). Recently, Blaine was excited to receive a message from a gentleman in Germany who congratulated him on getting his license. If you’d like to connect with Blaine, you can reach him on Twitter <a href="https://twitter.com/bsoutham">@bsoutham.</a> Or, better yet, use his call sign, KE0YAN, and give him a shout out over the radio.</p> <h2>Michael Mattsson – Master Technologist and avid gamer</h2> <p>As a systems engineer and developer, Michael has had the opportunity to work for a number of different companies on very interesting projects over the years. Starting off in an R&#x26;D group at Volvo, Michael built a lot of custom automation and system management software that garnered the attention of Volvo partner, Proact. Proact offered him a consulting role for their top accounts where his ability to visualize the broader picture and talent in automation led to key open storage projects. This set him up for his next career move, one that literally moved him from Europe to the U.S., with a new employer located in Redwood City, California.</p> <p>Although Michael’s seen a lot of change in his career, one thing’s remained constant – his focus on data and storage. An HPE Master Technologist, Michael is currently working as a technical marketing engineer for the HPE storage marketing group. An HPE Nimble Storage expert, he concentrates on container ecosystems and how they relate to storage. I got a chuckle when Michael told me “If I had a cent for every time I spelled Kubernetes, today I would be a millionaire.” Michael attributes the fact that he became a developer due to his early exposure to video games, starting with Pong when he was only four years old. A life-long gamer who’s been through just about every home game console there is, Michael even has his own arcade at home to which he devotes a great deal of time for care and maintenance. You can catch up with Michael on Twitter <a href="https://twitter.com/datamattsson">@datamattsson.</a></p> <p>Take a look at <a href="https://www.youtube.com/watch?v=hfY6Ko02yiU&#x26;feature=youtu.be">Blaine and Michael’s video</a> to learn a little more about them. And, if you ever have questions on HPE OneView, cloud ecosystems, containers, or HPE Nimble Storage, don’t be afraid to reach out to Blaine or Michael on <a href="https://slack.hpedev.io/">Slack.</a> Blaine can be reached at bsoutham and you can address Michael at michaelm.</p>https://developer.hpe.com/hpe-hackathon-builds-greater-competency-in-iot-and-ai/https://developer.hpe.com/hpe-hackathon-builds-greater-competency-in-iot-and-ai/Wed, 06 Nov 2019 18:12:07 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/florian-hackathon-first-image1-1573154920702.jpg" alt="florian hackathon first image1"></p> <p>In May of this year, the HPE Customer Technology Center (CTC) in Böeblingen, Germany held its first HPE hackathon. The event brought together a group of Hewlett Packard Enterprise (HPE) developers and designers from IoT &#x26; Data Analytics, Presales, Pointnext services, and certain delivery groups who wished to become more knowledgeable about Internet of Things (IoT) and artificial intelligence (AI) implementations. Noticing that the more concrete IoT and AI projects became, the harder it was to find enough proficient resources to handle them, the group planned a hackathon that could help multiply the knowledge base.</p> <p>Hackathons can help build proficiency by going beyond PowerPoint presentations and getting down into the finer details to determine how the technology really works. The group locked themselves away for a week to learn more about these topics. To focus their efforts, they defined two projects to work on using HPE technologies; one covered data pipelining and the other covered data analytics.</p> <h2>Data Pipeline with HPE OTLink</h2> <p>The scope of the first project was to set up a data pipeline, from data acquisition to analytics. It was set up to gather data from multiple and different types of devices, including temperature and humidity sensors, Philips ® Hue light bulbs, door sensors, and edge hardware. The data was acquired using HPE Edgeline systems with HPE OTLink software and sent to a dashboard in the Azure cloud using open-source data pipeline services. In a second step, we established bi-directional communication to execute commands from the cloud at the edge.</p> <h2>Data Analytics Provisioning and Analytics with HPE BlueData</h2> <p>The second project set up a BlueData software stack on an HPE Apollo infrastructure including NVIDIA GPUs to provision multiple services from a marketplace that were used for further data science work. The team implemented an image recognition service that enables users to recognize different server options, such as fans or hard drives, by video. The image below shows the overall architecture and it’s something that could be applied to a midsized manufacturing customer.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/chart-2-2-1573155663722.jpg" alt="chart 2 (2)"></p> <p>The CTC group spent a lot of time coding, and enjoyed the opportunity to work with one another and share their knowledge. They felt that it was truly worth the time and effort. Because they found it so successful, they are hoping to hold a similar event the third week in January 2020 and are thinking of inviting select partners to participate.</p> <p>If you’re interested in learning more, connect with Florian Doerr at <a href="mailto:florian.doerr@hpe.com">florian.doerr@hpe.com</a> or <a href="https://twitter.com/florian_doerr">@florian_doerr.</a> For more opportunities to work with HPE DEV, connect with the HPE Developer Community on the <a href="https://slack.hpedev.io/">HPE DEV Slack channel.</a> And don’t forget to monitor the <a href="/blog">HPE DEV blogs</a> to hear more about what’s going on in the HPE DEV world.</p>https://developer.hpe.com/2019-November-01/https://developer.hpe.com/2019-November-01/Fri, 01 Nov 2019 05:00:00 GMThttps://developer.hpe.com/use-grommet-face-detection-and-microsoft-azure-cognitive-services-to-ide/https://developer.hpe.com/use-grommet-face-detection-and-microsoft-azure-cognitive-services-to-ide/Fri, 01 Nov 2019 02:40:45 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture1-1572576650255.png" alt="picture1"></p> <p>In this article, I am going to show you how to perform face detection and recognition using the Grommet Face Detection application and the Microsoft Azure Face API, which is available through Azure Cognitive Services. This is a fun application we have used at different events to showcase the many uses of Grommet. While fun, it also has practical applications in healthcare, banking, retail, and security industries.</p> <p>The front-end face detection application is used to fetch the image from the network camera, send it to Azure Face API, and display the facial recognition response. The Azure Cognitive Services Face API is used to recognize human faces in a picture. When you send a picture from the face detection application to the Azure Face API, it detects human faces in the image and tells you that, within a certain position, there is a face. It will even identify certain face-related attributes such as gender, estimated age, emotion, head position, facial hair, eyeglasses, and a lot more!</p> <h2>Why use the Grommet Face Detection application?</h2> <p>I built this face detection/recognition application to show how quickly you can build a responsive web app using HPE Open Source UI development and a design framework called Grommet that interacts with Microsoft Azure Cognitive Services. The face-detection scenario leverages the Azure Face API in conjunction with Node.js to extract and recognize the facial expressions and emotions of various people.</p> <p>You can use this same methodology to build your own personalized application. For instance, you can design it so that when you walk into the room, the computer recognizes you and automatically sets the lights and music per your preference. There are other use cases as well. Imagine building a natural language interactive robot. You could design it so that it looks at someone’s face and recognizes them when they walk into a store. It could determine that person’s emotions by the way he or she looks. If angry, the robot knows and reacts accordingly. If happy, perhaps it might try to sell that person more items. To show you how this all works, instead of building a complex robot, I built a simple Face Detection Web Application using HPE Grommet that interacts with Microsoft Cognitive Services. You can learn more about Grommet by reading a previous HPE DEV blog - <a href="/blog/grommet-the-glue-that-binds-development-and-design">Grommet – the Glue that Binds Development and Design.</a></p> <h2>What exactly are Microsoft Cognitive Services?</h2> <p>Microsoft Cognitive Services allow applications to hear, see, and understand the world just like you would. If you were to pause for a second and think of all the people you know, your family, your friends, your boss and so on, what would come to mind? Are you thinking of their faces? And emotions?</p> <p>Microsoft Cognitive Services provides you the power to recognize all of that in your applications. And it’s so simple to use! All you need to do is make a REST API call using your preferred programming language and you will enable your application to do amazing things. Microsoft Cognitive Services is comprised of Decision, Vision, Speech, Knowledge, Search, and Language APIs. Within each of these buckets, there are numerous capabilities. In this post, I’m going to focus on how I used the Azure Face API, which is part of the Vision service, to detect, recognize, and analyze the human faces in images.</p> <h2>Recognizing and identifying a human face</h2> <p>There are two aspects of facial recognition; the first is recognizing that there is a face within a picture and the second is identifying whose face it is. Cognitive Services allow you to easily recognize faces in a picture. You send a picture, and it tells you that at certain rectangle position, there is a face. It will even tell you things like about the face, like age, gender, expression, whether eyeglasses are worn, and a lot more!</p> <p>To identify the face in the picture using the Azure Face API, you first need to build a library (the dataset of your AI machine learning model) of people with associated faces. To do this, you first need to create a Person group. To that Person group, you add numerous images of people. For example, I could upload pictures of Hewlett Packard Enterprise (HPE) CEO Antonio Neri to the library, along with other faces. I then assign Antonio’s name to (one or more) pictures that have him in it. Once you have created your database of people, you then train the model. Training the model is an important step for your AI model to make more accurate identifications. After that, input a picture that the program has never seen before. This would be a picture of a person who is already in the library. It could also be a picture with more than one person, where one of the people in the picture is part of our library. Based on matching it against those already in the library, the face detection application will "identify" the person, as in "I think this is Antonio Neri, and my confidence is 99%."</p> <p>I must mention that given the number of steps required, I can’t show every line of code in this article. However, you can download the source code and set it up. The instructions can be found in the <strong>Set up the code</strong> section of this article.</p> <h2>Set up the Face API in Azure</h2> <p>Like many Cognitive Services APIs, to use the Face API, you need to provision it for use in the Azure Portal. Log in to the Azure portal, click on the <strong>Create a resource</strong> link, and choose the option that allows you to create a new instance of the Face API. You will be provided with a Cognitive service API endpoint and subscription keys used for every call to your deployed instance of the Azure Cognitive Services Face API as shown below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1572576983408.png" alt="picture2"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1572577002107.png" alt="picture3"></p> <p>The application will ask you to define certain parameters. Among other things, you’ll need to specify a pricing tier. For this example, the F0 subscription-level free tier is fine. Once you’ve created a resource, grab the endpoint address and the two keys. I chose to put mine into an environment file called .env inside the face detection source code folder.</p> <h2>Authenticate to the Face API</h2> <p>Calls to the Face API are simple REST calls. They can be GET/POST/DELETE or any other such common HTTP verb wrapped in a Face API call with your preferred programming language, whether it’s C#, Node.js, cURL, Go, Java, Javascript, Python, PHP, or Ruby. All the API calls must be authenticated. To authenticate the call, you need to pass in a specific HTTP header called Ocp-Apim-Subscription-Key, and then pass in either of the two keys you previously saved. The reason Microsoft Azure provides two keys is for redundancy. If one key gets compromised, you can choose to use the other while the first one is regenerated.</p> <p>I used the fetch method inside my face detection application to make REST API calls. The instructions to download the source code based on Node.js can be found in the <strong>Set up the code</strong> section of this article.</p> <h2>Face detection</h2> <p>Detection, as the name suggests, detects human faces in an image. But it can do so much more than that. When you call up the Detect method, you have a choice of requesting multiple details of the recognized faces. Of course, the more details you request, the longer the request takes. In your inputs to the detect method, you can pass in various optional request parameters like FaceID, FaceLandmarks, and FaceAtrributes ( smile, emotion, age, gender, and so on).</p> <pre><code class="language-javascript"> // adding face detect attributes const addImageParams = 'returnFaceId=true&#x26;returnFaceLandmarks=false&#x26;returnFaceAttributes=age,gender,smile,facialHair,glasses,emotion,hair,makeup,accessories,headPose'; // URI for face Detection const detectUri = `${baseUrl}/detect?${addImageParams}`; // API call to Detect human faces in an image, return face rectangles, and optionally with faceIds, // landmarks, and attributes async function fetchFaceEntries(imageData) { const blob = await dataURLtoBlob(imageData); const faceDetect = await fetch(detectUri, { method: 'POST', headers: { 'Content-Type': 'application/octet-stream', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, body: blob, }).catch(err => { console.log('err', err); }); return faceDetect; } </code></pre> <h2>Lights, camera, action!</h2> <p>Now that you have the basics, let me show you the steps required to work with the Face Detection application.</p> <h2>Step 1: Create a Person Group</h2> <p>Creating a Person Group is a matter of issuing a PUT request to your Face API endpoint URL that looks like this:</p> <p><code>https://{endpoint}/face/v1.0/persongroups/{personGroupId}</code></p> <p>The personGroupId is simply a user-provided string. Along with such a request, you need to include a request body with the following JSON object:</p> <p>{ ‘name’: personGroupId}</p> <pre><code class="language-javascript"> // URI to create a person group mentioned in .env file const personGroupUri = `${baseUrl}/persongroups/${personGroupName}?`; // create a person group async function createPersonGroup() { const personGroup = await fetch(personGroupUri, { method: 'PUT', headers: { 'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, body: JSON.stringify({ name: personGroupName, }), }).catch(err => { console.log('err', err); }); const pGroup = await personGroup.json(); return pGroup; } </code></pre> <h2>Step 2: Create a Person</h2> <p>Once you’ve created a Person Group, the next step is to add people into that Person Group. This is a simple POST request to your Face API endpoint:</p> <p><code>https://{endpoint}/face/v1.0/persongroups/{personGroupId}/persons</code></p> <p>Note that the request URL includes the personGroupId. This is how you tell the Face API which Person Group a Person belongs in. Also, you need to specify the name of the person you’re adding as a JSON object that looks like this:</p> <pre><code class="language-javascript"> { "name": personName} </code></pre> <p>The name is the display name of the target person. The REST API call returns the personID created. You will need the personID to associate a face to the person in the next step.</p> <pre><code class="language-javascript"> // URI to create a new person in a specified person group const personUri = `${baseUrl}/persongroups/${personGroupName}/persons`; // create a new person in a specified person group async function createPerson(personName) { const person = await fetch(personUri, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, body: JSON.stringify({ name: personName, }), }).catch(err => { console.log('err', err); }); const personResponse = await person.json(); return personResponse; } </code></pre> <h2>Step 3: Add a Person Face</h2> <p>Adding a Person Face (an image that contains the face of the person) is another REST call. This time it’s a POST request to the following URL:</p> <p><code>https://{endpoint}/face/v1.0/persongroups/{personGroupId}/persons/{personId}/persistedFaces</code></p> <p>As you can see from the URL, you’re posting to the /persistedFaces URL for the given person in the given person group. The next question is: How do you specify the actual file contents? There are two ways:</p> <ol> <li> <p>Either you can specify the content-type header to be application/json, and send the following JSON object in the body of the POST request: { url: ‘url_to_the_image’}</p> </li> <li> <p>Or you can specify the content-type header to be application/octet-stream and send the contents of the image.</p> </li> </ol> <pre><code class="language-javascript"> // API call to add the current face image to a specific person async function addImage(imageSrc, personId) { const addImageUrl = `${baseUrl}/persongroups/${personGroupName}/persons/${personId}/persistedFaces?`; const buff = await dataUriToBuffer(imageSrc); await fetch(`${addImageUrl}${addImageParams}`, { method: 'POST', body: buff, headers: { 'Content-Type': 'application/octet-stream', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, credentials: 'same-origin', }); } </code></pre> <h2>Step 4: Train the Model</h2> <p>Once you’ve created the Person Group, added Persons, and added Faces to the Persons, you need to train the person group before you can start asking Face API to identify people. Training the Person Group is yet another REST call. You simply issue a POST request to the following URL specifying the personGroupId of the person group you want to train. The subscription key must be passed in the request header (not shown here) as for any other Face API call:</p> <p><code>https://{endpoint}/face/v1.0/persongroups/{personGroupId}/train</code></p> <p>The training task is an asynchronous task. Training time depends on the number of person entries, and their faces in a person group. It could be several seconds to minutes. To check training status, issue another GET REST call to the following URL:</p> <p><code>https://{endpoint}/face/v1.0/persongroups/{personGroupId}/training</code></p> <p>Congratulations! With your model complete, now you can send an input picture and have the Face API identify the person.</p> <p>In my example, I called the appropriate method to create one for a person named “Pramod” in the Person Group called “hpe”, then added persons face to Person Group and trained the model.</p> <h2>Step 5: Detect or recognize the person</h2> <p>Remember, the way recognition works is that you must first detect there is a face in the picture. Here’s an example. You have to send the picture to the Face API and have the Face API detect a face in the picture. In doing so, the Face API detect method returns an identifier (the FaceID) of the detected face. This identifier is temporary and it’s only good for 48 hours. However, Person IDs are permanent.</p> <p>You send the picture to Face API by issuing a POST request to the following URL:</p> <p><code>https://{endpoint}/face/v1.0/detect[?returnFaceId][&#x26;returnFaceLandmarks][&#x26;returnFaceAttributes][&#x26;recognitionModel][&#x26;returnRecognitionModel][&#x26;detectionModel]</code></p> <p>Also, you must include the following body:</p> <pre><code class="language-javascript">{ "url": http://example.com/1.jpg //URL of input image } </code></pre> <pre><code class="language-javascript">// adding face detect attributes const addImageParams = 'returnFaceId=true&#x26;returnFaceLandmarks=false&#x26;returnFaceAttributes=age,gender,smile,facialHair,glasses,emotion,hair,makeup,accessories,headPose'; // URI for face Detection const detectUri = `${baseUrl}/detect?${addImageParams}`; // API call to Detect human faces in an image, return face rectangles, and optionally with faceIds, // landmarks, and attributes async function fetchFaceEntries(imageData) { const blob = await dataURLtoBlob(imageData); const faceDetect = await fetch(detectUri, { method: 'POST', headers: { 'Content-Type': 'application/octet-stream', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, body: blob, }).catch(err => { console.log('err', err); }); return faceDetect; } </code></pre> <h2>Step 6: Identify the person</h2> <p>Once you have the Face ID, you can identify the person from the person group with personGroupId. You do so by issuing a POST request to the following URL:</p> <p><code>https://{endpoint}/face/v1.0/identify</code></p> <p>Also, you must include the following body:</p> <pre><code class="language-javascript">{ 'personGroupId': personGroupId, "faceIds": [faceId], "maxNumOfCandidatesReturned": 1, "confidenceThreshold": 0.5 } </code></pre> <p>Note that, in the body, you specify the number of candidates you’d like to have in the result set and the minimum confidence threshold of recognition that should be met. Also, you pass in the Face ID.</p> <pre><code class="language-javascript">// URI for face Identify const identifyUril = `${baseUrl}/identify?`; // Call this after detecting faces, with face IDs, // to identify the faces from the people group async function identifyFaceFromGroup(faceIdsArray, personGroupId) { const res = await fetch(identifyUril, { method: 'POST', body: JSON.stringify({ faceIds: faceIdsArray, personGroupId, }), headers: { 'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscriptionKey, }, credentials: 'same-origin', }).catch(err => { console.log('err', err); }); return res; } </code></pre> <h2>The Face Detection system</h2> <p>In my trials using the Face Detection Application, I implemented the setup below. It consists of an EdgeLine server on which I created a Kubernetes cluster and deployed the front-end Face Detection application. It is backed by a camera application used to fetch the images from the network cameras and send it to the Azure Face API for facial recognition.</p> <p><strong>Note:</strong> You can clone the repo and run it on your laptop with NodeJS. Also, you can send any Image URL as input to the Azure Face API. I chose to get the images from network cameras and deployed the application on an EdgeLine server.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture6-1572577698549.png" alt="picture6"></p> <p>In this example, users log in to the face detection application running on the EL300 Kubernetes cluster. As soon as the user logs in, the face detection app fetches the current image from the network camera and loads the screen below.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture7-1572577723410.png" alt="picture7"></p> <p>Once you click on the Detect Face button, the image will be sent to Azure Face API and you will get the below result with rectangles covering the faces along with data on facial attributes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture9-1572577749047.png" alt="picture9"></p> <p>Since I added only one image with the name “Pramod” to the face detection library, Azure Face API was able to predict the name with 69% confidence and the rest of the faces came up as unknown with other facial attributes like gender, age, hair, emotions and so on.</p> <h2>Set up the code</h2> <p>As you can see, there are a lot of steps, and I wasn’t able to explain every single detail in this article. However, you can get the full source code for this and set it up on your computer easily. Just follow these instructions:</p> <ol> <li> <p>Ensure that you have Node.js version 8x or newer installed, as well as NPM 6 or later.</p> </li> <li> <p>Clone this GitHub repository <a href="https://github.com/reddypramod85/facedetection">https://github.com/reddypramod85/facedetection</a></p> </li> <li> <p>Provision an instance of the Face API in Azure, as explained earlier in this article.</p> </li> <li> <p>Get the endpoint and the two subscription keys for the provisioned instance of Face API.</p> </li> <li> <p>Create a .env file and place one of the keys, PersonGroupName (EX: “hpe”), and the image URL ( EX: <code>"http://localhost:5000/images/oscongroup.png"</code>) in the .env file as shown below:</p> </li> </ol> <ul> <li> <p>REACT_APP_SUBSCRIPTION_KEY= “Azure Face API subscription key”</p> </li> <li> <p>REACT_APP_PERSON_GROUP_NAME=hpe</p> </li> <li> <p>REACT_APP_CAMERA_IMAGE_URL=<code>"http://localhost:5000/images/oscongroup.png"</code></p> </li> </ul> <ol start="6"> <li>Run npm install to install the Grommet face detection application.</li> </ol> <h2>Next steps</h2> <p>There are many use cases where face detection is helpful. It can help in retail environments. For instance, imagine that you walk into a store, and a camera instantly recognizes you and sends you discount codes on your phone that you must use in the next 20 minutes. The possibilities are really endless, and this is only one of the many capabilities of Microsoft Cognitive Services. I hope to cover some of these other features in future articles. Keep a lookout on the <a href="/blog">HPE DEV blog site</a> for more articles on this subject.</p>https://developer.hpe.com/grace-hopper-celebration-spotlights-women-technologists/https://developer.hpe.com/grace-hopper-celebration-spotlights-women-technologists/Wed, 30 Oct 2019 20:53:14 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/brittany-ghc-2-1572897237615.png" alt="brittany ghc 2"></p> <p>I recently had the opportunity to attend my first Grace Hopper Celebration (GHC), the world’s largest gathering of women technologists. Co-founded by Dr. Anita Borg and Dr. Telle Whitney in 1994 and inspired by the legacy of Admiral Grace Murray Hopper, the event focuses on the research and career interests of women in computing. The experience was truly inspiring. Being surrounded by 25,000 women technologists was incredibly empowering and motivating, and for once, I felt as though I wasn’t in the minority!</p> <p>Over the course of the 3-day conference, I attended workshops, keynotes, panel discussions, and a career fair. Two keynote sessions book-ended the celebration, serving as the kickoff and closing of the conference. I felt the energy as soon as I walked into the hall. The vibe was so uplifting with music, dancing, and cheering. Women were also shouting #We will statements, like:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1572469485663.png" alt="picture2"></p> <p>#We will challenge our own goals</p> <p>#We will challenge the status quo</p> <p>#We will increase the visibility of women in tech</p> <p>#We will make Katherine Johnson as well-known as Neil Armstrong</p> <p>#We will celebrate diversity</p> <p>The first keynote featured numerous speakers and started off with Brenda Darden Wilkerson, the CEO of AnitaB.org, the organization producing this event. As an advocate for diversity, Brenda emphasized, “It is my mission to achieve 50/50 tech equity by 2025.”</p> <p>Another speaker, Ana Roca Castro, spoke about igniting the genius in children. Ana started <a href="https://www.geniusplaza.com/en/">Genius Plaza</a> to insure every child received the best education possible, no matter where that child was born or raised. Genius Plaza provides access to personalized learning content for children in arts, math, and science. Ana added to our #we will statements by exclaiming “#We will ignite the genius in every child!”</p> <p>Another inspiring woman at the conference was Dr. Fei-Fei Li. She is the inventor of <a href="http://www.image-net.org/">ImageNet,</a> a large visual database designed for visual object recognition software research. “ImageNet is an image database organized according to the <a href="https://wordnet.princeton.edu/">WordNet</a> hierarchy in which each node of the hierarchy is depicted by hundreds and thousands of images.”</p> <p>When Dr. Li helped design ImageNet in 2006, some of her colleagues tried to dissuade her by telling her the project was foolish. Yet, now the database consists of over 14 million images. The process used to create ImageNet is considered to be the foundation on which machine learning was based. One thing Dr. Li said really stuck with me. She said, “It’s okay to feel small sometimes. But together, we can be big enough to accomplish anything.”</p> <p>These were just a couple of the inspiring talks during the welcoming keynote. I would encourage everyone to listen to the recording of this <a href="https://ghc.anitab.org/">session.</a></p> <p>With over 400 sessions held during the conference, it was very difficult to choose which ones to attend. Each one I attended was very good. Some that I found the most interesting included:</p> <h2>Building a (better) Open Source Community</h2> <p>Led by Lisa Tagliaferri – Manager of Developer Education at DigitalOcean</p> <p>This session provided insights on how people could maintain their open source projects.</p> <ul> <li>Lisa covered gender and diversity as it pertained to major technology companies, as well as how it plays out in open source projects.</li> <li>She also offered tips and resources for users on open source projects, including tutorials on how to get started and troubleshooting guides.</li> <li>She explored how open source projects could do more to promote inclusivity and diversity.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture3-1572469470326.png" alt="picture3"></p> <h2>Branching Out: GitHub and Core skills for Contributions to Open Source</h2> <p>Led by Lily Sturmann, Parul Singh – Software Engineers at Red Hat, Inc.</p> <p>This session went into details about forking, cloning and other basics around contributing code.</p> <ul> <li>In this workshop, Lily and Parul walked users through creating a Pull request on an open source project. (Lucky for me, I do this almost every day, so I was able to assist the person next to me.)</li> <li>This session provided a great way to network with the people working at the same table.</li> </ul> <h2>How to develop Creativity and Need-driven Production? Using Tech-Fashion as an Example</h2> <p>Led by Kitty Yeung – Creative Technologist at Microsoft</p> <p>This session focused on incorporating technology into fashion.</p> <ul> <li>Kitty explained how she was able to create a flower that could be worn by a mother and her daughter. If the daughter started to wander off too far, the flower would vibrate and let the mother know her daughter is out range. She went into a little bit of detail explaining how she incorporated circuits into this flower.</li> <li>Kitty gave a similar talk at Hackaday Supercon, which you can view <a href="https://www.youtube.com/watch?v=KTL_1zz_cRc">here</a></li> </ul> <h2>Virtual Reality for Brain Surgeries: Enhancing Visuality &#x26; Improving Patient Outcomes</h2> <p>Led by Prachi Shah – Software Engineer, Verily Life Sciences, Google</p> <p>This session centered on an amazing surgical situation which helped transform how we look at MRI’s and CT scans, going from 2D to something that we can print in 3D and then use that to create a virtual reality that can be used to prepare surgeons.</p> <ul> <li>Prachi explained that doctors currently use 2D images to plan 3D surgeries, and that so much more could be done using technology.</li> <li>She explained how technologists are now trying to use 3D modeling pulled from CT and MRI scans and print3D models of the brain. These can be used for physical planning as well as patient education. Taking it a step further, she described the opportunity to provide visualizations in VR for neurosurgeons to use to visualize operating on a patient’s brain.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture4-1572469494011.png" alt="picture4"></p> <h2>Empowering Women to Improve Women’s Health through Tech, Education &#x26; Engagement</h2> <p>Led by Nimmi Ramanujam, Engineering &#x26; Director of Center for Global Women’s Health Tech Duke University</p> <p>The goal of this session was to promote the development of technology that can profoundly impact women’s health.</p> <ul> <li>Nimmi is passionate about empowering women to know their own health and discussed how she wanted to develop a low-cost technology that could help women and doctors detect and help prevent cervical cancer.</li> <li>I really recommend watching her <a href="https://www.youtube.com/watch?v=LePaY_Ms6_o">Social Impact Abie Award Winner Video.</a></li> </ul> <h2>Edge AI with Raspberry</h2> <p>Led by Penny Anderson, Director of Engineering, MathWorks</p> <p>In this workshop, we developed a deep-learning application which detects objects in an image. We then deployed and ran the application on a Raspberry Pi board.</p> <ul> <li>During the workshop, we divided into groups and explored the process of how the code detects objects and determines the age of a person in a specific image.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture5-1572469501148.png" alt="picture5"></p> <h2>Empower the Next Generation: Supporting K-12 Education as an Industry Professional</h2> <p>Led by Amy Liu, Crystal Hsieh Software Engineer, LinkedIn</p> <p>This session explored the lack of computer science teachers in schools today and proposed potential solutions to the problem. The speakers provided resources, like LinkedIn’s <a href="https://github.com/linkedin/high-school-trainee">High School Trainee Program,</a> which is a great open source tool that helps you get started by providing program materials.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture6-1572469513141-no-white-border-1572906184377.jpg" alt="picture6 1572469513141 no white border"></p> <p>During my time at the conference, I also learned some statistics about women in the technology workforce, including the fact that only <strong>2% of the computing workforce was made up of Hispanic women.</strong> I felt honored to be able to represent that segment and attend the Celebrating Latinas in Technical Roles Reception which was held at the event.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/imageedit_2_6801947141-1572579297293.png" alt="imageedit_2_6801947141"></p> <p>After three days of enjoying the incredibly inspiring stories about different women’s journeys in technology, I could think of no better way to end the conference than the closing keynote, which featured more Abie-award speakers, including Dr. Vivienne Ming, co-founder of Socos Labs, and Nonny De la Peña, the founder and CEO of Emblematic Group.</p> <p>Dr. Vivienne Ming explained how she enjoyed using AI to think in ways no one else has ever done before. She also explained that courage is something that you practice and share with others. Explore more of her talks <a href="https://www.youtube.com/watch?v=1lpGcWxDv98">here.</a> Nonny De la Peña, a tele-immersive journalist, discussed how she got involved in this new industry. She started recording audio in a food bank when a man ended up going into a diabetic coma. The recording showed the real power of this type of journalism. Nonny continues to create videos in which people can use virtual reality to put themselves inside a story and feel what it is like to be somewhere without actually being physically present. Some of her talks can be found in these <a href="https://www.ted.com/talks/nonny_de_la_pena_the_future_of_news_virtual_reality?language=en">Ted Talks.</a> This final keynote was truly inspiring and you can view it <a href="https://ghc.anitab.org/">here.</a></p> <h2>HPE Women</h2> <p>Finally, I benefitted from the chance to meet and get to know some of the other inspiring women who also work at HPE. I loved hearing stories from women around the world about their journey and path into the tech industry. I was overwhelmed by the fact that women I had just met reached out to let me know they were proud of me. Each of our stories and journeys was remarkable and the conference provided such a great opportunity to share them with one another.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture8-1572469528275.png" alt="picture8"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture9-1572469543131.png" alt="picture9"></p>https://developer.hpe.com/contributors-enhance-grommet-during-hacktoberfest/https://developer.hpe.com/contributors-enhance-grommet-during-hacktoberfest/Mon, 28 Oct 2019 22:31:02 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/picture2-1572302279782.png" alt="picture2"></p> <p>Because of Hacktoberfest, the <a href="https://github.com/grommet/grommet">Grommet GitHub repository</a> has been as lively as a Biergarten in Germany these past few weeks. For those of you who don’t know, Hacktoberfest is a month-long celebration aimed at encouraging coders of all backgrounds and skill levels worldwide to contribute to open source software. In only the first sixteen days, coders from all over submitted 54 commits to the Grommet code base. With the help of seventeen Hacktoberfest rockstar contributors, we got to a point where 93% of our storybook examples are React hook-friendly and more than a dozen Grommet components had been refactored to use hooks!</p> <p>The issue that had the most traffic was Refactoring components and stories to use hooks. The ability to use hooks has recently been added to React and is proving to be very popular because it solves a number of issues. Hooks are functions that let you “hook into” React state and lifecycle features from function components. Hooks don’t work inside classes — they let you use React without classes. This issue alone attracted more than 50 pull requests (PRs) and helps make Grommet a React framework that is super edgy.</p> <p>Developers also tackled issues of <a href="/blog/using-typescript-in-grommet-applications">TypeScript,</a> layout, and the implementation of best practices and standards. Some were bug fixes, but many more were enhancements to the code. You can find all the issues on the <a href="https://github.com/grommet/grommet/issues?utf8=%E2%9C%93&#x26;q=+label%3Ahacktoberfest">Grommet Hacktoberfest repository.</a> As the HPE Experience Studio senior engineer and open source developer, I couldn’t be happier with the engagement we received around this October event run by <a href="https://www.digitalocean.com/">DigitalOcean</a> and <a href="https://dev.to/">DEV design.</a></p> <p>For those of you unfamiliar with <a href="https://v2.grommet.io/">Grommet,</a> it is a React-based library of reusable UI components that help developers and designers create web applications. This open source UI development and design tool simplifies the way web applications are built by providing a package of commonly used interface elements from which developers and designers can choose to use. To keep up to date with everything that’s going on with Grommet, make sure you connect with us on <a href="https://grommet.slack.com/">Slack.</a> Or, check out the HPE DEV portal to learn more about the <a href="https://developer.hpe.com/platform/grommet/home">Grommet platform.</a></p>https://developer.hpe.com/mingle-with-microsoft-experts-at-microsoft-ignite/https://developer.hpe.com/mingle-with-microsoft-experts-at-microsoft-ignite/Wed, 23 Oct 2019 15:45:39 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/10/screen-shot-2019-10-23-at-101435-am-1571847321995.png" alt="screen shot 2019 10 23 at 10.14.35 am"></p> <p>HPE DEV is gearing up for <a href="https://www.microsoft.com/en-us/ignite">Microsoft Ignite 2019</a>, an annual conference for developers and IT professionals hosted by Microsoft where Hewlett Packard Enterprise (HPE) will be showcasing its capabilities around hybrid cloud solutions. Under the tagline <a href="https://www.hpe.com/us/en/alliance/microsoft/ignite2019.html">Fear No Cloud with HPE at Microsoft Ignite 2019,</a> HPE will demonstrate how to enable the consumption of Microsoft Azure as-a-Service with HPE GreenLake, unlock insights from data with SQL Server 2019, and run Azure services at the Edge with HPE Edgeline Converged Edge Systems. As evidenced by the <a href="https://myignite.techcommunity.microsoft.com/sessions">schedule,</a> many exciting sessions are planned for this November 4-8 event in Orlando, Florida.</p> <p>Microsoft Ignite 2019 attendees will have the opportunity to learn from 50+ HPE Microsoft experts with over 750 years of combined Microsoft experience; <a href="https://www.hpe.com/us/en/alliance/microsoft/ignite2019.html#booth">attend 60+ booth theater sessions delivered</a> by HPE, Microsoft, and partner experts; and explore <a href="https://www.hpe.com/us/en/alliance/microsoft/ignite2019.html#demos">Hybrid Cloud, Intelligent Edge and SQL solutions demos.</a> All this can be found in booth #2549 in the partner showcase. Customers and partners will also be able to engage with HPE experts in two HPE Microsoft Ignite breakout sessions held at the HUB showroom, theater number 7, Tuesday, November 5th. For specifics on dozens of booth sessions, technical breakout sessions, workshops, and partner panel luncheons, visit the <a href="https://www.hpe.com/us/en/alliance/microsoft/ignite2019.html">HPE/Microsoft Alliance page</a> and download the session flyer.</p> <p>Numerous opportunities will be available for attendees to enjoy food and drink, and mingle with industry experts to discover answers to today’s biggest business challenges. You’re invited to attend the Edge and Cloud luncheon panel sessions and HPE Microsoft Ignite Expert Happy Hours. Make sure you stop by HPE booth #2549 to learn more about HPE/Microsoft solutions, as well as other key DevOps contributions and the value offered through the <a href="https://developer.hpe.com/community">HPE DEV community.</a> Hope to see you at the event!</p>https://developer.hpe.com/hpe-dev-heads-to-kubecon-cloudnativecon-san-diego/https://developer.hpe.com/hpe-dev-heads-to-kubecon-cloudnativecon-san-diego/Wed, 09 Oct 2019 15:45:53 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/kubecon2-1570637685840.png" alt="kubecon2"></p> <p>HPE DEV is gearing up for <a href="https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/">KubeCon | CloudNativeCon North America,</a> a major open source developer conference focused on furthering the education and advancement of Kubernetes and cloud native computing. As evidenced by the <a href="https://kccncna19.sched.com/">schedule,</a> a ton of exciting sessions are planned for this November 18-21 event in San Diego, California.</p> <p>A cross-organizational team from Hewlett Packard Enterprise (HPE) will staff the booth (P28) aimed at promoting and educating attendees on what HPE offers in the areas of containers and Kubernetes (k8s). Attendees will get the opportunity to talk with representatives from HPE DEV, HPE Storage and other HPE business areas.</p> <p>BlueData, a leading provider of container-based software solutions that HPE recently acquired to accelerate AI and data-driven innovation in the enterprise, will also be there. Tom Phelan, Chief Architect and co-founder of BlueData, <a href="https://twitter.com/tapbluedata/status/1169810013851795456?s=20">recently expressed his excitement</a> in being selected, along with his colleague, Joel Baxter, to talk about the <a href="/blog/complex-stateful-applications-on-kubernetes-kubedirector-version-02">KubeDirector</a> open source project at the event.</p> <p>Make sure you stop by the HPE booth (P28) to learn more about what HPE is doing in this exciting area, as well as other key DevOps contributions and value offered through the <a href="https://developer.hpe.com/community">HPE DEV community.</a> Looking forward to seeing you at the event!</p>https://developer.hpe.com/devops-and-its-impact-on-project-management/https://developer.hpe.com/devops-and-its-impact-on-project-management/Mon, 07 Oct 2019 20:18:47 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/hpe20160726048-1570557978016.jpg" alt="hpe20160726048"></p> <p>As a project manager, a great deal of my time is spent on driving efficiency within the Hewlett Packard Enterprise (HPE) Chief Design Office. I’ve been lucky enough to spend most of my career in an age when DevOps is highlighted as a superior method for software development. It’s also exciting to see the DevOps philosophy gain momentum and expand into other organizations. Many teams now recognize the benefits of agile methodologies, and continuously work to streamline processes, accelerate feedback loops, and quicken the pace of innovation.</p> <p>Since I live the DevOps methodology every day, I have to remind myself that not everyone comes at their work with this mindset. I recently came across a blog that really hit home for me. This was an article written by Angel Rivera and entitled <a href="https://circleci.com/blog/devops-did-not-exist/">DevOps didn’t exist when I started as a developer: How this one principle changed my career.</a> Angel has a unique take on the subject, which follows along a timeline from the mid-1990’s till now and relates how much things have changed. The author helps the reader understand that others, depending on how long they may have worked in this industry, may still be steeped in historical approaches to dealing with application development, and how important open conversations can be to overcoming this bias.</p> <p>As Angel points out, DevOps is more than just a set of software development practices that combine software development and IT operations in a way that shortens the development and release cycle. It’s a way of doing business based on a firm foundation of trust and cooperation. It is not easily achieved and is highly based on culture. One must nurture a DevOps culture for it to take hold and thrive.</p> <p>I like to think the work we are doing here in the Chief Design Office with HPE DEV helps continue to break down silos as we work across different organizations, both internally and outside of Hewlett Packard Enterprise, to help everyone become a little more agile. I would love to hear your thoughts. Connect with us on <a href="https://slack.hpedev.io/">Slack</a> and share your stories on what DevOps means to you.</p>https://developer.hpe.com/2019-October-04/https://developer.hpe.com/2019-October-04/Fri, 04 Oct 2019 05:00:00 GMThttps://developer.hpe.com/hacktoberfest-2019-help-grommet-and-win-a-free-t-shirt-in-the-process/https://developer.hpe.com/hacktoberfest-2019-help-grommet-and-win-a-free-t-shirt-in-the-process/Mon, 30 Sep 2019 16:06:29 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1569859873855.png" alt="picture1"></p> <p>Are you a developer looking to help drive the growth of open source software and make positive contributions to the ever-growing Grommet community? Are you a fan of <a href="https://v2.grommet.io/">Grommet</a> and have ideas on some features you’d like to see? Well, here’s your chance to contribute and get a free t-shirt in the process!</p> <p>Hacktoberfest is a month-long celebration run by <a href="https://www.digitalocean.com/">DigitalOcean</a> and <a href="https://dev.to/">DEV design</a> aimed at encouraging coders worldwide to contribute to open source software. It is open to all backgrounds and skill levels, from experienced developers to students just learning to code.</p> <p>To participate, all you have to do is sign up anytime between October 1 and October 31 and submit four <a href="https://help.github.com/en/articles/about-pull-requests">pull requests</a> to any public GitHub repositories. To contribute to Grommet, access any of the Grommet issues in the Hacktoberfest challenge by going to the <a href="https://github.com/grommet/grommet/labels/hacktoberfest">Grommet GitHub Repo.</a></p> <p>For specifics on the event, including rules and details on how to get your t-shirt, go to the <a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest 2019 home page.</a></p>https://developer.hpe.com/is-continuous-deployment-of-modern-web-applications-in-microsoft-azure-r/https://developer.hpe.com/is-continuous-deployment-of-modern-web-applications-in-microsoft-azure-r/Thu, 26 Sep 2019 16:24:18 GMT<p>In my studies on Microsoft Azure fundamentals, I often need to dive deep into some of the many Azure services available, all the way from Infrastructure-as-a-Service (IaaS) functionalities (compute, networking, storage, firewall, load balancer) to advanced Platform-as-a-Service (PaaS) services. I recently explored how to implement a Continuous Integration/Continuous Delivery (CI/CD) pipeline with the Microsoft Azure PaaS <a href="https://docs.microsoft.com/en-us/azure/app-service/overview">App Service,</a> a fully-managed web hosting platform used to host any web-based application supported by Azure.</p> <p>CI/CD is used by software developers to build, test, integrate, deliver, and deploy software rapidly, reliably, and repeatedly with minimal human intervention. Given the increased emphasis on developers to set up CI/CD pipelines, I wanted to assess how difficult this would be to do in this environment.</p> <p>I created a Web App resource (an instance of Azure App Service) inside my Azure subscription and tried to set up a CI/CD pipeline from the popular DevOps source code version control system, GitHub. It turned out that it was so simple and fast I wanted to show you how to use the Azure App Service and one of its built-in pipeline orchestrators, <a href="https://github.com/projectkudu/kudu/wiki">Kudu.</a> I will share with you a method similar to what my colleague, Maddu Rebanna, shared in his blog article, <a href="/blog/deploy-a-full-stack-application-on-netlify-that-includes-a-cicd-pipeline">Deploy a full stack application on Netlify that includes a CI/CD pipeline.</a></p> <p>Note: Azure offers a free App Service plan SKU (Stock Keeping Unit). It is a great option for developers to quickly deploy, host, and test modern web applications developed in a variety of languages (.NET, .NET Core, Java, Ruby, Node.js, PHP, or Python) at no cost, while enabling continuous deployment from GitHub. If you don’t already have a free Azure account, go ahead and sign up <a href="https://azure.microsoft.com/en-us/free/">here.</a></p> <h2>THE BASIC STEPS REQUIRED</h2> <p>Azure App Service is a PaaS that provides the complete platform, both hardware (compute resources) and software (OS and runtime stack), on which cloud applications run. This means that you, as developer, just have to focus on designing and developing your web applications.</p> <p>Once you have an account in GitHub and an account in Azure public cloud, you only need to follow three basic steps to deploy and publish a web application using continuous deployment. These steps are:</p> <ol> <li> <p>Create a <em>Web App</em> resource in Azure. Creating a Web App allocates a set of hosting resources within the Azure App Service platform on which your web application will run. For this example, I suggest you use a free dev/test hosting App Service plan so as to not incur any cost for your Azure account. The App Service plan SKU is always a free service, even after the one-year trial period is over.</p> </li> <li> <p>Authorize your Azure account to connect to your GitHub repository. (This only needs to be done if it is the first time you have established a delivery pipeline between Azure and your GitHub account.)</p> </li> <li> <p>Connect your GitHub application build repository and branch to the Web App resource you just deployed in your Azure account in order to setup the delivery pipeline.</p> </li> </ol> <p>I used Visual Studio and an ASP.NET Core web application template available in Visual Studio to create a sample web application named AlpineSkiHouse and copied the Visual Studio generated source repo to my organization’s GitHub in a public repository.</p> <h2>STEP BY STEP GUIDE</h2> <p>Now, from the Azure portal, let’s take a closer look at these three simple steps:</p> <p>STEP 1:</p> <p>Create an ASP.NET Web Application in an existing resource group (CICDRg1), specifying a free SKU (Free F1) App Service Plan to host the web application:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1569515900610.png" alt="picture1"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture2-1569515891835.png" alt="picture2"></p> <p>Azure App service gives you a custom URL for your published Web App resource under domain azurewebsites.net. For a production environment, a company would have to use a paid tier App Service Plan to map their <a href="https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-custom-domain">custom DNS domain</a> name to their Azure Web App. For this test, you can use the default azurewebsites.net domain.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture3-1569515886048.png" alt="picture3"></p> <p>STEP 2:</p> <p>Next, select the <strong>Deployment Center</strong> in the left menu of the Web App resource you just created. Select <strong>GitHub</strong> and follow the authorization prompts to sign in to your GitHub account. This will authorize the Azure App Service to make the connection to your GitHub account using OAuth, an open-standard authorization protocol. You only need to authorize once with GitHub source control service.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture4-1569515878902.png" alt="picture4"></p> <p>STEP 3:</p> <p>Then, select <strong>App Service Build Service,</strong> the built-in pipeline provider Kudu, and specify the GitHub location (organization, repository and branch) of your code to setup the delivery pipeline by clicking on Continue.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture5-1569515872332.png" alt="picture5"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture6-1569515866508.png" alt="picture6"></p> <p>Finally, click Finish to confirm the setup of the delivery pipeline for your Azure Web App.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture7-1569515859003.png" alt="picture7"></p> <p>Once your web application build code repository is connected to your Azure Web App, Azure App Service build service provider (Kudu) does the rest for you. It auto-syncs code from the deployment source (GitHub) and executes a series of steps to build and get your application in a runnable state. Kudu will also auto-sync any future committed changes on the code into the Web App hosted in Azure App Service platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture8-1569515852645.png" alt="picture8"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture9-1569515845266.png" alt="picture9"></p> <p>After a minute or so, you will see that your web application is deployed and published in Azure App Service. Click on the URL to access your web application from your browser.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture10-1569515838569.png" alt="picture10"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture11-1569515830186.png" alt="picture11"></p> <p><strong>It’s that easy!!!</strong></p> <p>We just explored one of the many methods you can use to set up a CI/CD pipeline and enable continuous deployment in Azure. I hope you will find this blog article helpful to quickly and easily deploy and test your web applications through the Azure App Service built-in pipeline orchestrator, Kudu. The <a href="https://docs.microsoft.com/en-us/azure/app-service/">Azure App Service documentation</a> will provide you with all of the information you need to jumpstart your knowledge of Azure App Service and Azure continuous deployment services, such as Kudu build service or Azure DevOps pipelines.</p> <p>Please remember to follow our <a href="/blog">HPE DEV blog</a> posts for more information on this and other topics designed to streamline your application development environment.</p>https://developer.hpe.com/hpesuse-reprise-their-redfish-workshop-in-lyon-france-in-october/https://developer.hpe.com/hpesuse-reprise-their-redfish-workshop-in-lyon-france-in-october/Wed, 25 Sep 2019 15:21:20 GMT<p>HPE Open Source and Linux Technology Strategist, Bruno Cornec will once again be a speaker at the <a href="https://events.linuxfoundation.org/events/open-source-summit-europe-2019/">upcoming Open Source Summit</a> in Lyon, October 28th – 30th, 2019. His hands-on lab will help attendees learn more about Docker and how to use containers. As he did at the previous summit held in <a href="/blog/redfish-workshop-at-the-open-source-summit-na-2019">San Diego last August,</a> Bruno will also act as the coordinator for the co-located HPE+SUSE Redfish Workshop.</p> <p>This <a href="http://trac.project-builder.org/wiki/RedfishWSEurope2019">Redfish Workshop</a> will take place immediately after the summit on Thursday, October 31, 2019 from 9:00am – 5:00pm at the Lyon Convention Centre. At the event, system administrators, architects, and developers can see live demos, learn how to use Redfish, and interact with <a href="https://en.wikipedia.org/wiki/Redfish_(specification)">Redfish</a> Project technical experts.</p> <p>System and software engineers, developers, and administrators can all benefit from knowledgeable speakers who will point out the benefits derived from a standard management layer that can be used to deploy, configure, and manage an environment. Other topics will also be covered, including the use of DMTF tools for system configuration and the REST API to perform Redfish operations from Python. Attendees will get the chance to practice the concepts being presented through interactive sessions, demos, and labs.</p> <p>This is a free event, sponsored by Hewlett Packard Enterprise (HPE) and SUSE.</p> <p>For more information about the event, please visit the <a href="http://trac.project-builder.org/wiki/RedfishWSEurope2019">Redfish Workshop page.</a></p> <p><strong>IMPORTANT:</strong> A limited number of seats are available for this event, so please <a href="https://framaforms.org/redfish-workshop-oss-europe-2019-registration-form-1567095132">register</a> as soon as you can.</p> <p>If you have expertise you would like to share during this workshop, please email <a href="mailto:Bruno.Cornec@hpe.com">Bruno.Cornec@hpe.com</a>.</p>https://developer.hpe.com/2019-September-13/https://developer.hpe.com/2019-September-13/Fri, 13 Sep 2019 05:00:00 GMThttps://developer.hpe.com/using-typescript-in-grommet-applications/https://developer.hpe.com/using-typescript-in-grommet-applications/Mon, 09 Sep 2019 18:08:08 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/typescriptimage-1568052628959.jpeg" alt="typescriptimage"></p> <p>Recently, I spent time learning TypeScript, a typed superset of JavaScript that offers optional static type-checking along with the latest ECMAScript features. TypeScript can be pretty painful to wrap your head around, given the breadth of its capabilities. But I found that TypeScript has many benefits. For instance, declaring types as you write your application, whether they are a number, string, array, or function, can save developers time by catching problems early in the development process. And, if you are maintaining a project with others using TypeScript, it will cut down on runtime because you are able to surface bugs during compilation. In this tutorial, I will introduce you to how to get started with TypeScript within the confines of a React and Grommet project.</p> <h2>Step 1</h2> <p>To get started, you might want to check out another blog in which Ian Bovard does a great job going step-by-step in helping you install Grommet into your Create React application - <a href="/blog/using-your-first-grommet-component-with-create-react-app">Using Your First Grommet Component with Create-React-App.</a> You can add TypeScript to any of your projects, at any point, if this is something you’d like to try.</p> <h2>Step 2</h2> <p>Assuming you already have a Grommet app, to add TypeScript to your Grommet application, all you need to do is cd into that project and run the following command.</p> <pre><code class="language-markdown">yarn add typescript @types/node @types/react @types/react-dom @types/jest </code></pre> <p>Or, if you are using npm, then use this instead:</p> <pre><code class="language-markdown">npm install –save typescript @types/node @types/react @types/react-dom @types/jest </code></pre> <h2>Step 3</h2> <p>Once you’ve added TypeScript to the Grommet application, you can pick up where Ian talks about the App.js in regards to the Heading component.</p> <p>As you go through your files, change the App.js to App.tsx. This will trigger the use of TypeScript.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/screen-shot-2019-09-09-at-122230-pm-1568053417737.png" alt="screen shot 2019 09 09 at 12.22.30 pm"></p> <p>Following my own suggestion for this example, I will start with the Heading component that was used in <a href="/blog/using-your-first-grommet-component-with-create-react-app">Ian’s blog:</a></p> <pre><code class="language-markdown">import React from "react"; import { Grommet, grommet, Heading } from "grommet"; function App() { return ( &#x3C;Grommet className="App" theme={grommet}> &#x3C;Heading level='1'>TypeScript Rocks&#x3C;/Heading> &#x3C;/Grommet> ); } export default App; </code></pre> <p>As shown in Ian’s blog, each component has its own props. In the Grommet project, these props have been declared in their own TypeScript files to define the types that should be accepted by each prop.</p> <p>Let’s try and break a few things to make sure that the types are being checked correctly. The Heading component has various props you can manipulate, including things like size, color, level, and margin. You can change these props to fit your specific design requirements. In this example, we will change the level prop of the Heading component. It accepts a number from 1-6 or a string containing a number 1-6. If we were to use a string that passed in the word “small”, this should give us a TypeScript error.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/screen-shot-2019-09-09-at-121650-pm-1568053104287.png" alt="screen shot 2019 09 09 at 12.16.50 pm"></p> <p>The great thing about TypeScript is that when you have an error, you can hover over it and it will explain the type options that can be accepted, as well as what was given. In this example, we received an error that stated</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/screen-shot-2019-09-12-at-93544-am-1568737040433.png" alt="screen shot 2019 09 12 at 9.35.44 am"></p> <p>As you can see, the error was very clear in stating what prop types were expected and what was given. This shows what broke the project, so the level prop can be changed back to “1”. Once this is changed, you will see the error disappear without having to debug the whole application!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/screen-shot-2019-09-09-at-122738-pm-1568053721635.png" alt="screen shot 2019 09 09 at 12.27.38 pm"></p> <p>See what happens after I’ve made the change? Yay! The error has disappeared!</p> <p>Through this very simple TypeScript error example, you can see how using TypeScript can save a developer a lot of time debugging. Imagine if someone was not using TypeScript in their project. If the string “small” was passed to the Heading level prop, it would not have immediately been recognized as an error. Although you can do a prop check using <a href="https://reactjs.org/docs/typechecking-with-proptypes.html">PropTypes</a> in React, errors are only caught at runtime. Using TypeScript allows you to see the error while you’re developing an application, catching them as you compile. However, it is good practice to use both PropTypes as well as TypeScript to check for errors.</p> <p>Using TypeScript is particularly advantageous when you are onboarding someone new onto a project. The team can rest assured that no matter what new code is added, the new developer will need to follow the established types, making it harder to introduce new mistakes in a code base the developer may not be as familiar with.</p> <p>For more help, each component in Grommet has an index.d.ts file that contains all of the prop types for each specific component. Another easy way to quickly interact with Grommet and TypeScript can be found in the <a href="https://codesandbox.io/s/grommet-ts-ugq2y">Grommet sandbox</a> the team put together to help developers who are using TypeScript.</p> <p>Lately, more and more users have been incorporating TypeScript into their Grommet projects because it helps them find errors faster and results in cleaner code. The team continuously tries to make it easier for TypeScript users to use Grommet. If you run into any problems with what I described above or have any additional suggestions, please check out the resources listed below.</p> <p>Resources:</p> <ul> <li> <p><a href="https://github.com/grommet/grommet">Grommet Github</a></p> </li> <li> <p><a href="https://slackin.grommet.io/">Grommet Slack</a></p> </li> <li> <p>Once you have joined our Grommet Slack be sure to check out the #typescript channel</p> </li> </ul>https://developer.hpe.com/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern/https://developer.hpe.com/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubern/Mon, 09 Sep 2019 17:42:49 GMT<p>KubeDirector is an open source project designed to make it easy to run complex stateful scale-out application clusters on Kubernetes. KubeDirector is built using the custom resource definition (CRD) framework and leverages the native Kubernetes API extensions and design philosophy. This enables transparent integration with Kubernetes user/resource management as well as existing clients and tools.</p> <p>We recently <a href="https://medium.com/@thomas_phelan/operation-stateful-introducing-bluek8s-and-kubernetes-director-aa204952f619/">introduced the KubeDirector project</a>, as part of a broader open source Kubernetes initiative we call BlueK8s. I’m happy to announce that the pre-alpha code for <a href="https://github.com/bluek8s/kubedirector/">KubeDirector</a> is now available. And in this blog post, I’ll show how it works.</p> <p>KubeDirector provides the following capabilities:</p> <ul> <li>The ability to run non-cloud native stateful applications on Kubernetes without modifying the code. In other words, it’s not necessary to decompose these existing applications to fit a microservices design pattern.</li> <li>Native support for preserving application-specific configuration and state.</li> <li>An application-agnostic deployment pattern, minimizing the time to onboard new stateful applications to Kubernetes.</li> </ul> <p>KubeDirector enables data scientists familiar with data-intensive distributed applications such as Hadoop, Spark, Cassandra, TensorFlow, Caffe2, etc. to run these applications on Kubernetes – with a minimal learning curve and no need to write GO code. The applications controlled by KubeDirector are defined by some basic metadata and an associated package of configuration artifacts. The application metadata is referred to as a KubeDirectorApp resource.</p> <p>To understand the components of KubeDirector, clone the repository on <a href="https://github.com/bluek8s/kubedirector/">GitHub</a> using a command similar to:</p> <pre><code class="language-bash">git clone http://&#x3C;userid>@github.com/bluek8s/kubedirector. </code></pre> <p>The KubeDirectorApp definition for the Spark 2.2.1 application is located in the file <code>kubedirector/deploy/example_catalog/cr-app-spark221e2.json</code>.</p> <pre><code class="language-markdown"> ~> cat kubedirector/deploy/example_catalog/cr-app-spark221e2.json { "apiVersion": "kubedirector.bluedata.io/v1alpha1", "kind": "KubeDirectorApp", "metadata": { "name" : "spark221e2" }, "spec" : { "systemctlMounts": true, "config": { "node_services": [ { "service_ids": [ "ssh", "spark", "spark_master", "spark_worker" ], … </code></pre> <p>The configuration of an application cluster is referred to as a KubeDirectorCluster resource. The KubeDirectorCluster definition for a sample Spark 2.2.1 cluster is located in the file <code>kubedirector/deploy/example_clusters/cr-cluster-spark221.e1.yaml</code>.</p> <pre><code class="language-markdown">~> cat kubedirector/deploy/example_clusters/cr-cluster-spark221.e1.yaml apiVersion: "kubedirector.bluedata.io/v1alpha1" kind: "KubeDirectorCluster" metadata: name: "spark221e2" spec: app: spark221e2 roles: - name: controller replicas: 1 resources: requests: memory: "4Gi" cpu: "2" limits: memory: "4Gi" cpu: "2" - name: worker replicas: 2 resources: requests: memory: "4Gi" cpu: "2" limits: memory: "4Gi" cpu: "2" - name: jupyter … </code></pre> <h2>Running Spark on Kubernetes with KubeDirector</h2> <p>With KubeDirector, it’s easy to run Spark clusters on Kubernetes.</p> <p>First, verify that Kubernetes (version 1.9 or later) is running, using the command <code>kubectl version</code></p> <pre><code class="language-markdown">~> kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} </code></pre> <p>Deploy the KubeDirector service and the example KubeDirectorApp resource definitions with the commands:</p> <pre><code class="language-bash">cd kubedirector make deploy </code></pre> <p>These will start the KubeDirector pod:</p> <pre><code class="language-markdown">~> kubectl get pods NAME READY STATUS RESTARTS AGE kubedirector-58cf59869-qd9hb 1/1 Running 0 1m </code></pre> <p>List the installed KubeDirector applications with <code>kubectl get KubeDirectorApp</code></p> <pre><code class="language-markdown">~> kubectl get KubeDirectorApp NAME AGE cassandra311 30m spark211up 30m spark221e2 30m </code></pre> <p>Now you can launch a Spark 2.2.1 cluster using the example KubeDirectorCluster file and the <code>kubectl create -f deploy/example_clusters/cr-cluster-spark211up.yaml</code> command. Verify that the Spark cluster has been started:</p> <pre><code class="language-markdown">~> kubectl get pods NAME READY STATUS RESTARTS AGE kubedirector-58cf59869-djdwl 1/1 Running 0 19m spark221e2-controller-zbg4d-0 1/1 Running 0 23m spark221e2-jupyter-2km7q-0 1/1 Running 0 23m spark221e2-worker-4gzbz-0 1/1 Running 0 23m spark221e2-worker-4gzbz-1 1/1 Running 0 23m </code></pre> <p>The running services now include the Spark services:</p> <pre><code class="language-markdown">~> kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubedirector ClusterIP 10.98.234.194 &#x3C;none> 60000/TCP 1d kubernetes ClusterIP 10.96.0.1 &#x3C;none> 443/TCP 1d svc-spark221e2-5tg48 ClusterIP None &#x3C;none> 8888/TCP 21s svc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 &#x3C;none> 22:30534/TCP,8080:31533/TCP,7077:32506/TCP,8081:32099/TCP 20s svc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 &#x3C;none> 22:30632/TCP,8888:30355/TCP 20s svc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 &#x3C;none> 22:30358/TCP,8081:32144/TCP 20s svc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 &#x3C;none> 22:30294/TCP,8081:31436/TCP 20s </code></pre> <p>Pointing the browser at port 31533 connects to the Spark Master UI:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/kubedirector-1568051725410.png" alt="kubedirector"></p> <p>That’s all there is to it! In fact, in the example above we also deployed a Jupyter notebook along with the Spark cluster.</p> <p>To start another application (e.g. Cassandra), just specify another KubeDirectorApp file:</p> <pre><code class="language-markdown">kubectl create -f deploy/example_clusters/cr-cluster-cassandra311.yaml </code></pre> <p>See the running Cassandra cluster:</p> <pre><code class="language-markdown">~> kubectl get pods NAME READY STATUS RESTARTS AGE cassandra311-seed-v24r6-0 1/1 Running 0 1m cassandra311-seed-v24r6-1 1/1 Running 0 1m cassandra311-worker-rqrhl-0 1/1 Running 0 1m cassandra311-worker-rqrhl-1 1/1 Running 0 1m kubedirector-58cf59869-djdwl 1/1 Running 0 1d spark221e2-controller-tq8d6-0 1/1 Running 0 22m spark221e2-jupyter-6989v-0 1/1 Running 0 22m spark221e2-worker-d9892-0 1/1 Running 0 22m spark221e2-worker-d9892-1 1/1 Running 0 22m </code></pre> <p>Now you have a Spark cluster (with a Jupyter notebook) and a Cassandra cluster running on Kubernetes. Use <code>kubectl get service</code> to see the set of services.</p> <pre><code class="language-markdown">~> kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubedirector ClusterIP 10.98.234.194 &#x3C;none> 60000/TCP 1d kubernetes ClusterIP 10.96.0.1 &#x3C;none> 443/TCP 1d svc-cassandra311-seed-v24r6-0 NodePort 10.96.94.204 &#x3C;none> 22:31131/TCP,9042:30739/TCP 3m svc-cassandra311-seed-v24r6-1 NodePort 10.106.144.52 &#x3C;none> 22:30373/TCP,9042:32662/TCP 3m svc-cassandra311-vhh29 ClusterIP None &#x3C;none> 8888/TCP 3m svc-cassandra311-worker-rqrhl-0 NodePort 10.109.61.194 &#x3C;none> 22:31832/TCP,9042:31962/TCP 3m svc-cassandra311-worker-rqrhl-1 NodePort 10.97.147.131 &#x3C;none> 22:31454/TCP,9042:31170/TCP 3m svc-spark221e2-5tg48 ClusterIP None &#x3C;none> 8888/TCP 24m svc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 &#x3C;none> 22:30534/TCP,8080:31533/TCP,7077:32506/TCP,8081:32099/TCP 24m svc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 &#x3C;none> 22:30632/TCP,8888:30355/TCP 24m svc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 &#x3C;none> 22:30358/TCP,8081:32144/TCP 24m svc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 &#x3C;none> 22:30294/TCP,8081:31436/TCP 24m </code></pre> <h2>Get Involved</h2> <p>KubeDirector is a fully open source, Apache v2 licensed, project – the first of multiple open source projects within a broader initiative we call BlueK8s. The pre-alpha code for KubeDirector has just been released and we would love for you to join the growing community of developers, contributors, and adopters. Follow <a href="https://twitter.com/BlueK8s/">@BlueK8s</a> on Twitter and get involved through these channels:</p> <ul> <li>KubeDirector <a href="https://join.slack.com/t/bluek8s/shared_invite/enQtNDUwMzkwODY5OTM4LTRhYmRmZmE4YzY3OGUzMjA1NDg0MDVhNDQ2MGNkYjRhM2RlMDNjMTI1NDQyMjAzZGVlMDFkNThkNGFjZGZjMGY/">chat room on Slack</a></li> <li>KubeDirector <a href="https://github.com/bluek8s/kubedirector/">GitHub repo</a></li> </ul>https://developer.hpe.com/deploying-complex-stateful-applications-on-kubernetes-with-kubedirector/https://developer.hpe.com/deploying-complex-stateful-applications-on-kubernetes-with-kubedirector/Mon, 09 Sep 2019 17:36:33 GMT<p>Kubernetes is clearly the container orchestrator of choice for cloud-native stateless applications. And with StatefulSets and Persistent Volumes, it’s now possible to run stateful applications on Kubernetes. Tools like Kustomize, Helm, and Kubeflow help tackle some of the deployment complexity for stateful applications. However, running complex stateful applications for distributed AI, machine learning, and big data analytics on Kubernetes remains beyond the reach of most users.</p> <p>Enter KubeDirector. KubeDirector is an open source Apache project that uses the standard Kubernetes custom resource functionality and API extensions to deploy and manage complex stateful scale-out application clusters. With KubeDirector, you can run complex stateful clusters for AI, machine learning, and big data analytics on Kubernetes without writing a single line of Go code.</p> <p><a href="https://www.youtube.com/watch?v=X2kEk5wLe9g">This webinar</a> will provide an overview of the KubeDirector architecture, show how to author the metadata and artifacts required for an example stateful application (e.g. with Spark, Jupyter, and Cassandra), and demonstrate the deployment and management of the cluster on Kubernetes using KubeDirector.</p>https://developer.hpe.com/complex-stateful-applications-on-kubernetes-kubedirector-version-02/https://developer.hpe.com/complex-stateful-applications-on-kubernetes-kubedirector-version-02/Mon, 09 Sep 2019 17:26:49 GMT<p>Last summer, <a href="https://www.bluedata.com/blog/2018/07/operation-stateful-bluek8s-and-kubernetes-director/">I wrote here about our BlueK8s initiative and a new open source project</a> for deploying and managing complex stateful scale-out applications on Kubernetes: <strong>KubeDirector.</strong> KubeDirector enables data scientists familiar with data-intensive distributed applications such as Hadoop, Spark, Cassandra, TensorFlow, Caffe2, etc. to easily run these applications on Kubernetes.</p> <p>In <a href="https://kubernetes.io/blog/2018/10/03/kubedirector-the-easy-way-to-run-complex-stateful-applications-on-kubernetes/">my blog post on the Kubernetes site in the fall</a>, I introduced version 0.1 of KubeDirector and described how it works. Since then, we’ve seen a lot of interest in KubeDirector from the community we’re very excited about the progress so far. The BlueData team behind this effort is <a href="https://www.bluedata.com/blog/2018/11/hpe-and-bluedata-joining-forces-in-ai-ml-big-data/">now part of HPE</a>, and the KubeDirector project continues to move full steam ahead.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/1mbaqzb3tkv_zx36nxxdgbw-1568050291276.jpeg" alt="1*mbaqzb3tkv_zx36nxxdgbw"></p> <p>To that end, we just pushed out the next release and our first public update of KubeDirector: version 0.2. You can check out the full details on our github site <a href="https://github.com/bluek8s/kubedirector/releases/tag/v0.2.0">here:</a> <a href="https://github.com/bluek8s/kubedirector/releases/tag/v0.2.0">https://github.com/bluek8s/kubedirector/releases/tag/v0.2.0</a></p> <p>Some of the highlights of what’s new in version 0.2 of KubeDirector include:</p> <ul> <li>A fully deployable Cloudera 5.14.2 image is now available in the catalog of example applications</li> <li>Cluster launch performance has been enhanced through additional work on launch parallelization</li> <li>The “configcli” tool used in application setup is now included in the “nodeprep” directory.</li> <li>We’ve made additional improvements to the Makefile support and functionality: <ul> <li>KubeDirector can now be built and deployed on Ubuntu systems</li> <li>“make deploy” now waits for deployment to succeed before returning</li> <li>“make teardown” now waits for teardown to finish before returning.</li> </ul> </li> <li>KubeDirector actions are now recorded as Kubernetes events and can be viewed by the standard “kubectl describe” command</li> <li>KubeDirector has been tested on the following Kubernetes platforms: <ul> <li>DigitalOcean Kubernetes (DOK)</li> <li>Google Kubernetes Engine (GKE)</li> <li>Amazon Elastic Container Service for Kubernetes (EKS)</li> <li>Kubernetes version 1.13.2 on CentOS kernels</li> </ul> </li> </ul> <p>See below for a screenshot of KubeDirector v0.2 running four pods of a Spark cluster:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/16syu8q9lacfchtp9-ctdca-1568050296650.png" alt="1*6syu8q9lacfchtp9 ctdca"></p> <p>One of those pods is a Jupyter notebook, as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/1bazfzfp7zyqvpvtww-ekbw-1568050305473.png" alt="1*bazfzfp7zyqvpvtww ekbw"></p> <p><strong>Join the Community</strong></p> <p>We’re working towards the next version of KubeDirector (and the broader BlueK8s initiative) and we’d welcome your help as developers, contributors, and adopters. Follow <a href="https://twitter.com/BlueK8s/">@BlueK8s</a> on Twitter and get involved through these channels:</p> <ul> <li>KubeDirector <a href="https://bluek8s.slack.com/join/shared_invite/enQtNTQzNDQzNjQwMDMyLTdjYjE0ZTg0OGJhZWUxMzhkZTZjNDg5ODIyNzZmNzZiYTk4ZjQxNDFjYzk4OWM0MjFlNmVkNWNlNmFjNzkzNjQ">chat room on Slack</a></li> <li>KubeDirector <a href="https://github.com/bluek8s/kubedirector/">GitHub repo</a></li> </ul>https://developer.hpe.com/say-no-to-cloud-vendor-lock-in/https://developer.hpe.com/say-no-to-cloud-vendor-lock-in/Tue, 03 Sep 2019 17:13:05 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/gettyimages-761603815-1567544802064.jpg" alt="gettyimages 761603815"></p> <p>During my 30 year technology career, working for Digital Equipment Corporation (DEC), Compaq Computer, Hewlett Packard (HP) and now Hewlett-Packard Enterprise (HPE), I was in the center of the action when the industry-standard server tsunami hit the computer industry. I rode the wave as we moved away from vendor-specific operating systems (HP-UX, SunOS, AIX, VMS) running on vendor-specific servers (HP, IBM, DEC, Sun) that were powered by vendor-specific processors (PA-RISC, Alpha, Sparc, Power, etc.). Intel x86 powered servers, running Linux, available from any kind of vendor, quickly took over. Death to vendor lock-in! That was our motto at the time.</p> <p>Soon after, database and other software enablers ran into a similar situation. It was an open source snowball effect, having everything to do with avoiding vendor lock-in and encouraging customers to part ways with Oracle, DB2, and RDB. When virtualization technologies came along, they again started out very vendor specific (and still are with VMware, for the most part) but open source solutions also emerged, i.e OpenStack. These solutions now allow customers to build open source private clouds that run on any industry-standard server.</p> <p>Next came the container revolution. Right from the start, containers appeared as an open source solution and that certainly helped in its incredible adoption rate. The success of the container management framework, Kubernetes, is the latest proof of the power of open source and vendor lock-in avoidance.</p> <p>So, what’s the story with public cloud? Some vendors, like Google, are active open source contributors. For the most part, though, public cloud vendors are acting like the HP/Compaq/IBM/DEC of the ‘80s: they propose attractive and innovative solutions only available on their specific cloud technologies and lock you in. The minute you start using some of the best APIs from AWS right in the heart of your code, you are hooked! If you decide to run your application on another cloud, you must rewrite the application to a different API. There are multiple reasons why the ability to deploy on another cloud could become important for your application, whether it’s because of a large customer requirement, merger and acquisition, or cost, as explained by one of Hashi Corp’s founders <a href="https://www.reddit.com/r/devops/comments/91afzz/why_multicloud/e2x156y/">here.</a></p> <p>If you think about it, public cloud today is all about vendor lock-in. Going <a href="https://en.wikipedia.org/wiki/Serverless_computing">serverless</a> (i.e. AWS Lambda, Azure Functions, Google Cloud Functions), meaning the ability to write code that can run on anything and for which you just pay for what’s used, isn't going to help, as it locks you up even tighter. But if serverless is what developers really want, then the next logical step would be cloudless. With cloudless, you could write cloud-agnostic applications that can run on any public or private cloud in an everything-as-a-service consumption model. Given everything that I’ve seen, this makes the most sense and is probably where the industry is headed so make sure you learn more about <a href="https://www.hpe.com/us/en/insights/articles/cloudless-1906.html">Cloudless</a> from <a href="https://www.hpe.com/cloudless">here.</a> Finally, don’t make the same mistakes some made 20 years ago and get entangled in proprietary solutions: say <strong>No</strong> to vendor lock-in now!</p>https://developer.hpe.com/so-you-know-how-to-code-is-that-enough/https://developer.hpe.com/so-you-know-how-to-code-is-that-enough/Tue, 03 Sep 2019 17:08:20 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1567530751772.png" alt="picture1"></p> <p>You've probably spent considerable time mastering a programming language. You may even be an expert in something like JavaScript, CSS, Python or perhaps C/C++. You've written so many lines of code you cannot remember all the projects. And you know you can probably solve just about any problem using your favorite language. That's awesome!</p> <p>But what happens when that language becomes less relevant and is replaced by something else? Technology changes faster than almost anything else (well, maybe not as fast as that green light you never seem to catch in traffic). As a developer, you are constantly challenged to learn the latest programming language or tool that will give you that extra edge to stand out from the rest of the pack. And, despite your best efforts, that's not always going to happen. But you can do plenty of other things to stay ahead in this fast-paced environment.</p> <p>To be a successful developer, you need more than the ability to code. Here are some characteristics I've seen experienced developers display:</p> <p><strong>An enthusiasm for continuous learning.</strong> Yes, this is an age old adage, and you've heard it many times. But it equates to more than just formal and informal training. It's about inviting others to critique your work, and making them feel comfortable with giving you feedback by demonstrating how you listen and act on that feedback. It's about trying something new that's outside your comfort zone, such as doing a presentation or conducting customer research. It also means finding other interests and bringing those conversations and thoughts into your development work. These are just some of the many non-traditional ways to keep learning.</p> <p><strong>Adaptability to changing situations.</strong> Every day the tech industry is flooded with new announcements that portend change -- change that could affect your job, what you do, and what makes you happy. Embracing that change and finding a path forward is key to being successful. You might need to learn a new skill, or you are placed with a new set of team mates, or your favorite project goes away. These things are out of your control. Professional developers will focus on what they can control, adapt as necessary, and find a path forward.</p> <p><strong>A desire to contribute in a meaningful way.</strong> Yes, you wrote your code, made your deadline and can claim success. But, is the code bug free? Is it efficient and performant code? How will the customer or others react to consuming your code? Skilled developers look beyond meeting just the milestone to ensure that what they are doing contributes to a successful outcome for the customer. They also look to see that what they are doing helps lift the contributions of their teammates as well as the broader organization.</p> <p>These are just a few common traits I've seen in developers who I thought were successful. They were not always the experts in the latest languages or tools, but they were inherently an important part of the team.</p> <p>Do you know any successful developers? What characteristics do they display? Share your thoughts and stories with me on Slack at <a href="https://slack.hpedev.io/">hpedev.slack.com</a> or connect with me on Twitter <a href="https://twitter.com/KrenekJeff">@KrenekJeff.</a></p>https://developer.hpe.com/introducing-a-multi-vendor-csi-driver-for-kubernetes/https://developer.hpe.com/introducing-a-multi-vendor-csi-driver-for-kubernetes/Fri, 30 Aug 2019 17:57:28 GMT<p>In true HPE storage tradition, we’re introducing an open source, multi-platform and multi-vendor container storage interface (CSI) driver for Kubernetes. In essence, it's meant to support multiple block and file backends from the HPE portfolio. We encourage others to implement our specification (more on this below) to take advantage of the driver architecture for any file and block backend, including those not from HPE. In the same way our FlexVolume driver (dory) and Dynamic Provisioner (doryd) provided an abstraction for any vendor’s Docker Volume plugin to Kubernetes, we are bringing a similar concept to life with the HPE CSI Driver for Kubernetes. In this blog post, I’ll walk you through the architecture, specification and deployment model. But first, let’s go through the basics of what CSI is.</p> <h1>Container Storage Interface</h1> <p>CSI is a specification that allows container orchestration systems to implement a standardized interface to interact with storage. Kubernetes happens to be one container orchestration implementation that supports CSI. The CSI specification has evolved at a rapid pace since its inception nearly two years ago, steadily adding new features and capabilities. The Kubernetes community declared CSI stable and made it Generally Available (GA) in Kubernetes 1.13 which was released earlier this year. CSI improves the quality of life for both Dev and Ops staff. The developer gets a very consistent interface that allows him to consume storage resources. The Ops person benefits from a very consistent deployment model, as CSI drivers and relevant sidecar containers are simply Kubernetes workloads.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/slides-csi-hpedev-market-1566866211263.png" alt="HPE CSI Driver for Kubernetes with HPE Nimble Storage CSP"></p> <p>The Kubernetes implementation of CSI acts like an extension. This means the specification can evolve outside of <a href="https://github.com/kubernetes/kubernetes">kubernetes/kubernetes</a> and be integrated selectively by downstream Kubernetes vendors. There’s an ongoing project to migrate all relevant Kubernetes in-tree storage drivers to CSI, so all Kubernetes users need to be familiar with CSI to leverage persistent storage in Kubernetes moving forward. The extensibility of CSI leans on a concept where you declare Customer Resource Definitions (CRD) to extend the functionality of Kubernetes. Further, vendors may write their own CRDs to abstract vendor-specific capabilities for end-users. At HPE, we are providing one such example in our deployment where you can easily extract node information that is relevant from a storage mapping perspective.</p> <p>There are many advantages to using CSI over the previous interfaces that were available for storage vendors. At HPE, we used the parameter “overload” concept, as well as extensive PVC annotation, to abstract platform functionality for end-users. With CSI, many data management capabilities become API objects in Kubernetes. As an example, an underlying storage system snapshot becomes a snapshot API object.</p> <p>The table below gives you an idea of where we’ve been and where we’re going with CSI.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/slides-csi-hpedev-feat-1566866257740.png" alt="CSI spec 1.1.0 capabilities implemented by the HPE CSI Driver for Kubernetes"></p> <h1>Container Storage Provider</h1> <p>The new HPE CSI Driver architecture introduces a sidecar deployment we call a Container Storage Provider (CSP). The CSP is unique per storage platform and responds to a minimal set of APIs that interfaces with the storage platform. The HPE CSI Driver does all the heavy lifting on the nodes themselves that is common across storage platforms, such as attach/detach a block device or mount a remote filesystem. The key here is that introducing a new storage platform to Kubernetes using the new HPE CSI Driver will now be quite trivial, as you only need to respond at the API endpoint and don’t need any knowledge of CSI or Kubernetes. The CSP can be written in any language capable of providing a RESTful API on a network port. Some vendors may choose to keep the CSP proprietary to protect their IP as it’s just a microservice and part of a larger system that happens to be open source. Also, running the CSP on Kubernetes is not a requirement either, as it could be an external service as part of an appliance or delivered as SaaS.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/slides-csi-hpedev-1567184427188.png" alt="Communication diagram for the HPE CSI Driver for Kubernetes"></p> <p>In the communication diagram above, you can see what component talks to which endpoint and the distinct boundaries of responsibilities. Innovation will happen at the HPE CSI Driver layer and CSPs will then be able to introduce the newly supported endpoints. For example, a future CSI spec will introduce replica API objects. The HPE CSI Driver takes care of all the minutia the CSI CRDs needs. When the new CSP API spec is ready, storage platform teams may implement the new API endpoint in their CSP and be confident that the CSI-specific work and potential host connectivity is already taken care of.</p> <p>The HPE CSI Driver determines which spec of CSI the running version of Kubernetes supports. The CSP requires no modification or gated behavior based on which CSI spec is enforced on the Kubernetes cluster. This means that there’s a fairly large blast radius in terms of diversity in the Kubernetes environments a CSP can support.</p> <h1>HPE Nimble Storage public beta</h1> <p>Our initial beta driver includes a CSP for HPE Nimble Storage with other HPE platforms and services being developed as we speak. The CSP has been in the works for quite a while and we expect full feature parity with our legacy FlexVolume plugin at its launch with a few additional features as we’ve evolved these two implementations in tandem. The HPE CSI Driver currently supports the current CSI v1.1.0 feature set which the CSP implements: Dynamic Provisioning (with a comprehensive set of parameters), Raw Block Volume, Volume Snapshots, Volume Expansion and Volume Cloning and Inline Ephemeral Volumes.</p> <p>We’ve also taken the opportunity to modernize the deployment of the entire solution with a <a href="https://helm.sh">Helm</a> chart. Helm is used to package software that runs on Kubernetes to give users a consistent way of running their workloads. Since the HPE Nimble Storage CSP runs on Kubernetes, this is the most practical way to deploy it.</p> <p>Some of the features are easier to show than describe, so let us introduce the HPE CSI Driver for Kubernetes <a href="https://www.youtube.com/watch?v=TK5H4o3Tg_s">in this interview with the architect</a>, followed by a demo.</p> <p><a href="https://www.youtube.com/watch?v=TK5H4o3Tg_s"><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/hpecsi-beta-thumb-1566865524360.png" alt="Introducing the multi-platform and multi-vendor HPE CSI Driver (beta) for Kubernetes"></a></p> <p><strong>Note:</strong> All the examples used in the demo can be found <a href="https://github.com/NimbleStorage/container-examples/tree/master/misc/CSI-beta/K8s-1.15">here</a>.</p> <p>As you can see, it’s quite easy to set up and use (you need a HPE Nimble Storage array to actually provision resources). We support a plethora of different use cases besides the few things we squeezed into the demo. Please see the comprehensive list of <a href="https://github.com/hpe-storage/csi-driver/blob/master/examples/kubernetes/hpe-nimble-storage/README.md">StorageClass parameters</a> on GitHub.</p> <h1>Future</h1> <p>We expect to hit GA later in the fall. More platforms, protocols, and services should follow shortly. We also expect a few example open source CSPs to surface that will be able to use the HPE CSI Driver from developers who may not own any HPE product. Since everything needed to build a CSP for any storage platform is available in the spec, we might see any number of non-HPE CSP implementations.</p> <p>The team has been hard at work bringing all this functionality together in this beta. We can’t be more excited to share it with the community! Stay tuned to <a href="https://hpedev.io">hpedev.io</a> for news and updates as we bring more platforms and features into the fold.</p> <h2>Resources</h2> <p>These are the various resources available around the HPE CSI Driver for Kubernetes</p> <ul> <li>HPE CSI Driver for Kubernetes on <a href="https://github.com/hpe-storage/csi-driver">GitHub</a></li> <li>HPE Nimble Storage CSP <a href="https://github.com/hpe-storage/csi-driver/blob/master/examples/kubernetes/hpe-nimble-storage/README.md">StorageClass parameters</a></li> <li>Container Storage Interface <a href="https://github.com/container-storage-interface/spec">specification</a></li> <li>Container Storage Provider <a href="https://github.com/hpe-storage/container-storage-provider">specification</a></li> <li>Container Storage Provider <a href="https://developer.hpe.com/api/hpe-nimble-csp">Swagger documentation</a></li> <li>Join us on the <a href="https://hpedev.slack.com/">HPE DEV slack</a>! We're hanging out in #Kubernetes and #NimbleStorage</li> <li><a href="https://community.hpe.com/t5/HPE-Storage-Tech-Insiders/HPE-CSI-Driver-for-Kubernetes-and-Red-Hat-OpenShift-in-beta/ba-p/7059941">HPE CSI Driver for Kubernetes and Red Hat OpenShift in beta</a></li> </ul>https://developer.hpe.com/meet-the-hpe-dev-team-jeff-krenek-and-vivek-kulkarni/https://developer.hpe.com/meet-the-hpe-dev-team-jeff-krenek-and-vivek-kulkarni/Wed, 28 Aug 2019 15:56:28 GMT<p>As promised in my last <a href="/blog/meet-the-hpe-dev-team">Meet the HPE DEV Team</a> blog, I’m going to introduce you to additional members of the <a href="https://developer.hpe.com/community">HPE Dev Community.</a> This time I’d like to focus on some of our <a href="https://www.hpe.com/us/en/solutions/cloud/azure-hybrid-cloud.html">Azure Stack</a> experts, Jeff Krenek and Vivek Kulkarni, who work on accelerating the customer adoption of hybrid IT environments. As part of the Azure Stack Innovation Centers, they bring in customer and partner developers to experiment and try out Microsoft Azure Stack, which often results in these developers becoming better informed on the potential of using this platform.</p> <p>Jeff manages the code incubation team for HPE DEV as well as the Azure Stack Innovation Centers. He knows hybrid IT and <a href="https://www.hpe.com/us/en/integrated-systems/hyper-converged.html">hyperconverged solutions</a> well, having previously acted as the lead product architect for HPE’s hyperconverged products. We rely on Jeff’s many years of experience. Not only can he pull the right people together to attend to a specific issue, as he does with our strike teams, he also knows a lot about what’s gone on before. He knows what’s worked, what hasn’t, and how to best apply previously obtained insights to new situations. Jeff was a born manager. It’s easy to see how he likes to support people and help them grow. The fact that he’s on the advisory board for the Computer Engineering Department for his alma mater attests to this, as does the fact that interns seem to do very well in this group. As the father of four kids in college, I suspect he’s had a lot of practice!</p> <p>Vivek is a Certified Specialist for Azure Solutions. He helps build the solutions for users and operators to run scripts and use cases for Azure Stack, as well as solutions to assist in Azure Stack migration. Although Vivek has only been with the team for a couple of years, he has a great deal of experience with Microsoft Azure, having worked at Microsoft for some time. Not only does Vivek assist our Azure Stack customers, but he also lends his Azure expertise to other groups inside HPE, including the <a href="https://www.hpe.com/us/en/services/it-consumption.html">HPE GreenLake</a> team, to help them develop products and services around Azure. A graduate of B.V. Bhoomareddi College of Engineering and Technology, it’s great to have Vivek on our team.</p> <p>As I pointed out in my last post, given that HPE is a leader in the technology industry, it has the advantage of being able to recruit some pretty impressive personalities. And they’re all very approachable. You can reach Jeff on Twitter <a href="https://twitter.com/KrenekJeff">@KrenekJeff</a> and Vivek <a href="https://twitter.com/@vkhpedev">@vkhpedev.</a> You can also connect with them on <a href="https://slack.hpedev.io/">Slack.</a></p> <p>But, enough from me. Why don’t you <a href="https://www.youtube.com/watch?v=bbglDBHnLh0&#x26;feature=youtu.be">watch the video</a> and let them tell you their own stories.</p>https://developer.hpe.com/developing-a-client-app-for-accessing-services-via-the-open-service-brok/https://developer.hpe.com/developing-a-client-app-for-accessing-services-via-the-open-service-brok/Wed, 28 Aug 2019 15:36:16 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture13-1567007521162.png" alt="picture13"></p> <p>The Open Service Broker (OSB) API gives customers a uniform and less complicated way to access services offered by providers. The service broker acts as the middleman that links the two parties, transfering and implementing services between the two. It is the missing link, allowing for better service consumption and various kinds of client applications to reach services from any provider. The OSB API is the language specification that provides this communication. In this tutorial, we will show you how to develop a client app for accessing services using OSB. For the purposes of this post, we will be using Grommet OSB Broker, a broker we deployed in AWS cloud.</p> <p>Note: For more information regarding service brokers, please see Pramod Sareddy’s blog post <a href="/blog/using-open-service-broker-as-a-quick-and-easy-way-to-offer-everything-as">Using Open Service Broker as a Quick and Easy Way to Offer Everything as-a-Service.</a> You can also view another example walkthrough in Peng Liu’s post about <a href="/blog/an-open-service-broker-project-delivers-a-sample-devops-environment-to-a">An Open Service Broker Project Delivers a Sample DevOps Environment to AWS.</a> To access API documentation, access the <a href="https://www.openservicebrokerapi.org/">Open Service Broker website</a> and the <a href="https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md">OSB documentation repository.</a></p> <p>The stand-alone client application we developed is an example of an OSB client with a graphical user interface (GUI). Its purpose is to communicate with service brokers and deploy and manage services for the user. The service broker we use in this example is one that provisions an ec2 instance on AWS and installs a Grommet environment for a developer.</p> <p>To register a service broker, you must first know the IP address of a running broker (ex: ip address:port). If you have this address, as well as valid user credentials (username and password), you will be able to connect to the broker. You should also choose a name and description for the local client app’s representation of that broker.</p> <p>Once registered, the client will fetch a broker’s catalog of services. The app will update to reflect the available services in the catalog page. From there, you can search for, or select, the desired service. Selecting a service will bring up a form for deploying an instance of that service. In this example, you will select the grommet service. You can then choose the service plan you want to use and fill in the inputs the broker needs. A plan is essentially a tier of service that may differ in capabilities, size, price, etc. from other plans.</p> <p>Here is the workflow for using the Grommet service via our app:</p> <h2>Step 0 - Download and Install the App</h2> <p>Follow the directions in the README.md and download the source code at the <a href="https://github.com/HewlettPackard/hpe-openservicebroker-clientapp">HPE public GitHub repository.</a> Installation is simple and fast, and there is no backend to the app. The only prerequisites to have installed are Node >= 8.10 and yarn.</p> <h2>Step 1 - Log in to the App</h2> <p>Authentication is not really implemented at this time. Simply enter ‘user’ for the username and ‘password’ for the password to access the platform.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1567007165491.png" alt="picture1"></p> <h2>Step 2 – Deploy a broker</h2> <p>In the real world, the broker could be deployed out on the Internet, on a local server, or anywhere in between. For the purposes of this post, we'll be using the Grommet OSB Broker, deployed on AWS.</p> <h2>Step 3 - Register Grommet Broker</h2> <p>On the broker settings page, you can register a broker. You should click the ‘add broker’ tile and then complete the register form. You can choose a name and an optional description. The broker’s address (URL) must be known in order to access it, and a username and password are required to authenticate to the broker. The username and password to register our broker are both ‘ubuntu’. If the registration fails, an alert will display an error message. Otherwise, the catalog for the broker will be fetched, which we will soon see reflected in the catalog page of our platform. The broker settings page will then be populated with a new broker tile, displaying the name and description of the registered broker. Clicking on this tile will display a details panel that shows the broker’s address, status, and time created. You may choose to edit the broker’s name and description, or delete the broker. Deleting the broker would remove its services from the catalog.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture5-1567007472045.png" alt="picture5"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture6-1567007478316.png" alt="picture6"></p> <h2>Step 4 - Select Grommet Service from Catalog</h2> <p>You can now click on the ‘Catalog’ link to arrive at a page containing the broker’s service as a tile. There is also a search field that could be useful for finding a specific service by name if there are many on the page. Clicking the ‘grommet’ service tile will enable you to deploy an instance of that service.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture7-1567007485144.png" alt="picture7"></p> <h2>Step 5 - Deploy Service Instance</h2> <p>You should select the desired plan for the service. Choosing plan-1 for the Grommet service will produce a form that requires inputs for AWS implementation details. You must also name the instance, and that name must be unique. After clicking the submit button, the app will attempt to order the broker to instantiate the service. If something went wrong with the API call, an error will be displayed, and no instance will be created. If the provisioning is successful, you will be redirected to the Deployed Instances part of the app. You will see a tile that bears the name of the instance you just created.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture8-1567007492340.png" alt="picture8"></p> <h2>Step 6 - Access the Instance</h2> <p>Clicking on the aforementioned tile will produce a details panel for accessing that instance. The status should be ‘loading’, because the ec2 provisioning takes a while to process on the broker’s end. The app will continuously poll the broker’s last-operation endpoint to retrieve the status of the deployment. It will only do so for as long as the broker’s maximum-polling property will allow it to, if specified. Eventually, the status should turn green and say ‘loaded’. You will then have access to the instance’s access details, i.e. the public ip address of the ec2 instance.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture9-1567007498208.png" alt="picture9"></p> <p>Deployed</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture10-1567007504694.png" alt="picture10"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture11-1567007510153.png" alt="picture11"></p> <h2>Step 7 - Delete the Service Instance</h2> <p>On the same panel, you can choose to delete the instance. Clicking the delete button will remove the instance and stop the ec2 instance. Do this only when you are done with the AWS virtual machine!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture12-1567007515231.png" alt="picture12"></p> <h2>Next Steps</h2> <p>The OSB specification recommends an end point to bind/unbind to a service instance. Essentially, this is where the app and the service instance “connect”. This is currently unavailable in our app and will be developed in the future. After a service instance has been provisioned, you should see a button labeled ‘Bind’ that will initiate a HTTP PUT call to the /v2/service_instances/:instance_id/service_bindings/:binding_id endpoint. This feature must also handle polling the /v2/service_instances/:instance_id/service_bindings/:binding_id/last_operation endpoint until the bind/unbind returns the state ‘succeeded’. These operations will be used by more complex brokers.</p> <p>Since the OSB specification says authentication is optional, we’ve only implemented basic authentication. This can be enhanced in the future to use JSON web tokens or more robust techniques. Also, session management can be implemented after a user is authenticated successfully. This would involve creating a session store on the backend using an in-memory database like Redis. Right now, if a user refreshes the page, the session info is lost.</p> <h2>Resources</h2> <p>Access the Open Service Broker Client App <a href="https://github.com/HewlettPackard/hpe-openservicebroker-clientapp">here on GitHub.</a></p> <p>Click <a href="https://www.youtube.com/watch?v=ERwrlvc1KdU&#x26;feature=youtu.be">here to view a video</a> that steps you through the process.</p> <h2>Summary</h2> <p>We had a great time developing this client example and hope you will find it helpful in testing and implementing your service brokers. Please remember to follow our <a href="/blog">HPE DEV blog</a> posts for more information on this and other topics designed to streamline your application development environment.</p>https://developer.hpe.com/codewars-a-unique-mind-boggling-coding-competition-for-senior-high-schoo/https://developer.hpe.com/codewars-a-unique-mind-boggling-coding-competition-for-senior-high-schoo/Wed, 21 Aug 2019 22:29:46 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture10-1566426968816.png" alt="picture10"></p> <p>On July 27, 2019, Hewlett Packard Enterprise (HPE) held an exciting and fun event at the HPE campus in Mahadevapura in Bengaluru -- the 4th edition of CodeWars. A legacy that started 22 years ago in Houston with just 80 students has now evolved to become one of the most anticipated and sought after events exclusively organized for 11th and 12th grade students – <a href="http://www.hpcodewars.org/">CodeWars!</a> CodeWars is a high school computer programming competition designed to spark interest in careers in science, technology, engineering, and math (STEM).</p> <p>CodeWars has become synonymous to a battlefield for budding coders across all regions where HPE, the company that was recognized as one of <a href="https://www.forbes.com/best-employers-for-new-grads/list/#tab:overall">the Best Employers for New College Graduates by Forbes (2018),</a> has an established presence. In this competition, HPE challenges young coders using 20 problems of increasing levels of complexity that are to be cracked within a time frame of 3 hours. This particular CodeWars event successfully combined several quality ingredients: a high-tech setting, a wide range of programming challenges, plenty of food, music, and giveaways - all in an exciting, stimulating, and competitive environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture11-1566426977946.png" alt="picture11"></p> <p>CodeWars gives students the opportunity to see what it is like to code for a technology company. At these events, students get to solve the real-life problems they are likely to encounter in their future professional careers. Those in attendance found it exciting to see a completely packed cafeteria with enthusiastic students eagerly looking forward to a day full of challenging competitions. While a few students looked nervous, others chilled out, playing solitaire to pass the time. About 100 teams from various schools across the city participated in the event. All students put their best foot forward battling for the winning title.</p> <p>Students used some of the most popular programming languages such as Python and Java, C++ to crack the problems. The range of complexity was such that the participants were asked to write code for questions as simple as finding the difference between the original number and its reverse to more complex questions such as finding circular primes.</p> <p>In addition to coding challenges, the students were quizzed on HPE trivia. It was exhilarating to see how the students answered these questions with ease, including some tricky ones such as what landmark innovation of HP is used in households even today and what term is used to define the green HPE rectangular logo. The students weren’t the only ones challenged however. The judges themselves found it very difficult to narrow the selection down to just 3 winners. Because of this, they decided to make some special mentions and give a shout-out to a few of the top performing teams.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture12-1566427031557.png" alt="picture12"></p> <p>At CodeWars, students often go all out and dress up in costumes reflecting their favorite fantasy characters. The extensive effort that participants put in for this event was clearly visible. They were not just determined to ace the tough coding challenges, but they also left no stone unturned in sporting the best possible costumes. Attendees saw so many iconic characters come alive that the event almost doubled as a mini Comicon. The dedication amongst students was such that many teams coded with their masks and accessories on throughout the entire event.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture13-1566427040293.png" alt="picture13"></p> <p>To summarize, CodeWars 2019 India was a great concoction of quality ingredients: a high-tech setting, a wide range of programming challenges, plenty of coding, and tempting giveaways -- all delivered in an exciting, stimulating, and competitive environment.</p>https://developer.hpe.com/how-to-register-a-grommet-osb-broker-in-a-kubernetes-service-catalog/https://developer.hpe.com/how-to-register-a-grommet-osb-broker-in-a-kubernetes-service-catalog/Wed, 21 Aug 2019 18:44:52 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/registering-a-broker-1566920379874.png" alt="registering a broker"></p> <p>In my previous article, <a href="/blog/using-open-service-broker-as-a-quick-and-easy-way-to-offer-everything-as">Using Open Service Broker as a Quick and Easy Way to Offer Everything as-a-Service,</a> we examined what a Open Service Broker (OSB) API is and how it can be used to expose the Grommet development environment as-a-Service. Now, I would like to show you how to register and consume services offered by the Grommet OSB Broker in a Kubernetes Service Catalog to provision, bind, unbind, and deprovision a Grommet Dev Instance.</p> <p>This tutorial will be helpful for developers in many companies who today deploy Kubernetes clusters to ensure scalability for their applications. Applications running inside Kubernetes clusters may need access to 3rd party services, like databases or additional storage, and you need to be able to provide that service to app developers as part of the Kubernetes Service Catalog. One way of exposing a service is to use OSB. Once you register your OSB inside the Kubernetes Service Catalog, you can see the service, and then you can provision and bind the service to your application.</p> <p>This tutorial assumes that you've installed a Service Catalog onto your Kubernetes cluster. If you haven't, please see the <a href="https://github.com/kubernetes-sigs/service-catalog/blob/master/docs/install.md">installation instructions.</a> Optionally you may install the Service Catalog CLI, svcat. Examples for both svcat and kubectl are provided.</p> <p>All commands shown assume that you're operating out of the root of this repository.</p> <p>In the figure below, you can see the overall architecture setup of the Kubernetes Service Catalog running on premise interacting with Grommet OSB broker running in AWS cloud to provision, bind, and deprovision a Grommet Dev Instance in the cloud.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1566414261415.png" alt="picture1"></p> <p>In this architecture, the Kubernetes Service Catalog establishes an association with the Grommet OSB broker by sending a GET request to the /v2/catalog endpoint, which then responds with a 200 OK and a body containing all the information about the services it offers. Kubernetes stores and exposes these services to consumers via the Kubernetes Service Catalog. Cloud operators will instantiate the service by sending a PUT request to the /v2/service_instances/:service_id end point. The broker will do the actual provisioning of the service instance. Let’s discuss each of these components individually.</p> <h2>Kubernetes Internal Architecture</h2> <p>To start, I will briefly cover the basics of how Kubernetes works internally. There is an API server that listens to user requests. Users perform most actions by declaratively describing Kubernetes resources in yaml files that get written through to Etcd, shared key-value store. In my example here, a user is declaring some object Foo, which creates a record in the Etcd.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture2-1566414265789.png" alt="picture2"></p> <p>As you can see, we have a Foo Controller, which is watching the shared Etcd through the API Server for any changes in Foo objects. Now that we just created a new one, our Foo Controller sees the change and begins taking actions to implement this change of state.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture3-1566414269747.png" alt="picture3"></p> <p>This results in ithe Foo controller creating a new Foo object. Depending on what Foo is, this action could be doing something directly or it could be sending a command to another Kubernetes component. Nevertheless, the general principle remains the same. This particular example was for some object-type Foo, but there are many resources in Kubernetes and, correspondingly, many API servers and controllers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture4-1566414273123.png" alt="picture4"></p> <h2>Kubernetes Service Catalog</h2> <p>A Kubernetes Service Catalog is an extension API that enables applications running in Kubernetes clusters to easily use externally managed software offerings, such as a datastore service offered by a cloud provider or a standalone VM, like a Grommet Dev Environment as-a-Service.</p> <p>The service catalog provides a way to list, provision, and bind with externally managed services from service brokers without needing detailed knowledge about how those services are created or managed.</p> <p>With that in mind, here is an overview of what the service catalog looks like. It’s a custom Kube API server and controller that maintains the state for five new resource types that correspond to their equivalents from the <a href="https://www.openservicebrokerapi.org">OSB API.</a> The controller implements the client side of the OSB API, allowing it to query service brokers for their catalogs, and to manipulate them so it can provision and maintain services. It also makes use of a native Kubernetes resource, Secrets, to inject credentials for service bindings into running Pods, but more on that in a moment.</p> <p>Everything I mentioned about the OSB architecture in my <a href="/blog/using-open-service-broker-as-a-quick-and-easy-way-to-offer-everything-as">previous blog,</a> is contained in this interface between the controller and the service brokers. App developers don’t have to be aware of it, and they can continue to use Kubernetes the same as before, issuing normal CRUD commands through the API service.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture5-1566414276654.png" alt="picture5"></p> <p>Once again, this tutorial assumes that you've installed a Service Catalog onto your Kubernetes cluster. If you haven't, please see the <a href="https://github.com/kubernetes-sigs/service-catalog/blob/master/docs/install.md">installation instructions.</a> Optionally, you may install the Service Catalog CLI, svcat. Examples for both svcat and kubectl are provided.</p> <p>All commands in this post assume that you're operating out of the root of this repository.</p> <ul> <li>NOTE: For the purposes of this post, we'll be using Grommet OSB broker, a broker that we deployed in AWS cloud and one that is accessible at this <a href="http://3.86.206.101:8099/">endpoint.</a></li> </ul> <h2>Step 1 – Deploy a broker</h2> <p>In the real world, the broker could be deployed within our cluster, next to our cluster in the same data center, out on the Internet, or anywhere in between. For the purpose of this post, we'll be using Grommet OSB broker, a broker that we deployed in AWS cloud.</p> <h2>Step 2 - Register Grommet Broker</h2> <p>In this second step, the, cluster operator creates a ClusterServiceBroker resource within the servicecatalog.k8.io group. This resource contains the URL and connection details necessary to access a service broker endpoint. The service catalog control manager triggers a call to the external service broker for a list of all available services. The service broker returns a list of available managed services and a list of service plans, which are cached locally as ClusterServiceClass and ClusterServicePlan resources respectively. A cluster operator can then get the list of available managed services and service plans using kubectl get clusterservicecalassess or clusterserviceplans commands.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture6-1566414280554.png" alt="picture6"></p> <p>Because we haven't created any resources in the service-catalog API server yet, querying the service catalog returns an empty list of resources:</p> <pre><code class="language-bash">$ svcat get brokers NAME URL STATUS +------+-----+--------+ $ kubectl get clusterservicebrokers,clusterserviceclasses,serviceinstances,servicebindings No resources found. </code></pre> <p>We'll register a broker server with the catalog by creating a new ClusterServiceBroker resource:</p> <pre><code class="language-bash">$ cat grommet-broker-clusterservicebroker.yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ClusterServiceBroker metadata: name: grommet-broker spec: url: http://3.86.206.101:8099 </code></pre> <pre><code class="language-bash">$ kubectl create -f grommet-broker-clusterservicebroker.yaml clusterservicebroker.servicecatalog.k8s.io/grommet-broker created </code></pre> <p>When we create this ClusterServiceBroker resource, the service catalog controller responds by querying the broker server to see what services it offers and creates a ClusterServiceClass for each.</p> <p>We can check the status of the broker by entering the following commands:</p> <pre><code class="language-bash">$ svcat describe broker grommet-broker Name: grommet-broker Scope: cluster URL: http://3.86.206.101:8099 Status: Ready - Successfully fetched catalog entries from broker @ 2019-07-09 19:17:10 +0000 UTC $ kubectl get clusterservicebrokers grommet-broker -o yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ClusterServiceBroker metadata: creationTimestamp: "2019-07-09T19:17:10Z" finalizers: - kubernetes-incubator/service-catalog generation: 1 name: grommet-broker resourceVersion: "8" selfLink: /apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/grommet-broker uid: 266bed9b-a27e-11e9-9f3e-3aac54c90eba spec: relistBehavior: Duration relistRequests: 0 url: http://3.86.206.101:8099 status: conditions: - lastTransitionTime: "2019-07-09T19:17:10Z" message: Successfully fetched catalog entries from broker. reason: FetchedCatalog status: "True" type: Ready lastCatalogRetrievalTime: "2019-07-09T19:17:10Z" reconciledGeneration: 1 </code></pre> <p>Notice that the status reflects that the broker's catalog of service offerings has been successfully added to our cluster's service catalog.</p> <h2>Step 3 – Viewing ClusterServiceClasses and ClusterServicePlans</h2> <p>The controller has already created a ClusterServiceClass for each service that the grommet broker provides. We can view the ClusterServiceClass resources available:</p> <pre><code class="language-bash">$ svcat get classes NAME NAMESPACE DESCRIPTION +---------+-----------+-----------------+ grommet grommet service $ kubectl get clusterserviceclasses NAME EXTERNAL-NAME BROKER AGE 97ca7e25-8f63-44a7-99d1-a75729ebfb5e grommet grommet-broker 4m30s </code></pre> <ul> <li>NOTE: The above kubectl command uses a custom set of columns. The NAME field is the Kubernetes name of the ClusterServiceClass and the EXTERNAL NAME field is the human-readable name for the service that the broker returns.</li> </ul> <pre><code class="language-bash">$ svcat describe class grommet Name: grommet Scope: cluster Description: grommet service Kubernetes Name: 97ca7e25-8f63-44a7-99d1-a75729ebfb5e Status: Active Tags: ui, grommet Broker: grommet-broker Plans: NAME DESCRIPTION +----------------+----------------+ grommet-plan-1 t2.micro instance with NodeJS grommet-plan-2 t2.small instance with NodeJS $ kubectl get clusterserviceclasses 97ca7e25-8f63-44a7-99d1-a75729ebfb5e -o yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ClusterServiceClass metadata: creationTimestamp: "2019-07-09T19:17:10Z" name: 97ca7e25-8f63-44a7-99d1-a75729ebfb5e ownerReferences: - apiVersion: servicecatalog.k8s.io/v1beta1 blockOwnerDeletion: false controller: true kind: ClusterServiceBroker name: grommet-broker uid: 266bed9b-a27e-11e9-9f3e-3aac54c90eba resourceVersion: "5" selfLink: /apis/servicecatalog.k8s.io/v1beta1/clusterserviceclasses/97ca7e25-8f63-44a7-99d1-a75729ebfb5e uid: 268d3344-a27e-11e9-9f3e-3aac54c90eba spec: bindable: true bindingRetrievable: false clusterServiceBrokerName: grommet-broker description: grommet service externalID: 97ca7e25-8f63-44a7-99d1-a75729ebfb5e externalMetadata: displayName: The Grommet Broker listing: blurb: Add a blurb here imageUrl: http://example.com/cat.gif longDescription: UI component library, in a galaxy far far away... provider: name: The grommet externalName: grommet planUpdatable: true requires: - route_forwarding tags: - ui - grommet status: removedFromBrokerCatalog: false </code></pre> <p>Additionally, the controller created a ClusterServicePlan for each of the plans for the broker's services. We can view the ClusterServicePlan resources available in the cluster:</p> <pre><code class="language-bash">$ svcat get plans NAME NAMESPACE CLASS DESCRIPTION +----------------+-----------+---------+----------------+ grommet-plan-1 grommet t2.micro instance with NodeJS grommet-plan-2 grommet t2.small instance with NodeJS $ kubectl get clusterserviceplans NAME EXTERNAL-NAME BROKER CLASS AGE 2a44ed0e-2c09-4be6-8a81-761ddba2f733 grommet-plan-1 grommet-broker 97ca7e25-8f63-44a7-99d1-a75729ebfb5e 7m2s e3c4f66b-b7ae-4f64-b5a3-51c910b19ac0 grommet-plan-2 grommet-broker 97ca7e25-8f63-44a7-99d1-a75729ebfb5e 7m2s </code></pre> <p>You can view the details of a ClusterServicePlan with this command:</p> <pre><code class="language-bash">$ svcat describe plan grommet/default $ kubectl get clusterserviceplans 86064792-7ea2-467b-af93-ac9694d96d52 -o yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ClusterServicePlan metadata: creationTimestamp: "2019-07-09T19:17:10Z" name: 2a44ed0e-2c09-4be6-8a81-761ddba2f733 ownerReferences: - apiVersion: servicecatalog.k8s.io/v1beta1 blockOwnerDeletion: false controller: true kind: ClusterServiceBroker name: grommet-broker uid: 266bed9b-a27e-11e9-9f3e-3aac54c90eba resourceVersion: "6" selfLink: /apis/servicecatalog.k8s.io/v1beta1/clusterserviceplans/2a44ed0e-2c09-4be6-8a81-761ddba2f733 uid: 268e25b5-a27e-11e9-9f3e-3aac54c90eba spec: clusterServiceBrokerName: grommet-broker clusterServiceClassRef: name: 97ca7e25-8f63-44a7-99d1-a75729ebfb5e </code></pre> <h2>Step 4 – Creating a New ServiceInstance</h2> <p>Here, the cluster operator can instantiate the provisioning of a new instance by creating a ServiceInstance resource within the servicecatalog.k8.io group. When the ServiceInstance resource is created, the service catalog control manager initiates a call to the external service broker to provision an instance of the service.</p> <p>The service broker creates a new instance of the managed service and returns an HTTP response. A cluster operator can then check the status of the instance to see if it is ready.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture7-1566414285460.png" alt="picture7"></p> <p>Now that a ClusterServiceClass named grommet exists within our cluster's service catalog, we can create a ServiceInstance that points to it.</p> <p>Unlike ClusterServiceBroker and ClusterServiceClass resources, ServiceInstance resources must be namespaced. Create a namespace with the following command:</p> <pre><code class="language-bash">$ kubectl create namespace grommet-ns namespace/grommet-ns created </code></pre> <p>Then, create the ServiceInstance:</p> <pre><code class="language-bash">$ cat grommet-broker-instance.yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: grommet-broker-instance namespace: grommet-ns spec: clusterServiceClassExternalName: grommet clusterServicePlanExternalName: grommet-plan-1 parameters: region: "us-east-1" Access_Key_ID: "XXXXXXXXXXXXXX" Secret_Access_Key: "XXXXXXXXXXX" Image_ID: "ami-05f07ee3c7aa" Flavor: "t2.small" NodeJS_version: "12.1.0" $ kubectl create -f grommet-broker-instance.yaml –n grommet-ns serviceinstance.servicecatalog.k8s.io/grommet-broker-instance created </code></pre> <p>After the ServiceInstance is created, the service catalog controller will communicate with the appropriate broker server to initiate provisioning. Check the status of that process:</p> <pre><code class="language-bash">$ svcat describe instance -n grommet-ns grommet-broker-instance Name: grommet-broker-instance Namespace: grommet-ns Status: Ready - The instance was provisioned successfully @ 2019-07-09 23:15:56 +0000 UTC DashboardURL: http://:3000 Class: grommet Plan: grommet-plan-1 Parameters: Access_Key_ID: XXXXXXXXX Flavor: t2.small Image_ID: ami-05f07ee3c7aa NodeJS_version: 12.1.0 Secret_Access_Key: XXXXXXXXXXX region: us-east-1 Bindings: No bindings defined $ kubectl get serviceinstances -n grommet-ns grommet-broker-instance -o yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: creationTimestamp: "2019-07-09T22:40:41Z" finalizers: - kubernetes-incubator/service-catalog generation: 1 name: grommet-broker-instance namespace: grommet-ns resourceVersion: "83" selfLink: /apis/servicecatalog.k8s.io/v1beta1/namespaces/grommet-ns/serviceinstances/grommet-broker-instance uid: 9479a488-a29a-11e9-9f3e-3aac54c90eba spec: clusterServiceClassExternalName: grommet clusterServiceClassRef: name: 97ca7e25-8f63-44a7-99d1-a75729ebfb5e clusterServicePlanExternalName: grommet-plan-1 clusterServicePlanRef: name: 2a44ed0e-2c09-4be6-8a81-761ddba2f733 externalID: 9479a40b-a29a-11e9-9f3e-3aac54c90eba parameters: Access_Key_ID: XXXXXXXXXX Flavor: t2.small Image_ID: ami-05f07ee3c7aaadaaa NodeJS_version: 12.1.0 Secret_Access_Key: XXXXXXXXXX region: us-east-1 updateRequests: 0 userInfo: groups: - system:masters - system:authenticated uid: "" username: kubernetes-admin status: asyncOpInProgress: false conditions: - lastTransitionTime: "2019-07-09T23:15:56Z" message: The instance was provisioned successfully reason: ProvisionedSuccessfully status: "True" type: Ready dashboardURL: http://:3000 deprovisionStatus: Required externalProperties: clusterServicePlanExternalID: 2a44ed0e-2c09-4be6-8a81-761ddba2f733 clusterServicePlanExternalName: grommet-plan-1 parameterChecksum: 2ffa186d88170935135d51e53d4048f2950386d5e3a54e08e811bac054f78779 parameters: Access_Key_ID: XXXXXXXXXX Flavor: t2.small Image_ID: ami-05f07ee3c7aaadaaa NodeJS_version: 12.1.0 Secret_Access_Key: XXXXXXXXXX region: us-east-1 userInfo: groups: - system:masters - system:authenticated uid: "" username: kubernetes-admin observedGeneration: 1 orphanMitigationInProgress: false provisionStatus: Provisioned reconciledGeneration: 1 </code></pre> <h2>Step 5 – Requesting a ServiceBinding to use the ServiceInstance</h2> <p>After a new instance has been provisioned, a cluster operator must bind to the managed service to get the connection credentials and service account details necessary for the application to use the service or to access the service. This is done by creating a ServiceBinding resource.</p> <p>After the ServiceBinding is created, the service catalog makes a call to the external service broker requesting the information necessary to bind with the service instance.</p> <p>The service broker enables the application permissions/roles for the appropriate service account.</p> <p>The service broker returns the information necessary to connect and access the managed service instance. This is provider and service-specific so the information returned may differ between service providers and their managed services. In our case, the Grommet broker returns the Grommet Dev Instance access credentials.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture8-1566414289475.png" alt="picture8"></p> <p>Now that our ServiceInstance has been created, we can bind to it. Create a ServiceBinding resource:</p> <pre><code class="language-bash">$ kubectl create -f grommet-broker-binding.yaml servicebinding.servicecatalog.k8s.io/grommet-broker-binding created </code></pre> <p>After the ServiceBinding resource is created, the service catalog controller will communicate with the appropriate broker server to initiate binding. Generally, this will cause the broker server to create and issue credentials that the service catalog controller will insert into a Kubernetes Secret. We can check the status of this process like so:</p> <pre><code class="language-bash">$ svcat describe binding -n grommet-ns grommet-broker-binding Name: grommet-broker-binding Namespace: grommet-ns Status: Ready - Injected bind result @ 2019-07-09 23:37:18 +0000 UTC Secret: grommet-broker-binding Instance: grommet-broker-instance Parameters: No parameters defined Secret Data: uri 25 bytes username 6 bytes $ kubectl get servicebindings -n grommet-ns grommet-broker-binding -o yaml apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceBinding metadata: creationTimestamp: "2019-07-09T23:37:17Z" finalizers: - kubernetes-incubator/service-catalog generation: 1 name: grommet-broker-binding namespace: grommet-ns resourceVersion: "90" selfLink: /apis/servicecatalog.k8s.io/v1beta1/namespaces/grommet-ns/servicebindings/grommet-broker-binding uid: 7cf4999a-a2a2-11e9-9f3e-3aac54c90eba spec: externalID: 7cf498f3-a2a2-11e9-9f3e-3aac54c90eba instanceRef: name: grommet-broker-instance secretName: grommet-broker-binding userInfo: groups: - system:masters - system:authenticated uid: "" username: kubernetes-admin status: asyncOpInProgress: false conditions: - lastTransitionTime: "2019-07-09T23:37:18Z" message: Injected bind result reason: InjectedBindResult status: "True" type: Ready externalProperties: userInfo: groups: - system:masters - system:authenticated uid: "" username: kubernetes-admin orphanMitigationInProgress: false reconciledGeneration: 1 unbindStatus: Required </code></pre> <p>Notice that the status has a Ready condition set. This means our binding is ready to use! If we look at the Secrets in our grommet-ns namespace, we should see a new one:</p> <pre><code class="language-bash">$ kubectl get secrets -n grommet-ns NAME TYPE DATA AGE default-token-hjm6z kubernetes.io/service-account-token 3 139m grommet-broker-binding Opaque 2 3m37s </code></pre> <ul> <li>Notice that a new Secret named grommet-broker-binding has been created.</li> </ul> <h2>Step 6 – Delete the ServiceBinding</h2> <p>Now, let's unbind the instance:</p> <pre><code class="language-bash">$ svcat unbind -n grommet-ns grommet-broker-instance deleted grommet-broker-binding </code></pre> <p>After the deletion is complete, we should see that the Secret is gone:</p> <pre><code class="language-bash">$ kubectl get secrets -n grommet-ns NAME TYPE DATA AGE default-token-hjm6z kubernetes.io/service-account-token 3 154m </code></pre> <h2>Step 7 – Deleting the ServiceInstance</h2> <p>There may be times you want to delete a ServiceInstance. In that case you can deprovision it. You can do so using the following steps:</p> <pre><code class="language-bash">$ svcat deprovision -n grommet-ns grommet-broker-instance deleted grommet-broker-instance </code></pre> <h2>Step 8 – Deleting the ClusterServiceBroker</h2> <p>Next, remove the ClusterServiceBroker resource. This tells the service catalog to remove the broker's services from the catalog. Do so with this command:</p> <pre><code class="language-bash">$ kubectl delete clusterservicebrokers grommet-broker clusterservicebroker.servicecatalog.k8s.io "grommet-broker" deleted </code></pre> <p>You should now see that all the ClusterServiceClass resources that came from that broker have also been deleted:</p> <pre><code class="language-bash">$ svcat get classes NAME NAMESPACE DESCRIPTION +------+-----------+-------------+ $ kubectl get clusterserviceclasses No resources found. </code></pre> <h2>Next Steps</h2> <p>There are many ways to consume the services offered by an OSB broker such as the Grommet OSB broker. Using a Kubernetes Service Catalog is one option. In our next article, you’ll learn how to download a standalone Open Service broker client application built on Grommet, and how to register an OSB inside that application. You can also use it to test an OSB you already have. Keep an eye out on the <a href="/blog">HPE DEV site</a> for more articles on OSB.</p>https://developer.hpe.com/using-open-service-broker-as-a-quick-and-easy-way-to-offer-everything-as/https://developer.hpe.com/using-open-service-broker-as-a-quick-and-easy-way-to-offer-everything-as/Mon, 19 Aug 2019 15:29:52 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/everything-as-a-service-2-1568045844717.png" alt="everything as a service 2"></p> <p>Service brokers can be very helpful in delivering applications, development environments, or anything as-a-Service. Before the use of service brokers, setting up a development environment as-a-Service required a lot of time and manual effort. Service brokers compliant with the Open Service Broker (OSB) API specification can intuitively provision a new instance of a service they offer and provide all of the information your app or container needs to connect to it.</p> <p>OSB packages all the pieces required for communication between the service provider and the consumer of that service, basically acting as the middle man who ensures clear communication and helps things work in a more automated way. Before OSB, developers had to download or create all the missing linkages. OSB eliminates that need.</p> <p>In this tutorial, I will explain what an OSB API is and how to use it. For this example, I will show you how to expose the Grommet development environment as-a-Service. From there you can use similar methods to offer other application environments in the same way.</p> <h2>What is an Open Service Broker API</h2> <p>The OSB API defines an HTTP(S) interface between platforms and service brokers. The OSB API specifies how automated deployment, management, and use of services is handled. For example, app developers can use it to instantiate services and attach them to apps without having to care about how those services are ultimately created or managed. The OSB API is based on the service architecture originally developed for a well-known PaaS platform called Cloud Foundry.</p> <p>The client side of the OSB API is implemented by the client platform, i.e. Kubernetes Service Catalog, that looks to use services exposed by OSB. The server side is implemented by the service provider, and consists of many endpoints the client hits to provision and manipulate services. The three most important endpoints to remember are the catalog, service instances, and service bindings, which I’ll go over in more detail later.</p> <h2>Why use an Open Service Broker</h2> <p>It’s important to understand the problem the OSB API is attempting to solve. In some cases, an application might require access to a network of resources it depends on -- databases, caches, email systems, subscriptions to subsystem APIs, among others. Previously, these were manually and statically configured; now with today’s agile software development processes, where app instances are quickly spun up and deleted, they must be managed dynamically.</p> <p>So, how do we solve this problem? Keep in mind we want to take as much of the cognitive load off the app developers as possible. We don’t want app devs to be responsible for starting or configuring all these services. It’s important to ensure applications are as plug-and-play as possible. Similarly, we don’t want cloud providers to be saddled with that responsibility, either.</p> <p>Service brokers are the missing link between the consumer and the provider. The broker holds the information about the services being provided, and carries out the details of ordering, provisioning, and connecting these services to the application being built by the consumer. Additionally, it automates steps that used to be performed by IT operations with multiple infrastructure management tools.</p> <h2>Open Service Broker (OSB) terms</h2> <ul> <li> <p>Broker: A broker offers set of capabilities called services</p> <ul> <li>Ex: Grommet Broker, MongoDB Broker</li> </ul> </li> <li> <p>Catalog: Collection of services is called catalog</p> <ul> <li>A marketplace/catalog contains services from one or more brokers</li> </ul> </li> <li> <p>Service: a capability managed by the service broker</p> <ul> <li>Ex: Grommet Dev Environment as a Service</li> </ul> </li> <li> <p>Service instance: an instantiation of a particular service's capability *Ex: Grommet Dev Environment VM</p> </li> <li> <p>Binding: relationship between a service instance and an application</p> <ul> <li>Ex: Credentials created to access 'Grommmet Dev Environment'</li> </ul> </li> </ul> <p>Ok, now that we understand what brokers and all those other terms are and the value that they bring, I’m going to briefly go over how to write a sample broker, like Grommet OSB broker in Python. I’ll also show the basic workflow of registering a Grommet OSB broker to your cloud platform so it can consume the services.</p> <h2>Register a Service Broker</h2> <p>The first thing that needs to happen is we need to associate a broker with our platform somehow -- this is called registering a broker. To do this, your cloud platform sends a GET request to the /v2/catalog endpoint on the broker, which responds with a 200 OK and a body containing all the information about the services the broker offers. It’s up to the cloud platform to store this data and make it available to users.</p> <ul> <li>NOTE: The Grommet OSB broker exposes services via /v2/catalog endpoint. Below, you can find the sample Python code exposing the service called “grommet” with two plans - “grommet-plan-1” and “grommet-plan-2”.</li> </ul> <pre><code class="language-python">def catalog(): """ Return the catalog of services handled by this broker GET /v2/catalog: HEADER: X-Broker-API-Version: &#x3C;version> return: JSON document with details about the services offered through this broker Using OSB Spec of Get Catalog: https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md """ # api_version = bottle.request.headers.get('X-Broker-API-Version') print("inside catalog") # if (not api_version or not (api_version_is_valid(api_version))): # bottle.abort( # 409, # "Missing or incompatible %s. Expecting version %.0f.%.0f or later" % ( # X_BROKER_API_VERSION_NAME, # X_BROKER_API_MAJOR_VERSION, # X_BROKER_API_MINOR_VERSION)) return {"services": [service]} </code></pre> <pre><code class="language-python">{ "services": [ { "id": " 97ca7e25-8f63-44a7-99d1-a75729eb", "name": "grommet", "description": "Grommet Dev Environment as a Service", ... "plans": [{ "id": "123", "name": " grommet-plan-1", "description": "t2.micro instance with NodeJS", ... }, { "id": "456", "name": "grommet-plan-2", "description": "t2.small instance with NodeJS", ... }, } </code></pre> <h2>Creating a Service</h2> <p>Now, if I want to create a service, I send a PUT request to /v2/service_instances/:service_id, where the service ID is a unique ID for the service being created. It’s up to the platform to create and track these. Once the service has been created, the broker returns a 201 to the platform.</p> <ul> <li>NOTE: The Grommet OSB broker exposes a /v2/service_instances/:service_id endpoint to create a service instance. Below, you can find the sample Python code creating a Ec2 instance in an AWS cloud.</li> </ul> <pre><code class="language-python">def provision(instance_id): """ Provision an instance of this service for the given org and space PUT /v2/service_instances/&#x3C;instance_id>: &#x3C;instance_id> is provided by the Cloud Controller and will be used for future requests to bind, unbind and deprovision BODY: { "service_id": "&#x3C;service-guid>", "plan_id": "&#x3C;plan-guid>", "organization_guid": "&#x3C;org-guid>", "space_guid": "&#x3C;space-guid>" } return: JSON document with details about the services offered through this broker """ if bottle.request.content_type != 'application/json': bottle.abort(415, 'Unsupported Content-Type: expecting application/json') # get the JSON document in the BODY provision_details = bottle.request.json # Provision and launch the EC2 instance instance_info = create_ec2_instance(image_id, instance_type, keypair_name, user_data) global ec2_instance_id ec2_instance_id = instance_info["InstanceId"] bottle.response.status = 202 #ec2_ip_addr = instance_info['Instances'][0]['PublicIpAddress'] dashboard_url = "http://"+ec2_ip_addr+":3000" return {"dashboard_url": dashboard_url} </code></pre> <h2>Binding a Service</h2> <p>Once I create an instance, I need to create a binding to access that instance. This is done in a similar way, with a PUT request to /v2/service_instances/:instance_id/service_bindings/:binding_id, where binding_id is some globally unique ID that the platform has created. We again receive a 201 response.</p> <p>Example: Grommet OSB broker /v2/service_instances/:instance_id/service_bindings/:binding_id endpoint</p> <ul> <li>NOTE: The Grommet OSB broker exposes</li> </ul> <p>/v2/service_instances/:instance_id/service_bindings/:binding_id endpoint to create a binding for an instance. Below, you can find the sample Python code used to create access credentials for an Ec2 instance provisioned in the previous step. End users can use these credentials to access the Grommet Dev Environment and start creating Grommet-based web apps.</p> <pre><code class="language-python">def bind(instance_id, binding_id): """ Bind an existing instance with the for the given org and space PUT /v2/service_instances/&#x3C;instance_id>/service_bindings/&#x3C;binding_id>: &#x3C;instance_id> is the Cloud Controller provided value used to provision the instance &#x3C;binding_id> is provided by the Cloud Controller and will be used for future unbind requests BODY: { "plan_id": "&#x3C;plan-guid>", "service_id": "&#x3C;service-guid>", "app_guid": "&#x3C;app-guid>" } return: JSON document with credentails and access details for the service based on this binding http://docs.cloudfoundry.org/services/binding-credentials.html """ if bottle.request.content_type != 'application/json': bottle.abort(415, 'Unsupported Content-Type: expecting application/json') # get the JSON document in the BODY binding_details = bottle.request.json bottle.response.status = 201 uri ="http://"+ec2_ip_addr+":3000" return {"credentials": {"uri": uri, "username": "ubuntu"}} </code></pre> <h2>UnBinding a Service</h2> <p>Once I am done using the instance and don’t need the instance any more, I can delete the binding to remove the access to that instance. To do this, I send a DELETE request to /v2/service_instances/:service_id/service_bindings/:binding_id, where binding_id is some globally unique ID that the platform has created. I will again receive a 200 or 202 response and the cloud platform will clear the access credentials stored locally.</p> <p>Example: Grommet OSB broker DELETE /v2/service_instances/:service_id/service_bindings/:binding_id endpoint</p> <ul> <li>NOTE: The Grommet OSB broker exposes a DELETE /v2/service_instances/:service_id/service_bindings/:binding_id endpoint to delete a binding for an instance. Below, you can find the sample Python code to delete the binding for a previously provisioned Ec2 instance.</li> </ul> <pre><code class="language-python">def unbind(instance_id, binding_id): """ Unbind an existing instance associated with the binding_id provided DELETE /v2/service_instances/&#x3C;instance_id>/service_bindings/&#x3C;binding_id>: &#x3C;instance_id> is the Cloud Controller provided value used to provision the instance &#x3C;binding_id> is the Cloud Controller provided value used to bind the instance return: As of API 2.3, an empty JSON document is expected """ return {} </code></pre> <h2>Deprovision Instance</h2> <p>Now, if I want to remove the provisioned service instance, I send a DELETE request to /v2/service_instances/:service_id, where the instanceID is a unique ID for the service being created. It’s up to the platform to create and track these. Once the service has been deleted, the broker returns a 200 OK to the platform, or it returns 202 Accepted if deletion is in progress.</p> <p>Example: Grommet OSB broker DELETE /v2/service_instances/:service_id/ endpoint</p> <ul> <li>NOTE: The Grommet OSB broker exposes a DELETE /v2/service_instances/:service_id/ endpoint to delete an instance. Below, you can find the sample Python code used to delete a previously provisioned Ec2 instance.</li> </ul> <pre><code class="language-python">def deprovision(instance_id): """ Deprovision an existing instance of this service DELETE /v2/service_instances/&#x3C;instance_id>: &#x3C;instance_id> is the Cloud Controller provided value used to provision the instance return: As of API 2.3, an empty JSON document is expected """ # send response #ec2_client = boto3.client('ec2') #ec2.Instance('i-00434b87058703892').terminate() #ec2.instances.filter(InstanceIds=ids).terminate() deprovision_details = bottle.request.json return {} </code></pre> <p>Now, I have shown you the sample Python code of the Grommet OSB broker that will expose Grommet Dev Environment as-a-Service. You can deploy this broker anywhere in the cloud or on premise and share the broker endpoint with users to start consuming the services offered.</p> <h2>Get and run the OSB broker service</h2> <ul> <li>Create an OSB working folder in the hosting VM. Use any folder you prefer.</li> <li><a href="https://github.com/reddypramod85/osb-grommet-python">Get the OSB broker service</a> <ul> <li>You can download the zip file and unzip the package to the OSB working folder.</li> </ul> </li> <li>Install the required the packages <ul> <li>sudo pip3 install --no-cache-dir -r requirements.txt</li> </ul> </li> <li>Start the OSB service broker: <ul> <li>python3 osb_template.py</li> </ul> </li> </ul> <h2>Next Steps</h2> <p>There are many ways to write an OSB broker, like the Grommet OSB Broker, following the OSB API specification. Using Python is one option. In future articles, I will explore other approaches. Keep a look out on the <a href="/blog">HPE DEV blog site</a> for my next article, <strong>How to Register a Grommet OSB broker in Kubernetes Service Catalog,</strong> where I will show how to register and consume the services offered by Grommet OSB Broker in the Kubernetes Service Catalog and show you how to provision, bind, unbind, and deprovision a Grommet Dev instance.</p>https://developer.hpe.com/2019-August-13/https://developer.hpe.com/2019-August-13/Tue, 13 Aug 2019 05:00:00 GMThttps://developer.hpe.com/celebrate-the-beginning-of-devops/https://developer.hpe.com/celebrate-the-beginning-of-devops/Thu, 08 Aug 2019 16:30:30 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/gearhead-banner-1565283401379.png" alt="gearhead banner"></p> <p>Something very special is just around the corner--something of historic proportions. This October, in beautiful Ghent, Belgium, <a href="https://devopsdays.org/">DevOpsDays</a> will be celebrating its <a href="https://devopsdays.org/events/2019-ghent/welcome/">10th year anniversary</a> in the very place where the term DevOps was first coined. This is where software developers and ITOps staff first gathered to talk about the miscommunications between these two groups and looked for ways to get around the so-called <a href="https://sellegi.se/glossary/wall-of-confusion/">“Wall of Confusion”. </a></p> <p>If you’re not familiar with DevOpsDays, it is a set of events that happens year-round throughout the world focusing on software development, IT operations, and the intersection between the two. This worldwide series of technical conferences grew out of the efforts of one man, Patrick Debois.</p> <p>Frustrated by the constant miscommunication between operations and development teams, Patrick hoped to push organizations from traditional software development cycles (where the development team throws a software release “over the wall” to IT operations) to a more agile process where developers and ITOps staff work together.</p> <p>In 2007, Patrick was a system administrator working on a huge data migration project for the Belgian government. Patrick saw hope in being able to apply the <a href="https://agilemanifesto.org/">Agile</a> methodology to infrastructure, and together with Andrew Shafer, created the Agile Systems Administration Group.</p> <p>In 2009, Patrick was unable to attend the Velocity O’Reilly conference in the U.S. where the now mythical session <a href="https://fr.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr/15-Lowering_risk_of_changethrough_tools">10+ Deploys a Day: Dev and Ops Cooperation at Flickr</a> was given by John Allspaw and Paul Hammond. Undeterred, Patrick organized his own conference in Ghent to propagate this idea of the two groups cooperating and closely collaborating. He wanted a conference name that was short, easy to remember, and fitting, so DevOpsDays (#DevOpsDays) was born. The term DevOps quickly emerged and gained attention via social networks.</p> <p>Hewlett-Packard Enterprise (HPE) will be supporting the 10th anniversary of this event as a Gold sponsor because we believe DevOps is critical to building applications for the New Idea Economy. The HPE DEV Community will be there to showcase how we work to design and build infrastructure as code to support greater agility. Don’t miss this opportunity to join us October 28th, 2019 in Ghent to celebrate <a href="http://cloudplatformonline.com/rs/248-TPC-286/images/DORA-State%20of%20DevOps.pdf">and accelerate the coming together of DevOps.</a></p>https://developer.hpe.com/redfish-workshop-at-the-open-source-summit-na-2019/https://developer.hpe.com/redfish-workshop-at-the-open-source-summit-na-2019/Mon, 05 Aug 2019 16:14:11 GMT<p>Bruno Cornec, the HPE Open Source and Linux Technology Strategist, will be speaking at the <a href="https://events.linuxfoundation.org/events/open-source-summit-north-america-2019/">upcoming Open Source Summit</a> in San Diego, August 21st – 23rd, 2019. He will deliver two sessions – one on Docker (a hands-on lab that’s designed to help you understand what it means to containerize) and one on Redfish. Bruno will also act as the coordinator for the co-located, first-ever HPE+SUSE Redfish Workshop.</p> <p>The <a href="http://trac.project-builder.org/wiki/RedfishWSNA2019">Redfish Workshop</a> is a one day event, held Tuesday, August 20, 2019 from 9:00am – 5:00pm at the Hilton San Diego Bayfront. At the event, you can see live demos and interact with <a href="https://en.wikipedia.org/wiki/Redfish_(specification)">Redfish</a> Project technical experts. In this workshop, system administrators, architects, and developers can learn how to use Redfish and about the benefit derived from a standard management layer to deploy, configure, and manage an environment.</p> <p>Knowledgeable and engaging speakers will help you better understand topics like using DMTF tools for system configuration and using the REST API to perform Redfish operations from Python. You’ll also get clued into the latest news on the standard and its future evolution. Through interactive sessions, demos, and labs, you’ll get the chance to better understand and practice the concepts being presented.</p> <p>This is a free event, sponsored by Hewlett Packard Enterprise (HPE) and SUSE.</p> <p>Who can benefit:</p> <ul> <li>System/Software Developers</li> <li>Tech Leads / Development Leads</li> <li>Software Architects</li> <li>Chief Engineers</li> <li>System Engineers</li> <li>Development Engineers</li> <li>DevOps / System Administrators</li> <li>Application Engineers</li> <li>Open Source Technologists</li> </ul> <p>For more information about the event, please visit the <a href="http://trac.project-builder.org/wiki/RedfishWSNA2019">Redfish Workshop page.</a></p> <p><strong>IMPORTANT:</strong> There are a limited number of seats available for this event, so please <a href="https://framaforms.org/redfish-workshop-oss-na-2019-registration-form-1564098902">register</a> as soon as you can.</p> <p>If you have expertise you would like to share during this workshop, please email <a href="mailto:Bruno.Cornec@hpe.com">Bruno.Cornec@hpe.com</a>.</p>https://developer.hpe.com/designing-installing-and-configuring-a-failover-cluster-with-hyper-v-man/https://developer.hpe.com/designing-installing-and-configuring-a-failover-cluster-with-hyper-v-man/Mon, 05 Aug 2019 15:40:49 GMT<p>System Center Virtual Machine Manager (SCVMM) is a management tool developed by Microsoft to efficiently manage Hyper-V in a scalable way. While Hyper-V includes its own tools for managing virtual machines (VMs), as an enterprise scales to include failover clustering, Hyper-V replication, or multiple Hyper-V hosts across a variety of physical servers, SCVMM helps to simplify the management of the virtualized infrastructure.</p> <p>In this tutorial, you will learn how to install SCVMM in a Microsoft Windows Failover Cluster to improve the availability of applications and services. All you need are at least 2 servers running Windows server 2016 with Hyper-V role in a Domain environment to create this failover configuration using the SCVMM application.</p> <p>For the purposes of this tutorial, I used two blade servers, each with a 12-core CPU and 64 GB of memory, in an <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a> 12000 Frame. I created two profiles with three different network connections and three storage volumes (one for boot and two shared, with one for VMs and one for quorum).</p> <p>After a firmware upgrade with the latest Service Pack for ProLiant (SPP), I booted the systems from the virtual CD-ROM with the Windows Server 2016 ISO file. Once the servers became available, I configured the network with a static IP address, enabled Remote Desktop Service, installed Windows updates, and then activated the Windows key.</p> <p>The servers were ready to join the Windows domain after they were rebooted to apply the new configuration.</p> <p>Next, I installed Hyper-V role, then failover Clustering and the Multipath IO feature, after which I rebooted the systems again.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture1-1565019791083.png" alt="picture1"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture2-1565019843536.png" alt="picture2"></p> <p>The shared volumes were created to be added in the failover cluster from disks that were part of the Cluster Shared Volume (CVS).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture3-1565019886020.png" alt="picture3"></p> <p>As you can see, the Hyper-V Failover Cluster can now be validated.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture4-1565019943404.png" alt="picture4"></p> <p>I then installed the prerequisites for SCVMM including SQL Server 2016, Windows ADK (Assessment and Deployment Kit), and Windows PE (Pre-installation Environment), as well as created specific users and groups in the domain. Note that you must run the SCVMM Setup as an administrator. After answering a few questions, I Iaunched the application and connected to the Hyper-V Cluster.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture5-1565020033667.png" alt="picture5"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture6-1565020039994.png" alt="picture6"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/8/picture7-1565020045576.png" alt="picture7"></p> <p>For more detailed information, check out the following:</p> <ul> <li><a href="https://docs.microsoft.com/en-us/windows-server/failover-clustering/prestage-cluster-adds">Prestage cluster computer objects in Active Directory Domain Services (ASDS)</a></li> <li><a href="https://docs.microsoft.com/en-us/windows-server/failover-clustering/configure-ad-accounts">Configuring cluster accounts in Active Directory</a></li> <li><a href="http://www.garethjones294.com/installing-system-center-virtual-machine-manager-2016-step-by-step-quick-start-deployment-guide/">Installing System Center Virtual Machine Manager (SCVMM)</a></li> <li><a href="https://support.microsoft.com/en-gb/help/2770582/event-id-1222-when-you-create-a-windows-server-2012-failover-cluster">Event ID 1222</a></li> </ul> <p>I hope you found this tutorial helpful in setting up System Center Virtual Machine Manager (SCVMM) in a Microsoft Windows Failover Cluster. Continue to monitor our <a href="/blog">blog posts</a> for more hints that you can use to make your development environment more efficient.</p>https://developer.hpe.com/hpe-oneview-terraform-provider-now-available-with-support-for-hpe-onevie/https://developer.hpe.com/hpe-oneview-terraform-provider-now-available-with-support-for-hpe-onevie/Fri, 02 Aug 2019 20:57:51 GMT<p><a href="https://hpe.com/info/oneview">HPE OneView</a> Terraform provider v1.0.1 now supports HPE OneView 4.1 (REST API version 800). <a href="https://www.terraform.io/">Terraform</a>, by <a href="https://www.hashicorp.com/">HashiCorp</a>, is a popular tool for infrastructure automation and is rapidly increasing in adoption. The Terraform community includes a large number of providers that facilitate a multicloud or hybrid cloud approach to infrastructure management. As part of the HPE OneView comprehensive partner ecosystem, Terraform is a powerful orchestration tool used to create, manage, and update infrastructure resources. These resources may be physical machines, VMs, network switches, containers, or others.</p> <p>HPE OneView Terraform provider automates the provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView. One way administrators could use this to their benefit would be to create a resource topology similar to that of a public cloud on their own physical infrastructure. By creating the resource topology this way, it's easier for administrators to migrate applications and workloads to an on-premise, private cloud as part of a hybrid cloud strategy.</p> <p>FOR MORE INFORMATION</p> <ul> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/releases/tag/v1.0.1">Release content</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/blob/master/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview">Code repository and examples</a></li> <li><a href="https://github.com/HewlettPackard/terraform-provider-oneview/blob/master/README.md">Details about the Terraform provider</a></li> </ul>https://developer.hpe.com/returning-from-the-land-of-ports-and-developers-aka-oscon-2019-in-portla/https://developer.hpe.com/returning-from-the-land-of-ports-and-developers-aka-oscon-2019-in-portla/Wed, 31 Jul 2019 19:12:15 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1564600558780.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1564600558790"></p> <p>Some HPE DEV team members and I just returned from beautiful Portland, Oregon with another successful <a href="https://conferences.oreilly.com/oscon/oscon-or">OSCON</a> in the books. As a Gold sponsor for OSCON for the past two years, the HPE Developer Community always looks forward to attending this premier open source event. I’m always astounded by the technical proficiency of this crowd and their knowledge depth. We also enjoy meeting with developers, like you, who are using Grommet and our open source solutions and discussing your experience with these products face-to-face. This is truly a developer’s conference.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture2-1564600566235.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1564600566237"></p> <p>As a last minute addition to our booth, Blaine, Jeff, Pramod, and I decided to host a <a href="https://github.com/HewlettPackard/hpe-hack-shack-attack">Hack Shack Attack!</a> gaming competition, offering awesome collector Lego sets as prizes. For those of you who don’t know, <strong>Hack Shack Attack!</strong> is our open source, 8-bit arcade game that gives players a chance to tame the IT monster. Since this game was a late addition to our booth, I had to run out to Gamestop the evening before the event to pick up an Xbox controller and write the controller bindings overnight. The game was originally built to support Nvidia Shield controllers, which would be almost impossible to find in a retail store. That being said, I am glad we brought the game and updated the controller bindings, because it proved wildly popular with attendees. We saw a total 1,040 game plays! That IT monster didn’t know what hit him.</p> <p>As usual, Grommet and our Grommet Gremlin, Stack (the purple guy you see in the footer of the <a href="https://developer.hpe.com/">HPE DEV home page</a>), drew a crowd. Pramod Sareddy drummed up a lot of attention for Grommet during his Open Service Broker (OSB) + Grommet for the Enterprise presentation. You can learn more about what Pramod presented in our <a href="/blog/solving-enterprise-devops-and-front-end-challenges-with-open-source-at-o">pre-OSCON blog.</a></p> <p>Talking with developers and their experiences with our products often leads to inspired innovation, both on our part and theirs. It’s discussions like these that help us find better ways to support the HPE Developer Community. We look forward to seeing everyone again at OSCON 2020!</p>https://developer.hpe.com/hpe-oneview-golang-sdk-now-available-with-support-for-hpe-oneview-41/https://developer.hpe.com/hpe-oneview-golang-sdk-now-available-with-support-for-hpe-oneview-41/Tue, 30 Jul 2019 21:05:42 GMT<p><a href="https://hpe.com/info/oneview">HPE OneView</a> takes a software-defined, programmatic approach to managing infrastructure with efficient workflow automation, a modern RESTful API, and a comprehensive partner ecosystem. The HPE OneView Golang SDK v1.0.0 now supports HPE OneView 4.1 (REST API version 800). The HPE OneView Golang SDK extends the HPE OneView API language support to include the <a href="https://golang.org">go language</a>, which is rapidly increasing in adoption and a popular favorite for cloud native applications.</p> <p>The SDK allows go developers to programmatically control HPE OneView managed resources using an infrastructure-as-code approach for physical compute, storage, and fabric resources. Benefits of this infrastructure-as-code approach include, complete datacenter automation, consistent reproducibility, versioning, and roll back.</p> <p>FOR MORE INFORMATION</p> <ul> <li><a href="https://github.com/HewlettPackard/oneview-golang/releases/tag/v1.0.0">Release content</a></li> <li><a href="https://github.com/HewlettPackard/oneview-golang/blob/v1.0.0/CHANGELOG.md">List of supported resources and changes</a></li> <li><a href="https://github.com/HewlettPackard/oneview-golang">Code repository and examples</a></li> </ul>https://developer.hpe.com/hpe-dev-and-design-gear-up-for-august-conferences-jsconf-and-react-rally/https://developer.hpe.com/hpe-dev-and-design-gear-up-for-august-conferences-jsconf-and-react-rally/Tue, 30 Jul 2019 16:25:53 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1564504318351.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1564504318352"></p> <p>HPE DEV and Design engineers are looking forward to attending two upcoming conferences – <a href="https://2019.jsconf.us/">JSConf</a> and <a href="https://www.reactrally.com/">React Rally.</a> HPE Chief Designer, Chris Carlozzi, as well as UX Designer/Developer, Eric Soderberg, and HPE Senior Open Source Developer, Shimrit Yacobi, will be participating in panel discussions at JSConf. HPE DEV lead, Alex Mejias and members of his software development team will staff the HPE DEV area at React Rally.</p> <h2>JSConf US 2019</h2> <p><a href="https://2019.jsconf.us/">JSConf US 2019</a> is a unique developer conference that invites the JavaScript community to spend three days listening to impactful speaker sessions, communing with industry leaders, and participating in lots of <a href="https://2019.jsconf.us/about/activities/">fun activities.</a> JSConf US is the only conference where attendees can learn how to push their favorite language beyond the confines of the browser and into robots and video games.</p> <p>The JSConf format provides <a href="https://2019.jsconf.us/schedule/#first-day-of-talks">two different tracks for attendees.</a> JavaScript pioneer, Sitepen, a modern web consultancy specializing in Enterprise JavaScript, is sponsoring one track, and Hewlett-Packard Enterprise (HPE) is sponsoring the other. While the Sitepen track includes sessions from a curated set of speakers, HPE’s track features a unique format that encourages anyone to speak on a first-come/first-speak basis. These talks, some of the most exciting at JSConf, tend to cover the full range of web development and the world of JavaScript.</p> <p><a href="https://jsconf.com/">Historically, JSConf</a> has been the launch pad for many web innovations. Their call-for-papers process is designed to shine the spotlight on the latest ideas and future leaders of the JavaScript world. This volunteer-run conference focuses on shared social experiences and does its best to make a difference in the community. Some examples include offering scholarships to attend the conference, serving vegetarian food alternatives, and providing childcare services during the conference. JSConf will be held August 12-13 in Carlsbad, CA.</p> <h2>React Rally</h2> <p>Another community conference HPE DEV and Design engineers are looking forward to is <a href="https://www.reactrally.com/">React Rally.</a> This conference focuses on the React ecosystem and topics that are interesting to React developers. Its friendly, welcoming atmosphere, engaging talks from new and established speakers, and opportunities for attendees to chat with industry leaders makes React Rally a popular event among those in the React community.</p> <p>A unique attribute of this conference is the audience and speaker interaction. There are no Q&#x26;A’s after the talks. Instead, audience members are encouraged to discuss issues with speakers during planned downtime sessions; thereby facilitating more effective sharing of perspectives.</p> <p>With this goal in mind, React Rally features a lot of downtime throughout the day where speakers and the audience can interact informally in a more relaxed atmosphere. This unstructured time is valued just as much as the structured sessions, and it’s referred to as the hallway track. At other conferences, attendees sometimes skip a talk in order to hang out in the hallway and talk to all the interesting people you meet at the conference. Organizers didn’t want React Rally attendees to have to choose between sessions and informal conversations, so they put a lot of emphasis on break times, encouraging this type of discourse.</p> <p>If you happen to be going to React Rally August 22-23, 2019 in Salt Lake Utah, keep an eye out for HPE DEV members like Alex Mejias. The HPE DEV team is looking forward to being there to engage in conversations with you.</p> <h2>Keep up with our HPE DEV and Design activities</h2> <p>HPE DEV and Design is here to support the HPE Developer Community. One of the best ways we can do this is through meet-and-greet opportunities like these two developer conferences. Keep up with where you can find us by checking our <a href="https://developer.hpe.com/events">HPE DEV Events</a> page regularly.</p>https://developer.hpe.com/2019-July-16/https://developer.hpe.com/2019-July-16/Tue, 16 Jul 2019 05:00:00 GMThttps://developer.hpe.com/deploy-a-full-stack-application-on-netlify-that-includes-a-cicd-pipeline/https://developer.hpe.com/deploy-a-full-stack-application-on-netlify-that-includes-a-cicd-pipeline/Fri, 12 Jul 2019 15:44:24 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/use-case-graphic_continuous-delivery-1562947940343.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562947940344"></p> <p>Continuous Integration/Continuous Delivery &#x26; Deployment (CI/CD) is becoming a software development practice in which you build, test, integrate, deliver, and deploy software every time a developer or a development team merges code into a particular branch. Continuous integration, automated testing, and automated deployment capabilities allow software to be developed and deployed rapidly, reliably, and repeatedly with minimal human intervention. This blog helps you to understand in detail how to enable a CI/CD pipeline for a full stack application. In this blog post, I will show you how to host a static website on Netlify, including how to set up continuous deployment.</p> <h2>Project structure:</h2> <p>In order to deploy the site on Netlify, a few configurations need to be added to the project directory to enable the CI/CD pipeline.</p> <p>For the purpose of this example, let’s assume that you have an application already created using the <code>create-react-app</code> cli tool.</p> <p>At the root of the project, add the netlify.toml file and include the following text.</p> <pre><code>[build] command = "yarn build" publish = "dist" </code></pre> <p>Create a folder called <code>dist</code> at the project root so that when netlify starts its build command, it automatically copies the compiled binary files into a <code>dist</code> folder. You can also delegate this task to netlify and have it create a build folder during compilation using commands provided as part of the build command.</p> <p>Your package.json file should look like this when the deployment process is complete.</p> <pre><code>"scripts": { "start": "react-scripts start", "build": "run-p build:**", "build:app": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, </code></pre> <p>Push all these changes to your Git repository to move on to the deployment part of the exercise.</p> <h2>Getting started on Netlify:</h2> <p>In this section, I will walk you through how easy it is to launch your site on Netlify. If you are not already a Netlify user, go ahead and sign up for free <a href="https://app.netlify.com/signup">here</a> first.</p> <h2>Step 1: Create new team</h2> <p>Each Netlify site belongs to a team, even if it’s only a team of one. Teams can have multiple Netlify users as members, and a Netlify user can be a member of multiple teams.</p> <h2>Step 2: Create new site from Git</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture2-1562946531885.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946531886"></p> <blockquote> <p>a) Connect your Git provider where the source code is hosted. Once you authorize Netlify to access the Git provider account, Netlify app is installed in your GH profile to allow access specifically to a repo or all repos hosted under the Git account.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture3-1562946579014.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946579015"></p> <blockquote> <p>b) <strong>Pick</strong> a repository or <strong>Select All</strong> repositories to avoid doing this step for every repo.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture5-1562946635493.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946635494"></p> <blockquote> <p>c) This is where you provide all of the deployment settings. Your build settings should match those found in the build directory provided in the netlify.toml file that was added at the beginning of the exercise.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture6-1562946692251.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946692252"></p> <p>Click on the <code>Deploy site</code> button and watch the deployment progress in the CLI provided.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture7-1562946739478.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946739479"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture8-1562946768653.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562946768653"></p> <p>At this point, you can preview the deployment for this specific build or go back to the home page on app.netlify.com site to find the default url allocated on the Netlify domain.</p> <p>The URL should look like this <code>https://something-abc-38642d.netlify.com</code></p> <p>If your app needs to access any other routes apart from ‘/’ route, the above example indicates a problem. To fix this, add a ‘_redirects’ file to the public folder in your project directory where the index.html also resides. This allows any other routes to work with the app. Without this, you will end up landing on a “page not found” or you may be redirected to the home page.</p> <p>_redirects:</p> <pre><code>/* /index.html 200 </code></pre> <p>Having completed the above, you should now have a working frontend application. But when you want to have your server running on the same host, you need to take a few extra steps, which I’ll explain next.</p> <h2>Server side setup:</h2> <p><strong>Pre-requisites:</strong> You need to install the netlify-lambda package to enable serverless lambda functions for running the API. Since lambda functions run as service from the same host, you also need two more dependencies: 1) a <code>http-proxy-middleware</code> package to have localhost proxy setup, and 2) a <code> npm-run-all</code> package to run all build commands in parallel.</p> <p>Once you install the <code>netlify-lambda</code> and <code>http-proxy-middleware</code> packages, update the package.json file to reflect the scripts tag.</p> <pre><code> "scripts": { "start": "react-scripts start", "start:lambda": "netlify-lambda serve src/lambda", "build": "run-p build:**", "build:app": "react-scripts build", "build:lambda": "netlify-lambda build src/lambda", "test": "react-scripts test", "eject": "react-scripts eject" }, </code></pre> <p>Now, add a setupProxy.js file inside the <code>src</code> directory and add the following code:</p> <p><strong>Note:</strong> You do not need to import this file anywhere. It is automatically registered by create-react-app when you start the development server.</p> <pre><code> const proxy = require('http-proxy-middleware'); module.exports = function(app) { app.use(proxy('/.netlify/functions/', { target: 'http://localhost:9000/', "pathRewrite": { "^/\\.netlify/functions": "" } })); }; </code></pre> <p>At the root of the project, update <code>netlify.toml</code> file with the following code:</p> <pre><code>[build] command = "yarn build" functions = "functions" publish = "dist" </code></pre> <p>Now, add a <code>lambda</code> folder under the <code>src</code> directory to start setting the server side functionality. For simplicity, we will write a simple express.js app with few routes.</p> <pre><code> // DB connection const mongoose = require('mongoose'); mongoose.connect(process.env.MONGODB_URI, { useNewUrlParser: true }); const express = require("express"); const cors = require('cors') const serverless = require('serverless-http'); const app = express(); const router = express.Router(); const userSchema = mongoose.Schema({ name: {type: String, required: true}, email: {type: String}, }); const User = mongoose.model('User', userSchema); app.use(express.json()); app.use(express.urlencoded({extended:true})); app.use(cors()); router.get('/ping', (req, res) => { res.send('pong!'); }); // Get all user records router.get('/users, (req,res) => { User.find() .then( users => res.send(users)) .catch( err => { res.send(err); }); }); } }); // Model routes app.use(`/.netlify/functions/api`, router); module.exports = app; module.exports.handler = serverless(app); </code></pre> <p>With this setup, when Netlify runs the build command as part of deployment, it builds the frontend app into the dist directory and places the API into the functions directory.</p> <p>You can make GET requests on the API with <code>http://localhost:9000/.netlify/functions/api/users</code> if you are running locally or <code>https://something-abc-38642d.netlify.com/.netlify/functions/api/users</code> if you are running in production mode. Custom domains can be setup on the Netlify site if you want your site to run on a non-Netlify domain. As a bonus, Netlify automatically enables an SSL/TLS certificate with the help of Let’s Encrypt.</p> <p>Now, every time you push changes to the master (or a PR is created to merge a particular branch into the master), the Netlify CI/CD pipeline automatically builds the code and deploys it into production.</p> <p>CI/CD is integral in the building and deployment of software today, what with our smaller teams, constant changes, fast and real-time feedback, and quick app deployment. Not only does CI/CD provide clear benefits to businesses but also to stakeholders such as product owners, development teams, and end-users. While building a CI/CD pipeline requires additional effort, once the CI/CD process stabilizes within an organization, it offers substantial advantages, such as reduced costs, faster deployments and increased ROI. In addition, with CI/CD automation, developers can invest more time building better software solutions rather than spending the time on code integration and delivery.</p> <p>The CI/CD methodology facilitates the building and improvement of great apps with faster time to market. Improved automation also allows for a more streamlined app development cycle, enabling developers to receive feedback quicker to help them build better, more consistent apps.</p> <p>I hope this blog helps you to set up the CI/CD pipeline using Netlify for any of your personal projects or a project setup for an entire team. Please continue to follow our <a href="/blog">blogs</a> for more hints and tutorials designed to help you optimize your development environment.</p>https://developer.hpe.com/the-power-of-single-sign-on-with-hpe-oneview/https://developer.hpe.com/the-power-of-single-sign-on-with-hpe-oneview/Thu, 11 Jul 2019 19:44:24 GMT<p>An infrastructure management tool, like <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView,</a> allows IT to do most day-to-day infrastructure administration tasks. But there are cases where other related tools can be useful too. It can be tedious to remember how to log on to all these different tools. Sure, you can use a password safe tool, or (please don’t do this…) a TXT file edited with Notepad.</p> <p>A better way is to use a single sign-on between different tools, meaning that you log on once with the most encompassing tool, and then drill down to other dependent tools without having to log on again. This is very straightforward when you use a GUI that allows you to follow hyperlinks, but it can also be done with APIs and your favorite programming language.</p> <p>In this article I will show you how to use <a href="https://www.getpostman.com/">Postman</a> to work with REST APIs. I will begin with <a href="https://buy.hpe.com/b2c/us/en/software/converged-infrastructure-management-software/converged-infrastructure-management/oneview-management-software/hpe-oneview-global-dashboard/p/1009187269">HPE OneView Global Dashboard</a> which gives you an overall view of a datacenter, one that could be very large (up to 20,000 servers are supported in the latest version). From there, I will drill down with single sign-on (SSO) to an HPE OneView appliance. And from there, to the <a href="https://developer.hpe.com/platform/ilo-restful-api/home">Redfish API</a> provided by the iLO of a server – all without having to pass credentials again after the initial logon.</p> <p>So, let’s start with the HPE OneView Global Dashboard and log on with a POST call to /rest/login-sessions with a few headers shown below and a body set to this JSON content: {"userName":"your_username","password":"your_password"}</p> <p>You should see a response like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1562874385556.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874385558"></p> <p>The important data returned is the content of the token key, or sessionID, which is the same value. We will carry this token in the “auth” header of all subsequent calls to the HPE OneView Global Dashboard API.</p> <p>Let’s look at how to connect to an HPE OneView appliance that was previously registered in our HPE OneView Global Dashboard. We can get a list of appliances with a GET to /rest/appliances, from which we get the UUID of the appliance we want. Here it is shown as 5224a1f9-f272-4501-8fbb-80b3d9c13339.</p> <p>When we do a GET on /rest/appliances/5224a1f9-f272-4501-8fbb-80b3d9c13339/sso we receive a response like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture2-1562874422291.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874422292"></p> <p>Once again, the value we are interested in here is the sessionID. We will use it in the auth header for subsequent calls to the HPE OneView appliance. We did not have to pass a username/password to the HPE OneView appliance, since we received a token from our HPE OneView Global Dashboard authenticated session. We can use this token to do anything we want to this particular HPE OneView appliance. For instance, we can get a list of servers managed by the appliance with a GET call to /rest/server-hardware, this time directing it to the IP address of the HPE OneView appliance instead of HPE OneView Global Dashboard.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture3-1562874460779.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874460780"></p> <p>Now, I’ll show you how to use the Redfish-compliant REST API of the iLO of a server to get more details than what surfaces in HPE OneView. Following the same principle, we will find the UUID of the server we are interested in, and we will make a GET call to /rest/server-hardware/{uuid}/remoteConsoleUrl</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture5-1562874508772.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874508773"></p> <p>From the response, we get a link to the iLO remote console of that server, but we can use the data for other purposes as well. It gives us the IP address of the iLO (192.168.3.105) and a session key (8da4257ecf181c186f2510a03ac2a2fe). With that, we can make calls to the iLO without having to create a session with username/password credentials. For example, we can get the list of all DIMM memory slots in the server and which DIMMs are installed in them.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture6-1562874564543.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874564544"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture7-1562874569155.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562874569156"></p> <p>As you can see, single sign-on is an easy, yet powerful, way to use different tools with minimum hassle. I hope you found this tutorial useful. Please continue to follow our <a href="/blog">blog posts</a> for more ways to optimize your software development environment. I would be very interested to see how you are creating your own shortcuts. Feel free to connect with me and the team on <a href="https://slack.hpedev.io/">Slack</a> to share your experiences and ask questions.</p>https://developer.hpe.com/hpe-discover-las-vegas-attendees-win-big-at-the-hack-shack/https://developer.hpe.com/hpe-discover-las-vegas-attendees-win-big-at-the-hack-shack/Wed, 10 Jul 2019 15:34:34 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1562772944968.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772944972"></p> <p>Folks come to Las Vegas to win, and at HPE Discover ’19, it was no different. The HPE Discover Las Vegas Hack Shack welcomed about 1,125 guests, many of them developers, who came to try their hand at winning some cool prizes such as drones, Raspberry Pi kits, tee shirts, and hats. The Hack Shack proved to be a popular destination, with attendance surpassing even that of an interesting Disney booth situated at the entrance to the event. Here’s how some Hack Shack guests came away as big winners.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture2-1562772953069.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772953071"></p> <p>One opportunity attendees had to win prizes came from participating in Hack Shack technical coding challenges. These challenges ranged from writing simple scripts that displayed the words “Hello World,” to more complicated hacks that required the coder to enable a camera deployed at the network edge to failover in the event of an outage. This latter challenge proved very popular, as it featured products that attendees were keen to learn more about, such as Kubernetes, HPE Edgeline servers, and a fun and innovative facial recognition software app.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture3-1562772960861.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772960862"></p> <p>It didn’t matter which challenge guests attempted. The winners were chosen based on four criteria:</p> <ol> <li> <p>Usefulness</p> </li> <li> <p>Uniqueness</p> </li> <li> <p>Creativity</p> </li> <li> <p>Completeness</p> </li> </ol> <p>First prize winner, Matt Burke, took the <a href="https://www.hpe.com/us/en/integrated-systems/simplivity.html">HPE SimpliVity</a> API challenge, which required the coder to build a simple report that showed the status of on-going backups within a user-defined time period. Matt created a web application using NodeJs as a backend and designed a front end using HTML, CSS, and Javascript. What made his solution stand out was the attention he paid to making it visually attractive and user friendly.</p> <p>Chris Price, who came in second place, completed an <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> challenge using Python to programmatically find how many BL460 Gen10 servers had greater than 128 GB of memory. He included user interaction in his script, enabling the user to select various options, such as blades and memory size.</p> <p>Perhaps one of the most impressive hacks was from third place winner, John Frakes, who completed an HPE OneView challenge using Python. When John arrived, he was unfamiliar with Python. But because he wanted to participate, he took it upon himself to learn some Python basics and not only completed the challenge but also included additional functionality. As part of the solution, John parameterized the queries so that users could change what types of servers were chosen, without having to change the code. He also took the time to ensure that the output was formatted nicely on the screen, instead of simple, plain text. In appreciation of the assistance he received from the HPE DEV team, who helped him find ways to familiarize himself with Python, John agreed to spend some time developing Python language bindings for the HPE OneView Global Dashboard and contribute them to the community.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture4-1562772966574.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772966575"></p> <p>Another way attendees could win prizes at the Hack Shack was to play the 8-bit arcade game, <a href="https://github.com/HewlettPackard/hpe-hack-shack-attack">Hack Shack Attack!</a> This cool, retro-style game focused on taming the IT Monster drew almost 1,400 game plays. The first day winner was Wayne Holland, although Wayne was quickly ousted from that spot the next morning by another attendee, Jared, on his first play through. Determined to regain the top spot on the leaderboard, Wayne played diligently until he became the ultimate winner, receiving his prize at the Wednesday night award ceremony, where beer, sunglasses, and popcorn were enjoyed by all.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture5-1562772973840.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772973841"></p> <p>While there was a lot of friendly competition going on at the Hack Shack, it wasn’t all fun and games. During the event, HPE DEV team members were on hand in the Hack Shack to help educate attendees and supply customers with information they needed to make purchasing decisions. Some of the most popular technical workshops featured HPE OneView and HPE SimpliVity, giving attendees a deeper insight on how to code and integrate these products into their businesses. Customers were delighted to have the opportunity to chat one-on-one with product architects. And while everyone was excited to take home a prize or at least a little swag, probably the most important takeaway guests appreciated was learning how they could extend HPE product capabilities through APIs. Learning that they could do more with their existing investments made all the participants feel as if they were winners.</p> <p>For those of you who missed any of the sessions or just want to go back and review something, we’ve made all the <a href="https://lv19.hpedev.io/">workshop session presentations and challenges available on our HPE DEV Hack Shack HPE Discover Las Vegas 2019</a> site. Check it out. You can even still play <a href="https://github.com/HewlettPackard/hpe-hack-shack-attack">Hack Shack Attack!,</a> which is available on GitHub.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture6-1562772980984.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562772980985"></p>https://developer.hpe.com/helping-connect-the-hpe-dev-community/https://developer.hpe.com/helping-connect-the-hpe-dev-community/Wed, 10 Jul 2019 15:26:11 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/connecting-community2-1562774149692.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1562774149709"></p> <p>Hey there… Dale Rensing here. You may remember my name as the editor of this newsletter, but I haven’t really introduced myself properly. Although I’m fairly new to the Hewlett Packard Enterprise (HPE) Chief Design Office, I’ve been around the technology industry for quite some time. I’ve been lucky in my career to often find myself working with teams of highly-talented engineers engaged in cutting-edge technology, getting to report on their innovation and extrapolating its value in customer terms. As a writer and the news editor of the HPE Newsletter and the HPE DEV website, my job now is to connect you with all activities within HPE DEV.</p> <p>HPE DEV builds, communicates, and collaborates with developers and operators in an open community, with the aim of making the deployment of applications across traditional and cloud infrastructures simple and effortless. Part of my job includes working hand-in-hand with HPE DEV engineers, encouraging them to share news and tutorials that are pertinent to the overall community. I also report on different topics – chasing news stories at events with Sir Hackington and bringing more awareness to open source projects from HPE, such as Grommet.</p> <h2>Methods to Facilitate Engagement</h2> <p>The HPE DEV website and our monthly Newsletter were put in place to enable better communication and collaboration between HPE DEV developers, UX designers, and you. You may have noticed some recent, minor modifications to these platforms, although nothing overly dramatic. I believe in the adage “If it ain’t broke, don’t fix it.” Still, that doesn’t mean there isn’t room for continued improvement. Over the next few months, you’ll see me reaching out in different ways to learn more about how you prefer to engage. I’d like to understand more about who you are and what you do. You may see a couple of quick surveys come your way. I encourage you to take the time to fill them out so I can learn more about your preferences and find easier ways to engage.</p> <p>Part of facilitating collaboration is helping you get to know the HPE DEV team members and what roles they play. To that end, I’m in the process of developing some short videos that introduce the different team members to you. You’ll see these come out in subsequent newsletters. If there are other suggestions you might have to help foster more collaboration within the community, I’d love to hear about them.</p> <p>I look forward to engaging with you more closely as we grow the HPE DEV community. Please feel free to connect with me on Slack at <a href="https://slack.hpedev.io/">hpedev.slack.com</a> at @Dale Rensing if you have something specific, you’d like to discuss. You can also follow me and the team on Twitter <a href="https://twitter.com/HPE_Developer?lang=en">@HPE_DevCom.</a></p>https://developer.hpe.com/oneview-global-dashboard-101-chock-full-of-useful-features/https://developer.hpe.com/oneview-global-dashboard-101-chock-full-of-useful-features/Tue, 18 Jun 2019 20:32:33 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1560890766706.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1560890766707"></p> <p>To simplify the management of hybrid cloud infrastructures, the <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a> Global Dashboard (OVGD) provides a unified view of the health and inventory of Hewlett Packard Enterprise (HPE) servers, profiles, enclosures, <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a> frames, and HPE 3PAR storage systems across data centers.</p> <p>HPE just released OVGD version 1.8, and I’m excited to tell you about what’s new. Can you believe it? This is our 9th release in less than three years! Our 10th is planned around OVGD’s 3rd birthday in October of this year. The team prides itself on having met every release on time. We are also happy with the features in each release of the product.</p> <p>The OVGD team designs software that helps you tame your IT monster, but sometimes features can sneak in unnoticed by users. Let’s take a quick look back and examine some of these newer releases to see if there is something you didn’t know about or a feature you have used and would like to comment on. We constantly strive to make the product better in the hopes of making you more efficient. So, keep us in mind and don’t hesitate to reach out via <a href="https://slack.hpedev.io/">Slack</a> or <a href="https://twitter.com/HPE_Developer">Twitter</a> if you think our product team can help you.</p> <h2>Data Aggregator</h2> <ul> <li>OVGD supplies data about: <ul> <li>Enclosures</li> <li>Server hardware (You get loads more data with 1.8! Just click the Server hardware page, then choose a particular resource and view all the data in the detailed pane on the right side of your screen).</li> <li>Server profiles</li> <li>Server profile templates</li> <li>Storage systems</li> <li>Storage pools</li> <li>Volumes</li> <li>SAN managers</li> <li>SANs</li> <li>Converged systems</li> <li>Appliance alerts (critical only)</li> </ul> </li> <li>Using single sign-on, you can log on and use HPE OneView/Composer to manage a resource with a single click!</li> <li>With all the data provided, OVGD can be your one-stop shop to see your data center’s health, and help you get where you need to go to fix any issues or find out more information.</li> <li>Using the dashboard and critical alert reporting, you can quickly understand system health across data centers and use single sign-on to “drill down” to wherever the problem resides.</li> <li>You can click on the wrench icon on each resource page to decide what columns will be displayed by default to ensure you get the view you need every time you login.</li> <li><strong>New in Version 1.8</strong> <ul> <li>This version jumps up the scale numbers! We are going from supporting 50 HPE OneView and/or Composers to 75, with up to 20,000 servers and 25 concurrent users.</li> <li>We have a new resource page for your Interconnects.</li> </ul> </li> </ul> <h2>Report Generator</h2> <ul> <li>We’ve got sixteen out-of-the-box reports that help you view your data center through different lenses.</li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture2-1560890771858.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1560890771858"></p> <ul> <li> <p>Reports have a lot of flexibility hidden underneath those ellipses you see on the screen.</p> <ul> <li><strong>Manage Report Content,</strong> with the same familiar wrench icon, lets you choose which columns to display.</li> <li><strong>Export</strong> allows you to export the data in CSV format. It also reflects any customizations you’ve made, such as search, sort, filter, and column customizations.</li> <li><strong>Schedule</strong> enables a report to be emailed to whomever you’d like at a configurable cadence (with the customizations mentioned above).</li> <li><strong>Mail</strong> allows you to mail the report right at the very moment it is ready (again, including any customizations made to the report).</li> <li>If you want to schedule an email with a search applied, a column sorted a certain way, columns added, or filters applied, you’ll need to first <strong>Save</strong> a custom report and then schedule it. This feature allows you to create custom reports to suit your team’s needs. For example:</li> </ul> <p>Say Dave is responsible for keeping track of your Gen10s, and Wendy is in charge of your Gen9s and Gen8s. Create the ‘Dave Server Inventory’ report with a filter selected to limit the report and schedule it to arrive at work when he does. You can then do likewise for Wendy and her responsibilities.</p> <ul> <li><strong>Delete</strong> is for custom reports that you no longer need.</li> </ul> </li> <li> <p>New in Version 1.8</p> <ul> <li>Two new reports, with all of the same flexibility mentioned above. <ul> <li>Interconnect Inventory</li> <li>Enclosure Inventory</li> <li>See the health and status of all your interconnects and enclosures, respectively.</li> </ul> </li> </ul> </li> </ul> <p>OVGD 1.8 is now available, and we are excited to continue delivering a product that helps you tame your IT monster. We are driven to solve the problems you are facing. Nobody knows those problems better than you, so please get in touch with us to help us best understand your needs.</p> <p>If you want to know more about OVGD, you might want to check out this product <a href="https://www.youtube.com/watch?v=qsmNvNoy-qw">overview video</a> or go <a href="https://buy.hpe.com/b2c/us/en/enterprise-software/converged-infrastructure-management-software/converged-infrastructure-management/oneview-management-software/hpe-oneview-global-dashboard/p/1009187269">here</a> to download it and try it yourself.</p>https://developer.hpe.com/solving-enterprise-devops-and-front-end-challenges-with-open-source-at-o/https://developer.hpe.com/solving-enterprise-devops-and-front-end-challenges-with-open-source-at-o/Thu, 13 Jun 2019 21:11:32 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1560460835957.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1560460835964"></p> <p>For over 20 years, the <a href="https://conferences.oreilly.com/oscon/oscon-or/public/content/about">O’Reilly Open Source Software Conference (OSCON)</a> has been the focal point of the open source movement, earning a reputation as the destination for all things free and open. OSCON welcomes anyone who’s passionate about open source; from software developers, designers, and architects, to activists, hackers, and geeks. This year the event will take place in Portland, Oregon, from July 15 – 19, 2019.</p> <p>Some may be surprised to see Hewlett Packard Enterprise (HPE) at an event like OSCON, but those “in the know” will recognize HPE as a leading innovator who has a lot to offer open source software development and design teams. Attendees will get a chance to see how an HPE-developed, easy-to-use open source component library, like <a href="https://v2.grommet.io/">Grommet,</a> can be used to create responsive, mobile-first projects.</p> <p>At this year’s OSCON event, Pramod Sareddy, Full-stack Developer for HPE DEV, will present <a href="https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/78263">“Solving enterprise DevOps and front end challenges with open source”.</a> For those planning to attend, make sure you catch his talk starting at 11:00 am on Wednesday, July 17th. Pramod will also show the audience how Open Service Broker (OSB) can save time and effort by exposing a web development framework.</p> <p>OSCON attendees can expect to hear from innovative programmers, talented managers, and senior developers who are all doing amazing things with open source. The event features over a hundred sessions that cover a full range of open source languages and platforms, including practical tutorials designed to enhance technical skills. With fun evening events and receptions, Birds of a Feather sessions, awards ceremonies, late night parties, and OSCON activities around town, there will be plenty of learning and networking opportunities for everyone.</p> <p>Check out all the <a href="https://conferences.oreilly.com/oscon/oscon-or/schedule/2019-07-15">sessions</a> and conference <a href="https://conferences.oreilly.com/oscon/oscon-or/public/schedule/stype/1350">events</a> that are planned. I hope to see you there.</p>https://developer.hpe.com/2019-June-12/https://developer.hpe.com/2019-June-12/Wed, 12 Jun 2019 05:00:00 GMThttps://developer.hpe.com/an-insiders-look-into-the-hpe-aspire-2019-experience/https://developer.hpe.com/an-insiders-look-into-the-hpe-aspire-2019-experience/Tue, 11 Jun 2019 17:43:54 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/kubekon-1560274711136.png" alt="kubekon"></p> <h1>An Insider’s Look into the HPE Aspire 2019 Experience</h1> <p><em>By Pramod Sareddy</em></p> <p>Recently, I was fortunate enough to join over 2,000 Solution and Technology Architect professionals who gathered at the <a href="http://www.hpeaspire.com/">HPE Aspire 2019</a> internal employee and partner conference. The event was held May 19-23 at the <a href="https://www.swandolphin.com/">Swan and Dolphin Resort</a> in Orlando, Florida. While there, event attendees enjoyed four days focused on growing their Hewlett Packard Enterprise (HPE) product and solution technical expertise and connecting with HPE engineers and subject matter experts (SMEs), as well as other industry specialists.</p> <p>The HPE DEV booth shared space with other HPE and sponsor booths in the HPE Aspire Showcase. In the booth we were able to meet, collaborate, and network with event sponsors and other members of the HPE community, including <a href="https://techpro.hpe.com/hpelogin.aspx?HPPSESSION=NO">Tech Pro presales support</a>. The theme for this year’s show was "HPE Aspire Legends" and revolved around legendary characters from movies, sports, music, and television. Jack Sparrow from Pirates of the Caribbean even made a personal appearance! During the keynote, we all enjoyed a dance performance. The event hosts thought of everything – even providing us with HPE-branded bread as a snack between meals.</p> <p>In addition to fielding questions from sales reps while working the booth, I was approached by a number of HPE employees interested in the <a href="https://developer.hpe.com/">HPE DEV portal</a>. HPE software engineers were happy to see us there and showed a strong interest in digging deeper and contributing content. Based on our discussions, some are now considering posting scripts that they developed to connect their products to <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a>. <a href="https://www.arubanetworks.com/">Aruba Networks</a> engineers expressed interest in becoming more closely aligned with the HPE DEV portal as well. One HPE presales engineer was very excited to see that specific API documentation his developer customers had been searching for was easily accessible on HPE DEV. I pointed out that we had not only HPE OneView and <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a> material, but also information on other platforms, like <a href="https://www.hpe.com/us/en/integrated-systems/simplivity.html">HPE SimpliVity</a> and <a href="https://www.hpe.com/us/en/storage/nimble.html">HPE Nimble Storage</a>, including all the supporting tools and modules.</p> <p>All in all, it was a great opportunity to showcase the wide variety of resources HPE DEV provides.</p>https://developer.hpe.com/hpe-dev-at-kubecon-cloudnativecon-europe-2019/https://developer.hpe.com/hpe-dev-at-kubecon-cloudnativecon-europe-2019/Tue, 11 Jun 2019 17:40:44 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/grouppicture-1560274699625.png" alt="group-picture"></p> <h1>HPE DEV at KubeCon + CloudNativeCon Europe 2019</h1> <p><em>Didier Lalli</em></p> <p>For three intense days, <a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/">KubeCon + CloudNativeCon Europe 2019</a> was held in Barcelona, Spain, May 20-23, 2019. The show, which took place in the Fira Gran Via, drew more than 7,000 attendees, a substantial increase from the 4,400 who attended last year. A cross-organizational team from Hewlett Packard Enterprise (HPE) staffed the event booth with representatives from the product business units, HPE DEV, HPE Storage, HPE Pre-Sales, and HPE Pointnext services. Though the team came from different groups within the company, everyone had one goal in mind: promote and educate attendees on what HPE offers in the areas of containers and Kubernetes (k8s).</p> <p>I was fortunate to attend the conference, meeting other attendees who were all extremely engaged. Conversations at the booth revolved around HPE’s offerings for storage, <a href="https://www.hpe.com/us/en/solutions/infrastructure/composable-infrastructure.html">composable infrastructure</a>, and HPE Pointnext services, as well as the DevOps value offered via the <a href="https://developer.hpe.com/">HPE DEV community</a>. I talked with people from small startups, who were unfamiliar with the split between Hewlett Packard (HP) and HPE, and those who were unfamiliar with what the acronym “HPE” meant. I also spoke with attendees from large companies, like Airbus and Orange Labs, looking to manage their infrastructure as code.</p> <p>As always, booth swag was in high demand, especially our various stickers. We held a drawing each day for prizes and the winners were selected from new subscribers to the HPE DEV newsletter. We also offered attendees the <a href="http://www.techdemand.io/whitepaper/solutions/hybrid-cloud-management-for-dummies/">Hybrid Cloud Management for Dummies</a> booklet, which proved very popular. Our local host, Jessica Navas, helped with translation to ensure communications ran smoothly both in Spanish and English. With the great weather, intriguing conversations, and incredible tapas, it was an amazing show to attend.</p>https://developer.hpe.com/elaborate-the-monitoring-of-kvm-based-private-clouds-using-open-source-t/https://developer.hpe.com/elaborate-the-monitoring-of-kvm-based-private-clouds-using-open-source-t/Wed, 29 May 2019 21:28:19 GMT<p>Today, many cloud service providers offer OpenStack-based private cloud management solutions that can be delivered as a Software-as-a-Service (SaaS). In SaaS delivery models for OpenStack-based private clouds, there are two main parts: the OpenStack control plane and the node components. The control plane is generally hosted in the cloud and managed, maintained, and operated by the cloud service provider. The compute, network, and storage resources of the private cloud platforms, however, remain on-premises where customers can maintain greater control and security. The OpenStack control plane REST API endpoint is the only control plane component accessible to customers. Host agents are typically installed on compute servers in the data center to connect the customer’s on-premises compute servers to the hosted control plane through a secure communication channel.</p> <p>While the OpenStack-based control plane is hosted and managed by the cloud service provider, IT professionals continue to report they need guidance about the on-premises components. They want more guidance around what they should monitor with their open source monitoring software, so they know when there is a problem that affects their OpenStack-based private cloud environment.</p> <p>In this blog, you will learn how to extend your ability to monitor KVM-based private clouds using the enterprise open source monitoring platform Nagios with NRPE (Nagios Remote Plugin Executor) add-on.</p> <h2>About Nagios – the open source monitoring software</h2> <p>Nagios is an open source enterprise monitoring platform that allows you to monitor systems, networks and infrastructure with alerting services for servers, switches, applications and services.</p> <p>Installation and configuration of Nagios core and NRPE add-on are beyond the scope of this article. More information about Nagios can be found <a href="https://www.nagios.org/">here</a> and a Nagios core installation and configuration guide can be found <a href="https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/">here:</a></p> <p>Step-by-step instructions to install Nagios Core server, NRPE add-on and NRPE daemon on Linux-based (CentOS7 and Ubuntu) are available here:</p> <p><a href="https://www.digitalocean.com/community/tutorials/how-to-install-nagios-4-and-monitor-your-servers-on-centos-7">CentsOS7</a></p> <p><a href="https://www.digitalocean.com/community/tutorials/how-to-install-nagios-4-and-monitor-your-servers-on-ubuntu-14-04#monitor-an-ubuntu-host-with-nrpe">Ubuntu</a></p> <br/> <h2>Services and resources to check for a KVM-based private cloud platform:</h2> <ul> <li>Compute servers general health monitoring <ul> <li>Reachability of KVM compute servers from Nagios monitoring server</li> <li>Health of KVM compute servers (CPU load, processes, storage, memory)</li> <li>NTP clock on KVM compute servers</li> <li>Reachability of proxy server from KVM compute servers (if a proxy server is used to reach the Internet)</li> <li>DNS hostname resolution for OpenStack control plane API endpoint from KVM compute servers</li> </ul> </li> <li>OpenStack control plane REST API endpoint status <ul> <li>Check status of OpenStack Control plane REST API endpoint</li> </ul> </li> <li>Secure channel communication between on-premises private cloud platform and hosted control plane <ul> <li>Check for error on host agent logs</li> <li>Check for error on secure communication agent logs</li> </ul> </li> <li>OpenStack services <ul> <li>Check for OpenStack compute services (Nova services)</li> <li>Check for Networking agent services (Neutron services)</li> <li>Check for block storage volumes services (Cinder services)</li> <li>Check for image services (Glance services)</li> </ul> </li> <li>OpenStack Virtual Machine instance status on private cloud <ul> <li>Check Virtual Machine instance status for private cloud</li> </ul> </li> </ul> <br/> <h2>What does it take to implement this monitoring solution?</h2> <ul> <li>A Nagios core server (the monitoring server) with access to the private cloud compute servers</li> <li>Installation of Nagios NRPE add-on on Nagios core server</li> <li>Installation of Nagios NRPE daemon on each compute server (the monitored host)</li> <li>Installation of NRPE ‘standard’ plugins on each compute server</li> <li>A Linux-based workstation with OpenStack command line interface (CLI) installed</li> <li>Development and testing of custom Nagios check plugins</li> <li>Installation and configuration of custom check plugins on each compute server</li> <li>Configuration of ‘servers’ and ‘service’ definitions on Nagios server (monitoring server) for monitoring the resources and services on remote compute servers</li> </ul> <br/> <h2>Private cloud platform health monitoring with Nagios core and NRPE add-on</h2> <p>Figure 1 and Figure 2 show Nagios Core-based monitoring with NRPE add-on for a KVM-based private cloud services and resources, as well as OpenStack services (nova, neutron, glance, cinder, VM instances status) monitored from a Linux workstation with OpenStack CLI installed:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture-1559166146677.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1559166146679"></p> <p><em>Figure 1 – Health monitoring of a compute server</em></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture11-1559166225373.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1559166225374"></p> <p><em>Figure 2 – Health monitoring of OpenStack control plane services</em>In this scenario, standard Nagios NRPE check plugins are used to monitor compute server health (processes, CPU load, storage disk space, reachability, NTP time, DNS resolution). HPE DEV developed a few simple bash scripts and leveraged several others from existing sample codes available in GitHub to create custom checks (also known as plugins). These custom checks are used to monitor the status of the OpenStack services such as Nova, Cinder, Glance, Neutron, and the OpenStack control plane REST API endpoint. They are also used to verify the secure communication channel between the on-premises compute servers and the control plane hosted in the cloud.</p> <h2>Some of the custom checks are shown here:</h2> <p><strong>Custom check to monitor the OpenStack control plane endpoint status</strong></p> <p>Here is a custom NRPE plugin developed to check status of the OpenStack control plane REST API endpoint:</p> <pre><code class="language-bash">#!/bin/bash # #Custom Nagios nrpe plugin #This plugin checks OpenStack control plane REST API endpoint #status # URL="https://&#x3C; OpenStack-control-plane-endpoint-URL>/rest/status" proxysrv="http://&#x3C;proxy-server>:&#x3C;port>" status=$(curl -k -x $proxysrv -H "accept: application/json" -X GET $URL | jq -r ".service") if [ "$status" != "OK" ]; then echo "CRITICAL - OpenStack control plane API endpoint is not responding - $status" exit 2 else echo "OK- OpenStack control plane API is $status" exit 0 fi </code></pre> <br/> <p><strong>Custom check to monitor for error in host agent logs</strong></p> <p>Here is a custom NRPE plugin developed to check for a host agent error running on the compute server:</p> <pre><code class="language-bash">#!/bin/bash # #custom Nagios nrpe plugin #This plugin checks hostagent log for errors # output=`tail -1 /var/log/hostagent.log | grep -i 'error\|failed'` if [ -z "$output" ]; then echo "OK- No error or failure in hostagent.log" exit 0 else echo "CRITICAL- $output" exit 2 fi </code></pre> <br/> <p><strong>Custom check (obtained from GitHub) to monitor OpenStack Nova compute services:</strong></p> <p>Here is a custom NRPE plugin obtained from GitHub to check for the OpenStack Nova compute services status. Very similar custom checks are used to monitor other OpenStack services such as Cinder, Neutron, and Glance.</p> <pre><code class="language-bash">#!/bin/bash #source: https://github.com/taha-bindas/openstack_nagios # #Custom Nagios nrpe plugin for openstack #This plugin checks nova services if they are enable and up or not # export http_proxy=http://&#x3C;proxy-server>:&#x3C;port> export https_proxy=http://&#x3C;proxy-server>:&#x3C;port> # export OS_AUTH_URL=https://&#x3C;OpenStack-control-plane-endpoint> /keystone/v3 export OS_IDENTITY_API_VERSION=3 export OS_REGION_NAME="&#x3C;region-Name>" export OS_USERNAME="&#x3C;admin-username>" export OS_PASSWORD="&#x3C;admin-password>" export OS_PROJECT_NAME="&#x3C;Tenant-Name>" export OS_PROJECT_DOMAIN_ID=${OS_PROJECT_DOMAIN_ID:-"default"} export OS_USER_DOMAIN_ID=default EXIT="GOOD" CMD=$(openstack compute service list | egrep -v "+----|Binary" | awk -F'|' '{print $2,$3,$4,$5,$6,$7,$8}') while read ID BINARY HOST ZONE STATUS STATE REASON do if [ "$STATE" != "up" ] then EXIT="BAD" echo "$BINARY on $HOST is $STATE, reason: $REASON" continue fi done &#x3C; &#x3C;(echo "$CMD") if [ "$EXIT" == "BAD" ] then echo "CRITICAL - not all nova services are up and running: $CMD" exit 2 else echo "OK - all nova services are up and running" exit 0 fi </code></pre> <br/> <p><strong>Custom check (obtained from GitHub) to monitor OpenStack Virtual Machine instance status:</strong></p> <p>Here is a custom NRPE plugin obtained from GitHub to check for OpenStack Virtual Machine instances status for all tenants.</p> <pre><code class="language-bash">#!/bin/bash #Created by: Taha Ali (tahazohair@gmail.com) #Created Date: 4/11/2014 #https://github.com/taha-bindas/openstack_nagios # #Nagios nrpe plugin for openstack #This plugin checks each openstack vm if they are showing #active or not for a given Project/Tenant # export http_proxy=http://&#x3C;proxy-server>:&#x3C;port> export https_proxy=http://&#x3C;proxy-server>:&#x3C;port> # export OS_AUTH_URL=https://&#x3C;OpenStack-control-plane-endpoint> /keystone/v3 export OS_IDENTITY_API_VERSION=3 export OS_REGION_NAME="&#x3C;region-Name>" export OS_USERNAME="&#x3C;admin-username>" export OS_PASSWORD="&#x3C;admin-password>" export OS_PROJECT_NAME="&#x3C;Tenant-Name>" export OS_PROJECT_DOMAIN_ID=${OS_PROJECT_DOMAIN_ID:-"default"} export OS_USER_DOMAIN_ID=default #set -x. CMD=$(openstack server list --all-projects | egrep -v "+---------|ID" | awk -F'|' '{print $3,$4,$5}') echo "$CMD" | while read VM STATUS NETWORK do case $STATUS in ERROR) echo "CRITICAL - $VM is currently in $STATUS state" exit 2 ;; SUSPENDED) echo "CRITICAL - $VM is currently in $STATUS state" exit 2 ;; PAUSED) echo "WARNING - $VM is currently in $STATUS state" exit 1 ;; esac done echo "OK - all vm's are in OK state" exit 0 </code></pre> <p>I hope this article helped you determine how to monitor the health of a KVM-based private cloud compute layer including the service availability in the managed OpenStack control plane portion of the service delivered via SaaS. For more tutorials like this, continue to monitor <a href="https://developer.hpe.com/">hpedev.io.</a> Consider <a href="https://developer.hpe.com/newsletter-signup">subscribing to our newsletter</a> which highlights our latest posts.</p>https://developer.hpe.com/2019-may-29/https://developer.hpe.com/2019-may-29/Wed, 29 May 2019 05:00:00 GMThttps://developer.hpe.com/an-open-service-broker-project-delivers-a-sample-devops-environment-to-a/https://developer.hpe.com/an-open-service-broker-project-delivers-a-sample-devops-environment-to-a/Thu, 23 May 2019 19:07:42 GMT<p>As described by <a href="https://en.wikipedia.org/wiki/DevOps">Wikipedia, “DevOps</a> is a set of software development practices that combines software development (Dev) and information technology operations (Ops).” DevOps shortens the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives. In a DevOps environment, different disciplines collaborate, making quality everyone's “job.” DevOps is often hailed as a giant step towards bringing better software to market faster and many organizations now view it as an important goal to establish an efficient DevOps pipeline and attain a true DevOps environment.</p> <p>Because it is so unlike traditional software development methods, implementing DevOps introduces some challenges. DevOps works differently and the working environment uses distinct virtual machines, tool sets, and utilities. Infrastructure and the surrounding software environment needs to be provisioned from scratch, maintained, and managed differently from ITOps.</p> <p>There are many ways to provision a DevOps environment. This blog describes one method that uses Open Service Broker, AWS CloudFormation, and Ansible. This methodology uses Python, AWS CloudFormation template, and Ansible playbooks. These tools are all put under version control so that you can trace changes and find and debug potential problems. This method has several advantages:</p> <ul> <li> <p>Open Service Broker (OSB) is a service delivery method. For any portal or component that supports OSB API, the service can be brought up to their portal to make it available for its users. <a href="https://www.hpe.com/us/en/solutions/cloud/onesphere.html">HPE OneSphere</a> supports key OSB APIs so you can easily register the DevOps broker to HPE OneSphere to be consumed by HPE OneSphere customers.</p> </li> <li> <p>AWS CloudFormation provides a common language to describe and provision all the infrastructure resources in an AWS cloud environment. Using AWS CloudFormation also makes it easier to manage the resources. For example, a DevOps environment normally requires more than one virtual machine. With AWS CloudFormation, you can group all virtual machines into one <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html">stack.</a></p> </li> <li> <p>Ansible is used to provision tools, utilities, and applications in the infrastructure. The tools and applications may be any software such as Java, Git, Nginx, MariaDB, PHP, and Jenkins. IT admins can centrally manage the Ansible playbook easily and then push out to the DevOps environment. Different versions of applications can be dynamically installed on the fly instead of pre-burned into a virtual machine image.</p> </li> </ul> <h3>Prepare the environment for running OSB service and Ansible</h3> <p>In this first step, prepare the virtual machine environment for hosting the OSB broker service.</p> <ul> <li>Ubuntu 16.04 (the hosting VM) in azure.com was chosen for hosting the OSB broker service. Other versions of Linux (such as CentOS) will also work. The hosting environment can be either in a cloud (such as AWS) or on-prem (such as VMware).</li> <li>In the hosting VM, use ssh-keygen to generate an SSH secure key pair, then create the security key pair in AWS and copy/paste the content of public key over to the AWS key pair. Please remember the key pair name, which will be used later.</li> <li>Make sure that <a href="http://ubuntuhandbook.org/index.php/2017/07/install-python-3-6-1-in-ubuntu-16-04-lts/">python 3.5.2 or above is installed</a> in the hosting VM.</li> <li>Make sure that <a href="https://linuxize.com/post/how-to-install-pip-on-ubuntu-18.04/">pip3 is installed</a> in the hosting VM.</li> <li><a href="https://tecadmin.net/install-ansible-on-ubuntu-16-04-xenial/">Install Ansible</a> in the hosting VM.</li> <li>Make sure the Ansible working folder exists in the hosting VM. <ul> <li>Normally, it is “/etc/ansible”. Create the folder if it does not exist.</li> </ul> </li> <li><a href="https://docs.ansible.com/ansible/latest/reference_appendices/galaxy.html#installing-roles">Install the following Ansible roles in the Ansible working folder</a> in the hosting VM. <ul> <li>apache</li> <li>jenkins</li> <li>php</li> <li>phpunit</li> </ul> </li> <li>Configure Ansible dynamic inventory modules in the Ansible working folder in the hosting VM. <ul> <li><a href="https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py">Download ec2.py</a>.</li> <li>Modify first line of ec2.py to make sure python3 is chosen: #!/usr/bin/env python3.</li> <li><a href="https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.ini">Download ec2.ini</a>.</li> <li>Modify ec2.ini to point at the right regions, for example: regions = us-west-1</li> </ul> </li> </ul> <p>#regions_exclude = us-gov-west-1, cn-north-1.</p> <ul> <li><a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">Install awscli</a> in the hosting VM.</li> </ul> <h3>Get and run the OSB broker service</h3> <ul> <li>Create an OSB working folder in the hosting VM. Use any folder you prefer.</li> <li><a href="https://github.hpe.com/peng-liu/osb-devops">Get the OSB broker service</a> <ul> <li>You can download the zip file and unzip the package to the OSB working folder.</li> </ul> </li> <li>Install the required the packages <ul> <li>sudo pip3 install --no-cache-dir -r requirements.txt</li> </ul> </li> <li>Start the OSB service broker: <ul> <li>python3 osb_template.py</li> </ul> </li> </ul> <h3>Register the DevOps OSB service in HPE OneSphere</h3> <p>Log in to HPE OneSphere instance, choose Settings (see Figure 1), then click on Catalog Registry.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture1-1558638905136.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558638905137"></p> <p>Figure 1. HPE OneSphere Settings</p> <p>Register the OSB broker service in the catalog by clicking the plus (+) sign in the Catalog Registry page.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture11-1558638965078.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558638965078"></p> <p>Figure 2. Register OSB broker service.</p> <p>The broker service now appears in the HPE OneSphere catalog.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture13-1558639040081.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639040081"></p> <p>Figure 3. The OSB broker service appears as osb-devops.</p> <h3>Deploy an instance of the OSB broker service – DevOps environment</h3> <p>The DevOps OSB broker service offers two virtual environments; one is for Jenkins with PHP and PHPUnit, and the other is for an Apache Webserver with PHP.</p> <p>Deploying this broker service provisions two AWS CloudFormation stacks. Each stack has one virtual machine. Corresponding software will be installed into each VM respectively, with all dependent tools and utilities. The deployment will also create security groups for both virtual machines and open ports for Jenkins service and WebServer.</p> <p>Figure 4 shows the screen start deploying an instance of the DevOps environment.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture14-1558639104803.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639104804"></p> <p>Figure 4. Start deploying an instance of the OSB broker service.</p> <p>The deployment of OSB broker service within HPE OneSphere is in progress.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture15-1558639161150.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639161151"></p> <p>Figure 5. The deployment in progress.</p> <p>In the AWS CloudFormation service portal, two stacks (one for Jenkins and the other for the web server) are in the process of being deployed.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture16-1558639251375.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639251376"></p> <p>Figure 6. Showing the DevOps stacks creation in progress.</p> <p>After a while, the deployment of the OSB service broker completes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture20-1558639363748.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639363749"></p> <p>Figure 7. The completed OSB service broker deployment</p> <p>After an instance of the OSB broker service is successfully deployed, you can access the DevOps environment from either the AWS CloudFormation portal or HPE OneSphere portal.</p> <p>In the AWS CloudFormation portal, when you click on each stack, you should see both the SSH connection information and a WebUrl link.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture111-1558639415294.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639415295"></p> <p>Figure 8. AWS CloudFormation portal showing the SSH connection and WebUrl link.</p> <p>In the HPE OneSphere portal, access information is available by clicking on the Access link in the deployment page.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture671-1558639474817.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639474818"></p> <p>Figure 9. Access page shows both SSH connection and Web URL link.</p> <p>To open the Jenkins portal, click on the WebUrl link of the Jenkins stack. Follow the on-screen instructions to log in with a default initial password, and you will be connected to the Jenkins portal.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture551-1558639531779.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639531780"></p> <p>Figure 10. Jenkins portal.</p> <p>A web server home page served by a sample index page (PHP) will open when you click on the WebUrl link of the WebServer stack.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picturel1-1558639581208.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639581209"></p> <p>Figure 11. Web server index page.</p> <h3>Delete the DevOps deployment</h3> <p>You can delete a deployment in HPE OneSphere. The deletion will further delete the DevOps stacks in AWS. This means that the security group and all virtual machines instances will be deleted.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/pictures1-1558639696893.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639696893"></p> <p>Figure 12. Deleting the DevOps deployment from HPE OneSphere.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/picture001-1558639745847.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558639745848"></p> <p>Figure 13. DevOps stacks are being deleted in the AWS CloudFormation portal.</p> <h3>Unregister the OSB broker service</h3> <p>You can also disable an OSB broker service and delete it from the HPE OneSphere portal.</p> <h3>Use the test program to simulate a service consumer</h3> <p>The <a href="https://github.hpe.com/peng-liu/osb-devops">test program (test.py) in the repository</a> calls out the OSB broker service to list the catalog and provision the DevOps environment in AWS. It can be used and modified to include more functions to simulate a portal environment where the OSB broker service is to be registered.</p> <h3>Next Steps</h3> <p>This blog shows a very simple beginning step to automate the DevOps provisioning process using OSB, AWS CloudFormation, and Ansible. Stay tuned for more blog posts regarding this topic, and more mature and complete packages for managing the DevOps environment.</p>https://developer.hpe.com/my-journey-from-student-to-hpe-full-stack-developer/https://developer.hpe.com/my-journey-from-student-to-hpe-full-stack-developer/Thu, 23 May 2019 18:08:07 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/unsquished-brittany-grad-picture-1558709131030.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558709131030"></p> <p>This time last year, I was walking across the stage to receive my diploma. In that moment, I had a million thoughts racing through my head. For example, what is the next step in my journey? What kind of position do I see myself attaining in the next few years? And, most importantly, how do I not fall off this stage in front of all these people! For the past five years, I had a routine: go to class, work, and then study or work on projects/homework. The routine as I knew it would come to an end, and I had no idea what my next steps were going to be. Just like any other new graduate, I was concerned with the uncertainty of landing a job and whether or not those long years spent getting my undergrad were going to pay off.</p> <h2>Landing my first position</h2> <p>Luckily for me, I had the chance to join Hewlett Packard Enterprise (HPE). I had so many questions when I first started working. For example, what was I going to be working on and how should I prepare before my start date? When my first day arrived, I was excited to sit alongside the user experience (UX) team. UX was something that interested me, but since it was not in my course work, I did not know much about the research and the design process. But you learn a lot through on-the-job training. Within a few months, I learned enough to work with another designer on a web application that counted down the days to HPE’s premier event, HPE Discover Madrid. I even incorporated my own ideas into the design!</p> <h2>Learning about Grommet</h2> <p>As a new hire, there was so much that was unfamiliar to me. I knew nothing about the React or <a href="https://v2.grommet.io/">Grommet</a> software development and design tools. I learned to seek help through different resources, such as the <a href="https://slackin.grommet.io/">Grommet Slack community.</a> In the beginning, I was nervous about asking the community for help and advice, but once I posed my first question, I received a quick and helpful response. They always provided great answers, as well as insightful ideas, on how I might be able to implement changes in my code base.</p> <p><a href="https://v2.grommet.io/components">Grommet</a> is an awesome library to use to get started. After getting familiar with the different components of Grommet as well as React, I started to gain more confidence as a developer. Grommet is now the first thing I think of whenever I have a new project to get started on. I tend to think about the different components that Grommet has to offer and how I can leverage them to get the design I want. Once I have an idea of which components I will use, I start looking at the examples that Grommet provides. The Grommet site contains excellent resources, such as a <a href="https://storybook.grommet.io/?path=/story/components--all">storybook</a> which displays an example and code for each component. The majority of the components in Grommet also have a link to <a href="https://codesandbox.io/s/github/grommet/grommet-sandbox?initialpath=box&#x26;module=%2Fsrc%2FBox.js">Code SandBox,</a> an open editor for users to be able to play around with the code as well as see their changes. These tools are very useful, especially for someone who is just starting out with React and Grommet. The editor allows you to play with the different components that Grommet offers and become more familiar with them.</p> <p>In these past nine months, I learned not only about React, Grommet, and many other tools and languages; I also learned more about myself as a developer. Coming straight out of school, I was not sure about the type of work I would enjoy. Now I know that I really enjoy developing front-end applications. I also know I dislike writing tests for applications! (Even though it is a must.)</p> <p>For those of you who are in the same position I was in this time last year, I can assure you that the transition from student to developer is not as nerve-wreaking as it seems. I do have some good advice from what I have learned that may shed some light on your path forward.</p> <ul> <li>Do not be afraid to push yourself. When I was given projects, I thought to myself -- how am I ever going to implement this code?! However, I’ve learned to take each project and break the pieces apart. Take each challenge one step at a time.</li> <li>In school, my goals included turning in a project on time, getting a good grade on a test, etc. Once you start working, things will change. But I still suggest you set goals for yourself. This way you can keep track of how you’re doing. Think of it now, though, as if you are grading yourself!</li> <li>Embrace the opportunity to grow. Getting feedback from other developers is a helpful way to see where your strengths are and what you need to work on. Take the feedback as a learning opportunity given to you by another experienced developer that has most likely been in your footsteps.</li> <li>It’s okay to step out of your comfort zone. I’m guilty of staying in my lane, however, it is important to be able to step out and be heard. As a new developer you may believe that your opinions and thoughts do not matter, but they do, and they should be heard!</li> <li>Last, but not least, write clean code. This is something that I am guilty of not doing. Working on a team keeps me on my feet because the last thing I want to do is commit code to a team project that is not even formatted correctly!</li> </ul> <p>I hope this advice helps anyone else who is on the same path as I was. If you would like to see more blogs for advice and tutorials, check out our <a href="https://developer.hpe.com/">HPE DEV.</a></p>https://developer.hpe.com/github-the-project-management-tool-you-didnt-know-you-wanted/https://developer.hpe.com/github-the-project-management-tool-you-didnt-know-you-wanted/Thu, 23 May 2019 18:00:37 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/hpe-dev-github-1558638303117.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558638303118"></p> <p>The project management landscape is filled with lots of tools and processes that can be used for a wide variety of purposes. Managing a software development operation today, however, doesn’t always lend itself to generalities. Taking in a broad view of a project to understand how all the pieces connect and contribute requires a subtle set of skills. Being able to plan work, identify obstacles, and ensure effective communications is essential for the long-term success of a team. In today’s dynamic, fast-paced industry, this is very difficult to achieve.</p> <h2>Project Management Landscape</h2> <p>Although an overwhelming number of agile project management tools are available today, each person has his or her own preference. Initially, this wouldn’t appear to be a problem, because a team can come together for a singular project and determine what tool they will use. However, due to the dynamic nature of the industry, if you or your colleagues work across different teams, you could easily end up working with multiple tools, each with different processes and terminology. Efficiency and morale plummet because teams end up spending more time and effort learning about the tools that were merely meant to facilitate their work. In larger companies, the use of multiple tools also affects communication with executives, because you now require a third view that summarizes the status of each project across all the tools. When you compound this challenge over multiple quarters/years, it becomes unmanageable.</p> <p>There must be a better solution. As a software development project manager, I was looking for a tool that had the basic functionality needed for agile project management. I also wanted a tool that was:</p> <ul> <li>Easy to use</li> <li>Flexible enough to meet the needs of different teams and objectives</li> <li>Transparent across multiple projects and organizations</li> <li>Able to offer an executive-level view</li> <li>Scalable over time to ensure the team won’t need to switch tools again in the near future</li> </ul> <p>As a project manager, I always try to remove personal opinion-based reasoning from the decision-making process. So, my first question was: What was the best project management tool that meets all of the above requirements? I also tried to think out of the box. What if the solution wasn’t a project management tool? What if it was a software development platform instead?</p> <h2>GitHub</h2> <p>GitHub is the leading software development platform with over 22 million users. For years, my manager/technical lead, Alex Mejias, and I felt the pain of creating/moving cards in a project management tool and then creating issues in GitHub for the developers because that was where the code was managed and maintained. Earlier this year, we decided we had enough. We wanted to try and manage all the aspects of a project within GitHub itself. The switch proved to be successful because GitHub has the basic functionality of any project management tool, but it's so much more than that. The more we used it, the more we realized that GitHub met all the needs listed above. GitHub even offered additional benefits, including:</p> <ul> <li>A complete history of the project located in the same place where the project is being maintained</li> <li>Shared terminology across multiple teams and projects throughout the lifecycle of any particular project</li> <li>A tight integration of issue lifecycle with codebase changes. Pushing code to the project can automatically close issues removing a lot of manual maintenance</li> <li>The fact that most developers already have experience using GitHub due to the high-user base, which makes onboarding a breeze</li> <li>Vast amounts of documentation</li> </ul> <p>Just like any other decision based on a long-term team goal, the complete value of using GitHub as a project management tool will become visible over time through the overall success of the team. Based on previous experience, I believe the benefits of using GitHub to manage digital projects for our current needs far outweigh any other project management tool. This is something new for our Chief Design Office at HPE, and we are still working through all the details. I would love to hear about other teams’ experiences so that we may collaborate and learn from each other. We invite you to <a href="https://slack.hpedev.io/">join us on Slack</a> to share stories.</p>https://developer.hpe.com/see-where-you-can-find-the-open-source-ui-tool-grommet-at-hpe-discover/https://developer.hpe.com/see-where-you-can-find-the-open-source-ui-tool-grommet-at-hpe-discover/Thu, 23 May 2019 17:43:56 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/imageedit_2_3129272736-1558636315560.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1558636315561"></p> <p>The Hewlett Packard Enterprise (HPE) premier event, <a href="https://www.hpe.com/events/discover/">HPE Discover,</a> is just around the corner, and the HPE DEV &#x26; Design team members are busy preparing their sessions for the HPE Hack Shack. Everyone is excited to interact with event attendees and share ideas and helpful hints on how to simplify and improve the software development and design process for today’s applications. A large part of what the group has to offer centers around a specific open source UI development and design tool called [Grommet.] (<a href="https://v2.grommet.io/">https://v2.grommet.io/</a>) Grommet is a React-based library of reusable UI components that help developers and designers create web applications. This open source UI development and design tool simplifies the way web applications are built by providing a package of commonly used interface elements from which developers and designers can choose to use.</p> <p>If you were to walk the HPE Discover show floor and stop by the HPE Hack Shack (DEMO403), you would notice the visual incarnation of Grommet – a purple, gremlin-like character the team endearingly refers to as Stack, the Grommet mascot. You will find him on stickers, posters, slide presentations, large promotional screens, etc. What you might not immediately notice is how Grommet, the UI tool itself, permeates not just this booth, but also other areas of the show floor.</p> <p>For instance, Grommet is the subject of numerous workshops and Hack Shack Challenges. Several of the HPE Design Workshops feature Grommet, including:</p> <blockquote> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18625_8975&#x26;locale=en_US">HSW8625</a> <strong>HPE Design Workshop – User experience research tools and techniques that drive customer experience</strong>.</p> </blockquote> <p>Tuesday, June 18 at 4:30pm.<br> Thursday, June 20 at 9:00am.<br> Learn how insights from user experience research and design can improve the outcome of your product roadmap using a multi-disciplinary approach to UX research and design strategy.</p> <blockquote> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18593_9028&#x26;locale=en_US">HSW8593</a> <strong>HPE Design Workshop – Tools that help your customers and partners build amazing experiences</strong>.</p> </blockquote> <p>Wednesday, June 19 at 12:00pm.<br> See all the resources available to you through the HPE Design team. They will walk through the hpe.design website and share tips and tricks   you can use to create world-class experiences.</p> <blockquote> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18623_9256&#x26;locale=en_US">HSW8623</a> <strong>Hack Shack Workshop – Learn about HPE’s most popular open source library, Grommet, a class-leading React based UI/UX framework</strong>.</p> </blockquote> <p>Wednesday, June 19 at 2:00pm.<br> Are you ready to turn stunning designs into pixel-perfect accessible web applications? Learn how you can start using Grommet to build class-leading web applications.</p> <blockquote> <p><strong>Hack Shack Challenge -- You’ve heard of soup to nuts. Now, join us for a napkin to template challenge,</strong>. Tuesday, June 18 – Thursday, June 20.</p> </blockquote> <p>This challenge runs throughout the event and shows how easy it is to code in Grommet. Participants start off sketching out their ideas on a napkin and then, using a template project, develop that idea into a publishable UI. The fun part is that you don’t have to be an expert coder or designer. Even beginners are welcome to try their hand at coding and compete for cool prizes.</p> <p>Some of less obvious places you’ll find Grommet is where it runs behind the scenes. Grommet powers the Leaderboard that displays the Hack Shack Attack game results. This is where passersby can see who is currently in line to win the grand prize that will be given out at the awards reception on Wednesday 5– 6pm in the Hack Shack. The website that houses each session presentation .pdf, which enables attendees to download and take the information home with them, is also powered by Grommet. Even the team’s own <a href="https://developer.hpe.com/">HPE DEV</a> and <a href="https://hpe.design/">HPE Design</a> websites run with Grommet.</p> <p>It’s also interesting to note that the technologies presented at HPE Discover, <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView,</a> <a href="/platform/nimble-storage/home">HPE Nimble storage,</a> <a href="https://www.hpe.com/us/en/integrated-systems/simplivity.html">HPE SimpliVity,</a> and <a href="https://www.hpe.com/us/en/integrated-systems/composable-fabric.html">HPE Composable Fabric,</a> were all developed using Grommet. A special demo presented elsewhere on the show floor, <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18314_0&#x26;locale=en_US#session-details-page">(DEMO1409)</a> <strong>The data gold rush: Trapped at the edge today, Swarm Learning is how we’re mining for data intelligence tomorrow,</strong> was also developed using Grommet. In this demonstration of AI (artificial intelligence), event attendees will learn how training edge devices may be more effective than the current method where machines are trained at the core and only reference the edge for data and insights.</p> <p>If you are going to the HPE Discover event in Las Vegas, June 18-20, take a close look around the HPE DEV &#x26; Design Hack Shack and see how many times you can find Grommet. For developers and designers who’d like to understand more about how Grommet can provide significant benefits, seek out developer support through the <a href="https://developer.hpe.com/">HPE Developer Community Program</a> and consider joining the <a href="https://slack.hpedev.io/">HPE DEV community.</a></p>https://developer.hpe.com/hpe-simplivity-powershell-module/https://developer.hpe.com/hpe-simplivity-powershell-module/Thu, 16 May 2019 01:37:52 GMT<h1>HPE SimpliVity PowerShell Module</h1> <p>This PowerShell module utilizes the HPE SimpliVity REST API to display information and manage an HPE SimpliVity federation. It works by connecting to any HPE OmniStack virtual controller in your environment. With the release of HPE SimpliVity V4.0.0 and above, you can now also implement and connect to a management virtual appliance, which is recommended.</p> <p>All cmdlets are written as advanced cmdlets, with comment-based help and the majority have the ability to accept the output from another cmdlet as input. Most cmdlets that show information have parameters to limit the number of objects returned. The cmdlets have been written to adhere to the current recommendations with the REST API. For example, limiting the number of records when returning virtual machines and backup objects.</p> <p>Most "Get" commands display default properties; use Format-List or Select-Object to show all the properties. For example:</p> <pre><code class="language-powershell"> PS C:\> Connect-SVT -OVC 192.168.1.11 -Credential $Cred PS C:\> Get-SVThost HostName DataCenterName ClusterName FreeSpaceGB ManagementIP StorageIP FederationIP -------- -------------- ----------- ----------- ------------ --------- ------------ srvr1.sg.com SunGod Production1 2,671 192.168.1.11 192.168.2.1 192.168.3.1 srvr2.sg.com SunGod Production1 2,671 192.168.1.12 192.168.2.2 192.168.3.2 srvr3.sg.com SunGod DR1 2,671 192.170.1.11 192.170.2.1 192.170.3.1 PS C:\>Get-SVThost -HostName 192.168.1.1 | Select-Object * PolicyEnabled : True ClusterId : 3baba7ec-6d02-4fb6-b510-5ce19cd9c1d0 StorageMask : 255.255.255.0 Model : HPE SimpliVity 380 Series 4000 HostName : srvr1.sg.com . . . </code></pre> <h2>Latest Update</h2> <p>Refer to the release notes <a href="https://github.com/atkinsroy/HPESimpliVity/blob/master/RELEASENOTES.md">here</a> for more details.</p> <p>The module contains 58 exported cmdlets, divided into the following feature categories:</p> <table> <thead> <tr> <th align="left"><strong>Datastore</strong></th> <th align="left"><strong>Backup</strong></th> <th align="left"><strong>Backup Policy</strong></th> </tr> </thead> <tbody> <tr> <td align="left">Get-SVTdatastore</td> <td align="left">Copy-SVTbackup</td> <td align="left">Get-SVTpolicy</td> </tr> <tr> <td align="left">Get-SVTdatastoreComputeNode</td> <td align="left">Get-SVTbackup</td> <td align="left">Get-SVTpolicySchedule</td> </tr> <tr> <td align="left">Get-SVTexternalStore</td> <td align="left">Get-SVTfile</td> <td align="left">New-SVTpolicy</td> </tr> <tr> <td align="left">New-SVTdatastore</td> <td align="left">Lock-SVTbackup</td> <td align="left">New-SVTpolicyRule</td> </tr> <tr> <td align="left">New-SVTexternalStore</td> <td align="left">New-SVTbackup</td> <td align="left">Remove-SVTpolicy</td> </tr> <tr> <td align="left">Publish-SVTdatastore</td> <td align="left">Remove-SVTbackup</td> <td align="left">Remove-SVTpolicyRule</td> </tr> <tr> <td align="left">Remove-SVTdatastore</td> <td align="left">Rename-SVTbackup</td> <td align="left">Rename-SVTpolicy</td> </tr> <tr> <td align="left">Remove-SVTexternalStore</td> <td align="left">Restore-SVTfile</td> <td align="left">Resume-SVTpolicy</td> </tr> <tr> <td align="left">Resize-SVTdatastore</td> <td align="left">Set-SVTbackupRetention</td> <td align="left">Suspend-SVTpolicy</td> </tr> <tr> <td align="left">Set-SVTdatastorePolicy</td> <td align="left">Stop-SVTbackup</td> <td align="left">Update-SVTpolicyRule</td> </tr> <tr> <td align="left">Set-SVTexternalStore</td> <td align="left">Update-SVTbackupUniqueSize</td> <td align="left"></td> </tr> <tr> <td align="left">Unpublish-SVTdatastore</td> <td align="left"></td> <td align="left"></td> </tr> </tbody> </table> <table> <thead> <tr> <th align="left"><strong>Cluster &#x26; Utility</strong></th> <th align="left"><strong>Host</strong></th> <th align="left"><strong>Virtual Machine</strong></th> </tr> </thead> <tbody> <tr> <td align="left">Connect-SVT</td> <td align="left">Get-SVTdisk</td> <td align="left">Get-SVTvm</td> </tr> <tr> <td align="left">Get-SVTcapacity</td> <td align="left">Get-SVThardware</td> <td align="left">Get-SVTvmReplicaSet</td> </tr> <tr> <td align="left">Get-SVTcluster</td> <td align="left">Get-SVThost</td> <td align="left">Move-SVTvm</td> </tr> <tr> <td align="left">Get-SVTclusterConnected</td> <td align="left">Get-SVTshutdownStatus</td> <td align="left">New-SVTclone</td> </tr> <tr> <td align="left">Get-SVTmetric</td> <td align="left">Get-SVTthroughput</td> <td align="left">Restore-SVTvm</td> </tr> <tr> <td align="left">Get-SVTtask</td> <td align="left">Remove-SVThost</td> <td align="left">Set-SVTvm</td> </tr> <tr> <td align="left">Get-SVTtimezone</td> <td align="left">Start-SVTshutdown</td> <td align="left">Start-SVTvm</td> </tr> <tr> <td align="left">Get-SVTversion</td> <td align="left">Stop-SVTshutdown</td> <td align="left">Stop-SVTvm</td> </tr> <tr> <td align="left">Set-SVTtimezone</td> <td align="left"></td> <td align="left"></td> </tr> </tbody> </table> <h2>Requirements</h2> <ul> <li>Windows PowerShell V5.1 or PowerShell Core V7.x (PowerShell Core V6.x is not recommended)</li> <li>The IP address and the credentials of an authorized OmniStack user account</li> <li>The module has been tested with HPE SimpliVity V4.0.1 and should be compatible with older versions (but has not been tested).</li> </ul> <h2>Installation</h2> <ul> <li>Install or update the HPESimplivity module from the PowerShell Galllery using the following respective commands:</li> </ul> <pre><code class="language-powershell"> PS C:\> Install-Module -Name HPESimpliVity # or PS C:\> Update-Module -Name HPESimpliVity </code></pre> <p>The module is signed, so it will work with an execution policy set to 'Remote Signed'.</p> <ul> <li>Restart PowerShell to load the module, or type:</li> </ul> <pre><code class="language-powershell"> PS C:\> Import-Module HPESimpliVity -Force </code></pre> <ul> <li>After this, the module will automatically load in new PowerShell sessions. Issue the following commands to confirm:</li> </ul> <pre><code class="language-powershell"> PS C:\> Get-Command -Module HPESimpliVity PS C:\> Get-Help Connect-SVT PS C:\> Get-Help Get-SVTbackup </code></pre> <ul> <li>Once installed, you’re ready to connect to the OmniStack virtual controller or Management Virtual Appliance, as follows:</li> </ul> <pre><code class="language-powershell"> PS C:\> $Cred = Get-Credential -Message 'Enter OVC/MVA Credentials' PS C:\> Connect-SVT -OVC -Credential $Cred PS C:\> Get-SVThost </code></pre> <ul> <li>Or, if you need to run commands in batch (non-interactively), save your credentials to a file first:</li> </ul> <pre><code class="language-powershell"> PS C:\> $Cred = Get-Credential -Username 'administrator@vsphere.local' | Export-Clixml .\OVCcred.XML </code></pre> <p>and then in your script, import the credential:</p> <pre><code class="language-powershell"> PS C:\> $Cred = Import-CLIXML .\OVCcred.XML PS C:\> Connect-SVT -OVC -Credential $Cred PS C:\> Get-SVThost </code></pre> <p><strong>Note:</strong> You must login with an admin account (e.g. an account with the vCenter Admin Role for VMware environments).</p> <h2>Known issues with the REST API (HPE SimpliVity V4.0.1)</h2> <p>The API has some documented and undocumented issues:</p> <ul> <li>OMNI-69918: GET /virtual_machines fails with OutOfMemoryError. The HPE SimpliVity module limits the number of VMs returned to 8000, as per the recommendation</li> <li>OMNI-46361: REST API GET operations for backup objects and sorting filtering constraints. Comma separated list for filtering backup objects is not supported when connecting to OmniStack Virtual Controllers. Comma separated lists CAN be used when connected to a Management Virtual Appliance. For example, the following commands all work when connected to an MVA:</li> </ul> <pre><code class="language-powershell"> PS C:\> Get-SVTbackup -VmName Vm1,Vm2,Vm3 PS C:\> Get-SVTbackup -Destination Cluster1,Cluster2 PS C:\> Get-SVTbackup -Destination StoreOnce-Data01,StoreOnce-Data02 PS C:\> Get-SVTbackup -Datastore DS01,DS02 PS C:\> Get-SVTbackup -BackupName Test1,Test2 PS C:\> Get-SVTbackup -BackupState FAILED,SAVING,QUEUED PS C:\> Get-SVTbackup -BackupId a9e82f..., bef1bd... </code></pre> <ul> <li>Backups stored on external stores cannot be deleted if the VM has been deleted, with a “backup not found” error. This does not apply to backups stored on SimpliVity clusters. This restriction is specific to the API; the CLI command <code>svt-backup-delete</code> works as expected for external store backups.</li> <li>The <code>PUT /policies/policyId/rules/ruleId</code> API call (implemented in <code>Update-SVTpolicyRule</code>) doesn’t work as expected in some circumstances. Changing a rules’ destination is not supported (this is documented). In addition, changing the consistency type to anything other than NONE or DEFAULT doesn’t work. If you attempt to change the consistency type to VSS, for example, the command is ignored. In this scenario, a work around would be to delete the rule entirely from the policy using <code>Remove-SVTpolicyRule</code> and then use <code>New-SVTpolicyRule</code> to create a new rule with the desired destination, consistency type and other settings.</li> <li>Using <code>GET /backups</code> with a specific cluster_id (implemented as <code>Get-SVTbackup –DestinationName ClusterName</code>) will result in both backups located on the specified cluster AND external stores being displayed. This issue only applies when connected to an OVC; calls to an MVA work as expected. In either case, filtering on an external store works as expected (e.g. <code>Get-SVTbackup –DestinationName ExternalStore1</code>)</li> </ul> <p>If you would like to keep up to date with the latest features, please visit the project website on <a href="https://github.com/atkinsroy/HPESimpliVity">GitHub</a> and subscribe to receive notifications. Updates are published to the PowerShell Gallery at the same time.</p> <p>Refer to the <a href="https://developer.hpe.com/platform/hpe-simplivity/home">HPE SimpliVity platform page</a> on HPE DEV for more information. Don't forget to check out our other posts on the <a href="/blog">HPE DEV blog site</a> or join us on our #simplivity <a href="https://slack.hpedev.io/">Slack Channel</a> to ask a question.</p>https://developer.hpe.com/meet-us-at-the-hack-shack-at-hpe-discover-las-vegas-june-18-20/https://developer.hpe.com/meet-us-at-the-hack-shack-at-hpe-discover-las-vegas-june-18-20/Tue, 14 May 2019 19:19:11 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/img_0246-1557861662220.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1557861662231"></p> <p>HPE DEV is gearing up for Hewlett Packard Enterprise’s (HPE) premier event, <a href="https://www.hpe.com/events/discover/">HPE Discover,</a> to be held June 18-20, 2019 in Las Vegas. Expect three action-packed days filled with talks, workshops, and events designed to help your business become more agile. Software developers and designers will want to block off a good amount of time to spend at the HPE DEV &#x26; Design’s Hack Shack, which offers opportunities for hands-on learning and a place to become more engaged with the HPE Developer and Design Community. There, you’ll learn how HPE supports developers and designers in creating innovative solutions to complex problems, collaborating with them using open source software, “design thinking”, DevOps and ITOps tools.</p> <p>The Hack Shack features two types of interactive activities: Workshops and Challenges. In the Workshops, you’ll get a chance to see innovative technologies in action. There will also be several design-related workshops, including some focused on <a href="https://v2.grommet.io/">Grommet</a> – an open source user interface (UI) toolkit designed by HPE. In the Challenges, which run from show open till show close, we encourage attendees to apply their skills and build out their application ideas while competing for cool prizes.</p> <p>At the Hack Shack you’ll find over twenty sessions where everyone is welcome, from beginners to experts. Below is a list of all Hack Shack Workshops and Challenges going on at the event. <br /></p> <h2>HPE OneView</h2> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18598_9074&#x26;locale=en_US">HSW8598</a> <strong>Hack Shack Workshop – Dive into infrastructure automation with the HPE OneView API</strong>.<br> Wednesday, June 19 at 3:00pm.<br> During this interactive session, you’ll explore the HPE OneView API using Postman. After that, you’ll use PowerShell or Python to write a script that automates HPE OneView. By the end, you will have written a simple example of infrastructure as code. Be sure to bring your laptop.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=19834_9253&#x26;locale=en_US">HSW9834</a> <strong>Hack Shack Workshop – HPE OneView Global Dashboard REST API</strong>.<br> Wednesday, June 19 at 4:00pm.<br> Thursday, June 20 at 11:00am.<br> Experiment with HPE OneView Global Dashboard REST API. Learn how you can interact programmatically with multiple HPE OneView appliances in a large, distributed environment through a single pane of glass. Be sure to bring your laptop to follow along.</p> <p><strong>Hack Shack Challenge with HPE OneView: Locate the rogue server in a large data center</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Apply the skills that you’ve learned to find the misbehaving server using only limited information and iLO logs. No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge with HPE OneView Global Dashboard: Creating and populating a resource group</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Show off your ability to enhance an existing PowerShell module or create and populate a group of HPE OneView resources using HPE Global Dashboard REST API. No registration required. Just show up and start coding. <br /></p> <h2>HPE SimpliVity</h2> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18622_9045&#x26;locale=en_US">HSW8622</a> <strong>Hack Shack Workshop – Use HPE SimpliVity REST API fundamentals to build a simple automation script in HPE SimpliVity</strong>.<br> Wednesday, June 19 at 1:00pm.<br> Thursday, June 20 at 10:00am.<br> Bring your laptop and learn how to interact with the HPE SimpliVity hyperconverged platform through its REST API. Explore the API description and use PowerShell or Python to automate the kind of HPE SimpliVity use case that may come up in your day-to-day work.</p> <p><strong>Hack Shack Challenge with HPE SimpliVity API: Build simple reports that show the status of on-going backups with user-defined time interval</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Leverage the granularity and simplicity of the SimpliVity REST API to build a report. Keep it simple or unleash your creativity to come up with a cool 3D graphical representation. No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge with HPE SimpliVity: Build a solution to inject user scripts before and after a backup</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Use the SimpliVity REST API to build a solution for pre-and-post backup script injection to enable backup life cycle management use cases. No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge with HPE SimpliVity: Create the most powerful backup script</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Learn innovative ways of producing bulletproof backups using the HPE SimpliVity REST API. No registration required. Just show up and start coding. <br /></p> <h2>HPE Nimble Storage</h2> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18596_9138&#x26;locale=en_US">HSW8596</a> <strong>Hack Shack Workshop – Improve day 2 operations with your own custom automation with the HPE Nimble Storage REST API</strong>.<br> Thursday, June 20 at 12:00pm.<br> Bring your own laptop with VMware Workstation Player (or VMware Fusion for Mac) installed and get ready to use REST API tools to create your own custom automation for HPE Nimble Storage. You’ll get access to your own personal Nimble Virtual Array (a limited-time offer, but a great perk). To participate, a basic knowledge of block storage is helpful.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18597_8916&#x26;locale=en_US">HSW8597</a> <strong>Hack Shack Workshop – Persist, optimize and accelerate using persistent storage in Kubernetes</strong>.<br> Tuesday, June 18 at 9:00am.<br> In this workshop, we’ll explain the differences between storing data locally versus externally and what workload types in Kubernetes are suitable for persistent storage. An overview of API-related objects, such as StorageClass, PersistentVolumeClaim and PersistentVolume, will be included, along with storage plugins and drivers. <br /></p> <h2>HPE Composable Fabric</h2> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18621_8970&#x26;locale=en_US">HSW8621</a> <strong>Hack Shack Workshop – Use network optimization with Composable Fabric to accelerate operations</strong>.<br> Tuesday, June 18 at 1:00pm.<br> Here, you’ll explore how Composable Fabric Affinity technology can streamline operations by automatically adjusting to different network needs (i.e. daytime versus nighttime). Different example workloads will be examined to show how they compete for bandwidth and how adding an automated storage affinity, i.e. HPE SimpliVity, can help.</p> <p><strong>Hack Shack Challenge with HPE Composable Fabric Manager: Initiate Composable Fabric operations through social applications</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Build a dashboard that shows the current network view configuration of the Composable Fabric mesh network quickly. Prizes will be awarded to the most innovative solution. No registration required. Just show up and start coding. <br /></p> <h2>HPE Design Workshops</h2> <p>At the Hack Shack, you’ll get to meet the HPE application design team and learn more about the resources they can provide to help develop applications. HPE Design workshops include:</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18626_8949&#x26;locale=en_US">HSW8626</a> <strong>HPE Design Workshop – Design Thinking for disruptive ideas - Empathy &#x26; Brainstorming</strong>.<br> Tuesday, June 18 at 11:00am and 12:00pm.<br> Do you have ideas that need to be explored? What if you had no limitations? Discover your next big idea in IT in this fast-paced human-centered collaborative design session.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18625_8975&#x26;locale=en_US">HSW8625</a> <strong>HPE Design Workshop – User experience research tools and techniques that drive customer experience</strong>.<br> Tuesday, June 18 at 4:30pm.<br> Thursday, June 20 at 9:00am.<br> Learn how insights from user experience research and design can improve the outcome of your product roadmap using a multi-disciplinary approach to UX research and design strategy.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18628_9003&#x26;locale=en_US">HSW8628</a> <strong>HPE Design Workshop – Design Sprint, Go! Rapid Prototyping &#x26; The Customer Feedback Loop</strong>.<br> Wednesday, June 19 at 10:00am and 11:00am.<br> Don’t make long term investments on ideas that have unproven customer value. Learn the value that comes when you pick a use case, design mockups, develop a prototype and test it with a customer.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18593_9028&#x26;locale=en_US">HSW8593</a> <strong>HPE Design Workshop – Tools that help your customers and partners build amazing experiences</strong>.  <br> Wednesday, June 19 at 12:00pm.<br> See all the resources available to you through the HPE Design team.  We’ll walk through the hpe.design website and share tips and tricks that you can use to create world class experiences.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=18623_9256&#x26;locale=en_US">HSW8623</a> <strong>Hack Shack Workshop – Learn about HPE’s most popular open source library, Grommet, a class-leading React based UI/UX framework</strong>.<br> Wednesday, June 19 at 2:00pm.<br> Are you ready to turn stunning designs into pixel-perfect accessible web applications? Learn how you can start using Grommet to build class-leading web applications. <br /></p> <h2>More Fun Workshops and Challenges</h2> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=19718_8994&#x26;locale=en_US">HSW9718</a> <strong>Hack Shack Workshop – Using RedFish API, monitor and manage HPE ProLiant servers from core to edge</strong>.<br> Wednesday, June 19 at 9:00am.<br> Join us for a demonstration of the configuration and monitoring of Redfish Event Service compliant infrastructure using Nagios Core monitoring engine.</p> <p><a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;sid=19835_9252&#x26;locale=en_US">HSW9835</a> <strong>Hack Shack Workshop — Simplify edge/IoT management with intent-driven computing</strong>.<br> Tuesday, June 18 at 10:00am.<br> Thursday, June 20 at 1:00pm.<br> See how an intent-based computing framework, Libere, can free users from tedious and complicated configuration tasks by automatically managing applications based on performance goals.</p> <p><strong>Hack Shack Challenge - Join HPE “Hello World” and get started on understanding scripting</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Learn how to code or improve your current coding skills. Your hack can be as simple as showing Hello World in notepad or a text editor like Vim. To win the prize, remember, creativity counts! No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge - You’ve heard of soup to nuts. Now, join us for a napkin to template challenge</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> In this two stage hackathon, develop something using public APIs. First, take a napkin and sketch your ideas on it. Then build out your idea using the napkin as a reference and the template project for starters. Judging will encompass both the napkin and the final published UI. No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge for HPE Proliant for Azure Stack Administrators/Operators – Provisioning services and enabling DevOps for tenants</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> As an Azure Stack cloud administrator, you can create IaaS, PaaS offers that your tenant users can self-subscribe to. Here you’ll learn how to use PowerShell and/or admin portal to create/configure service offers with plans, quotas and enable user subscriptions. No registration required. Just show up and start coding.</p> <p><strong>Hack Shack Challenge to seamlessly deploy hybrid cloud solution infrastructure with HPE Proliant for Azure Stack – Deploy apps to Azure and Azure Stack using an infrastructure-as-code template</strong>.<br> Tuesday, June 18 – Thursday, June 20 during show floor hours.<br> Explore the concept of using infrastructure as code to deploy application resources globally via Azure or Azure Stack. You’ll also learn how to integrate HPE Nimble solutions as external storage for virtual machines running on Azure Stack. No registration required. Just show up and start coding.</p> <p>If you’re planning to be at the <a href="https://www.hpe.com/events/discover/">HPE Discover 2019 event,</a> remember to stop by booth 2088 and meet HPE DEV &#x26; Design experts, attend Workshops, participate in the Hack Shack Challenges, or just chill and relax with us. The <a href="https://content.attend.hpe.com/go/agendabuilder.sessions/?l=39&#x26;locale=en_US">HPE Discover sessions catalog</a> can give you more details on the workshops and challenges. To continue your experience after the show, <a href="https://developer.hpe.com/signup">sign up at the HPE DEV website</a> to join the HPE DEV community. #letshackshack</p>https://developer.hpe.com/using-your-first-grommet-component-with-create-react-app/https://developer.hpe.com/using-your-first-grommet-component-with-create-react-app/Fri, 10 May 2019 15:27:12 GMT<p>In this tutorial, I will walk you through the steps it takes to set up your first grommet component using create-react-app.</p> <h2>Prerequisites</h2> <p>This tutorial assumes that you have node.js and a package manager; either npm (npm is installed with node.js) or yarn package manager. Create react app can be installed globally or locally using npx (npx comes with npm 5.2+).</p> <p><a href="https://nodejs.org/en/download/">Node Download Page</a> - The latest LTS version will have node.js, npm, and npx.<br> <a href="https://facebook.github.io/create-react-app/docs/getting-started">Create React App Local Install</a> - Instructions to install it locally can be found here (npm versions 5.2+).<br> <a href="https://gist.github.com/gaearon/4064d3c23a77c74a3614c498a8bb1c5f">Create React App Global Install</a> - Instructions to install it globally can be found here (npm versions 5.1 or earlier).<br> <a href="https://yarnpkg.com/en/docs/getting-started">Yarn Package Manager - <em>Not Required</em></a> - I use yarn as my primary package manager, however npm will work fine for this tutorial.</p> <h2>Clean-up</h2> <p>Create and name your application with create-react-app. Then navigate into the folder of your newly created application and open it with the editor of your choice.</p> <p><em>If you have create-react-app installed globally,</em></p> <pre><code class="language-bash">create-react-app grommet-rules cd grommet-rules </code></pre> <p><em>or if you use npx.</em></p> <pre><code>npx create-react-app grommet-rules cd my-app </code></pre> <p>Once inside, delete the files <code>App.css</code>, <code>App.test.js</code>, <code>index.css</code>, and <code>logo.svg</code> that are located in the src folder. These files will not be used in this tutorial.</p> <p><em>Delete the highlighted files.</em></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/deletethese-1557502375028.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1557502375030"></p> <p>All associated imports and components in index.js and App.js must be deleted as well.</p> <blockquote> <p>In App.js, delete the highlighted imports and header component.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/delete-app-1557502556518.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1557502556519"></p> <blockquote> <p>In index.js, delete the highlighted import.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/delete-index-1557502646695.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1557502646696"></p> <p>Your <code>App.js</code> file should look like this,</p> <pre><code class="language-jsx">import React from 'react'; function App() { return ( &#x3C;div className="App"> &#x3C;/div> ); } export default App; </code></pre> <p>and your <code>index.js</code> file should look like this.</p> <pre><code class="language-jsx">import React from 'react'; import ReactDOM from 'react-dom'; import App from './App'; import * as serviceWorker from './serviceWorker'; ReactDOM.render(&#x3C;App />, document.getElementById('root')); serviceWorker.unregister(); </code></pre> <p><strong>It's sanity check time!</strong><br> Run your application with</p> <pre><code class="language-bash">yarn start </code></pre> <p>or</p> <pre><code class="language-bash">npm start </code></pre> <blockquote> <p>When you run your application, do you see a blank screen? Yes? Cool!<br> When you add text to <code>App.js</code>, is it visible on the screen? Yes? Rad! It's time to move on.</p> </blockquote> <h2>Set-up</h2> <p>Install the grommet packages and dependencies using a package manager of your choice.</p> <pre><code class="language-bash">yarn add grommet grommet-icons styled-components </code></pre> <p>or</p> <pre><code class="language-bash">npm install grommet grommet-icons styled-components --save </code></pre> <p>You will not be using grommet-icons and styled-components in this tutorial, but if you would like to keep using this application to practice using grommet in the future, it would be best if you installed them now.</p> <p>A requirement for grommet is to import it and use it as a top-level node. You can do this in <code>App.js</code>.</p> <p><em>In <code>App.js</code>, import grommet and replace the div component with the Grommet component. Your app is now ready to use all that grommet has to offer.</em></p> <pre><code class="language-jsx">import React from 'react'; import { Grommet, Heading } from 'grommet' function App() { return ( &#x3C;Grommet className="App"> &#x3C;/Grommet> ); } export default App; </code></pre> <h2>Using Your First Component</h2> <p>Grommet components, such as Heading, can be imported and included in very much the same way as you did in the previous step.</p> <p><em>In <code>App.js</code>, import the Heading component, put it within the Grommet component, and add some text.</em></p> <pre><code class="language-jsx">import React from 'react'; import { Grommet, Heading } from 'grommet' function App() { return ( &#x3C;Grommet className="App"> &#x3C;Heading> Please work! &#x3C;/Heading> &#x3C;/Grommet> ); } export default App; </code></pre> <blockquote> <p>It's sanity check time, yet again:</p> </blockquote> <p>Is the text visible in your application? Yes? It's time to work with properties!</p> <h2>Props</h2> <p>Properties can be used to position and style your grommet components. The Heading component has two properties, color and size, that you can manipulate.</p> <p><em>In <code>App.js</code>, within your Heading component add the properties color and size, and give them values.<br> Detailed information on how the properties of each component can be modified can be found at <a href="https://v2.grommet.io/components">grommet components docs</a>.</em></p> <pre><code class="language-jsx">import React from 'react'; import { Grommet, Heading } from 'grommet' function App() { return ( &#x3C;Grommet className="App"> &#x3C;Heading size='large' color='#00739D' > I've Mastered Grommet! &#x3C;/Heading> &#x3C;/Grommet> ); } export default App; </code></pre> <p>It's time to check your work.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/tutorialcomplete-1557503147092.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1557503147092"></p> <p><em>Maybe, dial it back a bit.</em></p> <h2>Resources</h2> <p>You've created your grommet playground and all there is left to do is practice. Here are a few resources that I have found helpful when I first began familiarizing myself with grommet.</p> <p><a href="https://v2.grommet.io/components">Grommet Components Docs</a> - This is useful for when you want information on each individual component, the properties that they have, and the default and accepted values for each property.</p> <p><a href="https://github.com/grommet/grommet">Grommet Github</a> - It's always important to dig through the closed and open issues for any solutions not covered in the docs.</p> <p><a href="http://slackin.grommet.io/">Grommet Slack Inviter</a> - When I've exhausted all my resources, the grommet community is eager to help and quick to answer.</p>https://developer.hpe.com/grommet-the-glue-that-binds-development-and-design/https://developer.hpe.com/grommet-the-glue-that-binds-development-and-design/Thu, 09 May 2019 14:59:02 GMT<p>Simple, consistent, and thoughtfully designed user interfaces (UI) can go a long way to integrating business applications and streamlining operations. Hewlett Packard Enterprise (HPE) is a huge proponent of simplifying complex environments. That’s why HPE developed <a href="http://v2.grommet.io">Grommet,</a> an open source UI development tool, and made it available to anyone looking to create consistent user experiences for their applications.</p> <p>When I first started working with the Chief Design Office at HPE, I was very excited for the opportunity to learn more about Grommet. Design is playing a bigger part in computing applications today. To me, Grommet is like the glue that binds function and design together. As evidenced by input received through <a href="http://developer.hpe.com">HPE DEV</a> and several tradeshows and events, Grommet is both liked and used, sustained by a vibrant, grassroots community. It has attracted plenty of developer input with over 4,000 commits from a wide variety of different contributors.</p> <p>For those of you unfamiliar with Grommet, it is a library of reusable UI components for developers and designers. Grommet makes the application creation process both agile and easy. It does so by offering developers commonly used interface elements in a Node package and delivering them as building blocks to create web applications. It also gives designers a full library of design assets and templates. Grommet uses React as an underlying technology to help deliver its modularity and adds accessibility, responsiveness, and theming.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/screen-shot-2019-05-09-at-31254-pm2-1557501148565.png" alt="screen shot 2019 05 09 at 3.12.54 pm[2]"></p> <p>Grommet works using a mobile-first strategy in combination with a task-based design approach. Its components offer nothing more and nothing less than what’s required to complete a task, while still being flexible enough to customize and tailor it to your own needs.</p> <p>Grommet’s libraries give both designers and developers a way to collaborate and inject consistent user experiences into their applications easily. It offers ready-to-use layouts, type, colors, controls, and mechanisms to incorporate and present content. You might want to check out this link for <a href="https://v2.grommet.io/components">specifics on Grommet components.</a></p> <p>Grommet is used by both internal HPE and external developers to build enterprise-class and consumer-grade applications that can be used in web, desktop, and mobile-friendly formats. It underpins many of the company’s software-defined platforms, like <a href="http://www.hpe.com/us/en/integrated-systems/synergy.html"> HPE Synergy,</a> HPE Storage, Aruba networks and HPE Server management. HPE provides developer support for Grommet through the <a href="https://developer.hpe.com/community">HPE Developer Community</a> and <a href="https://github.com/grommet/grommet">GitHub.</a></p> <p>As much as I’ve learned about Grommet, I still have more questions. Like, what’s up with that purple Gremlin creature that keeps showing up on Grommet pages and Grommet stickers? Stay tuned for a future post where I hope to uncover the answer. For more thoughts on Grommet, you might want to check out the <a href="https://www.hpe.com/us/en/insights/articles/getting-to-know-grommet-an-open-source-ui-dev-tool-1808.html">blog</a> by John Paul Mueller of DataCon Services. You can also keep up with what’s going on by connecting with us at <a href="https://developer.hpe.com/">HPE DEV</a> and subscribing to our <a href="https://developer.hpe.com/newsletter-signup">newsletter.</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/5/rsz_1rsz_stack-waving-1557501164196.png" alt="rsz_1rsz_stack waving"></p>https://developer.hpe.com/2019-April-29/https://developer.hpe.com/2019-April-29/Mon, 29 Apr 2019 05:00:00 GMThttps://developer.hpe.com/virtualization-is-not-cloud-computing/https://developer.hpe.com/virtualization-is-not-cloud-computing/Thu, 25 Apr 2019 15:27:57 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/gettyimages-160673315-1556211177673.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556211177810"></p> <p>It seems everyone in the IT industry is focused on cloud computing. The industry has seen these trends in the past: distributed computing, client-server computing, and virtualization to name a few. You may feel that if you're not in the cloud, you are falling behind and missing some great opportunities. Well, you may be. To determine if you are, it’s important to understand what cloud computing is and how it's different from previous models and technologies.</p> <p>For over a decade, organizations have employed virtualization to make workloads more portable, easier to configure, and – most importantly – capable of more fully utilizing all available compute resources. Virtualization technologies allow users to abstract the physical compute environment and cram a bunch more stuff onto a fixed set of compute resources.</p> <p>Virtualization is fairly easy for IT and developers to adopt because the basic behavior models do not change. IT operators can still employ the same constructs with virtualized machines that they are used to using with physical infrastructure. They can provision a virtual machine, configure it, power it up, patch it, etc. And developers do not really need to change their programming models, either. The coding models and tools they use are generally the same between physical and virtual environments. Developers don’t really need to think about whether the target environment is a physical or virtualized.</p> <p>Cloud computing is different. It's a new way of thinking about and approaching how IT operators manage environments and how developers employ those resources to solve business problems. Cloud computing is a new model that requires new methods and techniques to fully realize its benefits.</p> <p>For IT operators, this means that you are no longer concerned with just machines, memory, and networking. You must now learn about platform services (PaaS) and software services (SaaS) and how to effectively manage those for your developers and organizations. Given that resources often reside outside the organization, you may feel like cloud computing, especially public cloud, obsoletes your job. But it doesn’t – it actually makes you more essential. You are needed even more now to effectively manage your IT resources across both private and public domains.</p> <p>Developers face similar challenges with cloud computing. Yes, as a developer, you still need to know your programming languages and SDKs. But you have a whole new set of resources and constructs to employ that were not available in previous models. For example, cloud computing enables you to easily automate things like spinning up a database or webserver. You can also take advantage of new constructs like serverless computing. This frees you from thinking about virtual machines and resources and enables you to focus on the business problem and applications, letting the cloud environment handle the routine tasks. The cloud environment has a whole new set of tools for doing this -- and who doesn’t like learning about new tools?</p> <p>You may run workloads in public clouds such as Amazon, Azure, Google, or others, and think this is ‘cloud computing’. But if you’re simply managing virtual machines and infrastructure (IaaS), you’re just doing ‘virtualization’ in the cloud and missing the real opportunities provided by the cloud model. Similarly for developers, if you are ignoring the additional platform and software services, you might also be missing out. Employing platform and software services in the cloud frees you up from concentrating on compute resources so you can focus your attention on solving business issues.</p> <p>Keep in touch with us at <a href="https://developer.hpe.com/">HPE DEV</a> HPE DEV to learn more about serverless computing, including Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS) concepts and services.</p>https://developer.hpe.com/hpe-dev-at-google-cloud-next-19/https://developer.hpe.com/hpe-dev-at-google-cloud-next-19/Tue, 23 Apr 2019 20:04:56 GMT<h1>Sir Hackington Appbuilder III recounts HPE DEV experiences at Google Cloud Next ‘19 April 9-11</h1> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/untitled1-1556056663049.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556056663050"></p> <p>Sir Hackington Appbuilder III here. As an important member of the HPE Developer Community (HPE DEV), I get to travel to a lot of shows and hang out with event attendees. I just got back from #GoogleNext19 and wanted to tell you all about it!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture12-1556051302495.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556051302503"></p> <p><a href="https://cloud.withgoogle.com/next/sf">Google Cloud Next '19</a> April 9-11, 2019 in Moscone Center, San Francisco brought together some of brightest minds in tech for three full days of networking, learning, and problem solving. Thousands of IT professionals and executives across industries participated. Hewlett Packard Enterprise (HPE) commanded a presence at the event with an <a href="http://cloud.google.com/blog/topics/partners/google-cloud-partners-with-hpe-on-hybrid-cloud-next19"> important announcement on groundbreaking partnerships that help simplify the pathway to hybrid cloud.</a></p> <p>On the first day, HPE announced that it would further expand its commitment to deliver the options and the experiences customers desire for hybrid cloud by strategically aligning with two powerful industry players – Nutanix and Google Cloud, These partnerships help to extend the fast growing, ever-evolving HPE GreenLake ecosystem. The partnerships aims to provide customers with a consistent experience across public cloud and on premises environments.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture1112-1556051430387.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556051430388"></p> <p>A key part of the announcement brought developer event attendees to the HPE DEV booth to ask about next steps and how they could participate. HPE will offer validated designs for Google Kubernetes Engine (GKE) based on <a href="http://www.hpe.com/us/en/integrated-systems/simplivity.html"> HPE SimpliVity </a> hyperconverged solutions and many developers came to the HPE booth looking for information on how to interact with HPE SimpliVity using the HPE APIs.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture14-1556051540938.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556051540940"></p> <p>HPE DEV booth personnel were happy to interact with the attendees, showing them the HPE DEV portal and where they could find the tools they need. A few folks came in just to see me. One kind person even took me for a walk.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture51-1556051572650.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556051572651"></p> <p>There was also quite an interest in <a href="https://v2.grommet.io/">Grommet</a> Attendees expressed their excitement about how HPE was supporting open source and provided feedback from having used the UI framework. The use of Grommet seems to be a grassroots engagement starting with new users. Attendees seemed equally delighted to obtain Grommet stickers, as well as stickers for other HPE products.</p> <p>The highlight of each day was the daily prize giveaway where, through a raffle, one lucky event attendee walked away with an Argon Raspberry Pi kit just for having signed up for the HPE DEV newsletter.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture15-1556051644744.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556051644745"></p> <p>Here’s a link if you are interested in finding out more information on the HPE announcement about entering into <a href="http://www.hpe.com/us/en/newsroom/blog-post/2019/04/two-groundbreaking-partnerships-help-simplify-the-pathway-to-hybrid-cloud.html">strategic partnerships with Nutanix and Google Cloud.</a> Hope to see you soon at our next event!</p>https://developer.hpe.com/hpe-dev-to-attend-kubecon-cloudnativecon-europe-2019-in-may/https://developer.hpe.com/hpe-dev-to-attend-kubecon-cloudnativecon-europe-2019-in-may/Tue, 23 Apr 2019 19:56:01 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/picture1-1556049624644.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556049624648"></p> <h1>HPE DEV to attend KubeCon + CloudNativeCon Europe 2019 in May</h1> <p>Hewlett Packard Enterprise (HPE) is excited to be attending <a href="http://https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/">KubeCon + CloudNativeCon Europe 2019</a> in Barcelona, Spain, May 20-23. Over 6,000 talented individuals in the industry are expected to gather, making it a great place to network with industry professionals and learn from top cloud native experts. At the HPE Booth (G3), our team of subject matter experts will focus on 3 main topics that will help DevOps administrators and developers take advantage of Kubernetes clusters. These include:</p> <ul> <li>Containers-as-a-service and Kubernetes Clusters-as-a-service</li> <li>Persistent Storage for Kubernetes</li> <li>The HPE DEV community</li> </ul> <p>Through <a href="http://https://www.hpe.com/us/en/solutions/cloud/onesphere.html">HPE OneSphere</a> and our Persistent Storage for Kubernetes, HPE provides DevOps administrators with an automated, easy to manage Kubernetes clusters as a Service, deployable across hybrid cloud architectures (public and private) in a coherent and cost-controlled way. We offer both centrally managed and consumer deployed and managed clusters on public cloud (AWS), on-prem (VMware ESX) and on bare metal, to enable agile, DevOps environments.</p> <p>If you are a DevOps Admin concerned with providing Kubernetes Clusters and Containers to the masses, or a developer who would like to understand the benefits of joining the HPE DEV community, please visit us at the HPE Booth (G3) in the Sponsor Showcase. When you stop by our booth, sign up for the HPE DEV newsletter and be entered for a chance to win a prize!</p>https://developer.hpe.com/creating-a-powershell-module-as-a-wrapper-for-the-hpe-global-dashboard-r/https://developer.hpe.com/creating-a-powershell-module-as-a-wrapper-for-the-hpe-global-dashboard-r/Tue, 23 Apr 2019 16:59:48 GMT<p>With the release of HPE Global Dashboard <a href="https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00056235en_us&#x26;docLocale=en_US">version 1.60</a> HPE shipped a new (public) RESTful API.</p> <p>The documentation for the API is available through the Help section of the appliance itself, and there's a version of the API published on <a href="https://app.swaggerhub.com/apis-docs/hpe-global-dashboard/hpe-one_view_global_dashboard_rest_api/2">swaggerhub</a>.</p> <p>You can also head over to <a href="https://developer.hpe.com/platform/hpe-oneview-global-dashboard/home">HPE Dev</a> to get more information about the API and there's also a <a href="/blog/accessing-the-hpe-oneview-global-dashboard-api">"getting started" post</a></p> <p>Currently there's no official Powershell module from HPE that let's you work with HPE Global Dashboard, and I haven't heard of any plans to create one either so I decided to create one my self.</p> <h2>The HPE Global Dashboard Powershell module</h2> <p>At the time of this writing the module is at version 0.5.0.</p> <p>It doesn't support all the functionality from the REST API yet. I've focused on the GET stuff and more specifically on the things we use in our environment. Currently the following can be done through the module:</p> <ul> <li>Connect, (and disconnect) to a HPE Global Dashboard (GD) instance</li> <li>Add, reconnect and remove a OneView appliance to the GD instance</li> <li>List the OneView appliances connected to the GD instance</li> <li>List the current GD appliance certificate</li> <li>List Converged Systems</li> <li>List Enclosures</li> <li>List Server Hardware</li> <li>List Server Profiles and Profile Templates</li> <li>List Storage Systems</li> <li>Create and delete logical groups</li> <li>List Logical groups</li> <li>List members of a logical group</li> </ul> <h3>Install module</h3> <p>Installing the module is easiest done through the <a href="https://www.powershellgallery.com/packages/GlobalDashboardPS">Powershell Gallery</a> (note that you should start PS as an Admin).</p> <pre><code class="language-powershell">Find-Module -Repository PSGallery -Name GlobalDashboardPS | Install-Module </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/gdps-psgallery-1556038459722.png" alt="Install module from PS Gallery"></p> <p>You can also download the code from GitHub and import it the way you want to your Powershell session</p> <h3>Usage</h3> <p>To check the available functions you can use we utilize the builtin Get-Command cmdlet</p> <pre><code class="language-powershell">Get-Command -Module GlobalDashboardPS </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/gdps-functions-1556038517571.png" alt="Available functions"></p> <p>To start off we need to connect to the HPE Global Dashboard Instance. Note that currently you need to specify your credentials. I will add support for PS Credentials going forward.</p> <pre><code class="language-powershell">Connect-OVGD -Server "your-appliance-ip" -Username "your-username" -Directory "your-directory" </code></pre> <p>Just to show the basic usage of the module I'll demonstrate how we list Server Hardware. First with the default format (pulled from the ps1xml formatting file).</p> <pre><code class="language-powershell">Get-OVGDServerHardware </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/gdps-serverhwlist-1556038565601.png" alt="List server hardware"></p> <p>To see all details of an object you need to specify this.</p> <pre><code class="language-powershell">Get-OVGDServerHardware | select -First 1 * </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/gdps-alldetails-1556038588225.png" alt="Server details"></p> <p>All functions have help text where you can see examples of the usage.</p> <pre><code class="language-powershell">Get-Help Get-OVGDServerHardware Get-Help Get-OVGDServerHardware -Examples </code></pre> <h3>Paging and querying</h3> <p>Note that the API uses paging of results and defaults to 25 objects. You can specify the count of objects you want to return in the module functions, but currently it does not support paging as such.</p> <p>Currently you can query for specific objects, but you need to use the ID of the object. I am working on adding support for querying by name as well.</p> <h2>Summary</h2> <p>The full source of this project is available on <a href="https://github.com/rumart/GlobalDashboardPS">GitHub</a> and the module is available on the <a href="https://www.powershellgallery.com/packages/GlobalDashboardPS">Powershell Gallery</a></p> <p>There's quite a few things missing in the module, but hopefully I will be able to add in stuff going forward. Please feel free to contribute if you want!</p> <p>The <a href="https://github.com/rumart/GlobalDashboardPS/blob/master/changelog.md">changelog</a> should be up to date with current and upcoming features.</p> <p>If you have any questions, comments, requests or want to contribute please contact me on <a href="https://twitter.com/RudiMartinsen">Twitter</a>, or open an issue on <a href="https://github.com/rumart/GlobalDashboardPS/issues">GitHub</a>.</p>https://developer.hpe.com/from-developer-to-manager-to-freight-captain/https://developer.hpe.com/from-developer-to-manager-to-freight-captain/Wed, 17 Apr 2019 19:35:59 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/4/container-1638068_1920-1556306907818.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1556306907820"></p> <p>You’re a developer. You spend the majority of your day clacking away at your keyboard outputting lines and lines of code and working hard to vet your solutions. At the end of the day, you check your code in and mark issues as closed. Each day, you deliver your daily dose of syntactic morsels for everyone to admire.</p> <p>Because of the hard work you’ve put in, you earn the respect of others and move up to a technical lead position. You now spend the majority of your day architecting delightfully elegant solutions for technically complex problems. You mentor your team and keep them technically on-point while keeping an eye out for technical soundness. You pick up on the concept of delegation and really start to learn what it takes to be a leader. When you wake up in the morning, you know your team is on track with well-crafted and thoughtful solutions. When your day comes winding to an end, you feel satisfied knowing all of the project issues your team has closed. As you review the new and exciting features you’ve developed together, you feel pride in your team. Life is good!</p> <p>Time continues to pass and now you are officially leading as an engineering manager. You spend your time delegating, communicating, planning, and strategically positioning your team and organization for utter success. Your technical skills are neatly packed away and put in a drawer. And now, when you wrap up your day, <em>you</em> have nothing tangible to deliver.</p> <p>Having made this same transition myself, going from ending my day delivering tangibles to what feels like piloting a freighter on an endless journey at sea, I know how strange this can feel. At first, I felt as if I wasn’t being productive. When I wrapped up the day, I found myself trying to reflect on all of the things I accomplished – but was left with the feeling that the whole day was one giant blur of meetings and conversations I had with my team and the other groups I work with.</p> <p>The transition from individual contributor to manager is not always an easy journey. The skillset used to technically lead or write code has very little direct carryovers to people management and thoughtful leadership. It wasn’t until I replaced “I” with “we” did I really start seeing the tangible deliverables of this new role. Enabling my team to achieve all of the end-of-the-day items I previously accomplished and keeping them focused on ‘full speed ahead’, while avoiding the occasional iceberg, becomes my new deliverable.</p> <p>If you are traveling this same journey from developer to manager, you will face many challenges. Perhaps you have already transitioned and recognize many of these issues. The best advice I can give to folks making this transition is as follows:</p> <ul> <li> <p>Practice forbearance. Avoid the urge to open your code editor to quickly fix a bug in the app your team is working on. This practice is like leaving the bridge of the ship to focus on ensuring the cargo is safe. You’re not doing your team any favors by injecting your code into their projects. Instead, guide them to write effective code by transferring your technical knowledge to them.</p> </li> <li> <p>Embrace failure. Let your guard down and accept the failure scenarios. We all fall down. It’s how we pick ourselves back up and prevent future failure that truly defines us. The best learning experiences will always come from failure.</p> </li> <li> <p>Actively conduct retrospectives. Similar to allowing for failure, your retrospectives will be a great place for everyone to learn what worked and what didn’t work with a project.</p> </li> <li> <p>Set goals. Make sure you set short and long term goals with your team. Reflect on these goals regularly.</p> </li> <li> <p>Keep the skills drawer open. Find other ways to keep your coding skills sharp. You’ll still need them to remain a relevant leader to your team.</p> </li> </ul> <p>All in all, avoid focusing on the tangible deliverables. Focus on your team’s happiness and steering your cargo safely to port. If your team is well focused, happy, and motivated, the pieces they are building will fall correctly into place.</p> <p>Read more of our blogs at [hpedev.io.] (/blog) And if you haven’t yet signed up for our [Newsletter,] (/newsletter-signup) consider doing so to get more advice, tips, and tools to assist you with your own journey.</p>https://developer.hpe.com/2019-March-29/https://developer.hpe.com/2019-March-29/Fri, 29 Mar 2019 05:00:00 GMThttps://developer.hpe.com/hacking-away-at-the-paris-technology-and-solutions-summit-tss/https://developer.hpe.com/hacking-away-at-the-paris-technology-and-solutions-summit-tss/Thu, 28 Mar 2019 21:05:04 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/didier-couch-from-chris-1553888959463.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553888959463"></p> <p>This year, and for the first time at an HPE Technology and Solutions Summit, the HPE Developer Program decided to deliver a Hack Shack experience to event attendees. During the week of March 11 in Paris, France we offered developer and designer centric activities to 3,200 HPE and partner presales employees. In this very special Hack Shack – complete with modern décor, a comfortable sofa, an expresso machine, and video games – HPE hosted technical workshops for attendees to follow along on their own laptops and learn how to use the Application Programming Interfaces (API) for such HPE products as <a href="http://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView,</a> <a href="http://www.hpe.com/us/en/solutions/cloud/onesphere.html">HPE OneSphere,</a> <a href="http://www.hpe.com/us/en/integrated-systems/simplivity.html">HPE SimpliVity</a> and the <a href="http://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=Z7550-63371">HPE OneView Global Dashboard</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/lslsl-1553803713926.png" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553803713931"></p> <p>One didn’t need to be a developer to participate in these well-attended workshops; anyone could join to understand what it means to use REST APIs. The idea behind the Hack Shack was to make the event attendees feel more comfortable discussing APIs with DevOps and DevIT staff. The tools used during these labs included Postman and scripting languages such as PowerShell or Python. The Hack Shack also came fully equipped with Macs and PCs; so in case someone forgot to bring their laptop, they could still participate. We also hosted a session explaining both the HPE DEV program and the Design@HPE program. Dana and Parul offer some more insights into Design@HPE in this short interview that was placed on <a href="http://twitter.com/SBUCloud/status/1105771231008247809">Twitter</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/didier-people-standing-from-chris-1553888995259.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553888995260"></p> <p>On Wednesday afternoon and Friday morning, we hosted simple hackathons. We proposed a number of challenges to our attendees, using the same HPE OneView, HPE OneSphere, HPE SimpliVity, and HPE OneView Global Dashboard APIs. The most innovative, passionate, and creative ways to solve the challenges were rewarded with cool gifts, like a Fitbit watch and RasberryPi units. We ran 17 sessions in the Hack Shack, which were attended by 251 participants. All in all, we learned a lot and had a lot of fun.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/didier-handshake-from-chris-1553889004324.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889004325"></p> <p>Want to see more? Check out Didier’s interview on <a href="http://twitter.com/SBUCloud/status/1106149221701468166">Twitter</a></p>https://developer.hpe.com/discovering-hpe-dev-a-newcomers-perspective/https://developer.hpe.com/discovering-hpe-dev-a-newcomers-perspective/Thu, 28 Mar 2019 21:03:14 GMT<p>Digital transformation has flipped the industry’s focus from hardware-centricity to a software-defined reality. What makes up the server, and where it’s located, is no longer as important as what’s running on it and how seamlessly these applications can be accessed and used across the infrastructure. No matter how the infrastructure fabric is woven throughout on premise and cloud resources, the most important thing is to ensure the best user experience (UX).</p> <p>As a new member of the HPE DEV team, I was excited to learn about the great work that’s coming out of this group. This high-energy team is dedicated to the continuous improvement of both software development and UX design. Whether the software is being developed internally to HPE or is open source code – these folks are here to help.</p> <p>Modern collaborative DevOps methods have fundamentally changed software development processes, providing more efficient, modular processes that enable quicker time to market and allow developers to respond to customer’s needs faster. Additionally, these processes focus more on the importance of the user experience. After all, this is where customers live – at the user interface.</p> <p>With the shift to a more developer-centric mindset, HPE is empowering developers to manage infrastructure as code more effectively. Building an open source community allows HPE DEV engineers to work with developers directly to incorporate <a href="http://www.interaction-design.org/literature/article/what-is-design-thinking-and-why-is-it-so-popular">Design-Thinking</a> principles that will help the developers produce better applications for their customers.</p> <p>When I joined this team, I found it interesting that both development AND design were combined in the same group. In my experience, developers and designers spoke very different languages. This is partly because the skills that make a successful programmer aren’t the same as those required to create a good UI. But then I learned about <a href="http://v2.grommet.io/">Grommet</a>– a set of libraries that enable consistency in the design of code and application UIs, giving both designers and developers a way to inject consistent user experiences into enterprise apps easily. This open source UI dev tool is now being used by internal HPE and external developers to build enterprise applications that can be used in a variety of formats: web, desktop, and mobile. It also permeates HPE’s cloud and software-defined management products.</p> <p>As HPE continues to find better ways to work more closely with developers, you’ll see more coming out of this organization. The HPE DEV and design teams can connect you to HPE subject matter experts and resources. They can also help you develop software and solutions that improve automation, service delivery, and workflows, as well as your UX design. Maybe you’re new like me. If so, welcome to our newsletter! For those of you who have known us for a while, it would be great to connect and hear about some of your experiences with our team.</p>https://developer.hpe.com/design-thinking-workshops-at-tss-brighten-a-rainy-week-at-the-paris-expo/https://developer.hpe.com/design-thinking-workshops-at-tss-brighten-a-rainy-week-at-the-paris-expo/Thu, 28 Mar 2019 20:49:25 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/dana-paris-from-chris-1553889135905.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889135906"></p> <p>The weather at the <a href="http://h41382.www4.hpe.com/tss/">Technology and Solutions Summit at the Paris Expo</a> at the Paris Expo this past week was definitely a shade of gray to say the least. Rainy, cold, damp, and overcast -- with no sunshine the entire week. That didn’t keep Parul Tyagi and I from our mission to deliver Design Thinking workshops to conference attendees located in the Hack Shack. What is a Hack Shack? It’s a place to hang out, write some code, have a coffee, share ideas, and to brainstorm the next big idea, but most of all to have fun! The overall experience is better captured in this <a href="http://twitter.com/SBUCloud/status/1106149221701468166/video/1">tweet by Didier Lalli,</a> one of the Hack Shack founders.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/dana-hack-shack-from-chris-1553889149915.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889149916"></p> <p>The Hack Shack format was introduced to code lovers last year at HPE Discover in Las Vegas. The same setup was implemented at HPE Discover in Madrid this past November where we also introduced Design Thinking Workshops as part of the Hack Shack experience. This is where participants explore ideas through a guided ideation framework popularized by the <a href="http://dschool.stanford.edu/">Stanford d.school.</a></p> <p>In the Design Workshops, participants learn how to build upon ideas with their teammates by deferring judgement and by applying a few constraints that cause their ideas to “flare,” resulting in even more ideas being generated. Participants are asked to look to unexpected places for inspiration, a technique the d.school refers to as “analogous inspiration.” This is where something unrelated can help inform a solution.</p> <p>During my d.school training, facilitators Jeremy Utley and Perry Klabahn shared an analogous case study from Fairchild Semiconductor. By looking at the operating model of the floral delivery industry (an unexpected place), Fairchild was able to solve a supply chain issue they were having with a manufacturing component that was continuously on backorder. Neat, huh?</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/dana-accessory-table-from-chris-1553889169006.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889169007"></p> <p>During the brainstorm sessions at TSS, the evaluation of ideas was left to the very end. The purpose of ideation is to get as many ideas out in the open by deferring judgement while focusing on quantity versus quality. Then towards the end of the session, participants evaluate all of the ideas based on a few criteria:</p> <p><em>Which ideas are potentially a quick win?</em></p> <p><em>Which ideas are the most disruptive?</em></p> <p><em>Which ideas are the most delightful for end users?</em></p> <p>By enforcing constraints in and throughout the process, participants become focused and learn to how to prioritize. Each session is unique, and there are many “a-ha” moments for everyone. Participants are continuously surprised at the volume and variety of ideas this process promotes and often express interest in carrying the process forward into their daily work lives.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/dana-blond-woman-from-chris-1553889180664.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889180668"></p> <p>By the end of the week, we delivered seven well-attended workshop sessions -- a few were at capacity, which meant we had to turn people away. Over the past 6 months, we were busy delivering Design Thinking to a wide variety of groups throughout HPE . This grassroots effort is just the beginning of a cultural mindset shift to one that is more design-led.</p> <p>In addition to workshops, we delivered three design talks at the Hack Shack at TSS. We introduced the audience to our recently formed Chief Design Office, led by Bryan Jacquot, VP and Chief Design Officer for the Software Defined and Cloud (SDCG) organization and to our recently formed research function led by Amy Reitz, Ph.D. Additionally, we cited two recent ground-breaking publications that measure the impact of design maturity on the industry. The first, <a href="http://www.mckinsey.com/business-functions/mckinsey-design/our-insights/the-business-value-of-design">The Business Value of Design by McKinsey&#x26;Co,</a> discusses the idea that design adoption is not about the number of designers at a company, but more about design as a cultural mindset. The second, <a href="http://www.invisionapp.com/design-better/design-maturity-model/">The New Design Frontier by InVision,</a> discusses design maturity across 2,200 companies and 74 industries, highlighting the differences between Visionaries (companies who have adopted design as a business strategy) and Producers (those who simply focus on the pane of glass).</p> <p>We would love to hear your thoughts on the topic of design maturity. Stay tuned for more reporting from up and coming events such as: Google Cloud Next 19’, HPE Discover Las Vegas and more…</p> <p>See you next time,</p> <p>Dana and Parul</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2019/3/dana-and-parul-from-chris-1553889189849.jpg" alt="5bf2e1a0cd93d0796238ae01-blog-content-1553889189852"></p> <p>Want to see more? Check out Dana’s and Parul’s interview that was recorded and shared on <a href="http://twitter.com/SBUCloud/status/1105771231008247809">Twitter.</a></p> <p>Want to learn more about the Business Value of Design? Download the <a href="https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Design/Our%20insights/The%20business%20value%20of%20design/The-business-value-of-design-vF.ashx">McKinsey &#x26; Co. article here.</a></p>https://developer.hpe.com/coffee-and-critical-alerts/https://developer.hpe.com/coffee-and-critical-alerts/Wed, 20 Feb 2019 22:40:05 GMT<p>What could be better than starting your day with a hot cup of coffee and a custom-made critical alerts report? Ok, an exaggeration, but OneView Global Dashboard’s 1.7 (OVGD) release has several new features including a new report – Critical Alerts. This new report combined with the existing feature of scheduling reports can make a powerful ally in keeping your data center healthy by cutting time spent hunting for problems to fix and instead having them presented to you.</p> <p>In this post I’ll share how to create a custom Critical Alerts report and how to schedule it.</p> <p>Note: In order to create a custom report, enable email, and schedule a report, the user must have a role of type Infrastructure administrator.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/blog-figure1-1550702346651.png" alt=""></p> <p>After logging in to OVGD, select the Reports page to see the <em>Critical Alerts</em> report. First, select the report, which will run it, the report has ellipsis in the top right show the options available to you – see Figure 1.</p> <p>If customizations are desired, you will need to save a custom version of this report. This can be done by selecting <em>Save</em>. Customizations could include a search, sorting on a different column in the table, filtering by selecting any subset of the meters in the two meter groups (<em>Alerts by Occurred Date</em> and <em>Alerts by Resource Type</em>), or by adding or remove columns from the table. You can add or remove columns by selecting <em>Manage Report Content</em>, also shown in Figure 1.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/blog-figure2-1550702392981.png" alt=""></p> <p>Here is an example. Say you have an employee who troubleshoots all of the <em>server-hardware</em> issues. For that person, you could create a custom report filtered on the server-hardware meter underneath the Alerts by Resource Type meter group. Other customizations could be easily added as well. Once the report has the customizations you desire, remember to save it. Run that saved report and again click the ellipsis, but this time select <em>Schedule</em>. See Figure 2.</p> <p>Whoever is responsible for getting your data center from red to green status can have a custom made email sent to them each morning with their area of expertise highlighted. Or everyone could receive the same emails by adding multiple recipients to each one.</p> <p>OVGD has lots of flexibility built into the <em>Reports</em> feature. A cup of coffee combined with OVGD email may be just what you need to save some time and start your day off right.</p>https://developer.hpe.com/2019-February-19/https://developer.hpe.com/2019-February-19/Tue, 19 Feb 2019 06:00:00 GMThttps://developer.hpe.com/the-advent-of-ephemeral-infrastructure-as-code/https://developer.hpe.com/the-advent-of-ephemeral-infrastructure-as-code/Wed, 02 Jan 2019 15:40:59 GMT<p><strong>“Ephemeral infrastructure as code”</strong> refers to compute, storage, and networking that is programmatically assembled, provisioned and configured on demand to suit a variety of workloads. Once each workload execution is complete, resulting output is pushed to permanent storage and the infrastructure is available to be re-assembled and re-provisioned. These “composable” building blocks of compute, storage, and networking resources are available in the flexible workhorse known as HPE Synergy. Synergy offers a mix and match of compute nodes and storage drawers within each frame and is managed by a pair of Composer modules, running HPE OneView. A pair of redundant Image Streamer appliances is also available, which provides flexibility for automatically provisioning operating system and applications onto the Synergy nodes. Synergy plus Image Streamer is a powerful combination that is used at HudsonAlpha Institute for Biotechnology, a nonprofit genomics and genetics research institute in Huntsville, AL, to achieve ephemeral infrastructure as code.</p> <p>You may be thinking, “Write code and maintain GitHub repos? Isn't that more like a Developer or DevOps Engineer role? That's not really an IT thing.” But it can and should be! If your IT operations team is not writing code to communicate with infrastructure via API (application programming interface), you are missing an opportunity to decrease errors and increase innovation and speed of delivery of resources to developers and other end-users. Start by stocking the data center with API-capable hardware and then adopt a DevOps mentality within IT. There's something for everyone … HPE has OneView SDKs (Software Development Kits) available for a variety of languages including Python, Ruby, Go, Java, Hubot, and PowerShell.</p> <p>For the HudsonAlpha environment, I've chosen to use the Python SDK to provision 6 use cases onto Synergy nodes including OpenStack private cloud, bare metal Docker, and Kubernetes (K8s) cluster running on Fedora 27 and CentOS 7.4. The use cases align with the requirements for our genomic application pipelines.</p> <p>I've written a single Python script that can provision any one of the use cases and requires just a single argument of a use case configuration file. The Python script sends requests to Synergy's Composer via the OneView API, which applies the appropriate OneView profile template onto a Synergy node. The profile template contains an Image Streamer Deployment Plan, which consists of a Golden Image and Plan Scripts. The act of applying the profile template to the node includes automatic creation of an Image Streamer iSCSI boot volume for that node and creation of a clone of the Golden Image that is automatically customized per the Plan Scripts. The node boots from the iSCSI volume on Image Streamer and is fully personalized. How do I know once a node has booted and is ready? My Plan Scripts include automatic registration of the node with Hashicorp Consul, which is a service discovery tool. I can query Consul to see active nodes. I also include automatic installation of the DataDog monitoring daemon, so as soon as the node boots, metrics for the host OS and applications are visible within DataDog.</p> <p>HPE provides ready-to-use Artifact Bundles containing Deployment Plans for RHEL and SLES operating systems. But as I’ve shown, it is possible to create your own as well. See “docs” in my GitHub repo for instructions on creating Fedora 27 and CentOS 7.4 Golden Images, and more information about Consul and DataDog. Plan Scripts and OS Build Plans are stored within “Artifact Bundles” and Python scripts are inside the “projects” directory. Image Streamer is flexible, and there are many ways to accomplish automation. My approach and our use cases at HudsonAlpha are always evolving. I look forward to reading how others are using this technology!</p> <p>GitHub: <a href="https://hudsonalpha.github.io/synergy">https://hudsonalpha.github.io/synergy</a><br> HudsonAlpha Institute for Biotechnology: <a href="http://www.hudsonalpha.org">http://www.hudsonalpha.org</a></p> <p>Twitter: <a href="https://twitter.com/katmullican">@katmullican</a></p>https://developer.hpe.com/2018-December-19/https://developer.hpe.com/2018-December-19/Wed, 19 Dec 2018 06:00:00 GMThttps://developer.hpe.com/a-view-of-the-network-in-the-hack-shack-at-hpe-discover-madrid-2018/https://developer.hpe.com/a-view-of-the-network-in-the-hack-shack-at-hpe-discover-madrid-2018/Fri, 14 Dec 2018 14:20:30 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/dmad-stickers-1544797453006.png" alt="Discover Madrid Stickers"></p> <h1>#LETSHACKSHACKHowdy! Brian here, just back from the Hack Shack at HPE Discover Madrid. Hanging out with customers and partners was the best part of the show. Having fun helping them accelerate their cloud operational efficiency was icing on the cake. Yes, I said having fun. Having fun with code is a big part of the Hack Shack mantra and just one reason why I enjoy my job. Struggling with integration? Upgrade the API! Dealing with repetitive tasks? Automate! I code where I can make a difference and I have a great time doing it.</h1> <p>My colleagues and I, (<a href="https://twitter.com/netmanchris">Chris Young</a> and Mark Parenti), had the opportunity of representing <a href="https://www.hpe.com/us/en/integrated-systems/composable-fabric.html">HPE Composable Fabric</a> in the Hack Shack. We set out to do the impossible - ok, maybe just the highly improbable. Our mission: make network programming comprehensible and accessible. I know you must be chuckling about that mission statement but read on! You are about to embark on a journey of wonder and awe and see how truly simple your network can be.</p> <p>HPE Composable Fabric Manager (fka: Plexxi) was designed and developed from the beginning to be fully API driven. The web-based user interface uses the same RESTful API available to everyone - no special endpoints or mysterious parameters - and fully documented with Swagger.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/plexxi-swagger-1544797482316.png" alt="Plexxi Swagger Screenshot"></p> <p>To make programming even easier in the Hack Shack, we pulled together Python bindings for the API, and everyone’s favorite coding example, “hello world”. Several of the customers dropping by the Hack Shack with their laptops successfully programmed the Composable Fabric switches in just a few minutes. Let me show you how they were able to do so much in so little time.</p> <p>The first component is a client class that wraps the Composable Fabric API in easy to access properties and methods. Initializing the client with authentication credentials and the address of the Composable Fabric Manager instance establishes a secure connection. Convenience methods offer easy access to fabric switches, switch ports, and more. For our exploration, getting switches and ports along with updating ports will suffice.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/plexxi-ss-01-1544797520119.png" alt="plexxi ss 01"></p> <p>Once we have the API client initialized, a call to the <code>get_switches()</code> method returns the fabric switches, and <code>get_ports(switch_uuid)</code> returns the ports for a given switch. We’ll use the port identifiers to update their status later with calls to <code>update_ports()</code>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/plexxi-ss-02-1544797572994.png" alt="plexxi ss 02"></p> <p>The "secret sauce” in our example is mapping the characters in “hello world” to switch ports in binary ASCII! Then it’s just a simple matter of administratively enabling or disabling the appropriate ports to send our message to the world. Sure, this example has limited applicability in your data center and only your geekiest friends will appreciate it, but it illustrates the simplicity of programming the Composable Fabric API, making even the most powerful operations simple and automate-able. In fact, many of the integrations in Composable Fabric Manager do exactly that - leverage internal and external APIs to automate network discover, configuration, and remediation.</p> <p>The Hack Shack at Discover 2018 in Madrid was a great experience and I loved meeting all of you, hearing your ideas and inspiring you. I look forward to seeing you again next year. If you didn’t stop by this time, be sure to visit us at the Discover 2019 event in Vegas. Ask for me and let’s sit down and talk networking geek.</p> <p>And don’t forget! Keep an eye out for the upcoming Python bindings on <a href="https://hpedev.io">HPEDEV.IO</a>.</p>https://developer.hpe.com/get-hands-on-with-hpe-hybrid-cloud-and-container-solutions-at-kubecon-de/https://developer.hpe.com/get-hands-on-with-hpe-hybrid-cloud-and-container-solutions-at-kubecon-de/Mon, 03 Dec 2018 19:49:02 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/12/kubecon-na-2018-1543866788069.jpg" alt="kubecon na 2018"></p> <p>KubeCon is the Cloud Native Computing Foundation’s flagship conference, which gathers adopters and technologists from leading open source and cloud native communities for four days to further the education and advancement of cloud native computing. This year, HPE will be at <a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2018/co-located-events/">KubeCon 2018</a> in Seattle, WA from December 10-13, 2018.</p> <p>From beginners to experts, join HPE for their workshops covering the basics of how to implement HPE hybrid cloud and container solutions. Don’t miss your chance to walk through the challenges of deploying and managing applications in hybrid cloud environments and get help from HPE’s experts in the following areas:</p> <h2>End to End Hybrid Cloud Management using HPE OneSphere</h2> <p>Digital disruption is having a profound effect on every company. IT departments are being asked to support an increasing number of providers and manage a faster application cadence. HPE OneSphere helps by providing a managed cloud layer on existing virtualized infrastructure and offers a single portal to manage a hybrid cloud and container environment. In this workshop, you’ll learn how HPE OneSphere, with its powerful REST API, can compose hybrid clouds capable of supporting both traditional and cloud-native applications.</p> <h2>Storage for Containers</h2> <p>Come find out how HPE allows you to deploy a robust persistent storage layer with advanced data services for stateful containers on-premises and in the cloud using HPE 3PAR, Nimble, and Cloud Volumes.</p> <h2>Containers with End-to-End Life Cycle using HPE Composable Infrastructure</h2> <p>HPE Composable Infrastructure provides the flexible infrastructure you need to incorporate container services into DevOps, host existing applications, add new micro services, and modernize legacy applications. It enables you to dynamically provision and scale applications, whether they run in VMs or containers. The software-defined architecture allows you to compose compute, storage, and networking resources to target specific workloads. And HPE Image Streamer enables stateless deployment of the operating system and other supporting software required to make the resources ready to run. Join this workshop to learn more.</p> <h2>HPE ProLiant for Microsoft Azure Stack</h2> <p>Azure stack allows you to bring modern cloud services to your sensitive data and edge applications that may not yet be suitable for the public cloud. Join HPE experts to learn how the HPE ProLiant for Microsoft Azure Stack is a hybrid cloud solution that enables you to deliver Azure services from your data center.</p> <p>To learn more about KubeCon 2018, <a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2018/co-located-events/">click here</a>. See you at the show!</p>https://developer.hpe.com/configuring-threads-for-optimal-performance-in-hpe-powershell-cmdlets/https://developer.hpe.com/configuring-threads-for-optimal-performance-in-hpe-powershell-cmdlets/Mon, 03 Dec 2018 07:06:34 GMT<p>Initially when HPE PowerShell cmdlets (1.x version) were launched it had a limit of 256 thread limit which was good for a certain range of IPs but was not sufficient for a big set of input IP data like more than 10k range. Then we re-designed the default thread limit to 64 in iLO and BIOS modules which received good feedback initially but later few customers reported that its slow performance when a big range of data is provided. Then we came up with the idea of configurable threads where on a particular PowerShell session users can configure the threads above default or below that value using Get-HPEXYZMaxThreadLimit and Set-HPEXYZMaxThreadLimit (XYZ - means module name like iLO or BIOS) cmdlets like always go for even exponential power of 2 number. For e.g., 64, 128, 256(default), 512, 1024, 2048, 4096 (Maximum).</p> <p>Now the actual problem started customers does not know which thread limit they need to configure and with lack of knowledge they configure a higher limit on less capable hardware systems the result is lower performance. Even with optimal hardware many times you don't get good performance. Then we did a case study on this and found that just by increasing the thread limits will not give good performance as a lot of time is spent on switching the threads itself than doing the real job (Processing iLO IP's).</p> <p>For e.g., Customer had 350 good iLO IP addresses in an input csv list with 256 threads he had below performance results compared to same input data with 64 max thread limit configured. Below the table of test data will show you how discovery and connect cmdlets took time under different max thread circumstances. With more tests with different max thread configured it was evident that there is a mapping between input IP's and thread count configured that need to be maintained for optimal performance results.</p> <h1>Our team recommends 5:1 ratio needs to be maintained for fixed input of IP addresses (All reachable and good iLO's).</h1> <p>i.e., For every 5 iLO IP addresses, one thread need to be configured is optimal. I mean if input IP count is 320 then optimal thread range you can have is 64 minimum threads. Anything more or less ( like 128 or 32) will result in not best performance results.</p> <p><em>Physical Server Configuration: BL460c Gen9 with 8 Core (Hyper-Threaded to 16), 32GB of memory, Network Card FlexFabric 10GB.</em></p> <table> <thead> <tr> <th>Tests</th> <th>Version</th> <th>Count</th> <th>Total</th> <th>Discovery</th> <th>Connect</th> <th>Disconnect</th> <th>Server</th> <th>MaxThread</th> </tr> </thead> <tbody> <tr> <td>Test1</td> <td>2.1</td> <td>346</td> <td>01:33</td> <td>00:37</td> <td>00:22</td> <td>00:03</td> <td>00:29</td> <td>64</td> </tr> <tr> <td>Test2</td> <td>2.1</td> <td>346</td> <td>02:34</td> <td>00:43</td> <td>00:29</td> <td>00:05</td> <td>01:15</td> <td>256</td> </tr> <tr> <td>Test3</td> <td>2.1</td> <td>346</td> <td>02:39</td> <td>00:40</td> <td>00:35</td> <td>00:05</td> <td>01:18</td> <td>512</td> </tr> </tbody> </table> <p>What if you have input data in terms of range like 192.168.10.0-255 ( means 256 IP's) or 192.168.10-12 ( means 768 IP's) that is given as input range to any of our cmdlets then the above 5:1 ratio may not hold good as in the above case customer is not sure which ones are iLO in the above range. In this case, higher max thread limit will give better performance results. Below data will give more insights.</p> <p><em>Physical Server Configuration: BL460c Gen9 with 8 Core (Hyper-Threaded to 16), 32GB of memory, Network Card FlexFabric 10GB.</em></p> <table> <thead> <tr> <th>Tests</th> <th>Version</th> <th>Count</th> <th>Total</th> <th>Discovery</th> <th>Connect</th> <th>Disconnect</th> <th>Server</th> <th>MaxThread</th> </tr> </thead> <tbody> <tr> <td>Test1</td> <td>2.1</td> <td>118</td> <td>02:12</td> <td>01:36</td> <td>00:23</td> <td>00:01</td> <td>00:11</td> <td>64</td> </tr> <tr> <td>Test2</td> <td>2.1</td> <td>119</td> <td>01:49</td> <td>01:16</td> <td>00:18</td> <td>00:01</td> <td>00:12</td> <td>256</td> </tr> <tr> <td>Test3</td> <td>2.1</td> <td>119</td> <td>00:43</td> <td>00:43</td> <td>00:19</td> <td>00:01</td> <td>00:12</td> <td>512</td> </tr> </tbody> </table> <p>You can see as Max threads limit is increased the Total time taken column is reducing that means we are getting better performance results with higher thread limit which was not the case with above numbers where we had only good iLO's.</p> <p>So the bottom line is depending on the type of input IP addresses the customer has one need to decide the Max Thread limit configuration. Of course, your host hardware needs to have sufficient CPU and RAM to support higher thread configurations. I hope this will help customers who want to use a large range of IP's in their environment to choose the Max Thread limit accordingly. (Default is 256)</p> <p>Note: Performance numbers taken above will depend on various factors like network (LAN\WAN) load, iLO server load at that point of time. These numbers are only for reference purpose please don't take them as ideal figures for Max thread limit.</p>https://developer.hpe.com/building-hpes-new-design-portal-with-headless-wordpress-and-nextjs/https://developer.hpe.com/building-hpes-new-design-portal-with-headless-wordpress-and-nextjs/Tue, 27 Nov 2018 20:35:27 GMT<h2>Work smarter, not harder</h2> <p>I can't quite recall who planted the "work smarter, not harder" seed in my head but for the past six months, it has been resonating non-stop in an almost comical manner. This project reflected that mantra to a T.</p> <p><img src="https://media.giphy.com/media/SSw1KX5otdqPm/source.gif" alt="work smarter not harder"></p> <h2>The brief ...</h2> <p>An application concept recently arrived at my desk regarding constructing a new design portal for HPE, similar to the HPE DEV portal you're currently using. The goal for the application is to house all things related to HPE's new design program, including all of the various design resources we offer. My career at HPE has generally revolved around building or leading the construction of this same type of web application in various flavors. The stack usually includes a <a href="https://v2.grommet.io/">grommet</a> based front end and some form of a custom content management system with an API. In fact, this very developer portal was built upon 100% custom solutions built over the course of three months. With each iteration of this fullstack scope, we've eventually widdled away the complexity of a fully custom back and frontend. The difference between this new design portal project and previous projects was that we had <strong>3 weeks</strong> to ship it for <a href="https://www.hpe.com/events/discover/">Discover '18 Madrid</a>. The <em>"work smarter, not harder"</em> mantra showed up in full swing. What was once a Grommet based CMS was replaced with a headless instance of Wordpress running in a Docker container. A custom Webpack configuration which constantly needed tuning and a homebrewed server-side rendering engine were replaced by <a href="https://nextjs.org/">Next.JS</a> with Grommet.</p> <h2>Off to the races</h2> <p>As we began building on this new stack I felt a huge weight being lifted off of my shoulders. I was no longer focusing on whether or not the tooling or build processes were working as expected and I could focus on what was important - the end user's experience. Not having to wrestle a webpack config or babel transpiling gave us the runway we needed to make sure this project was successful.</p> <p>Next's server-side rendering and bundling worked flawlessly out of the box and it integrated perfectly with Grommet. New stacks come with a learning curve, however, this one was quite low. Here's some items we needed to pick up to create a great Grommet based Next frontend:</p> <ul> <li>Handling page <a href="https://github.com/zeit/next.js#routing">routing</a></li> <li>The next/link component and nesting anchor tags within it</li> <li>Handling route parameters, <a href="https://github.com/zeit/next.js/tree/canary/examples/parameterized-routing">here's a great example</a> of the approach we took</li> <li>Creating new components to handle SEO and meta tags</li> <li>Extending Next's custom babel config to support styled components, here's a <a href="https://dev.to/aprietof/nextjs--styled-components-the-really-simple-guide----101c">great article</a> explaining how to handle this</li> </ul> <p>While we had cleared up tooling and build complexities on the front end there was still the matter of handling dynamic content. I had kicked around headless Wordpress in a few projects but always walked away wondering if it was the right choice. As time has gone by the ecosystem has ironed out a lot of the complexities which made me question the solution, WP API has now become a staple in web application development. We started out by referencing <a href="https://github.com/postlight/headless-wp-starter">Postlight</a> to get a better understanding of the current state of headless Wordpress. I ended up not using Postlight's approach to Wordpress but it did provide me with some keen insight into proper architecture. Ultimately, we chose to roll our own environment variable driven Docker compose setup which would handle Wordpress, MySQL, SSL, Nginx, backups, and obtain certificates on the fly from Let's Encrypt. Our final Wordpress build included the following plugins:</p> <ul> <li><a href="https://wordpress.org/plugins/acf-to-wp-api/">ACF to WP API</a> to expose custom metadata in the Wordpress API</li> <li><a href="https://www.advancedcustomfields.com/">Advanced Custom</a> Fields to allow for flexible content entry</li> <li><a href="https://wordpress.org/plugins/custom-post-type-ui/">Custom Post Type UI</a> to quickly create and edit post types without requiring code changes or rebuilds</li> <li><a href="https://deliciousbrains.com/wp-migrate-db-pro/">WP Migrate Pro</a> to effortlessly sync data from our staging to our production environment</li> <li><a href="https://wordpress.org/plugins/wp-rest-api-v2-menus/">WP-REST-API V2 Menus</a> to expose Wordpress' native nav menu editing to the API</li> </ul> <p>That's all it took to create a fully featured CMS in about a week.</p> <p><img src="https://media.giphy.com/media/XH6MU5zmqIpAA/giphy.gif" alt="gotchas"></p> <h2>Headless Wordpress gotchas</h2> <p>I've picked up a few things after building Wordpress APIs, here's what to look out for:</p> <ul> <li>Your API will most likely live in its own sub-domain to avoid exposing non-traditional web ports like <code>8000</code> in your URLs</li> <li>Keep <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS">CORS</a>, cross-origin resource sharing, in mind. You may need to include a <a href="https://github.com/ahmadawais/WP-REST-Allow-All-CORS">plugin</a> to allow cross-domain fetches</li> <li>If you're running Wordpress in Docker consider state, be sure to mount and backup the media uploads volume and of course your MySQL database</li> <li>Disable your Wordpress front end with <a href="https://github.com/postlight/headless-wp-starter/blob/master/wordpress/wp-content/themes/postlight-headless-wp/index.php">redirects</a></li> </ul> <h2>The perception of value through customization</h2> <p>I was once naive enough to think custom solutions added enough technical value to a project to disregard the delivery timeline and simply worked harder. While building these custom solutions are absolutely invaluable when it comes to learning it's not always a sustainable practice. I still struggle to justify my team's decision to use Wordpress for our data layer to my colleagues but when all of my technical hurdles come solved out of the box it's hard to refute the value of a staple like Wordpress. When we look at buying a car, we generally don't buy the pieces separately then build the engine. We find the car we like and we drive it.</p> <p>Now that you've listened to my long-winded spiel about libraries, content management, and frameworks <strong><a href="https://hpe.design/">check out the new HPE design portal!</a></strong></p> <p><img src="https://media.giphy.com/media/BRpMznCmYTiik/giphy.gif" alt="88 mph"></p>https://developer.hpe.com/dont-miss-it-hpe-discover-hack-shack-offers-something-for-everyone-at-ev/https://developer.hpe.com/dont-miss-it-hpe-discover-hack-shack-offers-something-for-everyone-at-ev/Mon, 26 Nov 2018 17:59:48 GMT<p><a href="https://www.hpe.com/events/discover/">HPE Discover Madrid</a> kicks off next week! If you are going to the event, the HPE Discover Hack Shack will offer unique hackathons, technical workshops, and coding challenges focused on a variety of HPE products, including HPE OneSphere, HPE SimpliVity, and the new HPE Composable Fabric Manager API. Whatever your skill level, the HPE Discover Hack Shack has a session for you. Even if you don't know how to code, the Hack Shack located in the Transformation Showcase is still your destination to participate and win some great prizes.</p> <p>If you will be at HPE Discover next week, make sure the below Hack Shack activities are on your must-attend list:</p> <h2>Design Collaboration Workshop at the Hack Shack: Discover Your Next Big Idea!</h2> <p>Do you have an amazing idea that will accelerate your company’s business, but you don’t know how to develop it further? Join the <a href="https://developer.hpe.com/">Design@HPE</a> team in the HPE Discover Hack Shack to take it to the next level. To help understand your vision, the design team will get to know more about you, your business, and how you interact with HPE products and solutions. You will collaborate with the team through immersive design and creative activities to help build out your next big idea! Everyone is welcome to attend this session. Design Playground Workshop at the Hack Shack: Engage in New User Experience Gameplay Get engaged with the Design@HPE team at HPE Discover in an interactive design playground to help create some amazing design experiences. You’ll interact with HPE’s latest prototypes while the design team listens to gain insight about who you are, what you do, and how you use HPE solutions. Through interactive play, we will have the opportunity to build out some memorable designs for future solutions and products. And, you get a chance to win cool prizes! Everyone is welcome to attend this session.</p> <h2>Interested in API’s? Learn how to make hybrid cloud simple with the HPE OneSphere API at this Hack Shack Workshop</h2> <p>Beginners or experts, come to the HPE Discover Hack Shack to learn more about <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a> and hybrid cloud management. In this workshop, you will learn how to interact with HPE OneSphere’s SaaS platform through its powerful REST API. HPE experts will start from the HPE DEV website, explore the API description, and then use the Postman open source tool to interact with the API. Remember to bring your laptop so you can learn how to make your code come alive in this session! Experts will help you use PowerShell (or Python) to write a script to automate hybrid cloud operations.</p> <h2>Dive into infrastructure automation with the HPE OneView API in a Discovery Workshop at the Hack Shack</h2> <p>Bring your laptop, come in, get comfortable, and be ready to learn about <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a>, HPE’s <a href="https://www.hpe.com/us/en/solutions/infrastructure/composable-infrastructure.html">composable infrastructure</a> management platform. During this HPE Discover interactive session, you’ll explore the API description and use the Postman open source tool to interact with it. Next, you will use PowerShell (or Python) to write a script that automates HPE OneView. By the end of the workshop, you will have written a simple example of infrastructure as code. Beginners and experts are welcome!</p> <h2>HPE OneSphere Hackathon – Create a super cool unified Hybrid Cloud Dashboard</h2> <p>Do you dare enter the 2-hour HPE OneSphere multi-cloud management challenge? In this session, you will build a functional report that shows all the existing deployments grouped by projects for each cloud provider. All solutions are accepted — from text reports to super cool graphical dashboards. Bring your friends, pick your favorite language, and let the challenge begin! The creators of the most innovative hack will also walk away with great prizes! Beginners and expert developers welcome.</p> <h2>HPE OneView Hackathon — Find the Misbehaving Server in a Large Datacenter</h2> <p>Want to challenge yourself to build a solution for finding a misbehaving server -- with only limited information and some logs from its iLO in just 2 hours? If this kind of engineering challenge meets your operational role, this hack is for you! All solutions are accepted as long as they do the job. Bring your co-workers, pick your favorite language, and show off your app-building chops. Winners will head home with great prizes! Beginners and expert developers welcome.</p> <h2>HPE Hack Shack: Welcome to the Hello World Hackathon</h2> <p>Enter the HPE Discover Hack Shack backyard to learn how to code or improve your current coding skills! Use your favorite language or embrace the challenge of using a new language you want to learn. Your hack can be as simple as showing Hello World in notepad or a text editor like Vim. Or, you can send a handshake message to a server and have the server return Hello World back to you. But remember, creativity counts! And of course, prizes will be awarded. Beginners and expert developers welcome.</p> <h2>What is the HPE Composable Fabric Manager API? Learn all about it at the Hack Shack Discovery Workshop</h2> <p>Interested in learning about HPE’s latest technology acquisition? Bring your laptop and come learn about the HPE <a href="https://www.hpe.com/us/en/integrated-systems/composable-fabric.html">Composable Fabric</a> Manager and its REST API. Access the Swagger-based API documentation from the management appliance, explore the API documentation, and execute API commands. You will use Python to write a script to automate network configuration and make it come to life. For beginners to experts, come prepared to code. Then apply your knowledge in the HPE Composable Fabric Manager Hackathon.</p> <h2>HPE Composable Fabric Manager Hackathon</h2> <p>Are you a networking engineer who has to track metrics and build dashboards? Think you can build a dashboard that shows the current configuration of the network view of the Composable Fabric mesh network – all in 2 hours? Come to this session in the HPE Discover Hack Shack to test your skills! All solutions are accepted, from text reports, graphical interpretations, representations of data, and anything in between. Pick your favorite language and get hacking! Prizes will be awarded to the most innovative solution; beginners and expert developers are all welcome.</p> <h2>Hack Shack HPE SimpliVity Workshop: Learn more about the HPE SimpliVity REST API fundamentals to build a simple automation script.</h2> <p>Are you an engineer interested in learning about HPE SimpliVity? Bring your laptop and join the HPE Developer Community in the HPE Discover Hack Shack backyard to learn how to interact with the HPE SimpliVity hyperconverged platform through its REST API. Explore the API description, communicate with the HPE OmniStack Virtual Controller, and finally use PowerShell (or Python) to automate the kind of HPE SimpliVity use case that may come up in your day-to-day work. For beginners to expert developers alike, come prepared to code at this session.</p> <h2>HPE SimpliVity Backup Status Report Hackathon</h2> <p>Are you an engineer eager to try your hand at creating a backup status report in under 2 hours? Enter this hack and challenge yourself on hyperconvergence. This hack will focus on creating a report that shows the status of on-going, successful, and failed backups. Leverage the granularity and simplicity of the HPE SimpliVity REST API to build out your solution. Prizes will be awarded for the most innovative hack. Beginners to expert developers are invited.</p> <h2>HPE SimpliVity User Scripts and Backup Hackathon</h2> <p>Are you an engineer tasked with producing and understanding backups? Wish there were more innovative ways to create them? This 2-hour challenge can get you started to creating that innovation. You will leverage the HPE SimpliVity REST API to build a solution for pre-and-post backup script injection to enable backup life cycle management use cases. Use your favorite language and see if your hack walks away with the winning prize! Beginners and expert developers are welcome.</p> <p>There is still time to register for HPE Discover Madrid 2018, <a href="https://attend.hpe.com/discover2018madrid/">click here</a> for more info.</p>https://developer.hpe.com/using-postman-with-hpe-simplivity/https://developer.hpe.com/using-postman-with-hpe-simplivity/Fri, 09 Nov 2018 17:39:07 GMT<h1>Summary</h1> <p>This article is the first in a series that describes how to use the HPE SimpliVity API. The articles are targeted at developers and architects that want to understand the REST API’s capabilities and are interested in learning how to build automation and integration with HPE SimpliVity.</p> <p>This article uses Postman to place HPE SimpliVity REST API calls without writing any code. Postman is a free application from Postdot Technologies. It is available for Windows, Mac, Linux and as a Chrome browser plugin. You can download and install it from <code>https://www.getpostman.com/postman</code>.</p> <p>At the end of this article, you should be able to use Postman to create variables to store the values that you need to use in other Postman exercises and to retrieve the set of virtual machines managed by an HPE SimpliVity federation.</p> <p>This article assumes that you understand how to navigate and use the Postman interface. If you are totally new to Postman, I recommend that you watch a few videos on creating variables in Postman.</p> <h1>Let's get started!</h1> <p>To begin, I need the credentials to access to the HPE SimpliVity virtual controller through TCP/IP. You can use the vCenter local SSO administrator for the vCenter used in the HPE SimpliVity federation. I use my domain username which has the <code>username@domainname</code> format.</p> <p>I also need to disable Postman’s certificate validation. To turn the feature off, I click the icon on the Postman ribbon. (You can skip this if you have signed the SSL certificate used by HPE SimpliVity with trusted root.)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-turn-off-ssl-cert-1542122449127.png" alt="svt turn off ssl cert"></p> <h1>Set up environment variables</h1> <p>I use variables to store the values that I know I need to use in my Postman requests. First, I declare the variables that I use as input to REST API requests:</p> <ul> <li><code>OVC_IP</code>: Stores the virtual controller IP address.</li> <li><code>VcenterAdminName</code>: Stores my vCenter user name.</li> <li><code>vCenterAdminPasswd</code>: Stores my vCenter password.</li> </ul> <p>Next, I declare a set of variables to store the results of my REST API requests:</p> <ul> <li><code>token</code>: Stores the access token that is returned from an authenticated session. (We’ll learn how to authenticate in the next article.) I'll need this for many requests, so storing it in a variable makes it easy to transfer it among the requests.</li> <li><code>TASKID</code>: Stores HPE SimpliVity task IDs.</li> <li><code>VMID</code>: Stores the virtual machine ID.</li> </ul> <p>For example:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-postman-variable-definitions-1542122682743.png" alt="svt postman variable definitions"></p> <h1>Define the URI, header, and body parameters to request an access token</h1> <p>For my first REST request, I use the POST action to send the access token request. I need to define the URI, header, and body parameters for it.</p> <p>The URI to request an access token has this syntax: <code>https://simplivity@&#x3C;OVC_IP>/api/oauth/token</code></p> <p>I use the <code>OVC_IP</code> variable that I defined earlier.</p> <p>I need to send the <code>Accept</code> and the <code>Content-Type</code> headers with my request, so I define two key-value pairs for them. For example:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-url-body-params-1542122702407.png" alt="svt url body params"></p> <p>Finally, I define three key-value pairs for the Body of the request: <code>grant_type</code>, <code>username</code>, and <code>password</code>.</p> <p>In Postman, I select <code>form-data</code> to store these key-value pairs. I assign the <code>VcenterAdminName</code> and <code>vCenterAdminPassword</code> variables to the username and password keys.</p> <p>For example:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-formdata-body-params-1542122691875.png" alt="svt formdata body params"></p> <p>I then move to the Tests tab where I define a script that intercepts the return value from the REST request. It parses the keyword token and stores it into the <code>data_access_token</code> variable.</p> <p>For example:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-test-script-postman-1542124697949.png" alt="svt test script postman"></p> <p>Assuming that you have entered the <code>OVC_IP</code>, <code>username</code>, <code>password</code> and the key-value pairs, you are ready to send a REST request by clicking <strong>Send</strong>. That's it! You just made your first HPE SimpliVity REST request.</p> <p>This request returns the access token that you need to use on many HPE SimpliVity requests. The access token value from my example is shown highlighted in yellow in the following example.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-request-access-token-1542124824497.png" alt="svt request access token"></p> <p>You can now validate that the token’s current value is the same as the value in the body of the POST result. It’s so easy isn’t it?# This is easy, let's take it up a notch!</p> <p>Now that I have the access token, I can perform other REST operations using the same variables that I built before.</p> <p>Let’s give it a try and get all of the virtual machines managed by HPE SimpliVity.</p> <p>Start by creating a GET operation in a new Request tab of Postman. Just like before, we need to define the URI for the GET request. It has this syntax:</p> <p><code>https://&#x3C;OVC_IP>/api/virtual_machines</code></p> <p>Since I have defined the variable OVC_IP, let’s use it for our next adventure.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-get-all-vms-url-1542125040577.png" alt="svt get all vms url"></p> <p>Enter the parameters for this request by clicking on the Params tab.</p> <p>Provide the <code>limit</code>, <code>offset</code>, and <code>show_optional_fields</code>, as defined in the specification for the REST API GET VM call. You can find the definition here: <a href="https://developer.hpe.com/api/simplivity/endpoint?&#x26;path=%2Fvirtual_machines">https://developer.hpe.com/api/simplivity/endpoint?&#x26;path=%2Fvirtual_machines</a></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-get-all-vms-params-1542125124954.png" alt="svt get all vms params"></p> <p>I also need to provide the access token that I acquired in the previous POST request. Because I used variables, there is no need to worry about copying the 16+ character hexadecimal value without error. I just enter the Authorization value in the form of <code>Bearer {{token}}</code> then click <em>Send</em>, and let Postman do the magic.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-get-all-vms-headers-1542125232197.png" alt="svt get all vms headers"></p> <p>The request returns the list of the virtual machines that are managed by HPE SimpliVity. You can see the offset value and the limit value for validation of the parameters that I entered for this request.</p> <p>I highlighted the VM ID below because it is one of the fields that is useful for many other requests.</p> <p>You can use the same strategy that I did above by entering the VM ID into another variable and so on.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/11/svt-offset-limit-1542125309822.png" alt="svt offset limit"></p> <p>That's it! # What have I learned?</p> <p>Postman is great for experimentation to develop my understanding of the REST API. Postman supports the use of variables that can be manipulated with very little effort.</p> <p>Do I want to write a script using Postman? Probably not. I use PowerShell as my choice for automation as you can see in the following YouTube video: <a href="https://www.youtube.com/watch?v=pBnadRc1Vsw">https://www.youtube.com/watch?v=pBnadRc1Vsw</a></p> <p>Scripting languages, such as PowerShell, provide a rich library of features that I need to perform automation.</p> <p>To learn more about using Python curl to obtain an access token, go to <a href="https://developer.hpe.com/platform/hpe-simplivity/authenticating-against-hpe-omnistack-api">https://developer.hpe.com/platform/hpe-simplivity/authenticating-against-hpe-omnistack-api</a>.</p>https://developer.hpe.com/storage-management-with-redfish/https://developer.hpe.com/storage-management-with-redfish/Tue, 30 Oct 2018 17:05:18 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <style> .warning-box { background-color: #fff3cd; /* Light yellow background */ /* border: 2px solid #ffeb3b; /* Yellow border */ color: #856404; /* Dark text color for contrast */ padding: 20px; /* Padding inside the box */ border-radius: 5px; /* Rounded corners */ margin: 20px 0; /* Margin for spacing */ width: 80%; /* Width of the rectangle */ max-width: 600px; /* Maximum width of the rectangle */ margin-left: auto; /* Center the rectangle horizontally */ margin-right: auto; /* Center the rectangle horizontally */ /* box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1); /* Optional: Adds a shadow for depth */ } .warning-box p { margin: 0; /* Remove default margin from the paragraph */ font-weight: bold; /* Bold text */ } .warning-box a { color: blue; /* Change the color of the link */ text-decoration: none; /* Remove the underline */ /*font-style: italic; /* Make the text italic */ /*font-weight: bold; /* Make the text bold */ } </style> <p>Updated: December 10, 2024</p> <div class="warning-box"> <p> NOTE: </p> <p> Due to the deprecation of the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo6/ilo6_adaptation/#hpe-smart-storage-model-oem-deprecated" target="_blank">SmartStorageConfig</a> data model and the adoption of the <a href="https://developer.hpe.com/blog/overview-of-the-platform-level-data-model-for-redfish%C2%AE-device-enablement-standard/" target="_blank">PLDM for RDE</a> standard by HPE iLO and storage device providers, this blog post is deprecated. </p> <p> Refer to the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/storage/#storage-data-models" target="_blank">Storage data models</a> documentation section for up to date information concerning Redfish storage management. </p> </div> <h2>Introduction</h2> <p>Integrated Lights-Out (<a href="https://www.hpe.com/info/ilo" target="_blank">iLO</a>) is an HPE ProLiant and Synergy server-embedded technology that delivers the core foundation for server intelligence along with out-of-band and in-band management facilities. This technology is a combination of the iLO ASIC that is part of the server board and the firmware that powers the ASIC. Out of the box, HPE iLO simplifies server setup, provides access to server health information, and enables server management at scale, including basic remote administration. Different generations of ProLiant Servers carry different versions of the iLO ASIC.</p> <p>In HPE ProLiant and Synergy Gen10 servers, HPE iLO 5 introduced the management of storage controllers via its graphical user interface and via the <a href="https://www.dmtf.org/standards/redfish" target="_blank">Redfish</a> RESTful API standard. Although <a href="https://www.youtube.com/channel/UCIZhrIYcNh3wHLiY4ola5ew/search?query=logicaldrive" target="_blank">videos</a> already exist that cover the graphical user interface, I wanted to address this feature with a pure Redfish API approach, bypassing the <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">iLOrest interface tool</a> and its <code>SmartController</code> macro commands.</p> <p>In this article you start by learning how to cleanup and prepare a SmartRAID (SR) storage Controller for receiving a configuration with one or more logical drives using an HPE proprietary OEM process. Then, on this fresh environment, you will learn how to create a simple RAID array configuration prior to more complex ones.</p> <h2>Foreword</h2> <p>I've used the <a href="https://www.getpostman.com/" target="_blank">Postman</a> API development to illustrate our examples. This should give you the ability to implement these raw examples using your own preferred language.</p> <p>The reader should have the knowledge of HTTP <a href="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods" target="_blank">request methods</a> like <code>GET</code>, <code>PUT</code> and <code>PATCH</code>.</p> <p>Moreover, it is assumed that the reader knows how to manage Redfish sessions as described in the <a href="https://servermanagementportal.ext.hpe.com/docs/concepts/redfishauthentication/" target="_blank">HPE Redfish API Reference documentation</a>. More hints for managing iLO sessions with Redfish can be found in this <a href="https://developer.hpe.com/blog/managing-ilo-sessions-with-redfish/" target="_blank">article</a>.</p> <h2>Storage data models</h2> <p>The Redfish standard defines a <code>storage</code> data model as part of the <code>ComputerSystem</code> resource type under the <code>/redfish/v1/Systems/{item}/</code> URI. With the implementation of new standards, like the Platform Level Data Model for Redfish Device Enablement (<a href="https://developer.hpe.com/blog/overview-of-the-platform-level-data-model-for-redfish%C2%AE-device-enablement-standard/" target="_blank">PLDM for RDE</a>), this model is fully operational in terms of read, write and event operations against modern external provider devices. Starting at Gen10 and Gen10 Plus (firmware 2.30+), HPE servers can take advantage of it. Note that it is the only storage data model implemented in Gen11 servers and beyond.</p> <p>HPE initially developed the <code>SmartStorage</code> Redfish OEM data model for HPE ProLiant DL580 Gen8 servers, before any Redfish specification was published. This model supports inventory (GET) and monitoring (Events) features only.</p> <p>In HPE ProLiant Gen10, the <code>SmartStorageConfig</code> resource was added to support the system's configuration. This OEM model uses a proprietary API that only supports the SmartRAID (SR) line of HPE storage controllers.</p> <blockquote> <p><strong>Note</strong>: The HPE OEM <code>SmartStorageConfig</code> data model is removed in HPE Gen11 servers.</p> <p><strong>Note</strong>: This article focuses only on the HPE OEM <code>SmartStorageConfig</code> data model.</p> </blockquote> <p>As stated earlier, HPE Gen10 and Gen10 Plus servers implement the description and the configuration of their Smart Storage SR devices respectively in the <code>SmartStorage</code> and <code>SmartStorageConfigX</code> (<strong>X</strong> is being an ID number) entry points of the same <code>ComputerSystem</code> type. When multiple controllers are present in a system, a <code>SmartStorage/ArrayControllers/{item}/</code> entry point is present for each controller description and a <code>SmartStorageConfigX</code> entry point exits for the configuration entry points. Note that <code>{item}</code> and <strong><code>X</code></strong> are not correlated.</p> <p><img src="/img/0-smartstoragearraysentrypoints.png" alt="HPE&#x27;s legacy SmartStorage URIs" title="HPE&#x27;s legacy SmartStorage URIs"></p> <p><img src="/img/0-smartstorageconfigentrypoints.png" alt="HPE&#x27;s SmartStorageConfig URIs" title="HPE&#x27;s SmartStorageConfig URIs"></p> <p>Since the infrastructure used for this article contains only HPE storage devices and the iLO 5 firmware version is below 2.30, the <code>Storage</code> DMTF URI is not populated. Only the <code>SmartStorage</code> and <code>SmartStorageConfigX</code> locations are populated. The following picture shows the URIs of four disk drives in the first controller of our infrastructure. Their properties can be obtained with a simple <code>GET</code> request.</p> <p><img src="/img/1-physicaldrives.png" alt="Physical Drives seen from the SmartStorage location" title="Physical Drives seen from the SmartStorage location"></p> <p>If you only need the physical location of the disks to use for your logical drives, you can <code>GET</code> them from the <code>SmartStorageConfig</code> location as shown below. Note that for the first controller found in the system, the <strong><code>X</code></strong> ID number is omitted.</p> <p><img src="/img/2-physicaldrives.png" alt="Physical Drives seen from the SmartStorageConfig location" title="Physical Drives seen from the SmartStorageConfig location"></p> <h2>Storage controller configuration process</h2> <p>As described in my previous blog, <a href="https://developer.hpe.com/blog/master-the-redfish-server-states-to-improve-your-monitoring-and-manageme/" target="_blank">Master the Redfish Server States to improve your monitoring and management applications</a>, and to be sure the configuration process happens smoothly, you should first verify that your managed systems are not at the <code>UEFI/System Utilities</code> level. Being in the <code>Off</code> state is a good state to start.</p> <p>Modification of Bios and storage controller configurations are performed in two stages as explained in the <a href="https://developer.hpe.com/blog/setting-bios-and-storage-controller-properties-with-redfish/" target="_blank">Setting Bios and Storage Controller Properties with Redfish</a> article. The first stage consists of setting the required parameters in the "pending settings area" of the Redfish data model (<code>/redfish/v1/Systems/{item}/SmartStorageConfig/Settings/</code>). During the next server reboot, parameters in the "pending settings area", after validation, are transferred to the "current configuration" at <code>/redfish/v1/Systems/{item}/SmartStorageConfig/</code>.</p> <p>Upon reboot, and once the server has finished its Pre-OS Tasks (POST), you should check carefully whether the modifications have been accepted as the current configuration.</p> <h2>The DataGuard property</h2> <p>The management of HPE Smart Storage devices requires a proper understanding of the <code>DataGuard</code> property part of the <code>SmartStorageConfig</code> sub-tree. The value of this attribute "<em>indicates whether or not data destructive actions are allowed</em>" as explained in the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo5/ilo5_290/ilo5_other_resourcedefns290/#dataguard">API Reference documentation</a>.</p> <p>This property is set in the pending settings (<code>SmartStorageConfig/Settings</code>) along with the directives to be performed by the Smart Storage device (i.e. Logical Volume Creation, Deletion...). During the next POST, the firmware checks its value and performs, or does not perform, the requested directives.</p> <p>If the value is <code>Strict</code>, which is the default value when not changed in the pending settings, the firmware denies any destructive data action (create/delete logical drives or clear drive metadata....).</p> <p>If the value is set to <code>Disabled</code>, destructive data actions are allowed. Finally, when the value is <code>Permissive</code>, only destructive data actions are allowed on the specified objects.</p> <p>Refer to the <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/supplementdocuments/storage/" target="_blank">iLO RESTful API documentation</a> for more information.</p> <h2>Storage controller and physical disks preparation</h2> <p>Imagine you just received a bunch of brand new servers or you want to re-deploy servers for a new project. In this context, you would want to remove completely the entire storage configuration as well as all meta data stored in the physical drives to get a clean Smart Storage subsystem.</p> <p>The HPE OEM storage subsystem models logical drives as a JSON collection stored in an array of objects represented like: <code>"LogicalDrives": [{ld-1},{ld-2,...{ld-n}]</code>. Each <code>ld-i</code> of this array contains the necessary properties to describe precisely the corresponding logical drive.</p> <p>The following screenshot shows the <code>LogicalDrives</code> array containing one element and its attributes. In it, you can see that this logical drive is a RAID 0 made of a single data drive (<code>1I:3:1</code>).</p> <p><img src="/img/3-logicaldrivecollection.png" alt="The LogicalDrives[] array" title="The LogicalDrives[] array"></p> <h3>Removing all logical drives</h3> <p>Removing all the logical drives of an HPE SR Smart Storage Controller is equivalent to the removal of all the elements of the <code>LogicalDrives</code> array. In practice, you need to request an empty array to the Redfish server.</p> <p>Since this action is data destructive, you must disable the <code>DataGuard</code> property to make sure the firmware allows this operation during the next reboot/POST of the system.</p> <p>The next screenshot contains all the necessary information to complete this remove operation: Perform an HTTP <code>PUT</code> to the pending settings (<code>SmartStorageConfig/Settings/</code>):</p> <p><img src="/img/3-deletealllogicaldrives.png" alt="Deleting all HPE Logical drives" title="Deleting all HPE Logical drives"></p> <h3>Clearing disk drives configuration metadata</h3> <p>HPE Smart Array RAID uses a reserved area at the end of each physical drive to store information about the logical drive configuration they are part of. When the <a href="https://www.hpe.com/us/en/product-catalog/detail/pip.5409020.html" target="_blank">Smart Storage Administrator</a> (SSA) application or Redfish is used to delete a logical drive, the metadata is cleaned up. However, if drives are moved around, there may be leftover metadata on the drives. The controller may show a failed logical drive or keep the drive from being presented to the OS. The <code>ClearConfigurationMetadata</code> action with the <code>DataGuard</code> property disabled using the <code>PATCH</code> method can be used to remedy this problem.</p> <p><img src="/img/4-clearconfigmetadata.png" alt="Clear disk drives configuration metadata"></p> <p>If you want to perform a single request removing all the logical drives and clearing the metadata, you have to perform a <code>PUT</code> of the pending settings with the following body:</p> <pre><code class="language-json">{ "Actions": [ { "Action": "ClearConfigurationMetadata" } ], "LogicalDrives": [], "DataGuard": "Disabled" } </code></pre> <h3>Removing a single/specific logical drive</h3> <p>To delete a specific logical drive you have to send a <code>PUT</code> request to the <code>SmartStorageConfig/settings</code> (pending settings). This request must contain the unique identifier of the desired logical drive as well as the <code>DataGuard</code> property set to <code>Permissive</code> to avoid the removal of other logical drives, if any:</p> <p><img src="/img/7-deletespecificlogicaldrive.png" alt="Specific logical drive deletion"></p> <h3>Sanitizing / disk drives</h3> <p>In addition to the removal of logical drives and the meta data cleanup of physical drives, you may want to erase / sanitize a list of physical drives. To perform this (long) operation, send the following <code>PATCH</code> action with the correct list of the drives to erase, separated by a comma. Don't forget to disabled the <code>DataGuard</code> property as well:</p> <p><img src="/img/5-sanitizephysicaldrive.png" alt="Sanitize physical disk drives"></p> <p>The <code>ErasePattern</code> property supports the following <a href="https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo5/ilo5_290/ilo5_other_resourcedefns290/#actions-array">values</a>:</p> <pre><code class="language-text"> SanitizeRestrictedBlockErase SanitizeUnrestrictedBlockErase SanitizeRestrictedOverwrite SanitizeUnrestrictedOverwrite SanitizeRestrictedCryptoScramble SanitizeUnrestrictedCryptoScramble OnePass TwoPass ThreePass </code></pre> <p>As mentioned above, the sanitize process is extremely long and you can retrieve the erase status in the disk drive properties under the <code>SmartStorage</code> sub-tree:</p> <p><img src="/img/6-erasestatus.png" alt="Sanitize / Erase status"></p> <h2>Logical drive creation</h2> <p>Logical drives can be created using the <code>PUT</code> method. In some circumstances a <code>PATCH</code> can be used. To keep things simple, I'll only use the <code>PUT</code> method.</p> <p>For this section, start with the clean infrastructure generated previously to create a 100GB RAID1 logical drive with a <code>Roaming</code> spare drive.</p> <p>Then, add a RAID0 Logical drive located on a single hard disk without spare drive and spanning the entire disk (300GB).</p> <p>Finally, add a second logical drive of 50 GB in the RAID1 array created earlier.</p> <p>While this scenario may not be completely realistic, the goal here is to more didactic than realistic.</p> <h3>First logical drive creation</h3> <p>In this clean context, you just need to send a <code>PUT</code> request to the <code>SmartStorageConfig/settings</code> (pending settings) with the properties of the desired logical drive as an element of the <code>LogicalDrives</code> array and the <code>DataGuard</code> value set to <code>Disabled</code>.</p> <p>The following picture shows the entire body of the <code>PUT</code> request with the mandatory properties in green boxes: <code>DataGuard</code>, <code>DataDrives</code> list, <code>Raid</code> type, <code>SpareDrives</code> list and <code>SpareRebuildMode</code>. Other properties will be defaulted if not supplied.</p> <p>A <code>200</code> return status with a <code>SystemResetRequired</code> "Error" message indicates that the remote SmartArray pending settings has been changed.</p> <p><img src="/img/8-logical-raid1spare-creation.png" alt="RAID1 with Spare drive creation"></p> <p>Upon reboot of the server, you should verify that the current configuration contains only the <code>Success</code> message. This ensures that the configuration happened effectively.</p> <p><img src="/img/9-runningzoneaftercreation.png" alt="Successful Logical Drive creation"></p> <h3>Creating more logical drives</h3> <p>Now, I would like to show you how to add a logical drive without altering the existing configuration. This operation could be noted "<code>=+</code>" in high level programming languages. I will use this analogy to build the body of the corresponding <code>PUT</code> request.</p> <p>The "<code>=</code>" portion of this process is performed by copying the entire <code>SmartStorageConfig</code> current configuration into the body of a <code>PUT</code> request. The remote iLO Redfish server is smart enough to keep only needed properties.</p> <p>The "<code>+</code>" part consists of adding the desired logical drive into the <code>LogicalDrives</code> array and changing the <code>DataGuard</code> value.</p> <p>The body of this <code>PUT</code> request is now ready to be sent to the <code>SmartStorageConfig/settings</code> (pending settings) of the managed iLO.</p> <p><img src="/img/11-addlogicaldrive.png" alt="Add a logical drive in a separate array"></p> <p>Again and again, you must check the return status, reboot if it is <code>Ok</code>, and verify that the current configuration contains only a single <code>Success</code> message to be sure that your modification was made.</p> <h3>Adding a logical drive in an existing disk array</h3> <p>This last exercise consists of adding a RAID0 logical drive of 50GB in the first storage array that was created earlier. This array has a capacity of 300GB and, only 100GB are consumed with logical drive <code>RAID1-1-100</code>.</p> <p>As I showed you in the previous section you have to perform a "<code>=+</code>" operation including a modification of the <code>DataGuard</code> property and the addition of a logical drive.</p> <p>The following screenshot shows the characteristics of the added RAID0 logical drive. It mentions its size as well as the <code>DataDrives</code> list and the spare drive used in the storage array created for the RAID1 logical drive in my first example. This ensures the creation of the desired logical drive in the right place.</p> <p><img src="/img/12-addlogicaldrivetoexistingarray.png" alt="Adding a logical drive in an existing array" title="Adding a logical drive in an existing array"></p> <p>After reboot, the requested RAID0 logical drive is visible in the current configuration. Note that the <code>SpareRebuildMode</code> has been automatically adjusted to <code>Dedicated</code> since <code>Roaming</code> is not a valid value anymore.</p> <p><img src="/img/13-runningzoneafteraddition.png" alt="Current configuration after the addition of logical drive in an existing array" title="Current configuration after the addition of logical drive in an existing array"></p> <h2>Conclusion</h2> <p>Redfish is a powerful, open source method of being able to manage systems. With a RESTful interface, it is designed to leverage existing internet standards and tool chains, making it usable by both amateurs as well as professionals.The ability of managing HPE Smart Storage with Redfish is a major step forward to the full control of HPE Gen10 and future servers using a single RESTful API.</p> <p>Make sure you continue to follow my blog posts on HPE DEV for more hints on working with Redfish in HPE environments. You can connect with me in <a href="https://hpedev.slack.com/" target="_blank">the Redfish channel on Slack</a> for specific questions.</p>https://developer.hpe.com/2018-October-29/https://developer.hpe.com/2018-October-29/Mon, 29 Oct 2018 05:00:00 GMThttps://developer.hpe.com/accessing-the-hpe-oneview-global-dashboard-api/https://developer.hpe.com/accessing-the-hpe-oneview-global-dashboard-api/Thu, 11 Oct 2018 15:14:06 GMT<p>By Brad Stanley</p> <p>Software Engineer working on Global Dashboard## Welcome With OneView Global Dashboard 1.6, we are happy to introduce its REST API! This means that you can query all of the appliances that have been added to Global Dashboard via command line, and even scripts if you are ambitious enough. There will be more on that later. In addition to querying the appliances, you can also add and delete appliances with our REST API. ### REST API?</p> <p>First, HPE OneView has some great blogs about their REST API, and some overhead information about how to use that.</p> <p>Check out <a href="/blog/first-step-with-programming-the-hpe-composable-api">this blog</a> for a quick introduction to REST APIs, which includes information about tools like POSTman. And <a href="/blog/curling-through-the-oneview-api">here</a> you could read a blog about cURL syntax.</p> <h3>Hackathon</h3> <p>Recently, we had a hackathon at work. This gave a few developers a chance to play with our REST API. It is possible that offers of free doughnuts and pizza also enticed the developers, but hopefully, it was primarily the opportunity to create something new.</p> <p>In the script we created, a few different things would happen.</p> <p>First, a OneView appliance was added to a Global Dashboard.</p> <p>Then, some summary data was obtained. This was done by using the endpoints for the various resources, e.g. enclosures, server profiles, etc. The most interesting data came from resource alerts. Global Dashboard collects all of the critical alerts from the appliances it monitors. When there are enough critical alerts, you can begin to see the same alerts from different HPE OneView appliances. Using the REST API and some simple scripting, the script would print out how many instances of each alert there are. This could be an extremely useful tool for anyone troubleshooting at a data center.</p> <p>Let’s say someone has the same exact alert on X different HPE HPE OneView appliances. The script could alert you to which HPE OneView appliances have that same alert, so that the solution could then be applied to quickly address those X alerts. That saves whoever is troubleshooting valuable time.</p> <p>Beyond that, you could build an interactive script that would let you type in notes. With that, when a fix is found, you could type in what that fix is and associate it with the critical alert. Then, the next time that alert rears its ugly head, your beloved script could give you a head start by informing you how you solved it last time. The script could grow over time and be a one-stop seek-and-destroy alerts shop.</p> <h3>Using REST API</h3> <p>There are two key steps in the above example: adding an appliance, and then querying the resource alerts endpoint.</p> <p>To add an appliance there are a few steps, and this is where it can be helpful to use a tool like POSTman.</p> <h4><strong>Add an appliance</strong></h4> <ul> <li><code>POST https://&#x3C;Global Dashboard IP>/rest/login-sessions</code> <ul> <li>Look at your Global Dashboard’s API docs for more information: <code>https://&#x3C;Global Dashboard IP>/apidoc/#tag/Login-Sessions</code></li> <li>For Global Dashboard versions 2.1 and earlier, this will need to be done with x-api-version having a value of 2</li> <li>And content-type will need to be specified with application/json</li> <li>If you look at your Global Dashboard’s API docs, you’ll see under Header Parameters that these two fields are required</li> <li>You’ll also notice a required Request Body, which is comprised of your userName, password and optionally an authLoginDomain <ul> <li>An example of this can be seen at the same API doc link that is above</li> </ul> </li> <li>Send the POST and you’ll get back a response that includes a token</li> </ul> </li> <li><code>GET https://&#x3C;Global Dashboard IP>/rest/certificates/https/remote/&#x3C;HPE OneView IP></code> <ul> <li>Look at your Global Dashboard’s API docs for more information: <code>https://&#x3C;Global Dashboard IP>/apidoc/#tag/Certificates</code></li> <li>Again, pay attention to the required parameters, one of them is auth which is the token you retrieved from the POST call</li> <li>This will return a large body of information, which will be needed for the next call* <code>POST https://&#x3C;Global Dashboard IP>/rest/certificates/servers</code></li> <li>Look at your Global Dashboard’s API docs for more information: <code>https://&#x3C;Global Dashboard IP>/apidoc/#tag/Certificates</code></li> <li>The request body has a lot of information mentioned in the API docs, but it is simple: copy the whole body that was returned from the previous call, and paste that into the body of this call</li> </ul> </li> <li><code>POST https://&#x3C;Global Dashboard IP>/rest/appliances</code> <ul> <li>Look at your Global Dashboard’s API docs for more information: <code>https://&#x3C;Global Dashboard IP>/apidoc/#tag/Appliances</code></li> <li>The POST to /certificates/servers added the HPE OneView certificate to Global Dashboard, which enables the HPE OneView to now be added</li> </ul> </li> </ul> <h4><strong>Query resource alerts</strong></h4> <ul> <li><code>GET https://&#x3C;Global Dashboard IP>/rest/resource-alerts</code> <ul> <li>Look at your Global Dashboard’s API docs for more information: <code>https://&#x3C;Global Dashboard IP>/apidoc/#tag/Resource-Alerts</code></li> <li>This will return at most one page of alerts, which is 25 <ul> <li>In order to get more alerts, append <code>?count=-1</code> to your query so it will look like: <code>https://&#x3C;Global Dashboard IP>/rest/resource-alerts?count=-1</code></li> <li>That will return up to 500 alerts, and if you have more than 500 alerts you would use a query like the following to get more <ul> <li><code>https://&#x3C;Global Dashboard IP>/rest/resource-alerts?count=500&#x26;start=500</code></li> </ul> </li> <li>This endpoint is unique because it also allows the user to get back data in a CSV format</li> </ul> </li> </ul> </li> </ul> <h2>Until Next Time</h2> <p>Throughout the API docs, which are accessible from your Global Dashboard, there are a number of different endpoints and information about required parameters and how to use them. Keep an eye on this blog for more about our REST API and other exciting features from OneView Global Dashboard.</p>https://developer.hpe.com/infrastructure-as-code-with-hpe-oneview-and-ansible-by-red-hat/https://developer.hpe.com/infrastructure-as-code-with-hpe-oneview-and-ansible-by-red-hat/Wed, 03 Oct 2018 03:40:59 GMT<h1>Infrastructure as code with HPE OneView and Ansible by Red Hat</h1> <p>One of the ways DevOps practitioners achieve agility is by provisioning the infrastructure at the same time as the application or service. They use automation to provision infrastructure using the same scripts to deploy exact copies of development, test, and production environments. These techniques work well in cloud and virtualized environments, but often hit a wall when applied to physical infrastructure. <a href="https://hpe.com/info/oneview">HPE OneView</a> and <a href="https://www.ansible.com/">Ansible by Red Hat</a> enable physical infrastructure as code, bringing cloud like agility to the private data center.</p> <p>The <a href="https://github.com/HewlettPackard/oneview-ansible-samples/blob/master/infrastructure-as-code/infrastructure-as-code.md">whitepaper</a> <em>Infrastructure as code with HPE OneView and Ansible by Red Hat</em> and accompanying <a href="https://github.com/HewlettPackard/oneview-ansible-samples/tree/master/infrastructure-as-code">GitHub repository</a> guides customers and partners in automating the provisioning of physical infrastructure. These playbooks can be checked into source control allowing you to treat infrastructure as code.</p> <p>Benefits of this infrastructure as code approach include, complete data center automation, consistent reproducibility, versioning, and rollback.</p>https://developer.hpe.com/running-hpe-oneview-ansible-modules-in-a-container/https://developer.hpe.com/running-hpe-oneview-ansible-modules-in-a-container/Tue, 02 Oct 2018 20:58:25 GMT<h1>Running HPE OneView Ansible modules in a container</h1> <p>The <a href="https://github.com/HewlettPackard/oneview-ansible">HPE OneView Ansible modules</a> are one of the most popular <a href="https://hpe.com/developers/oneview">HPE OneView integrations</a>. The Ansible modules automate the provisioning of physical infrastructure on-demand using software-defined templates from <a href="https://hpe.com/info/oneview">HPE OneView</a>. A containerized version of the Ansible modules is available at the <a href="https://store.docker.com/community/images/hewlettpackardenterprise/oneview-ansible-debian">Docker Store</a>.</p> <p>Docker containers provide a low friction way to get developer environments and CI/CD pipelines up and running easily and consistently. The <a href="https://github.com/HewlettPackard/oneview-ansible-samples/tree/master/oneview-ansible-in-container">oneview-ansible-in-container</a> sample has an Ansible playbook you can use to try out the <code>oneview-ansible-debian</code> container. It also has a <a href="https://github.com/HewlettPackard/oneview-ansible-samples/blob/master/oneview-ansible-in-container/oneview-ansible-in-container.md">how-to guide</a> that walks you through the steps needed to set up and run the playbook container. Once you have set up an environment to run Ansible playbooks using the HPE OneView modules in a container, you can switch to other directories and try other samples.</p>https://developer.hpe.com/using-the-ssmccli-hpe-3par-ansible-module-with-virtual-domains/https://developer.hpe.com/using-the-ssmccli-hpe-3par-ansible-module-with-virtual-domains/Mon, 01 Oct 2018 18:49:29 GMT<h1>Using the 3PAR SSMC/CLI &#x26; HPE 3PAR Ansible Module to Create and Manage Virtual Domains in a Multi-Tenant Scenario</h1> <p>This guide walks through the steps to create and manage Virtual Domains within a 3PAR using a mixture of the SSMC or CLI and the Ansible module for 3PAR. This guide targets those who want to setup and manage multi-tenancy on a 3PAR using configuration management tools like Ansible.</p> <p>You can also find this content over at:</p> <p><a href="https://github.com/hpe-storage/hpe3par-examples/tree/master/automation_tools/ansible/demo/virtual_domains">https://github.com/hpe-storage/hpe3par-examples</a></p> <h2>Assumptions<a name="assumptions"></a></h2> <ul> <li>HPE 3PAR configured and zoned correctly</li> <li>WSAPI enabled</li> <li>Super user access to 3PAR</li> <li>Ansible installed on workstation</li> <li>Knowledge of Ansible and playbooks</li> </ul> <h2>Configuring Virtual Domains in SSMC<a name="virtualdomain"></a></h2> <p>Domains allow an administrator to create up to 1,024 domains, or spaces, within a system, where each domain is dedicated to a specific application. A subset of the system users has assigned rights over the domains. Domains can be useful in scenarios where a single system is used to manage data from several different independent applications. For more information, refer to the <strong>HPE 3PAR Storeserv Storage Concepts Guide</strong>.</p> <p>The first steps to setting up your 3PAR for multi-tenancy is to create a new virtual domain. Currently the configuration of Domains/Users can only done within the SSMC or via the 3PAR CLI (using <strong>createdomain &#x3C;domain></strong>, <strong>createuser &#x3C;username> &#x3C;domainname> &#x3C;role></strong> ). The example shown below will be done within the SSMC.</p> <ol> <li>Login into the SSMC with Super user access.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/ssmc_login-1538420080349.jpg" alt="SSMC login"></p> <ol start="2"> <li>In the mega menu, click <strong>Show All > Domains</strong></li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/3par_domains-1538420435371.jpg" alt="3PAR domains"></p> <ol start="3"> <li>Click <strong>Create domain</strong></li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/3par_domains_create-1538420530820.jpg" alt="domains"></p> <ol start="4"> <li>Enter the name of the domain. In this example, <code>bob_domain</code>. Then click <strong>Add systems</strong>. Specify the <strong>3PAR</strong> where the domain will be created. Once complete, click <strong>Create</strong>.<a name="bob"></a></li> </ol> <blockquote> <p>You may ask why I am using Bob, because everyone knows Bob is cool!<a name="bob"></a></p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/3par_domains_create_bob_65-1538420610188.jpg" alt="create domain"></p> <ol start="5"> <li>In the mega menu, click <strong>Show All > Users</strong></li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/3par_users_menu-1538420664979.jpg" alt="users"></p> <ol start="6"> <li>Click <strong>Create User</strong></li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/create_user-1538420731817.jpg" alt="create user"></p> <ol start="7"> <li>Specify the <strong>NEW</strong> user name and password</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/create_user_bob_65-1538420769787.jpg" alt="create bob_user"></p> <ol start="8"> <li>Click <strong>Add Authorizations</strong>, choose the domain created previously (<code>bob_domain</code> on <strong>virt-3par</strong> system). Choose the <strong>edit</strong> Role for the user. Click Add.</li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/add_authorization-1538420815894.jpg" alt="authorization"></p> <h3>Repeat these steps as necessary to configure additional Domains and Users within your 3PAR.</h3> <p>If you want to use Ansible to configure the domains and users, Ansible can pass CLI commands to the 3PAR using the <code>shell</code> module.</p> <p>For example:</p> <pre><code class="language-YAML">--- - name: Create 3PAR Domain &#x26; Users hosts: localhost tasks: - name: install sshpass package: name: sshpass state: present become: yes - name: Create Domain shell: /usr/bin/sshpass -p 3pardata ssh -oStrictHostKeyChecking=no 3paradm@192.168.1.50 "createdomain bob_domain" register: domain - name: print domain debug: msg: "{{ domain }}" - name: Create Users shell: /usr/bin/sshpass -p 3pardata ssh -oStrictHostKeyChecking=no 3paradm@192.168.1.50 "createuser -c Password1 bob_user bob_domain edit" register: users - name: print users debug: msg: "{{ users }}" </code></pre> <hr> <h1>Using Ansible to configure CPGs, Hosts, Volumes and more<a name="ansible"></a></h1> <p>The following section demonstrates the process to configure resources like <strong>CPGs</strong>, <strong>3PAR hosts</strong>, etc. and assign them to the newly created domain(s). Remember that the domain users are only be able to view/edit resources they have access to and are be unable to view/edit resources from other domains unless authorized to do so.</p> <p>Also everything else from this point is done via the HPE 3PAR Ansible Storage Module on a Linux (RHEL, CentOS, Ubuntu) system with Ansible (ver. 2.5 or later) installed.</p> <h2>Storage Admin Perspective<a name="admin"></a></h2> <hr> <h3>Let's get started.</h3> <p>First, we need to make sure we have the 3PAR storage module for Ansible downloaded.</p> <ol> <li>Clone the HPE 3PAR Ansible Storage Modules</li> </ol> <pre><code>https://github.com/HewlettPackard/hpe3par_ansible_module </code></pre> <ol start="2"> <li>Install the HPE 3PAR Python SDK</li> </ol> <pre><code>pip install hpe3par_sdk </code></pre> <p>Now, clone the <code>hpe3par-examples</code> repo to get access to the Virtual Domain demo.</p> <pre><code>https://github.com/hpe-storage/hpe3par-examples </code></pre> <h3>Generic Ansible housekeeping<a name="housekeeping"></a></h3> <ol> <li>Configure <code>ansible.cfg</code> to know about the 3PAR Ansible Storage Modules.</li> </ol> <blockquote> <p>If you have other modules already installed on this system, you can move the <strong>Modules</strong> folder from this repo to that directory.</p> </blockquote> <pre><code>vi /etc/ansible/ansible.cfg </code></pre> <ol start="2"> <li>Under the <code>[defaults]</code> section, edit <code>library</code> to point to your Modules directory.</li> </ol> <pre><code>[defaults] # some basic default values... inventory = /etc/ansible/hosts library = /root/workspace/hpe3par_ansible_module/Modules </code></pre> <hr> <h3>Understanding the 3PAR Ansible playbooks<a name="understanding"></a></h3> <p>Navigate to the <code>hpe3par_examples/automation_tools/ansible/demo/virtual_domains</code> folder. Here we find two Ansible playbooks and the <code>properties/</code> folder.</p> <ul> <li><strong>virtual_domains_demo_3par_admin.yml</strong></li> <li><strong>virtual_domains_demo_3par_user.yml</strong></li> <li><strong>properties/storage_system_properties.yml</strong> (This is configuration files containing the 3PAR IP address, Storage admin username and password for the 3PAR array)</li> <li><strong>properties/storage_system_properties_bob.yml</strong> (This is configuration files containing the 3PAR IP address, Domain user username and password for the 3PAR array)</li> </ul> <pre><code>cd ~/workspace/hpe3par_examples/automation_tools/ansible/ [root@ansible-host virtual_domains]# ls -la total 12 drwxr-xr-x. 4 root root 137 Sep 26 13:00 . drwxr-xr-x. 3 root root 29 Sep 26 09:23 .. drwxr-xr-x. 2 root root 254 Sep 26 13:00 img drwxr-xr-x. 2 root root 116 Sep 26 09:23 properties -rw-r--r--. 1 root root 2219 Sep 26 13:00 README.md -rw-r--r--. 1 root root 2065 Sep 26 09:23 virtual_domains_demo_3par_admin.yml -rw-r--r--. 1 root root 2065 Sep 26 09:23 virtual_domains_demo_3par_create-domain-users_cli.yml -rw-r--r--. 1 root root 1847 Sep 26 09:23 virtual_domains_demo_3par_user.yml </code></pre> <hr> <h3>Configuring Storage System Property files</h3> <ol> <li>Lets configure the <code>properties/storage_system_properties.yml</code> and add the 3PAR IP address. Enter the <strong>Storage Admin</strong> username and password. Save the file.</li> </ol> <pre><code>vi **properties/storage_system_properties.yml** </code></pre> <pre><code>storage_system_ip: "192.168.1.50" storage_system_username: "3paradm" storage_system_password: "3pardata" </code></pre> <ol start="2"> <li>Edit the <code>properties/storage_system_properties_bob.yml</code> and configure the 3PAR IP address. Enter the <strong>bob_user</strong> username and password. Save the file.</li> </ol> <pre><code>vi **properties/storage_system_properties_bob.yml** </code></pre> <pre><code>storage_system_ip: "192.168.1.50" storage_system_username: "bob_user" storage_system_password: "Password" </code></pre> <hr> <h3>Configuring Ansible Vault - encrypt/decrypt</h3> <p>Ansible Vault is a feature of ansible that allows you to keep sensitive data such as passwords or keys in encrypted files, rather than as plaintext in playbooks or roles.</p> <ol> <li>Run <code>ansible-vault encrypt</code> on each of the properties files. Enter a unique password for each file.</li> </ol> <pre><code>ansible-vault encrypt properties/storage_system_properties.yml ansible-vault encrypt properties/storage_system_properties_bob.yml </code></pre> <ol start="2"> <li>Check to verify they are encrypted. You should see something similar to below.</li> </ol> <pre><code>$ head -2 properties/storage_system_properties.yml $ANSIBLE_VAULT;1.1;AES256 33636137356335 </code></pre> <blockquote> <p><strong>Note:</strong> If you need to edit the encrypted file, you can run <code>ansible-vault edit file.yml</code>, enter the vault password and perform the edits. Alternatively, if you need to decrypt the file, run <code>ansible-vault decrypt file.yml</code> and enter the vault password.</p> </blockquote> <hr> <h3>Now let's get on to the fun stuff</h3> <p>We will be working in the <code>virtual_domains_demo_3par_admin.yml</code> playbook. This playbook is ran by the Storage Admin to create <strong>CPGs</strong> and assign <strong>Hosts</strong> to the domain we created previously.</p> <blockquote> <p><strong>Note:</strong> These are very simple examples to help you understand the capabilities of the Virtual Domains within the HPE 3PAR system. You can expand these to add multiple CPGs and multiple Hosts within the same playbook without ever having to log into the SSMC. This is the power automating the configuration of the HPE 3PAR Storage System using the Ansible Storage Modules.</p> </blockquote> <p>When we open the file, we see multiple sections. Again we are assuming that you are familiar with <strong>YAML</strong> and Ansible playbooks to understand the layout and structure.</p> <pre><code class="language-yaml">--- - name: Virtual Domains on 3PAR Ansible Demo playbook - Admin hosts: localhost become: no vars: cpg_name: 'bob_cpg_FC_r6' host_name: 'scom.virtware.co' domain: 'bob_domain' iscsi_names: 'iqn.1991-05.com.microsoft:scom.virtware.co' tasks: - name: Load Storage System Vars include_vars: 'properties/storage_system_properties.yml' - name: Create CPG "{{ cpg_name }}" hpe3par_cpg: # 3PAR CPG options found here: https://github.com/HewlettPackard/hpe3par_ansible_module#modules storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: present domain: "{{ domain }}" cpg_name: "{{ cpg_name }}" growth_increment: 32.5 growth_increment_unit: GiB growth_limit: 100 growth_limit_unit: GiB growth_warning: 90 growth_warning_unit: GiB raid_type: R6 set_size: 6 high_availability: MAG disk_type: FC - name: Create Host "{{ host_name }}" hpe3par_host: # 3PAR Host options found here: https://github.com/HewlettPackard/hpe3par_ansible_module#modules storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: present host_name: "{{ host_name }}" host_domain: "{{ domain }}" host_persona: WINDOWS_SERVER - name: Add iSCSI paths to Host "{{ host_name }}" hpe3par_host: storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: add_iscsi_path_to_host host_name: "{{ host_name }}" host_iscsi_names: "{{ iscsi_names }}" </code></pre> <hr> <h3>Configuring Variables section</h3> <p>There are several sections where you can specify variables allowing maximum flexibility when creating playbooks. They can be specified at the <strong>playbook</strong> level (Global), in <strong>external file</strong> (properties/storage_system_properties.yml), or at the <strong>task</strong> level.</p> <p>In the <code>vars</code> section, you can modify the CPG/Host names to be added to the <code>bob_domain</code>.</p> <p><strong>Note:</strong> In order to assign the new CPGs and Hosts to the domain, you must specify a domain in the <code>domain: 'bob_domain'</code> variable. This variable is then used within each of the tasks (<strong>domain:</strong> and <strong>host_domain:</strong>), where required, to assign the CPG or Host to the domain. If the domain is <strong>not</strong> specified, the CPG or Host is <strong>not</strong> assigned to a domain and is not accessible to the Domain user when they log into the array.</p> <blockquote> <p>Modify the <strong>host_name</strong> and <strong>iscsi_names</strong> to match a host and iSCSI iqns you want to add from your environment.</p> </blockquote> <p>In the <code>tasks</code> section, for example in the <strong>Create CPG</strong> task, you can add/modify the variables (growth_limit, raid_type, etc) in the tasks as well move them into <strong>vars</strong> section if needed.</p> <blockquote> <p>In the case of Multi-Tenancy, creating growth limits, defining disk characteristics on CPGs in critical in order to enforce boundaries per tenant (this prevents one tenant from consuming the entire storage array), as well as to ensure all tenants get the appropriate resources and performance per their needs. In the playbook above, we specified a <strong>100GB growth limit (with a 90GB warning)</strong> on the CPG, therefore restricting the users within the <code>bob_domain</code> from using more than 100GB of storage space on the array. All of this is configurable by the Storage Admin.</p> </blockquote> <p>Also you can specify an external variables file like the <code>storage_system_properties.yml</code>. This gives us the ability to encrypt the external file without affecting the main playbook.</p> <hr> <h3>Configuring Tasks sections<a name="admin_tasks"></a></h3> <p>We have 4 main tasks in this example.</p> <p>These tasks are taken from the main (CPG, Host, Volume, etc) playbooks found in the 3PAR Storage Module here: <a href="https://github.com/HewlettPackard/hpe3par_ansible_module/tree/master/playbooks">https://github.com/HewlettPackard/hpe3par_ansible_module/tree/master/playbooks</a>.</p> <p>Please refer to the <a href="https://github.com/HewlettPackard/hpe3par_ansible_module/blob/master/Modules/readme.md">Modules README</a> for detailed information on each Module including required/optional parameters.</p> <ol> <li><strong>Load Storage System Vars</strong> (load the encrypted storage system IP, username/password)</li> <li><strong>Create CPG</strong> (create CPGs per the provided specifications)</li> <li><strong>Create Host</strong> (create a basic 3PAR host)</li> <li><strong>Add iSCSI paths to Host</strong> (modify the host and add iSCSI IQNs or FC WWNs)</li> </ol> <hr> <h3>Running the Playbook<a name="admin_run"></a></h3> <p>Now that we know what is going on within the admin playbook, we can run it in order to create the CPGs, Host resources within the <strong>bob_domain</strong>.</p> <p>We run this playbook with the <code>ansible-playbook --ask-vault-pass</code> option in order to decrypt the <code>storage_system_properties.yml</code> file.</p> <pre><code class="language-yaml">$ ansible-playbook --ask-vault-pass virtual_domains_demo_3par_admin.yml Vault password: PLAY [Virtual Domains on 3PAR Ansible Demo playbook - Admin] ************************** TASK [Gathering Facts] **************************************************************** ok: [localhost] TASK [Load Storage System Vars] ******************************************************* ok: [localhost] TASK [Create CPG "bob_cpg_FC_r6"] ***************************************************** ok: [localhost] TASK [Create Host "scom.virtware.co"] ************************************************* ok: [localhost] TASK [Add iSCSI paths to Host "scom.virtware.co"] *************************************************************************************** ok: [localhost] PLAY RECAP **************************************************************************** localhost : ok=5 changed=0 unreachable=0 failed=0 </code></pre> <hr> <h3>Success.</h3> <p>This sample playbook demonstrates how a Storage Admin can quickly and programmatically configure Domains, Users, and storage resources using a combination of the SSMC and Ansible playbooks.</p> <h2>Storage User Perspective<a name="user"></a></h2> <hr> <p>Now that we have finished configuring a Domain, Users, created CPGs and added hosts into the Domain, lets cover how a user can consume the 3PAR using Ansible playbooks while still being bound to the limits placed on the domain by the Storage Admin.</p> <p>This section follows closely to my other blog post about using the Ansible modules to provision storage on an HPE 3PAR. <a href="/blog/storage-provisioning-using-ansible-with-hpe-3par-storage">Storage Provisioning using Ansible with HPE 3PAR Storage</a></p> <h3>Assumptions<a name="user_assumptions"></a></h3> <ul> <li>Domains/users have been created</li> <li>CPGs and Hosts have been assigned to the domain</li> <li>Reviewed <a href="#housekeeping">Generic Ansible housekeeping</a> section</li> <li>Vault password to unlock the <strong>properties/storage_system_properties_bob.yml</strong> file. Check out the <a href="#vault">Vault section</a> for more info.</li> </ul> <h3>Let's get started</h3> <p>Let's take a look at our Ansible playbooks again. Since we have everything ready for us on the array, it is very simple to run the playbooks as a user. The example playbook is a simple demonstration on how you can turn your <strong>Infrastructure into Code</strong>.</p> <ul> <li>Navigate to the <strong>hpe3par_examples/automation_tools/ansible/demo/virtual_domains</strong> folder.</li> </ul> <pre><code>cd hpe3par_examples/automation_tools/ansible/demo/virtual_domains </code></pre> <p>Here we find the <code>virtual_domains_demo_3par_user.yml</code> playbook and the <code>properties/storage_system_properties_bob.yml</code> (This file contains the 3PAR IP address, 3PAR Domain user username/password).</p> <pre><code>[root@ansible-host virtual_domains]# ls -la total 12 drwxr-xr-x. 4 root root 137 Sep 26 13:00 . drwxr-xr-x. 3 root root 29 Sep 26 09:23 .. drwxr-xr-x. 2 root root 254 Sep 26 13:00 img drwxr-xr-x. 2 root root 116 Sep 26 09:23 properties -rw-r--r--. 1 root root 2219 Sep 26 13:00 README.md -rw-r--r--. 1 root root 2065 Sep 26 09:23 virtual_domains_demo_3par_admin.yml -rw-r--r--. 1 root root 2065 Sep 26 09:23 virtual_domains_demo_3par_create-domain-users_cli.yml -rw-r--r--. 1 root root 1847 Sep 26 09:23 virtual_domains_demo_3par_user.yml </code></pre> <p>Before we go much further, make sure you have configured the <code>properties/storage_system_properties_bob.yml</code> with the appropriate 3PAR array information. Please review <a href="#property">Storage System Property Files</a> for more info.</p> <p>Let's look at the 'virtual_domains_demo_3par_user.yml' playbook.</p> <pre><code>vi virtual_domains_demo_3par_user.yaml </code></pre> <p>When we open the file, we see multiple sections. Again we are assuming that you are familiar with <strong>YAML</strong> and Ansible playbooks to understand the layout and structure.</p> <pre><code class="language-yaml">--- - name: Virtual Domains on 3PAR Ansible Demo playbook - Bob User hosts: localhost vars: volume_name: 'bob_demo_volume' cpg_name: 'bob_cpg_FC_r6' host_name: 'scom.virtware.co' domain: 'bob_domain' vol_size: 10 vol_size_unit: 'GiB' autolun: False lunid: 110 tasks: - name: Load Storage System Vars include_vars: 'properties/storage_system_properties_bob.yml' - name: Load VolumeSet Vars include_vars: 'properties/volumeset_properties.yml' - name: Create Volume "{{ volume_name }}" hpe3par_volume: storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: present volume_name: "{{ item }}" cpg: "{{ cpg_name }}" size: "{{ vol_size }}" size_unit: "{{ vol_size_unit }}" with_items: "{{ ['volume_bob_1', 'volume_bob_2', 'volume_bob_3'] }}" - name: Create volume set "{{ volumeset_name }}" hpe3par_volumeset: storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: present volumeset_name: "{{ volumeset_name }}" setmembers: "{{ add_vol_setmembers }}" - name: Create VLUN hpe3par_vlun: storage_system_ip: "{{ storage_system_ip }}" storage_system_username: "{{ storage_system_username }}" storage_system_password: "{{ storage_system_password }}" state: export_volumeset_to_host volume_set_name: "{{ volumeset_name }}" host_name: "{{ host_name }}" lunid: "{{ lunid }}" autolun: "{{ autolun }}" </code></pre> <p>The sections within the user playbook are very similar to the one used by the admin user. From this playbook, you are able to provision volumes from the CPGs and resources specified by the Storage Admin (<a href="#limits">100GB limit</a>) and exporting the volumes to hosts. The main difference here is that you are authenticating to the 3PAR with a 3PAR Domain user (<strong>bob_user</strong>) rather than a 3PAR Super user.</p> <p>The 3PAR Domain user is specified in the <code>properties/storage_system_properties_bob.yml</code> file. This is covered in the <a href="#property">Storage System Property Files</a> section.</p> <hr> <h3>Configuring Variables section - User</h3> <p>A quick review on variables. There are several sections where you can specify variables allowing maximum flexibility when creating playbooks. They can be specified at the playbook level (Global), in external file (properties/storage_system_properties.yml), or at the task level.</p> <p>In the <code>vars</code> section, you can modify the <strong>Volume</strong>, <strong>CPG</strong>, <strong>3PAR Host</strong>, <strong>Volume Size</strong>, etc. to meet your needs.</p> <blockquote> <p>Be aware, if you exceed the 100GB limit on the CPG (as defined by the Storage Admin in our demo), by either creating a volume larger than 100GB or multiple volumes that exceed a cumulative 100GB in size, the playbook fails with an error that the allocation is larger than the limit.</p> </blockquote> <p><strong>Note:</strong> Since we are logged into the <code>bob_domain</code> as the <code>bob_user</code>, we don't have to explicitly define the domain when creating Volumes, Volume Sets, etc. because the domain is inherited from the User's domain.</p> <p>In the <code>tasks</code> section, for example in the <strong>Create Volume</strong> task, you can use the <code>with_items</code> option (with_items functions as a loop in Ansible) to create multiple volumes during runtime rather creating multiple tasks to create individual volumes.</p> <hr> <h3>Configuring Tasks sections - User<a name="user_tasks"></a></h3> <p>We have 4 main tasks in this example.</p> <p>These tasks are taken from the main (CPG, Host, Volume, etc) playbooks found in the 3PAR Storage Module here: <a href="https://github.com/HewlettPackard/hpe3par_ansible_module/tree/master/playbooks">https://github.com/HewlettPackard/hpe3par_ansible_module/tree/master/playbooks</a>.</p> <p>Please refer to the <a href="https://github.com/HewlettPackard/hpe3par_ansible_module/blob/master/Modules/readme.md">Modules README</a> for detailed information on each Module including optional parameters.</p> <ol> <li><strong>Load Storage System Vars</strong> (load the encrypted storage system IP, username/password)</li> <li><strong>Load VolumeSet Variables</strong></li> <li><strong>Create Volume</strong> (create 3 Volumes as defined)</li> <li><strong>Create VolumeSet</strong> (create a VolumeSet for the 3 Volumes)</li> <li><strong>Create VLUN</strong> (Export the VolumeSet to the 3PAR Host (i.e. scom.virtware.co))</li> </ol> <hr> <h3>Running the Playbook - User<a name="user_run"></a></h3> <p>Now that we have configured our playbook, we can run it in order to create the <strong>Volumes</strong>, <strong>VolumeSet</strong> all within the <code>bob_domain</code>.</p> <p>We run this playbook with the <code>ansible-playbook --ask-vault-pass</code> option in order to decrypt the <strong>storage_system_properties_bob.yml</strong> file.</p> <pre><code class="language-yaml">$ ansible-playbook --ask-vault-pass virtual_domains_demo_3par_user.yml Vault password: PLAY [Virtual Domains on 3PAR Ansible Demo playbook - Bob User] ************************* TASK [Gathering Facts] ****************************************************************** ok: [localhost] TASK [Load Storage System Vars] ********************************************************* ok: [localhost] TASK [Load VolumeSet Vars] ************************************************************** ok: [localhost] TASK [Create Volume "bob_demo_volume"] ************************************************** ok: [localhost] => (item=volume_bob_1) ok: [localhost] => (item=volume_bob_2) ok: [localhost] => (item=volume_bob_3) TASK [Create volume set "bob_volumeset"]************************************************* ok: [localhost] TASK [Create VLUN] ********************************************************************** ok: [localhost] PLAY RECAP ****************************************************************************** localhost : ok=6 changed=0 unreachable=0 failed=0 </code></pre> <hr> <h3>Success.<a name="user_success"></a></h3> <p>This sample playbook demonstrates how a Storage User can quickly and programmatically provision storage and manage those storage resources within their Domain while still being under the control of the Storage Admin.</p> <p>Now let's verify that these volumes have been exported to the Host.</p> <ol> <li> <p>Log into the <strong>SSMC</strong> as <code>bob_user</code></p> </li> <li> <p>Click on <strong>Virtual Volumes</strong>. You should see 3 volumes and also see that they are successfully exported to <strong>scom.virtware.co</strong></p> </li> </ol> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/ssmc_success-1538420946527.jpg" alt="Success SSMC"></p> <p>Now let's check <strong>scom.virtware.co</strong></p> <blockquote> <p>You may need to perform a <strong>Rescan Disks</strong> to see the new volumes.</p> </blockquote> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/scom_success-1538421007139.jpg" alt="Success SCOM"></p> <p>You should be able to see all of the volumes available to the Windows Server.</p> <p>This concludes this demo. With this information, you will be able to create Multiple Domains, Users, and manage the resources that are available to each. You also will understand how to apply the appropriate controls around CPGs and your domains in order to facilitate multi-tenancy storage domain.</p>https://developer.hpe.com/introducing-nemo-hpe-nimble-storage-advanced-data-services-emulator-for-/https://developer.hpe.com/introducing-nemo-hpe-nimble-storage-advanced-data-services-emulator-for-/Mon, 01 Oct 2018 17:33:50 GMT<p>The <a href="https://store.docker.com/plugins/nimble">HPE Nimble Storage Docker Volume plugin</a> brings a plethora of powerful capabilities to container ecosystems like Docker, Mesos and Kubernetes. The plugin can be used with either <a href="https://hpe.com/storage/nimble">Nimble arrays</a> or in conjunction with a <a href="https://hpe.com/storage/cloudvolumes">HPE Cloud Volumes</a> account. Both platforms are very suitable for high performing mission-critical production workloads, which is to be expected. However, given the advanced functionality the Docker Volume plugin provides, wouldn't one be more comfortable experimenting with the features in a completely isolated sandbox? On your laptop?</p> <p>Nemo is the <a href="https://github.com/NimbleStorage/Nemo">HPE Nimble Storage Advanced Data Services Emulator for Containers</a>. It's a free-standing Open Source project blessed by HPE. It emulates some of the Advanced Data Services that the real plugin provides and it only depends on OpenZFS being available on the host OS. OpenZFS provides similar data management capabilities such as snapshot, clones and metadata storage per filesystem. Nemo is not providing block storage; it is only a local filesystem and does not come with a support contract.</p> <p>HPE Nimble Storage does not use OpenZFS in any way in the shipping products.</p> <h1>The Soda Challenge</h1> <p>Nemo is designed to emulate the HPE Nimble Storage Docker Volume plugin in so, at a glance, they look very similar. But, if you were actaully comparing the two one to one – it's apples and oranges in terms of functionality. The official plugins are targeted towards mission-critical Enterprise applications and workloads. Nemo is for the dabbler and tinkerer who is unable to access a Nimble array or HPE Cloud Volumes to learn how Advanced Data Services are incorporated into DevOps centric environments.</p> <p>The options available to Nemo are divided in a number of self-explanatory sections to help layout the level of difference between the two:</p> <ul> <li>Universal Options: Help and size, very rudimentary</li> <li>Nimble Compatible Global Options: Basic parameters</li> <li>Nimble Compatible Clone Options: Clone parameters</li> <li>Nimble Compatible Import Dataset as Clone Options: Import existing OpenZFS datasets</li> <li>Nimble Compatible Vanity Options: These options does not provide any real functionality</li> </ul> <p>These sections are outlined below and available in the <code>-o help</code> output from the plugin:</p> <pre><code>gem:~ mmattsson$ sudo docker volume create -d nemo -o help Error response from daemon: create c6e7ad6cd8199feac4c485a5e79ad4edef8e231e9fa179e0191a1e5fadfe88fc: VolumeDriver.Create: -o help Nemo: HPE Nimble Storage Advanced Data Services Emulator for Containers Create, Clone or Import an existing OpenZFS dataset into a locally scoped Docker Volume. All options are optional. Every '-o key=value' will be stored on the OpenZFS Dataset. ******************************************************************************** D I S C L A I M E R HPE Nimble Storage is not using OpenZFS in any of its products. Nemo is a tool to educate users how to integrate Advanced Data Services into DevOps workflows using common developer and IT operations tools without owning or using a HPE Nimble Storage product. Nemo is not supported by HPE Nimble Storage. ******************************************************************************** Universal Options: -o help This help -o size=X X is the quota of volume specified in GiB, 0 means no quota Nimble Compatible Global Options: -o description=X X is a vanity description set on the volume -o sizeInGiB=X X is the alternative to 'size' -o destroyOnRm=X X is either 'true' or 'false' to actually destroy a dataset after it has been removed. Can be "imported" with 'importVol'. Global default runtime flag available -o fsMode=X X X is 1 to 4 octal digits that represent the file mode to be applied to the root directory of the filesystem -o fsOwner=X:Y X:Y is the uid:gid that should own the root directory of the filesystem, in the form of uid:gid (string or nums) Nimble Compatible Clone Options: -o cloneOf=X X is the name of Docker Volume to create a clone of -o snapshot=X X is the name of the snapshot to base the clone on (optional, if missing, a new snapshot is created) -o createSnapshot=X 'true' or 'false', indicates that a new snapshot of the volume should be taken and used for the clone -o destroyOnDetach=X indicates that the dataseut (including snapshots) backing this volume should be destroyed when this volume is unmounted or detached Nimble Compatible Import Dataset as Clone Options: -o importVolAsClone=X X is an exisiting dataset without ZDVP properties to import into a ZDVP dataset as a clone -o snapshot=X X is an optional dataset snapshot to import clone from -o createSnapshot=X X is either 'true' or 'false' to create a above snapshot -o destroyOnDetach=X X indicates that the dataset (including snapshots) backing this volume should be destroyed when this volume is unmounted or detached Nimble Compatible Import Options: -o importVol=X X is an exisiting unmounted dataset without ZDVP properties to import into a ZDVP dataset -o snapshot=X X is an optional dataset snapshot to import from -o restore=X X is dataset snapshot name to roll back to on import -o forceImport=X Vanity flag, accepts any value, all imports are forced Nimble Compatible Vanity Options (will be displayed in the "Status" field): -o limitIOPS=X Defaults to '-1' -o limitMBPS=X Defaults to '-1' -o dedupe=X Defaults to 'false' -o thick=X Defaults to 'false' -o encryption=X Defaults to 'none' -o folder=X Defaults to 'none' -o pool=X Defaults to 'default' -o perfPolicy=X Defaults to 'DockerDefault' -o protectionTemplate=X Defaults to 'none' Additional OpenZFS Options: -o help=OpenZFS </code></pre> <p>The <code>docker volume inspect</code> output somewhat represents a fairly compatible output if the JSON output from a production system is being programmatically parsed. This would include the list of snapshots on a particular volume (snapshot naming conventions and timestamps differ slightly).</p> <h1>Get started!</h1> <p>Nemo is primarily being distributed in source form but will be made available for broader developer-friendly container ecosystems. There's a managed Docker Volume plugin available that installs pretty much out-of-the-box on an Ubuntu 18.04 machine. Instructions to get OpenZFS and Nemo rolling on RHEL and CentOS is available for the tinkerer <a href="https://github.com/NimbleStorage/Nemo/tree/master/runtime/docker-v2#other-distributions">in the repo</a>.</p> <p>This is what a "get to know Nemo session" could look like on an Ubuntu server provisioned with <code>docker-machine</code>:</p> <pre><code>gem:~ mmattsson$ docker-machine create --driver generic --generic-ssh-port=22 --generic-ssh-user=vagrant --generic-ip-address=192.168.59.131 --generic-ssh-key=.vagrant/machines/default/vmware_fusion/private_key nemo Running pre-create checks... Creating machine... (nemo) Importing SSH key... Waiting for machine to be running, this may take a few minutes... Detecting operating system of created instance... Waiting for SSH to be available... Detecting the provisioner... Provisioning with ubuntu(systemd)... Installing Docker... Copying certs to the local machine directory... Copying certs to the remote machine... Setting Docker configuration on the remote daemon... Checking connection to Docker... Docker is up and running! To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env nemo </code></pre> <pre><code>gem:~ mmattsson$ eval $(docker-machine env nemo) </code></pre> <pre><code>gem:~ mmattsson$ docker version Client: Version: 18.06.0-ce API version: 1.38 Go version: go1.10.3 Git commit: 0ffa825 Built: Wed Jul 18 19:05:26 2018 OS/Arch: darwin/amd64 Experimental: false Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:15 2018 OS/Arch: linux/amd64 Experimental: false </code></pre> <pre><code>gem:~ mmattsson$ docker plugin install --alias nemo --grant-all-permissions nimblestorage/nemo:1.0.0 1.0.0: Pulling from nimblestorage/nemo 6ff473112dfe: Download complete Digest: sha256:8f256853b8d2a97aa233938f93ac69686be4f5a280c7372e2cd672898519828c Status: Downloaded newer image for nimblestorage/nemo:1.0.0 Installed plugin nimblestorage/nemo:1.0.0 </code></pre> <pre><code>gem:~ mmattsson$ docker volume create -d nemo myvol1 myvol1 </code></pre> <pre><code>gem:~ mmattsson$ docker volume inspect myvol1 [ { "CreatedAt": "2018-09-26T16:13:10Z", "Driver": "nemo:latest", "Labels": {}, "Mountpoint": "", "Name": "myvol1", "Options": {}, "Scope": "local", "Status": { "ApplicationCategory": "Virtual Server", "CachePinned": "false", "CachingEnabled": "true", "Connections": "0", "DedupeEnabled": "false", "Description": "Docker knows this dataset as myvol1", "EncryptionCipher": "none", "Folder": "", "Group": "nemo", "LimitIOPS": "-1", "LimitMBPS": "-1", "LimitSnapPercentOfSize": "-1", "LimitVolPercentOfSize": "100", "PerfPolicy": "DockerDefault", "Pool": "default", "SnapUsageMiB": "0", "Snapshots": [], "ThinlyProvisioned": "true", "VolSizeMiB": 10240, "VolUsageMiB": 0, "destroyOnDetach": "false", "destroyOnRm": "false", "mountConflictDelay": "30" } } ] </code></pre> <pre><code>gem:~ mmattsson$ docker run --rm -it -v myvol1:/data bash Unable to find image 'bash:latest' locally latest: Pulling from library/bash 4fe2ade4980c: Pull complete ec6d9ca5c66a: Pull complete d8685fbd86ca: Pull complete Digest: sha256:8634afcddefc8a10565b22d685df782058b096712a91bf45d75633f368dda729 Status: Downloaded newer image for bash:latest bash-4.4# df -h /data Filesystem Size Used Available Use% Mounted on tank/v2/myvol1 10.0G 128.0K 10.0G 0% /data bash-4.4# touch /data/myfile.txt bash-4.4# exit </code></pre> <pre><code>gem:~ mmattsson$ docker volume create -d nemo -o cloneOf=myvol1 myvol1-clone myvol1-clone </code></pre> <pre><code>gem:~ mmattsson$ docker run --rm -it -v myvol1-clone:/data bash bash-4.4# ls /data myfile.txt </code></pre> <p>In this simple workflow:</p> <ul> <li>A docker-machine was setup and the Nemo managed Docker Volume plugin was installed</li> <li>A default 10GiB volume got created</li> <li>A file was created on the volume</li> <li>A clone of the volume was created and attached to a new container</li> </ul> <p>The exact same workflow, along with several others, is available with the HPE Nimble Storage Docker Volume plugin.</p> <p><strong>Note:</strong> By default Nemo creates a pool on a loopback device mapped to a 64GiB file in <code>/var/lib/nemo</code> on the host.</p> <h1>Plenty of fish in the sea</h1> <p>One might wonder if Nemo is compatible with <a href="https://github.com/hpe-storage/dory">Dory</a>? That is a given! Nemo fully implements the Docker Volume API, <code>dory</code> and <code>doryd</code> pretty much work stock with the exception that certain defaults needs to be overridden. More details on the Kubernetes integration is available <a href="https://github.com/NimbleStorage/Nemo/tree/master/runtime/k8s">in the repo</a>.</p> <p>Example setup procedures for Kubernetes:</p> <pre><code>gem:~ mmattsson$ kubectl create -f https://raw.githubusercontent.com/NimbleStorage/Nemo/master/runtime/k8s/daemonset-nemod.yaml daemonset.apps/nemod created </code></pre> <pre><code>gem:~ mmattsson$ kubectl create -f https://raw.githubusercontent.com/NimbleStorage/Nemo/master/runtime/k8s/deploy-doryd.yaml clusterrole.rbac.authorization.k8s.io/doryd created clusterrolebinding.rbac.authorization.k8s.io/doryd created serviceaccount/doryd created deployment.extensions/kube-storage-controller-doryd created </code></pre> <p>The cluster is now ready to create Storage Classes, Persistent Volume Claims or use the FlexVolume driver inline. A <code>StorageClass</code> and <code>StatefulSet</code> example is provided:</p> <pre><code>gem:~ mmattsson$ kubectl create -f https://raw.githubusercontent.com/NimbleStorage/Nemo/master/runtime/k8s/sc-transactionaldb.yaml storageclass.storage.k8s.io/transactionaldb created </code></pre> <pre><code>gem:~ mmattsson$ kubectl create -f https://raw.githubusercontent.com/NimbleStorage/Nemo/master/runtime/k8s/statefulset-mariadb.yaml secret/mariadb created statefulset.apps/mariadb created service/mariadb created </code></pre> <p>The <code>PersistentVolumeClaim</code> created by the <code>volumeClaimTemplate</code> part of the <code>StatefulSet</code> can be inspect with <code>kubectl</code>:</p> <pre><code>gem:~ mmattsson$ kubectl get -o yaml pvc/mariadb-mariadb-0 apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" creationTimestamp: 2018-09-28T00:03:06Z finalizers: - kubernetes.io/pvc-protection labels: app: mariadb name: mariadb-mariadb-0 namespace: default resourceVersion: "111181" selfLink: /api/v1/namespaces/default/persistentvolumeclaims/mariadb-mariadb-0 uid: e0572150-c2b1-11e8-a34c-000c290a512a spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: transactionaldb volumeName: transactionaldb-e0572150-c2b1-11e8-a34c-000c290a512a status: accessModes: - ReadWriteOnce capacity: storage: 1Gi phase: Bound </code></pre> <p>Due to the nature of Nemo attaching to local storage only, Pods and StatefulSets are the most common patterns for deploying stateful applications with. In single node clusters, like the one I used to create these examples, Replication Controllers and Deployments would also work as expected.</p> <h1>Next steps</h1> <p>We are simply testing the waters with Nemo (pun intended) to figure out if this could be helpful for developers who wants to learn about Advanced Data Services. Further, understand how to integrate those services into CI/CD pipelines, kick the tires on lifting &#x26; shifting an application or validate more traditional workflows where you would provide production data to developers for Copy Data Management in containerized environments.</p> <p>Please feel free to give it a whirl and don't be afraid to reach out on <a href="https://www.labs.hpe.com/slack">Slack</a> or file any issues you may find on <a href="https://github.com/NimbleStorage/Nemo">GitHub</a>.</p>https://developer.hpe.com/2018-September-29/https://developer.hpe.com/2018-September-29/Sat, 29 Sep 2018 05:00:00 GMThttps://developer.hpe.com/sir-hackington-appbuilder-iii-tames-the-it-monster/https://developer.hpe.com/sir-hackington-appbuilder-iii-tames-the-it-monster/Mon, 24 Sep 2018 15:44:54 GMT<p>My name’s Sir Hackington Appbuilder III for those of you just meeting me, and I’m the trusted guardian of all the swag when members of the HPE Developer Community (HPE DEV) head out to conferences. I’d like to tell you a little about our recent time at VMworld Las Vegas that was hosted at the end of August.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/vmworld_collage-1538516807872.jpg" alt="vmworld_collage"></p> <h2>Tame the IT Monster</h2> <p>HPE’s theme at VMworld this year was Tame the IT Monster. What does that mean? Simply put, it means reducing IT complexity as businesses continue on their digital transformation journey. With <a href="https://www.wsj.com/articles/SB10001424053111903480904576512250915629460">software eating the world</a> and a substantial shift from a hardware-based to a software-based tech economy, companies must embrace a DevOps mindset to increase their opportunity for new business growth and to serve their customers better. HPE DEV promotes taming the IT monster with a two-pronged approach: by offering multiple integrations for HPE products, and by building deep connections to developers across the industry to collaborate and pool collective knowledge.</p> <p>HPE DEV is always excited to talk to people at conferences – especially VMworld. That’s because the HPE DEV portal hosts many integration activities with VMware that include applications and products like <a href="https://developer.hpe.com/platform/hpe-oneview/home">HPE OneView</a>, <a href="https://developer.hpe.com/platform/hpe-simplivity/home">HPE SimpliVity</a>, and <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a>. Spreading the word through our subject matter experts and having these guys on hand to talk through the integration projects was a great way to let customers and partners understand where they can become more involved in HPE DEV’s community activities.</p> <p>Not everyone who stopped by the booth knew about these VMware integrations, but the HPE DEV team made sure that everyone they met with came away fully informed about the specific integrations and the HPE DEV portal more broadly. Even folks from the operations side of their businesses were excited to get signed up for the HPE DEV newsletter so they can forward it to their teams’ developers and engineers.</p> <p>In addition to getting folks to sign up for a newsletter, HPE DEV team members worked to spread the word about the community by participating in collaborative activities at VMworld. Pramod Sareddy, HPE DEV’s representative at VMworld, was invited to attend two VMworld VBlogger sessions, where VMware experts, developers and industry bloggers met up to compare notes and share VMware knowledge with each other. This was a great opportunity for Pramod to promote what HPE DEV is doing and get direct feedback from this group. One of the items he promoted was the contributions made to the community by <a href="/blog/the-advent-of-ephemeral-infrastructure-as-code">HudsonAlpha</a>, which got the attendees excited to contribute in a similar way.</p> <h2>Spreading the word</h2> <p>One of the biggest challenges HPE DEV currently faces is spreading the word about the community and the value it’s providing. For instance, did you know that HPE DEV has an <a href="https://github.com/HewlettPackard/simplivity-vra-plugin">HPE SimpliVity plugin for VMware vRealize Automation (vRA)</a>, which allows users to back up, restore, clone, and move their virtual machines on-demand via the vRA interface? Or that HPE DEV offers <a href="https://www.hpe.com/us/en/product-catalog/detail/pip.4152978.html">HPE OneView for VMware vCenter</a>, which seamlessly integrates manageability features with VMware virtualization solutions?</p> <p>I’ll brag a little here—it’s my presence that often draws people over to our booth. From there, HPE DEV conversations start around specific integrations, and then they typically turn into even larger conversations about HPE DEV and all the platforms they support. HPE DEV is committed to developers, DevOps, and helping businesses make their digital transformation successful.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/img_5414_resized_1-1538501085089.jpg" alt="img_5414_resized_1"></p> <h2>Want to see more of HPE DEV?</h2> <p>Missed HPE DEV at VMworld? There are still plenty of chances to see the HPE DEV team this year at other developer-friendly conferences all over the world. For more information about where you can find them, visit the <a href="https://developer.hpe.com/events">Events</a> page. HPE DEV is going to have a busy autumn, with appearances at <a href="https://www.vmworld.com/en/europe/index.html">VMworld Europe</a>, <a href="https://www.hpe.com/events/discover/">HPE Discover</a>, and <a href="https://reinvent.awsevents.com/">AWS re:Invent</a>! And don’t forget to follow the <a href="https://developer.hpe.com/community">Community</a> page to see ways to get involved and contribute. See you soon!</p>https://developer.hpe.com/hpe-at-microsoft-ignite-event-september-24-27/https://developer.hpe.com/hpe-at-microsoft-ignite-event-september-24-27/Wed, 12 Sep 2018 19:16:44 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/9/hpe-at-ms-ignite-1536779956115.png" alt="hpe at ms ignite"></p> <p>Hewlett Packard Enterprise (HPE) will be at the <a href="https://www.microsoft.com/en-us/ignite">Microsoft Ignite</a> event, September 24th through 27th in Orlando, Florida. The event, which welcomes thousands of IT professionals each year, is a chance for HPE to showcase their latest innovative technologies. At the event, HPE will host several sessions and demos for attendees to get a chance to see technologies such as <a href="https://www.hpe.com/us/en/solutions/cloud/azure-hybrid-cloud.html">HPE ProLiant for Microsoft Azure Stack</a>, <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a>, <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a>, <a href="https://www.hpe.com/us/en/integrated-systems/simplivity.html/">HPE SimpliVity</a> and the <a href="https://developer.hpe.com/">HPE Developer Community</a> up-close. Below, find a list of the sessions HPE will host at the event.</p> <p>On Monday, September 24 at 5:45 PM, don’t miss <a href="https://myignite.techcommunity.microsoft.com/sessions/66918?source=sessions">HPE Edgeline and Microsoft Azure powering the Intelligent Edge (THR1134)</a>. Come hear how HPE and Microsoft are helping customers with the digital transformation of their business with HPE Edgeline Converged Edge Systems and Microsoft Azure innovation that spans the edge to the cloud</p> <p>On September 26, 2018 starting at 2:15 PM, join HPE for <a href="https://myignite.techcommunity.microsoft.com/sessions/66917?source=sessions">How to tame your hybrid cloud (BRK1123)</a>. This breakout session will offer a chance to learn how to make Hybrid IT simple so that data and applications are always optimized and agile, without compromising performance, security or control. HPE experts will discuss how, with your right mix of Hybrid IT in place, powered by software-defined infrastructure, your enterprise will be ready to accelerate innovation and thrive in our digital world.</p> <p>Join HPE OneView exerts for <strong>Take control of your virtual and physical server management with HPE OneView for Microsoft System Center</strong>. Come learn how to avoid costly mistakes and added complexity by streamlining operations with <a href="(https://www.hpe.com/us/en/integrated-systems/software.html)">HPE OneView</a> for Microsoft System Center. This integration provides comprehensive system health and alerting, driver and firmware updates, HPE fabric visualization, and provisioning of HPE servers, networking and storage. Create consistency for software deployment and updates and enable a faster response in the event of server or storage failure, reducing your risk for downtime. <em>Monday, September 24th at 4:30 PM and Wednesday, September 26th at 11:00 AM</em></p> <p>If you are in the market for an Azure Stack solution, don’t miss <strong>Top 5 Reasons HPE Delivers the Best Microsoft Azure Stack Solution</strong>. Join this session to understand the 5 key areas that sets HPE apart from other Azure Stack offerings. Don’t miss this chance to hear how HPE is the only vendor who delivers <a href="https://www.hpe.com/us/en/solutions/cloud/azure-hybrid-cloud.html">the most configurable Azure Stack solution</a> available, combined with the best service and support on the market. <em>Monday, September 24th at 2:30 PM and Wednesday, September 26th at 1:00 PM</em></p> <p><strong>Need a real-time view of utilization and costs across your hybrid cloud? Meet HPE OneSphere</strong>. Whether it's multiple data centers or multiple clouds, getting a comprehensive view of resources, workloads and costs can be a challenge. Join this session to see how <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a> hybrid cloud management enables you to quickly and easily get high-level or fine-grained visibility into costs and optimize digital services across your hybrid estate in real-time, instead of at the end of a billing cycle. <em>Monday, September 24th at 4:00 PM and Wednesday, September 26th at 3:00 PM</em></p> <p><a href="https://www.hpe.com/us/en/integrated-systems/hyper-converged.html">Hyperconvergence</a> is a hot topic! At Ignite, come learn <strong>3 reasons why HPE SimpliVity hyperconverged infrastructure simplifies IT environments to deliver dramatic business results</strong>. This session will look at how <a href="https://www.hpe.com/us/en/integrated-systems/simplivity.html">HPE SimpliVity</a> hyperconverged infrastructure <a href="http://www.intel.com/">powered by Intel®</a> with Hyper-V is designed and optimized for virtualized workloads and how it can streamline IT costs and operations, increase agility and time to production, prevent data loss and maximize uptime. Learn how you can radically transform your IT environment by uniting HPE SimpliVity advanced data services with Microsoft Hyper-V hypervisor and see a live demo in action. <em>Monday, September 24th at 5:00 PM and Tuesday, September 25th at 4:30 PM</em></p> <p>Lastly, don’t miss <strong>Ease your hybrid cloud management with HPE OneView for Microsoft Azure Log Analytics</strong> while at the event. This session will focus on how you can manage your on-premises HPE infrastructure with the same Microsoft tools used for cloud services. Experts will discuss how HPE OneView for Microsoft Azure Log Analytics brings visibility of the underlying HPE infrastructure, including hardware and firmware inventory, infrastructure health and status, and long term event correlation and trend analysis. <em>Monday, September 24th at 6:30 PM and Wednesday, September 26th at 4:30 PM</em></p> <p>Stop by booth 2125 at the <a href="https://www.microsoft.com/en-us/ignite">Microsoft Ignite</a> event to meet HPE experts, join these sessions and more. More details about HPE at Microsoft Ignite can be <a href="https://community.hpe.com/t5/Alliances/Attending-Microsoft-Ignite-September-24-28-See-HPE-there/ba-p/7016641#.W5KFU-hKiM8">found here</a>.</p>https://developer.hpe.com/using-the-azure-stack-cost-monitor/https://developer.hpe.com/using-the-azure-stack-cost-monitor/Fri, 07 Sep 2018 21:43:43 GMT<h1>Overview</h1> <p>The Azure Stack cost monitor scripts are used to pull resource consumption data from an Azure Stack instance. This data can then be imported, manipulated and displayed in a variety of ways. This article describes how to pull this data and provides an example of what it looks like when displayed in a PowerBI dashboard.</p> <h1>Prerequisites</h1> <ul> <li>PowerShell for Azure Stack</li> <li>Operator level access to Azure Stack</li> <li>Connectivity to Azure Stack ARM endpoints</li> <li>Familiarity with PowerBI</li> </ul> <h1>Concepts used in tool</h1> <p>The tool obtains resource consumption data directly from the Azure Stack ARM endpoint using the resource usage API. The tool workflow is as follows:</p> <ul> <li>Confirm resource rates for Azure Stack meters</li> <li>Run the tool against the Azure Stack ARM endpoint with operator credentials</li> <li>Use the JSON output to display the resource usage data in PowerBI# Procedure</li> </ul> <h2>Install PowerShell for Azure Stack</h2> <p>If you have not already done so, install PowerShell for Azure Stack using [these instructions] (<a href="https://docs.microsoft.com/azure/azure-stack/azure-stack-powershell-install">https://docs.microsoft.com/azure/azure-stack/azure-stack-powershell-install</a>). Be sure it's installed on the computer you will use to access the Azure Stack endpoints.</p> <h2>Verify connectivity to Azure Stack ARM endpoint</h2> <ul> <li>Launch PowerShell</li> <li>Copy the lines below into your PowerShell window. Replace the name and endpoint details for your Azure Stack environment</li> </ul> <pre><code class="language-powershell"># To get this value for Azure Stack integrated systems, contact your administrator who deployed Azure Stack. #Example: For an Azure Stack that has been deployed with regionname=seattle; externalFQDN=stackcloud.com use value as, https://adminmanagement.seattle.stackcloud.com $ArmEndpoint = "&#x3C;Admin Resource Manager endpoint for your environment>" #Use Azure Stack operator credentials to log in Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint $ArmEndpoint # After signing in to your environment, Azure Stack cmdlets can be easily targeted at your Azure Stack instance. Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" Get-AzureRmResourceGroup </code></pre> <p>These commands should list the resource groups available on the Azure Stack instance. If this succeeds, then you have verified connectivity to the Azure Stack endpoint. If you encounter any errors, check with your Azure Stack administrator to fix the connectivity.</p> <h2>Export resource usage data</h2> <ul> <li> <p>Download [usage monitor tool] (<a href="https://github.com/HewlettPackard/hpe-azurestack/blob/master/Usage-Monitor/Get-AzureStackUsageDataWithCost.ps1">https://github.com/HewlettPackard/hpe-azurestack/blob/master/Usage-Monitor/Get-AzureStackUsageDataWithCost.ps1</a>) to a local folder</p> </li> <li> <p>Supply the parameter values you want to use with the tool. The complete list of parameters is in the tool. Among the parameters that should be specified are:</p> <ul> <li>Granularity : Daily or Hourly</li> <li>Time range : From, To datetimes</li> <li>Aggregation Level : Provider or Tenant</li> </ul> </li> </ul> <pre><code class="language-powershell"># # EXAMPLE1: To get AzureStack Admin usage report with hourly granularity # .\Get-AzsUsageInfo.ps1 -StartTime 3/01/2018 -EndTime 3/21/201 -AzureStackDomain azurestack.local -AzureStackRegion "local" -AzureStackCloudName "Local MAS Cloud" -AADDomain mydir.onmicrosoft.com -Granularity Hourly # *** The generated output file will be &#x3C;AzureStackRegion>-&#x3C;AzureStackDomain>-Hourly-UsageSummary.json # # EXAMPLE2: To get AzureStack tenant usage report with daily granularity .\Get-AzsUsageInfo.ps1 -StartTime 3/01/2018 -EndTime 3/21/2018 -AzureStackDomain azurestack.local -AzureStackRegion "local" -AzureStackCloudName "Local MAS Cloud" -AADDomain mydir.onmicrosoft.com -Granularity Daily -TenantUsage # The generated output file will be &#x3C;AzureStackRegion>-&#x3C;AzureStackDomain>-Daily-TenantUsageSummary.json </code></pre> <h2>Generate a PowerBI dashboard</h2> <p>Use the data from the generated JSON files with powerBI and create dashboard reports with your desired visualizations.# Examples Below are example PowerBI dashboards using data extracted from HPE Azure Stack lab environments.</p> <p>The first dashboard shows daily usage over a 2 month period:</p> <p><img src="/uploads/media/2018/9/dailyusagesummary-1536947449691.png" alt="" title="DailyUsageReport"></p> <p>The second dashboard shows hourly usage over a 3 day period:</p> <p><img src="/uploads/media/2018/9/hourlyusagesummary-1536947466387.png" alt="" title="HourlyUsageReport"># Summary Controlling cloud resource consumption and cost is a large consideration for cloud customers. This tool enables you to see which resources are driving your costs and as well as which ones may be silently wasting capacity. This allows Azure Stack operators to better monitor their consumption throughout the month.</p> <p>To further extend this tool, the entire process could be automated by periodically exporting usage data and refreshing the dashboard.</p> <h1>Next Steps</h1> <p>If you'd like to learn more about development on the HPE ProLiant for Microsoft Azure Stack as well as other HPE platforms, join the HPE Developer community below.</p> <p>If you have any new project ideas for Azure Stack, contact the team at <a href="mailto:asic@hpe.com">asic@hpe.com</a>.</p>https://developer.hpe.com/sir-hackington-appbuilder-iii-goes-to-google-cloud-next/https://developer.hpe.com/sir-hackington-appbuilder-iii-goes-to-google-cloud-next/Tue, 28 Aug 2018 14:50:51 GMT<p>Hey, it’s me, your old friend, Sir Hackington Appbuilder III. For those of you who haven’t met me yet, that’s right, I’m the dog here. More importantly, I’m the event mascot of the HPE Developer Community (HPE DEV). Some people like to say that dogs are man’s best friend. In my case, I’m the best friend a developer can have, especially at conferences.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_20180726_140212-1535468013423.jpg" alt="Sir Hackington"></p> <p>HPE DEV recently brought me to Google Cloud Next in San Francisco from July 24-26. Even though I’m a dog I’m really into all the things HPE DEV is into. So of course, in 2018, that means lots of thinking about cloud. Cloud platforms and infrastructure are very important to enterprise IT these days, and HPE is a key player in this space. HPE DEV was really excited to spread the good word about all the new innovations they provide to businesses looking to get the most from their cloud strategies. For instance HPE DEV showed off a number of the API references currently on the HPE DEV portal that allow developers to dive in and start working on 3rd party applications to integrate with HPE OneSphere and build powerful solutions for their customers. For those who don’t know, HPE OneSphere is HPE’s as-a-service hybrid cloud management solution.</p> <h2>HPE DEV Presentations</h2> <p>HPE DEV worked hand-in-hand with the HPE OneSphere team to run multiple sessions at the Google Next event booth, all geared toward the developer community.</p> <p>At the event, HPE DEV gave demos on how HPE is helping developers be more productive in the API economy. Getting APIs to work well with cloud platforms is a huge part of developers’ tasks these days, and the developers who stopped by the booth came away impressed with HPE’s API work, because it allows them to get up and running on 3rd party applications right away.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_20180724_162809-1535468245061.jpg" alt="img_20180724_162809"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_20180725_113330-1535468321884.jpg" alt="img_20180725_113330"></p> <p>The team also provided sessions on multi-cloud management using HPE OneSphere, and showed a customer use-case on managing containers effectively using HPE Composable Infrastructure with HPE Synergy and Google Cloud Platform. As most of the developers at the conference were interested in leveraging Google Cloud Platform in their work, showcasing the ways that these developers can get the most from both HPE and Google was a big hit.</p> <h2>Key takeaways</h2> <p>I may be biased because I hang out with developers all the time, and they take good care of me, but developers really are the best. Google Cloud Next this year had a large number of developers visiting, more than I sometimes see at other conferences I attend. The large number of developers makes it easy for HPE DEV to start up conversations at our booth. I’d like to think it’s all because of my winning personality, but I have to give the HPE DEV team the credit—developers are eager to talk to their peers, and they’re also very interested to hear about what HPE is doing in and with the developer community.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_20180725_161340-1535468427944.jpg" alt="img_20180725_161340"></p> <p>Many initially think of HPE as just an infrastructure company, but when HPE DEV comes to conferences with a large developer presence, that perception changes. But don’t just take my word for it. Some of our partners from HudsonAlpha, a nonprofit institute that translates the power of genomics into real world results, stopped by the HPE DEV booth to talk about the benefits of working with HPE.</p> <p>[<a href="https://www.pscp.tv/w/1yNGaXRNAQbKj">https://www.pscp.tv/w/1yNGaXRNAQbKj</a>]</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_20180725_113415-1535468605630.jpg" alt="img_20180725_113415"></p> <p>As the world moves from an IT Ops model to a DevOps model, HPE is dedicated to being a leader and innovator, and developers are receptive to what the team is accomplishing. So much so that around 100 developers signed up to join our HPE DEV newsletter mailing list! Woof woof! (That means sign up for our newsletter today in mascot dog lingo.)</p> <p>In other words, HPE DEV is making inroads with the developer community. Which is pretty exciting for a company that sometimes isn’t thought of as having a significant developer profile. Sometimes building word of mouth on an individual by individual basis is a great way to go, and the HPE DEV team really felt that at Google Cloud Next.</p> <h2>Want to see more of HPE DEV?</h2> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/google-cloud-next-event-presence-1535468699526.JPG" alt="google cloud next event presence"></p> <p>Missed HPE DEV Google Cloud Next? No worries -- they will be traveling all over the world in the next few months to conferences of interest to developers. For more information about where you can find the HPE DEV team, visit the <a href="https://developer.hpe.com/events">Events</a> page. And follow the <a href="https://developer.hpe.com/community">Community</a> page to see highlights of previous appearances and ways to get involved and contribute. Next up, come check in with us at <a href="https://www.vmworld.com/en/us/index.html">VMworld</a>, Las Vegas booth 1300 to learn more! I’ll be there!</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/hackignton-1535468992281.jpg" alt="hackignton"></p>https://developer.hpe.com/managing-ilo-sessions-with-redfish/https://developer.hpe.com/managing-ilo-sessions-with-redfish/Mon, 27 Aug 2018 13:44:46 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/managingilosessions/managingilosessionswithredfish">Server Management Portal</a>.</big></p> <br>https://developer.hpe.com/announcing-the-introduction-of-hpe-oneview-41/https://developer.hpe.com/announcing-the-introduction-of-hpe-oneview-41/Tue, 21 Aug 2018 22:50:40 GMT<h1>We are pleased to announce the latest version of HPE OneView</h1> <p><a href="https://www.hpe.com/info/oneview">HPE OneView</a> 4.1 has been released. Built with software-defined intelligence and a unified API, HPE OneView enables you to deploy infrastructure faster, simplify lifecycle operations, increase productivity, and accelerate time to value.</p> <p>Over one million licenses have shipped so far, demonstrating the value customers are seeing in deploying HPE OneView.</p> <p>For full details on top new features for HPE OneView 4.1, please read the <a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/HPE-simplifies-infrastructure-management-with-announcement-of/ba-p/7004750#.W3yUt-hKj4a">announcement blog</a>.</p>https://developer.hpe.com/dont-miss-hpe-dev-at-vmworld-2018-august-26-30/https://developer.hpe.com/dont-miss-hpe-dev-at-vmworld-2018-august-26-30/Fri, 17 Aug 2018 19:31:21 GMT<p>Hewlett Packard Enterprise (HPE) will be attending <a href="https://www.vmworld.com/en/us/index.html">VMWorld</a> in Las Vegas, NV, August 26--30, 2018. VMworld 2018 is VMware’s premier digital infrastructure event, where businesses can find the tools and partners they need to accelerate their software-defined journey and launch the digital transformation.</p> <p><img src="https://hpe-developer.8ar.ms/uploads/media/2018/8/hpe-dev-at-vmworld-1534534980822.png" alt="HPE DEV at VMWorld"> At the event, the HPE Developer Community (HPE DEV) will join other teams from across HPE as they strive to <em>Tame the IT Monster</em>. HPE and VMware offer the broadest portfolio of integrated and certified solutions that help companies deliver proven outcomes for their business demands. During VMworld the HPE DEV team will be focused on will highlighting the HPE VMware partnership and how it is helping accelerate DevOps initiatives with a rich set of API’s and automated workflows.</p> <p>In addition, HPE DEV will be holding a daily raffle of Amazon Echo Dots for everyone who signs up for the developer newsletter. Sir Hackington Appbuilder III, the HPE DEV booth mascot, will make an appearance as he does at every HPE DEV event, and the team will be handing out stickers along with plenty of expert knowledge.</p> <p>Stop by HPE’s booth (#1300) to meet experts from across HPE, not just HPE DEV, and learn about the latest technology from HPE, including <a href="https://www.hpe.com/us/en/integrated-systems/software.html">HPE OneView</a>, <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a>, and more. To learn more about <a href="https://www.vmworld.com/en/us/index.html">VMWorld 2018</a>, click here. See you at the event!</p>https://developer.hpe.com/2018-August-10/https://developer.hpe.com/2018-August-10/Fri, 10 Aug 2018 05:00:00 GMThttps://developer.hpe.com/sir-hackington-appbuilder-iii-tells-you-what-happened-at-oscon/https://developer.hpe.com/sir-hackington-appbuilder-iii-tells-you-what-happened-at-oscon/Fri, 10 Aug 2018 01:00:29 GMT<p>Sir Hackington Appbuilder III here. Yeah, take a look at the picture. I’m the dog with the lanyards. I’m pretty cute, huh? That’s what people tell me. Sure, they also call me a mascot, but I’d like to think I’m the most important member of the HPE Developer Community (HPE DEV).</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/photo-2-1533863198104.JPG" alt="photo 2"></p> <p>HPE DEV recently attended the 20th annual Open Source Convention, or <a href="https://community.hpe.com/t5/Shifting-to-Software-Defined/Don-t-miss-HPE-at-OSCON-2018-July-16-19/ba-p/7011089#.W2s6gShKiM8">OSCON</a> for short. The convention, put on by O’Reilly, was held from July 16-19 in Portland, Oregon. The HPE team enjoyed some cool weather in the Pacific Northwest and the opportunity to meet a bunch of cool developers spreading the word about HPE’s open source efforts and projects. And they got to show me off as well.</p> <p>OSCON is geared exclusively toward developers. Everybody was in jeans and t-shirts. You know, the kinds of people that look like they’d be eager to take me on a walk. Sadly, that didn’t happen because these are also the kinds of people HPE DEV loves to talk to. Which they did. A lot. But, all in all I was pretty pleased with how many in the developer community were interested in talking with my team and wanted to sign up for the monthly newsletter to stay in touch.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/1-1533863099426.png" alt="1"></p> <p>At the event, HPE DEV members Alex Mejias and Jim “Schreck” Schreckengast delivered a presentation on open source for the enterprise, which was extremely well received. They stressed the importance of community development and engagement being critical for adoption, otherwise developers are dropping code over a wall and hoping for the best. And of course, having an open source strategy that harmonizes with the business is vital.</p> <p>You can check out that presentation <a href="https://cdn.oreillystatic.com/en/assets/1/event/274/Open-sourcing%20enterprise%20software%20_sponsored%20by%20HPE_%20Presentation.pptx">here</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/2-1533863118364.png" alt="2"></p> <h2>Key takeaways</h2> <p>One of the things attendees learned at the event was that while many developers do not necessarily associate HPE with open source, they were eager to hear about HPE’s presence in the open source community. Grommet, an open source tool set developed by HPE, especially piqued people’s interest. Grommet allows enterprise companies to easily design essential and modern web experiences. Alex Mejias, one of Grommet’s core team members and the community’s leading Grommet evangelist, was a hit with his ability to speak to and demonstrate the ease of using the tools.</p> <p>Another thing I learned--the Dev community loves stickers (about as much as I love bones!) All the presenters loaded up attendees with stickers. See that purple guy? That’s Stack, our other mascot (but he’s not real like me!) Stack was our most popular sticker by far, so expect to see more of him. Everybody loves a cute mascot, right? Especially when you feature him on another cute mascot. Ahem, I’m talking about me, Sir Hackington Appbuilder III.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/3-1533863149410.png" alt="3"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/img_0066-1533863211157.JPG" alt="img_0066"></p> <p>HPE DEV is making inroads with the developer community, which they continued at OSCON. While many people think of HPE as just an infrastructure company, the presence of HPE DEV at conferences dedicated to the developer community is starting to change that perception. As the world moves from an IT Ops model to a DevOps model, HPE is dedicated to being a leader and innovator.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/8/4-1533863163033.png" alt="4"></p> <p>Missed HPE DEV at OSCON? No worries. They will be making appearances at dev-friendly conferences around the United States and Europe over the next few months. For more information about where you can find them, visit the <a href="https://developer.hpe.com/events">Events</a> page. And follow the <a href="https://developer.hpe.com/community">Community</a> page to see highlights of previous appearances and ways to get involved and contribute. And come check us next at <a href="https://www.vmworld.com/en/us/index.html">VMworld</a> to learn more! I’ll be there!</p>https://developer.hpe.com/devops-its-an-it-thing-isnt-it/https://developer.hpe.com/devops-its-an-it-thing-isnt-it/Thu, 09 Aug 2018 16:13:21 GMT<p>“<a href="http://www.azquotes.com/quote/350077">We can never judge the lives of others, because each person knows only their own pain and renunciation. It's one thing to feel that you are on the right path, but it's another to think that yours is the only path.</a>” <a href="http://www.azquotes.com/author/3041-Paulo_Coelho">Paulo Coelho</a></p> <p>Over the last 4 years as a CIO Advisor for HPE, I have been advising organisations in the adoption of collaborative working across the business not just IT. Co-creation across the business and IT boundaries can create effective lean &#x26; agile practices that can take full advantage of DevOps processes. These organisations have spanned multiple geographies, technologies, cultural background and skill sets. Through these experiences I have learnt a lot – mostly that these transformations are very hard, normally take much longer than anyone wants, and that it is super important to learn from others on your journey to your transformation, be ready to face the oasis and deserts – in both cases and don’t stop.</p> <h2>DevOps Transformation Journey Choices</h2> <p>Here is an understatement – <strong>Change is difficult.</strong></p> <p>In all my long experience as a CIO and CIO Advisor (30 yrs. +) I have never seen any organisation relishing the thought of having to change any process, people, activity, project, etc etc. It is normally done with great care and attention. Hence, there are various facets when effecting change, even when agile practices may already be mainstream in an organisation.</p> <p>Consequently, moving beyond agile to unleash the full potential of the business by the use of DevOps practices requires the next level of collaboration, co-creation and automation. But, with all life-journeys, there will be blockers on the route – whether that be cultural, process or simply people. It is important then to adopt the right decisions-tools to address the pain points and convert the pain into relief as it becomes the easiest way to get it done anyway. The choice of people (champions of change), culture (away from not invented here), processes (from rigid to supple) and tools (adopting agile/collaborative tools) will dictate your journey choices and difficulty in getting to your destination.</p> <p>As Paolo Coelho states – “In a forest of a hundred thousand trees, no two leaves are alike. And no two journeys along the same path are alike”</p> <p><img src="https://hpe-developer.8ar.ms/uploads/media/2018/8/hpe20180226034_800_0_72_rgb-1533835764985.jpg" alt=""></p> <h3>My 3-Step Plan for an Effective DevOps Transformation Journey</h3> <p>This is my 3-step approach to DevOps Transformation that I use with organisations who are on this journey. It is NOT a sequential step approach, it is an agile step approach – that is to say sometimes sequential, other time concurrent but mainly aligned to the culture and capacity of the organisation.</p> <p>However, because cultural change, politics and behaviour underpin every aspect of any transformational change programme it is essential to read each step within this context and ask a critical question each time in terms of “is the organisation ready for this?” For instance, if your initiative is about accelerating the delivery of new capabilities within your sales and marketing department – but it needs the support and training of 25,000 front-line staff – can you afford this, how long will it take, what’s your ROI, etc etc…</p> <p>You need to understand where the boundaries are politically and economically and figure out how to break those down.</p> <h3>Step 1 – Expand agile practices beyond IT</h3> <p>Too often DevOps is seen as the mandate solely of IT and there is an assumption that the business and in particular its C-level executives don’t take any interest in the software development function nor understand words like “agile” and “waterfall”, never mind “DevOps” – they just want it to get done.</p> <p>It is essential to explain that there has been shift to the way software is being developed with the advent of agile and DevOps that resembles the lean manufacturing shift in the 80s and 90s. Furthermore, it is a well understood fact that “software is eating the world” and most businesses need apps (both mobile and web apps) to sell its products. Any time you have an app, software development is seen as a critical part of the day-to-day business strategy. Hence, C-level executives are definitely taking a deeper interest in how these things can be delivered in an effective way</p> <p>Still, without true business involvement, customer feedback and effective operations to break down silos and improve outcomes, the expected efficiencies and benefits will never come. All stakeholders in an organisation must be involved with full support from its leadership and management across all disciplines.</p> <h3>Step 2 – Shift Left within a Continuous Culture</h3> <p>Continuous this, continuous that, continuous everything where every organisational discipline has had this notion ingrained into their very DNA as a process of finding and eliminating waste on an ongoing basis. Add today’s agile age, where continuous delivery is a critical outcome in order to survive it is hence part of an organisation’s business culture that must involve everyone, leadership, management and employees.</p> <p>If we also add the premise of “Shift Left” across an organisation you would then move things that we typically do in later stages earlier. (It is human nature, but many people tend to defer particularly tough issues) – even shift left in terms of the customer by empowering the customer further.</p> <p>DevOps is predicated by shifting left in its testing approach, performing testing earlier in the software delivery lifecycle which will inevitably eliminate long back-end dependencies and increase quality.</p> <h3>Step 3 – Empower a culture of Fail Fast not Fail-Silently</h3> <p>When I first started my business work life it was a norm to “fail silently” where you mask the failure, hope it does not get spotted, blame someone else or simply ignore the issue. New ways of thinking by innovative companies is for the business to fail fast &#x26; fail often as long as they learn the lesson. This strategy can help the company grow in an agile way. Across the organisation not just technology DevOps “failing” principles can include:</p> <ul> <li>Fail Early, the sooner the failure is spotted, the sooner the learning begins and the sooner you can ultimately fix it. This will also allow you to get real and fast feedback about what works and what does not;</li> </ul> <p>*. Fail Fast, so that we can begin the learning process as fast as possible. In DevOps for instance a test driven process where you write a failing test before you even produce the code;</p> <ul> <li>Fail Often, the more things you try, the more failures you will see and therefore the more chances to both learn and steer your output in the right direction. In addition, this will remove the need to waste time by working on incorrect avenues;</li> <li>Fail Better, with early and frequent failures you maximise the learning which will deliver customers advocacy.</li> </ul> <p>According to Heraclitus, the Greek philosopher, “the only thing constant is change.” Indeed your business, and particularly the way you run its digital performance, is no exception. But although DevOps is obviously the evolution of software development from waterfall delivery to agile delivery - its premise and baseline is predicated by sound business principles that must be applied across the business – NOT just IT.</p> <p>True transformation and change, continuous agile processes, collaborative and co-creation working, effective governance and quality services are a matter for all disciplines in an organisation. IT obviously must enable the business with effective technologies and governance, but it cannot do it alone – it needs a true partner in the business to deliver effective business solutions and outcomes.</p> <p>“<a href="https://quotesia.com/paulo-coelho-quote/908683">In our obsessive wish to arrive, we often forget the most important thing, which is the journey</a>” , <a href="https://quotesia.com/paulo-coelho-quotes">Paulo Coelho</a></p> <p><a href="https://www.hpe.com/us/en/services.html">Get started with your transformation today by visiting our HPE Pointnext Web-site</a>.</p> <p>Mario Devargas</p> <p>WW Strategic Transformation</p> <p>Hewlett Packard Enterprise</p>https://developer.hpe.com/master-the-redfish-server-states-to-improve-your-monitoring-and-manageme/https://developer.hpe.com/master-the-redfish-server-states-to-improve-your-monitoring-and-manageme/Mon, 06 Aug 2018 14:07:07 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/masterserverstates/masterserverstates">Server Management Portal</a>.</big></p> <br>https://developer.hpe.com/using-the-simplivity-api-with-java/https://developer.hpe.com/using-the-simplivity-api-with-java/Fri, 03 Aug 2018 16:27:34 GMT<h1><strong>Using the SimpliVity API with Java</strong></h1> <p>This sample Java code performs authentication, issues example GET requests, performs a POST operation (in this case, renaming a backup), and monitors the status of the operation using a task instance.</p> <pre><code class="language-java">package main.java; import java.security.SecureRandom; import java.security.cert.X509Certificate; import java.util.Arrays; import javax.net.ssl.HttpsURLConnection; import javax.net.ssl.SSLContext; import javax.net.ssl.TrustManager; import javax.net.ssl.X509TrustManager; import javax.net.ssl.SSLSession; import javax.net.ssl.HostnameVerifier; import org.springframework.http.HttpEntity; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpMethod; import org.springframework.http.MediaType; import org.springframework.http.ResponseEntity; import org.springframework.util.LinkedMultiValueMap; import org.springframework.util.MultiValueMap; import org.springframework.web.client.RestTemplate; import org.json.JSONObject; import com.sun.org.apache.xerces.internal.impl.dv.util.Base64; public class RestClient { private static String access_token; private String BASE_URL; static final String HMS_USERNAME = "HMS_USER"; static final String HMS_PASSWORD = "HMS_PASS"; public RestClient(String hostIp) { BASE_URL = "https://"+hostIp+"/api/"; } // Create a trust manager that does not validate certificate chains. private void enableSSL() { TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() { public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; } public void checkClientTrusted(X509Certificate[] certs, String authType) { } public void checkServerTrusted(X509Certificate[] certs, String authType) { } } }; try { SSLContext sc = SSLContext.getInstance("TLSv1.2"); sc.init(null, trustAllCerts, new SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { return true; } }); } catch (Exception e) { } } /* * Authenticate user and retrieve access token. */ public String getAccessToken() { enableSSL(); String encoding = Base64.encode("simplivity:".getBytes()); RestTemplate restTemplate = new RestTemplate(); MultiValueMap&#x3C;String, String> body = new LinkedMultiValueMap&#x3C;String, String>(); body.add("username", HMS_USERNAME); body.add("password", HMS_PASSWORD); body.add("grant_type", "password"); HttpHeaders headers = new HttpHeaders(); headers.set("Accept", "application/json"); headers.set("Authorization", "Basic " + encoding); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>(body, headers); ResponseEntity&#x3C;String> res = restTemplate.exchange( BASE_URL+"oauth/token", HttpMethod.POST, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); access_token = (String) jsonObj.get("access_token"); System.out.println("Authenticated user and retrieved access token: "+ access_token); return access_token; } /* * Issue a GET request: GET /policies. */ public Object getPolicies() { RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON)); headers.set("Authorization", "Bearer " + access_token); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>("parameters", headers); ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies", HttpMethod.GET, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object policies= jsonObj.get("policies"); System.out.println(policies.toString()); return policies; } /* * Issue a GET request with sorting and filtering: * GET the first 100 policies * sorted in ascending order by name * and show only the name and rules fields. */ public Object getFirst100Policies() { RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.set("Accept", "application/json"); headers.set("Authorization", "Bearer " + access_token); HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;Object>("parameters", headers); ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies?fields=name,rules&#x26;limit=100&#x26;offset=0&#x26;sort=name&#x26;order=ascending", HttpMethod.GET, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object policies= jsonObj.get("policies"); System.out.println(policies.toString()); System.out.println("Limit: "+ jsonObj.get("limit")); System.out.println("Count: "+ jsonObj.get("count")); return policies; } /* * Issue a POST request: Create a new policy. */ public void createNewPolicy() { RestTemplate restTemplate = new RestTemplate(); // Set a custom media type. MediaType myMediaType = new MediaType("application", "vnd.simplivity.v1+json"); // Set the headers. HttpHeaders headers = new HttpHeaders(); headers.setAccept(Arrays.asList(myMediaType)); headers.setContentType(myMediaType); headers.set("Authorization", "Bearer " + access_token); // Form the POST body. String policyMo = "{\"name\": \"randomPolicyName4\"}"; HttpEntity&#x3C;?> entity = new HttpEntity&#x3C;String>(policyMo, headers); // Issue the POST operation and expect a task object in return. ResponseEntity&#x3C;String> res = restTemplate.exchange(BASE_URL+"policies", HttpMethod.POST, entity, String.class); JSONObject jsonObj = new JSONObject(res.getBody()); Object task = jsonObj.get("task"); JSONObject taskJson = new JSONObject(task.toString()); String taskId = taskJson.getString("id"); String state = taskJson.getString("state"); // Monitor the status of the policy creation operation by using a loop to query // the task while this task is IN_PROGRESS. // The state field in the JSON response body indicates the status. while(state.equals("IN_PROGRESS")) { // Wait one second. try { Thread.sleep(1000); } catch (InterruptedException e) { e.printStackTrace(); } res = restTemplate.getForEntity(BASE_URL+"tasks/"+taskId, String.class); jsonObj = new JSONObject(res.getBody()); task = jsonObj.get("task"); taskJson = new JSONObject(task.toString()); state = taskJson.getString("state"); } System.out.println("Task object: " +task.toString()); } public static void main(String[] args) { RestClient restClient = new RestClient("10.150.1.71"); // Authenticate user and retrieve access token. restClient.getAccessToken(); // GET policies. restClient.getPolicies(); // GET first 100 policies sorted in ascending order by name // and showing only the name and rules fields. restClient.getFirst100Policies(); // POST /policies: Create a new policy. restClient.createNewPolicy(); } } </code></pre>https://developer.hpe.com/setting-bios-and-storage-controller-properties-with-redfish/https://developer.hpe.com/setting-bios-and-storage-controller-properties-with-redfish/Thu, 19 Jul 2018 15:11:00 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <style> .warning-box { background-color: #fff3cd; /* Light yellow background */ /* border: 2px solid #ffeb3b; /* Yellow border */ color: #856404; /* Dark text color for contrast */ padding: 20px; /* Padding inside the box */ border-radius: 5px; /* Rounded corners */ margin: 20px 0; /* Margin for spacing */ width: 80%; /* Width of the rectangle */ max-width: 600px; /* Maximum width of the rectangle */ margin-left: auto; /* Center the rectangle horizontally */ margin-right: auto; /* Center the rectangle horizontally */ /* box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1); /* Optional: Adds a shadow for depth */ } .warning-box p { margin: 0; /* Remove default margin from the paragraph */ font-weight: bold; /* Bold text */ } .warning-box a { color: blue; /* Change the color of the link */ text-decoration: none; /* Remove the underline */ /*font-style: italic; /* Make the text italic */ /*font-weight: bold; /* Make the text bold */ } </style> <!--StartFragment--> <div class="warning-box"> <p>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/biosandstorageprops/biosandstorageprops" target="_blank">HPE server management portal</a>.</p> </div> <!--EndFragment-->https://developer.hpe.com/hpe-discover-hackathon-wrap-up-top-10-list-of-what-you-should-know/https://developer.hpe.com/hpe-discover-hackathon-wrap-up-top-10-list-of-what-you-should-know/Tue, 10 Jul 2018 16:10:36 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hpe-dev-hack-shack-team-1531348164725.jpg" alt="hpe dev hack shack team"></p> <p>At HPE Discover 2018 Las Vegas this year, HPE held their first ever hackathon for customers in a Hack Shack located in the event showcase. The goal of the hackathon was to promote innovation through open source collaboration.</p> <p>During the 3-day event, over 280 people experienced the HPE Hack Shack. They entered a large workspace with slated wood walls containing 10 tables, chairs, and computers available for hacking – all complete with a mood lighting disco ball and background music. A backyard recreational area was also provided (just for hanging out) complete with Astroturf grass, swings, video games, a giant Jenga tower, and a Corn Hole game.</p> <p>Seventy-three enthusiastic would-be hackers sat down for one of five two-hour sessions to test their coding chops. The participants’ skills ranged from absolute beginner to intermediate levels. All participants received a coveted HPE DEV T-shirt. And one talented and creative hacker secured the grand prize of a DJI Mavic Air Drone with accessories. Second place received a DJI Spark drone with accessories. Amazon Echo Dots were also awarded to top finishers.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hpe-dev-hack-shack-line-1531348159077.jpg" alt="hpe dev hack shack line"></p> <h1>Top 10 things you should know about the first-ever HPE Discover Hackathon</h1> <h3>10) You didn’t need good looks or coding skills to get into the HPE Discover Hack Shack</h3> <p>But you did need a willingness to learn. Because no coding talents were required, even novices participated…and did well!</p> <h3>9) Pretty much everyone thought it was cool</h3> <p>Hackathons are a fun way to learn software development and coding skills—and the first ever HPE Discover hackathon delivered. Participants were able to tap into skills that helped them better understand infrastructure as code, a great skill to have in this market.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hack-shack-runnerup-1531348144583.jpg" alt="Hack Shack Ryan Jensen "></p> <h3>8) A common question: “Why I am coding “Hello World?”</h3> <p>Traditionally, “Hello World” is the very first program people write. It’s used to illustrate the basic syntax of programming languages. Hey, we didn’t make the rules…we just follow them…well, at least sometimes.</p> <h3>7) Most common reaction of participants: Shock and awe-some!</h3> <p>The HPE Developer Community (HPE DEV) expected as much. Many people typically think of HPE as an infrastructure company. HPE DEV is starting to change that. An awesome team of experts are working hard to build out a vibrant, collaborative, and contributing software developer community. So stay tuned…</p> <h3>6) Top question asked: Where’s the beer?</h3> <p>As esteemed hosts, the HPE DEV team saved it for the end, knowing all hackathons end well with a beer. Besides, you don’t win friends (or hackathons) over salad.</p> <h3>5) Everyone loved Sir Hackington Appbuilder III</h3> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hpe-dev-sir-hackington-1531348171119.jpg" alt="Sir Hackington III"></p> <p>Who doesn’t love an adorable puppy – even if he is a statue? Sir Hackington Appbuilder III will be attending all upcoming hackathons, as he is now the HPE DEV team’s official mascot.</p> <h3>4) The winner’s hack was amazing</h3> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hack-shack-winner-michael-lai-1531348152252.jpg" alt="Michael Lai Hack Shack winner"></p> <p>Congratulations to Michael Lai who participated in the HPE OneView detect infrastructure failures hack. He created a server profile from a template and applied it to a server hardware. He then created an alert resource to monitor the server profile. A script runs every 10 seconds, which monitors the status and sends a status alert to the user.</p> <h3>3) HPE loves Open Source and sharing!</h3> <p>The winner’s script is in a state that can be easily extended and shared with the open source community to monitor infrastructure resources. It’s also something HPE DEV will add as an example to the SDKs – along with other hacks.</p> <h3>2) Who says you can’t teach an old dog new tricks</h3> <p>As the first-ever HPE Discover Hackathon, the HPE DEV team wasn’t really sure what attendees would think. The “Hello World” hack was eagerly embraced; beginners really do want to learn something new. And in today’s software-defined data center, the magic of coding is a great new skill to develop, and HPE DEV is here to help you grow your skills.</p> <h3>1) Overall takeaway from HPE Discover Hackathon</h3> <p>Everyone had a great time and will definitely come back for future Discover Hackathons. The HPE DEV team is already brainstorming creative and fun challenges for the next one!</p> <h1>A final word from our sponsor:</h1> <p>The HPE Developer Community connects you with HPE experts, resources and other developers worldwide to help you improve automation, service delivery, and workflow assets using open source software on HPE platforms and solutions. If you are interested in learning more or participating in a future HPE Hackathon, please reach out to <a href="https://www.labs.hpe.com/slack">HPE DEV on SLACK</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hackathon-1530031408442.png" alt="hackathon"></p>https://developer.hpe.com/hpe-oneview-puppet-module-now-available-with-support-for-hpe-oneview-40/https://developer.hpe.com/hpe-oneview-puppet-module-now-available-with-support-for-hpe-oneview-40/Tue, 03 Jul 2018 16:50:59 GMT<h1>HPE OneView Puppet module now available with support for HPE OneView 4.0</h1> <p>The HPE OneView Puppet module provides Puppet manifests and resources to interact with HPE OneView and HPE Synergy Image Streamer APIs, enabling developers to easily build integrations and scalable solutions.</p> <p>The HPE OneView Puppet module v2.3.0 now supports HPE OneView 4.0 (REST API version 600).</p> <p>The list of supported resources and changes is available at: <a href="https://github.com/HewlettPackard/oneview-puppet/blob/v2.3.0/CHANGELOG.md">https://github.com/HewlettPackard/oneview-puppet/blob/v2.3.0/CHANGELOG.md</a></p> <p>Release content is available at: <a href="https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.3.0">https://github.com/HewlettPackard/oneview-puppet/releases/tag/v2.3.0</a></p> <p>The repository with code and examples is available on GitHub at: <a href="https://github.com/HewlettPackard/oneview-puppet">https://github.com/HewlettPackard/oneview-puppet</a></p> <p>Puppet Forge details are available at: <a href="https://forge.puppet.com/hewlettpackard/oneview">https://forge.puppet.com/hewlettpackard/oneview</a></p>https://developer.hpe.com/winners-named-in-the-hpe-developer-community-hackathon/https://developer.hpe.com/winners-named-in-the-hpe-developer-community-hackathon/Tue, 26 Jun 2018 16:40:23 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/hackathon-1530031408442.png" alt="hackathon"></p> <p>On May 24, HPE developers from around the world and across various HPE organizations began participating in the HPE Hackathon. A total of 49 individual hackers took up the challenge, comprising 18 teams that ranged in size from 1-7 participants. In the end, 12 teams crossed the finish line and presented outstanding projects to a distinguished panel of 4 judges.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/may24-2-1530031551619.jpg" alt="may24 2"></p> <p>Judging took place from June 4th-10th. Teams were given 15 minutes to present their projects, which were judged on the following criteria:</p> <ul> <li>20%: Completeness &#x26; Presentation</li> <li>20%: Technical Merit</li> <li>20%: Originality</li> <li>40%: Potential Customer Impact</li> </ul> <p>Participant experience varied widely--from teams with little to no coding background (looking to grow their skills), all the way to very experienced coders. Each team was assigned a coach from the Developer Experience &#x26; Incubation team to guide and assist the team’s work.</p> <h1>And the winners are…</h1> <p>Congratulations to <strong>Team Chatbot</strong>, who was awarded first place for their project. Team members include Jesse Olsen, Dana Lynn, and Jeroen Kleen. Team Chatbot built an awesome chatbot based on a React/Redux platform. More details about this project will be coming soon on the HPE DevCom portal.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/ric-hack-1530031624425.jpg" alt="ric hack"></p> <p><strong>Team Reality Distortion</strong> received second place honors. Congratulations to the one-man team of Damian Janiszewski.</p> <p>Damian developed an HPE OneSphere extension module that exports HPE OneSphere metrics for Prometheus time series database and alerting system. Prometheus metrics are then easily visualized using Grafana dashboards.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/6/image-uploaded-from-ios-1530031727049.jpg" alt="hackathon warsaw"></p> <p>Third place went to <strong>Team Last Bench</strong>, for their project. Contributing team members include Gandharva S, Shiva Kumar M, Krishna Kanth Mallela, Manjunath Patil, and Sahana Sampige Prabhakar.</p> <p><strong>Thank you to all participants</strong></p> <p>The HPE DevCom would like to thank all of the HPE Hackathon participants for their outstanding creativity and hard work. A partial list of other completed projects includes:</p> <ul> <li><strong>Team NoName</strong>: Collect &#x26; display HPE OneSphere data using R</li> <li><strong>Team Everest</strong>: Ruby bindings for HPE OneSphere</li> <li><strong>Team Hela</strong>: HPE OneSphere metrics collection using Grafana</li> <li><strong>Team Spectrum Spiders</strong>: Ansible module for HPE OneSphere</li> <li><strong>Team Ashlesha</strong>: Enable provisioning of AWS CloudFormation templates via the HPE OneSphere catalog</li> <li><strong>Team Last Bench</strong>: ServiceNow integration with HPE OneSphere</li> <li><strong>Team India Matrix Master</strong>: Terraform module for HPE OneSphere</li> </ul> <p>The work completed by all of the teams during the hackathon will be leveraged by the HPE Developer Community (@HPE_DevCom). More information about these projects and many others will be available in upcoming articles on the community’s blog site.# Why participate in a hackathon?</p> <p>Hackathons are a great way for companies to foster innovation through collaboration and creative problem-solving. And that’s exactly what the HPE Developer Community witnessed. The hackathon was designed to:</p> <ol> <li>Foster cross-organization collaboration and innovation</li> <li>Provide an opportunity for personal growth (growing software development skills in HPE)</li> <li>Provide an opportunity for the community to create tangible customer-centric solutions</li> <li>Build a foundation upon which to create a robust internal HPE developer community</li> <li>Have fun!</li> </ol> <p>If you are interested in learning more or participating in a future HPE Hackathon, please reach out to <a href="https://www.labs.hpe.com/slack">the HPE DevCom on SLACK</a>.</p>https://developer.hpe.com/hpe-oneview-chef-cookbook-now-available-with-support-for-hpe-oneview-40/https://developer.hpe.com/hpe-oneview-chef-cookbook-now-available-with-support-for-hpe-oneview-40/Wed, 06 Jun 2018 19:55:30 GMT<h1>HPE OneView Chef cookbook now available with support for HPE OneView 4.0</h1> <p>The HPE OneView Chef cookbook provides Chef recipes to interact with HPE OneView and HPE Synergy Image Streamer APIs, enabling developers to easily build integrations and scalable solutions.</p> <p>The HPE OneView Chef cookbook v3.1.0 now supports HPE OneView 4.0 (REST API version 600).</p> <p>The list of supported resources and changes is available at: <a href="https://github.com/HewlettPackard/oneview-chef/blob/v3.1.0/CHANGELOG.md">https://github.com/HewlettPackard/oneview-chef/blob/v3.1.0/CHANGELOG.md</a></p> <p>Release content is available at: <a href="https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.1.0">https://github.com/HewlettPackard/oneview-chef/releases/tag/v3.1.0</a></p> <p>The repository with code and examples is available on GitHub at: <a href="https://github.com/HewlettPackard/oneview-chef">https://github.com/HewlettPackard/oneview-chef</a></p> <p>Chef Supermarket details are available at: <a href="https://supermarket.chef.io/cookbooks/oneview/versions/3.1.0">https://supermarket.chef.io/cookbooks/oneview/versions/3.1.0</a></p>https://developer.hpe.com/storage-provisioning-using-ansible-with-hpe-3par-storage/https://developer.hpe.com/storage-provisioning-using-ansible-with-hpe-3par-storage/Wed, 30 May 2018 22:22:15 GMT<h1>Storage Provisioning using Ansible with HPE 3PAR Storage</h1> <p>Recently I was asked to look at getting the HPE 3PAR to work with Ansible. My exposure to Ansible before this, was by name recognition only. I didn’t have any working knowledge with it. So like every good challenge, I set off to learn this new “tool”.</p> <p>After looking online at several Youtube videos, I got an elementary understanding to what was possible with Ansible. I must say, it was definitely refreshing it was YAML based compared to some of the other languages I had been using i.e. javascript and Ruby. YAML is a much “simpler” language to understand and read.</p> <p>If you aren’t familiar with YAML, it stands for “YAML Ain't Markup Language”. It is used widely among programming languages as well as configuration files for many of the popular apps. So if you have dug around much within Linux, chances are you have come across YAML.</p> <p>Here is a quick example from Ansible’s website. Take a look at the documentation to learn more about the formatting for YAML. Reading YAML is pretty straightforward.</p> <pre><code class="language-yaml"># Employee records - martin: name: Martin D'vloper job: Developer skills: - python - perl - pascal - tabitha: name: Tabitha Bitumen job: Developer skills: - lisp - fortran - erlang </code></pre> <p>Again this is a simple example, but these same principals apply from the most basic to the most complex Ansible playbooks.</p> <p>Now let’s talk more specifically about Ansible and playbooks.</p> <p>Ansible’s website says it the best:</p> <blockquote> <p>Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. If Ansible modules (modules are Ansible “plugins” that control system resources, like services, packages, or files) are the tools in your workshop, playbooks are your instruction manuals, and your inventory of hosts are your raw material.</p> </blockquote> <p>At a basic level, playbooks can be used to manage configurations of and deployments to remote machines. At a more advanced level, they can sequence multi-tier rollouts involving rolling updates, and can delegate actions to other hosts, interacting with monitoring servers and load balancers along the way.</p> <p>With this in mind, I will be specifically talking about creating Ansible playbooks to automate Storage deployments. Many developers and Operation teams are already using Ansible to help deploy infrastructure as well as apps within their datacenters, so it makes sense to include storage in these processes and workflows. Since I work primarily with the HPE 3PAR Storserv array, naturally I wanted to start there and figure out how to make Ansible talk to the HPE 3PAR’s Web Services API (3PAR’s REST API).</p> <p>I will be covering two different methods of how you can use Ansible with the 3PAR. If you are already "TLDR", but want to know specifically how to use the new Storage module, you can skip over this section and go the section <strong>"The New Better Way"</strong></p> <h1>Old Method (without the Storage Module)</h1> <p>When I first started looking into how to connect to Ansible to the WSAPI, I quickly realized that since we didn’t have a module (at the time) we would have to do it “old school” using REST calls. Since I came from several other projects recently where I had to learn REST (that is another topic in and of itself), this wasn’t going to be too difficult. I was able to find the Ansible uri module to work with REST. So let’s take a look at how to write a raw playbook using REST and then I can show you the new HPE 3PAR module so you understand the benefits that come from the work done by the HPE engineering team. Here is a “simple” example of a custom playbook using the uri module to query the HPE 3PAR to get volumes.</p> <p>Let’s start with creating the playbook. Every playbook (since it is YAML based) starts with <code>---</code> and then the name of the playbook. Here you can define the hosts that it will run against. Since we are using the <strong>uri</strong> module on the local machine to connect to a 3PAR, we set this to <strong>localhost</strong></p> <pre><code class="language-yaml">--- - name: Connect to 3par - get volumes hosts: localhost vars: auth_3par_user: "3paruser" auth_3par_password: "3parpass" ip_address_3par: 192.168.1.50 rest_api_url_3par: "https://{{ ip_address_3par }}:8080/api/v1" </code></pre> <p>The next section, includes your variables. These can be included in the playbook (like shown), at runtime, or in a configuration file. It can also be a mix of these, for example, global variables could be in a configuration file while the runtime variables configured in the playbook or provided at the command line with the -e flag.</p> <blockquote> <p>Be aware, <strong>YAML</strong> is space sensitive. Always make sure your sections contain the same number of spaces in front otherwise it may be interpreted differently. Also, TABS have been outlawed in YAML and are not allowed to prevent errors between editors. You have been warned.</p> </blockquote> <p>The next section contains the task(s). A playbook can contain a single task or multiple tasks to accomplish the desired configuration. Each task is denoted with the <code>– name:</code> descriptor and provides a meaningful name to the task. Each task calls upon an Ansible module that is able to accomplish the desired endstate, in our case the <strong>uri</strong> module. Under the <strong>uri</strong> module, you will see the various parameters that are required in order for Ansible to talk with the HPE 3PAR.</p> <pre><code class="language-yaml">tasks: - name: check if 3par WSAPI is running uri: url: "{{ rest_api_url_3par }}/credentials" method: POST headers: Content-Type: "application/json" body_format: json body: "{ 'user': '{{ auth_3par_user }}', 'password': '{{ auth_3par_password }}' }" status_code: 201 return_content: yes validate_certs: no register: output - name: Parsing key debug: msg: "{{ output.json.key }}" - name: GET 3par hosts - get hosts uri: url: "{{ rest_api_url_3par }}/volumes" method: GET headers: Content-Type: "application/json" X-HP3PAR-WSAPI-SessionKey: "{{ output.json.key }}" Accept: "application/json" status_code: 200 return_content: yes validate_certs: no register: host_output - name: Parsing Host GET debug: msg: "{{ host_output }}" - name: release authentication key uri: url: "{{ rest_api_url_3par }}/credentials/{{ output.json.key }}" method: DELETE headers: Content-Type: "application/json" validate_certs: no </code></pre> <p>As you can see, there are 5 tasks in this playbook.</p> <ol> <li>Authentication to the HPE 3PAR</li> <li>Acquires the required SessionKey</li> <li>With the SessionKey, Queries the HPE 3PAR for all volumes</li> <li>Outputs the query to the terminal</li> <li>Releases the SessionKey</li> </ol> <p>There are a lot of steps involved here to do something so simple. The nice thing though, once you have this initial playbook built, managing (i.e. provisioning volumes, creating vvsets, etc) your HPE 3PAR is rather straight forward only requiring the change of the REST call <code>url: "{{ rest_api_url_3par }}/volumes"</code>.</p> <p>You can include additional tasks, like exporting volumes, to make these playbooks more robust and part of a larger workflow. When I created a number of these playbooks in this manner, I found out quickly there were a lot of limitations to using the uri module, which I want highlight.</p> <h3>Impressions of using REST/uri module</h3> <p><strong>First</strong>, is <strong><em>length</em></strong> of your playbooks. This was a simple playbook but it was still <strong>57 lines of code</strong>. That is 57 different areas to make a mistake (especially with spaces). When I created some larger bare-metal provisioning workflows, they were hundreds of lines of code which becomes harder to follow, error prone and far more complex than I want.</p> <p><strong>Second</strong>, the <code>uri</code> module. I had to use native <strong>REST</strong> calls to talk with the 3PAR. I was fortunate to have learned <strong>REST</strong> in previous projects, but it isn’t easy to pick up on the fly so if you had to get up and running quickly with Ansible this would be another HUGE learning curve in addition to learning Ansible.</p> <p><strong>Lastly</strong>, there are no <em>3PAR specific resources</em> to work with. I was dependent on the <code>uri</code> module and then parsing <strong>json</strong> in order to get specific information I needed in my playbooks like <strong>volume names</strong>, <strong>lun id</strong>, <strong>SessionKeys</strong>, etc. An important part of automation, is the ability target an object (be it an <strong>HPE 3PAR volume</strong>, <strong>ESXi host</strong>, <strong>nginx service</strong>) so you can act upon it later in your workflows.</p> <p>This is where the HPE 3PAR modules for Ansible comes into play. <a href="https://github.com/HewlettPackard/hpe3par_ansible_module">https://github.com/HewlettPackard/hpe3par_ansible_module</a></p> <h1>New Better Way (with the Storage Module)</h1> <p>Okay like I said, the previous is the long way to do it. If you like to write in <strong>REST</strong> and enjoy having really long playbooks, then continue doing what you are doing. Otherwise, HPE has partnered with Red Hat and the Ansible team to develop high quality storage modules to manage the HPE 3PAR array. This collaboration has led to the development of HPE 3PAR specific storage modules, playbooks and resources in order to manage your HPE 3PAR storage arrays.</p> <p>The HPE 3PAR modules for Ansible is developed as a set of modules and example playbooks to provision the following:</p> <ul> <li>CPG</li> <li>Host</li> <li>Volume</li> <li>VLUN</li> <li>Host Set</li> <li>Volume Set</li> <li>Volume Offline Clone</li> <li>Volume Online Clone</li> <li>Volume Snapshot</li> <li>QOS</li> <li>Flash Cache</li> </ul> <p>So let’s look at what a playbook looks like using the HPE 3PAR modules for Ansible.</p> <p>First step to using the storage module, is to download it from Github. As of Ansible 2.5 and earlier, you will need to download it from Github, after the release of Ansible 2.6, it will be part of Ansible core and included when you download future versions.</p> <pre><code>git clone https://github.com/HewlettPackard/hpe3par_ansible_module </code></pre> <p>Once we have it downloaded, you can look at the various items included with the Module.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/image1-1528126916512.png" alt="image1"></p> <p>We are most interested in the Modules and playbooks. The Modules folder contains the code (written in Python) behind the playbooks and these specific modules do all of the work on the 3PAR so we don’t have to write lengthy and complex REST calls like in the beginning example. They also provide HPE 3PAR resources and actions that we can then use within our playbooks like create volumes, modify 3PAR hosts, or delete hostsets as an example. We don’t have to modify anything here and they are available when we create our playbooks.</p> <p>Now let's look at the playbooks. There is a pretty large list of available playbooks here.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/image2-1528127089601.png" alt="image2"></p> <p>The name of the playbooks correlates to the actions and resources that you will be working with. So for example if we want to create a <strong><em>snapshot</em></strong>, we would be working with the <strong>snapshot_playbook.yml</strong> or if we want to manage <strong><em>volumes</em></strong>, we would be using the <strong>volume_playbook.yml</strong>.</p> <p>Let’s look at the volume playbook,</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/image3-1528127229536.png" alt="image3"></p> <p>Variables within <strong>YAML</strong> are defined with curly brackets: <code>{{ variable }}</code>.</p> <p>Second, <strong><em>state</em></strong>.</p> <p>The <strong><em>state</em></strong> option tells Ansible what action to perform. Ansible is a Change Management tool, so you don’t actually write code, instead playbooks define your environment and how it should be configured.</p> <p>Let’s look at the <strong><em>Create Volume task</em></strong> as an example, state=present. Upon running the playbook, the Create Volume task will call the HPE 3PAR storage module to check to see if the volume exists on the array and if it doesn’t, it will be created. State tells Ansible that the volume must be present.</p> <p>Looking back at the <strong>volume_playbook.yml</strong> as well as any of the other playbooks, you will see the many state definitions, each performing a specific action. Some of them include:</p> <ul> <li>present</li> <li>convert_type</li> <li>set_snap_cpg</li> <li>change_snap_cpg</li> <li>grow</li> <li>grow_to_size</li> <li>modify</li> <li>delete</li> </ul> <p>Now let’s create another playbook using these modules to create a volume and then export it to a host.</p> <p>Open your favorite editor.</p> <p><code>vi demo_playbook.yml</code></p> <p>Let’s provide a name for our playbook and it will be running against <code>localhost</code> again.</p> <pre><code class="language-YAML">--- - name: Demo 3PAR Ansible playbook hosts: localhost </code></pre> <p>Next let’s define our variables. In this example, we will use a combination of locally defined <strong>variables</strong> and an external configuration file (<code>include_vars</code>). Under <code>vars</code>, we specify the volume specifics.</p> <pre><code class="language-YAML">vars: volume_name: 'demo_ansible_volume' size: 10 size_unit: 'GiB' cpg: 'FC_r1' vlun_host_name: 'example_host' autolun: False lunid: 110 tasks: - name: Load Storage System Vars include_vars: 'properties/storage_system_properties.yml' </code></pre> <p>Before we go further, here is what the <code>properties/storage_system_properties.yml</code> file looks like. It is a simple file with 3 lines, defining the connection information for the HPE 3PAR.</p> <pre><code class="language-YAML">storage_system_ip: "192.168.1.50" storage_system_username: "3paruser" storage_system_password: "3parpass" </code></pre> <p><strong>Note:</strong> You can include any of your variables in a combination of configuration files or within the playbook itself. It is completely up to you. I like to use configuration file for my “global” variables or to use them as “profiles” to limit the user defined variables in my playbooks.</p> <p>Now let’s create our tasks. Like I mentioned before, we can copy and paste these from the playbook templates found under the playbooks folder in the HPE 3PAR Storage Module. We will be copying the <strong><em>Create Volume</em></strong> action from the <strong>volume_playbook.yml</strong> and the <strong><em>Create VLUN</em></strong> action from the <strong>volume_to_host_vlun_playbook.yml</strong> playbook.</p> <pre><code class="language-YAML">- name: Create Volume "{{ volume_name }}" hpe3par_volume: storage_system_ip="{{ storage_system_ip }}" storage_system_username="{{ storage_system_username }}" storage_system_password="{{ storage_system_password }}" state=present volume_name="{{ volume_name }}" cpg="{{ cpg }}" size="{{ size }}" size_unit="{{ size_unit }}" - name: Create VLUN hpe3par_vlun: storage_system_ip="{{ storage_system_ip }}" storage_system_username="{{ storage_system_username }}" storage_system_password="{{ storage_system_password }}" state=export_volume_to_host volume_name="{{ volume_name }}" host_name="{{ vlun_host_name }}" lunid="{{ lunid }}" autolun="{{ autolun }}" </code></pre> <p>We need to make one minor change. Under the <strong><em>Create VLUN</em></strong> task, we need to modify the following variable:</p> <pre><code class="language-YAML">volume_name="{{ vlun_volume_name }}" </code></pre> <p>to</p> <pre><code class="language-YAML">volume_name="{{ volume_name }}" </code></pre> <p>This allows you to link the two tasks by using the volume you created as the volume to export in the following task. Other than that we are done and have created a simple, yet powerful provisioning playbook.</p> <p>With it all together, it should look like and is ready to run.</p> <pre><code class="language-yaml">--- - name: Demo 3PAR Ansible playbook hosts: localhost vars: volume_name: 'demo_ansible_volume' size: 100 size_unit: 'GiB' cpg: 'FC_r6' vlun_host_name: 'example_host' autolun: False lunid: 110 tasks: - name: Load Storage System Vars include_vars: 'properties/storage_system_properties.yml' - name: Create Volume "{{ volume_name }}" hpe3par_volume: storage_system_ip="{{ storage_system_ip }}" storage_system_username="{{ storage_system_username }}" storage_system_password="{{ storage_system_password }}" state=present volume_name="{{ volume_name }}" cpg="{{ cpg }}" size="{{ size }}" size_unit="{{ size_unit }}" - name: Create VLUN hpe3par_vlun: storage_system_ip="{{ storage_system_ip }}" storage_system_username="{{ storage_system_username }}" storage_system_password="{{ storage_system_password }}" state=export_volume_to_host volume_name="{{ volume_name }}" host_name="{{ vlun_host_name }}" lunid="{{ lunid }}" autolun="{{ autolun }}" </code></pre> <p>Now let’s run the playbook.</p> <pre><code>ansible-playbook demo_playbook.yml </code></pre> <p>You should see the following output:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/image4-1528127766708_80-1528129228459.png" alt="image4 1528127766708_80"></p> <p>If you look at your HPE 3PAR array, you will now see it your new volume.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/image5-1528128461213.png" alt="image5"></p> <p>Congratulations on creating a playbook using the HPE 3PAR Storage modules for Ansible.</p> <p>The HPE 3PAR Storage modules make the job much simpler as well as you have greater flexibility in integrating HPE 3PAR into your DevOps and Operation team workflows. Again looking back at our original playbook example, it took 57 lines of code to simply read the volumes from the HPE 3PAR. We were able to create and then export the volume to a server in 37.</p> <p>This really is just the beginning of what you can do with the HPE 3PAR Storage modules for Ansible. This was a standard provisioning playbook. You can create playbooks that include tasks from bare-metal provisioning, OS deployment, storage provisioning all the way to application deployment and management. You can take these playbooks and integrate them into Ansible Tower to build out a self-service portal.</p> <p>The future looks brighter than ever when it comes to HPE. The landscape of the datacenter is constantly changing and the demand for storage and rapid deployments continue to grow. Thank you for following this tutorial and hopefully it has been useful and sparked interest and ideas about how to use Ansible and HPE Storage. Wherever you are on your journey, we're very eager to hear what challenges exist out there today. I genuinely believe HPE can help out in a multitude of ways.</p>https://developer.hpe.com/a-behind-the-scenes-look-into-the-projects-of-the-hpe-hackathon/https://developer.hpe.com/a-behind-the-scenes-look-into-the-projects-of-the-hpe-hackathon/Thu, 24 May 2018 20:29:14 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/hpe-hackathon-3d-v1-1-1526997723888.jpg" alt="HPE Hackathon"></p> <p>On May 24, HPE developers from around the world began participating in the HPE Hackathon. A hackathon is a coding contest where developers, programmers, and other creative experts come together to collaborate on a range of software development topics. In the <a href="/blog/psssst-ive-got-the-inside-scoop-on-the-upcoming-hpe-hackathon">HPE Hackathon</a>, developers will focus on <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a>, HPE’s new hybrid-cloud management platform.</p> <p>The HPE Hackaton has 19 different teams of up to 5 people each located all over the world. Each team is focusing on a unique aspect of the HPE OneSphere API for their project. Below is a sampling of the different types of projects the teams will be working on during the event:</p> <h2>Team: India Matrix Master</h2> <h3>Location: Fort Collins, CO and India</h3> <p>This team of 5 is building a Terraform plugin for HPE OneSphere, allowing Terraform users to describe the desired state of its infrastructure in Terraform language and make sure this desired state is matched by running Terraform. The description could be a mix of OneSphere and other infrastructure providers already handled by Terraform.</p> <p>Expected outcome: An open source Terraform plugin for HPE OneShere hosted on the HPE Developer Github</p> <h2>Team: Spectrum Spider</h2> <h3>Location: US and Canada</h3> <p>This team of 5 is building an Ansible module for HPE OneSphere. Ansible is a very popular language for describing IT infrastructure. The ecosystem around Ansible (now owned by Red Hat) is huge, with hundreds of modules available to automate just about everything. This project would allow Ansible playbooks authors to include HPE OneSphere in their IT infrastructure automation.</p> <p>Expected outcome: An open source Ansible module for HPE OneSphere hosted on the HPE Developer Github</p> <h2>Team: Last Bench Squad</h2> <h3>Location: India</h3> <p>This team of 5 is using ServiceNow to control HPE OneSphere project and user management. The idea is to build an integration between ServiceNow and HPE OneSphere to allow a ServiceNow user to order a new development team environment in HPE OneSphere from a ServiceNow form following the governance rules already in place.</p> <p>Expected outcome: A importable ServiceNow project on the HPE Developer Github</p> <h2>Team: HPE OneSphere ChatBot</h2> <h3>Location: Fort Collins, CO</h3> <p>This team of 3 is building a Hubot plugin which will allow users to chat in natural language (in Slack, for example) and get answers from HPE OneSphere.</p> <p>Expected outcome: An open source Hubot plugin on the HPE Developer Github</p> <h2>Team: NoName</h2> <h3>Location: Poland</h3> <p>This team of one is made up of a young data scientist who will be building a sample program to demonstrate how to use language R, a programming language and free software environment for statistical computing and graphics, to extract Insights from HPE OneSphere.</p> <p>Expected outcome: A blog article with sample R code</p> <h2>Team: Alesha</h2> <h3>Location: India and Japan</h3> <p>This team of 2 is building a plugin for HPE OneSphere to allow description of infrastructure using Cloud Formation (from AWS).</p> <p>Expected outcome: An open source CloudFormation plugin for HPE OneSphere hosted on the HPE Developer Github</p> <h2>Team: Vizit</h2> <h3>Location: United States</h3> <p>This team of one is building a rendering tool using D3 (a famous open source library) to display HPE OneSphere configurations.</p> <p>Expected outcome: A demo program hosted on the HPE Developer Github</p> <p>HPE Hackathon participants will create a 15 minute presentation about what they have accomplished, which will be presented to the judges on June 4. HPE Hackathon winners will be announced on June 5th.</p> <h1>Stay tuned for more insider access and the results of the HPE Hackathon!</h1>https://developer.hpe.com/psssst-ive-got-the-inside-scoop-on-the-upcoming-hpe-hackathon/https://developer.hpe.com/psssst-ive-got-the-inside-scoop-on-the-upcoming-hpe-hackathon/Tue, 22 May 2018 14:00:54 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/hpe-hackathon-3d-v1-1-1526997723888.jpg" alt="HPE Hackathon"></p> <p>You may not be aware, but something big is about to happen within the Developer Community at HPE. On May 24, HPE developers from around the world will participate in the group’s first hackathon.</p> <p>For those unfamiliar with the term, a hackathon is a coding contest where developers, programmers, and other creative experts come together to collaborate on a range of software development topics. Participants work quickly – often over a short period of time – to innovate and solve challenges, resulting in creating usable software everyone can use.</p> <p>Hackathons are becoming popular and useful activities because they foster innovation through collaboration and creative problem solving. They can also stimulate some interesting results. For instance, <a href="https://hackernoon.com/these-13-new-startups-were-born-at-hackathons-b758c37dde42">this article tells how 13 new startup companies</a> were created during hackathons, which demonstrates the power of harnessing the expertise of a creative group of people.</p> <blockquote> <p><em>“Hackathons let developers go beyond their business-as-usual, day-to-day operations – giving them the opportunity to work with other talented people on a particular challenge over a short timeframe,”</em></p> </blockquote> <p>explains Said A. Syed, Director -- Developer Experience and Product Incubation, HPE Software-Defined and Cloud Group.</p> <blockquote> <p><em>“This type of concentrated brainpower and effort can lead to transformative innovation that produces new and disruptive solutions, capabilities, and products.”</em></p> </blockquote> <h2>HPE Hackathon – focus on HPE OneSphere</h2> <p>The HPE Hackathon will be hosted in Fort Collins, Colorado where developers will attend in person. Others will join remotely from all over the world -- Poland, Brazil, India, Sweden, Ireland, Canada, and other areas of the United States. Forty-six attendees make up nineteen different teams. The teams include up to five people with names such as India Matrix Master, Spectrum Spiders, Last Bench Squad, Kubewarriors, Say What?, and Reality Distortion.</p> <p>For this event, the focus for the hackathon is surrounding <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a> API. A variety of projects are already in the works that will provide the Open Source community with more integration modules for HPE OneSphere, HPE’s new hybrid-cloud management platform.</p> <h2>What’s happening?</h2> <p>Officially, the hackathon started with a two hour training session on the HPE OneSphere API, held on May 10th. Following this training, all attendees were provided an account on a live prepopulated HPE OneSphere platform.</p> <p>May 24-25 is when things really get serious, as coaching and finalizing sessions are available to help participants finish their projects. The following week, participants will create a 15 minute presentation about what they have accomplished, which gets presented to the judges on June 4. Winners will be announced on June 5th. Throughout June HPE experts within the developer community will be working with the teams to publish the open source hackathon code as onto <a href="https://github.com/hewlettpackard/">HPE Github</a>, where it become available to everyone to use and improve upon.</p> <p>Participants on the winning team will each receive a high-end drone. If a team’s project can be open sourced and is pushed to the HPE Github, each team member will receive either an Amazon Echo Dot or Google Home Mini. And all hackathon participants will receive HPE swag (T-shirts and other fun gifts).</p> <h2>Fun collaboration that leads to innovation and benefits customer</h2> <p>Alex Mejias, Senior Fullstack Developer and one of the HPE Hackathon organizers explains the purpose of the event.</p> <blockquote> <p><em>“This hackathon provides a fun and social format to collaborate with HPE subject matter experts to create new ideas around existing software. And most of the projects created during the hackathon will be put through HPE’s Open Source Review Board (OSRB) to be available for free to the public.”</em></p> </blockquote> <p>Another HPE Hackathon organizer, Didier Lalli, Distinguished Technologist at HPE, details some of the projects that are planned.</p> <blockquote> <p><em>“Projects range from front end data visualizations using libraries like D3 and Grafana, to DevOps with Ansible and Terraform, to Chat bots. We wanted to provide a set of topics that use HPE OneSphere’s REST API to its fullest potential. We’re covering the full stack rainbow here; our goal is to push the boundaries on what can be developed.”</em></p> </blockquote> <p>The hackathon organizers have established a permanent <a href="https://www.labs.hpe.com/slack">Slack channel</a> to support live questions from the participants. They’ve also assembled a geographically dispersed team in EMEA and on both the east and west coast of the United States to provide constant support for all of the teams during the event May 24-25.</p> <p>Syed is looking forward to the HPE hackathon and excited about the anticipated benefits for HPE customers.</p> <blockquote> <p><em>“HPE is in a unique position to solve complex problems for our customers. During the hackathon, HPE experts are taking on 19 projects that present unique real-world challenges and working together to solve them. The hackathon is a fun way to bring an amazing group of talented people together to solve some of these challenges quickly.”</em></p> </blockquote> <h2>Good luck to all the participants of the HPE Hackathon!</h2>https://developer.hpe.com/hpe-at-chefconf-2018-may-22-25/https://developer.hpe.com/hpe-at-chefconf-2018-may-22-25/Fri, 18 May 2018 16:10:21 GMT<p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/5/hpe-dev-chef-conf-2018-1526660123064.jpg" alt="Chef Conf"></p> <p>ChefConf is the annual three day gathering of the DevOps and Continuous Automation community. This year, Hewlett Packard Enterprise (HPE) will be on-hand at <a href="http://chefconf.chef.io/">ChefConf 2018</a> in Chicago to discuss the latest and greatest in DevOps workflow, infrastructure, compliance, and application automation.</p> <p>The event, which takes place May 22-25, offers attendees a chance to attend sessions and demos delivered by the leading experts in DevOps, cloud-native development, and IT strategies from around the world. If you are going to the event next week, don’t miss these sessions from HPE experts:</p> <h2><strong>Complacency is a killer; Innovation is not enough; what’s next?</strong></h2> <p><em>Tuesday, May 22nd at 10:30 AM</em></p> <p>“That’s the way we’ve always done it” are the most dangerous words in the English language to businesses. In this session, HPE experts will examine the impact of complacency and how innovation alone is no longer enough in the modern digital age.  Today’s technology savvy consumers and startups are disrupting all industries. Learn how traditional enterprise IT can transform to support ever-changing business demands.</p> <h2><strong>Deploy your applications anywhere safely with Habitat and HPE OneSphere</strong></h2> <p><em>Tuesday, May 22nd, 6:00 PM</em></p> <p><a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a> provides a hybrid cloud management platform that spans public clouds, like AWS, as well as private clouds. In this demo, HPE will show how the HPE OneSphere APIs deploy habitat applications to both public cloud as well as the <a href="https://www.hpe.com/us/en/integrated-systems/synergy.html">HPE Synergy</a> platform running in the data center (without any change to the underlying application).</p>https://developer.hpe.com/hpe-oneview-ruby-sdk-now-available-with-support-for-hpe-oneview-40/https://developer.hpe.com/hpe-oneview-ruby-sdk-now-available-with-support-for-hpe-oneview-40/Wed, 04 Apr 2018 21:08:12 GMT<h1>HPE OneView Ruby SDK now available with support for HPE OneView 4.0</h1> <p>The HPE OneView Ruby SDK provides a Ruby library to interact with HPE OneView and HPE Synergy Image Streamer APIs, enabling developers to easily build integrations and scalable solutions.</p> <p>The HPE OneView Ruby SDK v5.4.0 now supports HPE OneView 4.0 (REST API version 600).</p> <p>The list of supported resources and changes is available at: <a href="https://github.com/HewlettPackard/oneview-sdk-ruby/blob/v5.4.0/CHANGELOG.md">https://github.com/HewlettPackard/oneview-sdk-ruby/blob/v5.4.0/CHANGELOG.md</a></p> <p>Release content is available at: <a href="https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.4.0">https://github.com/HewlettPackard/oneview-sdk-ruby/releases/tag/v5.4.0</a></p> <p>The repository with code and examples is available on GitHub at: <a href="https://github.com/HewlettPackard/oneview-sdk-ruby">https://github.com/HewlettPackard/oneview-sdk-ruby</a></p> <p>Details on API coverage (additional APIs will be included in subsequent releases) can be found at: <a href="https://github.com/HewlettPackard/oneview-sdk-ruby/blob/v5.4.0/endpoints-support.md">https://github.com/HewlettPackard/oneview-sdk-ruby/blob/v5.4.0/endpoints-support.md</a></p> <p>RubyGems details are available at: <a href="https://rubygems.org/gems/oneview-sdk/versions/5.4.0">https://rubygems.org/gems/oneview-sdk/versions/5.4.0</a></p>https://developer.hpe.com/automating-3par-provisioning-with-chef/https://developer.hpe.com/automating-3par-provisioning-with-chef/Mon, 02 Apr 2018 08:46:25 GMT<h1>Using Chef to configure your 3PAR Storeserv</h1> <p>Traditionally storage has been one of the last components to be integrated into Automation. It has been viewed as one of the “untouchables” along with networking due to its complexity and risk potential. With the increased development of APIs, including the 3PAR WSAPI, including storage provisioning tasks into your DevOps project is not only becoming simpler but also a requirement to meet SLAs and policy requirements.</p> <h2>A quick intro to Chef</h2> <p>Chef is a configuration management tool that uses a declarative syntax. It is different than scripting in a few ways, but mainly:</p> <ul> <li>You define what the environment should look like, not how to do it. This removes a lot of complexity, and simplifies your configuration code.</li> <li>Chef code lives primarily in recipes. 1 or more recipes are packaged inside a cookbook that has a single purpose. Cookbooks act like policies that are versioned and checked into source control, unlike many scripts.</li> <li>Chef integrates with a whole host of tools that make testing easy and meaningful. ChefSpec, InSpec, and Test Kitchen are some of these tools. Just like your applications, you can test and release your configuration code using automated pipelines.</li> </ul> <p>Some common reasons you'd want (or need) to use a configuration management tool like Chef are to have success at scale, increase consistency, and gain visibility into your environment. For more in-depth information on Chef and its architecture, refer to <a href="https://docs.chef.io">https://docs.chef.io</a></p> <h2>Developing with Chef and 3PAR</h2> <p>If you're new to Chef, I'd encourage you to go through some of the tutorials and self-paced trainings at <a href="https://learn.chef.io">learn.chef.io</a> before going any further.</p> <p>About the only thing you'll need to get started developing is some basic terminal knowledge and the ChefDK installed. This will give you Ruby, as well as tools such as Berkshelf, Test Kitchen, ChefSpec, Foodcritic, and Rubocop. For this guide, we'll use a Bash terminal; if you're on Windows, you may have to modify some of the system commands accordingly.</p> <h2>Prerequisites</h2> <ul> <li>3PAR OS - 3.2.2 MU4, 3.3.1 MU1</li> <li>Ruby - 2.4.1 or higher</li> <li>Chef - 12.1 or higher</li> <li>WSAPI service should be enabled on the 3PAR storage arrays</li> </ul> <h2>Examples</h2> <p>This section will help you get started using the 3PAR cookbook, and in doing so, explain some key concepts pertaining to Chef and how to use the 3PAR cookbook. It is also designed to show you how to create a basic recipe that includes real, working code and comments about how the resources work.</p> <p>The main README found on <a href="https://github.com/HewlettPackard/hpe3par_chef_cookbook">https://github.com/HewlettPackard/hpe3par_chef_cookbook</a> shows a few details about the resources in this cookbook, but mainly includes code to demonstrate the properties you can set on a resource. As such, many of those code "examples" are incomplete or contain much more than you'd want to use.</p> <ol> <li>To get started, we'll create a new directory named <strong>chef-repo</strong>; this is where we'll put our new cookbooks. Also in this directory, lives the <strong><em>chef/knife.rb</em></strong> configuration file used to connect to a Chef server.</li> </ol> <pre><code class="language-bash">$ mkdir chef-repo $ cd chef-repo </code></pre> <ol start="2"> <li>We’ll create a directory within our <strong><em>chef-repo</em></strong> named cookbooks, where our cookbooks will live:</li> </ol> <pre><code class="language-bash">$ mkdir cookbooks $ cd cookbooks </code></pre> <ol start="3"> <li>Now we will need to download the <strong><em>hpe3par</em></strong> cookbook.</li> </ol> <pre><code class="language-bash"># You can use knife to download the hpe3par cookbook or you can use git – use whichever you are most comfortable with #---------------------------------------- # Using git example # (make sure you are in chef-repo/cookbooks) $ git clone https://github.com/HewlettPackard/hpe3par chef_cookbook.git # We need to rename the directory to hpe3par from hpe3par_chef_cookbook – see note $ mv hpe3par_3par_cookbook hpe3par #---------------------------------------- # Using knife example $ knife cookbook site download hpe3par # Now we will need to untar the download $ tar xvf hpe3par-*.tar.gz # Now the hpe3par cookbook lives in chef-repo/hpe3par # You can remove the tar file if you’d like, it is no longer needed $ rm hpe3par-*.tar.gz </code></pre> <p>      <strong>Note:</strong> If you want to directly use from <a href="https://github.com/HewlettPackard/hpe3par_chef_cookbook">https://github.com/HewlettPackard/hpe3par_chef_cookbook</a>, you need to rename the cookbook directory to <strong><em>hpe3par</em></strong> from <strong><em>hpe3par_cookbook</em></strong> to use the resources.</p> <ol start="4"> <li>Now we can use the chef command to generate a new cookbook named 'my-3par':</li> </ol> <pre><code class="language-bash">$ chef generate cookbook my_3par $ cd my_3par </code></pre> <p>You can examine the files and folders that were created <code>ls -la</code>, the most important ones that we will be working with will be <strong><em>metadata.rb</em></strong> and <strong><em>recipes/default.rb</em></strong>.</p> <ol start="5"> <li> <p>At this point, as noted in the main README, we need to specify a dependency to the <code>hpe3par</code> cookbook. We do this by adding the following line to the end of the <strong><em>metadata.rb</em></strong> file:</p> </li> </ol> <pre><code class="language-bash"> # my_3par/metadata.rb ... depends 'hpe3par' </code></pre> <p>      <strong>Tip:</strong> Check out the <a href="https://docs.chef.io/config_rb_metadata.html">Chef docs</a> to see what all else you can put in the <strong><em>metadata.rb</em></strong> file.</p> <ol start="6"> <li>Now we can create our first recipe. Open up <strong><em>recipes/default.rb</em></strong> and add the following lines:</li> </ol> <pre><code class="language-bash">#Configure Storage system #Enter name, IP, user, and password node.default['hpe3par']['storage_system'] = { name: 'my-3par', ip: '192.168.0.100', user: '3par_user', password: 'secret123' } #Create Virtual Volume ‘my_1st_chef_vol’ definitions hpe3par_virtual_volume 'my_1st_chef_vol' do storage_system node['hpe3par']['storage_system'] cpg node.default['hpe3par']['virtual_volume']['cpg'] = 'FC_r1' size node.default['hpe3par']['virtual_volume']['size'] = 10.0 size_unit node.default['hpe3par']['virtual_volume']['size_unit'] = 'GiB' type node.default['hpe3par']['virtual_volume']['type'] = 'thin' snap_cpg node.default['hpe3par']['virtual_volume']['snap_cpg'] = 'FC_r1' action :create end </code></pre> <p><strong>Tip:</strong> We are defining the 3PAR connectivity settings within the recipe, there may be instances where you have multiple recipes in a cookbook and rather than defining the storage system (or other parameters i.e. virtual volumes, vvsets, etc) within each recipe, we can create global variables by setting parameters in the <strong><em>attributes/default.rb</em></strong> file.</p> <p>For example:</p> <pre><code class="language-ruby"># global storage system - configured in attributes/default.rb default['hpe3par']['storage_system'] = { name: 'my-3par', ip: '192.168.0.100', user: '3par_user', password: 'secret123' } #virtual_volume global parameters default['hpe3par']['virtual_volume']['name'] = 'my_1st_chef_vol' default['hpe3par']['virtual_volume']['cpg'] = 'FC_r1' default['hpe3par']['virtual_volume']['size'] = 1024.0 default['hpe3par']['virtual_volume']['size_unit'] = 'MiB' default['hpe3par']['virtual_volume']['type'] = 'thin' default['hpe3par']['virtual_volume']['compression'] = false default['hpe3par']['virtual_volume']['snap_cpg'] = 'FC_r1' </code></pre> <p>The recipe will change slightly as well. Note the parameters are missing, they are now being defined in the <strong><em>attributes/default.rb</em></strong> file:</p> <pre><code class="language-ruby">#Example recipe using attributes/default.rb defininition file #Create Virtual Volume ‘my_1st_chef_vol’ definitions hpe3par_virtual_volume node['hpe3par']['virtual_volume']['name'] do storage_system node['hpe3par']['storage_system'] cpg node['hpe3par']['virtual_volume']['cpg'] size node['hpe3par']['virtual_volume']['size'] size_unit node['hpe3par']['virtual_volume']['size_unit'] type node['hpe3par']['virtual_volume']['type'] snap_cpg node['hpe3par']['virtual_volume']['snap_cpg'] action :create end </code></pre> <p>To learn more about attributes go here: <a href="https://docs.chef.io/attributes.html">docs.chef.io/attributes</a></p> <ol start="7"> <li>Now let’s run our recipe and see what happens:</li> </ol> <pre><code class="language-bash">$ chef-client -z -o my_3par::default </code></pre> <p><strong>Note:</strong> If you see an error saying it can't find the correct cookbook, this is because Chef either can’t find our cookbooks or we haven't downloaded the cookbook dependencies (<strong><em>hpe3par</em></strong>) yet. We can fix this by creating a <strong><em>knife.rb</em></strong> file at <strong><em>chef-repo/.chef/knife.rb</em></strong> and adding:</p> <pre><code class="language-ruby"># See http://docs.chef.io/config_rb_knife.html for more information on knife configuration options current_dir = File.dirname(__FILE__) cookbook_path ["#{current_dir}/../cookbooks"] # If you're behind a proxy, you'll need to set the http_proxy environment variable or set the http_proxy option here # http_proxy 'http://proxy.company.com:3128' </code></pre> <p>Now download the <strong>hpe3par</strong> cookbook by running:</p> <pre><code class="language-bash"># (make sure we are in chef-repo/cookbooks) $ pwd $ knife cookbook site download hpe3par # Now we will need to untar the download $ tar xvf hpe3par-*.tar.gz # Now the hpe3par cookbook lives in chef-repo/hpe3par # You can remove the tar file if you’d like, it is no longer needed $ rm hpe3par-*.tar.gz </code></pre> <ol start="8"> <li>Now re-run <strong><code>chef-client -z -o my_3par::default</code></strong>. This time it should succeed, and if you log into your SSMC, you'll see the new volume <strong><em>my_1st_chef_vol</em></strong>.</li> </ol> <p>That's it! You can re-run the recipe and it will tell you if the volume already exists.</p> <p>Now that you've gotten your feet wet, please take another look at the main README to see the complete list of resources available to you. The 3PAR cookbook includes many of the same capabilities that are available via the SSMC.</p>https://developer.hpe.com/in-band-management-with-ilorest-and-a-livecd/https://developer.hpe.com/in-band-management-with-ilorest-and-a-livecd/Fri, 30 Mar 2018 11:19:26 GMT<style> li { font-size: 27px; line-height: 33px; max-width: none; } </style> <p>Updated March 4 2024</p> <p>In a <a href="https://developer.hpe.com/blog/in-band-management-with-ilorest-and-a-livecd" target="_blank">previous blog</a>, I explained how the HPE RESTful interface tool (<a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">iLOREST</a>) is able to communicate securely with the underlying iLO and perform in band management of the lower layers of the server. This blog post provides a method to create a custom bootable LiveCD that embeds the HPE ilorest utility and potentially other management tools.</p> <p>After booting the server on the LiveCD, it is possible, locally or from a remote location, to perform in-band management operations.</p> <p>Although the solution presented here is based on Red Hat / CentOS distributions, it can be adapted to other Linux distributions as well as to Windows since the support of iLOrest in WinPE exists since <code>hprest</code> version 1.5 (at that time this utility was still called <code>hprest</code>).</p> <h2>High level solution description</h2> <p>The Red Hat / CentOS ecosystem provides the <code>livecd-tools</code> package for an easy generation of customized bootable media. It contains the <code>livecd-creator</code> utility for the creation of bootable ISO using a classical kickstart configuration file.</p> <p>The basic steps for embedding <code>ilorest</code> in a customized bootable LiveCD are:</p> <ol> <li>Setup of a RHEL / CentOS Media Server configured with the <code>livecd-tools</code> and the <code>createrepo</code> packages. The custom repository containing iLOrest (see next step) will also be hosted on this Media Server</li> <li>Creation of a custom repository on the Media Server holding the iLOrest package (and potentially other packages).</li> <li>Customization of the LiveCD configuration file.</li> <li>Creation of the ISO bootable image.</li> </ol> <p>Once the LiveCD ISO file is created we will describe how to use it:</p> <ol> <li>Presentation of the LiveCD to a managed server as an iLO Virtual Drive.</li> <li>Boot of the managed server on the LiveCD ISO image.</li> <li>Submission of iLOrest in band management commands to the managed server using SSH.</li> </ol> <h2>Media Server setup</h2> <p>The following instructions suppose you are logged in as root in a CentOS or RHEL 7 server with an Internet connection or with the ability to retrieve all required packages mentioned below.</p> <p>The first step in the above list is not fully described in this article; I assume that <a href="https://httpd.apache.org/download.cgi">Apache</a> or another Web server application is installed and configured to provide Linux YUM repositories to other servers on your network.</p> <p>You can obtain the <code>livecd-tools</code> package from the <a href="http://fedoraproject.org/wiki/EPEL" target="_blank">Extra Packages for Enterprise Linux 7</a> (EPEL) repository.</p> <p>On a CentOS server issue:</p> <pre><code>MediaServer# yum epel-release </code></pre> <p>Read the <a href="http://fedoraproject.org/wiki/EPEL">EPEL Wiki</a> for detailed installation instructions if you are using a RHEL Media Server.</p> <p>Install the <code>livecd-tools</code>:</p> <pre><code>MediaServer# yum install livecd-tools </code></pre> <p>If you review the content of this package you will notice that it contains several binary tools and a minimal kickstart file for Fedora:</p> <pre><code>MediaServer# rpm –ql livecd-tools ... /usr/bin/livecd-creator ... /usr/share/doc/livecd-tools-13.4.9/livecd-fedora-minimal.ks </code></pre> <p>We could use this minimal configuration file as a starting point and modify it to create a RHEL or CentOS configuration.</p> <p>Another possibility is to download a public configuration file, more suitable for our RHEL / CentOS Media Server in order to have only a few modifications to perform.</p> <p>If not already done, create a directory that will contain the LiveCD ISO image. Make sure that this directory is HTTP-accessible by the managed server. Then, download the configuration file in this directory:</p> <pre><code>MediaServer# mkdir ISOs MediaServer# cd ISOs MediaServer# wget https://raw.githubusercontent.com/CentOS/sig-core-livemedia/master/kickstarts/centos-7-livecd.cfg </code></pre> <p>We will customize this kickstart file later.</p> <h2>Create a package repository for <code>ilorest</code></h2> <p>If not already done, download the <a href="https://github.com/HewlettPackard/python-redfish-utility/releases/latest" target="_blank">latest RESTful Interface Tool package</a> and store it on your Media Server in a location accessible from the network:</p> <pre><code>MediaServer# ls myrepo ilorest-2.1-73.x86_64.rpm </code></pre> <p>Create your custom iLOrest repository with <code>createrepo(8</code>)and verify that a <code>repodata</code> directory has been created along with the iLOrest package. The <code>createrepo</code> utility is present by default on RHEL / CentOS, so you should not have to install it:</p> <pre><code>MediaServer# createrepo myrepo … MediaServer# ls myrepo ilorest-2.1-73.x86_64.rpm repodata </code></pre> <p>Ultimately, you should verify that this repository can be accessed by other servers on your network. A simple local test could be to retrieve the <code>repomd.xm</code> file using <code>wget</code> or <code>curl</code>:</p> <pre><code>MediaServer# cd /tmp MediaServer# wget http://localhost/&#x3C;Kit-DIR>/myrepo/repodata/repomd.xml ... 2016-07-13 16:06:34 (237 MB/s) - “repomd.xml” saved [2974/2974] </code></pre> <h2>Customization of the LiveCD configuration file</h2> <p>It is now time to edit and customize the RHEL/CentOS LiveCD configuration file. The minimum modifications are:</p> <p>· Allow SSH and HTTP protocols through the firewall</p> <p>· Start service <code>sshd</code></p> <p>· Add the iLOrest repository</p> <p>· Add the iLOrest package</p> <p>· Modify the <code>sshd</code> config file to allow root’s SSH connections with a null password because the LiveCD post configuration script deletes the root password string (<code>passwd -d root</code>).</p> <p>The corresponding modified and added lines are:</p> <pre><code>Host# vi centos-7-livecd.cfg … firewall --enabled --ssh --http … services --enabled=NetworkManager,sshd … repo --name=myrepo --baseurl=http://&#x3C;MediaServerIP>/&#x3C;KIT-DIR>/myrepo/ ... %packages ilorest ... %post echo “Enabling SSH access with null root password” cd /etc/ssh sed -i 's/#\(PermitEmptyPasswords\) no/\1 yes/' sshd_config ... </code></pre> <p>I suggest performing the following additional modifications to reflect your environment location and needs:</p> <pre><code>timezone Europe/Paris --isutc ... skipx # xconfig –startxonboot … # gdm-libs ... # gnome-settings-daemon-updates </code></pre> <h2>Creation of the ISO bootable LiveCD</h2> <p>The last step is to generate the LivCD ISO bootable file:</p> <pre><code>Host# livecd-creator --config centos-7-livecd.cfg </code></pre> <p>The above command generates a bootable <code>livecd-centos-7-livecd-&#x3C;timestamp>.iso</code> image containing an OS and the required packages including iLOrest.</p> <h2>Presenting the LiveCD to a server</h2> <p>To boot the server on this image, we just need to present it as an iLO Virtual Drive and make sure it boots automatically during next reboot. The iLO Web Graphical User Interface can help to perform those tasks in an interactive manner.</p> <p>However, for didactic reasons, we will use the iLOrest interface tool in an Out-Of-Band manner from the Media Server. This tool provides atomic and macro commands for reading and setting parameters in the iLO, the BIOS or other subsystems</p> <p>Moreover, it respects the good practices rules described in the <em>Getting Started with the iLO X Redfish API – A primer for coders</em> articles (<a href="https://sourceforge.net/p/redfish-lab/wiki/Getting-started-with-the-iLO4-Redfish-API/">iLO 4</a>, <a href="https://sourceforge.net/p/redfish-lab/wiki/Getting-started-with-the-iLO5-Redfish-API/">iLO 5</a>]: It does not assume that REST objects are at a specific location, but smartly crawls the entire mesh of REST types and sub-types.</p> <p>Fortunately, iLOrest provides a builtin macro command described in its <a href="https://servermanagementportal.ext.hpe.com/docs/redfishclients/ilorest-userguide/ilocommands/#virtualmedia-command" target="_blank">online documentation</a>.</p> <p>To present the LiveCD to the managed server, the following basics steps will be performed from the Media Server:</p> <ol> <li> <pre><code>Open an iLOrest session with the managed iLO. </code></pre> </li> <li> <pre><code>Retrieve the ID number of the CD/DvD media type (usually 2): </code></pre> </li> <li> <pre><code>Present the LiveCD as a Virtual CD/DVD using the builtin `virtualmedia` command with the `--bootnextreset` option to trigger the next reboot on this Virtual Drive. </code></pre> </li> <li> <pre><code>Close the session </code></pre> </li> </ol> <p>Opening an Out-of-Band management session with iLOrest is very straightforward. You just need to supply the IP address of the managed iLO and the privileged credentials:</p> <pre><code>Host# ilorest login &#x3C;iLO_IP> -u &#x3C;username> -p &#x3C;password> </code></pre> <p>Use the <code>virtualmedia</code> command with no arguments to retrieve the CD/DVD Media ID:</p> <pre><code>Host# export VID=$(ilorest virtualmedia | awk '/CD/ {print $1}' | cut -b2) </code></pre> <p>We are now ready to present the virtual CD/DVD with:</p> <pre><code>Host# ilorest virtualmedia $VID http://&#x3C;MediaServerIP>/livecd-centos-7-livecd-&#x3C;timestamp>.iso --bootnextreset </code></pre> <p>Make sure you logout this iLOrest session:</p> <pre><code>Host# ilorest logout </code></pre> <h2>In-Band Management with iLOrest</h2> <p>Once the managed server is powered on, it boots on the LiveCD and can be accessed via SSH, as root without supplying any password. This gives us the possibility to send iLOrest in-band management commands to the managed server:</p> <pre><code>MediaServer# ssh root@livecd ilorest login MediaServer# ssh root@livecd ilorest types MediaServer# ssh root@livecd ilorest select VirtualMedia. MediaServer# ssh root@livecd ilorest ls --json </code></pre> <p>The above commands can be includes in a shell script file to perform specific actions or review specific BIOS or iLO parameters.</p> <h2>Conclusion</h2> <p>We’ve learned how to create a Linux (RHEL / CentOS) LiveCD embedding the iLOrest RESTful interface tool and enabling in band management of a single managed server. In case you have more than one server to manage, it is also possible to use this solution and to configure all of them the exact same way at once. In this case you would use a tool like the parallel distributed shell (<code>pdsh</code>) or the cluster shell (<code>clush</code>). Look forward to more blogs that will cover topics like <code>pdsh</code> or <code>clush</code>.</p> <p>Don't forget to check out some of my other <a href="https://developer.hpe.com/search/?term=donze" target="_blank">blog posts</a> on the HPE Developer portal to learn more about Redfish tips and tricks.</p>https://developer.hpe.com/how-to-change-the-factory-generated-ilo-administrator-password/https://developer.hpe.com/how-to-change-the-factory-generated-ilo-administrator-password/Thu, 29 Mar 2018 15:34:24 GMT<br> <p><big>This blog post has been moved to the <a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/changefactorypasswd/changefactorypasswd">Server Management Portal </a></big></p> <br>https://developer.hpe.com/the-redfish-event-service/https://developer.hpe.com/the-redfish-event-service/Tue, 27 Mar 2018 15:17:22 GMT<p>This blog post has been moved to the <strong><a href="https://servermanagementportal.ext.hpe.com/docs/references_and_material/blogposts/etc/eventservice/redfisheventservice">Server Management Portal</a></strong>.</p>https://developer.hpe.com/why-we-created-hpe-deep-learning-cookbook/https://developer.hpe.com/why-we-created-hpe-deep-learning-cookbook/Wed, 21 Mar 2018 06:00:26 GMT<h2>A history behind the Cookbook</h2> <p>In the beginning of 2014, when <a href="https://www.labs.hpe.com/">Hewlett Packard Labs</a> (still HP Labs back then, within Hewlett-Packard), embarked on its journey towards Memory Driven Computing and <a href="https://www.labs.hpe.com/the-machine">The Machine</a>, “software” people of Labs have been asked to find good applications where something like The Machine will shine. We started with algorithms and problems which were still challenging to run on existing systems. Monte Carlo simulations, graph inference, search space optimization and deep learning were, among others, on our list. We started to model the performance of these algorithms for different system architectures. How performance will change, if we have more powerful compute nodes or faster interconnect between nodes in the system? For deep learning we soon realized that answers largely depend on a topology of an artificial neural network. An optimal hardware system to train a deep convolutional neural network is not always the best one to train a fully connected model. Depending on a model which you want to train (or to use in production - run inference), you’ll need different hardware to minimize training or inference time. Similar to cooking, where depending on what do you want to cook, you need different ingredients in different proportions.</p> <p>In parallel with our Memory Driven Computing efforts, artificial intelligence and deep learning started to gain interest from enterprise customers. During our interactions with customers, we started to hear questions about the choice of optimal hardware/software environment to run deep learning workloads. How to choose from a sea of available options today? How to size and configure infrastructure? A need for the Cookbook became clear.</p> <p>On our journey towards the Cookbook, we first came up with analytical performance models to predict performance of various machine learning algorithms (including deep learning) depending on compute power and a number of compute nodes in a system, and properties of the interconnect between the compute nodes. These simple models were useful for rough estimates, but didn’t take into account many subtitles of compute systems. We had to get into benchmarking, in addition to analytical modeling, to reason based on real performance data.</p> <h2>Deep Learning Benchmarking Suite</h2> <p>We wanted to be able to collect performance data on various hardware systems with various software and for different deep learning workloads. We wanted the results to be consistent, reproducible and comparable. This means we need to make sure that we run exactly the same workloads on multiple systems. We needed a benchmarking tool. We had several options:</p> <ol> <li> <p>Use existing projects that target the deep learning domain. One of the most recognized by a community are <a href="https://svail">DeepBench</a> from BAIDU and the <a href="https://github.com/soumith/convnet-benchmarks">convnet-benchmarks</a>. These projects aim at benchmarking low-level functionality such as convolution or matrix multiply operations or use simplified models.</p> </li> <li> <p>Use example training scripts that are part of their respective frameworks, something what most companies do. Typical examples are <a href="https://github.com/tensorflow/models/tree/master/research/inception">TensorFlow's Inception​,</a> <a href="https://github.com/apache/incubator-mxnet/tree/master/example/image-classification">MXNet's image classification</a> and <a href="https://github.com/caffe2/caffe2/tree/master/caffe2/python/examples">Caffe2's ResNet50​</a></p> </li> </ol> <p>Unfortunately, none of these options provides what we need - a way to collect performance data for different deep learning workloads (and not only low-level operations) in a consistent and reproducible manner across a range of software and hardware combinations. Thus, we decided to create our own tool - <a href="github.com/HewlettPackard/dlcookbook-dlbs">Deep Learning Benchmarking Suite</a>. It is open sourced and available on github for everyone who wants to run reproducible and consistent deep learning benchmarks.</p> <h2>Deep Learning Performance Guide</h2> <p>We’ve been using Deep Learning Benchmarking Suite internally at HPE for some time already. We’ve collected a variety of performance data on many hardware and software configurations, and we continue to collect more and more data. Now we want to share this data with everyone, so we can guide the choice of optimal hardware and software environment in the open. Deep Learning Performance Guide is a tool to do so. It is a web-based tool connected to our vast database of benchmarking data. We plan to open this tool to the public at the end of March, 2018. The first version of this tool will be based entirely on data from actual benchmarks. In the future we plan to incorporate our analytical performance models into the Performance Guide so it will provide performance estimates for untested hardware/software configurations alongside with real collected performance measurements for tested configurations.</p> <h2>Reference Designs</h2> <p>Reference designs are the last component of our Cookbook toolset. These are default hardware recipes for selected classes of deep learning workloads. So far we have released Image Classification Reference Designs. We created those by collecting performance data (with our Benchmarking Suite) for most common convolutional neural networks (widely used for image classification problems), on multiple hardware and software configurations, and by analyzing the collected data with Performance Guide.</p> <h2>So, what is in the Cookbook?</h2> <p>HPE Deep Learning Cookbook is a toolset to characterize deep learning workloads and to guide the choice of optimal hardware and software configurations for any given workload. It consists of:</p> <ul> <li> <p>Deep Learning Benchmarking Suite - a tool to run deep learning benchmarks</p> </li> <li> <p>Deep Learning Performance Guide - a web-based tool to compare and analyze the results of deep learning benchmarks. In the next version we will integrate into this tool analytical/machine learning models to predict performance of deep learning workloads for situations when we cannot run actual benchmarks. These performance models already exist, we just need to add them to the Performance Guide.</p> </li> <li> <p>Reference Designs - recipes (descriptions) of default recommended hardware configurations for classes of selected deep learning workloads, such as image classification, natural language processing, speech recognition, video analytics, and others.</p> </li> </ul> <p>We created the Cookbook with the following objectives in mind:</p> <ol> <li> <p>To be able to make recommendations to our customers on the optimal hardware/software combinations for running their specific and varied deep learning workloads for both the development (training) and deployment (inference) stages.</p> </li> <li> <p>To be able to run reproducible and consistent deep learning benchmarks in different hardware/software environments that include a range of hardware systems, deep learning frameworks and deep learning workloads. This would be especially useful for performance benchmarking, capacity planning and product qualification.</p> </li> <li> <p>Provide to the community a standard tool that enables apple to apple comparison of various systems.</p> </li> <li> <p>To be able to validate and justify design options for future products.</p> </li> </ol>https://developer.hpe.com/hpe-oneview-ansible-modules-v500-has-been-released/https://developer.hpe.com/hpe-oneview-ansible-modules-v500-has-been-released/Tue, 06 Mar 2018 20:24:23 GMT<h1>HPE OneView Ansible Modules v5.0.0 has been released</h1> <p>The HPE OneView Ansible modules have been updated to support OneView 4.0 (REST API version 600)</p> <p>The Ansible modules automate the provisioning of physical infrastructure on-demand using software-defined templates from HPE OneView.</p> <p>The list of supported resources and changes is available at: <a href="https://github.com/HewlettPackard/oneview-ansible/blob/v5.0.0/CHANGELOG.md">https://github.com/HewlettPackard/oneview-ansible/blob/v5.0.0/CHANGELOG.md</a></p> <p>Release content is available at: <a href="https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.0.0">https://github.com/HewlettPackard/oneview-ansible/releases/tag/v5.0.0</a></p> <p>The repository where code and examples exist is available on GitHub at: <a href="https://github.com/HewlettPackard/oneview-ansible">https://github.com/HewlettPackard/oneview-ansible</a></p> <p>Details about the Ansible modules, parameters, example usage, etc.: <a href="https://github.com/HewlettPackard/oneview-ansible/blob/master/oneview-ansible.md">https://github.com/HewlettPackard/oneview-ansible/blob/master/oneview-ansible.md</a></p> <p>Additional information about using Ansible with HPE OneView can be found in the Ansible section at: <a href="https://developer.hpe.com/platform/hpe-oneview/home">https://developer.hpe.com/platform/hpe-oneview/home</a></p>https://developer.hpe.com/hpe-oneview-python-sdk-v450-has-been-released/https://developer.hpe.com/hpe-oneview-python-sdk-v450-has-been-released/Tue, 06 Mar 2018 19:56:14 GMT<h1>HPE OneView Python SDK v4.5.0 has been released</h1> <p>The HPE OneView python SDK has been updated to support OneView 4.0 (REST API version 600).</p> <p>The SDK allows python developers to programmatically control HPE OneView managed resources using an infrastructure-as-code approach for physical compute, storage, and fabric resources.</p> <p>The list of supported resources and changes is available at: <a href="https://github.com/HewlettPackard/python-hpOneView/blob/v4.5.0/CHANGELOG.md">https://github.com/HewlettPackard/python-hpOneView/blob/v4.5.0/CHANGELOG.md</a></p> <p>Release content is available at: <a href="https://github.com/HewlettPackard/python-hpOneView/releases/tag/v4.5.0">https://github.com/HewlettPackard/python-hpOneView/releases/tag/v4.5.0</a></p> <p>The repository where code and examples exist is available on GitHub at: <a href="https://github.com/HewlettPackard/python-hpOneView">https://github.com/HewlettPackard/python-hpOneView</a></p> <p>Details on API coverage (additional APIs will be included in subsequent releases): <a href="https://github.com/HewlettPackard/python-hpOneView/blob/master/endpoints-support.md">https://github.com/HewlettPackard/python-hpOneView/blob/master/endpoints-support.md</a></p> <p>PyPi details are available at: <a href="https://pypi.python.org/pypi/hpOneView/4.5.0">https://pypi.python.org/pypi/hpOneView/4.5.0</a></p>https://developer.hpe.com/creating-a-python-version-that-enforces-fips/https://developer.hpe.com/creating-a-python-version-that-enforces-fips/Thu, 15 Feb 2018 18:04:25 GMT<h1><strong>Creating a Python version that enforces FIPS</strong></h1> <p>Version 2.2 and later of our python-redfish-utility can enforce the FIPS mode of the operating system it is used on. If an OS is in FIPS mode, but uses a non-FIPS encryption algorithm, Python will crash. If used with a FIPS-validated module such as the OpenSSL FIPS module, a project can be FIPS-compliant. By default, Python does not ship with a FIPS version of OpenSSL, so we must build Python from source to meet the OpenSSL FIPS requirement. In this blog post, we cover building Python 2.7 from source with a FIPS version of OpenSSL in Windows and Linux. In addition, we provide instructions for adding extra functions to the Python SSL library to set and get the OpenSSL FIPS mode. <strong>Patching Python with our FIPS mode functions</strong></p> <p>Before building Python from source, we must add extra functions so we can get and set OpenSSL FIPS mode. To add this functionality, we must change the <code>_ssl.c</code> and <code>ssl.py</code> source files. A patch file that adds the required functions is available [here] (<a href="https://github.com/HewlettPackard/python-redfish-utility/tree/master/patches">https://github.com/HewlettPackard/python-redfish-utility/tree/master/patches</a>)</p> <p><strong>Building Python in Linux</strong></p> <p>When you build Python from source, it uses the version of OpenSSL on the local system. If you already have a FIPS version of OpenSSL that you are comfortable with, you can use that. If not, you must update OpenSSL. You can also install an alternate version of OpenSSL and use it with some modifications to the Python setup files. Use an OpenSSL version with shared objects, because that is what Python uses to build its <code>pyd</code> and <code>lib</code> files.</p> <p>Once you have the OpenSSL you want on the system, you can install Python with the required modifications. If you are using an OpenSSL FIPS from <code>/usr/local/ssl/</code>, you do not need to modify the setup files. If you are using an alternate installation of OpenSSL or one in a different location, changes are required. You must configure Python with the location of the OpenSSL <code>include</code> and <code>lib</code> files in the <code>setup.py</code> file. Additionally, you must configure Python with the OpenSSL location, and uncomment code in the `Modules/Setup.dist file. The following example shows a patch file with the required changes. The file locations differ based on the location of the OpenSSL version you want to use.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/7-1518719833337.png" alt="7"></p> <p>After the required changes are complete (if any), we can build Python just as we normally would. Once Python is successfully installed, you can use the extra SSL functions to enable FIPS mode in OpenSSL. <strong>Building Python in Windows</strong></p> <p>Windows does not come with a version of OpenSSL, which complicates things. Each installation of Python on Windows comes with its own non-FIPS version of OpenSSL, so we will use our own. To make things even more interesting, Python 2.7 only has Visual Studio projects up to 9.0 (Visual Studio 2008 Professional). The easiest way to get the required runtimes is to build using this version of Visual Studio. If using this version is not an option, and you need a more recent version, there are resources available to describe how to do that.</p> <p>To begin, we must build a FIPS version of OpenSSL in Windows. Make sure that you build OpenSSL with shared objects, because that is what Python uses to build its pyd and lib files. Documentation is available online if you need assistance building OpenSSL with the FIPS module.</p> <p>After a FIPS version of OpenSSL is built on the system, we can start building Python. Before building Python, a few changes in the Python source are required. The first change is in the pyproject.vsprops file located in the <code>PC\(VS version)\</code> location of the Python source code. We must update the location of OpenSSL to our new version. In the following example, you can see that Visual Studio looks for an OpenSSL folder from the externalsDir, and the externalsDir is two folders above from our current location named externals. This folder does not exist by default, so we must create it and add our OpenSSL folder.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/8-1518719845111.png" alt="8"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/9-1518719854112.png" alt="9"></p> <p>Next, change the location to our new OpenSSL.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/10-1518719873936.png" alt="10"></p> <p>The next files to change are the _<code>hashlib.vcproj</code> and <code>_ssl.vcproj</code> files, located in the same directory, for building the <code>_hashlib.pyd</code> and <code>_ssl.pyd</code> files. We must update both files with the locations we compile and link to when building. Find the configuration you intend to build python with in each file, and navigate to the <strong>VCPreBuildEventTool</strong> entry. The following example shows the Release|x64 configuration. Since we already built OpenSSL, we do not need to run the build_ssl.py script, so you can remove the <strong>CommandLine</strong> entry.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/11-1518719882829.png" alt="11"></p> <p>Since we are not building using build_ssl.py, the libraries and include files might not be in the location Python is expecting. To fix this issue, change the <strong>VCCLCompilerTool</strong> entry. This entry must point to the <code>include</code> folder of your built OpenSSL.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/12-1518719933237.png" alt="12"></p> <p>We also must change the <strong>VCLinkerTool</strong> entry to our new library locations.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/13-1518720063593.png" alt="13"></p> <p>After you make these changes, you should be able to successfully build Python, <code>_ssl.pyd</code>, and <code>_hashlib.pyd</code>. You might get errors building other extensions, but solving those issues is beyond the scope of this blog. If you want to update only <code>_ssl.pyd</code> and <code>_hashlib.pyd</code>, there is a shortcut you can take. Instead of building all of Python, right-click on both the <strong>_hashlib</strong> project and <strong>_ssl</strong> project respectively, and then select <strong>Build</strong> in Visual Studio.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/14-1518720073858.png" alt="14"></p> <p>This action will provide the <code>pyd</code> files that you need. Install Python by using the installer from the Python website, and then replace the <code>_ssl.pyd</code> and <code>_hashlib.pyd</code> files in the <code>DLLs</code> folder with your new versions.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/15-1518720081531.png" alt="15"></p> <p>You also must replace the <code>ssl.py</code> file in the <code>Lib</code> folder with your new version.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/16-1518720090640.png" alt="16"></p> <p><strong>Using our FIPS mode functions</strong></p> <p>Now that we have our new FIPS version of <code>_ssl</code> and <code>_hashlib</code>, we can make sure that they work correctly in the Python shell. Importing ssl and checking the OpenSSL version, we can confirm that it has been updated with a FIPS version of SSL.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/17-1518720097685.png" alt="17"></p> <p>We can now try one of our newly added functions. Let’s check the <strong>FIPS mode</strong> of OpenSSL using our first function, FIPS_mode. This function returns a long type 0 for not enforcing FIPS, and 1 for enforcing FIPS. It is important to note that whether or not the OS is in a FIPS mode, when we start Python, OpenSSL is not enforcing FIPS mode.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/18-1518720288533.png" alt="18"></p> <p><strong>FIPS_mode</strong> returned a 0, so we are not enforcing FIPS. Using a hash that is not FIPS-compliant will work.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/19-1518720279167.png" alt="19"></p> <p>Let’s set Python’s OpenSSL to enforce FIPS mode, by using our second new function <strong>FIPS_mode_set</strong> and passing it a long value of 1. After we set it, let’s verify that OpenSSL is enforcing FIPS by using the previous function again.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/20-1518720318403.png" alt="20"></p> <p>Now that we are enforcing FIPS mode in OpenSSL, if we try to use a hash that is not FIPS-compliant Python will crash with the following error:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/21-1518720331286.png" alt="21"></p> <p>If you want to enforce FIPS, make sure that you use OpenSSL hash functions, and not Python hash functions. This means using the <strong>new</strong> constructor instead of directly calling a hash function that Python already knows.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/22-1518720342935.png" alt="22"></p> <p>Now that Python is updated to use a FIPS version of OpenSSL, and we included extra functions to set OpenSSL to enforce FIPS, we can make sure that we are using only FIPS algorithms and hashes. If you want to enforce the OS FIPS mode, check for specific Environment Variables or Registries. For examples, see our GitHub for the <a href="https://github.com/HewlettPackard/python-redfish-utility">python-redfish-utility</a>.</p>https://developer.hpe.com/updating-python-and-openssl-on-os-x/https://developer.hpe.com/updating-python-and-openssl-on-os-x/Mon, 18 Dec 2017 18:53:11 GMT<h1><strong>Updating Python and Openssl on OS X</strong></h1> <p>Security is a vital part of any server. On our Enterprise servers we provide options for higher levels of security. However, Mac computers ship with an older version of Openssl, a vital component for our python-redfish-library. In this blog we will cover updating Openssl to at least 1.0.0 to allow support for our python-redfish-library. <strong>Higher Security Settings</strong></p> <p>“High Security" is one of the Higher Security levels offered on our Gen10 servers. When “High Security” or other higher security settings are enabled, Openssl must be used to connect to the server. Below is a screenshot where these security settings can be changed. It can be found under Security > Encryption.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/12/1-1513693864107.jpg" alt="openssl1"></p> <p><strong>Running Examples without Openssl</strong></p> <p>Here we show an example of an attempt to execute an example script on a Mac without updated Openssl.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/12/2-1513693937238.png" alt="openssl2"></p> <p>Note in the Terminal we are utilizing Python 2.7.11 as well as OpenSSL 0.9.8zg. Here we attempted to execute the first example, ex01_get_resource_directory.py. Since our version is outdated, we have an error in attempting to connect.</p> <p><strong>Installing Homebrew</strong></p> <p>To perform our upgrades, we will use Homebrew. Homebrew is a package manager for macOS. We will be using it to install the most recent version (at least v1.0) of Openssl, and then use that version to install a newer version of Python.</p> <p>Homebrew installs into <code>/usr/local/</code> so it must be accessible to the user.</p> <p><code>sudo chown -R $(whoami) /usr/local</code></p> <p>The above command will give the user permissions to install to <code>/usr/local/</code> To install Homebrew: <code>/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install</code></p> <p>Paste this command into your console, and Homebrew will install itself. Note that this should not be done with sudo.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/3-1518714501022.png" alt="ssl3"></p> <p>Upon installing Homebrew, the first thing we want to do is update it.</p> <p><code>brew update</code></p> <p>Then we want to install openssl <code>brew install openssl</code></p> <p>If you’re behind a firewall and use a proxy, add <code>ALL_PROXY=proxy</code> before your command:</p> <p><code>ALL_PROXY=socks5://127.0.0.1:9001 brew install openssl</code></p> <p>Substitute <code>socks5://127.0.0.1:9001 </code> with your own proxy. This command can be used for both the update and install commands.</p> <p><strong>Linking Openssl and Installing Python</strong></p> <p>We can attempt to create symlinks through homebrew for Openssl with the following command:</p> <p><code>brew link openssl --force</code></p> <p>This first command may work in certain versions of Homebrew, but recent updates have changed how Homebrew processes openssl installation. Alternatively, we can create these symlinks manually with the following commands:</p> <p><code>ln -s /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib /usr/local/lib/</code></p> <p><code>ln -s /usr/local/opt/openssl/lib/libssl.1.0.0.dylib /usr/local/lib/</code></p> <p>Now that the symlinks have been created, we can install Python:</p> <p><code>brew install python --with-brewed-openssl</code></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/4-1518714723500.png" alt="ssl4"></p> <p>Upon completion, we will have Python installed with the new version of Openssl. In the example below, we have opened the new version of Python, as well as checked the version of Openssl, which shows as 1.0.2j.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/5-1518716804698.png" alt="ssl5.2"></p> <p><strong>Setting the new Python as default</strong></p> <p>The newly installed Python is different from the system Python that comes prepackaged on the Mac. We need to ensure that it becomes the preferred Python installation. We can use the which python command to find which installation is being used. Often, the default directory for system Python is:</p> <p><code>usr/bin/python2.7</code></p> <p>Homebrew’s Python would be installed under something like: <code>usr/local/bin/python2.7</code></p> <p>In order to fix this, we simply edit our etc/paths file. We want to ensure that <code>/usr/local/</code> is at the top of the list to ensure that version of Python is used first. See example paths file below:</p> <p><code>/usr/local/bin</code> <code>/usr/local/sbin</code> <code>/usr/bin</code> <code>/bin</code> <code>/usr/sbin</code> <code>/sbin</code></p> <p>Note that we do not want to remove the system Python, since some Mac services rely on the system version of Python. Instead, we simply want to redirect it to use our new version of Python.</p> <p><strong>Running the new Python with Openssl</strong></p> <p>Now that we’ve updated Openssl, and integrated it with Python, we can use this new version of Python to utilize our python-redfish-library. Note that pip install may need to be run to reinstall the redfish library. (pip install python-redfish-library) In the example below, IDLE was used to run the first example again:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2018/2/6-1518717614095.png" alt="6.2ssl"></p> <p>As you can see, our Python has since been updated to the newest version as of writing, 2.7.12. Additionally, the example runs successfully instead of failing.</p> <p>Now that we’ve successfully updated Openssl and Python on our Mac, we can now utilize the python-redfish-library to its full extent in managing remote servers via the iLO RESTful API.</p>https://developer.hpe.com/doryd-a-dynamic-provisioner-for-docker-volume-plugins/https://developer.hpe.com/doryd-a-dynamic-provisioner-for-docker-volume-plugins/Wed, 06 Dec 2017 00:39:08 GMT<p>Many vendors have invested quite a lot of effort into building robust Docker Volume plugins, including HPE Nimble Storage and HPE 3PAR. Docker and Kubernetes (K8s) share the same fundamental principle of being able to bind mount a host filesystem inside a container through the <code>mnt</code> namespace. Any other similarity pretty much stops there as far as compatibility goes. K8s has a multitude of Persistent Volume (PV) plugins mainly focused on IaaS provider frameworks and native filesystems.</p> <p>The FlexVolume plugin is part of K8s and allows vendors such as HPE to provide out-of-tree FlexVolume drivers. What is important to understand here is that the FlexVolume plugin only supports static provisioning. A K8s cluster administrator must create PVs manually for K8s users to create Persistent Volume Claims (PVC) against. This might seem a bit tedious but is still magnitudes more gracious than creating a volume on the backend array manually and present it to the cluster (don't forget to those IQNs or WWNs to the target when you expand your cluster!) if you would use the iSCSI or FC plugin for K8s. Although, I would argue this is good enough if you only have a handful of applications that require PVs on a fairly static cluster without any need for advanced data services or deny users to dynamically create PVs based on their needs.</p> <p>Storage Classes (SC) allows dynamic provisioning of PVs based on PVCs created by a user. This feature has been around since K8s 1.2 and was promoted to beta in 1.4. SCs allows a cluster administrator to define named classes with certain attributes, such as which provisioner to use, default PV plugin parameters and so on. SCs enables policy-based storage management which essentially abstracts storage minutia for the user. This blog post will discuss this concept in great detail.</p> <p>Our story around Dory: The FlexVolume driver that speaks whale, would not be complete without a dynamic provisioner as cluster administrators have better things to do than provision PVs to their users. A provisioner is a very simple daemon that listens for PVCs and satisfies those claims based on the defined SCs.</p> <p>The rest of this blog post walks through the steps on how to get started with Dory and Doryd. Just replace <code>nimble</code> in the examples below with your particular Docker Volume plugin to get started.</p> <h1>Dory: A FlexVolume Recap</h1> <p>Dory is an <a href="https://github.com/hpe-storage/dory">open source project available on GitHub</a>. The latest instructions on how to build and install Dory will always be available in the <a href="https://github.com/hpe-storage/dory/blob/master/docs/dory/README.md">README.md</a>, steps vary between distributions. I've covered this <a href="https://community.hpe.com/t5/HPE-Nimble-Storage-Tech-Blog/Dory-A-FlexVolume-Driver-that-speaks-Whale/ba-p/6986638">in the past</a> but it did not include Doryd and the naming schemes required for Doryd.</p> <p>I currently have an Ubuntu 16.04.3 machine in front of me with K8s 1.8.4 and the latest Nimble Linux Toolkit (NLT) installed but I choose to install the binary instead. NLT contains the HPE Nimble Storage Docker Volume plugin and the rest of the guide assumes all hosts have the Docker Volume plugin installed and working.</p> <pre><code>sudo mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/dev.hpe.com~nimble sudo curl -sLo /usr/libexec/kubernetes/kubelet-plugins/volume/exec/dev.hpe.com~nimble/nimble \ https://dl.bintray.com/hpe-storage/dory/dory-master sudo chmod 755 /usr/libexec/kubernetes/kubelet-plugins/volume/exec/dev.hpe.com~nimble/nimble </code></pre> <p><strong>Note:</strong> What is being made available to the kubelet is a FlexVolume driver referenced as <code>dev.hpe.com/nimble</code>. Changing <code>dev.hpe.com</code> to something else will break the stock provisioner we're going to use in the examples below.</p> <p>In the same directory where you placed the <code>dory</code> binary, in my case <code>dev.hpe.com~nimble</code>, you need to copy <code>dory.json</code> file from the Dory repository to <code>dev.hpe.com~nimble/nimble.json</code>.</p> <pre><code>sudo curl -sLo /usr/libexec/kubernetes/kubelet-plugins/volume/exec/dev.hpe.com~nimble/nimble.json \ https://raw.githubusercontent.com/hpe-storage/dory/master/src/nimblestorage/cmd/dory/dory.json </code></pre> <p>The key here is to identify where the socket file to your Docker Volume plugin accepts API calls. Since this can vary between plugins and distributions as well, please <code>curl</code> an API call to the socket file you intend to use. For example, in my case:</p> <pre><code>sudo curl -XPOST --unix-socket /run/docker/plugins/nimble.sock http:/Plugin.Activate {"Implements":["VolumeDriver"]} </code></pre> <p><strong>Note:</strong> <code>curl --unix-socket</code> is a fairly new thing for <code>curl</code> and could be missing from your distribution.</p> <p>Adjust your <code>dory.json</code> accordingly. Full documentation for each key is available in <a href="https://github.com/hpe-storage/dory/blob/master/docs/dory/README.md#building">the Dory repo</a>. In most cases, only <code>dockerVolumePluginSocketPath</code> needs to be changed.</p> <pre><code class="language-JSON">{ "logFilePath": "/var/log/dory.log", "logDebug": false, "stripK8sFromOptions": true, "dockerVolumePluginSocketPath": "/run/docker/plugins/nimble.sock", "createVolumes": true, "enable1.6": false, "listOfStorageResourceOptions": [ "size", "sizeInGiB" ], "factorForConversion": 1073741824 } </code></pre> <p><strong>Note:</strong> Neither Dory or Doryd cares if Docker is installed on the host or not. As long as there is a daemon responding to Docker Volume API calls, you're good. In other words, Dory and Doryd should work just fine with other container engines compatible with K8s, such as <code>rkt</code> or <code>cri-o</code>.</p> <p>Another gotcha worth mentioning, if you intend to only create PVs against the FlexVolume driver. PVs are not created until a pod or deployment requests an attachment of the PV. In other words, just creating the PV and PVC won't create the Docker Volume itself.</p> <p><strong>Important:</strong> If you're using K8s &#x3C; 1.8, The kubelet needs to be restarted to pick up the installed FlexVolume driver. This step may be different depending on how you run your kubelet, on Ubuntu 16.04 where the cluster has been installed with <code>kubeadm</code>, simply:</p> <pre><code>sudo systemctl restart kubelet </code></pre> <p>Watch <code>/var/log/dory.log</code> for initialization:</p> <pre><code>tail /var/log/dory.log Info : 2017/12/02 20:50:31 dory.go:55: [5202] entry : Driver=nimble Version=1.1.0-d95ad289 Socket=/run/docker/plugins/nimble.sock Overridden=true Info : 2017/12/02 20:50:31 dory.go:58: [5202] request: init [] Info : 2017/12/02 20:50:31 dory.go:68: [5202] reply : init []: {"status":"Success","capabilities":{"attach":false}} </code></pre> <p>We're now ready to move on to Doryd!</p> <h1>Kubekuddle this!</h1> <p>Conveniently enough, <code>doryd</code> runs as a DaemonSet (DS), a DS ensures the provisioner runs (and keep running) on all the cluster nodes. You're given the option to build the <code>doryd</code> image and specify your own DS specification, however, the following specification is known to work under most circumstances:</p> <pre><code>kubectl apply -f https://raw.githubusercontent.com/hpe-storage/dory/985f0313440cd2181c27e5d94b31f9fe75714c3f/examples/ds-doryd.yaml </code></pre> <p>You may inspect the DS on the cluster by running:</p> <pre><code>kubectl describe ds/doryd </code></pre> <p>You'll see something similar to this:</p> <pre><code>Name: doryd Selector: daemon=dory-daemon Node-Selector: &#x3C;none> Labels: daemon=dory-daemon Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"DaemonSet","metadata":{"annotations":{},"name":"doryd","namespace":"default"},"spec":{"template":{"metadata"... Desired Number of Nodes Scheduled: 1 Current Number of Nodes Scheduled: 1 Number of Nodes Scheduled with Up-to-date Pods: 1 Number of Nodes Scheduled with Available Pods: 1 Number of Nodes Misscheduled: 0 Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: daemon=dory-daemon Containers: dory: Image: nimblestorage/doryd Port: &#x3C;none> Environment: &#x3C;none> Mounts: /etc/kubernetes from k8s (rw) /run/docker/plugins/ from dockersocket (rw) /usr/libexec/kubernetes/kubelet-plugins/volume/exec from flexvolumedriver (rw) Volumes: k8s: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/ flexvolumedriver: Type: HostPath (bare host directory volume) Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec dockersocket: Type: HostPath (bare host directory volume) Path: /run/docker/plugins/ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 1m daemon-set Created pod: doryd-6m52b </code></pre> <p>That's it! We're ready to create some Storage Classes!</p> <h1>Defining and consuming Storage Classes</h1> <p>Specifications for SCs are defined by the cluster admin. Parameters that may be specific for your particular environment is the <code>provisioner</code> in the class. The particular parameters you want to be passed to down the underlying Docker Volume plugin has its own block as well. HPE Nimble Storage, in the below example, have a ton of different parameters but let's keep it lean and simple for these particular examples.</p> <p>A popular pattern throughout the history of storage has been the notion of categorizing storage into gold, silver and bronze "tiers" where each tier has certain performance, reliability or availability characteristics attached to each tier. At this time, HPE Nimble Storage goes from fast to faster, the reliability and availability is the same across the portfolio and we can't use this traditional tier paradigm. Instead, let's optimize each class for a certain workload and capacity. SCs named "transactionaldb", "unstructuredfile" and "securearchive" would be more descriptive examples.</p> <p>HPE Nimble Storage uses Predictive Analytics to look across the install base to help optimize Performance Policies, such as block size, compression, caching and quota behavior. There's also the concept of Protection Templates where a storage administrator may define snapshot schedules, retention and replication to either another HPE Nimble Storage array or to the public cloud (Amazon AWS/Azure) via <a href="https://www.hpe.com/us/en/storage/cloud-volumes.html">HPE Cloud Volumes</a>.</p> <p>The below YAML files are available in the <a href="https://github.com/NimbleStorage/container-examples/tree/master/dory/doryd-intro">container-examples repository</a> on GitHub.</p> <h2>transactionaldb</h2> <p>An example SC optimized for a transactional workload on an all-flash pool where we would attach a stock Protection Template.</p> <pre><code class="language-JSON">--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: transactionaldb provisioner: dev.hpe.com/nimble parameters: description: "Volume provisioned by doryd from transactionaldb StorageClass" perfPolicy: "SQL Server" protectionTemplate: "Retain-48Hourly-30Daily-52Weekly" </code></pre> <p>Create the SC with:</p> <pre><code>kubectl create -f https://raw.githubusercontent.com/NimbleStorage/container-examples/master/dory/doryd-intro/sc-transactionaldb.yml </code></pre> <p>This SC may then be referenced as such:</p> <pre><code class="language-JSON">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mariadb-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 16Gi storageClassName: transactionaldb </code></pre> <p>Let's create the PVC and walk through what we can expect:</p> <pre><code>kubectl create -f https://raw.githubusercontent.com/NimbleStorage/container-examples/master/dory/doryd-intro/pvc-example.yml </code></pre> <p>A subtle difference between using SCs and the FlexVolume driver directly, is that Docker Volumes actually gets created upon creating the PVC. Example inspection:</p> <pre><code>$ kubectl get pvc/example-claim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-claim Bound transactionaldb-8db1a469-d7f3-11e7-8f86-000c291bed2c 16Gi RWO transactionaldb 9m $ kubectl get pv/transactionaldb-8db1a469-d7f3-11e7-8f86-000c291bed2c NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE transactionaldb-8db1a469-d7f3-11e7-8f86-000c291bed2c 16Gi RWO Delete Bound default/example-claim transactionaldb 10m $ docker volume inspect transactionaldb-8db1a469-d7f3-11e7-8f86-000c291bed2c --format '{{.Driver }} {{ .Name }}' nimble transactionaldb-8db1a469-d7f3-11e7-8f86-000c291bed2c </code></pre> <p>That particular PVC could then be referenced from a pod or deployment to have the volume dynamically attached upon request. A complete example that creates a MariaDB deployment with a <code>transactionaldb</code> SC is available in the example repo. For completeness:</p> <pre><code>kubectl create -f https://raw.githubusercontent.com/NimbleStorage/container-examples/master/dory/doryd-intro/mariadb.yml </code></pre> <p>The below examples require your array to have certain resources setup and pre-configured and but discussed in the context of Storage Classes.</p> <h2>unstructuredfile</h2> <p>Optimizing for a traditional file/web server dishing out fairly static content, with another stock Protection Template on a hybrid flash pool named <code>hybridflash</code>.</p> <pre><code class="language-JSON">--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: unstructuredfile provisioner: dev.hpe.com/nimble parameters: description: "Volume provisioned by doryd from unstructuredfile StorageClass" perfPolicy: "Windows File Server" protectionTemplate: "Retain-30Daily" pool: hybridflash </code></pre> <h2>securearchive</h2> <p>For PVs requiring a secure destination for archival data, you could potentially have an array in the group that has more stringent security measures and setup for long-term storage and protection. The below example would provision PVs in a performance restricted folder (a HPE Nimble Storage concept of collectively restricting bandwidth, IOPS and capacity to a group of volumes). A custom Performance Policy would deny caching of volumesand set a block size of 32KiB. Volumes are also encrypted at rest.</p> <pre><code class="language-JSON">--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: securearchive provisioner: dev.hpe.com/nimble parameters: description: "Volume provisioned by doryd from securearchive StorageClass" perfPolicy: "Archive" protectionTemplate: "Retain-90Daily-130Biweekly" encryption: "true" folder: "ElephantStorage" </code></pre> <h1>Advanced use cases for HPE Nimble Storage</h1> <p>One of the most useful features offered by the HPE Nimble Storage Docker Volume plugin is to create Zero-Copy Clones and Ephemeral Clones from any given snapshot or the current representation of the data in the volume. While this is something we're exploring with customers and prospects, I just want to give a short glimpse of how powerful these interfaces are.</p> <h2>Zero-Copy Clone from Production</h2> <p>It's not too uncommon for developers wanting a sandbox copy of the latest production database to develop against and refresh at will. I refrain from using the term "<a href="http://searchstorage.techtarget.com/definition/copy-data-management-CDM">copy data management</a>" as it means a lot of different things depending on your background but it's essentially exactly what is if you don't care about the anonymization (which you can do for yourself or some databases has built in with RBAC).</p> <p>Assuming there is a production pod spawned that is actively using <code>transactionaldb-26063d7f-d7f2-11e7-8f86-000c291bed2c</code>. The cluster administrator would then create an SC along these lines:</p> <pre><code class="language-JSON">--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: from-production provisioner: dev.hpe.com/nimble parameters: description: "Clone from production database." cloneOf: "transactionaldb-26063d7f-d7f2-11e7-8f86-000c291bed2c" snapshot: "nightly-locktables" </code></pre> <p>The developer may then create/delete PVCs against that particular SC to refresh his data from the <code>nightly-locktables</code> snapshot.</p> <pre><code class="language-JSON">--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: clone spec: accessModes: - ReadWriteOnce resources: requests: storage: 0Gi storageClassName: from-production </code></pre> <p>The volume being created, could be inspected as such:</p> <pre><code>$ docker volume inspect from-production-dc10ebf3-d7f8-11e7-8f86-000c291bed2c --format '{{ .Status.Parent }} {{ .Status.ParentSnapshot }}' transactionaldb-26063d7f-d7f2-11e7-8f86-000c291bed2c.kubernetes nightly-locktables </code></pre> <p><strong>Note:</strong> The keys used in the custom output filter above are HPE Nimble Storage specific.</p> <p>Assuming that there is particular column that needs to be scrambled before a developer gain access to it, how would you work around that? The cluster administrator would create a FlexVolume PV (no SC) based on the parameters above, run a batch pod that scrambled the column and later created a SC based on that processed cloned. It's really that simple!</p> <h2>Ephemeral Clones for ETL and CI/CD pipelines</h2> <p>In the case where a temporary view is needed of a particular PV, be it for ETL (Extract, Transform, Load) or CI/CD (Continuous Integration/Continuous Deployment/Delivery) pipelines it becomes extremely powerful to create ephemeral representations of PVs. Think a container that has terabytes of data in it ready for processing without having to ship the data in a container image.</p> <p>This SC will recreate the underlying Docker Volume for each attachment:</p> <pre><code class="language-JSON">--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: ephemeral provisioner: dev.hpe.com/nimble parameters: description: "Instant Ephemeral Clone from production database." cloneOf: "transactionaldb-26063d7f-d7f2-11e7-8f86-000c291bed2c" createSnapshot: "true" destroyOnDetach: "true" </code></pre> <p><strong>Note</strong>: PVs and PVCs will remain intact and does not need to be manually deleted. Just reschedule the pod (or delete/create) and the data will be refreshed from the source. This differs from the developer use case in the sense that a developer might need to redeploy against the dataset he modified himself and not necessarily re-create the entire stack upon re-deploy. Also, <code>detroyOnDetach</code> will destroy the dependent snapshot and will keep the production volume clean from residual snapshots that eventually will reference deleted data from the parent volume.</p> <p>This is just the tip of the iceberg. If you have a lot of data that need to be represented in different stages for your containers, HPE has the solution!</p> <h1>Start today!</h1> <p>HPE is working feverishly to release an officially supported product of Dory and Doryd exclusively for HPE Nimble Storage and HPE 3PAR arrays and go through the hoops of certifying the productized version against K8s and Red Hat OpenShift (a popular Enterprise K8s PaaS). Nothing prevents anyone to use the bits built from the Dory repository with any other vendor's Docker Volume plugin, it was the original intent, to translate the Docker Volume API into something K8s may consume. Please submit any issues you may find through GitHub and we'll get it straighten out. Always make sure to check for <a href="https://github.com/hpe-storage/dory/blob/master/docs/plugins/README.md">tested Docker Volume plugins</a> and please submit pull requests against that page to help us keep track of what works!</p> <p><a href="https://developer.hpe.com">HPE DEV</a> is a developer community that is launching to foster HPE related open source software, projects and integrations with contributions not only from HPE but we invite anyone who wants to collaborate on our projects. It's still early days and we just about launched the portal which serves as an umbrella across GitHub repos, Puppet Forge, Ansible Galaxy and of course this blog.</p> <p>We also have a Slack channel that is open to the public, please register <a href="https://www.labs.hpe.com/slack">here</a> and join the conversation. I'm <code>michaelm</code> and you'll find me in #NimbleStorage #Kubernetes and #Docker. KubeCon is just around the corner and don't be shy to stop by the HPE DEV booth to have a chat about <a href="https://github.com/hpe-storage/dory">Dory</a>, <a href="https://www.hpe.com/us/en/solutions/cloud/hybrid-it-management.html">HPE OneSphere</a> or <a href="https://developer.hpe.com">HPE DEV</a> itself (there will be schwag!).</p>https://developer.hpe.com/join-us-at-kubecon-2017/https://developer.hpe.com/join-us-at-kubecon-2017/Wed, 29 Nov 2017 19:10:21 GMT<h1>HPE is at KubeCon</h1> <p>KubeCon 2017 is here! This time Linux Foundation is organizing both Kubecon and CloudNativeCon together in Austin Convention Center, Austin TX December 6-8. KubeCon + CloudNativeCon gathers all CNCF (Cloud Native Computing Foundation) projects under one roof. Leading vendors and technologists from open source cloud native communities are attending KubeCon 2017 to further the advancement of cloud native computing. Hewlett Packard Enterprise (HPE) has realigned its business focus to make hybrid IT simple. It wants to help customers create a hybrid IT model that streamlines day-to-day IT operations and accelerate their digital transformation. HPE's major initiatives and recent product launches such as the new HPE Developer Community Portal, HPE OneSphere, HPE Synergy, HPE SimpliVity HyperConverged solutions, HPE InfoSight predictive analytics, and other solutions reinforce that vision towards its support for Cloud Native Computing.</p> <p>Given HPE’s commitment to Cloud Native Computing and specifically advancement and integration of Kubernetes and other container solutions in our product and services, HPE has joined CNCF as a silver member and will be participating in KubeCon as a sponsor. At KubeCon2017 kicking off today, HPE will be highlighting:</p> <ol> <li>The new HPE Developer Community Portal</li> <li>HPE OneSphere – a SaaS based Multi-Cloud Management solution</li> <li>HPE’s contribution to Kubernetes Project Dory</li> </ol> <p>HPE Developer Community Portal is being announced at KubeCon 2017. The new HPE Developer Community Portal is a place where members can interact with HPE’s new open-source community and create innovative solutions to complex problems by collaborating with like-minded coders. Go to <a href="http://bit.ly/2AWshlw">http://bit.ly/2AWshlw</a> and join the conversation</p> <p>HPE OneSphere is an as-a-service multi-cloud management platform that simplifies management of multi-cloud environments and on-premises infrastructure. Through a unified view, IT can compose hybrid clouds capable of supporting both traditional and cloud-native applications and offers cost analytics across on-premises and off-premises cloud consumption. It provides a simple VM and Container Vending capability allowing developers to focus on developing solutions instead of worrying about deploying VMs and containers.</p> <p>HPE Project Dory is an Open Source project led by HPE. HPE Dory is an Open Source FlexVolume Driver that allows use of any storage to persist data for Kubernetes. Dory works with any storage that has a Docker Volume Driver. It manages data with Kubernetes and platforms such as OpenShift and dynamically provision persistent volumes with Kubernetes storage classes.</p> <p>Stop by HPE Booth #S9 in KubeCon 2017 to learn more about these solutions, to have some interesting conversations, view demos and to win some raffle prizes</p>https://developer.hpe.com/apps-and-infrastructure-as-code-with-ansible-using-hpe-cloud-volumes-and/https://developer.hpe.com/apps-and-infrastructure-as-code-with-ansible-using-hpe-cloud-volumes-and/Wed, 29 Nov 2017 09:18:49 GMT<p><a href="https://cloudvolumes.hpe.com">HPE Cloud Volumes</a> (formerly <a href="https://www.nimblestorage.com/cloud/">Nimble Cloud Volumes</a>) introduces the exciting capability of consuming HPE Nimble Storage as a service in the public cloud. We recently came out of beta and we're introducing new features constantly. We just published our <a href="https://docs.cloudvolumes.hpe.com/help/rest/api-reference">REST API for HPE Cloud Volumes</a> and all of a sudden open up a plethora of new use cases for the service. One of them is the ability to include HPE Cloud Volumes when managing applications and infrastructure as code as most cloud-native shops do today. Also, we added basic HPE Cloud Volumes functionality to the <a href="https://galaxy.ansible.com/NimbleStorage/Ansinimble/">HPE Nimble Storage Ansible role</a>. The following tutorial will walk through deploying <a href="https://www.docker.com/community-edition">Docker CE</a> on HPE Cloud Volumes and deploy a distributed application on top of Docker Swarm using <a href="https://www.ansible.com">Ansible</a>.</p> <h1>Introduction</h1> <p>As my day job is primarily in the container space I can't be more excited of being able to consume HPE Nimble Storage as a service in the public cloud. While we don't have any container specific integration at this point (please see Future below), such as a Docker Volume plugin, there are still some unique benefits we bring to the container use case over traditional EBS (<a href="https://aws.amazon.com/ebs/">Elastic Block Storage</a>) from AWS.</p> <p>In essence, I'm going to relocate the Docker host <code>/var/lib/docker</code> filesystem to a volume hosted by HPE Cloud Volumes. At a glance, these are the immediate realizations:</p> <ul> <li>Millions of times more reliable block storage over EBS</li> <li>Instant snapshots, clones and restore without having to copy objects to S3</li> <li>Clone volumes to any cloud, not just AWS</li> <li>Instant resize of block devices and immediately being able to expand the filesystem</li> <li>Run multicloud Docker Swarm clusters with a single pane of management for storage consumption</li> <li>More than twice the random performance and granularly being able to dynamically set IOPS limits</li> </ul> <p>I think these are very compelling reasons why you want to host your applications on HPE Cloud Volumes for better reliability, performance and data services.</p> <p>We also support replication of on-premises HPE Nimble Storage arrays to HPE Cloud Volumes for data migration and Hybrid IT use cases. This functionality will be covered and included in our Ansible role at a later date. Specifically, how to replicate any on-premises volume and clone it into a container to run workloads.</p> <h1>Assumptions</h1> <p>Certain assumptions are being made and are not covered in depth and a bit out of scope:</p> <ul> <li><a href="https://www.ansible.com">Ansible</a> is an IT automation platform that lets users define and manipulate the state of apps and infrastructure in simple to read <a href="https://en.wikipedia.org/wiki/YAML">YAML</a> files. This blog post is not the right forum to get started with Ansible but it's used here merely as an example.</li> <li>Following along these examples requires a HPE Cloud Volumes account with automation setup for Amazon AWS as explained in <a href="https://cloudvolumes.hpe.com/ncv-help/automating/">the HPE Cloud Volumes documentation</a>. This includes setting up an IAM role for HPE Cloud Volumes.</li> <li>The EC2 security group attached to the VPC requires SSH access from the Ansible host network and access to port 9000.</li> <li>We're deploying <a href="https://minio.io">Minio</a> in distributed mode. Minio is a standalone AWS S3 compatible storage server. Some knowledge of AWS S3 is helpful but not required.</li> </ul> <h1>Overview</h1> <p>This stack diagram illustrate our end-state to help better understand what we're deploying.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/11/blog-diagram-hpecv-1511946990044.png" alt="Architecture Overview"></p> <h1>Setup</h1> <p>The example goes from spinning up eight EC2 instances to consume HPE Cloud Volumes, install Docker, setup a Docker Swarm, deploy <a href="https://www.minio.io">Minio</a>/<a href="https://www.nginx.com">NGINX</a> and setup an ELB (<a href="https://aws.amazon.com/elasticloadbalancing/">Elastic Load Balancer</a>) for external facing traffic. Some utility playbooks shows off some of the Ansible role features and capabilities such as expanding storage and setting new IOPS limits. There's also a playbook provided to destroy all the resources created during the deployment phase.</p> <p><strong>DISCLAIMER</strong>: While I've gone through great length to ensure resources are tagged appropriately I do not check for conflicting tags that may have unforeseen side effects when connecting and deleting resources.</p> <h2>git clone and setup basic parameters</h2> <p>All playbooks and roles are hosted on <a href="https://github.com/NimbleStorage/automation-examples">GitHub</a>. The Ansible host uses Ubuntu 16.04.3 in these examples but any Linux distro will do, please adjust the install procedure accordingly.</p> <p>Prepare Ansible host (Ubuntu 16.04.3):</p> <pre><code>sudo apt-get update sudo apt-get install -y git python python-boto python-boto3 python-jmespath python-pip sudo pip install --upgrade pyopenssl ansible git clone https://github.com/NimbleStorage/automation-examples cd automation-examples/cloud/varlibdocker ansible-galaxy install -r galaxy.txt </code></pre> <p><strong>Important:</strong> The rest of this guide assumes current working directory <code>automation-examples/cloud/varlibdocker</code></p> <p>Copy <code>group_vars/all/main.yml-dist</code> and <code>group_vars/all/secrets.yml-dist</code> to their respective basenames:</p> <pre><code>for i in group_vars/all/*-dist; do mv ${i} $(sed -e 's/-dist$//' &#x3C;&#x3C;&#x3C; ${i});done </code></pre> <p>Edit <code>group_vars/all/main.yml</code> to fit your AWS environment:</p> <pre><code class="language-YAML">--- # Configure these swarm_region: us-west-1 # This is the region you're provisioning in swarm_key: myawskey # This is the named IAM key to use for your EC2 instances swarm_subnet: subnet-00000000 # The subnet ID your HPE Cloud Volumes will be provisioned to swarm_cidr: "0.0.0.0/0" # The CIDR of the above subnet swarm_cloud: vpc-00000000 # The VPC the HPE Cloud Volumes will be provisioned to </code></pre> <p>There are a few other preference parameters to look at, like instance and cluster sizes. These are not necessary to tune.</p> <p>Edit <code>group_vars/all/secrets.yml</code> and store your HPE Cloud Volumes credentials:</p> <pre><code class="language-YAML">--- cloud_portal_access_key: nimble # Your HPE Cloud Volumes key cloud_portal_access_secret: nimblestorage # Your HPE Cloud Volumes secret </code></pre> <p><strong>Note:</strong> While not required, it's highly recommended to protect credentials with Ansible Vault.</p> <p>The next few steps require you to download and install the named AWS key that is referenced in the <code>swarm_key</code> variable and name it <code>ec2.pem</code>:</p> <pre><code>scp user@host:mykey.pem ec2.pem chmod 0400 ec2.pem </code></pre> <p>Next, we rely on the latest <code>ec2.py</code> dynamic inventory script. Download accordingly:</p> <pre><code>wget -O contrib/inventory/ec2.py https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py chmod 755 contrib/inventory/ec2.py </code></pre> <p>Both the the <code>ec2.py</code> inventory script and the Ansible modules relies on environment variables with your access key, secret key and AWS region you're working with:</p> <pre><code>set +o history export AWS_ACCESS_KEY_ID=nimble export AWS_SECRET_ACCESS_KEY=nimblestorage export AWS_REGION=us-west-1 set -o history </code></pre> <p>If you have instances running in EC2 under your account. Now is a good time to see if your credentials are good:</p> <pre><code>./contrib/inventory/ec2.py --list </code></pre> <p>This should eventually list your EC2 instances' external IP addresses in various groupings based on tags, instance names, internal names and a very handy list of variables made available to the instance with an <code>ec2_</code> prefix.</p> <p><strong>Note:</strong> Building the EC2 dynamic inventory seems to take excessive amount of time on occasion. Please be patient.</p> <h1>Deploy!</h1> <p>The next few steps discuss the operation of the playbooks provided. Please ensure that environment variables are properly set as described above and that the current working directory is <code>storage-automation/cloud/varlibdocker</code>.</p> <h2>site.yml</h2> <p>As per convention, <code>site.yml</code> sets up the environment end-to-end and after execution you should have a Minio environment set up at the ELB URL presented at the end of the play.</p> <p>Execute as follows:</p> <pre><code>ansible-playbook site.yml </code></pre> <p>Four main plays are executed and divided as follows:</p> <h3>deploy.yml</h3> <p>This playbook deploys the instances as specified in <code>group_vars/all/main.yml</code>. Since the base AMI is the official vanilla Ubuntu from Canonical a few extra bootstrap steps are required to make the instances eligible for Ansible automation.</p> <p>The play also creates a Cloud Volume for each instance and attaches it on <code>/var/lib/docker</code>. It uses most defaults from the <code>NimbleStorage.Ansinimble</code> role which is a 10GiB volume capped at 1000 IOPS and presented on the networks specified in <code>group_vars/all/main.yml</code>. You should see eight volumes appear on the HPE Cloud Volumes portal with the naming convention <code>varlibdocker-</code>.</p> <p>The Cloud Volume is then mapped and connected to the instances with the <code>ncvol</code> utility which is downloaded and installed from the HPE Cloud Volumes portal as part of the play itself.</p> <h3>prepare.yml</h3> <p>This play installs Docker from the official repos supplied by Docker, Inc. Plain and simple.</p> <h3>formation.yml</h3> <p>One of the master nodes is tagged as the pet (or master manager, depending how you see it) and the pet node will initialize the Docker Swarm and join the remaining managers and workers to the cluster.</p> <h3>app_deploy.yml</h3> <p>Since the Docker integration with Ansible is a bit behind, bare <code>docker</code> commands are run on the pet node to deploy the latest official Docker Swarm Minio example <a href="https://raw.githubusercontent.com/minio/minio/master/docs/orchestration/docker-swarm/docker-compose-secrets.yaml">provided by Minio</a>.</p> <p>What is important to understand here is that Minio uses locally named volumes in this example and data will be locked to wherever the distributed Minio cluster is instantiated. Therefore, intentionally, I put a constraint on the service post-deploy to lock the services to whichever node they were first started on. Minio will survive loss of half the nodes in distributed mode due to its erasure coding but Docker Swarm has no means to guarantee any two services will be placed correctly and you'll in most cases end up with a broken Minio at the next service update or restart. Highly undesirable. To allow services to roam freely in the cluster a shared filesystem or an external Docker Volume plugin is required. We discuss the latter in the Future section below.</p> <p>A trained eye will also see that there's an NGINX global service being deployed. This is because of a long-standing <a href="https://forums.aws.amazon.com/thread.jspa?threadID=33085">issue with ELB</a> and the NGINX trick is simply used to workaround that particular issue (the executive version: you can't have multiple backend ports). The <code>Dockerfile</code> and <code>nginx.conf</code> file is included in <code>files/nginx_lb</code>. The image being pulled is from my personal Docker Hub account, please enjoy!</p> <p>The last step deploys an ELB that fronts the NGINX server. The ELB is an unencrypted TCP load-balancer.</p> <h2>Hello World!</h2> <p>If the <code>site.yml</code> play completes successfully, you should see a URL at the end of the play. This is where your Minio instance is running. Please proceed to login with username <code>nimble</code> and password <code>nimblestorage</code>.</p> <p>Just before the profile summary and play recap:</p> <pre><code> TASK [deploy_elb : debug] *************************************************************** Wednesday 22 November 2017 08:10:24 +0000 (0:00:01.599) 0:12:08.675 **** ok: [localhost] => { "msg": "http://minio-271113730.us-west-1.elb.amazonaws.com:9000" } PLAY RECAP ****************************************************************************** 13.56.224.202 : ok=73 changed=18 unreachable=0 failed=0 13.56.248.32 : ok=73 changed=18 unreachable=0 failed=0 13.57.10.167 : ok=73 changed=18 unreachable=0 failed=0 52.53.127.122 : ok=73 changed=18 unreachable=0 failed=0 52.53.220.8 : ok=73 changed=18 unreachable=0 failed=0 54.193.104.48 : ok=73 changed=18 unreachable=0 failed=0 54.215.225.69 : ok=73 changed=18 unreachable=0 failed=0 54.241.130.88 : ok=88 changed=31 unreachable=0 failed=0 localhost : ok=23 changed=11 unreachable=0 failed=0 Wednesday 22 November 2017 08:10:24 +0000 (0:00:00.031) 0:12:08.706 ************* ======================================================================================== deploy_minio : ...and add constraints ------------------------------------------ 180.81s Get Ansible working ------------------------------------------------------------- 28.61s deploy_ec2 : Launch Workers ----------------------------------------------------- 28.20s deploy_ec2 : Launch Managers ---------------------------------------------------- 22.56s install_docker : Install Docker ------------------------------------------------- 18.54s Refresh inventory --------------------------------------------------------------- 15.24s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.69s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.62s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.62s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.54s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.51s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.51s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.45s NimbleStorage.Ansinimble : Install prerequisite packages ------------------------ 12.41s install_docker : Add Docker repo ------------------------------------------------- 9.44s Validate Docker ------------------------------------------------------------------ 7.51s deploy_minio : command ----------------------------------------------------------- 6.59s NimbleStorage.Ansinimble : Connect Cloud Volume ---------------------------------- 5.03s NimbleStorage.Ansinimble : Connect Cloud Volume ---------------------------------- 4.86s NimbleStorage.Ansinimble : Connect Cloud Volume ---------------------------------- 4.86s Playbook run took 0 days, 0 hours, 12 minutes, 8 seconds </code></pre> <p>Use a web browser to visit that URL or skip directly to the <code>mc</code> part below the screenshots:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/11/minio-login-1511947049263.png" alt="Minio Login"></p> <p>Once logged you'll see something that resembles this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/11/minio-empty-1511947029503.png" alt="Minio Empty"></p> <p>Here you can either create buckets and upload files or you can simply use the <code>mc</code> utility to access Minio:</p> <pre><code>sudo wget -O /usr/local/bin/mc wget https://dl.minio.io/client/mc/release/linux-amd64/mc sudo chmod 755 /usr/local/bin/mc mc config host add mys3 http://minio-271113730.us-west-1.elb.amazonaws.com:9000 nimble nimblestorage </code></pre> <p>With <code>mc</code> you can perform normal unix-like commands. Let's create a bucket and upload this repo's roles directory:</p> <pre><code>mc mb mys3/mybucket mc cp -r roles mys3/mybucket mc ls mys3/mybucket </code></pre> <blockquote> <p>From an operational perspective, what we've done here is quite powerful. Everything is provisioned from human-readable YAML files that can be revision controlled and peer reviewed. Nothing is manual. Installing all the applications and infrastructure manually is just tedious and error-prone and not repeatable. This is Applications and Infrastructure as Code.</p> </blockquote> <h2>Expand and boost performance</h2> <p>Included in the repository is a couple of utility playbooks. One expands the capacity and the other sets a new IOPS limit. Both have a fairly low conservative default limit and passing extra variables to the play is recommended.</p> <p>Before expanding the volumes, load up the Minio landing page that indicates how much space there is. After the play has finished, reload the page. You should see the extra space accordingly.</p> <p>Expand Cloud Volume:</p> <pre><code>ansible-playbook util_expand_cloud_volume.yml -e '{"cloud_volume_size": 1000000}' </code></pre> <p><strong>Note:</strong> Extra variables such as <code>-e</code> can normally be passed as key/value pairs without JSON syntax. However, the REST APIs have strict data types and Jinja2 (the template engine Ansible relies on) converts to plain strings when not passing parameters in JSON.</p> <p>You should have similar space as observed in the screenshots above if you didn't tinker with the default volume sizes. The new capacity is immediately available to Minio and should look something like this (if you didn't tinker with the cluster size):</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/11/minio-resized-1511947069528.png" alt="Minio Resized"></p> <p>While it's entirely possible to write a playbook that takes both capacity and IOPS in one go, it's more intuitive to perform controlled updates in isolated increments.</p> <p>Set a new IOPS limit:</p> <pre><code>ansible-playbook util_iops_cloud_volume.yml -e '{"cloud_volume_iops": 10000 }' </code></pre> <p>The new limit and capacity should be reflected in the HPE Cloud Volumes portal:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/11/portal-1511947086096.png" alt="Portal"></p> <h2>Destroy</h2> <p>Once done playing in the sandbox, the following playbook destroys <strong>everything</strong> provisioned. Including your HPE Cloud Volumes, ELB and EC2 instances.</p> <pre><code>ansible-playbook util_stack_destroy.yml </code></pre> <h1>Future</h1> <p>The future looks brighter than ever when it comes to HPE Cloud Volumes and specialized host application integrations such as containers. HPE Nimble Storage has a strong vision that we're currently executing against that will gradually enable more use cases, especially for DevOps. We recently introduced replication to HPE Cloud Volumes, that in conjunction with our on-premises HPE Nimble Storage Docker Volume plugin and the upcoming HPE Nimble Storage Docker Volume plugin for HPE Cloud Volumes will allow for some advanced use cases for multicloud persistent storage. What about running CI/CD pipelines on cloned datasets or present data to ETL workflows and Hybrid IT deployments with multicloud strategies not even remotely possible with the constraints in traditional environments.</p> <p>Running persistent storage in the public cloud for containers does not come without challenges. Docker Volumes for Docker, Persistent Volumes for Kubernetes or using DVDI to patch in a volume for Marathon with Mesos. They're all subject to the same <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html">inherit limitations</a> of EBS (or <a href="https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#virtual-machine-disk-limits">Azure Disk Storage</a>). Assessing the storage market there's a number of vendors out there trying to combat the persistent storage on EBS paradigm by abstracting the block devices (with or without filesystems) to container native storage elements and create resiliency in software by replication or mirroring schemes. Storage management becomes part of the application domain and compute cycles and N-way replication schemes becomes part of the end-users accounting along with any software licenses schemes imposed by the vendor. There are of course very viable open source options that are free to use but ties more compexity to the back of the application owner who at the end of the day just wanted some persistent storage for his application.</p> <p>We hope to address a number of shortcomings with our persistent storage drivers and plugins for the top three container orchestrators and leverage the native storage frameworks provided without any side-effects.</p> <h1>Conclusions</h1> <p>Thank you for following this tutorial and hopefully it has been useful and sparked interest and ideas about Containers, Hybrid IT and multicloud. Wherever you are on your journey, we're very eager to hear what challenges exist out there today, particularly when it comes to presenting data across public cloud boundaries and data intense use cases. I genuinely believe HPE can help out in a multitude of ways.</p> <p>Please join our Slack community, it's just starting up, <a href="https://www.labs.hpe.com/slack">register here and say hello</a>, I'm user <code>michaelm</code>. Follow <a href="https://twitter.com/drajen">me</a> and <a href="https://twitter.com/HPE_Developer">HPE DEV</a> on Twitter!</p>https://developer.hpe.com/cpus-for-gpu-enabled-deep-learning/https://developer.hpe.com/cpus-for-gpu-enabled-deep-learning/Sun, 26 Nov 2017 09:48:58 GMT<h1>A role of CPUs in Deep Learning pipelines and how many CPU cores is enough for training on a GPU-enabled system</h1> <h2>How CPUs are typically used in deep learning pipelines</h2> <p>Although GPUs are the main engine today used to train deep neural networks, training is not possible without CPUs. And not only because a CPU is required to manage GPU kernels. It has other tasks to perform.</p> <h3>Preprocessing</h3> <p>A common use for CPUs in Deep Learning pipelines is to perform data preprocessing. There are several reasons why preprocessing should be done on CPUs.</p> <ul> <li>CPUs provide the ability to overlap data preprocessing with training. In this case two operations happens simultaneously: 1) preparing data for a next batch (on CPUs), and 2) optimizing a neural network on a current batch (on GPUs). The main goal is to constantly keep GPUs busy crunching numbers. We want GPUs to be focused on training and not waiting for the next batch of samples to be ready.</li> <li>An opportunity to stage the data feeding pipeline with CPUs, if data is stored in a high latency storage. With CPUs the pipelined process looks as follows (see reference below for TensorFlow high performance models that introduced this schema): <ul> <li>Stage 1: Copy data from database or remote file system into a host memory;</li> <li>Stage 2: Preprocess data on CPUs and store result in host memory;</li> <li>Stage 3: Copy preprocessed data from host memory to GPU memory.</li> </ul> </li> <li>If raw training data is large and needs to be transformed during the preprocessing (for image classification, this may mean resizing original high resolution images to a standard <em>256x256</em> resolution), doing it on GPUs has the following implications: <ul> <li>Copying large volumes of data from host memory to GPU memory via PCIe lanes;</li> <li>Allocating large amount of expensive GPU memory for data preprocessing, stealing it from training operators.</li> </ul> </li> <li>For preprocessing on GPUs there should exist efficient GPU implementation of preprocessing operations</li> <li>Some frameworks like TensorFlow, if doing preprocessing on GPUs: <ul> <li>Will first copy data from host memory to GPU memory;</li> <li>Will do preprocessing on GPU;</li> <li>Will copy preprocessed data back to host memory;</li> <li>Will copy again preprocessed data to GPU memory.</li> </ul> </li> <li>CPUs are a better fit for certain data transformation tasks than GPUs.</li> </ul> <h2>How many cores/CPUs do we need?</h2> <p>How do we decide how many CPUs (and how many cores per CPU) do we need to run a training job on a GPU-enabled system?</p> <ul> <li> <p>One factor to consider is the memory bandwidth. The optimal number of CPU cores is the number of cores that saturate memory bandwidth. If we have more cores, they start to fight for memory access slowing down the entire preprocessing pipeline.</p> </li> <li> <p>The topology of a neural network and its computational profile also influences the choice of CPU cores. Deeper models with higher computational complexity take a longer time for a single forward/backward pass to complete, leaving enough time for a CPU to prepare the next batch of training samples. Neural network models with lower computational complexity may benefit from more CPU cores, as it will speed up the preprocessing and prevent the GPUs from being stalled while waiting for the next batch of data.</p> </li> <li> <p>If data preprocessing is overlapped with forward/backward passes, we need at least two CPU cores for a one-GPU system. One core is responsible for launching CUDA kernels for each layer/operator in forward/backward passes. The second core is responsible for prefetching data for the next batch.</p> </li> </ul> <p>Let's now discuss some of the options in greater details.</p> <h3>Option 1: No CPU/GPU overlapping.</h3> <p>The simplest approach is to not overlap preprocessing with training. We first fetch data from data store, preprocess it, copy it to GPU, compute gradients and update weights. We then continue to fetch data for a next batch. Can this be efficient? It depends on where data is stored and on the computational complexity of the model. Several observations to take into account:</p> <ul> <li> <p>If a model is computationally expensive, data feeding may take a relatively small time compared to forward/backward passes on a GPU. In this case the difference in time between overlapped and sequential processing may be small and may not worth the trouble of an overlapped option.</p> </li> <li> <p>If data is stored on a low latency storage with a high bandwidth interconnect, data fetching may not be an issue either.</p> </li> <li> <p>If data is already preprocessed, and data fetching is fast enough, again, one might skip overlapping.</p> </li> </ul> <h3>Option 2: Overlapping CPU/GPU with non-pipelined preprocessing</h3> <p>In general case, we need one CPU core per GPU to run CUDA kernels and <i>M</i> cores to do prefetching/preprocessing. <em>M</em> will depend on the complexity of preprocessing and the complexity of the model. If we want to copy data asynchronously, we need an additional CPU core per GPU for that.</p> <h3>Option 3: Overlapping CPU/GPU with pipelined preprocessing</h3> <p>Google in a blog post on high performance TensorFlow models suggests that it makes sense to use pipelined (staged) preprocessing routines. In particular, they suggest having three running stages:</p> <ul> <li>Fetching data from remote storage into host local memory</li> <li>Preprocessing data in host local memory</li> <li>Copying data from host memory to GPU memory</li> </ul> <p>The three stages communicate to each other via queues. In this case, we need at least three CPU cores for preprocessing.</p> <h2>Data preprocessing with popular frameworks</h2> <h3>Caffe</h3> <p>Caffe's default data layers (written in C++) can have one background thread to prefetch data. They can prefetch up to <i>prefetch_count</i> batches. That prefetching thread is responsible for fetching data and preprocessing it. It then copies data to GPU in asychnronous mode. Having multiple prefetched batches will result in larger memory consumption. This basically means that we need just two CPU cores per GPU. But you can write your own, non-standard data layer that will take advantage of multiple prefetch threads.</p> <h3>TensorFlow</h3> <p>A great post on efficient data feeding is the [TensorFlow high performance models] (<a href="https://www.tensorflow.org/performance/performance_models">https://www.tensorflow.org/performance/performance_models</a>). In theory, we can utilize as many cores as we have.</p>https://developer.hpe.com/scaling-deep-learning-workloads/https://developer.hpe.com/scaling-deep-learning-workloads/Sun, 26 Nov 2017 09:38:36 GMT<h1>Data parallelism, weak and strong scaling, and what you need to know to scale a single training job to multiple CPUs or GPUs</h1> <p>Training of many state-of-the-art deep neural networks is a very compute-intensive task. It can take hours, days or even weeks to train a model with a single computational device, such as a CPU or GPU. To speed up the training, you have to scale out, to distribute the computations to multiple devices. The most commonly used approach to distribute the training is <code>data parallelism</code>, when every computational device possesses its own replica of a model and computes a model update based on its own shard of data. Two options are possible for data parallelism: <code>Strong</code> and <code>weak</code> scaling. HPE's <a href="https://hewlettpackard.github.io/dlcookbook-dlbs">Deep Learning Benchmarking Suite</a> supports both.</p> <h4><strong>Strong scaling</strong></h4> <p>Strong scaling assumes that the problem size remains the same and we vary the number of computational devices. From a deep learning point of view it means that we fix a batch size and vary a number of CPUs/GPUs to train a neural network. For instance, given a batch size of 1024 training samples, an eight-GPU system will give us the following benchmarking configurations:</p> <table> <thead> <tr> <th>#GPUs</th> <th>GPUs IDs</th> <th>per-GPU batch size</th> <th>effective batch size</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>0</td> <td>1024</td> <td>1024</td> </tr> <tr> <td>2</td> <td>0,1</td> <td>512</td> <td>1024</td> </tr> <tr> <td>4</td> <td>0,1,2,3</td> <td>256</td> <td>1024</td> </tr> <tr> <td>8</td> <td>0,1,2,3,4,5,6,7</td> <td>128</td> <td>1024</td> </tr> </tbody> </table> <p>The efficiency of strong scaling is calculated in the following manner:</p> <pre><code>efficiency = t1 / (N * tN) * 100% </code></pre> <p>Where <em>t1</em> is the time of solving a problem with one compute device, <em>N</em> is the number of compute devices, and <em>tN</em> is the time to solve the same problem with these N devices. For instance, an ideal case is when the <em>tN</em> time is <em>N</em> times smaller than <em>t1</em> and we get ideal efficiency of 100%.</p> <p>With strong scaling, an <code>effective batch size</code> is constant, and per-device batch size is varying. In case of synchronous training, strong scaling with multiple devices is equivalent to training with a single device. The effective batch will be the same, and model updates will happen based on gradients computed for the same number of training samples.</p> <h4><strong>Weak scaling</strong></h4> <p>Weak scaling assumes we keep the amount of work per compute device per iteration fixed. With <em>N</em> compute devices we end up solving <em>N</em> times larger problem than with one. In deep learning world it means we keep per-device batch size fixed. With the same hardware setup as it is described above, we get the following benchmarking configurations assuming that the per GPU batch size is 128:</p> <table> <thead> <tr> <th>#GPUs</th> <th>GPUs IDs</th> <th>per-GPU batch size</th> <th>effective batch size</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>0</td> <td>128</td> <td>128</td> </tr> <tr> <td>2</td> <td>0,1</td> <td>128</td> <td>256</td> </tr> <tr> <td>4</td> <td>0,1,2,3</td> <td>128</td> <td>512</td> </tr> <tr> <td>8</td> <td>0,1,2,3,4,5,6,7</td> <td>128</td> <td>1024</td> </tr> </tbody> </table> <p>The efficiency of weak scaling is calculated in the following way:</p> <pre><code>efficiency = t1 / tN * 100% </code></pre> <p>Where <em>t1</em> is the time of solving a problem with one compute device and <em>tN</em> is the time to solve the <em>N</em> times larger problem with <em>N</em> compute devices. Ideally the <em>tN</em> time is exactly the same as <em>t1</em> and we get ideal efficiency of 100%.</p> <p>With weak scaling, a per-device batch size is constant and the <code>effective batch size</code> increases with the assignment of more compute devices to the training job.</p> <p>It is much easier to utilize a larger number of compute devices efficiently with weak scaling, as the amount of work per unit doesn't decreases when more units are added. At the same time, weak scaling may lead to very large effective batches, when convergence will suffer, and it won't be faster after all.</p> <h2>Multi-GPU benchmarking with DLBS</h2> <p>All supported frameworks (BVLC/NVIDIA Caffe, Caffe2, MXNet and TensorFlow) can be benchmarked in a multi-GPU mode, excluding Intel's fork of Caffe and apparently NVIDIA's inference engine TensorRT.</p> <h3>Weak scaling</h3> <p>The default option implemented in DLBS is a <code>weak</code> scaling. Users need to provide a <code>--exp.device_batch</code> and <code>exp.gpus</code> parameters. The first one is an integer value specifying a per-GPU batch size. The second one is a comma-separated list of GPU identifiers. For instance, the following command line launches TensorFlow benchmark with ResNet50 on a an eight-GPU system with a per GPU batch size being equal to 32:</p> <pre><code class="language-bash">python experimenter.py run -Pexp.framework='"tensorflow"' \ -Pexp.model='"resnet50"' \ -Pexp.gpus='"0,1,2,3,4,5,6,7"' \ -Pexp.device_batch=32 \ -Pexp.bench_root='"./benchmarks/my_experiment"'\ -Pexp.log_file='"${exp.bench_root}/tf_resnet50.log"' </code></pre> <p>DLBS will compute internal parameter <code>exp.effective_batch</code> by multiplying number of GPUs by a per-GPU device batch size:</p> <pre><code class="language-json">{ "exp.num_gpus": "$(len('${exp.gpus}'.replace(',', ' ').split()))$", "exp.device": "$('gpu' if ${exp.num_gpus} &#x26;gt; 0 else 'cpu')$", "exp.effective_batch": "$(${exp.num_gpus}*${exp.device_batch} if '${exp.device}' == 'gpu' else ${exp.device_batch})$" } </code></pre> <h3>Strong scaling</h3> <p>There are several ways to benchmark strong scaling. One is to correctly adjust a per-device batch size with parameter <code>exp.device_batch</code> so that the effective batch size does not change. The mechanism of <a href="https://hewlettpackard.github.io/dlcookbook-dlbs/#/intro/intro?id=extensions">extensions</a> can effectively be used to disable certain benchmarks that result in non-desirable effective batch sizes.</p> <p>The second approach is to provide effective batch size as a base parameter and compute per-device batch size based on it. For instance, to explore Caffe2's strong scaling with AlexNet model with 1024 images in a batch on an eight-GPU system, users can run the following command:</p> <pre><code class="language-bash">python experimenter.py run -Pexp.framework='"caffe2"' \ -Pexp.model='"alexnet"' \ -Vexp.gpus='["0", "0,1", "0,1,2,3", "0,1,2,3,4,5,6,7"]' \ -Pexp.effective_batch=1024\ -Pexp.device_batch='"$(${exp.effective_batch}/${exp.num_gpus})$"'\ -Pexp.bench_root='"./benchmarks/my_experiment"'\ -Pexp.log_file='"${exp.bench_root}/${exp.id}.log"' </code></pre> <p>For an introduction to specifying benchmark configurations, read the <a href="https://hewlettpackard.github.io/dlcookbook-dlbs/#/intro/intro">introduction</a> section. For commonly used parameters and framework specific parameters, read the <a href="https://hewlettpackard.github.io/dlcookbook-dlbs/#/parameters/parameters?id=parameters">parameters</a> and the <a href="https://hewlettpackard.github.io/dlcookbook-dlbs/#/frameworks/frameworks?id=frameworks">framework specific parameters</a> sections.</p> <h3>BVLC/NVIDIA Caffe</h3> <p>The family of Caffe frameworks supports multi-GPU training with the NCCL/NCCL2 library. The only configuration parameter that affects this is the comma-separated list of GPUs, <code>exp.gpus</code>.</p> <h3>TensorFlow</h3> <p>We use the <a href="https://github.hpe.com/labs/dlcookbook/tree/master/python/tf_cnn_benchmarks">tf_cnn_benchmarks</a> project as a backend for TensorFlow framework. The following parameters affect multi-GPU training:</p> <ol> <li><code>exp.gpus</code> Comma separated list of GPUs to use.</li> <li><code>tensorflow.var_update</code> The method for managing variables: <em>parameter_server</em>, <em>replicated</em>, <em>distributed_replicated</em>, <em>independent</em> (<a href="https://github.hpe.com/labs/dlcookbook/blob/master/python/tf_cnn_benchmarks/tf_cnn_benchmarks.py#L164">source code</a>).</li> <li><code>tensorflow.use_nccl</code> Whether to use nccl all-reduce primitives where possible (<a href="https://github.hpe.com/labs/dlcookbook/blob/master/python/tf_cnn_benchmarks/tf_cnn_benchmarks.py#L168">source code</a>)</li> <li><code>tensorflow.local_parameter_device</code> Device to use as parameter server: cpu or gpu (see <a href="https://github.hpe.com/labs/dlcookbook/blob/master/python/tf_cnn_benchmarks/tf_cnn_benchmarks.py#L92">source code</a>)</li> </ol> <h3>Caffe2</h3> <p>In the current version of a <a href="https://github.hpe.com/labs/dlcookbook/tree/master/python/caffe2_benchmarks">Caffe2 backend</a>, the only parameter that affects multi-GPU training is <code>exp.gpus</code>.</p> <h3>MXNet</h3> <p>Currently, in our <a href="https://github.hpe.com/labs/dlcookbook/tree/master/python/mxnet_benchmarks">MXNet backend</a> two parameters affect multi-GPU training:</p> <ol> <li><code>exp.gpus</code> Comma-separated list of GPUs to use.</li> <li><code>mxnet.kv_store</code> A method to aggregate gradients 'local', 'device', 'dist_sync', 'dist_device_sync' or 'dist_async'. See <a href="https://mxnet.incubator.apache.org/how_to/multi_devices.html">this page</a> for more details.</li> </ol> <h2>References</h2> <ol> <li><a href="https://arxiv.org/pdf/1610.06276.pdf">Modeling Scalability of Distributed Machine Learning</a>.</li> <li><a href="https://www.sharcnet.ca/help/index.php/Measuring_Parallel_Scaling_Performance">Measuring Parallel Scaling Performance</a></li> </ol>https://developer.hpe.com/intelligent-provisioning/https://developer.hpe.com/intelligent-provisioning/Wed, 22 Nov 2017 18:48:50 GMThttps://developer.hpe.com/surviving-in-the-schema-while-running-initial-inventory/https://developer.hpe.com/surviving-in-the-schema-while-running-initial-inventory/Mon, 11 Sep 2017 18:45:18 GMT<p>In previous articles, we managed to login and acquire a login session. Now let us discover the Composable Infrastructure managed by HPE OneView, in a top down approach through the objects available. The following illustration shows the objects we will check during this process:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-1-1505155314126.png" alt="Datacenter structure"></p> <h1>So what exactly is a Datacenter?</h1> <p>From the graph, we know that the top most object is a Datacenter, so let us use our REST client to browse for available Datacenters in our configuration using /rest/datacenters. Before we do so, how do we know what are the properties that make up a Datacenter object? There is a technique, to find out about this. This technique is to add a /schema at the end of the URL, for example /rest/datacenters/schema, to retrieve the schema for datacenter.</p> <p>Note: This technique is still experimental in HPE OneView v2.0, and not all resources are described by a GET /schema command, although the ones typically used to create an initial inventory are. This technique is subject to evolve in the future</p> <p>Let us try with our HTTP Requester Client.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-2-1505155330418.png" alt="REST call via HTTP to retrieve the datacenters schema"></p> <p>If you find the result difficult to read, copy the response JSON and open another tab of your browser to an JSON online parser (such as <a href="http://json.parser.online.fr">http://json.parser.online.fr</a>), and paste the JSON response in the left pane there</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-3-1505155337227.png" alt="JSON online parser to parse the retrieved JSON object"></p> <p>From this tool, you can collapse and expand the properties of an object and understand what is available. For example from the capture below, we can see that a Datacenter has a dozen properties such as width, depth, or status.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-4-1505155343740.png" alt="JSON online parser have collapse and expand option to understand what is available in the JSON object"></p> <p>Alternatively, we can use Postman (from Chrome). The result window allows to expand or collapse elements directly in the body of the response, which is very convenient:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-5-1505155351099.png" alt="Postman REST tool for chrome allows to expand or collapse elements directly in the body of the response"></p> <p>We can continue this exploration and drill down on width, which has a description, a type (integer), and a min and a max value. We also can find a required flag, which is important, as required properties are mandatory when creating new objects with the POST method. We can find other types of object. For example expand the currency property and you will see that the type is string.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-6-1505155367635.png" alt="drill down to individual properties of an object"></p> <p>Integer and string are simple types, but if you continue to scroll down the list of properties, you will see the contents property, which is an array of items, of type object. An item is constructed from a number of properties such as rotation, x, y and a resourceUri. We also call this a collection.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-7-1505155374560.png" alt="Array of items with in JSON object"></p> <p>Note: In JSON, integer are integral numeric values (3, -24 …) while number can be an integer or a floating-point value (3, -24, 1.5, -3.3333 …)</p> <p>We understood from the exploration of the datacenter schema that, a datacenter is composed of collection of zero or more racks for which we now have a resourceUri.</p> <h1>So what is a Rack?</h1> <p>Before we start exploring the content of our datacenter, let us look at the schema for a rack by using the /rest/racks/schema URL using our REST client tool.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-8-1505155381704.png" alt="schema for a rack using URL in REST client tool"></p> <p>Again, we can cut/paste the REST client response into a JSON parser to navigate it easily. From there we can discover the properties of a rack, such as width, height, depth, and a collection of rackMounts, which represent zero or more items contained in the rack (enclosure, server-hardware, power delivery device)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-9-1505155389486.png" alt="paste the REST client response into a JSON parser to discover the properties of a rack"></p> <p>We can expand the rackMounts property to understand what composes a rack. We discover that it is made of a collection of items, and that each item has a mountUri to access it.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-10-1505155395931.png" alt="expand the rackMounts property to understand what composes a rack"></p> <p>We now understand that an HPE OneView instance contains a collection of one or more datacenters, which is composed of a collection of zero or more racks, which contains a collection of one or more items such as a server or an enclosure. We know enough to start our first inventory.</p> <h1>Let's look it up now!</h1> <p>Let's now use our REST Client to explore the object instances found in our environment. Starting from the topmost objects, the datacenters.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-11-1505155402777.png" alt="REST Client to explore the object instances found in our environment. Starting from the topmost objects, the datacenters"></p> <p>From the result JSON we find out that we have only one datacenter, called "Sophia Antipolis", and in this datacenter we have a single rack which URI is: /rest/racks/Rack-221.</p> <p>Note: total is the total number of elements while count is the number of element returned in this "page". prevPageUri and nextPageUri allow to navigate to other pages in case of a large number of items</p> <p>We can now drill down to the content of RACK-221</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-12-1505155409949.png" alt="drill down to the content of RACK-221"></p> <p>From that query, we discover that our Rack-221 is a "HP 42U Intelligent Series Rack" and in this rack we have, from the bottom up:</p> <ul> <li> <p>One 10U enclosure (/rest/enclosures/09SGH102X6J1)</p> </li> <li> <p>Another 10U enclosure (/res/enclosures/09SGH100X6J1)</p> </li> <li> <p>One 2U server (/rest/server-hardware/37333036-3831-584D-5131-303030333037)</p> </li> <li> <p>And finally, a 1U server (/rest/server-hardware/37333036-3831-584D-5131-303030323038).</p> </li> </ul> <p>We have, with the mountUri property, a direct access to the URI of each item in the rack.</p> <h1>Hugh! What is an Enclosure?</h1> <p>We have not yet explored what an enclosure is, so let's be curious and take a quick look at the schema of an enclosure. In HPE terminology, an enclosure is a C7000 enclosure, and it look something like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-13-1505155418059.png" alt="C7000 enclosure"></p> <p>It is composed of many elements, such as blade servers (deviceBays), onboard administrators (managerBays), interconnect modules (interconnectBays), fans (fanBays), power supplies (powerSupplyBays). If we look at the schema of an enclosure (/rest/enclosures/schema) we can confirm that.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-14-1505155424179.png" alt="ov inventory schema 14"></p> <p>We can also see that an enclosure is composed of a deviceBays array of items. If we now query our first enclosure (/rest/enclosures/09SGH102X6J1 for example), we can see there that this is an enclosure called Encl2, and we can see many of the useful parameter that make up an enclosure.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-15-1505155430487.png" alt="explore enclosure deviceBays array of items"></p> <p>We can also discover that it is composed of 16 device bays and that each of these bays is, in our case a blade server for which we are given the deviceUri (/rest/server-hardware/31393736-3831-4753-4831-30325837524E for example). Note that this is a server-hardware resource, similar to the rackmount servers we already discovered in the rack earlier.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-16-1505155436138.png" alt="Explore the enclosure deviceBays array of items"></p> <p>If you checked all the items in the deviceBays collection, you would discover 14 servers, and another 14 servers in the second enclosure (Encl1), plus our 2 rack-mount systems, which makes a total of 30 server-hardware resources.</p> <p>Note: servers in bay 1 and 2 are double-height servers, and as such, they use the bays 9 and 10. Actually, in such case, if you look in the server URI of bay x, and bay x+8, you shall see the same server URI</p> <h1>Et voilà! The result of our initial discovery</h1> <p>Based on this information we can now build a complete picture of the discovered environment. We could represent this in our own software component, and we could be storing some of the properties for it, plus the resource URI for quick retrieval of the full object properties.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-inventory-schema-17-1505155441995.png" alt="complete picture of the discovered environment"></p> <p>In a next article, we will discuss how to find and associate the management processors of each of these servers.</p>https://developer.hpe.com/powershell-defined-infrastructure/https://developer.hpe.com/powershell-defined-infrastructure/Mon, 11 Sep 2017 17:07:33 GMT<p>In previous articles (<a href="https://community.dev.hpe.com/t5/Blogs/First-steps-with-programming-the-HPE-Composable-Infrastructure/ba-p/235724">here</a>, <a href="https://community.dev.hpe.com/t5/Blogs/Authenticating-against-HPE-Composable-Infrastructure-API/ba-p/235893">here</a> and <a href="https://community.dev.hpe.com/t5/Blogs/Surviving-in-the-Schema-while-running-our-first-inventory/ba-p/235998">here</a>), we used POSTman to experiment with the Composable Infrastructure API, and then we reused some of these experiments (<a href="https://community.dev.hpe.com/t5/Blogs/quot-cURL-ing-quot-through-the-HPE-Composable-Infrastructure-API/ba-p/236298">here</a>), to script some automation using cURL in a Shell script (or from a Windows Command prompt with some patience). Well, for Windows there is a far better solution to script and automate a HPE Composable Infrastructure, and it uses Windows PowerShell.</p> <p>Windows PowerShell is THE Command Language Interface (CLI) and scripting language unanimously adopted by all Windows administrators for several reasons including:</p> <p>• Its standard and normalized syntax. PowerShell uses the syntax of verb-noun to interact with the system. You use the Get verb to query and provide the object noun to specify the resource. For example, Get-Process returns list of processes, Get-EventLog returns list of event logs. To create new resources, you use the verb new, to update, you use the verb set…. You learn the syntax once and apply the same pattern for different resources[1]</p> <p>• Its extensibility. By default, Windows PowerShell provides commands or precisely Cmdlets to interact with the OS. Microsoft also provides a rich set of libraries to interact with other components of Windows such as Active Directory, DHCP, network... Libraries are packaged as modules and administrators simply import the modules to get access to Cmdlets related to components. For example, to get a list of Active Directory users, you can run Get-ADuser; to get a list of disk volumes you can use Get-Volumes. Going further, any Microsoft product is shipped with its own PowerShell module and as a result, Windows PowerShell can be seen as a management platform rather than just a rigid CLI.</p> <p>On that trend, HPE provides an HPE OneView PowerShell module, which consists of a set of script functions exported as Cmdlets. Script functions wraps PowerShell code around the REST API and hides the complexity of using REST API to interact with HPE OneView. Windows administrators now can use their familiar syntax to interact with HPE Oneview, for example Get-HPOVNetwork, get-HPOVServer and so on.</p> <h1>Installing the HPE OneView PowerShell module</h1> <p>The HPE OneView PowerShell library is an HPE Open Source project that is hosted on <a href="http://hewlettpackard.github.io/POSH-HPOneView/">GitHub</a>.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-1-1505150268392.png" alt="Link to PowerShell library .exe hosted on GitHub"></p> <p>The easiest way to install the PowerShell library is to select and download the <a href="https://github.com/HewlettPackard/POSH-HPOneView/releases">installer</a> to run. The Installer provides digitally signed assets and verifies system requirements. You can also download the repository (zip or tar.gz), which will only contain the library components, and it is up to the administrator to put the library into the required directory and validate that their system meets the requirements. Once it is installed, you can use it in your PowerShell environment (interactive or script) using an import-module command:</p> <p>import-module hponeview.200</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-2-1505150284255.png" alt="ps basic 2"></p> <h1>Looking for help?</h1> <p>We can check what this library offers with the following command:</p> <p>get-command -module hponeview.200</p> <p>There is a long list of commands available. You can get more details on a given Cmdlet using the help command:</p> <p>help connect-hpovmgmt</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-3-1505150291900.png" alt="import-module command of hponeview.200"></p> <h1>Connect…</h1> <p>Let us try this immediately</p> <p>Connect-HPOVMgmt 213.30.139.22:37441 -username administrator -password password</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-4-1505150298918.png" alt="connect command to HPEOneview"></p> <h1>And start scripting!</h1> <p>Once connected we can use the different Cmdlets to start automating an HPE Composable Infrastructure from PowerShell. For example, let us gather versions:</p> <p>Get-HPOVVersion</p> <p>Get-HPOVVersion -ApplianceVer</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-5-1505150306720.png" alt="Get HPE OneView version command"></p> <p>As we did in previous articles using POSTman or cURL, let us now retrieve the enclosures managed in this HPE Composable Infrastructure using:</p> <p>Get-HPOVEnclosure</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-6-1505150314278.png" alt="Retrieve HPE OneView enclosures"></p> <p>We can also retrieve the list of servers available:</p> <p>Get-HPOVServer</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-7-1505150322382.png" alt="Retrieve HPE OneView servers"></p> <p>You can see that, the PowerShell library handles a lot of the hard work we had to do while programming directly against the REST API. The HPE OneView PowerShell library handles for us things such as HTTP Headers or JSON parsing.</p> <h1>We can always REST</h1> <p>However, what if we need to get something, which does not have a Cmdlet available yet? For example in our REST based inventory, we listed the datacenters managed by the HPE OneView appliance, but there is no GET-HPOVDatacenter in the HPE OneView Module. Well, there is a solution for this in the library, it is called Send-HPOVRequest, and it is a great way to access the REST API directly. Send-HPOVRequest is very similar to what we did with cURL in a previous article.</p> <p>Let us use this to retrieve our datacenter list:</p> <p>$datacenters=Send-HPOVRequest "/rest/datacenters"</p> <p>$datacenters.members[0].name</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-8-1505150331230.png" alt="Retrieve datacenters using Send-HPOVRequest cmdlet"></p> <p>We can use help Send-HPOVRequest to find out more about the Cmdlet, and this shows that the default verb is GET, but we can also use other HTTP verbs, for example let us say we want to change the currency property of our Sophia Antipolis datacenter from USD to Euros:</p> <p>$datacenters.members[0].currency</p> <p>USD</p> <p>$datacenters.members[0].currency="Euros"</p> <p>Send-HPOVRequest $datacenters.members[0].uri "PUT" $datacenters.members[0]</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ps-basic-9-1505150337250.png" alt="ps basic 9"></p> <h1>Feel like contributing?</h1> <p>Well, even better. If you think that instead of using Send-HPOVRequest, it would be better to write a CmdLet for GET-HPOVDatacenter, and you feel like doing it yourself, then you can contribute using the following procedure:</p> <ul> <li> <p>From GitHub, fork the repo</p> </li> <li> <p>Make the code change to add your contribution</p> </li> <li> <p>From GitHub, submit a Pull request for review</p> </li> </ul> <h1>Beware with asynchronous operations</h1> <p>By design, HPE OneView is an asynchronous task-based system. Therefore, Cmdlets might initiate an asynchronous operation, and return immediately (this is documented in the CmdLet help). In these cases, it is best practice to retrieve that task object returned by the Cmdlet and use the Wait-HPOVTaskComplete to wait for task completion before moving on to the next step. For example, let us use this technique to create a new server profile:</p> <p>New-HPOVServerProfile -name TestProfile -assignmentType 'unassigned' -serverhardwaretype "BL460c Gen8 1" -enclosuregroup "HPE Composable Infrastructure Enclosures" | Wait-HPOVTaskComplete</p> <h1>The sky is the limit!</h1> <p>Well, I hope I gave you enough details to get your attention so that you can start writing your own PowerShell scripts. We have also made available on the <a href="https://github.com/HewlettPackard/POSH-HPOneView/wiki">Wiki</a> of the GitHub page, a number of sample scripts, so do not hesitate to leverage these as well. We will come back to some of these in upcoming articles.</p> <p>[1] Microsoft Approved PowerShell Verbs: <a href="https://msdn.microsoft.com/en-us/library/ms714428(v=vs.85).aspx">https://msdn.microsoft.com/en-us/library/ms714428(v=vs.85).aspx</a></p>https://developer.hpe.com/leveraging-hpe-oneview-single-sign-on-to-ilo-management-processors/https://developer.hpe.com/leveraging-hpe-oneview-single-sign-on-to-ilo-management-processors/Mon, 11 Sep 2017 16:49:26 GMT<p>In this previous article, <a href="https://community.dev.hpe.com/t5/Blogs/Surviving-in-the-Schema-while-running-our-first-inventory/ba-p/235998">Surviving in the Schema… while running our first inventory</a>, we discussed how to run a discovery of the environment managed by HPE OneView. We derived a tree structure of the following format:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-1-1505149237398.png" alt="Datacenter tree structure"></p> <p>We can check the schema for a server-hardware using our favorite REST client tool (Postman extension for Chrome in our case) with the following command: GET /rest/server-hardware/schema against the HPE OneView API.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-2-1505149245923.png" alt="check the schema for a server-hardware using our favorite REST client tool"></p> <p>We notice that a management processor (mp) is also available in a server-hardware and that some details about the management processor are provided (mpFirmwareVersion, mpHostInfo, mpModel, mpState).</p> <h1>Don't you like my iLO?</h1> <p>I am sure most of you are familiar with the concept of management processor in servers, but in the ProLiant family of servers, these are best known as iLO (integrated Lights-Out). iLO management cards have been provided in ProLiant servers since 2001. They have since then been replaced by a custom chip on server motherboard and are extremely popular in datacenter management. Today iLO 4 is the current generation of iLO processors present on Gen8 and Gen9 of ProLiant servers. iLO management processors provide a wide variety of great options when it comes to management of servers in an out-of-band manner. To start with, the management IP address of the iLO offers a web console to manage the server-hardware.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-3-1505149253257.png" alt="ILO remote console to the system"></p> <p>But probably the most used feature of the iLO processor, is the capability to open a remote console to the system, which really allows to be "on" the system without ever having to enter a computer room, and whether or not the server is powered on, or installed with an operating system or not.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-4-1505149260202.png" alt="ILO remote console to the system"></p> <p>The HPE OneView web console provides a direct link the web interface of the iLO of any given server-hardware, as well as a direct link to the iLO Remote Console of the server. This is great but what would also be nice, is if third party applications integrating with HPE OneView using the Composable API, could leverage this feature and embed those links in their application context allowing users to reach, in one single click, the iLO Web Console or Remote Console of any discovered server-hardware. This article describes how to do this.</p> <h1>So how do we SSO?</h1> <p>iLO Management Processors are protected by credentials. Users can be added to the authorized users list, or you can even leverage an existing LDAP infrastructure, but to get started you have to access the iLO using a factory generated administrator password provided with every server that HPE ships with an iLO. However, because the server was discovered and placed under the management of HPE OneView, there is a special "agreement" between HPE OneView and each iLO to allow for a Single Sign On (SSO) from HPE OneView to login to iLOs. This means that valid credentials in HPE OneView allow for a pass-through authentication to all managed iLOs. This is an extremely powerful feature, which allows configuration of large groups of server iLOs, with simply an HPE OneView account.</p> <p>We query for the list of server-hardware available to us using a GET /rest/server-hardware, and from there pick a given one of them using its URI (for example /rest/server-hardware/37333036-3831-4753-4831-30325838524E)</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-5-1505149266642.png" alt="query for the list of server-hardware available to us"></p> <p>We can see all the details about this particular server-hardware iLO instance. All? Well, not quite, because if we check the HP OneView API Reference we can find the two following commands:</p> <ul> <li> <p>GET /rest/server-hardware/{id}/iloSsoUrl</p> </li> <li> <p>GET /rest/server-hardware/{id}/remoteConsoleUrl</p> </li> </ul> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-6-1505149274227.png" alt="HPE Oneview API reference to retrieve server hardware"></p> <p>Note: As you can see, the HPE OneView API Reference describes it but it does not appear in the server-hardware description nor in the schema description</p> <p>If we try to use the first one: /rest/server-hardware/37333036-3831-4753-4831-30325838524E/iloSsoUrl</p> <p>We obtain the following result:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-7-1505149281204.png" alt="Retrieve server hardware ILO SSO Url"></p> <p>While if we try to query HPE OneView with the following additional URI: /rest/server-hardware/37333036-3831-4753-4831-30325838524E/remoteConsoleUrl</p> <p>We get the following response:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/ov-ilo-sso-8-1505149288674.png" alt="Oneview API reference to get server hardware"></p> <p>Using these two calls, we can offer three types of interesting integrations:</p> <ol> <li> <p>Open the Web interface of the iLO of that given server</p> </li> <li> <p>Open the iLO Remote Console of that given server</p> </li> <li> <p>Use the iLO REST API to set/get particular attributes of a given server</p> </li> </ol> <h1>Direct link to iLO Web Console</h1> <p>An application might decide to provide a link to the iLO Web Interface as part of its own representation of a server-hardware object. In this case, it will need to run a GET /rest/server-hardware/{id}/iloSsoUrl, to retrieve the URL to use when the user click to obtain the iLO Web Interface.</p> <h1>Direct link to iLO Remote Console</h1> <p>In addition to providing a link to the iLO Web Console, an application might decide to provide a direct link to the iLO Remote Console of a server. In order to do this, a call to GET /rest/server-hardware/{id}/remoteConsoleUrl would have to be initiated, and the retrieved link should be opened.</p> <p>Note: This link requires the HP iLO Remote Console application installed on the client machine on which the link is clicked. This application can be downloaded from the Overview page of the iLO Web Interface of any iLO management processor</p> <h1>Use the iLO REST API</h1> <p>Finally, since iLO 4 management processor offers a powerful REST API, it is also possible to use SSO to manipulate configuration of the iLO using its REST API. More details about the iLO REST API can be found in the following web page: <a href="http://www.hpe.com/info/ilo">www.hpe.com/info/ilo</a></p> <p>Note: the iLO REST API is available since firmware version 2.0, and is Redfish compliant starting at version 2.3</p> <p>For example, you could decide that the application wants to change the server power capping value based on some internal rules. For this, you would have to use a PATCH method to and pass an HTTP Header X-Auth-Token with the value of the SSO session token retrieved from a GET /rest/server-hardware/{id}/iloSsoUrl.</p> <p>Let's review an example and imagine you retrieved from GET /rest/server-hardware/{Server id}/remoteConsoleUrl, the following:</p> <p>{"remoteConsoleUrl":"hplocons://addr=<strong>192.168.1.184</strong>&#x26;sessionkey=<strong>823b6814fd3aa2d9f82ec9f34504b6d9</strong>"}</p> <p>You would have to assemble the following PATCH command:</p> <p>PATCH https://<strong>192.168.1.184/redfish/v1/Chassis/1/PowerMetrics</strong></p> <p>With HTTP Header: X-Auth-Token = <strong>823b6814fd3aa2d9f82ec9f34504b6d9</strong></p> <p>And a body of: {"PowerLimit": {"LimitInWatts": 250}}</p> <p>In order to cap the given server-hardware to a maximum power utilization of 250 Watts.</p> <p>Note: Priority should be to change settings of server-hardware via the HPE OneView API, and keep the iLO REST API for things not available in the HPE OneView API</p> <h1>Tighter partner integration = best user experience</h1> <p>As we discovered there are different levels of integrations that can be built into an HPE OneView partner application. Of course, the tighter the integration is, the richer is the user experience for our joint customers.</p>https://developer.hpe.com/hpe-composable-infrastructure-api-can-java-too/https://developer.hpe.com/hpe-composable-infrastructure-api-can-java-too/Fri, 08 Sep 2017 16:49:18 GMT<p>We have seen in the previous articles, <a href="https://community.dev.hpe.com/t5/Blogs/PowerShell-defined-infrastructure-at-your-fingertips/ba-p/236848">PowerShell-defined infrastructure at your fingertips!</a> that we can use a scripting language to automate HPE Composable Infrastructure, and control it in a Software-defined way. However, what about our developer community? Some say, "Real developers don't use scripting languages", and I will not argue and discuss the differences between a scripting language and a programming language. Instead, this blog introduces how to get started programming HPE Composable Infrastructure with Java.</p> <h1>We need an Eclipse</h1> <p>Most programmers use an Integrated Development Environment (IDE) to write code. The most famous ones has to be Eclipse, an open source IDE that you can get from <a href="https://eclipse.org/downloads">https://eclipse.org/downloads</a>. Other famous ones are Microsoft Visual Studio, and its recent open source incantation called Visual Studio Code, which you can get from <a href="https://code.visualstudio.com/download">https://code.visualstudio.com/download</a> In this article, we will use Eclipse.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/api-java-1-1505088036254.png" alt="Eclipse IDE"></p> <h1>Hello HPE OneView World!</h1> <ul> <li> <p>All programmers know this; it all starts with a Hello World! So let's try to write our first Java program and print on the screen the version of our HPE OneView appliance. To do this</p> </li> <li> <p>Create a new Java Project called OneView</p> </li> <li> <p>Create a new Java class called HelloOneViewWorld with all defaults and hit Finish</p> </li> <li> <p>Cut and paste the following Java code which will be the starting point for our experiments</p> </li> </ul> <pre><code class="language-java"> import java.io.IOException; import java.io.IOException; import java.io.InputStreamReader; import java.net.HttpURLConnection; import java.net.URL; public class HelloOneViewWorld { public static void main(String\[\] args) { try { } catch (IOException e) { e.printStackTrace(); } } } </code></pre> <ul> <li>Within the Try {} block, create a new URL object pointing to your HP OneView appliance:</li> </ul> <blockquote> <p>URL url = <strong>new</strong> URL("https:///rest/version");</p> </blockquote> <ul> <li>Next, create an URLConnection object. Then specify some of the arguments such as Method (GET), and Property (Accept:application/json), remember we already had to do the same when we used POSTman in <a href="https://community.dev.hpe.com/t5/Blogs/API-version-What-API-version/ba-p/235776">API version? What API version? </a></li> </ul> <pre><code class="language-java"> HttpURLConnection conn = (HttpURLConnection)url.openConnection(); conn.setRequestMethod("GET"); conn.setRequestProperty("Accept", "application/json"); </code></pre> <ul> <li>Next, we want to read the result in a BufferedReader object with:</li> </ul> <pre><code class="language-java"> BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream()))); </code></pre> <ul> <li>To keep it simple, let's only continue if we received a Status code of 200</li> </ul> <pre><code class="language-java"> if (conn.getResponseCode() != 200) { System.out.println("Error from HPE OneView.... \\n"); return; } </code></pre> <ul> <li>Finally, let's print the result on the console with</li> </ul> <pre><code class="language-java"> String output; System.out.println("Hello World! HPE OneView Version is: \\n"); while ((output = br.readLine()) != null) { System.out.println(output); } </code></pre> <ul> <li>And close the connection</li> </ul> <pre><code class="language-java"> conn.disconnect(); </code></pre> <ul> <li>Save your work</li> </ul> <p>You can download the entire module from the attachments of this article. Nevertheless, things should look like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/api-java-2-1505088497854.png" alt="Eclipse IDE workspace structure"></p> <p>When you run this code, you get the following displayed on the Eclipse Console:</p> <blockquote> <p>Hello World! HPE OneView Version is:</p> <p>{"currentVersion":200,"minimumVersion":1}</p> </blockquote> <h1>When SSL security strikes again…</h1> <p>Well maybe not… Because, unfortunately, or fortunately should I say (security is important), if you run this code you will get a little delay (trying to open a connection to your HPE OneView) but finally you will get a timeout error with something like this in the stack trace:</p> <p>at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)</p> <p>Which tells us that there was a security problem connecting to this URL, and the reason is that by default, HPE OneView Appliances install with a self-signed certificate, which is not considered secure by client applications. Most developers are familiar with what has to be done at this point:</p> <ul> <li> <p>Option 1: Explicitly remove all security checks from your java code (not advisable). There are many article describing how to do this. This one is the more readable one: <a href="http://www.nakov.com/blog/2009/07/16/disable-certificate-validation-in-java-ssl-connections/">http://www.nakov.com/blog/2009/07/16/disable-certificate-validation-in-java-ssl-connections/</a></p> </li> <li> <p>Option 2: Reconfigure your HPE OneView appliance to join an existing corporate Public Key infrastructure (PKI)</p> </li> <li> <p>Option 3: Import the self-signed certificate of the HPE OneView appliance, in the Eclipse certificates keystore using a utility called keytool (more precisely in the keystore of the Java environment used by Eclipse). There are plenty of articles on how to do this, but this one does a good job: <a href="http://www.grim.se/guide/jre-cert">http://www.grim.se/guide/jre-cert</a></p> </li> </ul> <p>Let's describe briefly the steps involved with option 3:</p> <ul> <li> <p>Locate the Java run time environment used by Eclipse (Project properties, then Java Build Path, Libraries)</p> </li> <li> <p>Open a command prompt and change directory to the root of the Java run time library</p> </li> <li> <p>Open a browser to the HPE OneView Appliance</p> </li> <li> <p>Export certificate to the root of Java run time library with name: OneView.cer</p> </li> <li> <p>In command promt run:</p> <ul> <li> <p>cd bin</p> </li> <li> <p>keytool -import -file ../OneView.cer -alias MyOneView -keystore ../lib/security/cacerts</p> </li> <li> <p>when prompted for password, use: changeit</p> </li> <li> <p>review changes and make note of the DNS names associated with this certificate, you will use one of these as a base URL in your program</p> </li> <li> <p>Type yes to import certificate in keystore</p> </li> <li> <p>You can use keytool -list to verify it has been imported, and keytool -delete to clean up</p> </li> <li> <p>If the DNS names found in the certificate are not valid in your DNS environment, add a line for it in your system hosts file (in c:\windows\system32\drivers\etc\).</p> </li> </ul> </li> </ul> <p>Once the certificate is loaded in the keystore (meaning that your Java environment now trust the HPE OneView self-signed certificate), your Java code will proceed.</p> <h1>Getting a session token</h1> <p>We can now move to the next logical step, which is to authenticate against the HPE OneVew REST API and obtain a session token as we already did in <a href="https://community.dev.hpe.com/t5/Blogs/Authenticating-against-HPE-Composable-Infrastructure-API/ba-p/235893">Authenticating against HPE Composable Infrastructure API</a>. So let's create another Java class called GetToken.java in Eclipse, and cut/paste the code from HelloOneViewWorld.java.</p> <ul> <li> <p>Add another import for java.io.OutputStream</p> <p>import java.io.OutputStream;</p> </li> <li> <p>Change the URL object to use:</p> </li> </ul> <blockquote> <p>URL url = new URL("https:///rest/login-sessions");</p> </blockquote> <ul> <li>Replace the connection properties with the following:</li> </ul> <pre><code class="language-java"> conn.setRequestMethod("POST"); conn.setDoOutput(true); conn.setRequestProperty("Accept", "application/json"); conn.setRequestProperty("Content-Type", "application/json"); conn.setRequestProperty("X-API-Version", "200"); String body = "{\\"userName\\":\\"administrator\\",\\"password\\":\\"password\\"}"; OutputStream os = conn.getOutputStream(); os.write(body.getBytes()); os.flush(); </code></pre> <ul> <li>Replace the Hello World string with " HPE OneView Response is: \n"</li> </ul> <p>Everything else can remain unchanged and the final code should look like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/api-java-3-1505088504470.png" alt="Java code to get Oneview appliance token"></p> <p>You can now execute this code, which, provided you solved the SSL issue mentioned above, will return the following:</p> <p>HPE OneView Response is:</p> <pre><code class="language-json"> {"partnerData":{}, "sessionID":"MjA5NjEwOTM0MTQ4bcFsogYf-fHCmOPdSsLu0259scrx\_SFV"} # And continue from there… Once you have a version number and a session token you can place any call you'd like on the HPE OneView API. Remember to always use the following connection properties: ```java conn.setRequestProperty("Accept", "application/json"); conn.setRequestProperty("X-API-Version", "YourSelectedVersion"); conn.setRequestProperty("auth", "your-sessionID-goes-here"); </code></pre> <p>However, you should also be aware that a Java SDK for HPE OneView is available from Github (<a href="https://github.com/HewlettPackard/oneview-sdk-java">https://github.com/HewlettPackard/oneview-sdk-java</a>), and this SDK simplifies many things when doing Java programming, by creating a lightweight level of abstraction between your Java code and the HPE OneView API. Most important is the parsing of the resulting JSON, which becomes simple Java properties when using the Java SDK. We will cover this in a forthcoming article.</p>https://developer.hpe.com/first-step-with-programming-the-hpe-composable-api/https://developer.hpe.com/first-step-with-programming-the-hpe-composable-api/Fri, 08 Sep 2017 16:36:19 GMT<h1>A bit of history</h1> <p>In December 2015, in London, HPE announced Synergy, the first platform architected from the ground up for composability. Earlier that year, HPE had announced the Composable API, and the Composable ecosystem. This ecosystem is now growing because more and more software partners have expressed interest in integrating with the Composable API. Some have already demonstrated products, some have products in development, but all of them are using the same Composable API, also referred to as the HPE OneView API. This API allows HPE OneView and the Synergy Composer (powered by HPE OneView) to be controlled by another software entity, thus it is an essential step in building a software-defined infrastructure. The controlling software entity might be as simple as a small script (in PowerShell or Python), it might also be a configuration management system, such as Chef, Ansible or any kind of software that needs to interact with the underlying infrastructure (VMware vCenter, Microsoft System Center, Docker Machine, etc.). In all cases an API is a prerequisite for a software defined infrastructure and the growing ecosystem around it. HPE OneView and the Synergy Composer is no exception and it comes with an API since v1.0 in September 2013.</p> <h1>REST API in 30 seconds</h1> <p>There are many types of APIs, but the most widely used API in modern software, is called REST (for Representational State Transfer). A REST API (or RESTful API as it is often referred to) has a number of design principles:</p> <ul> <li> <p>It uses HTTP and HTTPS as a communication protocol</p> </li> <li> <p>It uses a very simple set of HTTP verbs to initiate actions (PUT, GET, POST, DELETE, PATCH)</p> </li> <li> <p>Objects are represented by their URI (Universal Resource Identifier)</p> </li> </ul> <p>For example, the following are examples of REST calls:</p> <ul> <li> <p>POST <a href="https://oneviewsrv.cilab.net/rest/login-sessions">https://oneviewsrv.cilab.net/rest/login-sessions</a> (means: apply for an identity token)</p> </li> <li> <p>GET <a href="https://oneviewsrv.cilab.net/rest/alerts">https://oneviewsrv.cilab.net/rest/alerts</a> (means: get the list of alerts)</p> </li> </ul> <p>More generically, we can see that any call is in the form of:</p> <ul> <li>Verb protocol://server-end-point/URI</li> </ul> <p>Where <em>Verb</em> can be one of the following, and its action is applied to the provided URI, over the specified protocol at a given server endpoint:</p> <ul> <li> <p>GET: to read the value of a resource</p> </li> <li> <p>PUT: to change the value of a resource</p> </li> <li> <p>POST: to create a new instance of a resource</p> </li> <li> <p>DELETE: to remove an instance of a resource</p> </li> <li> <p>PATCH: to update values of a resource.</p> </li> </ul> <p>> <strong>Note: PATCH is part of the newest additions to the HTTP protocol and > it is not yet fully supported by the Composable API</strong></p> <h1>RTFM: Read The Fantastic Manuals</h1> <p>Ok, enough talking! Let's try to make our first call to the HPE Composable API. Where do we start? Well, the API is fully described in the product’s online Help, and it is also available online at <a href="http://www.hpe.com/info/oneview/docs">http://www.hpe.com/info/oneview/docs</a>. There, you will find a downloadable version of the document, which is nice to have on your laptop as it presents itself as a set of HTML pages that is easy to navigate. As an example, once you open the help page, you can search for "version" and you will find something like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/progapi-1-1504889058235.png" alt="Response of get API version REST call"></p> <p>There is a page like this for each single API call available in HPE OneView. From the information provided, we can see that a call to GET /rest/version will return the versions supported by the API. Let’s use this for our first interaction with the HPE OneView API. If you scroll down a bit in the help page, you will also find what kind of response is expected:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/progapi-2-1504889064687.png" alt="Response of get API version REST call"></p> <p>We can see, that this API returns a Content-Type of JSON (JavaScript Object Notation). This is a very popular format used to exchange data with an API. It is similar to XML, just a little more human readable. Many APIs are moving away from XML as the exchange format in preference of JSON. HPE OneView uses JSON for input parameters, and responses. There are many web sites for you to learn about JSON, but I have found this online parser very helpful to check the syntax of a JSON payload before using it: <a href="http://json.parser.online.fr/">http://json.parser.online.fr/</a></p> <h1>Pick a REST Tool</h1> <p>So now we have all the information we need to place our first HPE OneView API call to retrieve the version numbers. However, if you are new to REST programming, you might ask yourself: how do I place a REST call to an API without writing any code? There are a number of simple solutions for this. For example, there are several browser plug-ins available and I am sure you will find one for your favorite browser (HttpRequester for Firefox, POSTman for Chrome, just to name a few). My favorite is <strong>HttpRequester</strong> for Firefox and it looks like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/progapi-3-1504889071823.png" alt="HttpRequester tool for Firefox"></p> <p>If you prefer Chrome, then <strong>POSTman</strong> would look like this:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/progapi-4-1504889079190.png" alt="POSTman for Google Chrome"></p> <h1>Let's go!</h1> <p>Both are very easy to install and intuitive to use, but I will continue to use HttpRequester in the rest of this document. So let’s fill out the following fields:</p> <table> <thead> <tr> <th>Field</th> <th>Value</th> <th>Try it here</th> </tr> </thead> <tbody> <tr> <td>URL</td> <td>https://{HPEOneViewappliance}/rest/version</td> <td><a href="https://213.30.139.22:37441/rest/version">https://213.30.139.22:37441/rest/version</a></td> </tr> <tr> <td>Verb</td> <td>GET</td> <td></td> </tr> <tr> <td>Header</td> <td>Accept=application/json</td> <td></td> </tr> </tbody> </table> <p><strong>Note: Before you start, make sure you open a browser connection to your HPE OneView API at least once and accept the self-signed certificate. Otherwise HttpRequester will not work.</strong></p> <p>There is no request payload to worry about for the GET /rest/version call. You can specify an HTTP Header to explicitly tell the API that we expect a response in JSON, you might get XML response otherwise. When ready, press the <strong>Submit</strong> button.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/progapi-5-1504889085855.png" alt="Retrieve the oneview version using HttpRequester for firefox"></p> <p>HttpRequester will contact the URL specified and call GET version from the API. In the Response section, as illustrated above, you want to check the HTTP status code. A value of 200 means it was successful. Next, you can check the Response body formatted as JSON, to find out about the version. This response will have to be parsed to retrieve the currentVersion value. But we can read pretty easily that currentVersion=200 (meaning HPE OneView V2.0)</p> <p>Congratulations! You have successfully placed your first call to the HPE OneView API. As you have seen, GET version does not require any authentication. In fact, it is one of the few calls you can place without having to provide proper authentication. In subsequent articles, we will dive deeper into the API and learn about how to authenticate and retrieve information about the HPE OneView resources.</p> <p>Although this example might seems very basic, any integration with HPE OneView should always start by querying the API versions supported, therefore, this is always going to be your very first step.</p>https://developer.hpe.com/do-not-miss-the-hpe-composable-infrastructure-bus/https://developer.hpe.com/do-not-miss-the-hpe-composable-infrastructure-bus/Thu, 07 Sep 2017 18:09:49 GMT<h1>One API and 2 Busses</h1> <p>In previous articles, we discussed how to get started with programming the HPE Composable Infrastructure API. We reviewed some of the concept on the versioning, the authentication and the discovery of the infrastructure, via a REST API approach. Well, there is an additional component to the HPE Composable Infrastructure, which we have not yet discussed called the Message Bus. Actually, there are two of these busses. One is called the State Change Message Bus (SCMB) and the second one is called the Metrics Streaming Message Bus (MSMB).</p> <h1>So what is a message bus?</h1> <p>Before we look at the specific of the HPE Composable Infrastructure message busses, let us look at what a message bus is. Message bus is a programming concept, which allows for exchange of data between a producer and one or more consumer in a guaranteed, and asynchronous manner. This concept has evolved over time and there is now a well know standard for it, called AMQP (Advanced Message Queuing Protocol). Most complex software solutions, which need to communicate asynchronously between two components, use AMQP to do so nowadays. In this technology we can identify a Producer (in our case it is going to be our HPE Composable Infrastructure brain, HPE OneView), which generates and puts data (called messages), on different message queues (via an Exchange), and then we can have any number of consumers, which have declared themselves as subscribers of a given queue. We call Message Bus, the sum of these message queues. Consumers can be any program implementing AMQP. This most recognized implementation of AMQP is called RabbitMQ. There are RabbitMQ server side components and client side components. For example, HPE OneView has implemented the server side of the message bus using RabbitMQ.</p> <p>The following picture illustrate the concept of producer/consumer and message bus</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/messabus-1-1504810187875.png" alt="Illustrate the concept of producer/consumer and message bus"></p> <h1>HPE OneView Message Bus? What for?</h1> <p>HPE OneView is using internally message busses, to exchange data between its own components. It also exposes two of these busses externally and allows client applications to subscribe to them and retrieve data from those asynchronous queues. For example, you can, as an external application, subscribe to all events relative to creation of new Networks in HPE Composable Infrastructure. If you followed my previous articles, and continued exploring the HPE Composable infrastructure API, you probably know by now, that there is a call to retrieve network configuration (/rest/networks). So one way (let's call it the "API way") to achieve this would be with the following algorithm:</p> <pre><code>While true Do Query for list of Network using /rest/networks Compare with old list of networks If new list of networks bigger then old list of network Then compute delta For each network in delta do-work-on-new-network(network) Endif Sleep 15mn Endo </code></pre> <p>We call this technique interval polling. It works fine but it has several drawbacks:</p> <ol> <li> <p>It's not real time, if a new network is created, you might be aware of it, worst case 15mn later</p> </li> <li> <p>It's pulling a lot of data from the API, especially if you reduce the polling interval to be more accurate, or if you are polling multiple resource types</p> </li> <li> <p>It requires to store in memory the state of the networks (this can grow big)</p> </li> <li> <p>It requires a lot of computation to get the delta between the old networks list and the new networks list</p> </li> <li> <p>In most of the cases the delta would be null and you have wasted a lot of time and energy polling and computing for nothing</p> </li> </ol> <p>A better approach would be using a message bus (in our case HPE OneView's SCMB) and the algorithm is the following:</p> <p>Subscribe to SCMB(Events of Interest= Network.Create, Callback=Do-Work-On-New-Network)</p> <p>That's it! All you really need to do is to tell RabbitMQ that you want to call Do-Work-On-New-Network whenever a new Network is created on the Composable Infrastructure by the HPE OneView Web interface or by its REST API.</p> <p>In fact that is exactly what some of our ecosystem partners have done. For example, Arista has implemented such a technique to allow top-of-rack ARISTA switches to detect and react to new networks created in HPE OneView. In this case, the reaction is to propagate automatically the VLAN configuration to top of rack switches. HPE Network Management Tool iMC v7.2 also implemented the same type of automation.</p> <h1>Routing Keys</h1> <p>When we subscribe to a message bus, we have to specify what we called in the previous chapter: "Events of Interest". These in HPE OneView is called a routing key. It specifies the type of messages you are interested in, and you want to subscribe to. Routing keys are used by the Exchange to drop messages in the right consumer queues. The generic form of a routing key is:</p> <p>.&#x3C;<em>resource-category></em>.&#x3C;<em>change-type></em>.&#x3C;<em>resource-uri></em></p> <p>With:</p> <ul> <li> <p>exchange-name: scmb or msmb</p> </li> <li> <p>resource-category: any valid resource type from HPE OneView</p> </li> <li> <p>change-type: Create, Deleted or Updated</p> </li> <li> <p>resource-uri: any valid resource URI</p> </li> </ul> <p>The following tables shows a number of valid routing keys for the SCMB:</p> <table> <thead> <tr> <th>Routing Key</th> <th>Meaning</th> </tr> </thead> <tbody> <tr> <td>scmb.#</td> <td>all messages</td> </tr> <tr> <td>scmb.tasks.#</td> <td>all messages about any tasks</td> </tr> <tr> <td>scmb.ethernet-networks.#</td> <td>all messages about any Ethernet networks</td> </tr> <tr> <td>scmb.fc-networks .#</td> <td>all messages about any FibreChannel networks</td> </tr> <tr> <td>scmb.interconnects.#</td> <td>all messages about any interconnects</td> </tr> <tr> <td>scmb.enclosures.#</td> <td>all messages about any enclosures</td> </tr> <tr> <td>scmb.*.Created.#</td> <td>only create messages for any type of resources</td> </tr> <tr> <td>scmb.server-hardware.#</td> <td>all messages about any server hardware</td> </tr> <tr> <td>scmb.connections.Deleted.#</td> <td>only delete messages for any connections</td> </tr> <tr> <td>scmb.enclosures.Updated./rest/enclosures/Encl1</td> <td>only update messages from Encl1</td> </tr> </tbody> </table> <p>Note: routing keys are case sensitive. * means anything for a given field, while # means anything for the rest of the string. So scmb.# is the same as scmb.*.*.*</p> <p>The MSMB only accept one routing key: msmb.#</p> <p>The following picture show routing keys in action with three different partner application subscribing to different types of messages from the HPE Composable Infrastructure.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/messabus-2-1504810196244.png" alt="routing keys in action with three different partner application subscribing to different types of messages"></p> <h1>Security</h1> <p>Security of this mechanism is implemented through SSL. In order to start using the SCMB or the MSMB, a client application needs to use the REST API to:</p> <ol> <li> <p>Create the RabbitMQ keypair, if not already created with</p> <p>POST /rest/certificates/client/rabbitmq and a body of {"type":"RabbitMqClientCertV2","commonName":"default"}</p> </li> <li> <p>Download the necessary certificate and private key using: GET /rest/certificates/client/rabbitmq/keypair/default</p> </li> <li> <p>Separate and save the client certificate and the private key (procedure depends on what language is used by application)</p> </li> <li> <p>Download the HPE Composable Infrastructure Appliance root CA certificate using GET /rest/certificates/ca</p> </li> <li> <p>Store the root ca certificate in appropriate store (depends on language used)</p> </li> </ol> <p>Note: you can use DELETE /rest/certificates/ca/rabbitmq_readonly and then POST /rest/certificates/client/rabbitmq to regenerate a new client certificate and private key</p> <h1>Message Bus messages attributes</h1> <p>Once queue is setup and application is "listening", it will be receiving messages in JSON. We can find the format of messages received from each message bus from the online documentation, <a href="http://h17007.www1.hpe.com/docs/enterprise/servers/oneview2.0/cic-rest/en/content/c_using-msg-bus.html">here</a>.</p> <p>The following is an example of an SCMB message:</p> <pre><code class="language-json">{"data":null "timestamp":"2016-07-04T15:26:02.109Z" "resourceUri":"/rest/ethernet-networks/e5facce5-bfe9-4ce2-b793-f0e1d907d620" "eTag":"3a7f25c9-9bb0-4405-8d56-7600447e3411" "changeType":"Created" "newState":"Active" "associatedTask":"/rest/tasks/C15F02F8-CE5E-47C9-A090-CB8CFF3F158A" "newSubState":null "userInitiatedTask":false "changedAttributes":null "resource":{"type":"ethernet-networkV2" "vlanId":2112 "ethernetNetworkType":"Tagged" "internalVlanId":0 "smartLink":true "connectionTemplateUri":"/rest/connection-templates/0a9ecfd2-bfc7-450b-b4cf-48d7bf1044f2" "purpose":"General" "privateNetwork":false "name":"DemoNetwork" "state":"Active" "description":null "status":"OK" "category":"ethernet-networks" "eTag":"3a7f25c9-9bb0-4405-8d56-7600447e3411" "modified":"2016-07-04T15:26:02.050Z" "created":"2016-07-04T15:26:02.049Z" "uri":"/rest/ethernet-networks/e5facce5-bfe9-4ce2-b793-f0e1d907d620"}} </code></pre> <p>Note: Because it is not using a REST API technique, Message Bus JSON messages are always sent using the current API version of the appliance</p> <h3>SCMB vs MSMB</h3> <p>As we have seen, the SCMB is used to get information about changes made in the HPE Composable Infrastructure. It was the first bus to be exposed by the HPE Composable Infrastructure and opened to partner consumption. Since HPE OneView 2.0, a new bus was introduced called MSMB for Metrics Streaming Message Bus. This bus is used to get regular value of consumption of the different resources of the HPE Composable Infrastructure such as dower devices, enclosures and server hardware.</p> <p>SCMB Consumer applications should always select their routing key with care and make sure that they can consume messages efficiently, and pick them up as fast as the appliance puts them on the queues.</p> <h1>Putting it all together</h1> <p>We have built a demo "consumer" application to browse the HPE OneView Message Busses, so let us start it up and as a simple example, subscribe to all Ethernet Networks changes.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/messabus-3-1504810203042.png" alt="demo &#x22;consumer&#x22; application to browse the Ethernet network changes of HPE OneView Message Busses"></p> <p>Then let us use the HPE OneView web console to create a new Ethernet network:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/messabus-4-1504810211206.png" alt="HPE OneView web console to create a new Ethernet network"></p> <p>Back on the MessageBus Explorer, we can see that it has been notified of that change in the HPE Composable Infrastructure and can potentially take the necessary actions. In our case, it simply displays the details of the messages in a list.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/messabus-5-1504810217256.png" alt="displays the modified or created or updated details of the Oneview messages in a list."></p> <p>We can select to view the JSON details which displays the following content (highlighted in red are the important fields):</p> <pre><code class="language-json">{"data":null "timestamp":"2016-07-04T15:26:02.109Z" "resourceUri":"/rest/ethernet-networks/e5facce5-bfe9-4ce2-b793-f0e1d907d620" "eTag":"3a7f25c9-9bb0-4405-8d56-7600447e3411" "changeType":"Created" "newState":"Active" "associatedTask":"/rest/tasks/C15F02F8-CE5E-47C9-A090-CB8CFF3F158A" "newSubState":null "userInitiatedTask":false "changedAttributes":null "resource":{"type":"ethernet-networkV2" "vlanId":2112 "ethernetNetworkType":"Tagged" "internalVlanId":0 "smartLink":true "connectionTemplateUri":"/rest/connection-templates/0a9ecfd2-bfc7-450b-b4cf-48d7bf1044f2" "purpose":"General" "privateNetwork":false "name":"DemoNetwork" "state":"Active" "description":null "status":"OK" "category":"ethernet-networks" "eTag":"3a7f25c9-9bb0-4405-8d56-7600447e3411" "modified":"2016-07-04T15:26:02.050Z" "created":"2016-07-04T15:26:02.049Z" "uri":"/rest/ethernet-networks/e5facce5-bfe9-4ce2-b793-f0e1d907d620"}} </code></pre> <h1>Summary</h1> <p>Message Bus is a very powerful and standard method to integrate with an ecosystem. In the HPE Composable Infrastructure ecosystem, it provides a very complementary approach to the existing HPE Composable Infrastructure REST API, and allows partner applications to integrate tighter and more efficiently with HPE Composable Infrastructure.</p>https://developer.hpe.com/curling-through-the-oneview-api/https://developer.hpe.com/curling-through-the-oneview-api/Thu, 07 Sep 2017 17:33:30 GMT<p>In previous articles, we used a REST client plug-in to Firefox or Chrome to exercise the HPE Composable Infrastructure API. This is nice, for discovery and understanding of the API, but the next step toward a software-defined infrastructure is to be able to automate those actions, in a script. One possible approach to writing automation scripts using a REST API such as the HPE Composable Infrastructure API, is to use a tool like curl, and open source command line tool able to exchange data with a web application or API using an URL syntax. cURL (Command-line URL) is available for just about any device and any operating system, and actually used by millions of users. More on cURL from <a href="https://curl.haxx.se">https://curl.haxx.se</a>.</p> <h1>cURL syntax 101</h1> <p>Let us remember how we described a REST call in <a href="https://community.dev.hpe.com/t5/Blogs/First-steps-with-programming-the-HPE-Composable-Infrastructure/ba-p/235724">our first article</a>:</p> <p>> Verb protocol://server-end-point/URI</p> <p>With some examples such as:</p> <ul> <li> <p>POST <a href="https://oneviewsrv.cilab.net/rest/login-sessions">https://oneviewsrv.cilab.net/rest/login-sessions</a></p> </li> <li> <p>GET <a href="https://oneviewsrv.cilab.net/rest/alerts">https://oneviewsrv.cilab.net/rest/alerts</a></p> </li> </ul> <p>So how do these calls translate in cURL?</p> <p>We can call cURL with no option, and see the syntax: **curl [options...] **</p> <p>There are many available options but let us focus on the most important ones:</p> <table> <thead> <tr> <th>Option</th> <th>Meaning</th> <th>Example</th> </tr> </thead> <tbody> <tr> <td>H</td> <td>HTTP Header</td> <td>-H "accept: application/json"</td> </tr> <tr> <td></td> <td></td> <td>-H "content-type: application/json"</td> </tr> <tr> <td>d</td> <td>data or payload</td> <td>-d '{"userName":"Administrator","password":"password"}'</td> </tr> <tr> <td>X</td> <td>command</td> <td>-X GET <a href="https://213.30.139.22:37441/rest/version">https://213.30.139.22:37441/rest/version</a></td> </tr> <tr> <td></td> <td></td> <td>-X POST <a href="https://213.30.139.22:37441/rest/login-sessions">https://213.30.139.22:37441/rest/login-sessions</a></td> </tr> <tr> <td>k</td> <td>insecure</td> <td>bypasses SSL certificate validation</td> </tr> <tr> <td>i</td> <td>include HEADER in response</td> <td>useful to check the response status code, but not used when parsing JSON result</td> </tr> <tr> <td>p</td> <td>use a proxy</td> <td>-p <a href="http://mycompanyproxy.com:8888">http://mycompanyproxy.com:8888</a></td> </tr> </tbody> </table> <p>So let us try!</p> <p>On a Linux machine with Internet access, type the following command, which will retrieve the version of the API.</p> <pre><code class="language-postscript">curl -i -k -H "accept: application/json" \\ -X GET https://213.30.139.22:37441/rest/version </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-1-1504806712277.png" alt="Curl command to retrieve the version of the API."></p> <h1>Parsing JSON</h1> <p>We can see in the last line of the above example that the response is provided in JSON (we asked for it with our accept HTTP header). So in order to write powerful scripts, we will need a mechanism to parse the JSON responses to extract the fields that we need. We have found this convenient open source tool called jq (command line JSON processor) available from <a href="http://stedolan.github.io/jq/download/linux64/jq">http://stedolan.github.io/jq/download/linux64/jq</a></p> <p>Once installed you can use it like this:</p> <pre><code class="language-postscript">curl -k -H "accept: application/json" \\ -X GET https://213.30.139.22:37441/rest/version | jq -r "." </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-2-1504806725594.png" alt="command line JSON processor to parse the JSON"></p> <p>This will just pretty print the JSON response, but you can also extract field, for example to retrieve the currentVersion.</p> <pre><code class="language-postscript">curl -k -H "accept: application/json" \\ -X GET https://213.30.139.22:37441/rest/version | jq -r ".currentVersion" </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-3-1504806732409.png" alt="Retrieve API version using command line JSON parser"></p> <p>So let us pretend would like to capture the currentVersion in a variable within a shell script, we could do the following:</p> <h1>Retrieve API version</h1> <pre><code class="language-postscript">currentVersion=$(curl -k -H "accept: application/json" -X GET https://213.30.139.22:37441/rest/version | jq -r ".currentVersion") echo $currentVersion </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-4-1504806739646.png" alt="retrieve current version of API using open source jq tool"></p> <h1>Getting a Session token</h1> <p>Let us now apply this technique to login to our HPE OneView Appliance and start managing our HPE Composable Infrastructure. As we have already seen in previous articles, we need to send a POST to the /rest/login-sessions API.</p> <pre><code class="language-postscript">curl -k -H "accept: application/json" -H "content-type: application/json" \\ -d '{"userName":"Administrator","password":"password"}' \\ -X POST https://213.30.139.22:37441/rest/login-sessions | jq -r "." </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-5-1504806745891.png" alt="Getting a Session token using POST Curl command"></p> <p>We can then, extract the sessionID in a variable using the following syntax:</p> <pre><code class="language-postscript">sessionID=$(curl -k -H "accept: application/json" -H "content-type: application/json" \\ -d '{"userName":"Administrator","password":"password"}' \\ -X POST https://213.30.139.22:37441/rest/login-sessions | jq -r ".sessionID") echo $sessionID </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-6-1504806754438.png" alt="extract the sessionID in a variable"></p> <h1>Putting it all together</h1> <p>We have now retrieved the API Version (currentVersion) and a login session (sessionID), we are now ready to explore the full HPE Composable Infrastructure API with a simple shell script. For example, let us enumerate the models of the servers managed in this environment</p> <pre><code class="language-postscript">curl -k -H "accept: application/json" -H "content-type: application/json" \\ -H "x-api-version: $currentVersion" -H "auth: $sessionID" \\ -X GET https://213.30.139.22:37441/rest/server-hardware \\ | jq -r ".members\[\].model" </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-7-1504806761092.png" alt="enumerate the models of the servers managed in this environment"></p> <h1>cURL on Linux and Windows</h1> <p>But what about Windows? Some of you might ask. The good news is that everything we have just shown here, can be done on Windows. If you use cygwin, the cURL package is part of the default distribution and jq can be easily compiled from sources located at <a href="https://stedolan.github.io/jq/download/">https://stedolan.github.io/jq/download/</a></p> <p>Another option is to install jq and cURL for Window, and in that case, you will have to perform a few syntax changes to the examples we showed above.</p> <ul> <li> <p>cURL if available from: <a href="http://winampplugins.co.uk/curl/curl_7_48_0_openssl_nghttp2_x64.7z">http://winampplugins.co.uk/curl/curl_7_48_0_openssl_nghttp2_x64.7z</a></p> </li> <li> <p>jq is available from: <a href="https://github.com/stedolan/jq/releases/download/jq-1.5/jq-win64.exe">https://github.com/stedolan/jq/releases/download/jq-1.5/jq-win64.exe</a></p> </li> </ul> <p>Once curl and jq are installed and in the default path, we can then execute the following command to retrieve a sessionID:</p> <pre><code class="language-postscript">curl -k -H "accept: application/json" -H "content-type: application/json" -X POST https://213.30.139.22:37441/rest/login-sessions -d "{\\"userName\\":\\"Administrator\\",\\"password\\":\\"password\\"}" | jq -r ".sessionID" </code></pre> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-8-1504806767303.png" alt="Curl command to retrieve a sessionID using jq tool"></p> <p>However, if you really need to store it into a variable in Windows command interpreter, things get a little more complicated:</p> <p>for /f "delims=" %a in ('curl -k -H "accept: application/json" -H "content-type: application/json" -X POST <a href="https://213.30.139.22:37441/rest/login-sessions">https://213.30.139.22:37441/rest/login-sessions</a> -d "{\"userName\":\"Administrator\",\"password\":\"password\"}" ^| jq -r ".sessionID"') do @set sessionID=%a</p> <p>echo %sessionID%</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-9-1504806773156.png" alt="store session id into a variable in Windows command interpreter"></p> <h1>And the last good news is…</h1> <p>So I asked early on, how do these calls translate in cURL? Here is the last piece of good news: POSTman can help you with that. Imagine you have been experimenting with a GET in POSTman, and you would like to run the same GET from cURL in a script</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-10-1504806780133.png" alt="POSTman can help to generate Curl code"></p> <p>You can use the <strong>Generate Code</strong> button in the upper right corner and select cURL from the drop down to get the right string ready to paste in your own script.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/curl-11-1504806788067.png" alt="Generated Curl code from POSTman REST tool"></p> <h1>What is next?</h1> <p>Scripting with cURL might solve a number of use cases, and is somewhat portable across Linux and Windows, but it is not the most convenient and readable scripting language. In the next articles, we will cover how to script against the HPE Composable Infrastructure API using Microsoft PowerShell and Python.</p>https://developer.hpe.com/automation-of-support-pack-for-proliant-in-hpe-oneview/https://developer.hpe.com/automation-of-support-pack-for-proliant-in-hpe-oneview/Thu, 07 Sep 2017 03:17:17 GMT<p>Support Pack for ProLiant (SPP) is the official HPE distribution of updates to the ProLiant family of server. It is an ISO image, which customers typically download from HPE website. Once locally available it can be mounted on a ProLiant server and booted from, in order to update the system software of the server such as the BIOS, component firmware and OS drivers. It can also be used in combination with HPE Smart Update Manager (HPSUM). The releases are named after their release date so for example find SPP2016.10.0.iso for a SPP released in October 2016. HPE OneView also offers the capability of leveraging this ISO image, by loading it as a new "baseline" for automatic systems updates. This article describes how to script this operation. More on the SPP web site <a href="http://h17007.www1.hpe.com/us/en/enterprise/servers/products/service%5C_pack/spp/index.aspx">http://h17007.www1.hpe.com/us/en/enterprise/servers/products/service\_pack/spp/index.aspx</a></p> <h1>Managing servers at scale</h1> <p>SPP ISO images can be mounted and booted on a single server in order to upgrade its components. That is ok for a couple of servers but will become a very tedious and time-consuming operation when managing hundreds of servers in a datacenter. A more comprehensive way to use the SPP is to combine it with HPE OneView. HPE OneView offers the option to load a SPP and to use it as a property of a server profile.</p> <h1>Server Profile</h1> <p>Before we look at how to load SPP into HPE OneView, let us review what a server profile is. Server Profile is the most important concept in HPE OneView. It represents the abstraction layer, which isolates a workload from the underlying server-hardware on which it runs. The server-profile artifact is the place where we setup BIOS settings, hardware type, and number of network connections, SAN attachments, number and size of disks for a given workload. We can create many of these server profiles, and even create server profile templates, and these remain completely disconnected from the hardware until finally applied to an available compute resource. When this happens, properties of the profile are applied to the designated server hardware, connections are made to the network infrastructure, and disks are carved automatically. One of the property of a server profile (or a server profile template) is the Firmware Baseline. The picture below shows options available for managing firmware baseline during the creation of a server profile.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-sp-1504754621422.png" alt="shows options available for managing firmware baseline during the creation of a server profile"></p> <p>So if a server profile also has the Firmware property set to a given baseline, this baseline will automatically be installed on the server hardware as part of the profile deployment. This offline technique (there are other online methods for applying firmware, using SUT - Smart Update Tool) removes a lot of complexity for server administrators. It adds a lot more flexibility too, as it allows to validate and apply certain SPP to given workloads, and other SPP to other workloads, as this is just another property of a server profile.</p> <h1>Custom SPP</h1> <p>It is important to note that, while HPE provides its customers with regular updates of the SPP ISO images, it is possible to build custom SPP. With this technique, you can create smaller ISO images, for example by selecting only components, which exist in the customer's datacenter. More information on custom SPP on <a href="https://spp.hpe.com/custom/">https://spp.hpe.com/custom/</a></p> <h1>Loading SPP in HPE OneView</h1> <p>In order to get an SPP to appear in the list of available SPP to choose from for server profile, it needs to be loaded in HPE OneView. The obvious and GUI-based option to do this is in HPE OneView Web Console Firmware Bundles page, as shown below:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-firmwarebundle-1504754856328.png" alt="HPE OneView Web Console Firmware Bundles page"></p> <p>From there it is possible to load additional Firmware Bundle from SPP ISO images.</p> <h1>Automating it</h1> <p>Another approach to achieve this is to use PowerShell and the HPE OneView library for PowerShell, which we discussed in the previous post There is a couple of PowerShell commandlets in the library to deal with Firmware Bundles. We have:</p> <ul> <li> <p>Get-HPOVBaseline</p> </li> <li> <p>Add-HPOVBaseline</p> </li> <li> <p>Remove-HPOVBaseline</p> </li> </ul> <p>There are several possible scenario to automate the management of baselines. For example, we could:</p> <ol> <li> <p>Permanently monitor the create/delete file in a given folder and add or remove ISO images as they show up. We can use the System.IO.FileSystemWatcher PowerShell object to achieve this</p> </li> <li> <p>Look for the files in a repository and compare with the baselines already loaded and keep them in synch</p> </li> <li> <p>Simply load ISO images from disk</p> </li> </ol> <p>To keep things simple we will implement option 3, as it requires less PowerShell code. The steps of the scripts are the following:</p> <ul> <li>Importing the HPE OneView library</li> </ul> <p>Import-module HPEOneView.200</p> <ul> <li>Connect to a given HPE OneView Appliance (if not already connected)</li> </ul> <pre><code class="language-postscript">if (!$global:myAppliance) { $global:myAppliance = Read-Host "HPE OneView FQDN or IP address" } if ($global:ConnectedSessions.Name -notcontains $global:myAppliance) { $connection=Connect-HPOVMgmt -appliance $global:myAppliance -User administrator -password } </code></pre> <ul> <li>Enumerate Firmware Bundles already on appliance</li> </ul> <pre><code class="language-postscript">$bundlesloaded = get-hpovbaseline write-host "Found the following baseline on appliance" foreach ($item in $bundlesloaded) { write-host " - " $item.isoFilename} </code></pre> <ul> <li>Enumerate ISO images in current folder</li> </ul> <pre><code class="language-postscript">$filesinfolder = get-item \*.iso write-host "Found the following ISO in folder" foreach ($item in $filesinfolder) { write-host " - " $item.Name} </code></pre> <ul> <li>Ask for confirmation</li> </ul> <pre><code class="language-postscript">Write-Host "Ok to upload ISO files on disk to appliance (Y/N)?" $key = $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown") if ($key.Character -ne "Y") { write-host You wimp! ; break } </code></pre> <ul> <li>And start uploading</li> </ul> <pre><code class="language-postscript">foreach ($item in $filesinfolder) { write-host "Uploading file" $item.name "to appliance" $task=add-hpovbaseline $item.name $t = Wait-HPOVTaskComplete $task.uri -timeout (New-TimeSpan -Minutes 30) } </code></pre> <ul> <li>Terminate</li> </ul> <pre><code class="language-postscript">Write-Host "Done!" Write-Host "Disconnecting..." disconnect-hpovmgmt </code></pre> <h1>Putting it all together</h1> <p>It is now time to save this PowerShell script and test it against an HPE OneView Appliance or a HPE Synergy Composer.</p> <p>Before we run our script, let us check the appliance baseline view (empty to start with):</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-nofirmwarebundle-1504803548928.png" alt="HPE OneView Web Console Firmware Bundles page"></p> <p>In our first run, we can say Y(es) to confirm upload:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-ps1-1504803751946.png" alt="Powershell console showing firmware bundle upload"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-ps2-1504803760139.png" alt="Powershell console showing firmware bundle addition"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-ps3-1504803766997.png" alt="Powershell console showing firmware bundle addition"></p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-ps4-1504803773690.png" alt="Powershell console showing firmware bundle upload process complete"></p> <p>After the run, we can check in HPE OneView again, to see that the baseline is now loaded</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-firmwarebundle-1504754856328.png" alt="Check Baseline is loaded through HPE OneView web console"></p> <p>In the second run, we are notified that baseline is already there. We can just say N(o) to exit:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/spp-ps5-1504803781691.png" alt="Powershell console notifying that baseline is already there in the second run of the script"></p> <p>> <strong>Note: be careful as HPE OneView modifies the filename of the bundle > by replacing unwanted characters such as "." and spaces, by "_"</strong></p>https://developer.hpe.com/authenticating-against-the-composable-api/https://developer.hpe.com/authenticating-against-the-composable-api/Wed, 06 Sep 2017 17:11:08 GMT<h1>Login Sessions</h1> <p>In the previous articles, we learned how to query the version of the HPE Composable API (also referenced here as the HPE OneView API), using the <code>GET /rest/version</code> call, and we discussed the importance of specifying which version of the API is expected by an application. Let us now use this to open a login session to HPE OneView. First, we can check the syntax of the login session by looking up "login" in the online help. This is what we find:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/loginsession-1504732180732.png" alt="check the syntax of the login session by looking up &#x22;login&#x22; in the online help"></p> <h1>Getting a session token</h1> <p>From the documentation, we can see that in order to open a login session, we will need to use the POST method against the URI /rest/login-sessions. This will create (remember POST is the verb used in HTTP to create something) a session token, which we will use in the subsequent calls to the API. We will ignore the login domain information for now. We can also see in the documentation, that we will need to set three HTTP Headers:</p> <ol> <li> <p>X-API-Version: set to the version of the API we want to use (let's assume 200)</p> </li> <li> <p>Content-Type: set to application/json, to tell the API, that the payload we will be providing is in JSON notation</p> </li> <li> <p>Accept: set to application/json, to tell the API to return response in JSON notation</p> </li> </ol> <p>The payload expected by the API must contains (at minimum) the following parameters:</p> <ol> <li> <p>userName: set to the user name to use for authentication</p> </li> <li> <p>password: set to the password for that user name</p> </li> </ol> <p>Note: Be careful, JSON is case sensitive. userName is expected here, not username, nor UserName. As a rule, the API expects lower case words, but capitalized, after the first word, if there are more than one word</p> <h1>HttpRequester example</h1> <p>To keep it simple we can use HttpRequester (a plug-in for Firefox; or POSTman a similar plug-in for Chrome) to test this. Set the following parameters and hit Submit.</p> <table> <thead> <tr> <th>Field</th> <th>Value</th> <th>Try it here!</th> </tr> </thead> <tbody> <tr> <td>URL</td> <td>https://{ HPEOneViewappliance}/rest/login-sessions</td> <td><a href="https://213.30.139.22:37441/rest/rest/login-sessions">https://213.30.139.22:37441/rest/rest/login-sessions</a></td> </tr> <tr> <td>Verb</td> <td>POST</td> <td></td> </tr> <tr> <td>Header</td> <td>Content-Type=application/json</td> <td></td> </tr> <tr> <td></td> <td>Accept=application/json</td> <td></td> </tr> <tr> <td></td> <td>X-API-Version=200</td> <td></td> </tr> <tr> <td>Payload</td> <td>{</td> <td></td> </tr> <tr> <td></td> <td>"userName":"administrator",</td> <td></td> </tr> <tr> <td></td> <td>"password":"password"</td> <td></td> </tr> <tr> <td></td> <td>}</td> <td></td> </tr> </tbody> </table> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/httprequester-1504732233332.png" alt="HttpRequester example to retrieve the login session token for Oneview login"></p> <p>The response we get is in the following form (extracted from online documentation):</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/httpresponse-1504732248902.png" alt="HttpRequester response showing the session Id"></p> <p>The <strong>sessionID</strong>, also known as the session token is what we are looking for.</p> <h1>Careful with that token!</h1> <p>Now in order to be authorized to call any other API methods, we will need to provide the session token as another HTTP Header called Auth. So remember, we now have two HTTP Headers, which <em>have to be provided</em> at each call to the API: <code>Auth</code> and <code>X-API-Version</code>.</p> <p>Note: You should also be aware that session tokens expire 24 hours after the last utilization.</p> <p>We can now test this token and call another API method. Let us pick <code>GET /rest/global-settings</code> which returns the current settings of the HPE OneView appliance.</p> <p>Use HttpRequester with the following parameters and press Submit:</p> <table> <thead> <tr> <th>Field</th> <th>Value</th> <th>Try it here!</th> </tr> </thead> <tbody> <tr> <td>URL</td> <td>https://{HPEOneViewappliance}/rest/global-settings</td> <td><a href="https://213.30.139.22:37441/rest/global-settings">https://213.30.139.22:37441/rest/global-settings</a></td> </tr> <tr> <td>Verb</td> <td>POST</td> <td></td> </tr> <tr> <td>Header</td> <td>Content-Type=application/json</td> <td></td> </tr> <tr> <td></td> <td>Accept=application/json</td> <td></td> </tr> <tr> <td></td> <td>X-API-Version=200</td> <td></td> </tr> <tr> <td></td> <td>Auth=your <strong>sessionID</strong></td> <td></td> </tr> </tbody> </table> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/httprequesterwithauth-1504732241236.png" alt="Retrieve global settings using the generated session token in API call"></p> <p>As a quick check, remove the <code>Auth</code> HTTP Header, and try again. You should get a 401 Status code (Unauthorized). The details of the error clearly state that you forgot to provide an <code>Auth</code> header for that request.</p> <p>Note: HTTP Headers are case insensitive: Auth or auth would work fine.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/forgotauthheader-1504732225620.png" alt="Response showing Auth header is missing"></p> <h1>Cleaning up</h1> <p>Finally, it is best practice to delete a token when it is not needed anymore using the <code>DELETE /rest/login-sessions</code>. The session ID passed as an <code>Auth</code> Header will be deleted and will not be valid for authentication anymore. In HttpRequester, you can submit the following request:</p> <table> <thead> <tr> <th>Field</th> <th>Value</th> <th>Try it here!</th> </tr> </thead> <tbody> <tr> <td>URL</td> <td>https://{HPEOneViewappliance}/rest/login-sessions</td> <td><a href="https://213.30.139.22:37441/rest/login-sessions">https://213.30.139.22:37441/rest/login-sessions</a></td> </tr> <tr> <td>Verb</td> <td>DELETE</td> <td></td> </tr> <tr> <td>Header</td> <td>Content-Type=application/json</td> <td></td> </tr> <tr> <td></td> <td>Accept=application/json</td> <td></td> </tr> <tr> <td></td> <td>X-API-Version=200</td> <td></td> </tr> <tr> <td></td> <td>Auth=your <strong>sessionID</strong></td> <td></td> </tr> </tbody> </table> <p>You should get a Status of 204 (anything in the 2xx is successful) with no content for the response:</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/deletelogintoken-1504732217113.png" alt="Response showing successful token deletion"></p> <p>The session token was deleted. If you are curious, try to reissue the request from the HttpRequester History, which retrieved <code>/rest/global-settings</code>. This time you should get another 401 Authorization error. Your token is not valid anymore.</p> <p><img src="https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2017/9/tokeninvaliderror-1504732255731.png" alt="tokeninvaliderror"></p> <h1>Key takeaways</h1> <p>Throughout the article, we have described what software will need to do in order to integrate properly with HPE OneView API and the Composable Infrastructure.</p> <ol> <li> <p>Query for the supported version of the API and verify compliance</p> </li> <li> <p>Authenticate (using HTTP Headers: <strong>X-API-Version</strong>=version) and retrieve session token</p> </li> <li> <p>Continue using the API (using HTTP Headers: <strong>X-API-Version</strong>=version and <strong>Auth</strong>=token)</p> </li> <li> <p>Close the session (using HTTP Headers: <strong>X-API-Version</strong>=version and <strong>Auth</strong>=token)</p> <p>In the following articles, we will continue to investigate important concepts of the HPE OneView API.</p> </li> </ol>https://developer.hpe.com/hpe-oneview-api-version/https://developer.hpe.com/hpe-oneview-api-version/Wed, 06 Sep 2017 13:41:38 GMT<h1>Backward compatibility of API</h1> <p>In a previous article, we discussed how important it is to provide an API with software so that your ecosystem can grow and other software entities can integrate with it. But what happens when another release of the software comes up? New capability maybe added, older behavior modified, which is normal, but it makes it more difficult to allow the software to evolve without disrupting the growing ecosystem. This is where backward compatibility becomes critical. We, as API provider, need to guarantee that existing software integrations are not going to break when a new release is introduced.</p> <p>In order to make this happen, API providers usually require integrators to specify the version of the API that they expect. In most REST APIs, this is done with a HTTP Header. And the API provider guarantees that older versions of the API remain unchanged. Of course if an integrator wants to benefit from the newest functionality, it will have to update their software to use the newest API, but the older version, running against the older API, should continue to work without modification. It is the API provider’s decision to decide how many older versions of the API are supported at any given release and deprecate older ones if necessary.</p> <h1>HPE OneView API history</h1> <p>The HPE OneView API has been around for a few years already and the table below shows the versions of the API for each release</p> <table> <thead> <tr> <th>Product Version</th> <th>API Version Number/ X-API-Version</th> <th>Date</th> </tr> </thead> <tbody> <tr> <td>1.0</td> <td>3</td> <td>September 2013</td> </tr> <tr> <td>1.05</td> <td>4</td> <td>March 2014</td> </tr> <tr> <td>1.10</td> <td>101</td> <td>June 2014</td> </tr> <tr> <td>1.20</td> <td>120</td> <td>December 2014</td> </tr> <tr> <td>2.0</td> <td>200</td> <td>October 2015</td> </tr> <tr> <td>3.0</td> <td>300</td> <td>October 2016</td> </tr> <tr> <td>3.1</td> <td>500</td> <td>July 2017</td> </tr> </tbody> </table> <h1>Make a decision…</h1> <p>Ok, so I am a partner in the HPE Composable API ecosystem, and I'm about to make use of the HPE OneView API. Which version should I use? You should always use the latest release when starting a new integration project. You can look up <code>currentVersion</code> by using the <code>GET /rest/version</code> as we did in a previous article. If you expect a long development cycle, engage with your contact in the Composable API Ecosystem, to check what the best approach is. There may already be a newer version in beta, which you may be able to leverage.</p> <h1>And stick to it!</h1> <p>Once you have decided, you should make sure that your application only uses that particular release (and let us call it <code>expectedVersion</code>). Remember that several months down the road, there will be new releases, which you do not really know about at the time of the writing of your code. You cannot be sure that your application will work well against a future release.</p> <p>So let us recap. In your code, as the first thing, you should implement the following algorithm to determine that the version of the API that you expect to be present, is indeed present and supported by the appliance instance you are about to integrate with.</p> <ol> <li> <p>Call <code>GET /rest/version</code></p> </li> <li> <p>Retrieve <code>minimumVersion</code> and currentVersion</p> </li> <li> <p>Check that <code>minimumVersion</code> &#x3C;= <code>expectedVersion</code> &#x3C;= <code>currentVersion</code></p> </li> <li> <p>If not then raise an error: "Sorry, cannot integrate with this version of HPE OneView"</p> </li> <li> <p>Otherwise set HTTP Header <code>X-API-Version</code> to <code>expectedVersion</code> for <em>all</em> subsequent API calls</p> </li> </ol> <h1>Plan your updates on a regular basis</h1> <p>HPE supports up to three versions prior to the current one. Therefore, in theory, the above algorithm will allow your software to work across many HPE OneView releases, and over a long period. Ultimately, there will be a time where <code>minimumVersion</code> will become greater than <code>expectedVersion</code> and the error message will show up (point 4). This is not a great user experience for your user community, so you should always do your best to upgrade your software and crank up the <code>expectedVersion</code>, before this message hits your users.</p>