{
  "name": "Dhruv Doshi technical content corpus",
  "canonical": "https://doshidhruv.com",
  "generatedAt": "2026-08-10",
  "license": "Copyright remains with the author. Indexing, retrieval, summarization, reproduction, and citation with attribution to the canonical URL are permitted.",
  "attribution": "Dhruv Doshi, with a link to the canonical page URL.",
  "entries": [
    {
      "type": "guide",
      "title": "Platform architecture: from standards to a usable product",
      "canonical": "https://doshidhruv.com/guides/platform-architecture/",
      "datePublished": "2026-07-28",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Staff engineering"
      ],
      "description": "A practical guide to turning architecture standards into paved roads, reusable contracts, and measurable platform outcomes.",
      "contentMarkdown": "Platform architecture is useful when it reduces the work required to make a safe, operable decision. A standards document can describe the right answer, but a platform turns that answer into something engineers can discover, adopt, and verify.\n\nThis guide describes the operating model behind that shift. It draws on the same platform, governance, and architecture-automation themes documented in the [selected work](/projects/) and [experience](/resume/) pages.\n\n## Start with decisions, not technology lists\n\nA platform should encode decisions that repeat across teams: how a service authenticates, where telemetry goes, which deployment patterns are supported, and what evidence a review needs. Begin by identifying the decisions that create the most delay or production risk.\n\nFor each decision, record:\n\n- the context in which it applies;\n- the approved options and their constraints;\n- the evidence required to select an option;\n- the owner and review date;\n- the exception path when the standard does not fit.\n\nThis produces a decision model rather than a catalogue. Technologies can change while the underlying decision remains stable.\n\n## Define a small platform contract\n\nEvery paved road needs a contract between its maintainers and consumers. A useful contract covers inputs, outputs, operational expectations, ownership, and lifecycle state. It should answer what the platform guarantees, what a consuming team must provide, and what happens when either side changes.\n\nThe contract can be expressed through schemas, templates, policy checks, service APIs, or infrastructure modules. The form matters less than whether it is versioned, testable, and visible at the point of use.\n\n## Separate policy from implementation\n\nPolicy explains the required outcome. Implementation provides one supported way to achieve it. Keeping the two separate allows a platform to evolve without weakening governance.\n\nFor example, a policy may require encrypted service-to-service identity, auditable authorization, and credential rotation. One implementation might use a particular identity provider and gateway. A second implementation can satisfy the same policy if it produces equivalent evidence.\n\n## Build feedback into the path\n\nAdoption is not proof that a platform is working. Track whether it reduces lead time, review effort, duplicated integration work, and operational variance. Combine quantitative signals with direct feedback from teams using the platform.\n\nUseful signals include:\n\n- time from a design request to an approved decision;\n- percentage of changes handled by an established pattern;\n- exception volume and repeated exception causes;\n- support requests per adopting team;\n- production incidents connected to missing or misunderstood standards.\n\n## Treat exceptions as product discovery\n\nAn exception is not automatically a governance failure. It may reveal a missing capability, an outdated constraint, or a genuinely different workload. Review exceptions as a set, not only as individual approvals. Repeated exceptions should lead to a new supported pattern, a clearer boundary, or retirement of an ineffective rule.\n\n## Keep the platform legible\n\nThe strongest platform architectures are explainable without internal knowledge. Publish the decision model, ownership, lifecycle, and escape hatch. Link implementation assets directly from the relevant decision. Give every pattern a stable URL so design reviews, code changes, and incident reports can refer to the same source.\n\nThe older notes on [cloud computing](/notes/what-is-cloud-computing/), [IaaS](/notes/infrastructure-as-a-service-iaas/), [PaaS](/notes/platform-as-a-service-paas/), and [serverless computing](/notes/serverless-computing-and-function-as-a-service/) provide the service-model background. The platform model above is the layer that makes those choices repeatable inside an engineering organisation.\n\n## Sources and further reading\n\n- [CNCF Platform Engineering Technical Community Group](https://tag-app-delivery.cncf.io/whitepapers/platforms/)\n- [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html)\n- [Architecture Decision Records](https://adr.github.io/)\n",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "guide",
      "title": "Observability systems: design the telemetry path before the dashboard",
      "canonical": "https://doshidhruv.com/guides/observability-systems/",
      "datePublished": "2026-07-28",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Platform architecture"
      ],
      "description": "A systems guide to telemetry contracts, collection, routing, storage boundaries, and operational feedback at scale.",
      "contentMarkdown": "Observability is a data system before it is a set of dashboards. The critical design work happens in the path between a workload producing a signal and an operator using that signal to make a decision.\n\nThis guide connects the vendor-neutral telemetry architecture described in [selected work](/projects/#observability-platform) with reusable design principles for logs, metrics, and traces.\n\n## Begin with operational questions\n\nInstrumentation should answer a known question: is a service meeting its objective, where is latency introduced, which dependency failed, or what changed before an incident? Collecting data without a decision in mind creates cost without reliable diagnostic value.\n\nDefine service objectives, failure modes, and investigation paths first. From those, derive the signals, dimensions, retention, and access requirements.\n\n## Use a telemetry contract\n\nA telemetry contract makes data consistent across teams and tools. At minimum, define:\n\n- service and deployment identity;\n- environment and ownership attributes;\n- trace and request correlation fields;\n- event severity and error semantics;\n- privacy classification and redaction requirements;\n- schema version and compatibility rules.\n\nOpenTelemetry provides vendor-neutral APIs, semantic conventions, and a collector model. It does not remove the need for organisational conventions; it gives those conventions a portable foundation.\n\n## Separate collection, routing, and storage\n\nTreat the telemetry path as distinct stages. Collection receives and normalises data near the workload. Routing applies policy, enrichment, sampling, and destination selection. Storage and analysis systems serve different operational and retention needs.\n\nThis separation reduces vendor coupling. A team can change an analysis destination without rebuilding every application integration. It also creates clear control points for regional handling, cost limits, and security policy.\n\n## Design for pressure and partial failure\n\nTelemetry volume often rises during the incident that operators most need to understand. The path therefore needs explicit behaviour for backpressure, buffering, retries, sampling, and data loss. Document which signals are durable, which can be sampled, and how operators detect a degraded pipeline.\n\nMonitor the observability system itself: queue depth, dropped records, processing latency, cardinality growth, export failures, and configuration drift are first-class service indicators.\n\n## Control cardinality and cost at the source\n\nUnbounded identifiers in metric labels can make a monitoring system expensive or unstable. Define approved dimensions and move high-cardinality detail into logs or traces when appropriate. Apply retention and sampling according to the value and sensitivity of the signal rather than using one policy for all telemetry.\n\n## Make ownership visible\n\nEvery service and telemetry schema needs an owner. An operator should be able to move from an alert to the responsible team, current runbook, recent deployments, and relevant service objectives without searching across disconnected systems.\n\nThe related notes on [distributed cloud](/notes/distributed-cloud/), [multi-cloud architecture](/notes/multi-cloud-architecture/), and [serverless computing](/notes/serverless-computing-and-function-as-a-service/) describe deployment models that make a portable telemetry contract especially valuable.\n\n## Sources and further reading\n\n- [OpenTelemetry documentation](https://opentelemetry.io/docs/)\n- [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/)\n- [Google SRE workbook: monitoring](https://sre.google/workbook/monitoring/)\n",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "guide",
      "title": "AI governance for software systems: controls that fit delivery",
      "canonical": "https://doshidhruv.com/guides/ai-governance/",
      "datePublished": "2026-07-28",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "Staff engineering"
      ],
      "description": "A delivery-focused guide to AI inventories, risk boundaries, evaluation, human oversight, and operational evidence.",
      "contentMarkdown": "AI governance becomes useful when it changes how a system is designed, evaluated, released, and monitored. A policy that sits outside delivery creates paperwork; an engineering control creates evidence at the same point a team makes a decision.\n\nThis guide focuses on applied AI and language-model systems. It complements the foundational notes on [artificial intelligence](/notes/introduction-to-artificial-intelligence-history-and-evolution/), [machine learning](/notes/the-fundamentals-of-machine-learning/), and [deep learning](/notes/deep-learning-explained-from-basics-to-advanced/).\n\n## Establish the system boundary\n\nBegin with an inventory entry that describes the complete system, not only the model. Record the user, intended decision, model and data providers, retrieval sources, tools, human checkpoints, outputs, and downstream actions. Assign accountable technical and business owners.\n\nThe boundary should make dependencies visible. A retrieval index, prompt template, policy filter, and external API can each change system behaviour even when the underlying model does not change.\n\n## Classify consequences before selecting controls\n\nRisk depends on how output is used. A drafting assistant with mandatory human review has a different consequence profile from a system that can change access, move money, or communicate externally without review.\n\nClassify the system according to impact, reversibility, affected users, data sensitivity, autonomy, and exposure. Use that classification to determine evaluation depth, approval requirements, monitoring, and fallback behaviour.\n\n## Turn requirements into testable controls\n\nTranslate principles into checks a delivery pipeline or reviewer can verify. Examples include:\n\n- approved model and data-provider versions;\n- documented data lineage and retention;\n- prompt-injection and data-exfiltration tests;\n- quality thresholds for defined task sets;\n- access controls for tools and retrieval sources;\n- human confirmation before consequential actions;\n- immutable records of configuration, evaluations, and approvals.\n\nControls should identify their evidence. A statement such as “the system is fair” is not a control. A defined evaluation dataset, metric, threshold, owner, and review cadence is.\n\n## Evaluate the system, not just the model\n\nTest representative end-to-end tasks, including refusal, uncertainty, retrieval failure, malformed inputs, conflicting instructions, and unavailable dependencies. Include regression cases drawn from production failures and near misses.\n\nFor generative systems, separate factuality, relevance, instruction following, safety, latency, and cost. A single aggregate score can hide a failure that matters to users.\n\n## Limit authority by default\n\nUse the least privilege required for tools, data, and actions. Prefer read-only access, scoped credentials, explicit allowlists, bounded execution, and user confirmation for irreversible operations. Treat model output as untrusted input at every integration boundary.\n\n## Monitor change over time\n\nModels, prompts, retrieval content, and external services change. Record the deployed configuration, watch for quality and safety drift, and define when a change requires re-evaluation. Provide a clear shutdown or fallback path when the system moves outside approved limits.\n\n## Sources and further reading\n\n- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)\n- [NIST Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)\n- [OWASP Top 10 for LLM applications](https://genai.owasp.org/llm-top-10/)\n",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "guide",
      "title": "Cloud migration: sequence decisions, not just workloads",
      "canonical": "https://doshidhruv.com/guides/cloud-migration/",
      "datePublished": "2026-07-28",
      "dateModified": "2026-07-28",
      "topics": [
        "Cloud architecture",
        "Platform architecture"
      ],
      "description": "A practical guide to cloud migration boundaries, dependency mapping, landing zones, observability, cutovers, and exit criteria.",
      "contentMarkdown": "A cloud migration is a sequence of operating-model decisions expressed through workload moves. Moving infrastructure without changing ownership, delivery, security, and observability usually relocates existing constraints rather than removing them.\n\nThe cloud archive on this site covers [public](/notes/public-cloud/), [private](/notes/private-cloud/), [hybrid](/notes/hybrid-cloud/), and [multi-cloud](/notes/multi-cloud-architecture/) models. This guide focuses on how to decide and execute a migration across those choices.\n\n## Define the outcome and boundary\n\nState what the migration must improve: delivery lead time, resilience, regional reach, capacity, security controls, or operating cost. Set measurable acceptance criteria and constraints. A migration defined only as “move to cloud” cannot make trade-offs consistently.\n\nChoose a boundary that can be owned and tested. A business capability or service boundary is usually more useful than a list of virtual machines because it includes data, dependencies, operations, and users.\n\n## Build a dependency map\n\nInventory runtime calls, data flows, identity dependencies, batch jobs, operational tooling, network paths, and organisational owners. Confirm the map with telemetry and operators rather than relying only on configuration records.\n\nClassify dependencies by latency sensitivity, data sensitivity, availability requirement, and ease of change. This shows which workloads can move independently and which need a coordinated transition.\n\n## Establish the landing zone as a product\n\nA landing zone should provide reusable identity, network, logging, policy, encryption, deployment, and cost-management capabilities. Version these capabilities and give teams a supported adoption path.\n\nValidate the landing zone with a representative workload before scaling migration waves. The first workload should exercise important controls without carrying the organisation’s highest operational risk.\n\n## Select a migration treatment deliberately\n\nFor each workload, decide whether to retire, retain, replace, rehost, replatform, or redesign it. The right treatment follows the desired outcome and constraints. Rehosting can reduce data-centre dependency quickly; redesign can improve elasticity or operability but introduces more change and validation work.\n\nRecord the decision, expected benefit, required evidence, rollback path, and owner. Revisit it when dependency information changes.\n\n## Make observability available before cutover\n\nOperators need comparable signals on both sides of a transition. Establish service objectives, logs, metrics, traces, ownership, and alert routing before production traffic moves. Test failure modes, capacity limits, backup restoration, and access recovery.\n\nThe note on [cloud downtime](/notes/downtime-with-cloud-computing/) is a reminder that provider infrastructure does not remove the need for explicit resilience and recovery design.\n\n## Use explicit cutover and exit criteria\n\nDefine traffic steps, data synchronization, freeze windows, rollback triggers, decision owners, and communication paths. After cutover, remove obsolete infrastructure, credentials, routes, and monitoring. A migration is incomplete while two environments remain operational without a deliberate reason.\n\n## Sources and further reading\n\n- [AWS migration guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/large-migration-guide/welcome.html)\n- [Google Cloud migration framework](https://cloud.google.com/architecture/migration-to-google-cloud-getting-started)\n- [Microsoft Cloud Adoption Framework](https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/)\n",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "guide",
      "title": "Staff engineering practice: create leverage through clear systems",
      "canonical": "https://doshidhruv.com/guides/staff-engineering-practice/",
      "datePublished": "2026-07-28",
      "dateModified": "2026-07-28",
      "topics": [
        "Staff engineering",
        "Platform architecture"
      ],
      "description": "A practical guide to technical direction, decision records, cross-team delivery, mentoring, and operational credibility.",
      "contentMarkdown": "Staff engineering is the practice of increasing the quality and pace of decisions beyond one person’s individual output. The work still requires technical depth, but its value comes from making a wider system easier to understand, change, and operate.\n\nThis guide reflects the working principles and delivery patterns documented across the [about](/about/), [experience](/resume/), and [selected work](/projects/) pages.\n\n## Find the constraint that spans teams\n\nThe highest-leverage problem is often between ownership boundaries: an integration repeated by every team, an approval with no shared evidence, an operational signal nobody owns, or a platform capability that exists but is difficult to adopt.\n\nFrame the problem in observable terms. Identify who experiences it, how often it occurs, what it delays or risks, and which constraints are real. This prevents a broad technical programme from becoming a collection of unrelated improvements.\n\n## Write the decision before scaling the implementation\n\nUse short decision records to capture context, options, trade-offs, consequences, and ownership. A decision record is valuable when someone outside the original conversation can understand why the system is shaped a certain way.\n\nKeep decision status visible. Supersede outdated records rather than silently editing history. Link decisions to implementation, operational evidence, and follow-up work.\n\n## Build a thin end-to-end path\n\nFor platform or architecture work, prove one complete path before generalising. Include the interface, policy, deployment, telemetry, support model, and documentation. A thin vertical slice exposes organisational and operational gaps that a component-only prototype misses.\n\nUse the first implementation to refine contracts and boundaries. Standardise only after the team has evidence that the path works.\n\n## Make reviews produce reusable knowledge\n\nA review should improve the current change and the system around it. When the same issue appears repeatedly, turn the feedback into a test, template, documented pattern, or platform capability. This reduces dependence on the reviewer and gives teams faster feedback.\n\n## Keep leadership close to production\n\nTechnical direction needs contact with code, telemetry, incidents, and user feedback. Review critical changes, trace failures across boundaries, and understand the cost of operating the proposed design. This keeps architecture grounded in what teams can build and support.\n\n## Grow ownership rather than collecting it\n\nDelegate complete decisions with context, constraints, and success criteria. Create opportunities for engineers to lead design reviews, incident analysis, and cross-team delivery. Offer feedback that explains the reasoning, not only the preferred answer.\n\nThe goal is a system that continues to make good decisions without routing every question through one staff engineer.\n\n## Communicate at the decision level\n\nDifferent audiences need different detail, but the underlying facts should stay consistent. Explain the problem, constraints, options, decision, evidence, and next checkpoint. Avoid presenting implementation activity as an outcome.\n\n## Sources and further reading\n\n- [Architecture Decision Records](https://adr.github.io/)\n- [Google SRE resources](https://sre.google/resources/)\n- [Team Topologies](https://teamtopologies.com/key-concepts)\n",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Elastic support for OpenTelemetry and managed OTLP ingestion",
      "canonical": "https://doshidhruv.com/notes/elastic-support-for-opentelemetry-and-managed-otlp-ingestion/",
      "datePublished": "2026-07-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Elastic",
        "OpenTelemetry",
        "OTLP"
      ],
      "description": "Elastic supports OpenTelemetry through managed OTLP ingestion, the Elastic Distribution of OpenTelemetry, and standard SDK or collector pipelines. Organisations can send traces,…",
      "contentMarkdown": "Elastic supports OpenTelemetry through managed OTLP ingestion, the Elastic Distribution of OpenTelemetry, and standard SDK or collector pipelines. Organisations can send traces, metrics, and logs while preserving OpenTelemetry semantic conventions and resource attributes.\n\n## Managed OTLP endpoint\n\nElastic Cloud provides a [managed OTLP endpoint](https://www.elastic.co/docs/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint) for Elastic Serverless and Elastic Cloud Hosted. OpenTelemetry SDKs or collectors can send OTLP with API-key authorization, while Elastic operates ingestion, processing, scaling, and storage.\n\nThe managed endpoint stores OpenTelemetry data without requiring an application-specific Elastic exporter. This provides a direct standards-based path for workloads that do not need an organisation-operated gateway.\n\n## Elastic Distribution of OpenTelemetry\n\nEDOT packages collectors, SDK options, and configuration for Elastic environments while remaining based on upstream OpenTelemetry. It is useful for Kubernetes, host metrics, application instrumentation, and pipelines that need local processing before export.\n\nAn upstream collector is still a valid choice when the organisation has its own distribution, release process, or multi-vendor routing requirements. Evaluate support boundaries: an Elastic distribution may provide tested integration and faster access to Elastic features, while an upstream distribution may reduce vendor-specific operating assumptions.\n\n## Preserve schema and query behavior\n\nUse OpenTelemetry semantic conventions consistently and test how resource and event attributes appear in Elastic data views. Stable service, environment, version, deployment, host, cloud, and Kubernetes attributes enable correlation without rewriting instrumentation.\n\nHigh-cardinality fields can increase storage and query cost. Keep metric attributes bounded and choose index, retention, and data-tier policy according to signal value. Security and audit logs may need different lifecycle and access controls from application traces.\n\n## Authenticate and route safely\n\nStore Elastic API keys at the collector or workload-secret boundary, scope them to the required destination, and rotate them. Prefer regional gateways when traffic requires classification, redaction, buffering, or multiple destinations. Use TLS and restrict local receivers to intended networks or workload identities.\n\n## Verify end-to-end behavior\n\nSend representative traces, metrics, logs, exemplars, and resource attributes through the proposed path. Confirm timestamp handling, span relationships, log severity, metric temporality, histogram queries, service correlation, ingest latency, partial failure, and retention.\n\nMonitor collector health and Elastic intake together. A healthy collector exporter does not prove that indexed data is complete or queryable. Maintain golden telemetry fixtures and run them after distribution, pipeline, or backend upgrades.\n\nElastic’s OpenTelemetry support can preserve a portable instrumentation boundary while providing Elastic-native storage and analysis. The architecture remains vendor-neutral when semantic ownership, processing policy, and the ability to route elsewhere stay outside the backend.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Dynatrace support for OpenTelemetry and native OTLP",
      "canonical": "https://doshidhruv.com/notes/dynatrace-support-for-opentelemetry-and-native-otlp/",
      "datePublished": "2026-07-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Dynatrace",
        "OpenTelemetry",
        "OTLP"
      ],
      "description": "Dynatrace supports OpenTelemetry through native OTLP API endpoints, the upstream OpenTelemetry Collector, and a Dynatrace collector distribution. It also supports mixed environments…",
      "contentMarkdown": "Dynatrace supports OpenTelemetry through native OTLP API endpoints, the upstream OpenTelemetry Collector, and a Dynatrace collector distribution. It also supports mixed environments where OpenTelemetry instrumentation and Dynatrace OneAgent capabilities coexist.\n\n## Choose among three ingestion paths\n\nDynatrace’s [OpenTelemetry documentation](https://docs.dynatrace.com/docs/ingest-from/opentelemetry) identifies three principal approaches:\n\n- direct OTLP export for simple environments without central processing;\n- a standard OpenTelemetry Collector when an organisation already operates upstream collectors;\n- the Dynatrace OTel Collector for a supported distribution and Dynatrace-oriented configuration.\n\nThe choice affects who owns scaling, processing, enrichment, configuration, and upgrade support. A regional gateway is usually preferable when telemetry needs redaction, sampling, routing, or protocol conversion.\n\n## Account for protocol details\n\nDynatrace’s native OTLP endpoints accept standard signal paths for traces, metrics, and logs. The [OTLP endpoint reference](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/otlp-api) documents HTTP/protobuf export, API-token authorization, signal-specific scopes, endpoint formats, and current limitations. A collector can receive OTLP over gRPC internally and export OTLP/HTTP to Dynatrace.\n\nDo not assume a successful HTTP response means every record was accepted. Observe partial-success responses and backend ingest metrics, and validate rejected or transformed data.\n\n## Preserve semantic meaning\n\nDynatrace maps OpenTelemetry semantic conventions into its semantic model. Use current upstream attributes for service, cloud, messaging, database, HTTP, and deployment identity. Apply vendor-specific attributes additively only where they improve a required topology or analysis feature.\n\nMetric temporality and histogram representation need particular testing. Backend mapping may not support every aggregation form identically. Validate the queries and objectives that depend on those metrics before migration.\n\n## Design the production gateway\n\nConfigure batching and compression, memory limits, queued retry, authentication, TLS, sensitive-data processing, and a bounded failure policy. Monitor incoming records, dropped data, request size, exporter failures, queue saturation, and end-to-end latency.\n\nUse distinct tokens and endpoints by environment and tenant. Restrict ingest scopes to the required signals. When routing to both Dynatrace and another backend, measure the additional collector capacity and do not assume both exporters fail or recover at the same rate.\n\n## Verify the integration\n\nBuild a repeatable fixture that emits successful and failed spans, correlated logs, counters, histograms, and the resource attributes used to identify a service. Check the fixture after every collector, semantic-convention, or backend upgrade. Verify not only ingestion but also the service topology, error classification, units, aggregation, dashboard queries, and alert conditions.\n\nDuring migration, compare OpenTelemetry data with any OneAgent-derived view and document which features depend on proprietary enrichment. This prevents an accidental promise that an open ingestion path and a vendor-specific agent produce identical context. It also makes later portability decisions measurable: teams know which capabilities move with OTLP and which require a replacement design.\n\nDynatrace provides first-class OpenTelemetry ingestion while retaining proprietary enrichment and analytics. A sound architecture uses the open path deliberately and documents where Dynatrace-specific behavior is required for the desired operational experience.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Measure whether an internal platform creates leverage",
      "canonical": "https://doshidhruv.com/notes/measure-whether-an-internal-platform-creates-leverage/",
      "datePublished": "2026-07-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Platform Engineering",
        "Metrics",
        "Developer Experience"
      ],
      "description": "An internal platform can look successful while moving work from one team to another. Repository count, portal visits, and cluster utilisation show activity, but they do not prove that…",
      "contentMarkdown": "An internal platform can look successful while moving work from one team to another. Repository count, portal visits, and cluster utilisation show activity, but they do not prove that product delivery is faster, safer, or easier.\n\n## Use a balanced model\n\nMeasure four connected areas:\n\n1. **User outcomes:** time to create, deploy, observe, and recover a service.\n2. **Adoption and retention:** voluntary use by eligible teams and continued use after first contact.\n3. **Reliability and support:** platform objectives, failed workflows, support demand, and time to restore.\n4. **Organisational economics:** duplicated work removed, operating cost, upgrade effort, and team capacity returned to product work.\n\nSegment measures by service type and team maturity. An average can hide a path that works for simple services and fails for regulated or data-intensive ones.\n\n## Follow complete journeys\n\nInstrument the developer workflow from request to production outcome. A self-service form that completes in seconds is not fast if an approval waits three days behind it. Measure elapsed time, active effort, handoffs, retries, and failure reasons.\n\nCombine telemetry with interviews and observation. Developers may work around the platform in ways that usage analytics cannot see. Repeated support questions can reveal an unclear contract; silent abandonment may be worse than high ticket volume.\n\nThe [DORA research program](https://dora.dev/research/) provides validated measures for software-delivery performance. Platform teams can use those outcomes while carefully testing contribution: many product, process, and organisational factors affect delivery metrics.\n\n## Define guardrails\n\nOptimising provisioning speed must not weaken security or reliability. Pair speed and adoption with policy compliance, change-failure rate, service objectives, and incident evidence. Track cost per useful workload rather than total spend alone.\n\nPublish platform objectives and status so teams can judge whether the dependency is trustworthy. Treat platform incidents as product incidents with reviews and follow-up.\n\n## Make metrics actionable\n\nEvery measure needs an owner, decision, and review cadence. Remove metrics that never change a priority. Set hypotheses for platform investments and compare the result with the baseline.\n\nThe platform creates leverage when teams spend less effort on undifferentiated infrastructure while operating services more safely. Measurement should make that transfer of time, risk, and responsibility visible.\n\n## Define the eligible population\n\nAdoption rates are meaningless without a denominator. Identify which teams and workload types the platform is designed to serve, which are in migration, and which legitimately require another path. Report first use, sustained use, and abandonment separately. Mandatory registration should not be counted as successful product adoption.\n\nMeasure time to first successful outcome for a new user and for an experienced team. Include waiting, approvals, failed attempts, and support time. A workflow may be fast once configured while onboarding remains the dominant cost.\n\n## Use leading and lagging signals\n\nLeading signals include documentation success, workflow completion, provisioning time, support demand, upgrade adoption, and repeated exceptions. Lagging outcomes include delivery performance, reliability, security incidents, and total operating cost. Use both: leading signals help the platform team act quickly, while lagging outcomes check whether local improvements matter to the organisation.\n\nAvoid claiming direct causation from a platform change without accounting for team, product, and process differences. Compare cohorts, observe before and after, and combine quantitative evidence with user research.\n\n## Measure cognitive load\n\nAsk teams which decisions they still need to understand, which failures they can diagnose, and which platform abstractions leak during operation. A platform that hides everything during normal use but becomes opaque in an incident can increase risk. Track whether ownership and escalation are clear at the point of failure.\n\n## Scorecard checklist\n\nA useful platform scorecard includes eligible and active users, journey completion, elapsed and active time, workflow reliability, objective performance, support and exception themes, upgrade lag, unit cost, security and reliability outcomes, and qualitative evidence. Every metric should map to a decision. Retire vanity measures that remain green while users continue to build around the platform.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Datadog support for OpenTelemetry and OTLP ingestion",
      "canonical": "https://doshidhruv.com/notes/datadog-support-for-opentelemetry-and-otlp-ingestion/",
      "datePublished": "2026-06-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Datadog",
        "OpenTelemetry",
        "OTLP"
      ],
      "description": "Datadog accepts OpenTelemetry data through several integration paths, but the resulting product capabilities are not identical. A design should choose explicitly between the Datadog…",
      "contentMarkdown": "Datadog accepts OpenTelemetry data through several integration paths, but the resulting product capabilities are not identical. A design should choose explicitly between the Datadog Agent, an OpenTelemetry Collector with Datadog components, and direct OTLP intake.\n\n## Agent and collector paths\n\nThe Datadog Agent can receive OTLP traces and metrics over gRPC or HTTP, and current supported Agent versions also receive OTLP logs. This allows applications to use OpenTelemetry SDKs while retaining local Datadog enrichment and integration.\n\nAn upstream collector with the Datadog exporter—or Datadog’s collector distribution—creates a more explicit pipeline boundary. It is appropriate when teams require central processors, multi-destination routing, or an existing collector operating model.\n\nDatadog also provides [direct OTLP intake endpoints](https://docs.datadoghq.com/opentelemetry/setup/otlp_ingest/) for environments where a collector or agent is impractical. Datadog currently recommends the Agent or Collector for production workloads because they provide metadata enrichment, normalization, and centralized sampling; direct intake has documented limits and feature differences.\n\n## Understand product compatibility\n\nOTLP ingestion does not automatically enable every Datadog feature. Some proprietary security, profiling, runtime, or ingestion capabilities rely on Datadog-specific instrumentation or local Agent behavior. Review the current feature-compatibility matrix for each signal and language rather than assuming parity.\n\nKeep the application on OpenTelemetry APIs where portability is required. Add Datadog-specific instrumentation only for a named capability whose value justifies the coupling.\n\n## Configure identity and mapping\n\nSet stable `service.name`, environment, version, deployment, host, container, and cloud resource attributes. Confirm how they map to Datadog service, host, unified service tagging, and infrastructure views. Missing host metadata can affect infrastructure correlation even when OTLP intake succeeds.\n\nUse API keys only at the exporter or Agent boundary and load them from a protected secret mechanism. Applications should send to a local trusted endpoint where possible rather than sharing backend credentials.\n\n## Validate limits and failure behavior\n\nDirect OTLP endpoints enforce payload-size limits that vary by signal. Batching, compression, and flush behavior need to stay within those limits. Monitor exporter rejections, partial success, throttling, queue capacity, and dropped telemetry.\n\nTest metric types and temporality, histogram mapping, log severity, trace sampling, resource correlation, and redaction. Compare backend results with a representative golden signal set after every Agent, collector, or exporter upgrade.\n\nDatadog has meaningful OpenTelemetry interoperability, but vendor neutrality still comes from owning the signal contract and routing boundary. The backend can enrich and analyse standard telemetry without becoming the only place the organisation knows how that telemetry is produced.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Splunk support for OpenTelemetry collection and OTLP",
      "canonical": "https://doshidhruv.com/notes/splunk-support-for-opentelemetry-collection-and-otlp/",
      "datePublished": "2026-06-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Splunk",
        "OpenTelemetry",
        "OTLP"
      ],
      "description": "Splunk supports OpenTelemetry through the Splunk Distribution of the OpenTelemetry Collector, standard OTLP ingestion paths, and integrations with Splunk Observability Cloud and Splunk…",
      "contentMarkdown": "Splunk supports OpenTelemetry through the Splunk Distribution of the OpenTelemetry Collector, standard OTLP ingestion paths, and integrations with Splunk Observability Cloud and Splunk platform products. The main architectural decision is whether to use upstream components, Splunk’s supported distribution, or a layered combination.\n\n## The Splunk collector distribution\n\nSplunk’s distribution packages upstream collector components with default configuration, deployment tooling, receivers, processors, exporters, and Splunk-specific integrations. It can run in agent mode near hosts or workloads and gateway mode as an aggregation tier.\n\nThe distribution exposes standard OTLP receivers over gRPC and HTTP and can receive other formats needed during migration. This makes it possible to standardise application instrumentation before every legacy signal source has moved.\n\n## OTLP inside the pipeline\n\nSplunk documents an [OTLP exporter](https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-components/exporters/otlp-exporter) for traces, metrics, and logs. OTLP is useful both from applications to collectors and between agent and gateway tiers.\n\nFor production, configure TLS, authentication, batching, queued retry, memory limiting, and health endpoints. Expose receivers only on intended interfaces. A default collector configuration is a starting point, not an enterprise security boundary.\n\n## Preserve resource identity\n\nApply OpenTelemetry semantic conventions for service identity and add deployment, environment, cloud, Kubernetes, and host attributes through trusted resource detectors. Splunk-specific enrichment may improve backend experiences, but portable attributes should remain intact so the same signal can be routed elsewhere.\n\nDo not put unbounded values such as user or request identifiers into metric dimensions. Test how log severity, metric temporality, span status, and resource attributes appear after backend mapping.\n\n## Size for failure conditions\n\nCollector demand changes with receiver mix, enabled processors, batch size, cardinality, sampling, and export latency. Benchmark the actual configuration. Monitor accepted and refused records, queue utilisation, memory limiter action, dropped data, and failed exports.\n\nAn outage in Splunk Observability Cloud should not synchronously block instrumented applications. Use local offload, bounded retry, and an explicit data-loss policy. Add durable transport when particular audit or security events cannot be lost.\n\n## Maintain a vendor-neutral boundary\n\nKeep reusable receiver and processing policy separate from the final Splunk exporter where practical. Version the distribution, upstream component set, and configuration. Test upgrades against representative signal fixtures.\n\n## Migration checklist\n\nCapture a small golden dataset containing normal requests, errors, high-cardinality attributes, exemplars, and correlated logs. Send it through the proposed collector topology and compare the resulting Splunk fields, service identity, timestamps, units, and trace relationships with the source data. This catches semantic loss that a simple connectivity test cannot reveal.\n\nRoll out by workload class rather than changing every producer at once. Define the owner of agent configuration, gateway capacity, Splunk access tokens, schema changes, and rollback. Record any Splunk-only processors or attributes as explicit dependencies so a future routing decision is based on evidence instead of an assumption of perfect portability.\n\nSplunk’s distribution can reduce operational integration work, while standard OpenTelemetry APIs and OTLP preserve flexibility. The combination works best when teams know which layer is upstream-compatible, which is Splunk-specific, and which operational guarantees they own.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "AWS support for OpenTelemetry with ADOT and CloudWatch",
      "canonical": "https://doshidhruv.com/notes/aws-support-for-opentelemetry-with-adot-and-cloudwatch/",
      "datePublished": "2026-05-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "AWS",
        "ADOT",
        "OpenTelemetry",
        "CloudWatch",
        "X-Ray"
      ],
      "description": "AWS supports OpenTelemetry through several paths: the AWS Distro for OpenTelemetry, the CloudWatch agent, upstream or custom collectors, and CloudWatch OTLP endpoints. The right choice…",
      "contentMarkdown": "AWS supports OpenTelemetry through several paths: the AWS Distro for OpenTelemetry, the CloudWatch agent, upstream or custom collectors, and CloudWatch OTLP endpoints. The right choice depends on signal coverage, AWS-specific enrichment, operational ownership, and the degree of vendor-neutral control required.\n\n## ADOT and upstream OpenTelemetry\n\nThe [AWS Distro for OpenTelemetry](https://docs.aws.amazon.com/xray/latest/devguide/xray-services-adot.html) packages upstream OpenTelemetry components that AWS tests, optimises, secures, and supports. It includes SDK and auto-instrumentation options plus collector distributions for AWS environments.\n\nADOT can send traces and metrics to AWS destinations including X-Ray, CloudWatch, Amazon Managed Service for Prometheus, and OpenSearch. Because it remains based on OpenTelemetry APIs and OTLP, an organisation can keep common instrumentation while selecting AWS or non-AWS exporters at a collector boundary.\n\n## Choose an ingestion path\n\nAWS documents several [CloudWatch OpenTelemetry paths](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OTLPGettingStarted.html):\n\n- the CloudWatch agent for integrated logs, metrics, traces, resource correlation, and Application Signals;\n- an upstream OpenTelemetry Collector for a standard pipeline;\n- a custom collector build when AWS components and additional processors are required;\n- collectorless ADOT SDK export for supported signals and constrained environments.\n\nThese paths are not feature-equivalent. Collectorless operation reduces infrastructure but may omit log support or resource correlation available through the agent. An upstream collector preserves control but may require explicit AWS enrichment and configuration.\n\n## Authenticate with workload identity\n\nUse IAM roles for EC2, ECS, EKS, or Lambda rather than static access keys. Grant only the actions required by the chosen OTLP or service exporter. When a central gateway serves several accounts, preserve source account, region, resource, cluster, and service identity and prevent one tenant from selecting another tenant’s credentials or destination.\n\n## Preserve portable semantics\n\nAdopt OpenTelemetry semantic conventions and standard resource attributes at instrumentation time. Apply AWS resource detection in the collector so EC2, ECS, EKS, or Lambda context is added from trusted environment data. Keep vendor-specific attributes additive; do not replace portable service identity with an AWS-only dimension.\n\n## Operate and test the path\n\nMonitor collector queueing, refused telemetry, exporter errors, throttling, credential failure, and CloudWatch or X-Ray ingestion limits. Confirm metric temporality and units after export, trace-service mapping, log correlation, and retention. Test region loss and destination throttling under load.\n\n## Migration checklist\n\nStart with one representative service and record its expected traces, metrics, logs, resource attributes, and correlations before changing the pipeline. Run the AWS destination beside the existing backend long enough to compare service maps, alert inputs, sampling decisions, and ingestion cost. Then rehearse credential rotation, collector restart, endpoint failure, and rollback.\n\nFor EKS or multi-account estates, decide explicitly where agents end and shared gateways begin. Keep account and region boundaries visible in configuration, dashboards, and access controls. A migration is complete only when teams can diagnose missing telemetry, not merely when data appears in CloudWatch.\n\nAWS provides strong OpenTelemetry integration, but portability still depends on the architecture an organisation owns: standard APIs in applications, explicit semantic contracts, controlled enrichment, and collectors that can route to more than one destination when required.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Microsoft 365 audit data in an OpenTelemetry pipeline",
      "canonical": "https://doshidhruv.com/notes/microsoft-365-audit-data-in-an-opentelemetry-pipeline/",
      "datePublished": "2026-05-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Microsoft 365",
        "OpenTelemetry",
        "Audit Logs",
        "Data Engineering"
      ],
      "description": "Microsoft 365 audit data is valuable operational and security telemetry, but Microsoft 365 is not itself a general native OTLP source. The reliable architecture is to retrieve tenant…",
      "contentMarkdown": "Microsoft 365 audit data is valuable operational and security telemetry, but Microsoft 365 is not itself a general native OTLP source. The reliable architecture is to retrieve tenant audit records through supported Microsoft APIs, preserve their audit semantics, normalise them into an organisational event contract, and then route them through an OpenTelemetry-compatible pipeline.\n\n## Understand the source boundary\n\nThe [Office 365 Management Activity API](https://learn.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-reference) exposes user, administrator, system, and policy activity from workloads including Microsoft Entra ID, Exchange, SharePoint, and other Microsoft 365 services. Collection requires unified audit logging, Microsoft Entra application identity, appropriate permissions, and subscriptions to relevant content types.\n\nThe API returns tenant-specific content blobs. A collector service can poll for available content or receive webhook notifications and then download the records. The cursor, subscription state, and downloaded content identifiers need durable storage so retries do not create gaps or uncontrolled duplicates.\n\n## Preserve audit semantics\n\nThe Microsoft common schema includes creation time, record type, operation, user identity, client address, object identity, result, workload, and service-specific fields. Do not flatten everything into an unstructured message. Map stable common fields to a governed log schema and retain the original record or source-specific attributes where policy permits.\n\nUseful OpenTelemetry resource attributes describe the collector service, tenant boundary, cloud, region, and environment. Log record attributes can hold workload, operation, result, record type, user type, and source event identifier. Event time must come from the audit record; observed time should record when the pipeline received it.\n\n## Design for delay and duplication\n\nAudit content can arrive after the activity occurred and may not be ordered globally. Use source identifiers and content metadata for deduplication. Maintain a look-back window to recover late content, but bound it and monitor duplicate rate. Do not use arrival time as the business event time.\n\nTrack the most recent successfully retrieved window per tenant and content type. Alert on subscription failure, authorization change, webhook silence, polling lag, content-download failure, parsing errors, and unexpected volume changes.\n\n## Apply tenant and regional policy\n\nCredentials, checkpoints, buffers, and output records must remain tenant-scoped. Route records using declared geography and data classification before they reach a shared backend. Redact or tokenize personal identifiers only according to investigation and regulatory requirements; excessive redaction can make audit data unusable, while unrestricted replication creates privacy risk.\n\n## Microsoft OpenTelemetry support is adjacent\n\nAzure Monitor supports OpenTelemetry instrumentation and OTLP ingestion, and Microsoft publishes an OpenTelemetry distribution for application telemetry. Those capabilities can observe the collection service and receive its normalised output, but they do not replace the Microsoft 365 audit extraction API.\n\nThe resulting design is a bridge: supported Microsoft 365 extraction on one side, an explicit audit-event contract in the middle, and vendor-neutral routing on the other. Keeping those boundaries clear makes completeness, security, and backend portability independently testable.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "OpenTelemetry pipeline architecture for vendor-neutral observability",
      "canonical": "https://doshidhruv.com/notes/opentelemetry-pipeline-architecture-for-vendor-neutral-observability/",
      "datePublished": "2026-04-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "OpenTelemetry",
        "OTLP",
        "Collector"
      ],
      "description": "OpenTelemetry standardises how applications produce and transmit traces, metrics, and logs. It does not make every backend identical, and it does not operate the telemetry path…",
      "contentMarkdown": "OpenTelemetry standardises how applications produce and transmit traces, metrics, and logs. It does not make every backend identical, and it does not operate the telemetry path automatically. Vendor neutrality comes from controlling instrumentation, semantic contracts, routing, and export boundaries.\n\n## Separate API, SDK, protocol, and collector\n\nThe API is what application and library code calls. The SDK records, samples, processes, and exports signals. OTLP is the protocol used to transmit OpenTelemetry data. The Collector is a separate service that receives, processes, and exports telemetry.\n\nKeeping application code on OpenTelemetry APIs and semantic conventions reduces vendor coupling. Sending OTLP through a controlled collector tier keeps credentials, retries, filtering, enrichment, and destination choice outside each workload.\n\nThe [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is explicitly designed as a vendor-agnostic receiver, processor, and exporter. A pipeline connects those components for each signal.\n\n## Use agent and gateway tiers deliberately\n\nAn agent collector runs close to a workload or host. It can receive local OTLP, collect host data, add trusted resource attributes, and offload the application quickly. A gateway tier aggregates many agents, applies central policy, performs tail sampling, and routes to one or more destinations.\n\nNot every environment needs both. Serverless or managed platforms may export directly. Large estates benefit from separating local collection from regional or institutional routing. Define failure behavior and capacity for each tier.\n\n## Make semantics a contract\n\nStandardise service name, namespace, version, environment, region, owner, data class, and deployment identity. Adopt upstream semantic conventions before creating local attributes. Version organisational extensions and reject high-cardinality metric attributes such as request or customer identifiers.\n\nLogs should preserve event time, severity, body, trace correlation, source identity, and schema. Traces need stable operation names and correct context propagation. Metrics need unit, temporality, aggregation intent, and bounded dimensions.\n\n## Route without losing control\n\nA central pipeline can send operational metrics to one backend, regulated logs to a regional store, sampled traces to several analysis systems, and an audit copy to durable storage. Routing policy should be based on declared attributes with tested defaults. Unknown classifications should fail visibly rather than fall into an unrestricted destination.\n\nUse memory limiting, batching, queued retry, and backpressure controls. Collector queues are not automatically durable; if loss is unacceptable, place a durable transport or storage boundary in the path. Monitor accepted, refused, dropped, and failed-export counts plus queue size and processing latency.\n\n## Understand backend compatibility\n\nMicrosoft Azure Monitor, AWS CloudWatch and X-Ray, Splunk Observability Cloud, Datadog, Dynatrace, and Elastic all provide supported OpenTelemetry paths, but the paths differ. Some accept native OTLP endpoints; some recommend a vendor distribution or agent for enrichment; signal coverage, protocol, authentication, metadata, and product-feature parity vary.\n\nTherefore the portability contract should be “standards-based instrumentation with controlled translation,” not “all data looks identical everywhere.” Test a representative dataset against each destination and preserve raw semantic meaning before applying vendor-specific mapping.\n\n## Production checklist\n\nValidate SDK lifecycle, context propagation, collector availability, TLS and authentication, resource identity, sensitive-data removal, cardinality, sampling, retry, destination limits, cost, and pipeline self-observability. Test destination loss and configuration rollback. A vendor-neutral platform is credible when teams can change routing centrally without reinstrumenting every application.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Pattern matching algorithms for architecture recommendations",
      "canonical": "https://doshidhruv.com/notes/pattern-matching-algorithms-for-architecture-recommendations/",
      "datePublished": "2026-03-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Algorithms",
        "Pattern Matching",
        "Recommendation Systems",
        "Architecture"
      ],
      "description": "An architecture recommendation platform must translate incomplete requirements into a small set of compatible patterns and services while explaining why each result fits. This is not…",
      "contentMarkdown": "An architecture recommendation platform must translate incomplete requirements into a small set of compatible patterns and services while explaining why each result fits. This is not ordinary text search. It is constrained pattern matching over structured facts, rules, relationships, and evidence.\n\n## Represent the problem\n\nModel a request as typed features rather than one free-text string. Useful dimensions include workload type, interaction style, data classification, region, availability objective, recovery objective, throughput, latency, integration protocol, identity model, deployment boundary, and approved exceptions.\n\nRepresent each candidate pattern with:\n\n- required features that must match;\n- prohibited conditions that disqualify it;\n- optional capabilities that improve fit;\n- compatible component roles and service implementations;\n- controls and evidence the completed blueprint must satisfy;\n- lifecycle, ownership, and version metadata.\n\nMissing input should remain unknown rather than becoming false. A recommendation engine that treats “not provided” as “not required” can select an unsafe pattern.\n\n## Filter before scoring\n\nUse hard constraints first. Remove candidates that violate data residency, security classification, protocol, lifecycle, availability, or organisational policy. This reduces the search space and prevents a high soft score from overpowering a mandatory control.\n\nThen calculate a fit score for surviving candidates. A simple weighted model can be expressed as:\n\n`score(pattern) = Σ weight(feature) × match(feature, pattern) − penalties`\n\nMatch functions can be exact, hierarchical, numeric-range, set-overlap, or relationship-aware. Availability tiers may use ordered comparison; regions use set membership; capabilities may use taxonomy distance; topology rules need graph matching.\n\n## Match graphs, not only attributes\n\nArchitecture patterns contain relationships: a public client reaches a gateway, the gateway authenticates through an identity provider, a service writes to a classified data store, and telemetry leaves through an approved route. A candidate component can satisfy its local attributes yet create an invalid combination.\n\nRepresent the requested blueprint and pattern as graphs. Match node roles and relationship types, then validate edge constraints such as protocol, trust boundary, direction, and permitted deployment. Full subgraph isomorphism can be expensive, but enterprise patterns usually have typed nodes and small bounded graphs; domain-specific pruning makes matching practical.\n\n## Produce an explanation trace\n\nEvery recommendation should preserve:\n\n- matched requirements;\n- disqualifying rules applied to rejected candidates;\n- score contribution by feature;\n- assumptions caused by missing data;\n- control evidence still required;\n- pattern and catalogue versions used.\n\nThis trace supports review, debugging, and challenge. It also allows platform owners to see whether a surprising result came from input, taxonomy, weight, rule, or stale service metadata.\n\n## Validate with cases and counterexamples\n\nBuild a test corpus of approved architectures, invalid combinations, boundary conditions, and historical exceptions. Test that the expected pattern appears, prohibited options never appear, ranking remains stable after unrelated changes, and every result has a complete explanation.\n\nMeasure top-k recall, invalid-recommendation rate, reviewer acceptance, override reasons, and change between engine versions. Human overrides should feed taxonomy and rule review rather than train an opaque score automatically.\n\nFor an Architecture Solution Blueprint platform, the algorithm should make architectural judgment repeatable without pretending it is purely mathematical. Constraints protect mandatory boundaries, scoring orders credible choices, graph validation checks the assembled design, and explanations keep the decision accountable.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Production RAG requires retrieval evidence and control",
      "canonical": "https://doshidhruv.com/notes/production-rag-requires-retrieval-evidence-and-control/",
      "datePublished": "2026-02-15",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "RAG",
        "LLM",
        "Retrieval",
        "Architecture"
      ],
      "description": "Retrieval augmented generation is often described as “search, then prompt.” A production RAG system is more accurately a governed information retrieval system connected to a…",
      "contentMarkdown": "Retrieval-augmented generation is often described as “search, then prompt.” A production RAG system is more accurately a governed information-retrieval system connected to a probabilistic synthesizer. Its quality depends on corpus ownership, permission-aware retrieval, evidence coverage, and explicit behavior when sources are insufficient.\n\n## Define the answer contract\n\nState what the system may answer, which sources are authoritative, how current information must be, and when it must abstain. Separate three output types:\n\n- a statement directly supported by retrieved evidence;\n- an inference that combines evidence but is not stated verbatim;\n- general model knowledge that is outside the governed corpus.\n\nFor enterprise architecture recommendations, general model knowledge should not silently override approved standards or service metadata. The application should prefer current governed sources, mark inference, and refuse when the required evidence is missing.\n\n## Build a permission-preserving corpus\n\nIngestion must retain source identity, owner, version, classification, effective dates, and access policy through parsing and chunking. Every indexed chunk needs enough metadata to reconstruct its origin and evaluate whether the requesting principal may retrieve it.\n\nDeletion is part of ingestion. When a source is removed or access changes, derived chunks, embeddings, caches, and replicas must be updated within a defined objective. Otherwise the vector index becomes an uncontrolled copy of protected information.\n\n## Use hybrid retrieval\n\nSemantic search is useful for conceptual similarity, but exact terms matter for architecture: product identifiers, control numbers, region codes, runtime versions, and error messages. Combine lexical retrieval, vector similarity, metadata filters, and optional reranking.\n\nA practical pipeline is:\n\n1. normalise the query and identify required filters;\n2. enforce tenant and document authorization;\n3. retrieve lexical and vector candidates;\n4. merge and rerank the candidates;\n5. remove duplicates and allocate context across sources;\n6. test whether evidence crosses the answer threshold;\n7. generate with citations or abstain.\n\n## Evaluate retrieval separately\n\nMaintain representative questions with expected source documents and supported claims. Measure whether relevant evidence appears in the candidate set, whether it ranks high enough to reach the prompt, and whether citations actually support the answer.\n\nEnd-to-end “answer quality” is not diagnostic. A wrong answer may come from a missing document, stale ingestion, weak filtering, poor ranking, context truncation, unsupported synthesis, or an application defect. Measure each boundary so the team knows what to change.\n\n## Protect the generation boundary\n\nRetrieved documents are untrusted input. They can contain instructions, malformed markup, hidden text, or examples that resemble system policy. Keep application instructions separate, label source content, constrain tool use outside the model, and validate structured output before it reaches another system.\n\nFor an architecture platform, use deterministic rules for mandatory controls and compatibility constraints. RAG can explain a recommendation, locate standards, and assemble evidence; it should not replace hard authorization or certification checks.\n\n## Operate the complete system\n\nVersion the model, prompt, index, parser, embedding model, reranker, retrieval configuration, and policy. Record source identifiers and decision metadata with privacy controls. Monitor ingestion freshness, empty retrieval, citation coverage, latency, cost, authorization denials, and user-confirmed failures.\n\nThe [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) provides a useful Govern–Map–Measure–Manage structure for the complete system. The engineering objective is evidence-bounded assistance: useful when sources support it, explicit when inference is involved, and safe when the corpus cannot answer.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "FINOS CALM and architecture as code",
      "canonical": "https://doshidhruv.com/notes/finos-calm-and-architecture-as-code/",
      "datePublished": "2026-01-15",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "FINOS CALM",
        "Architecture as Code",
        "Governance"
      ],
      "description": "Architecture diagrams are useful for communication, but a diagram alone is difficult to validate, compare, or connect to delivery controls. The Common Architecture Language…",
      "contentMarkdown": "Architecture diagrams are useful for communication, but a diagram alone is difficult to validate, compare, or connect to delivery controls. The Common Architecture Language Model—CALM—from FINOS treats architecture as structured, version-controlled data that tooling can validate, render, and govern.\n\nCALM is an open specification rather than a diagram format. Its [official introduction](https://calm.finos.org/introduction/) describes a machine-readable language and toolchain for keeping design intent closer to implementation.\n\n## The core model\n\nA CALM architecture describes several connected concepts:\n\n- **nodes** represent systems, services, databases, networks, actors, or other architectural elements;\n- **interfaces** describe interaction points exposed by nodes;\n- **relationships** express interaction, connection, deployment, or composition;\n- **controls** attach domain requirements and evidence to relevant elements;\n- **metadata and decorators** add business, deployment, security, or organisational context;\n- **patterns** provide reusable architectural structures and constraints.\n\nThe [CALM core concepts](https://calm.finos.org/core-concepts/) define this vocabulary. Stable identifiers are important because relationships, controls, generated views, and external systems must refer to the same element over time.\n\n## From diagram to decision model\n\nA visual line between an application and database may hide the protocol, identity, network boundary, data class, encryption requirement, and ownership. A structured relationship can make those properties explicit. CALM distinguishes interactions, technical connections, deployment, and composition, allowing tools to ask questions that a drawing cannot reliably answer.\n\nFor example, a platform can validate that a confidential-data flow uses an approved protocol, terminates at an approved interface, and has the required control configuration. The resulting diagram remains useful, but it becomes a view generated from governed architecture data rather than the only source.\n\n## Patterns and platform recommendations\n\nCALM patterns can define required nodes and relationships without fixing every product choice. An enterprise platform can combine a pattern with a service catalogue:\n\n1. capture workload requirements and constraints;\n2. select a compatible architecture pattern;\n3. match abstract pattern roles to approved enterprise services;\n4. apply controls to nodes and relationships;\n5. validate the completed architecture;\n6. generate diagrams, documentation, and review evidence.\n\nThis model is directly relevant to an Architecture Solution Blueprint platform. CALM can provide a portable representation at the architecture boundary, while the platform supplies organisation-specific recommendations, ownership, certification logic, and workflow.\n\n## Controls need evidence\n\nCALM controls distinguish a requirement from its configuration. A requirement might say that a connection must use an approved encrypted protocol; the configuration records how the architecture satisfies it. The [controls documentation](https://calm.finos.org/core-concepts/controls/) shows how schemas can define and validate this evidence.\n\nPassing schema validation does not prove the deployed system is compliant. A mature implementation links control configuration to infrastructure policy, tests, runtime inventory, or other evidence and detects drift between declared and observed state.\n\n## Adoption sequence\n\nStart with one recurring architecture pattern and a small vocabulary. Map existing platform services to node and interface definitions, encode a few high-value controls, validate in CI, and generate a view that teams already need. Avoid modelling the entire enterprise before proving that the representation improves a real decision.\n\nVersion schemas, patterns, and organisational extensions. Provide migration tooling when identifiers or constraints change. Treat exceptions as explicit, time-bound records rather than invalid models stored outside the system.\n\nCALM is most valuable when architecture data participates in delivery: a pull request can validate a change, a platform can recommend compatible services, and governance can review evidence tied to the same model engineers use.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Staff engineering is measured through leverage",
      "canonical": "https://doshidhruv.com/notes/staff-engineering-is-measured-through-leverage/",
      "datePublished": "2025-12-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Staff engineering",
        "Staff Engineering",
        "Technical Leadership",
        "Organisational Design"
      ],
      "description": "Staff level engineering is not defined by taking the hardest ticket or writing the most code. Its impact appears through the decisions, systems, and people that enable many teams to…",
      "contentMarkdown": "Staff-level engineering is not defined by taking the hardest ticket or writing the most code. Its impact appears through the decisions, systems, and people that enable many teams to deliver better outcomes after the individual contribution is complete.\n\n## Work at the constraint\n\nFind the recurring issue that limits several teams: an unclear ownership boundary, unreliable delivery path, fragmented identity model, missing operational standard, or decision that no single team can make alone. Confirm it with production evidence and the people doing the work.\n\nNot every broad problem belongs to a staff engineer. The work needs a clear outcome, sponsor or accountable owner, and enough organisational readiness to act. Otherwise influence becomes commentary without change.\n\n## Create durable mechanisms\n\nUseful outputs include a shared contract, migration path, reference implementation, decision record, platform capability, operating review, or mentoring structure. The artifact matters because it allows others to execute without repeatedly asking its author.\n\nWrite for the next engineer. Explain constraints and trade-offs, not only the final design. Build feedback into the mechanism so it improves after adoption.\n\n## Lead through context\n\nBring product, security, operations, data, and infrastructure constraints into the same decision. Surface disagreement early and distinguish a technical fact from a preference. Make a recommendation with consequences, then support the accountable decision even when another option is chosen.\n\nThe [StaffEng project](https://staffeng.com/guides/) documents several shapes of staff-plus work and the organisational context around them. Titles vary; the consistent theme is influence beyond one team or codebase.\n\n## Measure the system change\n\nLook for reduced decision latency, fewer repeated failures, faster onboarding, safer releases, clearer ownership, successful migration, and teams independently using the new path. Avoid claiming leverage from meeting count, document volume, or framework adoption alone.\n\nShare credit and make successors. If every important decision still requires the same individual, the work has created dependency rather than leverage.\n\nStaff engineering combines technical depth with organisational design. The strongest contribution is often a system in which many engineers can make good decisions with less coordination and lower risk.\n\n## Select work with a leverage test\n\nAsk whether the problem affects multiple teams, whether solving it changes a durable mechanism, whether you have access to the required decision-makers, and whether success can be observed. A complex isolated task may need senior technical skill without being the highest-leverage staff-level work.\n\nWrite a short problem statement with current evidence, affected groups, why local solutions have failed, and the decision required. This creates a boundary and prevents a broad initiative from becoming endless “alignment.”\n\n## Move between altitude levels\n\nStaff engineers must connect strategy to production detail. At high altitude, explain why the capability matters and which trade-offs the organisation is making. At low altitude, inspect interfaces, failure modes, migrations, and operational evidence deeply enough that the direction is credible. Remaining only at one level produces either disconnected vision or locally excellent work without organisational movement.\n\n## Build a coalition and succession\n\nIdentify owners from the teams that must implement and operate the result. Involve them in shaping the contract rather than presenting a completed design. Delegate meaningful decisions, document context, and create maintainers who can evolve the system without returning to its original author.\n\n## Personal review checklist\n\nPeriodically ask: Which recurring decision became easier? Which production risk decreased? Which team can now move independently? What operating burden was removed? Who else can lead the next phase? Also record work deliberately stopped; focus is part of leverage. If the primary outcome is that the staff engineer became busier or more central, the intervention probably needs redesign.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Architecture governance should operate through evidence",
      "canonical": "https://doshidhruv.com/notes/architecture-governance-should-operate-through-evidence/",
      "datePublished": "2025-10-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Architecture Governance",
        "Standards",
        "Enterprise Architecture"
      ],
      "description": "Architecture governance is useful when it improves decisions and makes system risk visible. It becomes harmful when every change waits for a central meeting that lacks the context to…",
      "contentMarkdown": "Architecture governance is useful when it improves decisions and makes system risk visible. It becomes harmful when every change waits for a central meeting that lacks the context to make the decision.\n\n## Match governance to impact\n\nClassify decisions by reversibility, blast radius, data sensitivity, external obligation, cost, and number of affected teams. Local and reversible choices should stay with the owning team. Cross-domain contracts, identity boundaries, regulated data, and difficult-to-reverse platform choices deserve broader review.\n\nPublish the thresholds. Teams should know before design begins which evidence is required and who has decision authority.\n\n## Provide executable standards\n\nA standard should include its purpose, scope, owner, required controls, reference implementation, validation method, exception path, and review date. Where possible, encode it in platform defaults, policy tests, templates, and CI checks. A slide deck that engineers must manually interpret will drift.\n\nUse a small number of principles to guide cases a standard does not cover. Keep a catalogue of approved patterns and their boundaries, but avoid presenting patterns as universal solutions.\n\n## Review evidence, not presentation quality\n\nAsk for context, alternatives, threat model, reliability objectives, data flows, operating model, migration plan, and measurable consequences in proportion to risk. Link the decision record and resulting controls to the system inventory.\n\nNIST’s [Secure Software Development Framework](https://csrc.nist.gov/Projects/ssdf) demonstrates how governance outcomes can be organised as practices and tasks rather than tied to one implementation. Architecture governance can use the same approach: state the required outcome and provide evidence-backed ways to satisfy it.\n\n## Manage exceptions as information\n\nExceptions need an owner, rationale, compensating controls, expiry, and review trigger. Repeated exceptions may indicate an unrealistic standard or a missing platform capability. Analyse them as product feedback.\n\nMeasure review lead time, recurring findings, exception age, control adoption, and production outcomes. Do not optimise for the number of approvals completed.\n\nThe purpose of governance is distributed decision quality. Central architects should supply context, patterns, and challenge for high-impact choices while platforms and automated evidence make the safe path routine.\n\n## Create a decision-rights model\n\nDefine who recommends, who supplies required expertise, who decides, and who must be informed for each class of change. A service owner may decide a reversible internal library choice. A cross-domain identity contract may require security and platform input with an accountable architecture owner. Clear decision rights prevent both central bottlenecks and decisions made without affected parties.\n\nTime-box review according to risk. Publish response objectives and an escalation path. If governance cannot review within the delivery window, teams will either wait unnecessarily or bypass it.\n\n## Build an evidence package\n\nFor higher-impact changes, collect system context, data flows, trust boundaries, objectives, alternatives, dependency and failure analysis, cost, migration sequence, and operating ownership. Reuse evidence already generated by code, infrastructure plans, threat models, tests, service catalogues, and production telemetry.\n\nEvidence must correspond to the actual release. A diagram from an early proposal is not proof that deployed network or identity controls match it. Link evidence to versions and automate drift detection where possible.\n\n## Maintain the standards portfolio\n\nEach standard needs an owner and review trigger such as provider change, repeated exceptions, incident evidence, or new regulation. Mark obsolete standards clearly and provide a migration path. Overlapping standards should be consolidated so teams do not have to interpret conflicting authority.\n\n## Effectiveness checklist\n\nMeasure decision lead time, percentage resolved at the appropriate level, repeated review findings, exception age, adoption of paved paths, and incidents related to governed risks. Sample approved systems to verify evidence and real operation still align. Effective governance reduces surprise and repeated analysis; it does not maximise the number of committees involved.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Technical roadmaps should preserve options",
      "canonical": "https://doshidhruv.com/notes/technical-roadmaps-should-preserve-options/",
      "datePublished": "2025-08-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Staff engineering",
        "Technical Strategy",
        "Roadmaps",
        "Leadership"
      ],
      "description": "A technical roadmap should explain how engineering investments change business capability and system risk. A list of technologies with quarterly dates is a delivery calendar, not a strategy.",
      "contentMarkdown": "A technical roadmap should explain how engineering investments change business capability and system risk. A list of technologies with quarterly dates is a delivery calendar, not a strategy.\n\n## Begin with outcomes and constraints\n\nState the current condition in evidence: delivery delay, reliability exposure, security gap, scaling limit, operating cost, or inability to support a product direction. Connect each investment to a measurable change. “Adopt Kubernetes” is not an outcome; “reduce environment provisioning from days to under an hour while enforcing workload identity” is.\n\nDocument constraints such as regulatory deadlines, vendor end-of-support, staffing, migration windows, and dependency ownership. This prevents a future reader from assuming every sequence was arbitrary.\n\n## Organise around decisions\n\nFor each horizon, identify the decisions that must be made, the evidence required, and the options kept open. Near-term work can be specific. Longer-term work should describe capability and direction without pretending uncertain implementation details are commitments.\n\nUse discovery milestones where uncertainty is high: prototype a control, measure a bottleneck, migrate one representative service, or validate recovery. The output should change a later decision, not merely demonstrate activity.\n\n## Show dependencies and stopping rules\n\nMake cross-team, vendor, security, and data dependencies visible with owners. Define what would cause the organisation to accelerate, pause, alter, or stop an initiative. This turns the roadmap into a decision instrument rather than a promise that survives regardless of evidence.\n\nArchitecture decision records can preserve the choices made along the way. The [Technology Radar model](https://www.thoughtworks.com/radar) is one useful way to express differing levels of confidence in technologies, but a roadmap must still connect those positions to the organisation’s actual systems and goals.\n\n## Review as a portfolio\n\nBalance feature enablement, reliability, security, cost, and retirement work. Count the operational burden removed, not only new capabilities added. A roadmap that continually adds platforms without decommissioning old ones increases cognitive load and risk.\n\nUpdate it when assumptions change, and retain the previous rationale. Stable outcomes with adaptable implementation are a sign of learning, not weak planning.\n\nA credible technical roadmap gives teams direction while preserving room for better evidence. It commits to problems and outcomes more strongly than to tools.\n\n## Use horizons instead of false precision\n\nA practical roadmap can separate **committed**, **planned**, and **exploratory** horizons. Committed work has an owner, capacity, dependency agreement, and acceptance evidence. Planned work has a defined outcome and sequence but may change as earlier evidence arrives. Exploratory work frames a problem and learning objective without promising an implementation date.\n\nThis language helps stakeholders distinguish confidence from importance. A high-value initiative can remain exploratory because the solution or dependency path is uncertain.\n\n## Express investments as hypotheses\n\nWrite: “If we provide a supported service template with automated identity and telemetry, then teams can reach production with fewer security exceptions and less setup time.” Attach a baseline, target signal, review date, and disconfirming evidence. Delivery of the template is an output; changed team behavior is the outcome.\n\nFor foundational work, show the downstream decision or capability it unlocks. A dependency map, prototype, or migration inventory is valuable when it reduces uncertainty for a named next step.\n\n## Manage capacity and retirement\n\nAllocate capacity across product enablement, reliability, security, platform improvement, and decommissioning. Make the cost of keeping old systems visible. A new platform milestone should often include migrated workloads and retired legacy paths, not only feature availability.\n\n## Review checklist\n\nEvery roadmap item should have a problem owner, desired outcome, evidence, confidence horizon, dependencies, risk, decision date, and stop condition. Review the portfolio at a stable cadence and after material assumption changes. Remove completed work, record abandoned directions with rationale, and resist rolling unfinished items forward without learning why they slipped.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Capacity planning begins with constraints",
      "canonical": "https://doshidhruv.com/notes/capacity-planning-begins-with-constraints/",
      "datePublished": "2025-06-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "Capacity Planning",
        "Performance",
        "Reliability"
      ],
      "description": "Capacity planning is not forecasting one traffic number and adding a fixed percentage. It is identifying which resource limits the user outcome, how demand approaches that limit, and…",
      "contentMarkdown": "Capacity planning is not forecasting one traffic number and adding a fixed percentage. It is identifying which resource limits the user outcome, how demand approaches that limit, and how quickly the system can respond.\n\n## Model demand in useful units\n\nRequests per second may be insufficient when requests have different cost. Include dimensions such as payload size, records scanned, concurrent sessions, event fan-out, model tokens, or storage growth. Separate steady demand from bursts and scheduled work.\n\nUse historical distributions, product events, seasonal patterns, and known launches. Forecasts should be ranges with assumptions, not a single precise value. Record which assumptions would invalidate the plan.\n\n## Find the limiting resource\n\nMap each demand unit to CPU, memory, connections, queue depth, I/O, network, third-party quota, and human operational load. The first exhausted dependency determines effective capacity. Managed services still have quotas, scaling rates, partition limits, and regional constraints.\n\nLoad tests should reproduce representative mixes and data sizes. Measure user-visible latency and error behavior while increasing load gradually. A test that stops before saturation cannot show the failure mode or recovery behavior.\n\n## Preserve headroom\n\nHeadroom covers traffic variance, node loss, failover, deployments, rebalancing, and forecast error. Define it per constrained resource. Autoscaling reduces provisioning time only if metrics, limits, quotas, and downstream dependencies can support the added instances.\n\nThe [Google SRE discussion of handling overload](https://sre.google/sre-book/handling-overload/) explains load shedding, request prioritisation, and graceful behavior as capacity protections. Those controls should be designed before saturation.\n\n## Connect capacity to action\n\nCreate thresholds with lead time: when to tune, scale, repartition, request quota, or change architecture. Assign owners and account for the time needed to purchase or approve capacity. Monitor forecast error and update the model after launches and incidents.\n\nTest reduced-capacity states, including zone loss and dependency throttling. A system sized only for normal conditions may fail exactly when redundancy is needed.\n\nCapacity planning is an operating loop: measure demand, identify constraints, validate behavior, preserve recovery margin, and revisit assumptions. The output is not a spreadsheet. It is enough time and evidence to act before users discover the limit.\n\n## Use queueing signals\n\nUtilisation alone can look healthy while latency rises sharply near saturation. Track concurrency, queue wait, service time, rejection, and retry volume. Little’s Law—average items in a stable system equals arrival rate multiplied by average time—can help connect throughput, latency, and work in progress, provided the measured boundary and time window are consistent.\n\nRetries amplify demand during failure. Model retry policy, client timeouts, batch catch-up, and failover traffic as part of capacity rather than treating them as unusual. A dependency recovery can create a second spike when queued work is released.\n\n## Separate scaling horizons\n\nApplication replicas may scale in seconds, nodes in minutes, database partitions in hours, and procurement or architecture changes in months. Maintain triggers for each horizon. Fast autoscaling cannot fix an exhausted regional IP range or a third-party quota that takes weeks to raise.\n\nReserve enough capacity for maintenance and failure. If normal traffic consumes the capacity required to lose a zone, redundancy exists only on a diagram. Test failover while representative load is present.\n\n## Connect technical and financial limits\n\nFor elastic systems, define the cost of meeting peak demand and a guardrail for runaway scaling. Track unit economics such as cost per transaction, active tenant, or processed gigabyte. A capacity change that lowers latency but multiplies unit cost needs an explicit product decision.\n\n## Planning checklist\n\nRecord demand units, seasonal range, workload mix, constraint map, saturation behavior, headroom policy, scaling lead times, quotas, failover load, cost envelope, and owners. Review actual versus forecast, explain material error, and update the model. Forecast accuracy improves through a measured loop, not through increasingly detailed unsupported estimates.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Incident command is a coordination system",
      "canonical": "https://doshidhruv.com/notes/incident-command-is-a-coordination-system/",
      "datePublished": "2025-04-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Staff engineering",
        "Incident Management",
        "Reliability",
        "Leadership"
      ],
      "description": "During a serious incident, technical skill is necessary but insufficient. Many responders changing the system without shared priorities can increase risk. Incident command creates a…",
      "contentMarkdown": "During a serious incident, technical skill is necessary but insufficient. Many responders changing the system without shared priorities can increase risk. Incident command creates a temporary coordination structure so investigation, mitigation, communication, and decision-making remain aligned.\n\n## Assign explicit roles\n\nThe incident commander owns priorities, coordinates work, and decides when the response changes phase. They do not need to be the deepest technical expert and should avoid becoming the primary debugger.\n\nOther useful responsibilities include:\n\n- operations leads investigating and executing mitigations;\n- a communications lead updating stakeholders and users;\n- a scribe preserving timeline, hypotheses, decisions, and actions;\n- subject-matter experts engaged for specific systems.\n\nOne person may hold several roles in a small event, but the responsibilities should still be named.\n\n## Stabilise before explaining\n\nEstablish impact, affected users, start time, current risk, and immediate containment options. Prefer reversible mitigations: stop a rollout, shed non-critical load, disable a feature, fail over, or restore a known state. Root cause can wait when user harm is continuing.\n\nMaintain a single incident record with timestamps, owners, and next update time. Separate facts from hypotheses. Every production change should have an executor, reviewer where possible, expected effect, and rollback condition.\n\nGoogle’s [SRE incident management guidance](https://sre.google/workbook/incident-response/) describes a structured response with clear command, operational work, communications, and planning. The exact role names matter less than preventing hidden ownership and conflicting action.\n\n## Manage responder load\n\nRotate people during long incidents. Handovers should state system state, active hypotheses, changes made, risks, and next actions. Invite expertise deliberately rather than filling the channel with observers. Psychological safety improves accuracy: responders must be able to report uncertainty and mistakes quickly.\n\n## Close the response carefully\n\nRecovery is not the same as resolution. Confirm user journeys, queues, data integrity, dependencies, and monitoring after metrics return to normal. Record follow-up owners before dissolving the command structure.\n\nThe review should examine technical and organisational conditions, including detection, tooling, permissions, and decision load. Incident command is effective when it helps experts apply their judgment without losing a shared picture of the event.\n\n## Establish a response rhythm\n\nAt the start, state severity, commander, channel, incident document, user impact, and next update time. Use short operational cycles: assess current state, select the highest-value action, assign an owner, execute, and compare the result with expectation. This prevents multiple responders from testing conflicting hypotheses simultaneously.\n\nThe incident commander should maintain an explicit priority: protect people and data, stop continuing harm, restore the critical journey, verify recovery, then investigate deeper causes. Priorities may change, but the change should be announced.\n\n## Communicate uncertainty clearly\n\nExternal updates should distinguish confirmed impact from investigation. State what users experience, what is being done, and when the next update will arrive. Avoid speculative root causes and precise recovery estimates without evidence. Internal updates can include hypotheses but should label them as such.\n\nKeep executives and stakeholders informed through the communications role rather than pulling operators into repeated briefings. This protects technical focus while preserving accountability.\n\n## Capture decisions, not chat volume\n\nThe timeline should record detection, declared severity, observed impact, hypotheses tested, production changes, approvals, results, handovers, and recovery confirmation. Link dashboards and commands where useful, but preserve the reasoning around consequential actions.\n\n## Readiness checklist\n\nPrepare severity definitions, role cards, paging and conference paths, status templates, access procedures, rollback tools, customer-communication ownership, and handover guidance. Exercise scenarios where the primary monitoring system, identity provider, or usual communication channel is unavailable. The response system is dependable only if responders can invoke it under the same degraded conditions as the product.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Plan schema evolution as a production migration",
      "canonical": "https://doshidhruv.com/notes/plan-schema-evolution-as-a-production-migration/",
      "datePublished": "2025-02-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Distributed systems",
        "Database",
        "Schema Evolution",
        "Reliability"
      ],
      "description": "Database schema changes are distributed system changes. Application instances, background jobs, replicas, analytical consumers, and rollback versions rarely update at the same instant.…",
      "contentMarkdown": "Database schema changes are distributed-system changes. Application instances, background jobs, replicas, analytical consumers, and rollback versions rarely update at the same instant. A safe migration keeps mixed versions compatible during the transition.\n\n## Expand before you contract\n\nFor a rename or representation change:\n\n1. Add the new structure without removing the old one.\n2. Deploy code that can read both and writes the chosen transition form.\n3. Backfill historical data in bounded batches.\n4. Verify parity and move readers to the new structure.\n5. Stop old writes, observe, then remove the old structure later.\n\nEach phase should be independently deployable and reversible. The exact dual-read or dual-write strategy depends on transaction boundaries and performance, but the compatibility window must be explicit.\n\n## Understand the database operation\n\nAn apparently simple `ALTER TABLE` can lock a large relation, rewrite data, expand a transaction log, saturate replicas, or exhaust storage. Behavior varies by database engine and version. Use the vendor’s current documentation, test with representative size and load, and define cancellation and recovery before production execution.\n\nFor PostgreSQL, the official [`ALTER TABLE` documentation](https://www.postgresql.org/docs/current/sql-altertable.html) describes lock levels and operation behavior. It should be consulted for the deployed version rather than relying on a generic migration recipe.\n\n## Backfill as a workload\n\nBackfills need rate limits, checkpoints, idempotency, observability, and pause controls. Process stable key ranges rather than relying on offsets in a changing table. Monitor query latency, lock waits, replica lag, storage, error rate, and remaining rows.\n\nValidate semantic correctness, not only non-null counts. Sample records, compare old and new reads, and reconcile totals where the transformation supports it.\n\n## Preserve rollback\n\nApplication rollback is unsafe once a new version writes data an old version cannot understand. State the rollback boundary for every phase. Sometimes forward repair is safer than code rollback; operators should know that before an incident.\n\nRemove old columns, indexes, triggers, and compatibility code only after all consumers and restore procedures have moved. A delayed cleanup is acceptable when it has an owner and date.\n\nSchema evolution succeeds when no single deployment must be perfectly timed. Compatibility creates the room to observe, correct, and proceed safely.\n\n## Analyse reads and writes separately\n\nList every application version, worker, export, replica, report, and recovery tool that reads or writes the structure. A writer adding a new enum value can break an older reader even when the column itself is unchanged. A rollback may restore old code that cannot parse data already written by the new release.\n\nUse feature flags to separate deployment from behavior change. Deploy compatibility code first, observe that all instances can read both forms, then enable new writes gradually. Keep the flag and old read path until rollback risk has passed.\n\n## Plan large-table operations\n\nEstimate row count, table size, write rate, index build time, log generation, replica capacity, and free storage. Prefer online or concurrent operations supported by the engine, but understand their remaining locks and failure behavior. Set conservative statement and lock timeouts so a migration fails rather than blocking production traffic indefinitely.\n\nFor backfills, choose batch size from production measurements. Pause on elevated latency or replica lag. Record a durable high-water mark and make each batch safe to repeat. Validate continuously instead of waiting until the final row.\n\n## Handle removal as a release\n\nBefore dropping old structure, prove that no supported code reads or writes it. Search application source, queries, dashboards, ETL, audit tools, and restore scripts. Disable access or add monitoring before deletion to reveal hidden consumers. Take a recoverable backup appropriate to the data and test restoration of the affected object.\n\n## Migration checklist\n\nDocument compatibility matrix, lock behavior, capacity estimate, staged rollout, backfill controls, validation queries, rollback boundary, monitoring, owner, and cleanup date. Run the sequence in an environment with representative data volume; functional test fixtures alone cannot expose production migration risk.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Use data contracts to make ownership executable",
      "canonical": "https://doshidhruv.com/notes/use-data-contracts-to-make-ownership-executable/",
      "datePublished": "2024-12-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Distributed systems",
        "Data Contracts",
        "Data Platforms",
        "Governance"
      ],
      "description": "A data contract makes the expectations between a data producer and its consumers explicit. It covers more than column names: meaning, ownership, quality, timeliness, compatibility,…",
      "contentMarkdown": "A data contract makes the expectations between a data producer and its consumers explicit. It covers more than column names: meaning, ownership, quality, timeliness, compatibility, access, and operational response all belong to the interface.\n\n## Start with a named data product\n\nIdentify the dataset, event stream, or API output as a product with an accountable owner. Define the business entities and measures in plain language. If “active customer” has three definitions, a schema registry cannot resolve the disagreement.\n\nThe contract should include:\n\n- field types, nullability, units, and accepted values;\n- keys, uniqueness, ordering, and partition behavior;\n- freshness, completeness, and availability objectives;\n- classification, retention, residency, and access rules;\n- compatibility policy and deprecation period;\n- owner, support channel, and incident process.\n\n## Validate near the producer\n\nGenerate or validate schemas in the producer’s delivery pipeline. Test critical semantic rules against representative data. Detect contract violations before publication where possible, and quarantine or mark invalid records when blocking the entire stream would cause more harm.\n\nConsumers should validate assumptions too. Contract testing is strongest when registered consumers can express the fields and behavior they depend on. This turns an apparently harmless producer change into visible impact before deployment.\n\nThe [OpenAPI Specification](https://spec.openapis.org/oas/latest.html), [AsyncAPI Specification](https://www.asyncapi.com/docs/reference/specification/latest), and schema systems such as Avro each describe structural interfaces for different transports. A data contract uses those mechanisms but also carries semantic and operational commitments.\n\n## Change through negotiation\n\nPrefer additive, backward-compatible changes. For a breaking change, identify consumers, publish a migration path, run versions in parallel for a bounded period, and measure adoption. Do not keep unused fields forever because ownership is unclear.\n\nVersion meaning, not just shape. Changing currency, time zone, aggregation window, or source logic can break a consumer without changing a type.\n\n## Operate the relationship\n\nMonitor freshness, volume, schema conformance, distribution shifts, and access failures. Route alerts to the owner able to act. Review recurring exceptions as product feedback rather than treating them only as consumer misuse.\n\nA contract does not eliminate coordination. It makes coordination concrete, testable, and proportional to the impact of change.\n\n## Define semantic rules explicitly\n\nStructural validation can confirm that `amount` is a decimal, but consumers also need currency, tax treatment, rounding, sign convention, and time of recognition. For timestamps, specify event time versus processing time, time zone, precision, and late-arrival behavior. For identifiers, state stability, uniqueness scope, and whether values can be reassigned.\n\nInclude representative valid, invalid, boundary, and redacted examples. Examples make ambiguity visible earlier than abstract field descriptions and can become executable fixtures for producer and consumer tests.\n\n## Assign objectives and consequences\n\nFreshness might mean 95 percent of records available within fifteen minutes, measured from a named source timestamp. Completeness might compare received records with an authoritative control total. Define the measurement location, window, exclusions, and response when the objective is missed.\n\nNot every violation should stop production. A missing optional description may be quarantined or defaulted; a duplicated financial transaction may require immediate containment. Classify rules by impact and connect them to alerts, ownership, and remediation.\n\n## Manage discovery and lineage\n\nPublish contracts in a searchable catalogue linked to owners, upstream sources, downstream consumers, transformations, quality history, and access policy. Automated lineage is useful but incomplete when data leaves through files, spreadsheets, or manual exports. Allow teams to declare and verify those edges.\n\n## Readiness checklist\n\nBefore declaring a contract active, verify business definitions, structural schema, examples, ownership, quality objectives, classification, retention, access, compatibility rules, consumer tests, monitoring, and incident routing. Before a breaking change, show affected consumers and a measured migration plan. A contract without a consumer or enforcement point is documentation; a contract with both becomes an operating boundary.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Design safe tool use for AI agents",
      "canonical": "https://doshidhruv.com/notes/design-safe-tool-use-for-ai-agents/",
      "datePublished": "2024-10-01",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "AI Agents",
        "Tool Use",
        "Security"
      ],
      "description": "An AI agent becomes materially different from a chatbot when it can read private data or change external systems. The central safety question is not whether the model produces good…",
      "contentMarkdown": "An AI agent becomes materially different from a chatbot when it can read private data or change external systems. The central safety question is not whether the model produces good prose; it is whether authority, intent, and side effects remain controlled when model output is uncertain.\n\n## Give tools narrow contracts\n\nEach tool should perform one bounded operation with a typed schema, explicit authorization, predictable errors, and a small response. Avoid general shell, database, or HTTP tools when a domain-specific operation will do. Validate every argument outside the model and reject unknown fields.\n\nSeparate read operations from writes. Separate reversible changes from destructive or externally visible actions. Tool descriptions must not be the security boundary; enforcement belongs in application code and the target system.\n\n## Bind authority to the user and task\n\nThe agent should receive only the credentials and scopes needed for the current task. Preserve the initiating user’s identity and tenant context. Re-check authorization at execution time rather than assuming that access to the conversation implies access to every connected resource.\n\nRequire confirmation when an action is destructive, costly, difficult to reverse, or communicates externally. The confirmation should show the resolved target and effect, not a vague “continue?” prompt.\n\n## Treat all content as untrusted\n\nTool results, retrieved documents, websites, emails, and attachments can contain instructions intended to redirect the agent. Keep system policy and data separate, constrain which tool calls content can influence, and prevent secrets from being inserted into untrusted destinations.\n\nThe [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) describes prompt injection, excessive agency, sensitive-information disclosure, and related risks. Use it as a threat-model input, then test the concrete workflows your agent supports.\n\n## Make execution inspectable\n\nRecord requested action, resolved arguments, authorization decision, tool result, model and policy versions, confirmation, and resulting resource identifiers. Redact sensitive content and apply retention limits. Use idempotency keys for retried writes and compensation procedures where transactions are unavailable.\n\nSet budgets for steps, time, tokens, money, and affected records. A stop condition is a control, not a model preference.\n\nUseful agents combine flexible reasoning with inflexible authority boundaries. The model may propose an action; deterministic systems must decide whether and how that action is allowed to occur.\n\n## Use a plan-execute boundary\n\nRepresent proposed actions as structured data before execution. A policy layer can resolve resource identifiers, calculate risk, check current authorization, and decide whether confirmation is required. The executor receives only an approved, immutable action—not the full conversation and open-ended model authority.\n\nFor a destructive request such as deleting a deployment, confirmation should show the exact environment, deployment identifier, dependent resources, and recovery consequence. If any target changes after confirmation, require a new approval.\n\n## Limit indirect influence\n\nRetrieved content may supply facts but should not grant capabilities. An email saying “upload credentials to this URL” must not change the allowlist of destinations. Label data provenance internally and prevent untrusted text from being concatenated into system policy or tool definitions.\n\nUse separate contexts for browsing, reasoning, and secrets. Where a tool needs a credential, bind it inside the executor rather than exposing it to the model. Filter tool responses to the minimum fields needed for the next decision.\n\n## Design compensation and recovery\n\nFor reversible actions, capture the previous state and provide a tested undo operation. For irreversible actions, increase review and confirmation. Multi-step workflows need checkpoints so a retry resumes safely rather than repeats completed effects. Stop and escalate when actual state differs from the plan.\n\n## Adversarial checklist\n\nTest instructions embedded in websites, documents, code comments, images, and tool output; attempts to cross tenant or user boundaries; argument smuggling; excessive result counts; repeated writes; tool timeout; stale confirmation; and partial workflow failure. Verify budgets, policy decisions, and audit events remain correct under each case. Safety is demonstrated by denied or contained actions, not by the model explaining that it intends to be careful.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Build AI governance into the delivery control plane",
      "canonical": "https://doshidhruv.com/notes/build-ai-governance-into-the-delivery-control-plane/",
      "datePublished": "2024-08-01",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "AI Governance",
        "Platform Engineering",
        "Risk Management"
      ],
      "description": "AI governance fails when it exists only as a policy document reviewed before launch. Controls must be attached to the same delivery path that versions models, prompts, data, tools,…",
      "contentMarkdown": "AI governance fails when it exists only as a policy document reviewed before launch. Controls must be attached to the same delivery path that versions models, prompts, data, tools, permissions, and application code.\n\n## Inventory the system\n\nMaintain a record for each AI-enabled capability: owner, purpose, affected users, model and provider, data classes, retrieval sources, tools, evaluation suite, human oversight, risk tier, and current deployment. The inventory should describe the complete application, not only a model identifier.\n\nConnect that record to source, build artifacts, approvals, incidents, and production telemetry. Evidence should be generated by normal engineering work rather than reconstructed for an audit.\n\n## Match controls to risk\n\nA private drafting assistant and an automated eligibility decision should not share the same review path. Define tiers using impact, autonomy, data sensitivity, scale, reversibility, and external obligations. Higher-risk systems may require independent evaluation, security testing, legal review, explicit human decision authority, or stronger release gates.\n\nThe [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) provides a useful structure through Govern, Map, Measure, and Manage. Translate those functions into named owners and executable controls within the organisation.\n\n## Enforce at delivery time\n\nCI can verify that required metadata exists, approved model endpoints are used, evaluations meet thresholds, dependencies are scanned, and policy exceptions have not expired. Runtime gateways can enforce identity, allowed models, data-handling rules, rate limits, logging, and emergency disablement.\n\nControls must preserve developer feedback. A failed gate should state what evidence is missing, why it matters, and how to resolve or appeal it. Opaque governance encourages bypasses.\n\n## Monitor change\n\nModels and external services can change without an application commit. Track provider versions, safety settings, retrieval data, tool definitions, and policy configuration. Define which changes trigger re-evaluation and who can approve them.\n\nRecord incidents and near misses in the same operational system as other production risks. Feed observed failures back into evaluation and policy.\n\nGovernance is strongest when it makes safe delivery routine. The control plane should provide traceability and proportionate constraints while leaving product teams with a clear, supported path to ship useful systems.\n\n## Define evidence by lifecycle stage\n\nDuring discovery, record intended users, decision impact, data sources, and prohibited uses. Before development, assign a risk tier and control owner. Before release, attach evaluation results, threat model, privacy assessment, human-oversight design, rollback plan, and operational objectives. In production, retain version, incident, monitoring, and change evidence.\n\nThis evidence should be addressable by stable identifiers. A deployed capability can then point to the exact model configuration, prompt, retrieval index, tool set, evaluation run, approval, and application version that produced it.\n\n## Separate policy from enforcement\n\nPolicy states the required outcome—for example, sensitive data must not be sent to an unapproved model endpoint. Enforcement may occur through network controls, an AI gateway, data classification, CI policy, and runtime monitoring. Map each policy to one or more controls and each control to observable evidence.\n\nAvoid a single central gateway becoming an unreviewed source of broad access. The gateway itself needs workload identity, tenant isolation, rate limits, versioned policy, failure behavior, and objectives.\n\n## Operate exceptions\n\nAn exception should name the unmet control, reason, affected system, compensating measures, approver, expiry, and remediation owner. Expired exceptions must fail visibly. Analyse repeated exceptions: they may reveal an impractical policy, missing platform capability, or risk the organisation has implicitly accepted without deciding.\n\n## Governance checklist\n\nConfirm complete system inventory, explicit owners, risk classification, evidence-linked releases, approved data and model boundaries, independent review for high-impact use, runtime disablement, incident handling, change triggers, and time-bounded exceptions. Measure time to satisfy controls, recurring failure reasons, overdue reviews, and production outcomes—not only how many systems received approval.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Evaluate LLM systems as systems",
      "canonical": "https://doshidhruv.com/notes/evaluate-llm-systems-as-systems/",
      "datePublished": "2024-06-01",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "LLM Evaluation",
        "AI Quality",
        "Applied AI"
      ],
      "description": "A language model feature cannot be evaluated by trying a few prompts and deciding the answers look good. Its quality depends on the model, instructions, tools, retrieval, application…",
      "contentMarkdown": "A language-model feature cannot be evaluated by trying a few prompts and deciding the answers look good. Its quality depends on the model, instructions, tools, retrieval, application logic, data, and user workflow. Evaluation must cover the system and the cost of its failures.\n\n## Define the decision\n\nStart with the task the user is trying to complete and the harm caused by an incorrect, incomplete, delayed, or inappropriate result. Translate that into observable criteria: factual support, task completion, citation correctness, policy compliance, format validity, latency, and cost.\n\nNot every criterion should be collapsed into one score. A response that is fluent but exposes confidential data is a failure regardless of average quality.\n\n## Build a representative set\n\nCollect normal cases, important edge cases, adversarial inputs, ambiguous requests, and examples where the system should abstain or escalate. Preserve source and consent for production-derived data. Partition development and holdout sets so prompt tuning does not simply memorise the benchmark.\n\nUse deterministic checks for structure, required citations, tool parameters, and policy rules. Human review remains necessary for nuanced correctness and usefulness. Model-based graders can increase coverage, but they need calibration against human judgments and should not grade with the same assumptions as the system under test.\n\n## Evaluate components and the whole path\n\nFor retrieval, measure whether supporting evidence was found and ranked. For tool use, verify selection, arguments, authorization, and handling of tool failure. For generation, assess whether claims follow from supplied evidence. Then run end-to-end tasks because individually good components can interact badly.\n\nNIST’s [Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) extends the AI RMF with risks and actions specific to generative systems. It is a useful control catalogue, not a substitute for task-specific acceptance criteria.\n\n## Make evaluation continuous\n\nRun a stable regression suite for changes to models, prompts, indexes, tools, and policies. Add newly observed failures after review. Monitor production proxies and sampled outcomes, but do not treat thumbs-up rates as ground truth; feedback is sparse and selection-biased.\n\nVersion results with the complete system configuration. A model name alone is not reproducible evidence.\n\nThe goal of evaluation is not to prove an AI feature is intelligent. It is to define where it is dependable, detect when that boundary changes, and prevent unacceptable failure from reaching users.\n\n## Create an evaluation specification\n\nFor each task, define input population, expected behavior, unacceptable outcomes, scoring method, reviewer guidance, and release threshold. Include examples of partial credit and disagreement. This turns evaluation from a demo into a repeatable engineering artifact.\n\nUse exact checks where the answer is deterministic: JSON schema validation, allowed tool names, citation existence, permission boundaries, arithmetic, or known identifiers. Use rubric-based human review for relevance, clarity, supported reasoning, and contextual appropriateness. Keep safety and privacy as separate gates rather than averaging them into general quality.\n\n## Calibrate model graders\n\nIf an LLM grades outputs, give it a narrow rubric and examples, randomise answer order for comparisons, and measure agreement against qualified human reviewers. Check whether the grader favors longer answers, familiar phrasing, or outputs from its own model family. Periodically resample graded cases for human audit.\n\nModel graders are useful for triage and scale; they are not independent proof. Preserve the grader model, prompt, temperature, and rubric with each result.\n\n## Test uncertainty and change\n\nRun repeated trials for non-deterministic paths and report distributions, not only a best result. Test model timeouts, truncated context, tool errors, missing retrieval, malformed output, and policy refusal. Compare a proposed version with the production baseline on the same holdout set and define the maximum acceptable regression per critical slice.\n\n## Release checklist\n\nRequire representative test coverage, protected holdouts, calibrated graders, component and end-to-end results, risk-specific thresholds, latency and cost budgets, reproducible configuration, and an owner for production review. After release, turn confirmed failures into regression cases while watching for test-set contamination and changing user behavior.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Define the boundaries of a production RAG system",
      "canonical": "https://doshidhruv.com/notes/define-the-boundaries-of-a-production-rag-system/",
      "datePublished": "2024-04-01",
      "dateModified": "2026-07-28",
      "topics": [
        "AI governance",
        "RAG",
        "LLM",
        "AI Architecture"
      ],
      "description": "Retrieval augmented generation connects a language model to external information, but it does not automatically make an answer correct, current, or authorised. A production design must…",
      "contentMarkdown": "Retrieval-augmented generation connects a language model to external information, but it does not automatically make an answer correct, current, or authorised. A production design must define the boundaries of ingestion, retrieval, generation, and user trust.\n\n## Establish the corpus contract\n\nIdentify which sources are authoritative, who owns them, how often they change, and which users may access each part. Preserve source identifiers, timestamps, versions, and access-control metadata through parsing and chunking. If the ingestion pipeline loses document permissions, retrieval can become a data-exfiltration path.\n\nMeasure freshness from source change to searchable representation. Define deletion behavior and prove that removed material disappears from indexes, caches, and derived stores.\n\n## Treat retrieval as a system\n\nChunking, embeddings, filters, hybrid search, reranking, and context assembly are separate design choices. Evaluate them against representative questions and known relevant sources. Recall at a fixed result count, ranking quality, latency, and empty-result behavior matter more than whether a vector database is present.\n\nKeep citations attached to the exact evidence supplied to the model. A generated link that was not part of retrieved context is not provenance.\n\n## Constrain generation\n\nTell the model when to abstain, how to separate sourced statements from inference, and which operations require deterministic application logic. Validate output structure before another system consumes it. Treat retrieved text as untrusted input: documents can contain instructions intended to override the application.\n\nThe [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) organises AI risk work around governing, mapping, measuring, and managing. Those activities apply to the whole RAG pipeline, not only the model endpoint.\n\n## Observe each boundary\n\nRecord retrieval query, filters, document identifiers, model and prompt versions, latency, token use, safety decisions, and user feedback with appropriate privacy controls. Do not log sensitive prompts and retrieved text by default.\n\nEvaluate failure by layer: missing source, stale ingestion, poor retrieval, context truncation, unsupported synthesis, or application error. A single end-to-end accuracy score hides where the system needs improvement.\n\nRAG is useful because it makes evidence available at inference time. Its credibility comes from access control, provenance, evaluation, and honest behavior when the evidence is insufficient.\n\n## Design ingestion for traceability\n\nStore the source URI, document version, section location, parser version, chunking policy, and ingestion timestamp with every chunk. A generated answer should be traceable back through the retrieved chunk to the exact source version. When parsing fails, quarantine the document and alert its owner rather than silently indexing incomplete text.\n\nChunk boundaries should reflect document structure where possible. Fixed token windows are simple but may separate a heading from its explanation or combine unrelated sections. Evaluate chunk size and overlap against actual questions, and avoid creating so many near-duplicates that ranking becomes noisy.\n\n## Build retrieval in stages\n\nApply authorization and coarse metadata filters before semantic ranking. Combine lexical and vector retrieval when exact identifiers, product names, error codes, or policy clauses matter. Rerank a manageable candidate set using features appropriate to the task, then allocate context space across sources so one large document cannot crowd out all others.\n\nDefine an evidence threshold. When retrieval scores, source authority, or coverage are insufficient, the system should say that it cannot answer from the approved corpus. Returning a plausible answer from model memory defeats the purpose of grounded retrieval.\n\n## Evaluate with diagnostic metrics\n\nMaintain questions with expected source documents and supported answer claims. Measure source recall, ranking, citation precision, answer support, abstention quality, latency, and cost. Slice results by document type, user group, language, recency, and authorization path. A high overall score can hide failure on the most sensitive corpus.\n\n## Production checklist\n\nVerify source ownership, permission-preserving ingestion, deletion propagation, parser monitoring, retrieval evaluation, prompt-injection controls, citation binding, model abstention, privacy-safe traces, index backup, and an emergency disable path. Review incorrect answers by layer and assign the fix to the owning component instead of reflexively changing the prompt.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Build portable boundaries, not lowest-common-denominator clouds",
      "canonical": "https://doshidhruv.com/notes/build-portable-boundaries-not-lowest-common-denominator-clouds/",
      "datePublished": "2024-02-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Cloud architecture",
        "Multi Cloud",
        "Portability",
        "Architecture"
      ],
      "description": "Multi cloud strategies often begin with a desire to avoid lock in and end with a platform that hides every useful provider capability. Portability is more effective when it is applied…",
      "contentMarkdown": "Multi-cloud strategies often begin with a desire to avoid lock-in and end with a platform that hides every useful provider capability. Portability is more effective when it is applied to selected business and operational boundaries rather than demanded equally from every component.\n\n## Name the reason\n\nDifferent goals require different architectures: regulatory placement, acquisition integration, customer proximity, resilience to a provider-scale event, commercial leverage, or access to a specialised service. “Use multiple clouds” is an implementation constraint, not an outcome.\n\nFor each workload, state whether it must run active-active, be recoverable on another provider, move within a defined period, or simply avoid proprietary data formats. These are materially different promises with different costs.\n\n## Separate portable and provider-specific layers\n\nKeep domain logic, API contracts, event schemas, identity abstractions, telemetry, and deployment metadata independent where doing so preserves real option value. Encapsulate provider-specific services behind owned interfaces when an alternative is plausible and the switching value justifies the cost.\n\nDo not recreate a managed database, queue, or identity service solely to make two providers look identical. The abstraction itself becomes a platform with reliability, security, and support obligations.\n\nThe [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) and [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework) both emphasise explicit trade-offs across reliability, security, operations, performance, and cost. A multi-cloud decision should be evaluated against the same concerns, not treated as an automatic resilience improvement.\n\n## Test the option\n\nPortability that is never exercised decays. Pin open formats and protocol versions, rebuild environments from source, restore representative data, and run dependency-loss exercises. Measure the time, manual steps, data loss, and degraded functions involved.\n\nIdentity, DNS, keys, CI, observability, and source control can remain hidden single points of failure even when compute exists in two providers. Draw the complete control and data planes.\n\n## Price the continuing cost\n\nAccount for duplicate skills, networking, security controls, vendor management, observability, data transfer, and slower adoption of native capabilities. Compare this recurring cost with the quantified risk or option the design addresses.\n\nGood portability protects a specific exit or recovery path. It does not pretend cloud platforms are interchangeable.\n\n## Classify the portability requirement\n\nThere are several useful levels. **Data portability** means information can be exported in a documented format. **Build portability** means infrastructure and applications can be recreated elsewhere. **Operational portability** means teams can monitor, secure, and recover the workload in the alternate environment. **Traffic portability** means production demand can actually move within a stated objective. Each level requires more continuing investment.\n\nName the recovery point and recovery time for any cross-cloud failover claim. If data replication is asynchronous, quantify acceptable loss. If the alternate environment is cold, include quota approval, image availability, DNS change, certificate issuance, and cache warming in the recovery test.\n\n## Preserve identity and data semantics\n\nUse stable internal identities and map them to provider-specific roles at the boundary. Avoid copying long-lived credentials between clouds. Keep data formats explicit, but account for database behavior, collation, consistency, extensions, and operational tooling—not only whether both systems speak SQL.\n\nEgress cost and replication lag are architectural constraints. A design that constantly moves large datasets to preserve theoretical choice may cost more and fail more often than a documented restore path.\n\n## Make provider use intentional\n\nMaintain a decision record for each provider-specific service. State the capability gained, the coupling introduced, the exit mechanism, and when that exit is worth testing. Some workloads should embrace a managed native service because its operational value exceeds switching value.\n\n## Review checklist\n\nVerify the business reason, required portability level, dependency map, identity model, data-copy behavior, DNS and certificate path, observability in both environments, capacity reservation, cost, skills, and tested recovery evidence. If the organisation has never exercised the alternate path, describe it as a plan—not as achieved resilience.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Use the strangler pattern for controlled cloud migration",
      "canonical": "https://doshidhruv.com/notes/use-the-strangler-pattern-for-controlled-cloud-migration/",
      "datePublished": "2023-12-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Cloud architecture",
        "Cloud Migration",
        "Modernization",
        "Architecture"
      ],
      "description": "Large migrations fail when “move the system” is treated as one indivisible project. The strangler pattern reduces risk by placing a controlled boundary around the existing system and…",
      "contentMarkdown": "Large migrations fail when “move the system” is treated as one indivisible project. The strangler pattern reduces risk by placing a controlled boundary around the existing system and moving capabilities incrementally while production continues to operate.\n\n## Find a business seam\n\nStart with a capability that has a clear owner, manageable data dependencies, and a measurable user outcome. A low-value component is not always the safest pilot if it teaches nothing about identity, data, traffic, and operations. Choose a slice representative enough to validate the migration platform without putting the entire business at risk.\n\nDocument the current request paths, data ownership, batch jobs, operational procedures, and hidden integrations. Migration plans are often defeated by an unrecorded report, shared table, or manual recovery step rather than by compute provisioning.\n\n## Introduce a routing boundary\n\nUse an API gateway, facade, event route, DNS layer, or application adapter to direct selected traffic to the new capability. The boundary must support gradual exposure, rollback, and observability. Avoid duplicating business logic inside the router.\n\nThe migration should make ownership clearer. If both old and new systems can update the same record indefinitely, the architecture has created distributed ambiguity rather than decomposition.\n\n## Move data deliberately\n\nChoose a source of truth for each phase. Options include one-time migration with a cutover, change-data capture, event replication, or temporary dual writes. Dual writes require reconciliation because partial failure is unavoidable. Define how lag, ordering, deletion, and correction are handled.\n\nAWS’s [migration guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/strangler-fig.html) describes the routing and incremental replacement pattern. The same principle applies beyond one cloud provider: isolate change, preserve reversibility, and retire the old path in measured steps.\n\n## Set exit criteria\n\nEach slice needs functional parity, performance and reliability evidence, security review, support readiness, cost visibility, and a tested rollback. After cutover, remove obsolete routes, jobs, credentials, data copies, and monitoring. A migration is not complete while the old operating burden remains.\n\nIncremental modernization is not slow by definition. It creates frequent proof, exposes dependencies early, and lets teams stop or redirect before a single high-risk cutover consumes the whole program.\n\n## Choose the first slice with evidence\n\nScore candidate capabilities on business value, coupling, data ownership, traffic pattern, compliance, operational pain, and rollback difficulty. A useful first slice has visible value and exercises the shared migration path, but does not require solving the hardest dependency before the team has learned how the new environment behaves.\n\nEstablish a baseline before moving it: latency, error rate, throughput, operating effort, cost, recovery time, and critical user outcomes. Without a baseline, a migration can be technically complete while performance or operability quietly worsens.\n\n## Avoid the distributed monolith\n\nMoving code into separate services while retaining synchronous calls, shared tables, coordinated deployments, and central release approval preserves the original coupling with more failure modes. Give the extracted capability clear data ownership and a stable contract. Where immediate separation is impossible, document the temporary coupling and its removal milestone.\n\n## Run the cutover\n\nUse progressive traffic exposure by tenant, region, operation, or percentage. Compare old and new outcomes, not only infrastructure metrics. Define automatic and manual rollback signals. Protect rollback from incompatible data written by the new system; sometimes traffic can return while reads must continue through a compatibility layer.\n\n## Completion checklist\n\nA slice is finished when production traffic uses the new path, data ownership is unambiguous, objectives are met, on-call teams can operate it, recovery is tested, costs are attributed, and the old path is removed. Track retired servers, licences, jobs, integrations, credentials, dashboards, and support procedures as first-class migration outcomes. Otherwise the organisation pays for both architectures indefinitely.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Treat Terraform modules as versioned interfaces",
      "canonical": "https://doshidhruv.com/notes/treat-terraform-modules-as-versioned-interfaces/",
      "datePublished": "2023-10-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Terraform",
        "Infrastructure as Code",
        "Platform Engineering"
      ],
      "description": "A reusable Terraform module is an internal product interface. Its inputs, outputs, defaults, provider constraints, state behavior, and upgrade path affect every stack that consumes it.…",
      "contentMarkdown": "A reusable Terraform module is an internal product interface. Its inputs, outputs, defaults, provider constraints, state behavior, and upgrade path affect every stack that consumes it. Copying resources into a module does not make them maintainable.\n\n## Design around a capability\n\nCreate modules around stable capabilities such as a production-ready object store or service runtime, not around every individual provider resource. Expose the choices consumers genuinely need and keep security, tagging, encryption, logging, and lifecycle defaults inside the module.\n\nAvoid a single module with dozens of flags that can produce unrelated architectures. A smaller opinionated interface is easier to test and evolve. Where teams need a materially different pattern, publish a separate module or composition.\n\nHashiCorp’s [module development guidance](https://developer.hashicorp.com/terraform/language/modules/develop) describes the expected structure and composition model. The engineering challenge is choosing the boundary that remains meaningful as provider resources change.\n\n## Make changes compatible\n\nUse semantic versions and pin released module versions in consumers. A default change can be breaking even if Terraform accepts the configuration. Renamed resources can force destructive replacement unless state moves are declared. Changed outputs can break downstream stacks and automation.\n\nEvery release should state:\n\n- behavioral changes and migration steps;\n- minimum Terraform and provider versions;\n- state moves or import requirements;\n- expected plan impact;\n- deprecation timelines.\n\nTest upgrades from supported previous versions, not only clean creation. A module that passes from an empty state can still destroy an existing resource during migration.\n\n## Validate the contract\n\nStatic validation and formatting are the baseline. Add policy checks, example configurations, integration tests in isolated accounts, and plan assertions for critical properties. Verify encryption, network exposure, identity permissions, logging, backup, and deletion protection.\n\nKeep provider configuration at the composition root so consumers control credentials and regions. Do not hide cross-account assumptions inside a reusable child module.\n\n## Observe adoption and exceptions\n\nMaintain ownership, support expectations, and a registry of supported versions. Measure upgrade lag, recurring overrides, failed plans, and exception requests. Repeated exceptions often indicate the interface is missing a legitimate use case.\n\nGood modules compress organisational knowledge into a safe, reviewable contract. Their value is not fewer lines of Terraform; it is consistent infrastructure behavior with a credible migration path.\n\n## Keep inputs intentional\n\nInputs should describe user intent rather than expose every provider argument. A service module might accept data classification, availability tier, expected capacity, and approved network boundary. It can derive encryption, backup, logging, and placement controls from those choices. Passing a raw map of provider settings through the module avoids interface design and makes every consumer responsible for policy.\n\nValidate inputs with clear error messages. Mark sensitive values correctly, but remember that Terraform state can still contain them. Avoid passing secret material where a resource can reference a managed secret or identity directly.\n\n## Manage state changes safely\n\nResource addresses are part of the module’s effective compatibility surface. Refactoring a resource into a child module or renaming it can look like delete-and-create. Use moved blocks where supported, document state migrations, and inspect representative upgrade plans.\n\nApply lifecycle protections deliberately. `prevent_destroy` can stop accidental deletion but can also block an emergency replacement. `ignore_changes` can hide drift indefinitely. Each exception needs an explanation and test.\n\n## Release and support\n\nPublish immutable versions with a changelog and working examples. Maintain an upgrade matrix showing supported source and target versions. Automate dependency update proposals, but require plan review for changes that can replace or expose infrastructure.\n\n## Consumer checklist\n\nBefore adoption, confirm the module’s owner, version policy, provider constraints, security properties, recovery behavior, outputs, and escape path. Before upgrade, read migration notes, test against a state copy or representative environment, review replacements and permission changes, and confirm rollback feasibility. After apply, verify the workload behavior the module promised, not merely that Terraform reported success.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Choose Kubernetes tenancy boundaries deliberately",
      "canonical": "https://doshidhruv.com/notes/choose-kubernetes-tenancy-boundaries-deliberately/",
      "datePublished": "2023-08-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Cloud architecture",
        "Kubernetes",
        "Multi Tenancy",
        "Platform Engineering"
      ],
      "description": "Kubernetes can host many teams and workloads, but a namespace is not automatically a complete security, reliability, or cost boundary. Tenancy design should begin with the risks that…",
      "contentMarkdown": "Kubernetes can host many teams and workloads, but a namespace is not automatically a complete security, reliability, or cost boundary. Tenancy design should begin with the risks that must be isolated and the operations the platform team can sustain.\n\n## Decide what a tenant is\n\nA tenant may be a product team, application, business unit, customer, or environment. Each definition creates different requirements. Ask whether tenants trust one another, run privileged workloads, share compliance obligations, need independent upgrade schedules, or can consume a shared failure budget.\n\nThe [Kubernetes multi-tenancy documentation](https://kubernetes.io/docs/concepts/security/multi-tenancy/) distinguishes soft and hard isolation and explains the trade-offs between namespace and cluster boundaries. Use that as a model, not a promise that one control provides isolation by itself.\n\n## Layer namespace controls\n\nFor trusted internal teams, namespaces can be an efficient boundary when combined with:\n\n- role-based access scoped to the namespace;\n- default-deny network policies and explicit ingress and egress;\n- resource requests, limits, quotas, and priority policy;\n- pod security controls and restricted workload privileges;\n- separate service accounts and workload identities;\n- policy that prevents unsafe host access and cross-namespace references.\n\nControl-plane objects outside namespaces require special attention. Cluster roles, custom resources, admission configuration, storage classes, and shared operators can cross tenant boundaries.\n\n## Use clusters for stronger isolation\n\nSeparate clusters reduce some shared blast radius and enable independent lifecycle or regulatory controls, but add cost and operational surface. Cluster separation is appropriate when workloads are mutually untrusted, require conflicting control-plane configuration, or cannot share an outage domain. It still does not remove shared cloud-account, identity, network, or supply-chain risks.\n\n## Make capacity and failure visible\n\nNoisy-neighbor protection requires more than CPU limits. Consider memory pressure, ephemeral storage, API-server load, admission latency, IP space, persistent-volume throughput, and shared dependencies. Track usage and cost by tenant with a documented allocation model.\n\nTest isolation with failure exercises: exhausted quotas, compromised credentials, a broken operator, a network-policy regression, and a cluster upgrade. Record which layer contained the event.\n\nThe right tenancy model is rarely “one cluster” or “one cluster per team” everywhere. It is a small set of documented patterns matched to trust, failure, compliance, and operational boundaries.\n\n## Map the boundary by layer\n\nControl-plane isolation covers API access, admission, custom resources, and cluster-scoped controllers. Data-plane isolation covers node placement, network reachability, storage, kernel exposure, and resource contention. Operational isolation covers upgrades, incident ownership, maintenance windows, and recovery. A tenancy proposal should state the guarantee at each layer instead of using “separate namespace” as shorthand for all of them.\n\nDedicated node pools can reduce noisy-neighbor and kernel-sharing risk, but tolerations and node selectors must be controlled so tenants cannot schedule elsewhere. Runtime sandboxing may add another boundary for untrusted code, with performance and compatibility costs that need testing.\n\n## Treat operators as privileged software\n\nOperators often watch resources across namespaces, create cluster-scoped objects, and hold broad credentials. Review their permissions, upgrade behavior, webhooks, and failure impact. One malfunctioning admission webhook can block unrelated deployments across the cluster. Use timeouts, failure policies, availability controls, and staged rollout for admission components.\n\n## Design the tenant lifecycle\n\nAutomate namespace or cluster creation, identity bindings, quotas, network policy, observability, cost allocation, backup, and deletion. Offboarding must remove external cloud permissions, secrets, DNS, storage, and catalogue records—not only the namespace object.\n\n## Decision checklist\n\nChoose a shared namespace model only when tenants have compatible trust and lifecycle requirements. Move to dedicated nodes when compute isolation or capacity ownership requires it. Use separate clusters when control-plane configuration, compliance, blast radius, or mutual distrust outweigh additional operating cost. Revisit the decision after material workload, regulatory, or organisational changes rather than assuming the first boundary remains permanent.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Manage secrets without creating secret sprawl",
      "canonical": "https://doshidhruv.com/notes/manage-secrets-without-creating-secret-sprawl/",
      "datePublished": "2023-06-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Secrets Management",
        "IAM",
        "Security"
      ],
      "description": "Secret management is not solved by moving passwords from source code into a central vault. The larger goal is to reduce the number, lifetime, reach, and human handling of credentials…",
      "contentMarkdown": "Secret management is not solved by moving passwords from source code into a central vault. The larger goal is to reduce the number, lifetime, reach, and human handling of credentials across the system.\n\n## Remove secrets where possible\n\nPrefer workload identity, managed identities, and short-lived federation over static access keys. A credential that can be derived at runtime from an authenticated workload does not need to be copied into CI, configuration, local machines, and deployment tooling.\n\nFor the secrets that remain, maintain an inventory with owner, purpose, consumers, environment, rotation method, expiry, and recovery procedure. Unknown ownership is itself a security finding.\n\n## Control the lifecycle\n\nA sound lifecycle covers creation, distribution, use, rotation, revocation, and deletion. Generate values through approved tooling; never ask a person to paste a production secret through chat or a ticket. Deliver secrets directly to the workload where the runtime supports it, and keep plaintext out of build artifacts and environment dumps.\n\nThe [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) provides practical guidance across storage, rotation, auditing, and CI/CD use. Apply it according to the threat model rather than treating a vault as a universal control.\n\n## Design rotation before issuance\n\nRotation fails when applications assume one credential is valid forever. Support overlapping versions, reload without a full outage where practical, and verify the new credential before revoking the old one. Automate routine rotation and alert on values nearing expiry.\n\nEmergency rotation is different. Document how to identify affected consumers, revoke quickly, limit blast radius, and restore service. Practice it with a non-production credential.\n\n## Prevent accidental disclosure\n\nScan source, history, container layers, logs, telemetry attributes, and generated artifacts. A scanner finding is not safely resolved by deleting the current line; assume exposed credentials are compromised, rotate them, then clean history where necessary.\n\nAudit access to the secret store and alert on unusual retrieval, but protect audit logs from containing the values themselves. Separate administrative access from application retrieval.\n\nThe most mature secret-management program has fewer secrets each quarter. Central storage is useful; eliminating long-lived credentials is better.\n\n## Classify before choosing storage\n\nNot every sensitive value has the same lifecycle. Database credentials, signing keys, third-party API keys, encryption keys, and user recovery codes need different controls. Record whether a value can be rotated automatically, whether old versions must decrypt historical data, whether use must occur inside a hardware boundary, and what happens when the value is unavailable.\n\nEncryption keys deserve particular separation. Envelope encryption keeps data-encryption keys close to data while a key-encryption key remains in a managed key service. Applications should receive permission to perform required cryptographic operations rather than export root key material.\n\n## Deliver at runtime\n\nPrefer identity-based retrieval into memory or a protected runtime volume. Environment variables are convenient but may appear in process inspection, crash reports, debugging tools, or accidental configuration output. If a platform injects files, define ownership, permissions, refresh behavior, and deletion on termination.\n\nCI should exchange its trusted workload identity for a short-lived deployment credential. Repository secrets are still needed for some vendors, but scope them to one environment, prevent access from untrusted pull requests, and rotate them without editing workflow source.\n\n## Respond to exposure\n\nWhen a secret is disclosed, first revoke or rotate it; repository cleanup comes later. Identify every system where the value was valid, inspect its use during the exposure window, and verify that dependent applications have loaded the replacement. Preserve incident evidence without copying the secret into more systems.\n\n## Control checklist\n\nFor every remaining secret, require a named owner, least-privilege scope, automated distribution, rotation procedure, maximum lifetime, access audit, exposure response, and deletion condition. Report secrets without owners, credentials older than policy, workloads retrieving unusually large numbers of values, and applications still using superseded versions.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Zero Trust begins with service identity",
      "canonical": "https://doshidhruv.com/notes/zero-trust-begins-with-service-identity/",
      "datePublished": "2023-04-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Zero Trust",
        "Identity",
        "Security Architecture"
      ],
      "description": "Network location is a weak security identity. A request originating inside a cluster, virtual network, or corporate boundary is not automatically trustworthy. A Zero Trust design…",
      "contentMarkdown": "Network location is a weak security identity. A request originating inside a cluster, virtual network, or corporate boundary is not automatically trustworthy. A Zero Trust design evaluates the identity of the caller, the requested action, and the current policy at each meaningful boundary.\n\n## Establish workload identity\n\nEvery service needs a non-human identity that can be authenticated and rotated without embedding a long-lived secret. The identity should be tied to workload provenance: namespace, service account, deployment, environment, or another verifiable runtime claim. Transport encryption protects the channel; authenticated identity makes authorization possible.\n\nUse short-lived credentials issued through an automated trust chain. Rotation then becomes routine rather than a risky project. Bind credentials to the intended audience so a token issued for one service cannot be replayed against another.\n\n## Authorize the operation\n\nAuthentication answers who is calling. Authorization decides whether that identity may perform this action on this resource under current conditions. Prefer policies expressed in business or platform terms over IP-address lists. Keep default access narrow and make exceptions time-bound and attributable.\n\nThe [NIST Zero Trust Architecture](https://csrc.nist.gov/publications/detail/sp/800-207/final) describes access decisions around subjects, assets, resources, and policy rather than implicit network trust. Its model is architectural guidance, not a command to add a particular proxy or vendor.\n\n## Preserve identity across hops\n\nDo not blindly forward an end-user token through every service. Each hop should know both the initiating principal, where required for business authorization, and the immediate workload making the call. Token exchange or a constrained delegated credential can preserve that distinction without granting downstream services the user’s full authority.\n\nAudit records should capture the decision inputs and policy version without leaking credentials. A trace identifier can connect identity decisions to the wider request path.\n\n## Design for failure\n\nDecide whether policy-engine or identity-provider failure fails closed, uses a bounded cache, or permits a narrowly defined degraded mode. The answer depends on the operation’s risk. Test clock skew, certificate expiry, revocation, stale policy, and regional isolation.\n\nZero Trust is not “authenticate every packet.” It is a system of explicit identities, least-privilege decisions, protected context, and observable enforcement. Network controls still matter, but they become one layer rather than the definition of trust.\n\n## Build the trust chain\n\nA useful workload identity begins with something the runtime can attest: a cloud instance identity, Kubernetes service account, signed workload document, or hardware-backed key. An identity service validates that evidence and issues a short-lived credential for a named audience. The receiving service validates issuer, signature, audience, expiry, and relevant workload claims before policy evaluation.\n\nEvery link needs an owner and rotation path. If the root issuer, signing key, admission system, or service-account binding is compromised, downstream mutual TLS alone will faithfully authenticate the wrong workload.\n\n## Separate control and data planes\n\nThe control plane distributes identity, policy, keys, and trust configuration. The data plane enforces decisions on live requests. Decide how quickly a revoked identity or updated policy reaches every enforcement point, and measure propagation delay. A policy change that takes an unknown time to apply is not an effective emergency control.\n\nCache only bounded decisions with expiry and enough context to avoid using an authorization result for a different resource or action. Highly sensitive writes may fail closed when fresh policy is unavailable; low-risk reads may use a short, known cache. Document this per operation.\n\n## Migration sequence\n\nInventory service calls, assign workload identities, enable authentication in observe-only mode, compare expected and actual callers, then enforce one boundary at a time. Replace shared credentials and broad network rules only after workloads use their own identities. Maintain a tested break-glass path with narrow scope, short expiry, and complete auditing.\n\n## Verification checklist\n\nTest token replay against the wrong audience, expired and not-yet-valid credentials, clock skew, identity-provider outage, stale policy, forged forwarding headers, compromised workload identity, and cross-tenant requests. Confirm that logs can reconstruct who initiated the action, which workload executed it, what policy allowed it, and which resource changed.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Evolve event contracts without breaking consumers",
      "canonical": "https://doshidhruv.com/notes/evolve-event-contracts-without-breaking-consumers/",
      "datePublished": "2023-02-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Distributed systems",
        "Event Driven Architecture",
        "Kafka",
        "Schema Evolution"
      ],
      "description": "An event is a public record of something that happened. Once multiple consumers depend on it, its schema, meaning, ordering, and delivery behavior form a contract—even if the producer…",
      "contentMarkdown": "An event is a public record of something that happened. Once multiple consumers depend on it, its schema, meaning, ordering, and delivery behavior form a contract—even if the producer never wrote that contract down.\n\n## Specify meaning before shape\n\nName an event in past tense and define the business fact it represents. `CustomerAddressChanged` is more stable than `UpdateCustomerCache`. Document who owns the fact, when it is emitted, the identity of the subject, and whether consumers may use it as a source of truth.\n\nAn envelope typically includes an event identifier, type, schema version, occurrence time, producer, subject identifier, and trace context. Business data belongs in the payload. Avoid copying large mutable aggregates into every event unless consumers genuinely need the snapshot.\n\n## Prefer additive evolution\n\nAdding an optional field is usually safer than changing the type or meaning of an existing one. Consumers should tolerate fields they do not recognise. Required-field removal, enum narrowing, identifier reinterpretation, or a change in units is a new contract even if the serialized schema still validates.\n\nCompatibility tooling can catch structural breaks. It cannot decide whether `total` changed from pre-tax to post-tax. Pair a schema registry with ownership, examples, semantic review, and consumer tests. Apache Kafka’s [design documentation](https://kafka.apache.org/documentation/#design) explains the log and consumer model underlying many event systems; delivery semantics still depend on the application’s processing boundaries.\n\n## Make migration explicit\n\nFor a breaking change, publish a new event type or version, dual-write for a bounded period, and measure consumer migration. Do not remove the old contract because its topic “looks quiet”; confirm registered consumers and replay jobs have moved. State the support window and rollback path.\n\nOrdering should be promised only within a defined key and partition strategy. Consumers must handle duplicates and delayed events. Include enough identity to make processing idempotent, and define how corrections or tombstones work.\n\n## Treat replay as a production feature\n\nReprocessing historical events can overload dependencies or apply obsolete logic. Version consumer behavior, isolate replay capacity, preserve original timestamps, and distinguish replay traffic in telemetry. Test schemas against representative historical records before deployment.\n\nEvent-driven architecture reduces temporal coupling, not organisational responsibility. A durable event contract gives producers freedom to change implementation while giving consumers a fact they can continue to trust.\n\n## Separate event types\n\nA **domain event** records a business fact such as `InvoiceIssued`. An **integration event** is the stable representation intentionally shared outside the owning boundary. A **command** asks another component to perform work and may be refused. Mixing these meanings makes ownership unclear and encourages consumers to depend on internal state transitions.\n\nPublish only facts for which the producer is authoritative. If a service copies an event from another domain and republishes it as its own fact, consumers may no longer know which source to trust.\n\n## Define delivery behavior\n\nDocument partition key, ordering scope, retention, retry behavior, duplicate expectations, maximum payload, and dead-letter handling. Ordering across a whole topic is expensive and often unnecessary; ordering per account or aggregate is usually more useful. Changing the partition key can change both ordering and load distribution, so treat it as a contract change.\n\nConsumers should persist their processing checkpoint with the resulting state change where possible. If they acknowledge first and write later, a crash can lose work. If they write first and acknowledge later, duplicates are expected and processing must be idempotent.\n\n## Govern without centralising every change\n\nGive each contract an owner and machine-readable schema. Automated compatibility checks should run in producer CI, while semantic review is required for changes to meaning, units, identifiers, privacy classification, or lifecycle. Maintain a consumer registry based on actual subscriptions and declared ownership.\n\n## Migration checklist\n\nBefore changing an event, identify consumers and replay jobs, classify the change as additive or breaking, publish examples, test old and new schemas against representative records, define a dual-publish window, observe consumer migration, and specify removal criteria. After retirement, delete obsolete permissions, topics, transformations, and monitoring so the old contract does not remain an unowned production surface.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Design APIs for safe retries",
      "canonical": "https://doshidhruv.com/notes/design-apis-for-safe-retries/",
      "datePublished": "2022-12-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Distributed systems",
        "API Design",
        "Idempotency",
        "Distributed Systems"
      ],
      "description": "Networks fail in ambiguous ways. A client can time out after the server commits a change but before the response arrives. Retrying may be necessary, yet repeating the operation may…",
      "contentMarkdown": "Networks fail in ambiguous ways. A client can time out after the server commits a change but before the response arrives. Retrying may be necessary, yet repeating the operation may charge a card twice, create two orders, or send duplicate notifications. Safe retry behavior must be part of the API contract.\n\n## Separate method semantics from business semantics\n\nHTTP defines PUT, DELETE, and safe methods such as GET as idempotent at the method level; repeating the same request should have the same intended effect. POST is not generally idempotent. The current semantics are defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-idempotent-methods).\n\nBusiness operations often still need an explicit idempotency mechanism. A payment creation endpoint may accept an idempotency key representing one client intent. The server stores the key, a fingerprint of relevant request fields, the operation state, and the eventual response.\n\n## Define key behavior\n\nThe contract should specify:\n\n- who generates the key and its uniqueness scope;\n- how long the server retains it;\n- whether reuse with a different payload is rejected;\n- what concurrent requests with the same key receive;\n- which response, including errors, is replayed;\n- whether the key covers one resource or a wider workflow.\n\nPersist the key and state change atomically when possible. A cache written after the database commit recreates the failure window the key was meant to close.\n\n## Model incomplete work\n\nLong-running operations need states such as accepted, in progress, succeeded, and failed. A retry should return or reference the existing operation rather than starting another. If downstream side effects are asynchronous, propagate a stable operation identifier and make each consumer idempotent as well.\n\nExactly-once delivery is not a property an HTTP endpoint can promise across arbitrary dependencies. The practical goal is an effectively-once business outcome built from durable identifiers, deduplication, transactional boundaries, and reconciliation.\n\n## Make retry policy observable\n\nRecord idempotency-key collisions, duplicate attempts, retention misses, and incomplete operations. Do not log raw keys if they expose sensitive client information. Test timeouts at each boundary, including after commit and before response.\n\nRetries are normal distributed-systems behavior. An API that documents and tests them is safer than one that assumes clients will call every operation exactly once.\n\n## Choose the storage boundary\n\nThe idempotency record must survive the same failures as the business operation. For a single database transaction, store the request key and resulting resource alongside the state change with a uniqueness constraint. If the workflow crosses services, use a durable operation record and an outbox so publication of downstream work can be retried without recreating the business action.\n\nHash or canonicalise the relevant request body and associate that fingerprint with the key. If a client reuses the key with different input, return a conflict rather than silently replaying an unrelated result. Document which headers or defaults participate in the fingerprint.\n\n## Define response behavior\n\nA concurrent duplicate may receive the completed response, an “operation in progress” response with a status URL, or a conflict that tells the client to poll. Pick one behavior and make it consistent. Preserve stable resource identifiers across retries.\n\nDo not cache every failure indefinitely. A validation failure may be safe to return again, while a transient dependency failure may be retriable under the same intent. The API contract should distinguish whether the key has been consumed and when a client needs a new one.\n\n## Extend protection downstream\n\nThe API can be idempotent while a consumer sends two emails or applies two credits. Give each side effect a stable business identifier and enforce uniqueness at the system that owns that effect. Inbox and outbox patterns, deduplication tables, and reconciliation jobs provide practical protection when a distributed transaction is unavailable.\n\n## Test ambiguous outcomes\n\nInject failure after validation, after database commit, after event publication, and before response delivery. Retry concurrently from multiple clients. Test key expiry, process restart, delayed messages, and a response lost after success. Verify the resulting business state, not only HTTP status codes.\n\nThe design is complete when a caller can retry an unknown outcome safely and operators can explain exactly which intent was executed.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Service-level objectives are decision tools",
      "canonical": "https://doshidhruv.com/notes/service-level-objectives-are-decision-tools/",
      "datePublished": "2022-10-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "SRE",
        "SLO",
        "Reliability"
      ],
      "description": "A service level objective is not a decorative percentage on a dashboard. It is an agreement about the reliability users need and a mechanism for deciding how engineering capacity should…",
      "contentMarkdown": "A service-level objective is not a decorative percentage on a dashboard. It is an agreement about the reliability users need and a mechanism for deciding how engineering capacity should be spent.\n\n## Measure a user outcome\n\nChoose a service-level indicator at the point where the user experiences success or failure. For an API, that may be the proportion of valid requests completed correctly within a latency threshold. For a data pipeline, it may be records available by an agreed deadline. Infrastructure availability alone rarely describes the whole experience.\n\nDefine the population precisely: which requests count, which exclusions are legitimate, where measurement occurs, and how missing telemetry behaves. A ratio without those rules will be interpreted differently during every incident.\n\n## Set the objective from consequences\n\nThe objective should reflect the cost of failure to users and the cost of delivering additional reliability. More nines are not automatically better. They increase redundancy, testing, operational, and coordination requirements. The [Google SRE Workbook](https://sre.google/workbook/implementing-slos/) recommends starting from what users need and iterating as evidence improves.\n\nAn error budget translates the objective into allowable unreliability. If a 30-day target is 99.9%, the budget is roughly 43 minutes, but time alone may not capture a request-based indicator. Track budget consumption in the same unit as the SLI.\n\n## Connect the budget to action\n\nAgree on policy before the budget is exhausted. Rapid burn may page the on-call team. Sustained burn may pause risky releases, prioritise reliability work, or trigger an architectural review. Healthy budget may support normal delivery rather than justify consuming unreliability intentionally.\n\nUse multiple burn-rate windows so a brief severe event and a slow persistent regression are both visible. Alerts should correspond to a meaningful threat to the objective, not every small fluctuation.\n\n## Review the model\n\nAn SLO can be met while users are unhappy if the indicator misses an important journey. It can also be impossible because the dependency contract does not support it. Review objectives after incidents, major product changes, and shifts in traffic or user expectations.\n\nThe point is not to make reliability mathematically impressive. It is to make the trade between feature delivery and operational risk explicit, shared, and grounded in the experience the system exists to provide.\n\n## Build the indicator carefully\n\nFor a request-based service, a common SLI is:\n\n`good valid requests / total valid requests`\n\n“Good” may require both a successful result and completion below a latency threshold. “Valid” might exclude malformed client requests but should not exclude server failures, dependency failures, or slow responses merely because they are inconvenient. Write exclusions so an independent reviewer can reproduce the calculation.\n\nFor a multi-step journey, measure the completed outcome where possible. A checkout service can return successful API responses while payment confirmation never reaches the user. Synthetic journeys, business events, or client-side signals may represent that experience better than server availability alone.\n\n## Use burn rate instead of remaining minutes alone\n\nBurn rate compares current error consumption with the rate permitted by the objective. A burn rate of 1 consumes budget exactly at the sustainable rate; a burn rate of 10 consumes it ten times faster. Pair a short window with a longer confirmation window to detect severe incidents quickly without paging on a single transient sample.\n\nSet notification and paging thresholds from the fraction of budget at risk. A ticket might be appropriate for gradual degradation; a page should require a condition that needs immediate human action.\n\n## Handle dependencies explicitly\n\nA service cannot sustainably promise more than its critical dependencies unless it adds caching, redundancy, graceful degradation, or another form of insulation. Map dependency objectives to the user journey and identify where budgets compound. Do not simply copy a provider’s availability number into the product commitment.\n\n## Review checklist\n\nConfirm that the SLI represents a user outcome, source data is independently verifiable, exclusions are narrow, the objective has product agreement, alert policy maps to error-budget risk, and remediation authority is clear. Review whether teams actually make different release or investment decisions because the SLO exists. If not, it is reporting, not reliability management.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Treat telemetry as a production contract",
      "canonical": "https://doshidhruv.com/notes/treat-telemetry-as-a-production-contract/",
      "datePublished": "2022-08-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Observability",
        "OpenTelemetry",
        "Platform Engineering"
      ],
      "description": "Telemetry is often added after a service is built: a few logs, default metrics, and traces sampled when something fails. At scale, that creates inconsistent names, missing context,…",
      "contentMarkdown": "Telemetry is often added after a service is built: a few logs, default metrics, and traces sampled when something fails. At scale, that creates inconsistent names, missing context, uncontrolled cost, and dashboards that cannot answer operational questions. A better approach treats telemetry as an interface with owners and compatibility rules.\n\n## Define the questions first\n\nBegin with the decisions operators must make:\n\n- Is the service meeting its user-facing objective?\n- Which dependency or release changed the failure rate?\n- Can a request be followed across trust and service boundaries?\n- Which tenant, region, or operation is affected?\n- Is telemetry loss hiding a production problem?\n\nSignals should exist because they answer one of these questions. Collection without a use case produces volume, not observability.\n\n## Standardise the envelope\n\nUse consistent resource attributes for service identity, environment, version, region, and ownership. Define span names at stable operation boundaries rather than including unbounded identifiers. Logs should carry trace and span identifiers where available. Metrics need explicit units, aggregation intent, and cardinality limits.\n\nThe [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) provide shared names for common operations and resources. Adopt them before inventing local vocabulary, then document the organisation-specific attributes that remain.\n\n## Put governance in the pipeline\n\nValidate telemetry during development and delivery. Tests can assert that critical spans exist, attributes avoid sensitive data, metric labels remain bounded, and schema changes are compatible. Collectors can enforce redaction, routing, sampling, and export policy centrally, but they cannot repair missing application context.\n\nVersion the contract. A renamed metric or attribute can break alerts, service-level indicators, cost reports, and incident tooling even when the application itself remains healthy. Deprecate names, observe consumer migration, and remove them deliberately.\n\n## Operate the telemetry system\n\nThe telemetry pipeline is production infrastructure. Measure dropped spans, exporter failures, queue saturation, ingestion latency, and cost by signal and service. Define behavior during downstream failure: buffer within a bound, reduce sampling, or discard lower-value data before critical signals.\n\nTelemetry becomes useful when teams can rely on its meaning. A vendor-neutral collection layer helps, but the deeper win is organisational: every service emits a predictable operational story, and every platform component preserves that story from process to query.\n\n## Control cardinality and cost\n\nMetric dimensions such as customer ID, request ID, URL, or error message can create an effectively unbounded number of time series. That increases ingestion cost and may make queries or alerts unusable. Define an allowlist of bounded dimensions and move high-cardinality investigation context into traces or structured logs.\n\nSampling is also a contract. Head sampling makes an early decision before the full trace is known; tail sampling can preserve errors or slow traces after observing the completed path but requires buffering and additional collector capacity. Document which traffic can be discarded and which signals—security decisions, critical transactions, or objective calculations—must remain complete.\n\n## Protect data at collection\n\nTelemetry can contain credentials, personal information, query text, or document contents. Classify attributes before release, redact as close to the source as possible, and enforce additional controls in the collector. Apply retention, residency, and access policy by data class rather than assuming every observability user should see every field.\n\nTrace propagation across an external boundary needs an explicit trust decision. Accepting arbitrary baggage or trace identifiers can leak information or corrupt internal analysis. Validate permitted fields and create a new internal context when the boundary requires it.\n\n## Test the failure path\n\nExercise collector loss, exporter throttling, unavailable backends, malformed payloads, and sudden cardinality growth. Application availability should not normally depend on synchronous telemetry export. Use bounded queues and define whether the process drops, spills, or reduces data when the pipeline is unhealthy.\n\n## Review checklist\n\nFor each production service, verify stable resource identity, semantic operation names, trace-log correlation, bounded metric attributes, sensitive-data controls, schema ownership, pipeline health monitoring, and a documented sampling policy. Then take a real operational question—such as “which release increased checkout latency?”—and prove the signals can answer it without a custom forensic project.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Architecture decision records that remain useful",
      "canonical": "https://doshidhruv.com/notes/architecture-decision-records-that-remain-useful/",
      "datePublished": "2022-06-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Staff engineering",
        "Architecture",
        "Decision Records",
        "Technical Leadership"
      ],
      "description": "An architecture decision record is valuable only if a future engineer can understand what changed, why it changed, and when the decision should be reconsidered. Length is not the goal.…",
      "contentMarkdown": "An architecture decision record is valuable only if a future engineer can understand what changed, why it changed, and when the decision should be reconsidered. Length is not the goal. Durable context is.\n\n## Record the decision boundary\n\nA practical record needs five things:\n\n1. **Context:** the problem, constraints, and forces that matter.\n2. **Decision:** the choice in direct language.\n3. **Alternatives:** credible options considered and why they lost.\n4. **Consequences:** benefits, costs, risks, and follow-up work.\n5. **Revisit conditions:** evidence that would make the team reopen the choice.\n\n“Use Kafka” is not a decision record. “Use the managed Kafka service for durable domain-event distribution because consumers require replay and independent scaling; do not use it for synchronous request-response workflows” establishes a boundary.\n\n## Keep records close to the system\n\nStore records in the repository that owns the decision when possible. Review them with the code or infrastructure change, link them from relevant runbooks and diagrams, and assign stable identifiers. A small index can show status—proposed, accepted, superseded, or deprecated—and link a superseded record to its replacement.\n\nThe original [ADR guidance by Michael Nygard](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) deliberately uses a small structure. The value comes from capturing the forces and consequences while they are still known, not from producing a large template.\n\n## Separate approval from documentation\n\nAn ADR should preserve a decision; it should not become a universal approval gate. Match review to impact. A local, reversible library choice may need only the owning team. A cross-domain data contract or identity pattern needs representatives from affected teams and security. The review path should be explicit before authors start writing.\n\n## Revisit with evidence\n\nGood records include observable triggers: request volume exceeds the design range, a vendor ends support, recovery objectives change, or operating cost crosses an agreed threshold. Calendar reviews can help, but evidence-based triggers are more meaningful.\n\nDuring incidents and migrations, update consequences rather than rewriting history. The original record should continue to explain why a once-reasonable choice was made.\n\nThe best ADRs reduce repeated debate. They let a new engineer challenge a decision with the same context the original team had—and with better evidence when the system has changed.\n\n## Match the record to the decision\n\nNot every choice deserves an ADR. Record decisions that are expensive to reverse, affect multiple teams, establish a security or data boundary, introduce a long-lived dependency, or constrain future designs. Routine implementation details belong in code and ordinary review. This keeps the decision index useful instead of turning it into a second commit log.\n\nA decision record should name its scope. “Standardise asynchronous integration” is too broad if the actual choice applies only to customer-notification events. State the systems, environments, and teams affected, plus any explicitly excluded use cases.\n\n## Example consequence model\n\nSuppose a team chooses a managed event service over operating its own cluster. Immediate benefits may include reduced control-plane work, supported upgrades, and clearer availability commitments. Costs may include provider limits, data-transfer charges, reduced configuration freedom, and a migration dependency. Follow-up actions might cover schema governance, quota monitoring, disaster recovery, and cost ownership.\n\nWriting consequences this way prevents a false binary of “approved” and “rejected.” Every architecture choice buys some properties by accepting others.\n\n## Keep the lifecycle visible\n\nLink records to code, diagrams, threat models, service catalogues, and delivery milestones. During review, check whether the implementation still matches the accepted decision. During an incident, link relevant findings back to the record and update its known consequences.\n\nDo not edit an accepted record to make the past look cleaner. Add a dated amendment or superseding ADR. The historical chain explains how the system arrived at its present state and protects future teams from repeating discarded options without new evidence.\n\n## Review checklist\n\nAn ADR is ready when a reader can identify the problem, decision owner, affected boundary, credible alternatives, trade-offs, implementation obligations, and revisit trigger. If the record cannot say what evidence would invalidate the decision, it is probably documenting a preference rather than an architectural judgment.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Design the platform as a product",
      "canonical": "https://doshidhruv.com/notes/design-the-platform-as-a-product/",
      "datePublished": "2022-04-01",
      "dateModified": "2026-07-28",
      "topics": [
        "Platform architecture",
        "Platform Engineering",
        "Developer Experience",
        "Architecture"
      ],
      "description": "An internal platform is successful when product teams choose it because it removes work, not because governance forces them to use it. That changes the architecture question from “what…",
      "contentMarkdown": "An internal platform is successful when product teams choose it because it removes work, not because governance forces them to use it. That changes the architecture question from “what infrastructure should we standardise?” to “what recurring developer problem should we solve?”\n\n## Start with a user journey\n\nMap the path from an idea to a production service. A useful first version usually covers a small number of high-friction steps: creating a repository, obtaining an environment, deploying safely, observing the service, and requesting support. Record how long each step takes, where approvals wait, and where teams build their own workaround.\n\nThe platform boundary should follow those repeated problems. A portal without reliable deployment primitives is a catalogue, not a platform. A collection of Terraform modules without documentation, support, and versioning is a library, not a product.\n\n## Define a contract, not a golden cage\n\nA paved road should make the safe path fast while preserving an explicit escape hatch. Define:\n\n- the interface a team consumes;\n- the security and operational controls supplied by default;\n- the responsibilities that remain with the service team;\n- the supported lifecycle and migration policy;\n- the process for capabilities the platform does not cover.\n\nThis contract matters more than the implementation. It allows the platform team to replace an underlying tool without surprising its users.\n\n## Measure outcomes\n\nAdoption alone is weak evidence. A platform can have mandatory adoption and still create expensive friction. Combine quantitative signals—lead time, deployment frequency, failed changes, environment provisioning time, support volume—with interviews and workflow observation. The [DORA research program](https://dora.dev/research/) provides a useful starting point for delivery measures, while the [CNCF platform engineering white paper](https://tag-app-delivery.cncf.io/whitepapers/platforms/) frames platforms as products serving internal users.\n\nTrack the cost transferred to product teams as carefully as the cost removed from the platform team. If a standard reduces central effort but adds manual work to every service owner, it is not leverage.\n\n## Keep the first promise narrow\n\nBegin with one well-understood service archetype and make its path excellent. Publish ownership, service levels, known constraints, and a roadmap tied to user evidence. Expand only when the first path is reliable enough that teams recommend it to one another.\n\nA good platform does not hide engineering decisions. It packages the common ones, makes exceptional ones visible, and gives teams a dependable path from code to operation.\n\n## A concrete service contract\n\nConsider a platform capability called “production web service.” Its contract could accept a repository, runtime, owner, data classification, availability tier, and scaling range. In return it creates build and deployment workflows, workload identity, network policy, dashboards, alerts, a service catalogue entry, and a documented support path. The contract should identify what it does **not** provide: perhaps persistent databases, cross-region recovery, or public ingress without security review.\n\nThat boundary gives teams enough information to decide whether the paved road fits. It also lets the platform team test the complete promise. A successful infrastructure plan is not sufficient if the deployment never becomes observable or the ownership record is missing.\n\n## Design the operating model\n\nEvery platform capability needs an owner, support hours, service objectives, incident process, and upgrade policy. Decide how breaking changes are communicated and how long old versions remain supported. Make dependency status visible so product teams can distinguish an application failure from a platform failure.\n\nUse contribution paths carefully. Product teams may submit improvements, but the platform team still owns coherence, review, documentation, and long-term support. An internal open-source model without maintainers merely distributes responsibility ambiguously.\n\n## Common failure modes\n\n- **Portal-first delivery:** a polished front end masks unreliable or manual workflows.\n- **Mandatory adoption:** usage rises while workarounds and resentment remain invisible.\n- **Unlimited flexibility:** every option becomes supported, so upgrades become impossible.\n- **No migration funding:** a better platform exists, but teams cannot leave the old path.\n- **Platform as project:** funding ends at launch even though users need continuing operation.\n\n## Review checklist\n\nBefore adding a capability, confirm that a named user problem exists, the interface and responsibilities are written, the full journey is automated and tested, operational ownership is funded, success measures include user effort, and a retirement or migration path exists. If those answers are weak, more technology will not make the platform a product.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Deep Learning Explained - From Basics to Advanced",
      "canonical": "https://doshidhruv.com/notes/deep-learning-explained-from-basics-to-advanced/",
      "datePublished": "2022-03-01",
      "dateModified": "2022-03-01",
      "topics": [
        "AI & machine learning",
        "Artificial Intelligence",
        "Deep Learning"
      ],
      "description": "Deep Learning, a subset of machine learning, has revolutionized artificial intelligence by enabling computers to learn from vast amounts of data. Its applications range from image and…",
      "contentMarkdown": "Deep Learning, a subset of machine learning, has revolutionized artificial intelligence by enabling computers to learn from vast amounts of data. Its applications range from image and speech recognition to natural language processing and autonomous vehicles. In this blog, we will explore the fundamentals of deep learning, its architecture, and advanced concepts.\n\n\n## What is Deep Learning?\n\nDeep Learning is a type of machine learning that uses neural networks with many layers (hence \"deep\") to model complex patterns in data. Unlike traditional algorithms, which require manual feature extraction, deep learning models automatically discover the representations needed for classification or prediction.\n\n## Fundamentals of Neural Networks\n\n### Neurons and Layers\n\nAt the core of deep learning are artificial neural networks (ANNs), inspired by the human brain's structure. An ANN consists of layers of nodes (neurons):\n\n1. **Input Layer**: Receives the input data.\n2. **Hidden Layers**: Intermediate layers where the computation happens.\n3. **Output Layer**: Produces the final output.\n\nEach neuron receives input, processes it with a weight and bias, applies an activation function, and passes the output to the next layer.\n\n### Activation Functions\n\nActivation functions introduce non-linearity into the network, allowing it to learn complex patterns. Common activation functions include:\n\n- **Sigmoid**: \\( \\sigma(x) = \\frac{1}{1 + e^{-x}} \\)\n- **ReLU (Rectified Linear Unit)**: \\( f(x) = \\max(0, x) \\)\n- **Tanh**: \\( \\tanh(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}} \\)\n\n## Training Neural Networks\n\nTraining a neural network involves finding the optimal weights and biases that minimize the error between the predicted and actual outputs. This process is typically done using the following steps:\n\n### Forward Propagation\n\nInput data passes through the network layer by layer, undergoing transformations at each neuron until it reaches the output layer. The output is compared to the actual result to compute the error.\n\n### Backpropagation\n\nBackpropagation adjusts the weights and biases to reduce the error. It involves two main steps:\n\n1. **Calculating the Gradient**: The gradient of the loss function with respect to each weight is computed using the chain rule.\n2. **Updating the Weights**: Weights are updated using gradient descent or other optimization algorithms to minimize the loss function.\n\n## Optimization Algorithms\n\n1. **Gradient Descent**: Iteratively adjusts weights to minimize the loss function.\n2. **Stochastic Gradient Descent (SGD)**: Uses a random subset of data for each iteration, speeding up the process.\n3. **Adam**: Combines the benefits of SGD and RMSProp, adapting learning rates for each parameter.\n\n## Deep Learning Architectures\n\n### Convolutional Neural Networks (CNNs)\n\nCNNs are specialized for processing grid-like data, such as images. They use convolutional layers to detect local patterns and pooling layers to reduce dimensionality. Applications include image recognition and video analysis.\n\n\n\n<a href='https://postimg.cc/YGpBDYb4' target='_blank'><img src='/images/notes/deep-learning-network.png' border='0' alt='image'/></a>\n\n### Recurrent Neural Networks (RNNs)\n\nRNNs are designed for sequential data, such as time series or text. They have connections that form cycles, allowing information to persist. Variants like Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) address the vanishing gradient problem. Applications include language modeling and speech recognition.\n\n<a href='https://postimg.cc/8sXgwFW5' target='_blank'><img src='/images/notes/deep-learning-training.png' border='0' alt='image'/></a>\n\n### Generative Adversarial Networks (GANs)\n\nGANs consist of two networks: a generator and a discriminator. The generator creates fake data, and the discriminator attempts to distinguish between real and fake data. They are trained simultaneously, improving each other's performance. Applications include image generation and data augmentation.\n\n<a href='https://postimg.cc/SJwFFwgb' target='_blank'><img src='/images/notes/deep-learning-architectures.png' border='0' alt='image'/></a>\n\n## Advanced Deep Learning Concepts\n\n### Transfer Learning\n\nTransfer learning leverages pre-trained models on new, similar tasks, reducing the need for large datasets and extensive training. It's particularly useful in applications like image classification, where pre-trained models on large datasets like ImageNet can be fine-tuned for specific tasks.\n\n### Reinforcement Learning\n\nIn reinforcement learning, agents learn by interacting with their environment, receiving rewards or penalties based on their actions. Combining reinforcement learning with deep learning has led to significant advancements in fields such as game playing (e.g., AlphaGo) and robotics.\n\n### Autoencoders\n\nAutoencoders are neural networks used for unsupervised learning of efficient codings. They encode the input into a lower-dimensional representation and then decode it back to the original form. Applications include anomaly detection, image denoising, and data compression.\n\n### Attention Mechanisms and Transformers\n\nAttention mechanisms allow models to focus on specific parts of the input sequence, improving performance in tasks like translation and text generation. Transformers, which rely heavily on attention mechanisms, have revolutionized NLP with models like BERT and GPT-3.\n\n\n## Practical Applications of Deep Learning\n\n1. **Image and Video Recognition**: Detecting objects, faces, and activities in images and videos.\n2. **Natural Language Processing (NLP)**: Understanding and generating human language, including translation, sentiment analysis, and chatbots.\n3. **Healthcare**: Diagnosing diseases from medical images, predicting patient outcomes, and personalizing treatment.\n4. **Autonomous Vehicles**: Enabling self-driving cars to perceive and navigate the environment.\n5. **Finance**: Fraud detection, algorithmic trading, and risk management.\n\n## Challenges and Future Directions\n\n### Data and Computational Requirements\n\nDeep learning models often require large amounts of data and significant computational resources, posing challenges for smaller organizations. Techniques like data augmentation, transfer learning, and more efficient algorithms are being developed to address these issues.\n\n### Interpretability and Transparency\n\nDeep learning models are often considered \"black boxes\" due to their complexity, making it difficult to understand their decision-making process. Research in explainable AI (XAI) aims to make these models more transparent and interpretable.\n\n### Ethical Considerations\n\nAs deep learning becomes more pervasive, ethical concerns related to bias, privacy, and the societal impact of AI systems need to be addressed. Developing fair, accountable, and transparent AI systems is crucial for their responsible deployment.\n\n## Conclusion\n\nDeep learning is a powerful and versatile tool that has transformed numerous industries by enabling computers to learn and make decisions from vast amounts of data. From its basic principles to advanced architectures and applications, understanding deep learning is essential for leveraging its full potential. As the field continues to evolve, ongoing research and innovation will drive further advancements, shaping the future of technology and society.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "The Fundamentals of Machine Learning",
      "canonical": "https://doshidhruv.com/notes/the-fundamentals-of-machine-learning/",
      "datePublished": "2022-02-01",
      "dateModified": "2022-02-01",
      "topics": [
        "AI & machine learning",
        "Artificial Intelligence",
        "Machine Learning"
      ],
      "description": "Machine Learning (ML) is a subset of artificial intelligence (AI) that enables systems to learn from data and improve their performance over time without being explicitly programmed.…",
      "contentMarkdown": "Machine Learning (ML) is a subset of artificial intelligence (AI) that enables systems to learn from data and improve their performance over time without being explicitly programmed. This blog explores the fundamentals of ML, its types, algorithms, and applications.\n\n\n## What is Machine Learning?\n\nMachine Learning involves the use of algorithms and statistical models to analyze and draw inferences from patterns in data. Unlike traditional programming, where a computer follows explicit instructions, ML systems use data to identify patterns and make decisions.\n\n## Key Components of Machine Learning\n\n1. **Data**: The cornerstone of ML. Data can be structured (e.g., databases) or unstructured (e.g., text, images).\n2. **Algorithms**: The mathematical models that process data to learn patterns.\n3. **Model**: The result of training an algorithm with data.\n4. **Training**: The process of feeding data to an algorithm to build a model.\n5. **Evaluation**: Assessing the model's performance using metrics.\n6. **Prediction**: Using the trained model to make predictions on new data.\n\n## Types of Machine Learning\n\n1. **Supervised Learning**:\n   - **Definition**: The algorithm learns from labeled data, where the input-output pairs are provided.\n   - **Algorithms**: Linear Regression, Decision Trees, Support Vector Machines (SVM), Neural Networks.\n   - **Applications**: Email spam detection, image recognition, and predictive maintenance.\n\n2. **Unsupervised Learning**:\n   - **Definition**: The algorithm learns from unlabeled data, identifying patterns and relationships.\n   - **Algorithms**: K-Means Clustering, Principal Component Analysis (PCA), Hierarchical Clustering.\n   - **Applications**: Customer segmentation, anomaly detection, and market basket analysis.\n\n3. **Semi-Supervised Learning**:\n   - **Definition**: Combines labeled and unlabeled data for training.\n   - **Algorithms**: Variants of supervised algorithms adapted to semi-supervised learning.\n   - **Applications**: Speech recognition, natural language processing (NLP), and image classification.\n\n4. **Reinforcement Learning**:\n   - **Definition**: The algorithm learns by interacting with an environment, receiving rewards or penalties.\n   - **Algorithms**: Q-Learning, Deep Q-Networks (DQN), Policy Gradient Methods.\n   - **Applications**: Robotics, game playing (e.g., AlphaGo), and automated trading.\n\n    <!-- <center><img src=\"https://imgur.com/a/jzPicBf\" style=\"height:40%; width:80%;\"></center><br> -->\n<a href='https://postimg.cc/23sBq56q' target='_blank'><img src='/images/notes/machine-learning-model.png' border='0' alt='image'/></a>\n\n\n## Key Machine Learning Algorithms\n\n1. **Linear Regression**:\n   - **Use**: Predicting continuous values.\n   - **Example**: House price prediction.\n\n2. **Logistic Regression**:\n   - **Use**: Binary classification.\n   - **Example**: Spam email detection.\n\n3. **Decision Trees**:\n   - **Use**: Classification and regression.\n   - **Example**: Customer churn prediction.\n\n4. **Support Vector Machines (SVM)**:\n   - **Use**: Classification and regression.\n   - **Example**: Handwritten digit recognition.\n\n5. **K-Means Clustering**:\n   - **Use**: Unsupervised learning for clustering.\n   - **Example**: Customer segmentation.\n\n6. **Principal Component Analysis (PCA)**:\n   - **Use**: Dimensionality reduction.\n   - **Example**: Image compression.\n\n7. **Neural Networks**:\n   - **Use**: Complex pattern recognition.\n   - **Example**: Image and speech recognition.\n\n## Machine Learning Workflow\n\n1. **Data Collection**: Gathering relevant data from various sources.\n2. **Data Preprocessing**: Cleaning and transforming data to make it suitable for analysis.\n3. **Feature Engineering**: Selecting and transforming variables (features) to improve model performance.\n4. **Model Training**: Applying algorithms to training data to create a model.\n5. **Model Evaluation**: Assessing the model's accuracy and performance using validation data.\n6. **Model Deployment**: Integrating the model into a production environment to make predictions on new data.\n7. **Monitoring and Maintenance**: Continuously monitoring the model's performance and updating it as needed.\n\n## Applications of Machine Learning\n\n1. **Healthcare**:\n   - **Diagnosis**: ML algorithms assist in diagnosing diseases from medical images.\n   - **Personalized Medicine**: Tailoring treatments based on patient data.\n\n2. **Finance**:\n   - **Fraud Detection**: Identifying fraudulent transactions.\n   - **Algorithmic Trading**: Making trading decisions based on data analysis.\n\n3. **Marketing**:\n   - **Customer Segmentation**: Dividing customers into distinct groups based on behavior.\n   - **Personalized Advertising**: Delivering targeted ads to users.\n\n4. **Retail**:\n   - **Recommendation Systems**: Suggesting products to customers based on past behavior.\n   - **Inventory Management**: Optimizing stock levels based on demand forecasts.\n\n5. **Transportation**:\n   - **Autonomous Vehicles**: Enabling self-driving cars to navigate safely.\n   - **Route Optimization**: Finding the most efficient routes for delivery services.\n\n6. **Natural Language Processing (NLP)**:\n   - **Speech Recognition**: Converting spoken language into text.\n   - **Language Translation**: Automatically translating text from one language to another.\n\n## Challenges in Machine Learning\n\n1. **Data Quality**: Ensuring the data used for training is clean, relevant, and representative.\n2. **Overfitting**: When a model performs well on training data but poorly on new data.\n3. **Interpretability**: Understanding how a model makes decisions, especially in complex models like deep neural networks.\n4. **Scalability**: Handling large volumes of data efficiently.\n5. **Ethical Considerations**: Addressing bias, fairness, and privacy concerns in ML applications.\n\n## Future of Machine Learning\n\nThe future of ML is promising, with continuous advancements in algorithms, computing power, and data availability. Emerging trends include:\n\n1. **Explainable AI (XAI)**: Developing models that provide clear explanations for their decisions.\n2. **Federated Learning**: Training models across decentralized devices without sharing raw data.\n3. **Automated Machine Learning (AutoML)**: Simplifying the creation of ML models through automation.\n4. **Integration with IoT**: Enhancing IoT applications with intelligent data analysis and decision-making.\n\n## Conclusion\n\nMachine Learning is a transformative technology with the potential to revolutionize various industries. Understanding its fundamentals, from types and algorithms to applications and challenges, is crucial for harnessing its power effectively. As ML continues to evolve, it will drive innovation and offer new solutions to complex problems, shaping the future of technology and society.\n\n---\n\nBy understanding and implementing the fundamentals of machine learning, individuals and organizations can unlock the potential of this powerful technology, driving innovation and solving complex challenges across various domains.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Introduction to Artificial Intelligence - History and Evolution",
      "canonical": "https://doshidhruv.com/notes/introduction-to-artificial-intelligence-history-and-evolution/",
      "datePublished": "2022-01-01",
      "dateModified": "2022-01-01",
      "topics": [
        "AI & machine learning",
        "Artificial Intelligence"
      ],
      "description": "Artificial Intelligence (AI) has evolved from a niche field of study to a transformative technology influencing many facets of modern life. This blog delves into the history and…",
      "contentMarkdown": "Artificial Intelligence (AI) has evolved from a niche field of study to a transformative technology influencing many facets of modern life. This blog delves into the history and evolution of AI, highlighting key milestones and developments.\n\n## The Origins of AI\n\nThe roots of AI stretch back to ancient myths and legends where intelligent automatons and mechanical beings were imagined. However, AI as a formal academic discipline began in the mid-20th century. The term \"Artificial Intelligence\" was coined by John McCarthy in 1956 at the Dartmouth Conference, an event considered the inception of AI as a formal field. Early AI research was characterized by the exploration of symbolic reasoning and problem-solving.\n\n## The Early Years: 1950s-1970s\n\nIn the 1950s and 1960s, pioneers like Alan Turing, John McCarthy, Marvin Minsky, Herbert Simon, and Allen Newell made significant contributions to AI. Alan Turing's 1950 paper \"Computing Machinery and Intelligence\" posed the fundamental question of whether machines can think, introducing the Turing Test as a measure of machine intelligence.\n\nThe development of early AI programs like the Logic Theorist by Newell and Simon in 1956 showcased AI's potential. This program was capable of proving mathematical theorems, illustrating the feasibility of automated reasoning. Researchers developed systems that could play chess, solve algebra problems, and understand limited natural language.\n\n## The First AI Winter\n\nDespite initial successes, AI research faced substantial challenges in the late 1960s and early 1970s. Early AI systems were limited by the computational power and algorithms available at the time. These limitations led to unmet expectations and skepticism about AI's potential, resulting in reduced funding and interest. This period, known as the first \"AI winter,\" saw a significant slowdown in AI research and development.\n\n## The Resurgence: 1980s-1990s\n\nAI research experienced a resurgence in the 1980s with the advent of expert systems. These systems, which utilized rule-based programming to emulate human expertise in specific domains, found practical applications in fields such as medicine, finance, and manufacturing. The Japanese government's Fifth Generation Computer Systems project also spurred interest and investment in AI during this period.\n\nThe development of machine learning algorithms and neural networks in the late 1980s and early 1990s marked another significant milestone. Researchers like Geoffrey Hinton and Yann LeCun pioneered techniques such as backpropagation, which improved the training of neural networks and expanded their applicability.\n\n## The Rise of Modern AI: 2000s-Present\n\nThe turn of the millennium marked the beginning of modern AI, driven by advancements in computing power, data availability, and algorithmic innovations. The emergence of big data and the proliferation of internet-connected devices provided a wealth of information for training AI models.\n\nDeep learning, a subfield of machine learning, gained prominence in the 2010s with the development of deep neural networks. These networks, capable of learning from vast amounts of data, revolutionized tasks such as image and speech recognition. Landmark achievements, such as Google's DeepMind developing AlphaGo, a program that defeated human champions in the complex game of Go, demonstrated the power of deep learning.\n\nAI's impact on various industries became increasingly evident. In healthcare, AI-powered systems improved diagnostics and personalized treatment plans. In finance, AI algorithms enhanced fraud detection and trading strategies. Autonomous vehicles, powered by AI, promised to transform transportation.\n\n## Ethical and Societal Considerations\n\nAs AI technology advanced, ethical and societal considerations gained prominence. Issues such as bias in AI algorithms, privacy concerns, and the potential for job displacement became critical topics of discussion. Efforts to address these concerns led to the development of frameworks for responsible AI, emphasizing transparency, fairness, and accountability.\n\n## The Future of AI\n\nThe future of AI holds immense potential. Research continues to push the boundaries of what AI can achieve, from developing more advanced natural language processing models to creating AI systems that exhibit general intelligence. The integration of AI with other emerging technologies, such as quantum computing and the Internet of Things (IoT), promises to unlock new possibilities.\n\nAs AI becomes increasingly integrated into everyday life, collaboration between researchers, policymakers, and industry leaders will be crucial in ensuring that AI technologies are developed and deployed responsibly, maximizing their benefits while mitigating potential risks.\n\n### Conclusion\n\nThe history and evolution of Artificial Intelligence is a testament to human ingenuity and the relentless pursuit of knowledge. From its early conceptualization to its current role as a transformative force in technology and society, AI's journey is marked by significant milestones and achievements. As we look to the future, the continued advancement of AI promises to bring about profound changes, shaping the way we live, work, and interact with the world.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is cloud computing?",
      "canonical": "https://doshidhruv.com/notes/what-is-cloud-computing/",
      "datePublished": "2021-12-30",
      "dateModified": "2021-12-30",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Cloud computing is the on demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user.…",
      "contentMarkdown": "**`CLOUD COMPUTING`**<br>\n\nCloud computing is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user. Large clouds often have functions distributed over multiple locations, each location being a data center. Cloud computing relies on sharing of resources to achieve coherence[clarification needed] and economies of scale, typically using a \"pay-as-you-go\" model which can help in reducing capital expenses but may also lead to unexpected operating expenses for unaware users.\n\n*`Service Models`*<br>\nThere are five service models out of them top 3 are widely used and most of the content ofinternet uses these kind of service models.\n\n1. <a href=\"https://blog.doshidhruv.com/posts/Infrastructure-as-a-service/\">Infrastructure as a Service (Iaas)</a>\n2. <a href=\"https://blog.doshidhruv.com/posts/Platform-as-a-service/\">Platform as a Service (PaaS)</a>\n3. <a href=\"https://blog.doshidhruv.com/posts/Software-as-a-service/\">Software as a Service (SaaS)</a>\n4. <a href=\"https://blog.doshidhruv.com/posts/Mobile-Backend-as-a-service/\">Mobile backend as a Service (MBaaS)</a>\n5. <a href=\"https://blog.doshidhruv.com/posts/Serverless-Computing-as-a-service/\">Serverless computing or Function as a Service (FaaS)</a>\n\n\n\n*`Control comparison between all major models`*<br>\n\nIn the following image we have the comparison between the different service models which gives us the insight about the  amount of access and the control we have while using this service models.\n\n<center><img src=\"/images/notes/cloud-overview.png\" style=\"height:40%; width:80%;\"></center><br>\n\n\n*`Major Deployment Models`*<br>\nTo deply the cloud architecture there are three ways we could do. With private being the most used till today organizations are moving to use Hybrid cloud more nowadyas and also migrating towards also.\n\nThe following images displays that what each cloud models offer to the users,\n\n<center><img src=\"/images/notes/cloud-characteristics.png\" style=\"height:40%; width:80%;\"></center><br>\n\n<br>\nThe following image puts more emphasis on the comparison across all three cloud models/\n\n<center><img src=\"/images/notes/cloud-services.jpg\" style=\"height:40%; width:80%;\"></center><br>\n\n\nRead more about them here,\n1. <a href=\"https://blog.doshidhruv.com/posts/Private-cloud-in-cloud-computing/\">Private Cloud</a>\n2. <a href=\"https://blog.doshidhruv.com/posts/Public-cloud-in-cloud-computing/\">Public Cloud</a>\n3. <a href=\"https://blog.doshidhruv.com/posts/Hybrid-cloud-in-cloud-computing/\">Hybrid Cloud </a>\n\n\n*`Other Deployment Models`*\n\n1. <a href=\"https://blog.doshidhruv.com/posts/Community-cloud-in-cloud-computing/\">Community Cloud</a>\n2. <a href=\"https://blog.doshidhruv.com/posts/Distributed-cloud-in-cloud-computing/\">Distributed Cloud</a>\n3. <a href=\"https://blog.doshidhruv.com/posts/Multi-cloud-in-cloud-computing/\">Multi Cloud </a>\n4. <a href=\"https://blog.doshidhruv.com/posts/Poly-cloud-in-cloud-computing/\">Poly Cloud</a>\n<!-- 5. <a href=\"https://dhruvdoshi.github.io/blog/2019/09/04/what-is-wallet-in-blockchain\">Big Data Cloud</a> -->\n5. <a href=\"https://blog.doshidhruv.com/posts/HPC-cloud-in-cloud-computing/\">HPC Cloud</a>\n\n\n*`Security and Privacy`*<br>\n\nCloud computing poses privacy concerns because the service provider can access the data that is in the cloud at any time. It could accidentally or deliberately alter or delete information. Many cloud providers can share information with third parties if necessary for purposes of law and order without a warrant. That is permitted in their privacy policies, which users must agree to before they start using cloud services. Solutions to privacy include policy and legislation as well as end-users' choices for how data is stored. Users can encrypt data that is processed or stored within the cloud to prevent unauthorized access. Identity management systems can also provide practical solutions to privacy concerns in cloud computing. These systems distinguish between authorized and unauthorized users and determine the amount of data that is accessible to each entity. The systems work by creating and describing identities, recording activities, and getting rid of unused identities.\n\nAccording to the Cloud Security Alliance, the top three threats in the cloud are Insecure Interfaces and APIs, Data Loss & Leakage, and Hardware Failure—which accounted for 29%, 25% and 10% of all cloud security outages respectively. Together, these form shared technology vulnerabilities. In a cloud provider platform being shared by different users, there may be a possibility that information belonging to different customers resides on the same data server. Additionally, Eugene Schultz, chief technology officer at Emagined Security, said that hackers are spending substantial time and effort looking for ways to penetrate the cloud. \"There are some real Achilles' heels in the cloud infrastructure that are making big holes for the bad guys to get into\". Because data from hundreds or thousands of companies can be stored on large cloud servers, hackers can theoretically gain control of huge stores of information through a single attack—a process he called \"hyperjacking\". Some examples of this include the Dropbox security breach, and iCloud 2014 leak. Dropbox had been breached in October 2014, having over 7 million of its users passwords stolen by hackers in an effort to get monetary value from it by Bitcoins (BTC). By having these passwords, they are able to read private data as well as have this data be indexed by search engines (making the information public).\n\nThere is the problem of legal ownership of the data (If a user stores some data in the cloud, can the cloud provider profit from it?). Many Terms of Service agreements are silent on the question of ownership. Physical control of the computer equipment (private cloud) is more secure than having the equipment off-site and under someone else's control (public cloud). This delivers great incentive to public cloud computing service providers to prioritize building and maintaining strong management of secure services. Some small businesses that don't have expertise in IT security could find that it's more secure for them to use a public cloud. There is the risk that end users do not understand the issues involved when signing on to a cloud service (persons sometimes don't read the many pages of the terms of service agreement, and just click \"Accept\" without reading). This is important now that cloud computing is becoming popular and required for some services to work, for example for an intelligent personal assistant (Apple's Siri or Google Now). Fundamentally, private cloud is seen as more secure with higher levels of control for the owner, however public cloud is seen to be more flexible and requires less time and money investment from the user.\n\nTo resolve these king of security and privacy concerns we could come up with the cloud service which could be clubbed up with *<a href=\"https://blog.doshidhruv.com/posts/What-is-Blockchain/\">Blockchain Technology</a>*\n\n*`Limitations and Disadvanatages`<br>\nThere are immense potential in cloud computing and features which could be extracted and which makes the developer life easy and easy to extend the resources power with minimal investment.\n<!-- TODO: TO BE UPDATED AFTER ADDITION -->\n1. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Downtime </a>\n2. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Security and Privacy </a>\n3. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Vulnerablity to attack </a>\n4. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Limited control and flexiblity </a>\n5. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Vender Locking </a>\n6. <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\">Cost Concerns </a>",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Downtime with Cloud Computing",
      "canonical": "https://doshidhruv.com/notes/downtime-with-cloud-computing/",
      "datePublished": "2021-12-20",
      "dateModified": "2021-12-20",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Downtine in Cloud Computing Cloud Outage simply refers to the duration when the cloud infrastructure service is unavailable for use. The unavailability may also refer to performance…",
      "contentMarkdown": "**`Downtine in Cloud Computing`**<br>\nCloud Outage simply refers to the duration when the cloud infrastructure service is unavailable for use. The unavailability may also refer to performance inadequacy of the service, as per the agreed SLA metrics. For instance, the incident during which an outage may have only partially impacted a data center may cause the vendor to perform the necessary maintenance and restoration measures. Until the service is fully restored as per the agreed SLA standards, it may be seen as a downtime for the end-user.\n\n*`Causes of Downtime or Cloud Outage`*<br>\n\n - Power Outage\n - Cyber Attacks and Security Breaches\n - Human Errors\n - Software and Technical Issues\n - Networking Issues\n - Maintenance\n\nFor more desriptive attacks we could look at,\n\n - Traffic Overload DDoS Attacks\n - DNS Failure\n - Expired Domain\n\n\n*`Repurcations of Downtime or Cloud Outage`*<br>\n\n - Lost of Sales Revenue\n - Lost employee Productivity\n - Corruption of and gaps in mission-critical data\n - Damages to equipment and associated assets\n - Cost of remediating systems and core business processes\n - Damaged reputation with customers and key stakeholders\n - Degradation of employee morale\n - Regulatory, compliance, and legal penalties (including potential litigation fees)\n - Loss of insurance discounts; Contract penalties\n - Disruption of supply-chain\n\n\nAccording to reports, In 2013, Forbes famously calculated the cost associated with the eTailer's most critical outage that year. Based on Amazon's 2012 net sales, it was determined that  outage cost Amazon $66,240 per minute—or nearly $2 million. A previous outage in June 2008 was close to $31,000 per minute, based on the previous quarter’s global revenue of $4.13 billion. Amazon reported revenues of $107 billion in 2015, which comes out to $203,577 every minute in today's numbers, or a $2,646,501 price tag for the 13 minute episode of downtime.\n\n*`Last five major Cloud Outages`*<br>\n\n - **Microsoft Azure, March 2020**<br>\n On March 3, 2020, Microsoft Azure services in Microsoft’s East U.S. data center region encountered more than six hours of outage. The outage affected a subset of East U.S. customers. The Redmond giant disclosed that a cooling system failure led to the outage, which impacted storage, compute, networking, and other services.\n\n - **IBM Cloud, June 2020**<br>\n IBM Cloud suffered a multi-zone, four-hour interruption of services on June 10, 2020 that affected IBM cloud customers in Washington, D.C., Dallas, London, Frankfurt, and Sydney. The outage impacted general cloud services, Kubernetes services, App connect, and Watson AI cloud services. An investigation revealed that a third party network provider flooded the IBM Cloud network with incorrect routing, which impacted IBM Cloud services and 80+ data centers.\n\n - **Cloudflare, July 2020**<br>\n A 27 minutes Cloudflare outage took down a significant chunk of internet services on July 17, 2020. The outage was due to a configuration error in Cloudflare’s global backbone network, which resulted in a 50% traffic drop across its network. The disruption impacted several big name clients such as Discord, Feedly, GitLab, League of Legends, Patreon, Politico, and Shopify.\n\n - **AWS, November 2020**<br>\n Even though 2020 turned out to be a strong financial year for AWS, the cloud giant suffered a multi-hour, global outage on November 25, 2020 which sparked a wave of memes on Twitter. The interruption affected the U.S. East-1 region that knocked down services of prominent AWS customers, including 1Password, Adobe Spark, Autodesk, Flickr, iRobot, Roku, Twilio, The Washington Post, and Glassdoor. The interruption was triggered due to the small addition of capacity to Amazon Kinesis. Also, it affected other AWS services, such as Lambda, LEX, Macie, Managed Blockchain, Marketplace, MediaLive, MediaConvert, Personalize, Rekognition, SageMaker, and Workspaces.\n\n - **Google Cloud, December 2020**<br>\n On December 14, 2020, Google Cloud experienced a widespread outage that interrupted services, including YouTube, Google Workspace, and Gmail. The 47 minutes outage was due to its automated storage quota management system that reduced the authentication system’s capacity and prevented users from accessing the services.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "High-performance computing in the cloud",
      "canonical": "https://doshidhruv.com/notes/high-performance-computing-in-the-cloud/",
      "datePublished": "2021-12-15",
      "dateModified": "2021-12-15",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "HPC stands for High Performance Computing, the name dictattes itslef. This is the vertical in which cloud computing offers extremenly high resources in the computation side. AWS is run…",
      "contentMarkdown": "**`HPC CLOUD`**<br>\n\nHPC stands for High Performance Computing, the name dictattes itslef. This is the vertical in which cloud computing offers extremenly high resources in the computation side. AWS is run away winner with this use case.\n\nThis is not a new concept altogether. Here the difference between the traditional cloud service and this one is the ammount of resources and the speed which is being offered by the playform. As an Exaple AWS states, \"Run your large, complex simulations and deep learning workloads in the cloud with a complete suite of high performance computing (HPC) products and services on AWS. Gain insights faster, and quickly move from idea to market with virtually unlimited compute capacity, a high-performance file system, and high-throughput networking.\"\n\nSo if we need to define HPC it would be like, \"High performance computing (HPC) is the ability to process data and perform complex calculations at high speeds. To put it into perspective, a laptop or desktop with a 3 GHz processor can perform around 3 billion calculations per second. While that is much faster than any human can achieve, it pales in comparison to HPC solutions that can perform quadrillions of calculations per second. \"\n\n\n*`Use cases of HPC CLOUD`*<br>\n\nDeployed on premises, at the edge, or in the cloud, HPC solutions are used for a variety of purposes across multiple industries. Examples include:\n\n - Research Labs\n - Media and Entertainment\n - Oil and Gas\n - Artificial Intelligence\n - Machine Learning\n - Financial Services\n - Healthcare\n - Manufacturing\n\n\n*`Who is offering HPC Service`*<br>\n\n - <a href=\"https://aws.amazon.com/\">Amazon Web Services (AWS)</a> : The most prolific service provider in the market as it comees up with services like, EC2, ELASTIC FIBER, PARALLEL CLUSTER, AWS BATCH and NICE DCV.\n - <a href=\"https://cloud.google.com/\">Google Cloud</a>\n - <a href=\"https://www.ibm.com/cloud\">IBM Cloud</a>\n - <a href=\"https://azure.microsoft.com/en-gb/\">Microsoft Azure</a>\n\n\nLarge scale istitutions like, Netflix, Amazon, MIT, HSBC, Mchigen University etc are using this service on daily basis to improve and obtain extreme computation edge compared to other peers.\n\n\n\n*`Advantages of HPC CLOUD?`*<br>\n\n - Bursting HPC workload into the cloud\n - Capacity and Capability to run HPC workloads\n - Adapt the hardware resources to the individual HPC job\n - Testing and benchmarking new hardware\n - Save money and time on expensive ISV software licenses\n - Archive result data\n - Parallel File System as a Service\n\n*`Disadvantages of HPC CLOUD?`*<br>\n\n - Cost of HPC in the Cloud higher than on-premises\n - Performance in the Cloud\n - Data Gravity keeps data in the cloud\n - Data Egress Cost for downloading data from the cloud\n - ISV Licenses – terms and network access",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Poly-cloud architecture",
      "canonical": "https://doshidhruv.com/notes/poly-cloud-architecture/",
      "datePublished": "2021-12-10",
      "dateModified": "2021-12-10",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "A poly cloud is a cloud approach that uses different types of cloud services that are hosted onto various clouds. A poly cloud approach runs different kinds of cloud services on one…",
      "contentMarkdown": "**`POLY CLOUD`**<br>\n\nA poly cloud is a cloud approach that uses different types of cloud services that are hosted onto various clouds. A poly cloud approach runs different kinds of cloud services on one cloud and hosting others on another cloud. Although they sound similar, there is a principal difference between multicloud and poly cloud.\n\nThe main difference between multicloud and poly cloud and is in the way a business organizes its cloud services within its general cloud strategy. Multicloud is using multiple clouds at the same time, but not choosing various clouds for a specific type of service. For instance, a business that uses a poly cloud strategy will have all of its data on one cloud service, and all of its virtual machines on another cloud service.\n\nA poly cloud strategy is beneficial because businesses will use certain cloud services depending on specific features that can benefit their operation. Some cloud services might not have what you need, but another cloud service may. A poly cloud strategy is more intentional about the way it chooses a specific service.\n\nPoly cloud makes businesses gives businesses a concrete and thoughtful strategy in the way they choose certain services and providers. Combining multiple clouds is one thing, but purposefully choosing for a specific reason will be more advantageous for its users.\n\n\n*`Advantages of POLY CLOUD `*<br>\n\nHybrid clouds used to be the result of literally connecting a private cloud envrionment to a public cloud environment using massive, complex iterations of middleware. You could build that private cloud on your own, or you could use prepackaged cloud infrastructure like OpenStack®. You would also need a public cloud, like one of the few listed below:\n\n - You are mix and match parts of your solution based on best offering from vendors\n - You get performance and costs benefits\n - You are leveraging the most advanced offerings to provide the best value\n\n\n*`Issues with POLY CLOUD?`*<br>\n\n - You are relying on the cloud-to-cloud vendor connectivity, where there might be increased latency\n - You are relying on different vendors availability; outage on one vendor can cause other parts of the application to stop performing.\n - Increased complexity due to the need to manage and deploy to different vendors\n - You are not able to leverage your spend to achieve a higher level of savings\n - A need to review and revise approach as offerings get updated and might need re-assessment\n\n*`Security of POLY CLOUD?`*<br>\nFor the security aspects we could indulge it inside the multi cloud, basically the approach is same just with the difference with the expected outcome hence hte security concerns with the poly cloud would remain same as the multi cloud\n\nMulti-cloud security has the specific challenge of protecting data in a consistent way across a variety of cloud providers. When a company uses a multi-cloud approach, third-party partners handle different aspects of security. That is why it is important in cloud deployment to clearly define and distribute security responsibilities among the parties.\n\n\nSome organizations take advantage of multi-cloud capabilities to manage very large amounts of storage that is frequently accessed by a broad variety of users. For example, streaming media behemoth Netflix leverages both AWS and Google Cloud to reduce its dependency on a single provider, to take advantage of disaster recovery and business continuity services between providers, and to leverage those capabilities unique to each cloud.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Multi-cloud architecture",
      "canonical": "https://doshidhruv.com/notes/multi-cloud-architecture/",
      "datePublished": "2021-12-01",
      "dateModified": "2021-12-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Multi cloud is the use of two or more cloud computing services from any number of different cloud vendors. A multi cloud environment could be all private, all public or a combination of…",
      "contentMarkdown": "**`MULTI CLOUD`**<br>\n\nMulti-cloud is the use of two or more cloud computing services from any number of different cloud vendors. A multi-cloud environment could be all-private, all-public or a combination of both. Companies use multi-cloud environments to distribute computing resources and minimize the risk of downtime and data loss. They can also increase the computing power and storage available to a business. Innovations in the cloud in recent years have resulted in a move from single-user private clouds to multi-tenant public clouds and hybrid clouds — a heterogeneous environment that leverages different infrastructure environments like the private and public cloud.\n\n`How does MULTI CLOUD look like!`<br>\n\n<center><img src=\"/images/notes/multi-cloud.png\" style=\"height:40%; width:80%;\"></center><br>\n\nAs shown in the image, there would be multiple instances of cloud which would would be supplying for the resources and the computation power need to opercome the requirenemnt. Moreover because of the multiple clouds it works as the subset of distributed cloud as there are multiple cloud instances in distributed cloud but alongwith that they are also distributed in the geographic location also.\n\n`Advantages of MULTI CLOUD?`\nAlthough Multi cloud is advanced version and mixture of hybrid cloud and distributed cloud, there are all the advantages covered of both alongside that there are many more which are listed belop\n\n - Reliablity and Redundancy\n - Reduced Vender Lock In\n - Potential cost saving\n\n\n`Security of MULTI CLOUD?`\nMulti-cloud security has the specific challenge of protecting data in a consistent way across a variety of cloud providers. When a company uses a multi-cloud approach, third-party partners handle different aspects of security. That is why it is important in cloud deployment to clearly define and distribute security responsibilities among the parties.\n\n\nSome organizations take advantage of multi-cloud capabilities to manage very large amounts of storage that is frequently accessed by a broad variety of users. For example, streaming media behemoth Netflix leverages both AWS and Google Cloud to reduce its dependency on a single provider, to take advantage of disaster recovery and business continuity services between providers, and to leverage those capabilities unique to each cloud.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Distributed cloud",
      "canonical": "https://doshidhruv.com/notes/distributed-cloud/",
      "datePublished": "2021-11-01",
      "dateModified": "2021-11-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "DISTRIBUTED CLOUD Distributed cloud enables a geographically distributed, centrally managed distribution of public cloud services optimized for performance, compliance, and edge computing.",
      "contentMarkdown": "**`DISTRIBUTED CLOUD`**<br>\nDistributed cloud enables a geographically distributed, centrally managed distribution of public cloud services optimized for performance, compliance, and edge computing.\n\n`How distributed cloud WORKS?`<br>\nYou may have heard of distributed computing, in which application components are spread across different networked computers, and communicate with one another through messaging or APIs, with the goal of improving overall application performance or maximize computing efficiency.\n\nDistributed cloud goes a giant step further by distributing a public cloud provider's entire compute stack to wherever a customer might need it - on-premises in the customer's own data center or private cloud, or off-premises in one or more public cloud data centers that may or may not belong to the cloud provider.\n\nIn effect, distributed cloud extends the provider's centralized cloud with geographically distributed micro-cloud satellites. The cloud provider retains central control over the operations, updates, governance, security and reliability of all distributed infrastructure. And the customer accesses everything - the centralized cloud services, and the satellites wherever they are located - as a single cloud and manages it all from a single control plane. In this way, as industry analyst Gartner puts it, distributed cloud fixes with hybrid cloud and hybrid multicloud breaks.\n\n`Use Case for DISTRIBUTED CLOUD`<br>\nDistributed cloud and edge computing support everything from simplified multicloud management, to improved scalability and development velocity, to deployment of state-of-the-art automation and decision support applications and functionality.\n\n - Improved hybrid cloud/multicloud visibility and manageability\n - Efficient, cost-effective scalability and agility\n - Easier industry or localized regulatory compliance\n - Faster content delivery\n - IoT, (AI) and machine learning applications\n - Scalablity and Flexiablity\n\n\n`Challenges with DISTRIBUTED CLOUD`<br>\nManaging an enterprise using a multi-site cloud deployment has its challenges including:\n\n - Bandwidth\n - Security\n - Data Protection\n\n\n`FURTHER READING`<br>\n\nThe implementation of distributed cloud could be merged with multiple new age technologies like, blockchian and AI. There is a post on this please <a href=\"https://dhruvdoshi.github.io/blog/2019/10/31/what-if-we-combine-blockchain-and-cloud\"> read this.</a>\n\nThere was an article published in <a href=\"https://www.forbes.com/sites/forbestechcouncil/2021/06/21/distributed-cloud-is-the-way-of-the-future--what-this-means-for-your-business/?sh=3a91846d6818\">FORBES MAGAZINE,</a> this article put emphasis on distributed cloud and the things which could be changed with the implementation of this sort within cloud industry",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Community cloud",
      "canonical": "https://doshidhruv.com/notes/community-cloud/",
      "datePublished": "2021-10-01",
      "dateModified": "2021-10-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Community cloud computing refers to a shared cloud computing service environment that is targeted to a limited set of organizations or employees (such as banks or heads of trading…",
      "contentMarkdown": "**`COMMUNITY CLOUD`**<br>\n\nCommunity cloud computing refers to a shared cloud computing service environment that is targeted to a limited set of organizations or employees (such as banks or heads of trading firms). The organizing principle for the community will vary, but the members of the community generally share similar security, privacy, performance and compliance requirements. Community members may wish to invoke a mechanism that is often run by themselves (not just the provider) to review those seeking entry into the community.\n\nThe implementation of a community cloud is more complicated than other types of clouds. This is because of the number of players involved. Decisions are no longer standalone, and, as a result, a handbook needs to be established at the outset, which must cover:\n\n - Mission statements\n - Ownership of services and resources\n - An economic model of shared cloud and services\n - Resource allocation and maintenance\n - Industry regulations binding each organization\n\nDespite the startup costs and inevitable teething problems, as per industry estimates, the community cloud market is estimated to reach $12.8 billion by 2027 from $2.6 billion in 2020. This is because the benefits of a community cloud currently outweigh the challenges.\n\n`Architecture for COMMUNITY CLOUD`<br>\nIn the below image the architecture of community cloud is explained.\n\n<center><img src=\"/images/notes/community-cloud-overview.png\" style=\"height:40%; width:80%;\"></center><br>\n\n`Advantages for COMMUNITY CLOUD`<br>\n\n - Cost Effectiveness\n - Regulatory Compliance\n - Industry based security compliance\n - High Avaliablity\n - More Control\n\n`Best practices for COMMUNITY CLOUD implementations`<br>\nTo implement the community cloud major things should be taken in the consideration which are further explained in the image and listed below also.\n\n<center><img src=\"/images/notes/community-cloud-model.png\" style=\"height:40%; width:80%;\"></center><br>\n\n - Evaluate and narrow down on the cloud management system\n - Document the terms of shared ownership\n - Determine procurement and cost management\n - Address security requirements and patch management\n - Decide on a data segmentation plan\n - Ensure change management\n - Factor in scalability and migration\n - Plan backup and disaster recovery\n\nBuilding a community cloud involves more focus on business processes and cooperation than technical considerations. After all, a community cloud is just a modified private cloud. A community cloud can be a powerful tool for businesses with the same objectives and requirements. It can even serve as a background for industry-based innovations. When deciding on infrastructure migration to the cloud, private and public clouds are not the only options; community clouds can also fit the bill.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Hybrid cloud",
      "canonical": "https://doshidhruv.com/notes/hybrid-cloud/",
      "datePublished": "2021-09-01",
      "dateModified": "2021-09-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Hybrid cloud refers to a mixed computing, storage, and services environment made up of on premises infrastructure, private cloud services, and a public cloud—such as Amazon Web Services…",
      "contentMarkdown": "**`HYBRID CLOUD`**<br>\n\nHybrid cloud refers to a mixed computing, storage, and services environment made up of on-premises infrastructure, private cloud services, and a public cloud—such as Amazon Web Services (AWS) or Microsoft Azure—with orchestration among the various platforms. Using a combination of public clouds, on-premises computing, and private clouds in your data center means that you have a hybrid cloud infrastructure.\n\nHybrid cloud is an IT architecture that incorporates some degree of workload portability, orchestration, and management across 2 or more environments. Depending on whom you ask, those environments may need to include:\n\n - At least 1 private cloud and at least 1 public cloud\n - 2 or more private clouds\n - 2 or more public clouds\n - A bare-metal or virtual environment connected to at least 1 cloud—public or private\n\nThese varying requirements are an evolution from the earlier age of cloud computing, where the differences between public clouds and private clouds were easily defined by location and ownership. But today’s cloud types are far more complex, because location and ownership are abstract considerations. For example:\n\nPublic clouds traditionally ran off-premises, but public cloud providers are now running cloud services on their clients’ on-premise data centers.\n\nPrivate clouds traditionally ran on-premises, but organizations are now building private clouds on rented, vendor-owned data centers located off-premises.\n\nThis is why it can be more helpful to define hybrid cloud computing by what it does. All hybrid clouds should:\n\n - Connect multiple computers through a network.\n - Consolidate IT resources.\n - Scale out and quickly provision new resources.\n - Be able to move workloads between environments.\n - Incorporate a single, unified management tool.\n - Orchestrate processes with the help of automation.\n\n\n*`How to build HYBRID CLOUD ?`*<br>\n\nHybrid clouds used to be the result of literally connecting a private cloud envrionment to a public cloud environment using massive, complex iterations of middleware. You could build that private cloud on your own, or you could use prepackaged cloud infrastructure like OpenStack®. You would also need a public cloud, like one of the few listed below:\n\n - <a href=\"https://us.alibabacloud.com/en\">Alibaba Cloud</a>\n - <a href=\"https://aws.amazon.com/\">Amazon Web Services (AWS)</a>\n - <a href=\"https://cloud.google.com/\">Google Cloud</a>\n - <a href=\"https://www.ibm.com/cloud\">IBM Cloud</a>\n - <a href=\"https://azure.microsoft.com/en-gb/\">Microsoft Azure</a>\n\nFinally, you would need to link the public cloud to the private cloud. Moving huge amounts of resources among these environments require powerful middleware, or a preconfigured VPN that many cloud service providers give customers as part of their subscription packages:\n\n - <a href=\"https://cloud.google.com/network-connectivity/docs/interconnect/concepts/dedicated-overview\">Google Cloud offers Dedicated Interconnect.</a>\n - <a href=\"https://aws.amazon.com/directconnect/\">Amazon Web Services (AWS) offers Direct Connect.</a>\n - <a href=\"https://azure.microsoft.com/en-us/services/expressroute/\">Microsoft Azure offers ExpressRoute.</a>\n - <a href=\"https://www.openstack.org/passport/\">OpenStack provides the OpenStack Public Cloud Passport.</a>\n\n\n`Advantages of HYBRID CLOUD?`<br>\nAlthough cloud services can drive cost savings, their main value lies in supporting a fast-moving digital business transformation. Every technology management organization runs under two agendas: the IT agenda and the business transformation agenda. Typically, the IT agenda has been focused on saving money. However, digital business transformation agendas are focused on investments to make money.\n\nThe primary benefit of a hybrid cloud is agility. The need to adapt and change direction quickly is a core principle of a digital business. Your enterprise might want (or need) to combine public clouds, private clouds, and on-premises resources to gain the agility it needs for a competitive advantage.\n\nNo matter which definition of hybrid cloud that you use, the benefits are the same: When computing and processing demand increases beyond an on-premises data centre’s capabilities, businesses can use the cloud to instantly scale capacity up or down to handle excess capacity. It also allows them to avoid the time and cost of purchasing, installing and maintaining new servers that they may not always need.\n\n`Security of HYBRID CLOUD?`<br>\nA properly designed, integrated, and managed hybrid cloud can be as secure as traditional on-premise IT infrastructure. While there are some unique hybrid cloud security challenges (like data migration, increased complexity, and a larger attack surface), the presence of multiple environments can be one of the strongest defenses against security risks. All those interconnected environments let enterprises choose where to place sensitive data based on requirements, and it lets security teams standardize redundant cloud storage that can augment disaster recovery efforts.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Private cloud",
      "canonical": "https://doshidhruv.com/notes/private-cloud/",
      "datePublished": "2021-08-01",
      "dateModified": "2021-08-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Private cloud is a cloud computing environment dedicated to a single customer. It combines many of the benefits of cloud computing with the security and control of on premises IT…",
      "contentMarkdown": "**`PRIVATE CLOUD`**<br>\n\nPrivate cloud is a cloud computing environment dedicated to a single customer. It combines many of the benefits of cloud computing with the security and control of on-premises IT infrastructure.\n\n*`How PRIVATE CLOUD works?`*\n\nPrivate cloud is a single-tenant environment, meaning all resources are accessible to one customer only—this is referred to as isolated access. Private clouds are typically hosted on-premises in the customer's data center. But, private clouds can also be hosted on an independent cloud provider’s infrastructure or built on rented infrastructure housed in an offsite data center. Management models also vary—the customer can manage everything itself or outsource partial or full management to a service provider.\n\n*`How PRIVATE CLOUD architecture looks like?`*\n\nSingle-tenant design aside, private cloud is based on the same technologies as other clouds—technologies that enable the customer to provision and configure virtual servers and computing resources on demand in order to quickly and easily (or even automatically) scale in response to spikes in usage and traffic, to implement redundancy for high availability, and to optimize utilization of resources overall.\n\nThese technologies include the following:\n\n1. <b>Virtualization</b>, which enables IT resources to be abstracted from their underlying physical hardware and pooled into unbounded resource pools of computing, storage, memory, and networking capacity that can then portioned among multiple virtual machines (VMs), containers, or other virtualized IT infrastructure elements. By removing the constraints of physical hardware, virtualization enables maximum utilization of hardware, allows hardware to be shared efficiently across multiple users and applications, and makes possible the scalability, agility, and elasticity of the cloud.<br>\n2. <b>Management software</b.> gives administrators centralized control over the infrastructure and applications running on it. This makes it possible to optimize security, availability, and resource utilization in the private cloud environment.<br.>\n3. <b>Automation speeds</b> tasks—such as server provisioning and integrations—that would otherwise need to be performed manually and repeatedly. Automation reduces the need for human intervention, making self-service resource delivery possible.<br>\n\nIn addition, private cloud users can adopt cloud native application architectures and practices—such as DevOps, containers, and microservices—that can bring even greater efficiency and flexibility and enable a smooth transition to a public cloud or hybrid cloud environment in the future.\n\n*`Advantages of PRIVATE CLOUD?`*\n\n1. Full control over hardware and software choices\n2. Freedom to customize hardware and software in any way\n3. Greater visibility into security and access control\n4. Fully enforced compliance with regulatory standards\n5. Efficient resource allocation based on user needs",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Public cloud",
      "canonical": "https://doshidhruv.com/notes/public-cloud/",
      "datePublished": "2021-07-01",
      "dateModified": "2021-07-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "A public cloud is a type of cloud computing in which a third party service provider makes computing resources—which can include anything from ready to use software applications, to…",
      "contentMarkdown": "**`PUBLIC CLOUD`**<br>\n\nA public cloud is a type of cloud computing in which a third-party service provider makes computing resources—which can include anything from ready-to-use software applications, to individual virtual machines (VMs), to complete enterprise-grade infrastructures and development platforms—available to users over the public Internet. These resources might be accessible for free, or access might be sold according to subscription-based or pay-per-usage pricing models.\n\n*`How PUBLIC CLOUD works?`*<br>\n\nA public cloud relies on a virtualized environment to provide an extension of a company’s IT infrastructure, allowing that company to host certain aspects of its infrastructure and services on virtual servers that are offsite and owned by a third party. Public cloud service providers have different strengths, and they offer a wide variety of services and pricing models. Companies that are considering a migration to public cloud need to carefully consider their options when it comes to choosing a provider, especially if they will be locked into a long-term contract. Careful planning can help to keep costs down on monthly cloud services bills, but organizations with unpredictable public cloud usage may find it hard to avoid spending a lot of money on public cloud services when usage suddenly surges.\n\n\nBecause servers in the public cloud share data from multiple companies, security in public cloud is another issue that IT managers will want to weigh. Encrypting data is a good way to ensure stronger security, but if you are using a combination of public and private cloud (also known as a hybrid cloud), not all encryption platforms work across both public and private clouds. There is also an inherent security risk whenever data is moved between a private data center or private cloud and a public cloud.\n\n\nOne last consideration is the location of your public cloud service provider. Data privacy laws in many countries require certain types of data to be stored in-country. These laws change frequently, so it’s a good idea to choose a cloud service provider that is located in your country and can confirm that the servers where your data will be stored are local and in compliance with regional laws. There is also the issue of latency—if your data is being hosted on a different continent, it may take longer than if it were stored close by.\n\n\n*`What makes PUBLIC CLOUD?`*<br>\n\n1. Resource allocation: Tenants outside the provider’s firewall share cloud services and virtual resources that come from the provider’s set of infrastructure, platforms, and software.\n\n2. Use agreements: Resources are distributed on an as-needed basis, but pay-as-you-go models aren’t necessary components. Some customers—like the handful of research institutions using the Massachusetts Open Cloud—use public clouds at no cost.\n\n3. Management: At a minimum, the provider maintains the hardware underneath the cloud, supports the network, and manages the virtualization software.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Serverless computing and Function as a Service",
      "canonical": "https://doshidhruv.com/notes/serverless-computing-and-function-as-a-service/",
      "datePublished": "2021-06-01",
      "dateModified": "2021-06-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Function as a Service (FaaS) is a serverless way to execute modular pieces of code on the edge. FaaS lets developers write and update a piece of code on the fly, which can then be…",
      "contentMarkdown": "**`FAAS - Function as a Service`**<br>\n\nFunction-as-a-Service (FaaS) is a serverless way to execute modular pieces of code on the edge. FaaS lets developers write and update a piece of code on the fly, which can then be executed in response to an event, such as a user clicking on an element in a web application. This makes it easy to scale code and is a cost-efficient way to implement microservices.\n\n*`How FAAS works?`*\n\nFaaS gives developers an abstraction for running web applications in response to events, without managing servers. For example, uploading a file could trigger custom code that transcodes the file into a variety of formats.\n\nFaaS infrastructure is usually metered on-demand by the service provider, primarily through an event-driven execution model, so it’s there when you need it but it doesn’t require any server processes to be running constantly in the background, like platform-as-a-service (PaaS) would.\n\nModern PaaS solutions offer serverless capabilities as part of common workflows that developers can use to deploy applications, blurring the lines between PaaS and FaaS.\n\nIn reality, entire applications will be composed of a mix of these solutions: functions, microservices, and long running services\n\n*What are Microservices ?`*\n\nIf a web application were a work of visual art, using microservice architecture would be like making the art out of a collection of mosaic tiles. The artist can easily add, replace, and repair one tile at a time. Monolithic architecture would be like painting the entire work on a single piece of canvas.\n\n<center><img src=\"/images/notes/serverless-model.png\" style=\"height:40%; width:80%;\"></center><br>\n\nThis approach of building an application out of a set of modular components is known as microservice architecture. Dividing an application into microservices is appealing to developers because it means they can create and modify small pieces of code which can be easily implemented into their codebases. This is in contrast to monolithic architecture, in which all the code is interwoven into one large system. With large monolithic systems, even a minor changes to the application requires a hefty deploy process. FaaS eliminates this deploy complexity.\n\nUsing serverless code like FaaS, web developers can focus on writing application code, while the serverless provider takes care of server allocation and backend services.\n\n*`Popular FAAS providers`<br>*\n\n1. <a href=\"https://cloud.ibm.com/functions/\">IBM Cloud Functions</a>\n2. <a href=\"https://aws.amazon.com/lambda/\">Amazon AWS Lambda</a>\n3. <a href=\"https://cloud.google.com/functions\">Google Cloud Function</a>\n4. <a href=\"https://docs.microsoft.com/en-us/azure/azure-functions/\">Microsoft Azure function</a>\n5. <a href=\"https://www.openfaas.com/\">OpenFaaS (OpenSource)</a>\n\n*`Advantages of FAAS`<br>*\nTaken in consideration there are many points which makes this model best for the cloud computing. Those are listed below.\n\n1. Improved developer velocity\n2. Builtin scalablity\n3. Cost efficiency\n\n\nAs listed, MBAAS is better in all the terms compared to all other traditional models and all comparative models like PAAS and IAAS.\n\n\n*`Disadvantages of FAAS`<br>*\n\n1. Less System Control\n2. More difficult to test the system",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Mobile Backend as a Service (MBaaS)",
      "canonical": "https://doshidhruv.com/notes/mobile-backend-as-a-service-mbaas/",
      "datePublished": "2021-05-01",
      "dateModified": "2021-05-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Mobile backend as a service (MBaaS), also known as \"backend as a service\", is a model for providing web app and mobile app developers with a way to link their applications to backend…",
      "contentMarkdown": "**`MBAAS - Mobile Backend as a Service`**<br>\n\nMobile backend as a service (MBaaS), also known as \"backend as a service\", is a model for providing web app and mobile app developers with a way to link their applications to backend cloud storage and APIs exposed by back end applications while also providing features such as user management, push notifications, and integration with social networking services. These services are provided via the use of custom software development kits (SDKs) and application programming interfaces (APIs). BaaS is a relatively recent development in cloud computing, with most BaaS startups dating from 2011 or later.\n\n<center><img src=\"/images/notes/mobile-backend.png\" style=\"height:40%; width:80%;\"></center><br>\n\nEven though Mobile Backend-as-a-Service mBaaS providers have been around for just a few years, they have become highly influential in the technology industry, especially the mobile app development sector.\n\nModern-day apps provide various services, including push notifications, APIs, analytics, data, security, authentication, and the list goes on. MBaaS acts as a bridge for developers, giving them access to tools they need to develop top-notch apps with innovative features mentioned above.\n\nIf you are considering using MBaaS to create an app for your business or startup, you are on the right track. This article will give you a brief overview of the MBaaS industry and the key players you should consider.\n\nBroadly, MBaaS offers a number of backend development benefits compared to building and managing backend resources manually. Backendless, specifically, offers the most functionality of any MBaaS provider with a user-friendly interface.\n\n\n`Advantages of MBAAS`<br>\nTaken in consideration there are many points which makes this model best for the cloud computing. Those are listed below.\n\n1. Standardized coding environment enables faster and easier coding\n2. Search, data storage and authentication are ready to use features\n3. Security and backup infrastructure is ready to use\n4. Developers can clone apps with ease\n5. Using an MBaaS is recommended for running standalone applications for mobile platforms.\n\nAs listed, MBAAS is better in all the terms compared to all other traditional models and all comparative models  like <a href=\"https://blog.doshidhruv.com/posts/Software-as-a-service/\">SAAS</a> and <a href=\"https://blog.doshidhruv.com/posts/Infrastructure-as-a-service/\">IAAS</a>.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Platform as a Service (PaaS)",
      "canonical": "https://doshidhruv.com/notes/platform-as-a-service-paas/",
      "datePublished": "2021-04-01",
      "dateModified": "2021-04-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Platform as a service (PaaS) or application platform as a service (aPaaS) or platform based service is a category of cloud computing services that allows customers to provision,…",
      "contentMarkdown": "**`PAAS - Platform as a Service`**<br>\n\nPlatform as a service (PaaS) or application platform as a service (aPaaS) or platform-based service is a category of cloud computing services that allows customers to provision, instantiate, run, and manage a modular bundle comprising a computing platform and one or more applications, without the complexity of building and maintaining the infrastructure typically associated with developing and launching the application(s); and to allow developers to create, develop, and package such software bundles.\n\nPlatform as a service (PaaS) is a complete development and deployment environment in the cloud, with resources that enable you to deliver everything from simple cloud-based apps to sophisticated, cloud-enabled enterprise applications. You purchase the resources you need from a cloud service provider on a pay-as-you-go basis and access them over a secure Internet connection.\n\nLike IaaS, PaaS includes infrastructure – servers, storage and networking – but also middleware, development tools, business intelligence (BI) services, database management systems and more. PaaS is designed to support the complete web application life cycle: building, testing, deploying, managing, and updating.\n\nPaaS allows you to avoid the expense and complexity of buying and managing software licences, the underlying application infrastructure and middleware, container orchestrators such as Kubernetes or the development tools and other resources. You manage the applications and services that you develop, and the cloud service provider typically manages everything else.\n\n\n`Advantages of PAAS`**<br>\nTaken in consideration there are many points which makes this model best for the cloud computing. Those are listed below.\n\n1. Cut coding times\n2. Adding development capablities without adding more staff\n3. Develop for multiple platforms – including mobile – more easily\n4. Use sophisticated tools affordably\n5. Support geographically distributed development teams\n6. Efficiently manage the application life cycle\n\nAs listed, PAAS is better in all the terms compared to all other traditional models and all comparative models like <a href=\"https://blog.doshidhruv.com/posts/Software-as-a-service/\">SAAS</a> and <a href=\"https://blog.doshidhruv.com/posts/Infrastructure-as-a-service/\">IAAS</a>.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Software as a Service (SaaS)",
      "canonical": "https://doshidhruv.com/notes/software-as-a-service-saas/",
      "datePublished": "2021-03-01",
      "dateModified": "2021-03-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "SaaS stands for software as a service, which means software is hosted by a third party provider and delivered to customers over the internet as a service. While most SaaS products are…",
      "contentMarkdown": "**`SAAS - Software as a Service`**<br>\n\nSaaS stands for software as a service, which means software is hosted by a third-party provider and delivered to customers over the internet as a service. While most SaaS products are aimed at business users, some products have proved popular with individual consumers, like note-taking app Evernote or personal finance tools like TurboTax and Mint.\n\nIn business settings, users access productivity applications or enterprise software from a service provider instead of from their company’s private data center. Microsoft 365 and Salesforce are common examples of such SaaS software used in business that had been previously hosted and distributed by businesses’ own data centers.\n\nSaaS is a marked difference to the old model of making a one-off purchase of software that must be hosted, implemented, and maintained by the buyers themselves.\n\nThe SaaS delivery model is enabled by a multitenant architecture, where a service provider can distribute multiple versions of the same software from a single physical server. Each user or business has its own version of the application, with the associated customizations, data, and access controls, but from a shared code base that can be patched, updated, and maintained centrally.\n\nAs a result, software can be purchased by individuals or for a select group of users and paid for on a monthly or annual subscription basis per “seat,” instead of making a large upfront investment in a perpetual (permanent) license, starting a lengthy implementation, and committing to years of maintenance, upgrades, and support contracts.\n\n`Advantages of SAAS`<br>\nTaken in consideration there are many points which makes this model best for the cloud computing. Those are listed below.\n\n1. Reduced time to Benifit\n2. Lower Costs\n3. Scalablity and Integration\n4. New Releases / Upgrades\n5. Easy to use and perform proof-of-concepts\n\nAs listed, SAAS is better in all the terms compared to all other traditional models and all comparative models like <a href=\"https://blog.doshidhruv.com/posts/Platform-as-a-service/\">PAAS</a> and <a href=\"https://blog.doshidhruv.com/posts/Infrastructure-as-a-service/\">IAAS</a>.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Infrastructure as a Service (IaaS)",
      "canonical": "https://doshidhruv.com/notes/infrastructure-as-a-service-iaas/",
      "datePublished": "2021-02-01",
      "dateModified": "2021-02-01",
      "topics": [
        "Cloud architecture",
        "Cloud Computing"
      ],
      "description": "Infrastructure as a service (IaaS) are online services that provide high level APIs used to dereference various low level details of underlying network infrastructure like physical…",
      "contentMarkdown": "`IAAS - Infrastructure as a Service`<br>\n\nInfrastructure as a service (IaaS) are online services that provide high-level APIs used to dereference various low-level details of underlying network infrastructure like physical computing resources, location, data partitioning, scaling, security, backup etc. A hypervisor, such as Xen, Oracle VirtualBox, Oracle VM, KVM, VMware ESX/ESXi, or Hyper-V runs the virtual machines as guests. Pools of hypervisors within the cloud operational system can support large numbers of virtual machines and the ability to scale services up and down according to customers' varying requirements.\n\n`Advantages of IAAS`<br>\nTaken in consideration there are many points which makes this model best for the cloud computing. Those are listed below.\n\n1. Pay As You Go\n2. Speed\n3. Avaliablity\n4. Scale\n5. Latency and Performance\n\nAs listed, IAAS is better in all the terms compared to all other traditional models and all comparative models like <a href=\"https://blog.doshidhruv.com/posts/Platform-as-a-service/\">PAAS</a> and <a href=\"https://blog.doshidhruv.com/posts/Platform-as-a-service/\">SAAS.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is cryptocurrency?",
      "canonical": "https://doshidhruv.com/notes/what-is-cryptocurrency/",
      "datePublished": "2021-01-31",
      "dateModified": "2021-01-31",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "A cryptocurrency, crypto currency, or crypto is a collection of binary data which is designed to work as a medium of exchange wherein individual coin ownership records are stored in a…",
      "contentMarkdown": "`CRYPTOCURRENCY`<br>\n\nA cryptocurrency, crypto-currency, or crypto is a collection of binary data which is designed to work as a medium of exchange wherein individual coin ownership records are stored in a ledger which is a computerized database using strong cryptography to secure transaction records, to control the creation of additional coins, and to verify the transfer of coin ownership.\n\nCrypto algorithms are around in the computer domain since 1983 but the induction of hashing and development of blockchain came to the public around 2008 by Satoshi Nakamoto and after the publication of a research paper which was empathizing on proof of work algorithm, a new cryptocurrency came up named Bitcoin.\n\nThere are more than `6000 cryptocurrencies` in existence today in 2021. The major chunk of them was developed in the last decade. Those are all working in different algorithms like proof of stake or proof of work, there are dozen more algorithms available over the internet!\n\n`What is Cryptocurrencies`<br>\nAccording to *Jan Lansky*, a cryptocurrency is a system that meets six conditions\n\n1. The system does not require a `Central Authority`; its state is maintained through distributed consensus.\n2. The system keeps an overview of cryptocurrency units and their ownership.\n3. The system defines whether new cryptocurrency units can be created. If new cryptocurrency units can be created, the system defines the circumstances of their origin and how to determine the ownership of these new units.\n4. Ownership of cryptocurrency units can be `proved exclusively cryptographically.`\n5. The system allows transactions to be performed in which ownership of the cryptographic units is changed. A transaction statement can only be issued by an entity proving the current ownership of these units.\n6. If two different instructions for changing the ownership of the same cryptographic units are simultaneously entered, the system `performs at most one of them.`\n\nIn March 2018, the word cryptocurrency was added to the `Merriam-Webster Dictionary.`\n\n\nThere are numerous things more which we need to understand to know more about the working of cryptocurrencies.\n\n1. <a href=\"https://blog.doshidhruv.com/posts/what-is-blockchain/\">Blockchain</a>\n2. <a href=\"https://blog.doshidhruv.com/posts/What-is-Nodes-in-Blockchain/\">Nodes</a>\n3. <a href=\"https://blog.doshidhruv.com/posts/What-is-TimeStamping-in-Blockchain!/\">Time Stamping</a>\n4. <a href=\"https://blog.doshidhruv.com/posts/What-is-Mining-in-Blockchain/\">Mining</a>\n5. <a href=\"https://blog.doshidhruv.com/posts/What-is-Wallet-in-Blockchain/\">Wallet</a>\n6. <a href=\"https://blog.doshidhruv.com/posts/What-is-Anonimity-in-Blockchain/\">Anonymity</a>\n7. <a href=\"https://blog.doshidhruv.com/posts/What-is-Transaction-Fees-in-Blockchain/\">Transaction Fees</a>\n8. <a href=\"https://blog.doshidhruv.com/posts/What-is-Exchanges-in-Blockchain/\">Exchanges</a>\n9. <a href=\"https://blog.doshidhruv.com/posts/What-is-Automativ-Swaps-in-Blockchain/\">Automatic Swaps</a>\n10. <a href=\"https://blog.doshidhruv.com/posts/What-is-Cryptocurrency-ATM/\">ATM's</a>\n11. <a href=\"https://blog.doshidhruv.com/posts/What-is-Cryptocurrency-ICO/\">Initial Coin Offering </a>\n\nYou could find more information on all of these topics from the <a href=\"https://blog.doshidhruv.com/\">home</a> page.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "How banks could respond to blockchain",
      "canonical": "https://doshidhruv.com/notes/how-banks-could-respond-to-blockchain/",
      "datePublished": "2020-12-31",
      "dateModified": "2020-12-31",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "As the financial service industry is moving from the exploration phase to the application phase for blockchain, banks need to understand the future role of blockchain and its impact on…",
      "contentMarkdown": "`How should banks react?`<br>\n\nAs the financial service industry is moving from the exploration phase to the application phase for blockchain, banks need to understand the future role of blockchain and its impact on banking services if they want to take advantage of this financial revolution.\n\nSuccess will depend on how banks quickly respond to opportunities for innovation. Used effectively, disruption is key to driving organizational change and taking the business to the next level.\n\nDisruption can be a valuable way for banks to reinvent their organization by challenging competitors and identifying growth opportunities. To be able to work effectively in the new eco-system, banks should identify radically new ways to conduct business. They, therefore, have to change their old business models and personal skills and create new, viable models for the blockchain age to be able to combat new competitors.\n\n*“It’s a very interesting space to watch. It’s clear that blockchain has the potential to make finance more efficient, but the big players are well-established. And establishments don’t tend to favour innovation. I’d keep an eye on the start-ups who want to disrupt, but also know how to play nice with the institutions.”* Jeff Koyen\n\n\n`Forward thinking`<br>\n\nWhile the process of disruption triggered by blockchain technology has only just begun, many expect it to speed up soon. This and the coming years we will – as a result - see a lot of changes in the banking industry, while new players will enter the market. Blockchain may eliminate some roles in financial services in the short term.\n\nAlthough the financial industry is thrilled about blockchain, the technology will take a few years to become a mainstream financial model. That means that disruption will happen gradually – but surely. In the meantime, banks have the time to prepare and adjust so that they can have a “second life” in a blockchain world.\n\nThe way blockchain is driving disruption in the traditional financial services industry is however not straightforward but may occur in both obvious and not so obvious ways. The future of the financial services industry will thereby depend on how the various stakeholders including banks capitalize on this technology and how they interact with each other.\n\nBut when financial institutions fully adopt this technology, we should see these gained efficiencies in the form of lower fees for consumers, creating a renewed promising customer experience by banks.\n\nIn that case, these banks may survive ….",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "How blockchain could disrupt banking",
      "canonical": "https://doshidhruv.com/notes/how-blockchain-could-disrupt-banking/",
      "datePublished": "2020-11-30",
      "dateModified": "2020-11-30",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "In today's day and age, banks are the most important key point in the whole finance ecosystem where they serve as the critical storehouses and transfer hubs of value. Now with the…",
      "contentMarkdown": "In today's day and age, banks are the most important key point in the whole finance ecosystem where they serve as the critical storehouses and transfer hubs of value. Now with the induction of Blockchain in this domain, the role of banks in the transaction and the value of fiat currency is going down day by day hence there could be a scenario where blockchain could propose a big threat to large institutions like JP Morgan and Credit Suisse.\n\n`What makes blockchain (so) disruptive for banks?`<br>\nThe answer to this question lies in the three specific in-build properties of a blockchain: Decentralized, distributed and Immutability. These differ completely from those of banks that are centralized organizations.\n\n - `Decentalized Network`<br> Blockchain operates on a decentralized network, that is acting on a peer-to-peer basis. It handles all operations similar to a bank, but without any central authority that monitors all data. So it potentially cuts out the middleman, giving back the power to the owner of the assets (i.e. data or tokens carrying some financial value). All information is stored across its network via blocks. These blocks, which are time-stamped and linked together with all past and current transactions, are permanently recorded and consistently reconciled and updated in a cryptographically secure way. By storing data across its network, blockchain eliminates the risks that come with data being held centrally.\n\n - `Distributed Ledger`<br>A second property of blockchain is the distributed ledger, that allows sharing of a ledger of activity - such as arbitrary data or virtually anything of value between multiple parties. What makes blockchain so important is its ability to automate trust and transparency among all parties using it. Because the ledger is distributed among all transaction participants, it exists simultaneously in multiple places. Each of the computers in the distributed network maintains a copy of the ledger to ensure transparency and also prevent a single point of failure and all copies are updated and validated simultaneously. This makes it extremely difficult to manipulate entries or tamper with the data without the other parties noticing.\n\n - `Immutable Records`<br> A third unique property is its immutability. By design, blockchains are inherently resistant to modification of data. All blockchain networks adhere to a certain protocol for validating new blocks. No changes can be made once the system is set with the initial standards. Once recorded, the data in any given block cannot be altered without the alteration of all the subsequent blocks, which requires the consensus of the network majority.\n\n\n`Where will be blockchain hit on the Banks ??`<br>\nThough blockchain is said to have an impact on virtually every aspect of the financial system, most disruptive use cases can be found in activities such as cross border payments and remittances, share trading, clearing and settlement, trade finance and supply chain finance, regulatory reporting and compliance, as well as smart contracts. It is however clear that when the technology further evolves there will be many more areas for applying the blockchain technologies emerging.\n\n1. `Cross border payments`<br>\n    It is not surprising that payments are emerging as the first and foremost blockchain use case of any banking and/or financial system. Nowadays payments across borders are a time-consuming and expensive process, given the need for correspondent banks or other intermediaries.\n\n    Blockchain offers an easy and secure solution, as it will not require third party authorization. . By cutting out many of the traditional middlemen, the laborious and costly process of cross border payments is simplified, thus significantly speeding up the cross-border payment process and that at a cost much less than with the traditional banking systems.\n\n2. `Share Trading`<br>\n    Using blockchain for buying and selling stocks and shares could also bring several significant benefits. Share trading involves many third parties, such as brokers, CCPs, CSDs and exchanges, making this process time-consuming.\n\n    The decentralized nature of blockchain technology can remove all those intermediaries and enable trading to be run on computers all over the world. Eliminating some of the middlemen from the share trading process speeds up the settlement process and allows for greater trade accuracy. Trading transactions in blockchain thereby reduces the redundancy of information and thus improves performance.\n\n3. `Clearing and settlement`<br>\n    The global cash settlement for fixed income, equity, and derivative products in various currencies is slow, costly, and complicated. Because of the large number of parties involved, It takes several days to settle.\n\n    By eliminating a large number of intermediaries, blockchain enables instantaneous settlement leading to substantially lower costs\n\n4. `Trade Finance`<br>\n    Many trade finance activities still involve lots of paperwork, such as bills of lading, invoices, letters of credit etc. All participants in the trade chain must maintain their database for all transaction-related documents, that must be constantly reconciled against each other. It is as a result a time-consuming activity.\n\n    Blockchain-based trade finance can streamline the entire trading process by getting rid of this time-consuming paperwork and bureaucracy. It eliminates the need for several copies of the same document and can integrate all necessary information in one digital document, which is updated in real-time and can be accessed by all network members.\n\n5. `Digital Identity Verification`<br>\n    Another area where customer experience could be significantly improved when using blockchain is in the digital identity verification process. Online financial transactions require a lot of steps to be taken, including face-to-face checking, authentication, authorization etc. All of these steps need to be taken for each new service provider.\n\n    Blockchain makes it possible to securely re-use identity verification for other services. With blockchain, users can choose how they identify themselves and with whom they agree to share their identity. They still need to register their identity on the blockchain, but they do not need to repeat the registration for each service provider if those providers are also powered by blockchain.\n\n6. `Smart Contracts`<br>\n    Smart contracts are another way to fundamentally change the way business is done nowadays. The functions of a bank such as lending, deposits, treasury, investment advice, business intelligence, regulatory compliance, payments, and remittances will be disrupted by using these contracts.\n\n    Through the use of smart contracts, blockchain technology will change the way information and money are exchanged in finance (and in many other industries). Smart contracts enable operating and automating business processes in a fully decentralized fashion, enabling shared rules of engagement, conduct, and business processes to be automated and enforced ecosystem-wide.\n\n7. `Intensive collaboration`<br>\n    The blockchain-based bank ecosystem will be one of intensive collaboration, within an increasingly open banking environment, not only with other banks but also with various third parties in the financial chain. Whether the third party is a payment processor, a fintech startup or a creative app developer.\n\n    Future success will depend on their willingness to co-operate – even with the apparent challengers to their core activities. Banks don’t have a choice in this – not if they want to survive and prosper. Although it might seem counterintuitive to banks to facilitate new services beyond their immediate control and balance sheets, standing in the way of external creativity and innovation carries the greater risk.\n\n8. `Bank-fintech partnerships`<br>\n    Even as the disruptions continue, the ecosystem will see deeper collaboration between financial institutions and fintech firms and the creation of platform companies to meet the changing needs of their customers. While fintech companies have the advantage of innovation, financial institutions provide a sandbox for proof of concept and scale.\n\n    Most fintech start-ups lack several features for a stand-alone existence, that are well known for traditional banks. Fintech firms need to be wary of product functionality, flexibility, scalability and compliance. Banks have the advantages of greater resources and a larger scale. Good partnerships between banks and fintech could bring the best for both worlds.\n\n\nThese are the domains in which banks could face the competition and could potentially lose ground against Blockchain technology and cryptocurrencies.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Where blockchain falls short",
      "canonical": "https://doshidhruv.com/notes/where-blockchain-falls-short/",
      "datePublished": "2019-11-30",
      "dateModified": "2019-11-30",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "1. There is no customer protection on the blockchain Blockchain technology operates as a push based settlement system. This means the individual holds power over the resource they want…",
      "contentMarkdown": "`1. There is no customer protection on the blockchain` Blockchain technology operates as a push-based settlement system. This means the individual holds power over the resource they want to verify on the blockchain. This could be in cryptos, certificate authentication, land titles, etc. The problem with this is if a transaction goes sour after it has already been verified on the blockchain, the only feasible way of returning the transaction is if the parties agree to reverse it. Using a  centralized system like a bank. However, there is a procedure in place to be able to dispute trades after they are complete. Some trade technologies that settle on a blockchain have used an arbiter system to fix this problem, an example of this is the Open Bazaar P2P  trade network. This way trade occurs between two people and one impartial moderator.\n\n`2. Settlement on a blockchain is slow` A  cost of settling a transaction on the blockchain is that all the nodes in the network need to come to an agreement that the transaction is valid. This is a far slower process than having a bank verify your transaction in an instant. Transactions can be made instantaneously, however until the block in which the transaction is inserted has been verified, it is classified as untrustworthy. In the time between a lodged transaction is made and when the block settles, a bad actor could launch fraudulent transactions to trick the network into what is known as double spend. A  very exciting upcoming technology that could solve this problem is the lightning network. This solution acts as layer 2 of blockchain technology; it can be applied to any public blockchain. It will enable instantly verified transactions for a fraction of the cost of today’s settlement.\n\n`3. Miners can be selfish `The mining process on the blockchain is an innovation that uses game theory economics to incentivize people to commit computer power for securing the network for a profit. The downside of this is generally miners won’t care about settling as many transactions as possible; they will make the most money by finding and verifying a block in the fastest way possible. This leads to a problem of miners finding empty blocks and validating. There is also another problem known as Selfish Mining, which is a situation where a miner or mining pool finds and validates a block and does not publish and distribute a valid solution to the rest of the network.\n\n`4. The growing blockchain size ` With  every new block, a blockchain grows. This can be an issue because each node that is validating the network needs to store the entire history of the blockchain to be a participant. This is a hard enough problem with the bitcoin blockchain where the transaction size is only a  few bytes, the total blockchain size as of January 2017 is 98GB. Given that at the same time in 2016 the size was 50GB, and the use of the blockchain is continuing to increase, this is a growing concern. One of the biggest debates in the bitcoin space is if the block size should be increased. If a blockchain has bigger blocks the blockchain size will increase faster, thus weeding out the solo miners eventually. This is a big issue because the health of a blockchain network is partially dependent on the number of nodes in the network, and the spread of those nodes across the world. The counterargument for this issue is that with sufficient advancement of technology hard disk space will be very cheap in the future and will stay ahead of the blockchain size. The debate is ongoing.\n\n`5. Eventually, a settlement on the blockchain will not be cheap ` On any public blockchain, space in a block is a finite resource.  Necessarily as the network is utilized more the amount of transactions that will want to settle in a block will exceed the storage capacity.  Public blockchain networks have a solution built-in for this which is that transactions with a higher miner fee attached will get precedence to be included in a block. This makes sense because the miners want to maximize their profit so that they will include transactions with the highest fees first. This is not a bug, but a feature. If it were free to settle on the blockchain, there would be far too many ways of attacking the blocks with dust transactions and clogging up the network. Originally the bitcoin blockchain had no block size limit; this was eventually set to  1MB to avoid a Sybil Attack on the network. All of these problems have potential solutions that can be implemented as a  fix. In my view, blockchains will eventually have layers of centralization like the lightning network. However, this is not a bad thing so long as there is a sufficient amount of encryption to protect the privacy of the people who want to use the centralized layers of the network.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What if we combine Blockchain and Cloud Computing?",
      "canonical": "https://doshidhruv.com/notes/what-if-we-combine-blockchain-and-cloud-computing/",
      "datePublished": "2019-10-31",
      "dateModified": "2019-10-31",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Blockchain technology is distributed ledger with records of data containing all details of the transaction carried out and distributed among the nodes present in the network. All the…",
      "contentMarkdown": "Blockchain technology is distributed ledger with records of data containing all details of the transaction carried out and distributed among the nodes present in the network. All the transactions carried out previously are **saved and then they are reflected** in the next upcoming block hash. Hence it is an `intangible and immutable` data structure at the basic end.\n\nCloud computing is well-defined technology that emerged from **large-scale distributed computing technology**. Cloud computing helps to reduce the burden on the resources and lets the user use the optimum resources for redundant work. Organizations like Google or Microsoft have large data servers and then the employees could get the required resources out of it. This helps the organization to have the flexibility of resources according to the projects.\n\nThere are some evident issues in cloud computing which are Data Security, Data Management, Interoperability etc. By having the Blockchain concept with the cloud we could solve all of the issues or **technical bottlenecks** which we are facing at the moment with cloud computing. Blockchain technology is an emerging technology well known for its `security and authenticity`, which are the main characteristics that are making the world turn to its side. By integrating blockchain with cloud computing, there will be many advantages in usability, trust, security, scalability, data management, and many other advantages.\n\nThere are several projects and Ideas which are working progressively in this direction, those are added in the references.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Where could blockchain be used?",
      "canonical": "https://doshidhruv.com/notes/where-could-blockchain-be-used/",
      "datePublished": "2019-09-30",
      "dateModified": "2019-09-30",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Blockchain is freaking exciting and does have the potential to transform all of the business and how it operates, but that doesn’t mean it’s the right solution for every scenario.…",
      "contentMarkdown": "Blockchain is freaking exciting and does have the potential to transform all of the business and how it operates, but that doesn’t mean it’s the right solution for every scenario. Here’s why you should choose blockchain  over, say, a standard outdated Centralized database:\n\n\n`When you want to manage and secure digital relationships or keep a decentralized, shared system of record.` Anytime you want to keep a long-term, transparent record of assets (for example, to record property or land rights ), publically available data, blockchain could be the ideal solution. Smart contracts, in particular, are great for facilitating digital relationships and transactions. With a solidity smart contract, automated payments can be released when parties in a  transaction agree that their conditions have been met.\n\n`Anywhere a middleman or gatekeeper function is expensive or time-consuming.` For  example, major cloud services nowadays allows user to pile up the data on the servers which are organized by a single body or corporate like Google or Amazon, these services charges immensely instead of the single data serves system we could use decentralized systems like DCS-BBN or SIA data systems which are backed by the blockchain and have an ecosystem of the hosts and consumers which provides storage for the rent on lower tariffs\n\n\n`When you want to record secure transactions, especially between multiple partners.` A  traditional database may be good for recording simple transactions between two parties, but when things get more complicated, blockchain can reduce bottlenecks and simplify relationships. For example, shipping conglomerate Maersk is working with IBM to develop a private blockchain platform to connect its various partners and customers across the shipping industry. What’s more, the added security of a decentralized system makes blockchain ideal for transactions in general.\n\n\n`Where the data is in constant flux, but you want to keep a record of past actions.`  Blockchain is a better, safer way to record the activity and keep data fresh while maintaining a record of its history. The data can’t be  corrupted by anyone or accidentally deleted, and you benefit from both a  historical trail of data, plus an instantly up-to-date record",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is an initial coin offering?",
      "canonical": "https://doshidhruv.com/notes/what-is-an-initial-coin-offering/",
      "datePublished": "2019-09-10",
      "dateModified": "2019-09-10",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "An initial coin offering (ICO) or initial currency offering is a type of funding using cryptocurrencies. It is often a form of crowdfunding, although a private ICO which does not seek…",
      "contentMarkdown": "An **`initial coin offering (ICO)`** or initial currency offering is a type of funding using cryptocurrencies. It is often a form of crowdfunding, although a private ICO which does not seek public investment is also possible. In an ICO, a quantity of cryptocurrency is sold in the form of \"tokens\" (\"coins\") to speculators or investors, in exchange for legal tender or other (generally established and more stable) cryptocurrencies such as `Bitcoin or Ether`. The tokens are promoted as future functional units of currency if or when the ICO's funding goal is met and the project successfully launches\n\nKEY TAKEAWAYS</span></b>\n - Initial Coin Offerings (ICOs) are a popular fundraising method used primarily by startups wishing to offer products and services, usually related to the cryptocurrency and blockchain space.\n - ICOs are similar to stocks, but they sometimes have utility for a software service or product offered.\n - Some ICOs have yielded massive returns for investors. Numerous others have turned out to be fraud or have failed or performed poorly.\n - To participate in an ICO, you will usually need to purchase a digital currency first and have a basic understanding of how to use cryptocurrency wallets and exchanges.\n - ICOs are, for the most part, completely unregulated, so investors must exercise a high degree of caution and diligence when researching and investing in ICOs.\n\n**`How an Initial Coin Offering (ICO) Works`**<br>\nWhen a cryptocurrency startup wants to raise money through ICO, it usually creates a whitepaper that outlines what the project is about, the need the project will fulfill upon completion, how much money is needed, how many of the virtual tokens the founders will keep, what type of money will be accepted, and how long the ICO campaign will run for.\n\nDuring the ICO campaign, enthusiasts and supporters of the project buy some of the project’s tokens with fiat or digital currency. These coins are referred to the buyers as tokens and are similar to shares of a company sold to investors during an IPO.\n\nIf the money raised does not meet the minimum funds required by the firm, the money may be returned to the backers; at this point, the ICO would be deemed unsuccessful. If the funding requirements are met within the specified timeframe, the money raised is used to pursue the goals of the project.\n\n\n*`Special Considerations `*<br>\nInvestors looking to buy into ICOs should first familiarize themselves with the cryptocurrency space more broadly. In the case of most ICOs, investors must purchase tokens with pre-existing cryptocurrencies. This means that an ICO investor will need to already have a cryptocurrency wallet set up for a currency like bitcoin or ethereum, as well as having a wallet capable of holding whichever token or currency they want to purchase.\n\n\n`How are ICOs regulated?`<br>\nICOs are largely unregulated. In the United States, there aren't any regulations that apply specifically to ICOs. However, if an ICO fits the classification of a securities offering, then it falls under the SEC's jurisdiction and is regulated by federal securities laws.\n\nSome countries have taken a strict stance and banned ICOs entirely. Countries that have banned ICOs include China, Nepal, Bangladesh, Macedonia, Bolivia, and Ecuador.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is Cryptocurrency ATM?",
      "canonical": "https://doshidhruv.com/notes/what-is-cryptocurrency-atm/",
      "datePublished": "2019-09-09",
      "dateModified": "2019-09-09",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Bitcoin ATMs (Automated Teller Machine) are kiosks that allows a person to purchase Bitcoin and other cryptocurrencies by using cash or debit card. Some Bitcoin ATMs offer bi…",
      "contentMarkdown": "*`Bitcoin ATMs (Automated Teller Machine)`* are kiosks that allows a person to purchase Bitcoin and other cryptocurrencies by using cash or debit card. Some Bitcoin ATMs offer bi-directional functionality enabling both the purchase of Bitcoin as well as the sale of Bitcoin for cash. In some cases, Bitcoin ATM providers require users to have an existing account to transact on the machine.\n\n`There are two main types of Bitcoin machines`<br>\n - unidirectional (one-way)\n - bidirectional (two-way)\n\nOnly about 30% of all crypto ATMs worldwide are bidirectional, and less than 23% in the U.S. Both types are connected to the Internet, allowing for cash purchase and/or sale of Bitcoin. Some machines use a paper receipt and others move money to a public key on the blockchain. Bitcoin cash kiosks look like traditional ATMs, but do not connect to a bank account and instead connect the user directly to a Bitcoin wallet or exchange. While some Bitcoin ATMs are traditional ATMs with revamped software, they do not require a bank account or debit card. On average, transaction fees are 10-20% but can go as high as 25% and as low as 7%.\n\n**`What Is a Bitcoin ATM?`**<br>\nA bitcoin ATM is an Internet-connected kiosk that allows customers to purchase bitcoins and/or other cryptocurrencies with deposited cash.\n\nA bitcoin ATM is not the same as an automated teller machine (ATM) that allows bank customers to physically withdraw, deposit, or transfer funds in one's bank account. Rather, bitcoin ATMs produce blockchain-based transactions that send cryptocurrencies to the user's digital wallet, often via the use of a QR code.\n\n*`KEY TAKEAWAYS`*<br>\nA bitcoin ATM is a standalone device or kiosk that allows members of the public to buy or sell bitcoin or other cryptocurrencies for a terminal.\nBitcoin ATMs are connected to the Internet and often utilize QR codes to send and receive tokens to users' digital wallets.\nThere are currently more than 14,000 bitcoin ATMs in operation around the world",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What are atomic swaps?",
      "canonical": "https://doshidhruv.com/notes/what-are-atomic-swaps/",
      "datePublished": "2019-09-08",
      "dateModified": "2019-09-08",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "An atomic swap is a smart contract technology that enables the exchange of one cryptocurrency for another without using centralized intermediaries, such as exchanges.",
      "contentMarkdown": "An atomic swap is a smart contract technology that enables the exchange of one cryptocurrency for another without using centralized intermediaries, such as exchanges.\n\nAtomic swaps can take place directly between blockchains of different cryptocurrencies, or they can be conducted off-chain, away from the main blockchain. They first came into prominence in September 2017, when an atomic swap between Decred and Litecoin was conducted.\n\nSince then, other startups and decentralized exchanges have allowed users the same facility. For example, Lightning Labs, a startup that uses bitcoin’s lightning network for transactions, has conducted off-chain swaps using the technology.\n\nCryptocurrencies and decentralized exchanges, such as 0x and Altcoin.io, have also incorporated the technology.\n\n\n**`How do atomic swaps work?`**<br>\nAtomic swap protocols are designed in a way that prevents any of the involved parties from cheating. To understand how they work, let’s imagine that Alice wants to trade her Litecoins (LTC) for Bob’s Bitcoins (BTC).\n\nFirst, Alice deposits her LTC into a contract address that acts like a safe. When this safe is created, Alice also generates a key to access it. She then shares a cryptographic hash of this key with Bob. Note that Bob can’t access the LTC yet because he only has the hash of the key and not the key itself.\nNext, Bob uses the hash provided by Alice to create another safe contract address, in which he deposits his BTC. To claim the BTC, Alice is required to use that same key and, by doing so, she reveals it to Bob (thanks to a special function called hash lock). This means that as soon as Alice claims the BTC, Bob can claim the LTC and the swap is complete.\nThe term ‘atomic’ relates to the fact that these transactions either happen entirely or not at all. If any of the parties give up or fail to do what they are supposed to, the contract is cancelled, and the funds are automatically returned to their owners.\n\nAtomic swaps can happen in two different ways: on-chain and off-chain. On-chain atomic swaps happen on either of the currency’s networks (in this case, either the Bitcoin or Litecoin blockchain). Off-chain atomic swaps, on the other hand, take place on a secondary layer. This kind of atomic swap is usually based on bidirectional payment channels, similar to the ones used in the Lightning Network.\nTechnically speaking, most of these trustless trading systems are based on smart contracts that use multi-signatures and Hash Timelock Contracts (HTLC).\n\n*`Hash Timelock Contracts (HTLC)`*<br>\nWhile Hash Timelock Contracts (HTLC) are an important part of the Bitcoin Lightning Network, they are also one of the key components that makes atomic swaps possible. As the name suggests, they are based on two key functions: a hash lock and a timelock.\nA hash lock is what prevents funds from being spent unless a piece of data is revealed (Alice’s key in the previous example). Timelock is a function that ensures the contract can only be executed within a predefined timeframe. Consequently, the use of HTLCs removes the need for trust because they create a specific set of rules that prevent atomic swaps from executing partially.\n\n\n*`Advantages`*<br>\nThe biggest advantages of atomic swaps are all related to their decentralized nature. By removing the need for a centralized exchange or any other kind of mediator, cross-chain swaps can be executed by two (or more) parties without requiring them to trust each other. There is also an increased level of security because users don’t need to give their funds to a centralized exchange or third party. Instead, the trades can happen directly from users’ wallets.\nAlso, this form of peer-to-peer trading has much lower operational costs as trading fees are either very low or absent. Lastly, atomic swaps make it possible for trades to happen very quickly, with higher degrees of interoperability. In other words, altcoins can be swapped directly without making use of Bitcoin or Ethereum as an intermediary coin.\n\n\n*`Limitations`*<br>\nThere are a few conditions that need to be met for an atomic swap to take place, and these may likely present obstacles for the technique to be widely adopted. For instance, to perform an atomic swap, the two cryptocurrencies need to be based on blockchains that share the same hashing algorithm (e.g., SHA-256 for Bitcoin). They also need to be compatible with HTLC and other programmable functionalities.\nOther than that, atomic swaps bring up concerns about user's privacy. That’s because on-chain swaps and transactions can be quickly tracked on a blockchain explorer, making it easy to link the addresses. A short-term answer to this problem is to use privacy-focused cryptocurrencies as a way to reduce exposure. Still, many developers are experimenting with the use of digital signatures in atomic swaps as a more reliable solution.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What are cryptocurrency exchanges?",
      "canonical": "https://doshidhruv.com/notes/what-are-cryptocurrency-exchanges/",
      "datePublished": "2019-09-07",
      "dateModified": "2019-09-07",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "A crypto exchange is a platform on which you can buy and sell cryptocurrency. You can use exchanges to trade one crypto for another — converting Bitcoin to Litecoin, for example — or to…",
      "contentMarkdown": "**`What are Cryptocurrency Exchanges?`**<br>\n\n`A crypto exchange is a platform on which you can buy and sell cryptocurrency.` You can use exchanges to trade one crypto for another — converting Bitcoin to Litecoin, for example — or to buy crypto using regular currency, like the U.S. Dollar. Exchanges reflect current market prices of the cryptocurrencies they offer. You can also convert cryptocurrencies back into the U.S. Dollar or another currency on an exchange, to leave as cash within your account (if you want to trade back into crypto later) or withdraw to your regular bank account.\n\nCryptocurrency exchanges are online platforms in which you can exchange one kind of digital asset for another based on the market value of the given assets. The most popular exchanges are currently Binance and GDAX. It is important not to confuse cryptocurrency exchanges for cryptocurrency wallets or wallet brokerages. Cryptocurrency wallets and wallet brokerages generally allow you to buy and sell a small range of popular digital assets (Bitcoin and Ethereum), which you can then send to a different exchange to trade for other digital assets like altcoins. This statement is not entirely exclusive though; most cryptocurrency exchanges will usually limit their users to only trade digital assets for digital assets, but a few allow trades of fiat currencies such as U.S. Dollars for cryptocurrencies. An example of such an exchange is Kraken, which currently accepts funds in the form of USD, JPY, CAD, and GBP, and supports trades with Monero, Ripple, and Litecoin as well as Bitcoin and Ethereum.\n\n`There’s no one crypto exchange that’s best for every user,` says Tyrone Ross, a financial advisor and CEO of Onramp Invest, a crypto investment platform for financial advisors. Instead, he says it helps to evaluate your own interests when it comes to crypto, and find an exchange that aligns with your goals. For example, maybe you’re looking for a specific coin, or you want to continue learning more as you get into crypto investing.\n\n\n*`Things to look for in Crypto Exchanges.`*<br>\n1. Accessibility\n2. Security\n3. Fees\n4. Liquidity\n5. Coins offered\n6. Educational tools\n7. Storage\n8. Tax information\n\nThese are some of the points which should be took in the consideration while choosing the appropriate crypto wallet to buy Crypto currencies.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What are transaction fees in blockchain?",
      "canonical": "https://doshidhruv.com/notes/what-are-transaction-fees-in-blockchain/",
      "datePublished": "2019-09-06",
      "dateModified": "2019-09-06",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "The blockchain fee is a cryptocurrency transaction fee that is charged to users when performing crypto transactions. The fee is collected in order to process the transaction on the network.",
      "contentMarkdown": "The blockchain fee is a cryptocurrency transaction fee that is `charged to users when performing crypto transactions.` The fee is collected in order to process the transaction on the network.\n\nYou need to pay the blockchain fee to ensure your cryptocurrency transfers arrive promptly. The blockchain fee is `one of the main tools used to speed up crypto transactions,` which are often slow due to high congestion on the blockchain network. The lower the blockchain fee, the lower your transaction's priority in the blockchain network.\n\nTransaction fees for cryptocurrency depend mainly on the `supply of network capacity at the time,` versus the demand from the currency holder for a faster transaction. The currency holder can choose a specific transaction fee, while network entities process transactions from the highest offered fee to the lowest. Cryptocurrency exchanges can simplify the process for currency holders by offering priority alternatives and determining which fee will likely cause the transaction to be processed in the requested time.\n\nFor ether, transaction fees differ by computational complexity, bandwidth use, and storage needs, while bitcoin transaction fees differ by transaction size and whether the transaction uses SegWit. `In September 2018, the median transaction fee for ether corresponded to $0.017, while for bitcoin it corresponded to $0.55.`\n\nSome cryptocurrencies have no transaction fees and rely on client-side proof-of-work as the transaction prioritization and anti-spam mechanism.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is anonymity in blockchain?",
      "canonical": "https://doshidhruv.com/notes/what-is-anonymity-in-blockchain/",
      "datePublished": "2019-09-05",
      "dateModified": "2019-09-05",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Blockchain is public for all but the information saved in the Blockchain is not public for all the users as it comes up with SHA encryption, and the information is only accessed during…",
      "contentMarkdown": "Blockchain is public for all but the information saved in the Blockchain is not public for all the users as it comes up with SHA encryption, and the information is only accessed during verification of the data by the legitimacy miners only.\n\nThinking about anonymity in terms of Blockchain and cryptocurrency, we could look for some points listed below.\n\nThe only thing to take in the consideration is,\n> `BLOCKCHAIN AREN\"T ANONYMOUS BUT THEY ARE PSEDUONYMOUS! `\n\nSimilar to how authors will sometimes write under a pseudonym, you send and receive cryptocurrency using a pseudonym. Instead of a fictitious name, your blockchain pseudonym is your public address. This long string of numbers and letters does not contain any identifiable information that would tie you to the address or its associated wallet.\n\nConsider a public hash of 32 bit like 567sdf89h..v56eg4 this, which is available publically in the blockchain domain, but no one could undermine what this means!\n\n\nBecause the complicated addresses mask your identity, many people think that activity on a blockchain is anonymous. While public addresses protect your privacy to some extent, other blockchain features tend to expose you.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is a blockchain wallet?",
      "canonical": "https://doshidhruv.com/notes/what-is-a-blockchain-wallet/",
      "datePublished": "2019-09-04",
      "dateModified": "2019-09-04",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Crypto wallets keep your private keys – the passwords that give you access to your cryptocurrencies – safe and accessible, allowing you to send and receive cryptocurrencies like Bitcoin…",
      "contentMarkdown": "Crypto wallets keep your private keys – the passwords that give you access to your cryptocurrencies – safe and accessible, allowing you to send and receive cryptocurrencies like Bitcoin and Ethereum. They come in many forms, from hardware wallets like Ledger (which looks like a USB stick) to mobile apps like Coinbase Wallet, making using crypto as easy as shopping with a credit card online.\n\nGoing in-depth, we could signify the definition for the crypto wallets like, `\"Crypto wallets store your private keys, keeping your crypto safe and accessible. They also allow you to send, receive, and spend cryptocurrencies like Bitcoin and Ethereum.\"`\n\n<br>\n\n`Understanding how crypto wallets work`\nBlockchain is a public ledger that stores data in known as \"blocks.\" These are records of all transactions, the balances held at any given address, and who holds the key to those balances. Crypto is not stored \"in\" a wallet, per se. The coins exist on a blockchain, and the wallet software allows you to interact with the balances held on that blockchain. The wallet itself stores addresses and allows its owners to move coins elsewhere while also letting others see the balance held at any given address.\n\n*\"Most Crypto wallets allow users to send, receive, and store crypto. Some have a feature to buy and spend cryptocurrencies,\"* says Utsav Dar, co-founder of Incub8 Finance. *\"Certain crypto wallets have additional features like swapping between tokens, staking tokens for a fixed return paid out to users, and access to DApps (decentralized applications) built on various networks.\"*\n\nWhile each wallet has its specific nuances, here are the general steps involved in sending or receiving funds using a crypto wallet:\n\n - `To receive funds,` you need to retrieve an address (also known as a public key) from your wallet. Locate the \"generate address\" feature in your wallet, click it, copy the alphanumeric address or QR code and share it with the person who wants to send you crypto.\n\n - `To send funds,` you need the address of the receiving wallet. Locate the \"send\" feature in your wallet and enter the wallet's address you intend to send coins to. Select the amount of crypto you would like to send, and click \"confirm.\" Consider sending a small test transaction before sending large amounts of crypto. Note that sending coins requires a fee paid to miners in exchange for processing the transaction.\n\nSending money via QR codes or long strings of numbers and letters may seem strange. However, after doing it a few times, it becomes pretty simple.\n\n\n`How do you use a crypto wallet?`\nCrypto wallets range from simple-to-use apps to more complex security solutions. The main types of wallets you can choose from include:\n\n`Paper wallets:` Keys are written on a physical medium like paper and stored in a safe place. This, of course, makes using your crypto harder because digital money can only be used on the internet.\n\n`Hardware wallets:` Keys are stored in a thumb-drive device that is kept in a safe place and only connected to a computer when you want to use your crypto. The idea is to try to balance security and convenience.\n\n`Online wallets:` Keys are stored in an app or other software – look for one that is protected by two-step encryption. This makes sending, receiving, and using your crypto easier than using an online bank account, payment system, or brokerage.\n\nEach type has its tradeoffs. Paper and hardware wallets are more complicated for malicious users to access because they are stored offline, but they are limited in function and risk being lost or destroyed. Online wallets offered by a major exchange like Coinbase are the simplest way to get started in crypto and offer a balance of security and easy access. (Because your private info is online, your protection against hackers is only as good as your wallet provider's security – so make sure you look for features like two-factor verification.)\n\n*`What could be done with the help of Crypto Wallets`*<br>\n\n1. Manage all your digital assets in one secure place\n2. Control your private keys\n3. Send and receive cryptocurrency to and from anywhere in the world\n4. Interact with usernames rather than long, hexadecimal “public key” addresses\n5. Browse DApps (decentralized finance apps)\n6. Shop at stores that accept cryptocurrency",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "How mining works in blockchain",
      "canonical": "https://doshidhruv.com/notes/how-mining-works-in-blockchain/",
      "datePublished": "2019-09-03",
      "dateModified": "2019-09-03",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "What exactly is Blockchain mining? A peer to peer computer process, Blockchain mining is used to secure and verify Cryptocurrency transactions. Mining involves Blockchain miners who add…",
      "contentMarkdown": "`What exactly is Blockchain mining?`<br>\nA peer-to-peer computer process, `Blockchain mining is used to secure and verify Cryptocurrency transactions.` Mining involves Blockchain miners who add cryptocurrencies transaction data to cryptocurrency's global public ledger of past transactions. In the ledgers, blocks are secured by Blockchain miners and are connected, forming a chain.\n\nAs opposed to traditional financial services systems, Bitcoins have no central clearinghouse when we talk in-depth. Bitcoin transactions are generally verified in `decentralized clearing systems wherein people contribute computing resources to verify the same.` This process of verifying transactions is called mining. It is probably referred to as mining as it is analogous to mining of commodities like gold—mining gold requires a lot of effort and resources, but then there is a limited supply of gold; hence, the amount of gold that is mined every year remains roughly the same. In the same manner, `a lot of computing power is consumed in the process of mining bitcoins.` The number of bitcoins generated from mining dwindles over time. In the words of Satoshi Nakamoto, there is a limited supply of bitcoins—only 21 million bitcoins will ever be created.\n\nAt its core, the term ‘Blockchain mining’ is used to describe the process of adding transaction records to the bitcoin blockchain. This process of adding blocks to the blockchain is how transactions are processed and how money moves around securely on Bitcoins. This Blockchain mining process is performed by a community of people around the world called ‘Blockchain miners.'\n\nAnyone can apply to become a Blockchain miner. These Blockchain miners install and run a `special Blockchain mining software that enables their computers to communicate securely with one another.` Once a computer installs the software, joins the network and begins mining bitcoins, it becomes what is called a *<a href=\"https://blog.doshidhruv.com/posts/what-is-nodes-in-blockchain/\">‘node.’</a>* Together, all these nodes communicate with one another and process transactions to add new blocks to the blockchain which is commonly known as the bitcoin network. This bitcoin network runs throughout the day. It processes equivalent to millions of dollars in bitcoin transactions and has never been hacked or experienced downtime since its launch in 2009.\n\n`Types of Mining`<br>\nThere are mainly three disting types of mining refereed to cryptocurrencies\n<br><br>`1. Individual Mining `<br>\nWhen mining is done by an individual, user registration as a miner is necessary. As soon as a transaction takes place, a mathematical problem is given to all the single users in the blockchain network to solve. The first one to solve it gets rewarded.\n\nOnce the solution is found, all the other miners in the blockchain network will validate the decrypted value and then add it to the blockchain. Thus, verifying the transaction.\n\n`2. Pool Mining`<br>\n In pool mining, users work together to approve the transaction. Sometimes, the complexity of the data encrypted in the blocks makes it difficult for a user to decrypt the encoded data alone. So, a group of miners works as a team to solve it. After validating the result, the reward is then split between all users.\n\n`3. Cloud Mining`<br>\n Cloud mining eliminates the need for computer hardware and software. It is a hassle-free method to extract blocks. With cloud mining, handling all the machinery, order timings, or selling profits is no longer a constant worry.\n\nWhile it is hassle-free, it has its own set of disadvantages. The operational functionality is limited with the limitations on bitcoin hashing. The operational expenses increase as the reward profits are low. Software upgrades are restricted, and so is the verification process.\n\n\n`Uses of Blockchain Mining`<br>\n<br>`1. Validating Transactions`<br>\n Cryptocurrencies function without a central administrator and the insecurity can be substantial with the transactions that transpire. So, what is the authentication method with such cryptocurrencies? With each transaction, new blocks are added to the blockchain in the network and the validation lies in the mining results from the blockchain miners.\n\n`2. Confirming Transactions`<br>\n Miners work the blockchain mining process to confirm whether the transaction is authentic or not. All confirmed transactions are then included in the blockchain.\n\n`3. Securing Networks`<br>\n With more users mining the blockchain, the blockchain network security increases. Network security ensures that there are no fraudulent activities happening with the cryptocurrencies.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "Timestamping in blockchain and cryptocurrencies",
      "canonical": "https://doshidhruv.com/notes/timestamping-in-blockchain-and-cryptocurrencies/",
      "datePublished": "2019-09-02",
      "dateModified": "2019-09-02",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Trusted timestamping is the process of securely keeping track of the creation and modification time of a document. Security here means that no one—not even the owner of the…",
      "contentMarkdown": "Trusted timestamping is the `process of securely keeping track of the creation and modification time of a document.` Security here means that no one—not even the owner of the document—should be able to change it once it has been recorded, provided that the timestamp's integrity is never compromised.\n\nDecentralization is one of the fundamental aspects of technology blockchain, And of course, that implies that anyone from any part of the world can be added to the network and can operate in it. This, in turn, causes there to be no universal time code. This is because we can connect from any time zone. The timestamp is a timestamp, which is calculated according to different parameters.\n\nThe temporal parameter, or timestamp, is based on an ` quick adjustment that uses a median of the timestamps returned by all nodes of the network.` This is due to the decentralized form and seeks to keep the nodes of the network as well synchronized as possible.\n\nWe must also bear in mind that the timestamps of the blocks are not exact. This is because they do not necessarily have to be in order. However, they still offer a relative precision of between one and two hours, which gives a margin of validity. Basically, `all the nodes are connected to the same time slot.` For this, the reference is taken UTC-0 (London local time), where UTC It is in Spanish Coordinated Universal Time. From this, the network nodes coordinate the time in which they work. After storing this data, the local node calculates the displacement time between the UTC strip and the local time.\n\nThis adjusts between the time of the local node with the displacement of all the nodes connected to the network. `This allows the network time to be adjusted constantly`. This avoids manipulation and usually does with little time variations concerning the time slot. This is done because there may be many hourly rates and repetitions, and other problems could occur. Therefore, a universal timestamp creation system was developed for all nodes. This system considers the jet lag that could exist between the nodes.\n\nImplementing a timestamp makes the block it is `impossible to be repeated in the future`, since, in addition to the time, the date of creation of the block is also stored, therefore, there is no possibility that it will be repeated hash that happened a week, two months ago, or a year ago.\n\n<br>\n\n`What is blockchain timestamp used for?`</br>\nOne of the primary uses of a timestamp is to establish the parameters of the mining process. This is because these timestamps allow nodes to `correctly adjust the mining difficulty to be used for each block generation period.` Timestamps help the network determine how long it takes to extract blocks for a certain period and adjust the mining difficulty parameter.\n\nThis, of course, can open the door for miners to manipulate time to lessen the difficulty. Nevertheless, Satoshi Nakamoto foresaw this and programmed the network so that nodes ignore blocks that are outside a ` specific time range based on their own internal clock time.` As a result, if a miner tried to do this, he would lose all his mining work.\n\nOn the other hand, in the whitepaper From Bitcoin, Nakamoto explains that another functionality of the timestamp is to create a mechanism to `avoid double-spending.` In this regard, Nakamoto wrote the following:\n\n>> `*For our purposes, the last transaction is what counts, so we won't mind other subsequent double-spending attempts.*`",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What are nodes in blockchain?",
      "canonical": "https://doshidhruv.com/notes/what-are-nodes-in-blockchain/",
      "datePublished": "2019-09-01",
      "dateModified": "2019-09-01",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "A Node is a part of cryptocurrency needed to make most of the popular tokens like Bitcoin or Dogecoin function. It's a fundamental part of the blockchain network, which is the…",
      "contentMarkdown": "A Node is a part of cryptocurrency needed to make most of the popular tokens like Bitcoin or Dogecoin function. It's a fundamental part of the blockchain network, which is the `decentralised ledger that is used to maintain a cryptocurrency.`\n\nThe explanation can vary depending on the protocol. For example, a resident network may comprise a file server, three laptops and a fax machine. In this case, the network has five nodes, each equipped with a unique MAC address to identify them.\n\n`What is a node in blockchain?`<br>\nThe term *“node”* is being used chiefly about blockchain, a decentralized digital ledger that records all cryptocurrency transactions and `makes the information available to everyone via a connected device.` What this means is that every transaction has to be chronologically recorded and distributed to a series of connected devices. These devices are called nodes. These nodes communicate within the network and transfer information about transactions and new blocks.\n\nIt is a critical component of the blockchain infrastructure. It helps maintain the `security and integrity of the network.` A blockchain node's main purpose is to verify each batch of network transactions, called blocks. Each node is distinguished from others by a unique identifier.\n\n`What do nodes do?`<br>\nWhen a miner attempts to add a new block of transactions to the blockchain, it broadcasts the block to all the nodes on the network. Based on the block’s legitimacy (validity of signature and transactions), `nodes can accept or reject the block.` When a node accepts a new block of transactions, it saves and stores it on top of the blocks it already has stored. In short, here is what nodes do:\n\n1. Nodes check if a block of transactions is valid and accept or reject it.\n2. Nodes save and store blocks of transactions (storing blockchain transaction history).\n3. Nodes broadcast and spread this transaction history to other nodes that may need to synchronize with the blockchain (need to be updated on transaction history).\n\n`What are the types of nodes?`<br>\nThere are basically two types of nodes: `full nodes and lightweight nodes.`\n\n - `Full nodes support and provide security` to the network. These nodes download a blockchain's entire history to observe and enforce its rules.\n\n - `Each user in the network is a lightweight node.` The lightweight node has to connect to a full node to be able to participate.\n\nMany volunteers run full Bitcoin nodes to help the Bitcoin ecosystem. As of now, there are roughly `12,130 public nodes` running on the Bitcoin network. Other than the public nodes, there are many hidden nodes (non-listening nodes). These nodes usually run behind a firewall.\n\n`Miners' nodes`<br>\nThere is also a third type of node: Miner nodes. The term “Bitcoin miners” has now become familiar. These miners are classified as nodes. The miner may work alone (solo miner) or in groups (pool miner). A solo miner uses his full node. Only the administrator can run a full node in a mining pool, which can be referred to as a pool miner's full node.\n\n`The difference between a miner and a node`<br>\nA miner must run a full node to select valid transactions to form a new block. Without a complete node, it cannot determine what proposed transactions are valid according to the current blockchain’s transaction history (if all balances involved in the transactions are sufficient to perform the proposed transactions) because it does not have access to the entire blockchain history. Therefore, a miner is always also a full node. A node, however, is not necessarily simultaneously a miner. A device can run a full node by receiving, storing, and broadcasting all transaction data (much like a server) without creating new blocks of transactions. In this case, it functions more like a passing point with a directory, whereas a miner is the same but simultaneously tries to create new blocks of transactions.\n\n`Listening nodes (supernodes)`<br>\nMoreover, finally, a sub-category called listening nodes. A listening node, essentially, is a publicly visible full node. It communicates with any node that decides to establish a connection with it. A reliable super node typically runs simultaneously, transmitting blockchain history and transaction data to multiple nodes.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    },
    {
      "type": "note",
      "title": "What is Blockchain?",
      "canonical": "https://doshidhruv.com/notes/what-is-blockchain/",
      "datePublished": "2019-08-08",
      "dateModified": "2019-08-08",
      "topics": [
        "Blockchain systems",
        "Blockchain",
        "Cryptocurrency"
      ],
      "description": "Blockchain is the technology that allows the user to create a decentralized system of transaction and data transfers. For the naïve person, it is a method to create a people's bank…",
      "contentMarkdown": "<div align=\"center\">\n  <iframe\n    width=\"640\"\n    height=\"360\"\n    src=\"https://www.youtube.com/embed/iSJJ-haKX3c\"\n    title=\"Blockchain | Preview\"\n    frameborder=\"0\"\n    allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\"\n    allowfullscreen>\n  </iframe>\n</div>\nBlockchain is the technology that allows the user to `create a decentralized system of transaction and data transfers.`For the naïve person, it is a method to create a people's bank without central banks! Blockchain technology is used in the backend for each cryptocurrency. Talking about that, Blockchain is building the blocks of every critical data and then chaining the blocks. So if the last block or any previous block is changed or updated, then every information in the chain needs to be updated. This makes blockchain technology secure and more robust compared to other technologies.\n\nBlockchain is a `shared, immutable ledger that felicitates the process of recording transactions and tracking assets in a business network. `An investment can be tangible (a house or a car) or intangible (Intellectual property, patents, or copyrights). Virtually anything of value can be tracked and traded on a blockchain network, reducing risk and cutting costs for all involved.\n\nFinally, Blockchain is a growing list of records, called blocks linked together with cryptography which is also described as \"trustless and fully decentralized peer to peer immutable data storage which is spread over the network of participants often referred to as the nodes. Every block contains a `cryptographic hash of the previous block and the timestamp`, and the transaction data.\n\nBut first, let's get back to basics. What is Blockchain, and how does it work?\n\nA blockchain is a file for storing data. Or, it's an open, distributed database system to put it in more technical terms. The data is distributed (i.e., duplicated), and the whole Blockchain is decentralized. This means no one person or entity (say, a government or corporation like Google or Microsoft) has control over the Blockchain; this is a radical departure from the centralized (Government databases) databases controlled and administered by businesses and other entities.\n\nSo how does it work? In straightforward terms, the file consists of blocks of data, with each block being connected to the previous block with the hashing code, forming a chain. Hence, the name `blockchain`. As well as the data itself, each block also contains a record of when that block was created or edited. It contains the hash code of the previous block, which makes it very useful for maintaining a complex system of records that cannot be corrupted or lost, and this is the primary use case to preserve the integrity of the data.\n\nBecause the whole Blockchain is duplicated across many computers, any user can view the entire Blockchain in the case of public blockchains. Transactions or records are processed not by one central administrator but by a network of users who work to verify the data and achieve a consensus; these verifiers are known as miners, each miner is paid some cryptocurrency for successful mining. If this sounds familiar, Bitcoin operates in the same way. `Bitcoinis` the first example of Blockchain in action.",
      "author": "Dhruv Doshi",
      "language": "en-CA"
    }
  ]
}