I set up Grafana, Prometheus, Loki, Tempo, and Alloy in my homelab, wired everything together, made dashboards, and felt pretty good about the whole thing. Dashboards rendered. Metrics scrolled. Graphs went up and to the right. Then I actually audited it. The first pass found five real bugs. One path had been failing to produce useful Loki streams. One dashboard referenced metrics that did not exist in my environment. Another panel displayed a number that was mathematically meaningless. Later, when I added Kubernetes telemetry, a warning panel made a healthy cluster look scarier than it was because the query matched raw message text instead of parsed severity labels. The lesson I keep coming back to is simple: observability infrastructure needs to be observed. If you do not measure the metrics and log pipeline itself, you do not know whether you have monitoring. You have the illusion of monitoring. This is the honest version of "I built a monitoring stack."

The Stack

The core stack runs on a dedicated Docker host:

• Grafana for dashboards and visualization
• Prometheus for metrics storage and PromQL queries
• Loki for log aggregation and LogQL queries
• Tempo for distributed traces
• Alloy as the collector that receives, processes, and forwards signals

The data sources are mixed on purpose. Host metrics come from node_exporter. Container metrics come from cAdvisor. Application metrics come from Prometheus scrape targets and OpenTelemetry where the service supports it. Logs flow through Alloy into Loki. Traces flow into Tempo. Storage sits on a 4TB SSD. Prometheus and Loki are configured for up to 180 days of history, with Prometheus also capped by size. Tempo uses its own shorter trace retention and search window rather than inheriting the 180-day policy. A few days of data is enough for alerting; months of metrics and logs are what let me notice slow changes such as disk I/O drift and usage shifts over a quarter. The newer Kubernetes layer feeds into the same stack instead of running a second in-cluster Grafana deployment. An in-cluster Alloy instance scrapes Kubernetes API server metrics, kubelet, cAdvisor, kube-state-metrics, node-exporter, Flux, Envoy, cert-manager, MetalLB, Longhorn, CoreDNS, and tunnel metrics. Metrics are remote-written to the central Prometheus with a cluster="homelab" label. Pod logs and Kubernetes Events push to the central Loki. In-cluster OTLP endpoints are exposed for future app instrumentation. That shape matters. Grafana stays the place I look first, but Kubernetes gets real telemetry instead of being a blind spot. The setup was the easy part. The Grafana docs are excellent, the Alloy reference is detailed, and there are dozens of community dashboards you can import. In under a weekend, I had something that looked like a monitoring stack. Looking like one and being one are different things.

Bug 1: The Loki Pipeline That Looked Healthy

This is the one that bothered me most. I had OpenTelemetry logs flowing into Alloy from a handful of services. Alloy had a Loki exporter configured. Loki was running. Grafana could query Loki. Every component reported healthy. The pipeline looked fine. The actual useful log streams I expected to see in Loki: nothing. I noticed because I opened Loki Explore expecting to see service logs and got an empty result. No obvious error. No broken dashboard. Just no logs where logs should have been. The first thing I checked was the Alloy metrics endpoint. The otelcol.exporter.loki component itself does not expose component-specific debug information, but the downstream loki.write component does. So I checked the write path directly: That is the kind of check I should have had on a dashboard from day one. A log pipeline can be configured, healthy, and still not be producing the streams you think it is producing. The root cause was my understanding of how OTLP attributes become Loki labels. Grafana's docs for otelcol.exporter.loki are specific: OTLP log attributes are not converted into Loki labels by default. If you want resource attributes like service.name or host.name to become labels, the log needs hint attributes such as loki.resource.labels. I had treated "the exporter is configured" as equivalent to "the logs are labeled and queryable." That was wrong. The fix was to insert an attributes processor that adds the Loki hint before the logs hit the exporter: After that, the downstream write counters moved and the logs became queryable by the labels I actually cared about. The important thing here is not the exact processor block. It is the verification habit. A collector being healthy means the collector accepted its config. It does not prove the output is useful. For logs, I now check the collector output counters and then query Loki directly before trusting any dashboard built on top of it.

Bug 2: Dashboards That Reference Metrics That Do Not Exist

Community dashboards are great until they are not. I imported an infrastructure dashboard that had been well-rated and came from a reputable publisher. It rendered beautifully on first load, which is what these things always do when there is no data. Most panels filled in as Prometheus scraped the hosts. A few panels stayed empty. When a dashboard panel shows "No data" forever, there are usually three possibilities: the query syntax is wrong, the metric label filters do not match your environment, or the metric itself does not exist. I checked all three. Three panels referenced metrics that did not exist in my Prometheus instance. Not renamed. Not missing a label. Just absent. I verified by querying Prometheus directly: Zero matches. The fix was unglamorous: I replaced those panels with equivalent queries using metrics that did exist, like node_cpu_seconds_total and node_memory_MemAvailable_bytes. The takeaway is broader than one dashboard. Dashboards are executable code. They can be wrong in ways that look exactly like "no data yet." If you import a dashboard, validate every panel against your own Prometheus labels before you trust it. Empty panels train you to ignore the dashboard, and once that happens, you have lost the value of the dashboard entirely.

Bug 3: The Phantom Cost Panel

One of my dashboards had a "Cost (USD)" panel for my AI automation. It showed "Waiting for data..." for weeks. I kept telling myself I would look at it later. Eventually I did, and the problem was simple: that service did not emit a cost metric over OpenTelemetry. It never had. The panel was querying a metric that Prometheus had never seen. This one stung because the fix was easy and I had ignored it. A "Waiting for data" panel is a signal, not a state. Either the metric is coming and you know when to expect it, or the metric is not coming and the panel is wrong. The right move was to delete the cost panel and replace it with panels built from metrics that actually existed. Tokens by model was useful. Cache hit rate was useful. Both are good proxies for cost trends without pretending I had a dollar metric in the telemetry path: The lesson is the same one I learned from the imported dashboard, except this time I had done it to myself. Before you commit a panel, verify the metric exists in your own Prometheus instance. A metric that will never arrive looks identical to a metric that has not arrived yet.

Bug 4: Counter Used Where a Gauge Was Needed

This one was embarrassing in retrospect. I had a panel showing something like "total context size" for an AI service. The intent was a current snapshot of active conversation context size. The panel showed a number in the tens of billions. That could not possibly be right. The metric was defined as a Prometheus counter. Counters only go up. Every time a context grew, the code incremented the counter. Every time a context was deleted or trimmed, the counter did not decrement, because counters do not work that way. Over weeks, the value became a cumulative sum of every context that had ever existed, not the current size. That is not "current context size." It is a meaningless artifact. The fix was to change the metric type to a gauge and set the current value on each update: The Prometheus metric type docs are clear, and I have used Prometheus long enough to know this. I still got it wrong because the metric name was ambiguous. A name like context_bytes_current pushes you toward a gauge. A name like context_bytes_total pushes you toward a counter. If you are naming a metric, decide whether the consumer needs the current value, a cumulative value, or a rate over time. The name should make that obvious. If the dashboard panel has to explain what the metric "really" means, the metric name is probably wrong.

Bug 5: Datasource UIDs vs Placeholder Names

This one only matters when you start linking across data sources, but it matters a lot when you do. Grafana datasources each have a uid, which is the stable identifier used in dashboard JSON and cross-datasource links. Clicking a log line in Loki to jump to a trace in Tempo depends on those UIDs lining up. When you provision datasources with YAML, the uid is whatever you set it to. When you add datasources through the UI, Grafana generates one for you. Exported dashboards preserve those UIDs. That means a dashboard exported from another Grafana instance might reference a Loki datasource with uid: "loki", while your local Loki datasource is something like b2f-3c4d-abcd-1234. The dashboard imports. Most panels render. Cross-datasource links fail quietly or throw an error that references a UID you have never seen. I had spent time setting up Tempo and configuring derived fields in Loki so a trace ID in a log line would become a clickable link to the trace. It looked wired up in the UI, but nothing was clickable in actual logs. The immediate fix was to inspect the real datasource UIDs and update the dashboard JSON: The durable fix is better: provision datasources with deterministic UIDs and version-control the dashboards that reference them. If the UID is config, it cannot drift silently between environments.

Bug 6: Warning Panels That Matched Words Instead of Severity

The Kubernetes rollout added a different kind of observability bug. After I started shipping pod logs and Kubernetes Events into Loki, I built a dashboard panel for warning and error logs. The first version used a raw text match: That seemed reasonable until the panel started making a healthy cluster look noisy. The problem was that raw log text is not severity. A message can contain error="<nil>" and be reporting success. A metric or resource name can contain errors without being an error. Kubernetes klog warnings can include Err: in the message body even when the actual severity prefix is W, which is warning, not error. The fix was to parse severity into a label in Alloy, then query that label in Grafana. For Kubernetes klog lines, the prefix is authoritative: Then the dashboard query becomes label-based: That changed the workflow immediately. The panel stopped being a text search for scary words and became a view of parsed severity. When something appeared there, I still checked cluster truth first: pods, Flux state, Longhorn health, and API readiness. But the panel was no longer crying wolf because a healthy log line happened to contain the word "error." This is a subtle failure mode. Bad observability can create false incidents just as easily as it can hide real ones.

What I Actually Measure Now

A few design choices have held up. Pipeline health before application health. I watch scrape target health, remote-write success, Loki write throughput, and drop counters. If the telemetry path is broken, every application dashboard downstream is suspect. node_exporter everywhere it makes sense. Application metrics tell you what your code is doing. Node metrics tell you what the host is doing to your code. Disk pressure, memory pressure, CPU throttling, and network saturation often show up at the host layer before the application explains them clearly. Kubernetes object state through kube-state-metrics. Container CPU and memory are useful, but they do not tell you whether a Deployment has unavailable replicas, a PVC is stuck, or a node is NotReady. kube-state-metrics turns Kubernetes objects into Prometheus data, which is exactly what dashboards and alerts need. Old log replay protection. When Alloy restarts in Kubernetes, it can discover buffered pod logs that are old enough for Loki to reject as stale samples. I drop pod log lines older than 55 minutes at collection time. That creates an expected, visible drop counter instead of a confusing stream of stale-sample errors after every collector restart. Resource sizing for the collector itself. The first Alloy Kubernetes deployment was healthy but under-requested for memory. It was running around 600-750 MiB with peaks above 850 MiB against a 256 MiB request. I moved the request to 768 MiB and the limit to 1536 MiB. The collector is infrastructure. It deserves the same sizing discipline as the apps it watches. I see teams skip this layer because managed observability platforms make the ingestion path feel invisible. That is fine until the bill spikes, a collector starts dropping data, or a dashboard quietly stops matching reality. Running the stack myself makes the ingestion path impossible to ignore.

Tech Stack

• Grafana for dashboards and visualization
• Prometheus for metrics storage and PromQL
• Loki for log aggregation and LogQL
• Tempo for distributed traces
• Alloy for metric, log, trace, and Kubernetes telemetry pipelines
• node_exporter for host metrics
• kube-state-metrics for Kubernetes object state
• Docker Compose for the central LGTM stack
• Kubernetes for in-cluster telemetry collection and service discovery

If you are standing up your first monitoring stack, build the dashboards last. Build the health metrics for your pipeline first, then the application metrics, then the dashboards. That order feels backwards because dashboards are the fun part. It is still the right order. The next time the collector is healthy but the pipeline is not producing useful data, I want to know in five minutes, not five weeks.