Jul 23 01:26:43.439478 ip-10-0-136-186 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 01:26:43.439489 ip-10-0-136-186 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 01:26:43.439495 ip-10-0-136-186 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 01:26:43.439702 ip-10-0-136-186 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 01:26:53.674959 ip-10-0-136-186 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 01:26:53.674976 ip-10-0-136-186 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 712d7ebf077e465aaaa0a0c795d61073 -- Jul 23 01:28:49.041573 ip-10-0-136-186 systemd[1]: Starting Kubernetes Kubelet... Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.429942 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434387 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434400 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434403 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434407 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434421 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434424 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434427 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:49.713299 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434430 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:49.509030 ip-10-0-136-186 systemd[1]: Started Kubernetes Kubelet. Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434432 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434435 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434438 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434440 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434443 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434445 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434448 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434450 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434453 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434455 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434458 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434461 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434463 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434466 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434469 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434472 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434474 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434477 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:49.715117 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434481 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434485 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434488 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434491 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434494 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434496 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434499 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434502 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434505 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434508 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434511 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434514 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434517 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434520 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434522 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434525 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434528 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434530 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434534 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434537 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:49.717217 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434540 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434542 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434545 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434547 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434551 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434553 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434556 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434558 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434561 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434563 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434566 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434568 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434571 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434574 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434576 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434579 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434582 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434586 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434590 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:49.718706 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434593 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434596 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434598 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434601 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434603 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434606 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434610 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434612 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434615 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434617 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434619 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434622 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434625 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434628 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434631 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434633 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434636 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434638 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434641 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434643 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:49.719811 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434647 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434651 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434655 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434658 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434660 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434663 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434666 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434669 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434672 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434674 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434677 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434679 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434682 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.434685 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435405 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435412 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435415 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435418 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435421 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435425 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:49.720833 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435428 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435430 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435433 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435436 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435438 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435441 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435443 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435446 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435449 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435452 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435455 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435457 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435461 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435465 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435467 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435470 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435473 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435476 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435478 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435482 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:49.721950 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435485 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435488 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435490 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435493 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435496 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435499 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435501 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435504 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435506 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435509 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435512 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435516 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435519 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435521 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435524 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435526 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435528 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435531 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435533 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435536 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:49.723023 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435538 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435541 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435543 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435546 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435550 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435554 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435557 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435559 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435562 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435565 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435568 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435571 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435574 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435578 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435581 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435584 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435586 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435589 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435591 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:49.724133 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435594 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435598 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435600 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435603 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435605 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435608 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435611 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435613 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435616 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435619 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435621 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435627 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435629 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435632 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435634 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435637 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435639 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435642 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435644 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435647 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:49.725004 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435649 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435652 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435654 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435657 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435662 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435665 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435667 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435670 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435672 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435675 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435677 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435680 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435682 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.435685 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437308 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437319 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437323 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437326 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437329 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437332 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:49.726088 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437335 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437338 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437343 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437345 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437348 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437351 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437353 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437356 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437359 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437361 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437364 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437367 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437370 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437372 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437375 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437377 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437380 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437382 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437385 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437387 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:49.726968 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437390 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437393 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437396 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437398 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437401 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437404 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437407 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437410 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437413 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437416 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437420 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437424 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437427 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437430 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437433 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437436 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437439 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437442 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437444 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437447 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:49.728046 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437450 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437452 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437455 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437457 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437461 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437463 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437466 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437468 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437471 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437473 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437476 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437478 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437481 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437483 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437486 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437488 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437491 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437494 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437499 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:49.728933 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437501 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437507 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437511 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437514 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437517 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437520 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437522 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437525 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437528 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437531 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437533 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437537 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437539 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437542 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437545 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437547 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437550 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437553 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437555 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:49.729804 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437558 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437561 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437563 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437568 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437571 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437574 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437576 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437579 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437582 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437584 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437587 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437589 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437593 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437596 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.437600 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437675 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437683 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437692 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437697 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437702 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437705 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 01:28:49.730677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437709 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437714 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437717 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437720 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437723 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437727 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437730 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437733 2554 flags.go:64] FLAG: --cgroup-root="" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437736 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437738 2554 flags.go:64] FLAG: --client-ca-file="" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437741 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437745 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437750 2554 flags.go:64] FLAG: --cluster-domain="" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437753 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437756 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437759 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437763 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437767 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437770 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437773 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437776 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437779 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437782 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437787 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 01:28:49.731566 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437790 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437794 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437799 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437802 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437805 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437807 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437810 2554 flags.go:64] FLAG: --enable-server="true" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437813 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437818 2554 flags.go:64] FLAG: --event-burst="100" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437821 2554 flags.go:64] FLAG: --event-qps="50" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437823 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437826 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437829 2554 flags.go:64] FLAG: --eviction-hard="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437833 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437835 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437838 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437841 2554 flags.go:64] FLAG: --eviction-soft="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437844 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437847 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437850 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437853 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437856 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437859 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437862 2554 flags.go:64] FLAG: --feature-gates="" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437866 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 01:28:49.732642 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437869 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437872 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437875 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437886 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437890 2554 flags.go:64] FLAG: --help="false" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437893 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-136-186.ec2.internal" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437897 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437900 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437905 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437908 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437912 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437915 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437918 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437921 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437924 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437927 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437930 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437932 2554 flags.go:64] FLAG: --kube-reserved="" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437936 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437938 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437941 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437944 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437947 2554 flags.go:64] FLAG: --lock-file="" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437949 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 01:28:49.733958 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437953 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437955 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437961 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437964 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437968 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437971 2554 flags.go:64] FLAG: --logging-format="text" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437973 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437977 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437979 2554 flags.go:64] FLAG: --manifest-url="" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437982 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437986 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437989 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437993 2554 flags.go:64] FLAG: --max-pods="110" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.437996 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438002 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438005 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438009 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438012 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438015 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438018 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438025 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438029 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438032 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 01:28:49.737115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438035 2554 flags.go:64] FLAG: --pod-cidr="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438038 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438044 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438046 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438049 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438052 2554 flags.go:64] FLAG: --port="10250" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438055 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438058 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5eff8d886bd3287" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438061 2554 flags.go:64] FLAG: --qos-reserved="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438064 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438067 2554 flags.go:64] FLAG: --register-node="true" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438070 2554 flags.go:64] FLAG: --register-with-taints="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438073 2554 flags.go:64] FLAG: --registry-burst="10" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438076 2554 flags.go:64] FLAG: --registry-qps="5" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438079 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438083 2554 flags.go:64] FLAG: --reserved-memory="" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438087 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438090 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438093 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438096 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438099 2554 flags.go:64] FLAG: --runonce="false" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438102 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438104 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438107 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438111 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 01:28:49.738214 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438116 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438119 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438122 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438125 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438128 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438131 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438133 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438136 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438139 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438142 2554 flags.go:64] FLAG: --system-cgroups="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438145 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438151 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438154 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438157 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438161 2554 flags.go:64] FLAG: --tls-min-version="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438164 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438167 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438169 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438172 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438175 2554 flags.go:64] FLAG: --v="2" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438180 2554 flags.go:64] FLAG: --version="false" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438184 2554 flags.go:64] FLAG: --vmodule="" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438188 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.438192 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438318 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:51.033610 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438322 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438325 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438328 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438331 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438333 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438336 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438341 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438344 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438347 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438350 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438352 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438355 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438358 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438360 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438363 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438368 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438372 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438375 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438378 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:51.035540 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438380 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438383 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438386 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438388 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438391 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438394 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438397 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438399 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438402 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438405 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438407 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438411 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438414 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438416 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438419 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438421 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438424 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438426 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438429 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438433 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:51.036455 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438437 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438439 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438442 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438445 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438447 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438450 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438452 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438455 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438457 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438460 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438462 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438465 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438467 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438470 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438472 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438475 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438477 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438480 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438483 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438485 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:51.037467 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438488 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438490 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438493 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438495 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438499 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438502 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438504 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438507 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438509 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438512 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438514 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438518 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438522 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438524 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438527 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438530 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438532 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438535 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438537 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438540 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:51.038348 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438542 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438545 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438549 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438552 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438555 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438558 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438561 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438564 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438566 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438569 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438571 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438574 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438576 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438579 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438581 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438584 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438586 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438589 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:51.039431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.438593 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.439204 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.445896 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.445910 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.445944 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.445954 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446015 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446020 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446023 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446026 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446029 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446032 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446035 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446038 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446041 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:51.040487 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446043 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446046 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446049 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446051 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446054 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446056 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446059 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446061 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446064 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446066 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446069 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446072 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446074 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446077 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446080 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446082 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446085 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446088 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446090 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:51.041353 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446096 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446101 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446104 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446107 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446110 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446113 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446115 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446118 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446121 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446123 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446126 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446128 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446131 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446133 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446137 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446140 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446143 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446146 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446148 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:51.042422 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446151 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446154 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446157 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446160 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446163 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446165 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446167 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446170 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446174 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446177 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446180 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446183 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446187 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446191 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446194 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446197 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446200 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446202 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446205 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446207 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:51.043313 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446210 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446212 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446215 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446217 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446220 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446223 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446225 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446229 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446231 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446235 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446238 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446259 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446263 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446268 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446270 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446273 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446276 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446278 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446281 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446284 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:51.044187 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446286 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446289 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446291 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446294 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446296 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446300 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446303 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446305 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446309 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446312 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446314 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446317 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.446322 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446456 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446461 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:51.045037 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446464 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446467 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446470 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446473 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446477 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446480 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446483 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446485 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446488 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446490 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446493 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446496 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446498 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446501 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446503 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446506 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446508 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446512 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446516 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:51.045687 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446519 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446521 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446524 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446527 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446531 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446533 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446536 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446539 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446542 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446545 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446548 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446550 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446553 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446555 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446558 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446560 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446563 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446566 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446569 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446572 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:51.046550 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446574 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446577 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446579 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446582 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446584 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446587 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446590 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446592 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446594 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446597 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446599 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446602 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446604 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446608 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446611 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446615 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446619 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446622 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446625 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446628 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:51.047431 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446631 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446634 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446637 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446639 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446642 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446644 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446647 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446649 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446652 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446656 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446658 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446661 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446663 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446666 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446668 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446671 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446673 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446676 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446679 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:51.048323 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446681 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446684 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446686 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446689 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446692 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446694 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446696 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446699 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446701 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446705 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446709 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446712 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446715 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446718 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446722 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446725 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446728 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446730 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:51.049220 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:49.446733 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.446738 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.446840 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.450866 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.452183 2554 server.go:1019] "Starting client certificate rotation" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.452290 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.452321 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.454562 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.458902 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.470113 2554 log.go:25] "Validated CRI v1 runtime API" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.474782 2554 log.go:25] "Validated CRI v1 image API" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.476589 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 01:28:51.050109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.481534 2554 fs.go:135] Filesystem UUIDs: map[168c804f-9dda-4777-a2bc-f730bbb09836:/dev/nvme0n1p4 271822c2-9edc-4e1d-8717-294de5569d06:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 23 01:28:51.050720 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.481550 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 01:28:51.050720 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.483751 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.485824 2554 manager.go:216] Machine: {Timestamp:2026-07-23 01:28:49.485348292 +0000 UTC m=+0.321803310 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102325 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7f5d5eac187a4758333f7d8d94e2 SystemUUID:ec2b7f5d-5eac-187a-4758-333f7d8d94e2 BootID:712d7ebf-077e-465a-aaa0-a0c795d61073 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:d2:c8:dc:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:d2:c8:dc:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:ac:68:5f:3e:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.485932 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.486005 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489007 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489050 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-186.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489714 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489725 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489732 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.489742 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.491355 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.496835 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.496962 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.496989 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497001 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497023 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497036 2554 kubelet.go:398] "Adding apiserver pod source" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497045 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 01:28:51.053381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497413 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.497431 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.500118 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.502292 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.502320 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503648 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503662 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503668 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503674 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503682 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503688 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503696 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503701 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503708 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503714 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503724 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.503730 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.504982 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.504990 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.506357 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8hq8" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.506546 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-186.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.507020 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.507036 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.508131 2554 server.go:1268] "Started kubelet" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.508197 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.508228 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.508308 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.509462 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.509797 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.513851 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8hq8" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.515417 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519415 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519422 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.519603 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519897 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519898 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519920 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.519999 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.520010 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521411 2554 factory.go:55] Registering systemd factory Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521425 2554 factory.go:223] Registration of the systemd container factory successfully Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521632 2554 factory.go:153] Registering CRI-O factory Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521643 2554 factory.go:223] Registration of the crio container factory successfully Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521683 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521700 2554 factory.go:103] Registering Raw factory Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.521710 2554 manager.go:1196] Started watching for new ooms in manager Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.522007 2554 manager.go:349] Starting recovery of all containers Jul 23 01:28:51.054461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.523383 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.526801 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-186.ec2.internal\" not found" node="ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.531793 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.537612 2554 manager.go:354] Recovery completed Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.565204 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567258 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567300 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567318 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567919 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567934 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567948 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.567956 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.568735 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.568765 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.568775 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.569795 2554 policy_none.go:49] "None policy: Start" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.569807 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.569816 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.570990 2554 policy_none.go:47] "Start" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.605838 2554 manager.go:337] "Starting Device Plugin manager" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.606020 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606034 2554 server.go:87] "Starting device plugin registration server" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606279 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606289 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606383 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606475 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.606484 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.606991 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.607031 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.705719 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.705754 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.705779 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.705812 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.706370 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.708372 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.708406 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.708413 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.708422 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.708548 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.714797 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.714815 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-186.ec2.internal\": node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:49.731464 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.806914 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal"] Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.807000 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.808645 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.808668 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.301874 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.808677 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.809991 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.810155 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.810183 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.810846 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.810866 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.810876 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.811236 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.811263 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.811275 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.812230 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.812264 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.812897 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.812918 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.812928 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.821722 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/233b0e2a1e1f48ac6102330b8172b615-config\") pod \"kube-apiserver-proxy-ip-10-0-136-186.ec2.internal\" (UID: \"233b0e2a1e1f48ac6102330b8172b615\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.821742 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:49.821757 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:50.888583 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:50.892765 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-186.ec2.internal\" not found" node="ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:50.896933 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-186.ec2.internal\" not found" node="ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979338 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979383 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/233b0e2a1e1f48ac6102330b8172b615-config\") pod \"kube-apiserver-proxy-ip-10-0-136-186.ec2.internal\" (UID: \"233b0e2a1e1f48ac6102330b8172b615\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.303812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979470 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/233b0e2a1e1f48ac6102330b8172b615-config\") pod \"kube-apiserver-proxy-ip-10-0-136-186.ec2.internal\" (UID: \"233b0e2a1e1f48ac6102330b8172b615\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:50.979441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8152f8ad5810793ce03a5b897a6076b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal\" (UID: \"a8152f8ad5810793ce03a5b897a6076b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:50.989415 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.089830 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.190313 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.196446 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.199362 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.304524 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.290691 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.391141 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.491638 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-186.ec2.internal\" not found" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.508795 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.508928 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.508961 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 01:28:51.541616 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.530589 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.554514 2554 apiserver.go:52] "Watching apiserver" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.563169 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.564468 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-2x8kp","openshift-multus/multus-jnsg2","openshift-multus/network-metrics-daemon-rbvvn","openshift-network-diagnostics/network-check-target-kdgzl","openshift-network-operator/iptables-alerter-vzcsh","kube-system/konnectivity-agent-85j9x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb","openshift-dns/node-resolver-dh5gm","openshift-multus/multus-additional-cni-plugins-tv78p","openshift-ovn-kubernetes/ovnkube-node-k2wsn","kube-system/global-pull-secret-syncer-6qtsd","openshift-cluster-node-tuning-operator/tuned-qtrvb"] Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.565908 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.567263 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.568477 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.568506 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.568632 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.568712 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zskbn\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.569731 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.569843 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570031 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570486 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570506 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-46kww\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570550 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570550 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.570801 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.572362 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.572784 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6gjdg\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.572846 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.572993 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.573726 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.573914 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.574228 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.574444 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pfn6r\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.574556 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.574614 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.575415 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.575817 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576144 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576292 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ch6k7\"" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576444 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576858 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.704794 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576857 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 01:23:50 +0000 UTC" deadline="2028-03-15 23:39:42.428511269 +0000 UTC" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576914 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.576914 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14446h10m50.851600415s" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.577293 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.577458 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.577805 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2btv7\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.577967 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.578025 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.578958 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.578985 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhn5g\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579037 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579054 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579300 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579306 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579380 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.579705 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.580904 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.580969 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582129 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfsxb\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582197 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582287 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582611 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9hr\" (UniqueName: \"kubernetes.io/projected/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-kube-api-access-8t9hr\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582630 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-systemd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582645 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-device-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-iptables-alerter-script\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582689 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-kubelet\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.706444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582708 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-kubelet\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582726 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582741 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-node-log\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582777 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-config\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582835 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-bin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582859 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582873 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-env-overrides\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582973 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7032703b-c778-409b-a540-5d373902d38e-serviceca\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.582998 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-system-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-log-socket\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583046 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-netd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583074 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-socket-dir-parent\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583090 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-netns\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583103 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-etc-kubernetes\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583116 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-etc-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.707913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-os-release\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583211 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-cni-binary-copy\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583225 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5dr\" (UniqueName: \"kubernetes.io/projected/88247bad-8b16-464f-8958-de0736e36ee2-kube-api-access-6z5dr\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-host-slash\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583262 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqd7b\" (UniqueName: \"kubernetes.io/projected/1694d2d6-8f82-4d67-85fb-8d40817f9992-kube-api-access-kqd7b\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583289 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-multus-certs\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583325 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583354 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-script-lib\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583367 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7032703b-c778-409b-a540-5d373902d38e-host\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583380 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz85f\" (UniqueName: \"kubernetes.io/projected/2e49321b-6027-4071-8cae-7ab0f3aea156-kube-api-access-pz85f\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzv2z\" (UniqueName: \"kubernetes.io/projected/f80e49a0-6d13-4628-930d-0b65a47c9033-kube-api-access-lzv2z\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583425 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88247bad-8b16-464f-8958-de0736e36ee2-tmp-dir\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583449 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-slash\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583463 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-netns\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.709072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583478 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-var-lib-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583510 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-k8s-cni-cncf-io\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-multus\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583549 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583573 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltf4\" (UniqueName: \"kubernetes.io/projected/c44e022b-202a-48e4-8c9f-23ef0751decf-kube-api-access-wltf4\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqqj\" (UniqueName: \"kubernetes.io/projected/7032703b-c778-409b-a540-5d373902d38e-kube-api-access-2bqqj\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583616 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-conf-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583636 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-socket-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583672 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583690 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-bin\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583714 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583737 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-cnibin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583752 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583786 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tpr\" (UniqueName: \"kubernetes.io/projected/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-kube-api-access-n8tpr\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583803 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-systemd-units\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583818 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-ovn\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.710048 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583836 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583892 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-daemon-config\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583924 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583969 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-hostroot\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.583984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-registration-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88247bad-8b16-464f-8958-de0736e36ee2-hosts-file\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-etc-selinux\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584057 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-sys-fs\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584080 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cnibin\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584110 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-os-release\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584971 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.584971 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.585059 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbv27\"" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.591136 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.594848 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.594914 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.595030 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal"] Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.606724 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.607016 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal"] Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.607454 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.611647 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lr99n" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.617423 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lr99n" Jul 23 01:28:51.711068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.677822 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684541 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-dbus\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684571 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-modprobe-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684593 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266c9325-3995-4e57-9222-e616ba4721ed-agent-certs\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684623 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-registration-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88247bad-8b16-464f-8958-de0736e36ee2-hosts-file\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-conf\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684679 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-registration-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684692 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-etc-selinux\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684715 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88247bad-8b16-464f-8958-de0736e36ee2-hosts-file\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684728 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-sys-fs\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684752 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cnibin\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-os-release\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684780 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-sys-fs\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684799 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-etc-selinux\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684819 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cnibin\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684800 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9hr\" (UniqueName: \"kubernetes.io/projected/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-kube-api-access-8t9hr\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.712539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684852 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-systemd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-device-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684894 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-iptables-alerter-script\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684933 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-kubelet\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684939 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-os-release\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-kubernetes\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684986 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-systemd\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685014 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-systemd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685031 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-tuned\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.684987 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-device-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685055 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-kubelet\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685087 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685114 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-node-log\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685121 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-kubelet\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685135 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-config\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685193 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-node-log\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713198 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685239 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-kubelet\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685265 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-sys\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685299 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266c9325-3995-4e57-9222-e616ba4721ed-konnectivity-ca\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685283 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-bin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685418 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-bin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685473 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685499 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-env-overrides\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685539 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7032703b-c778-409b-a540-5d373902d38e-serviceca\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-system-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685595 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-log-socket\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-netd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685630 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-iptables-alerter-script\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685640 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-socket-dir-parent\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685661 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-netns\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685694 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-etc-kubernetes\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685733 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-config\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.713875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-kubelet-config\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685790 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-etc-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685809 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685815 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-os-release\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685875 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-cni-binary-copy\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685901 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5dr\" (UniqueName: \"kubernetes.io/projected/88247bad-8b16-464f-8958-de0736e36ee2-kube-api-access-6z5dr\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-socket-dir-parent\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685924 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-host-slash\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685960 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqd7b\" (UniqueName: \"kubernetes.io/projected/1694d2d6-8f82-4d67-85fb-8d40817f9992-kube-api-access-kqd7b\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685989 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-log-socket\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685993 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-run\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686018 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-netd\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686027 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7032703b-c778-409b-a540-5d373902d38e-serviceca\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686045 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-multus-certs\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686059 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-netns\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686070 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:51.714441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686091 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-etc-kubernetes\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686107 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686119 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-etc-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686133 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-script-lib\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686167 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7032703b-c778-409b-a540-5d373902d38e-host\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686230 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz85f\" (UniqueName: \"kubernetes.io/projected/2e49321b-6027-4071-8cae-7ab0f3aea156-kube-api-access-pz85f\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686234 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686257 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzv2z\" (UniqueName: \"kubernetes.io/projected/f80e49a0-6d13-4628-930d-0b65a47c9033-kube-api-access-lzv2z\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686281 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-os-release\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686293 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88247bad-8b16-464f-8958-de0736e36ee2-tmp-dir\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686327 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-multus-certs\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686333 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-slash\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-netns\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686375 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-slash\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686394 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-var-lib-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686423 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-var-lib-kubelet\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686449 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-k8s-cni-cncf-io\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.715978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686472 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-multus\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686497 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686515 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-cni-binary-copy\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686529 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltf4\" (UniqueName: \"kubernetes.io/projected/c44e022b-202a-48e4-8c9f-23ef0751decf-kube-api-access-wltf4\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.685963 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-system-cni-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686572 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-netns\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686581 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-var-lib-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-host-slash\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686622 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-run-k8s-cni-cncf-io\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686658 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-openvswitch\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686702 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-lib-modules\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686738 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-host\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686791 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7032703b-c778-409b-a540-5d373902d38e-host\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686827 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-host-var-lib-cni-multus\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-tmp\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br64r\" (UniqueName: \"kubernetes.io/projected/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-kube-api-access-br64r\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686932 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqqj\" (UniqueName: \"kubernetes.io/projected/7032703b-c778-409b-a540-5d373902d38e-kube-api-access-2bqqj\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686963 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 01:28:51.716623 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-conf-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687015 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-env-overrides\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687124 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88247bad-8b16-464f-8958-de0736e36ee2-tmp-dir\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.686973 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-conf-dir\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-socket-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687233 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687277 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-bin\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687304 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687319 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f80e49a0-6d13-4628-930d-0b65a47c9033-socket-dir\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687329 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687373 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-cnibin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687375 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687375 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-cni-bin\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687397 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687423 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tpr\" (UniqueName: \"kubernetes.io/projected/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-kube-api-access-n8tpr\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687449 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-systemd-units\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.717449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-ovn\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687502 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-systemd-units\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687421 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687530 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysconfig\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687557 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-daemon-config\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687566 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-run-ovn\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687425 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-cnibin\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687582 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687609 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687637 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-hostroot\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.687668 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.687752 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:28:52.187715855 +0000 UTC m=+1.967239779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687752 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e49321b-6027-4071-8cae-7ab0f3aea156-hostroot\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687785 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1694d2d6-8f82-4d67-85fb-8d40817f9992-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.687899 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.688139 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.718037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.688205 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e49321b-6027-4071-8cae-7ab0f3aea156-multus-daemon-config\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.690453 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.697088 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.697121 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.697150 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.697238 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:52.197209461 +0000 UTC m=+1.976733373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.699632 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5dr\" (UniqueName: \"kubernetes.io/projected/88247bad-8b16-464f-8958-de0736e36ee2-kube-api-access-6z5dr\") pod \"node-resolver-dh5gm\" (UID: \"88247bad-8b16-464f-8958-de0736e36ee2\") " pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.699684 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzv2z\" (UniqueName: \"kubernetes.io/projected/f80e49a0-6d13-4628-930d-0b65a47c9033-kube-api-access-lzv2z\") pod \"aws-ebs-csi-driver-node-nd9sb\" (UID: \"f80e49a0-6d13-4628-930d-0b65a47c9033\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.700210 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqd7b\" (UniqueName: \"kubernetes.io/projected/1694d2d6-8f82-4d67-85fb-8d40817f9992-kube-api-access-kqd7b\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.700276 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqqj\" (UniqueName: \"kubernetes.io/projected/7032703b-c778-409b-a540-5d373902d38e-kube-api-access-2bqqj\") pod \"node-ca-2x8kp\" (UID: \"7032703b-c778-409b-a540-5d373902d38e\") " pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.700324 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9hr\" (UniqueName: \"kubernetes.io/projected/29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6-kube-api-access-8t9hr\") pod \"iptables-alerter-vzcsh\" (UID: \"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6\") " pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.700941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tpr\" (UniqueName: \"kubernetes.io/projected/a0df5ea7-1285-44e5-894d-c4ecbdfac36b-kube-api-access-n8tpr\") pod \"multus-additional-cni-plugins-tv78p\" (UID: \"a0df5ea7-1285-44e5-894d-c4ecbdfac36b\") " pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.709245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1694d2d6-8f82-4d67-85fb-8d40817f9992-ovnkube-script-lib\") pod \"ovnkube-node-k2wsn\" (UID: \"1694d2d6-8f82-4d67-85fb-8d40817f9992\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.709559 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz85f\" (UniqueName: \"kubernetes.io/projected/2e49321b-6027-4071-8cae-7ab0f3aea156-kube-api-access-pz85f\") pod \"multus-jnsg2\" (UID: \"2e49321b-6027-4071-8cae-7ab0f3aea156\") " pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.718657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.709564 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltf4\" (UniqueName: \"kubernetes.io/projected/c44e022b-202a-48e4-8c9f-23ef0751decf-kube-api-access-wltf4\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.730242 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tv78p" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.747012 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788106 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysconfig\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788143 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-dbus\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788161 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-modprobe-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788195 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266c9325-3995-4e57-9222-e616ba4721ed-agent-certs\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788217 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-conf\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-kubernetes\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysconfig\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-dbus\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788273 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-systemd\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788317 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-systemd\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788334 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-tuned\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-kubernetes\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-conf\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788377 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-sys\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788409 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266c9325-3995-4e57-9222-e616ba4721ed-konnectivity-ca\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.941880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788414 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-sys\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788445 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-sysctl-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788455 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-kubelet-config\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788495 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-kubelet-config\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-run\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788574 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-modprobe-d\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788597 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-var-lib-kubelet\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788640 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-lib-modules\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788664 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-host\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788686 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-tmp\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788710 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br64r\" (UniqueName: \"kubernetes.io/projected/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-kube-api-access-br64r\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.788740 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.789209 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.789239 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266c9325-3995-4e57-9222-e616ba4721ed-konnectivity-ca\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.789262 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-lib-modules\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:51.789282 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:28:52.289262988 +0000 UTC m=+2.068786897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.789317 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-host\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.789373 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-run\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.789479 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-var-lib-kubelet\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.791293 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-etc-tuned\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.791488 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266c9325-3995-4e57-9222-e616ba4721ed-agent-certs\") pod \"konnectivity-agent-85j9x\" (UID: \"266c9325-3995-4e57-9222-e616ba4721ed\") " pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.791554 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-tmp\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.800500 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8152f8ad5810793ce03a5b897a6076b.slice/crio-fa09986f6d4c8f247676f04fe99ccc8fafdcd88713c1fd983df734b01ae97070 WatchSource:0}: Error finding container fa09986f6d4c8f247676f04fe99ccc8fafdcd88713c1fd983df734b01ae97070: Status 404 returned error can't find the container with id fa09986f6d4c8f247676f04fe99ccc8fafdcd88713c1fd983df734b01ae97070 Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.801356 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233b0e2a1e1f48ac6102330b8172b615.slice/crio-a5c694cbb936fedc7837a7f13cd1adb30fa348a2f3d1582e22538d591684098c WatchSource:0}: Error finding container a5c694cbb936fedc7837a7f13cd1adb30fa348a2f3d1582e22538d591684098c: Status 404 returned error can't find the container with id a5c694cbb936fedc7837a7f13cd1adb30fa348a2f3d1582e22538d591684098c Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.802254 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br64r\" (UniqueName: \"kubernetes.io/projected/7ed4e708-1edc-4049-95b1-2ad8bbec3b1c-kube-api-access-br64r\") pod \"tuned-qtrvb\" (UID: \"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.808047 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.909193 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2x8kp" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.917083 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7032703b_c778_409b_a540_5d373902d38e.slice/crio-52380ff5c1ca82436c37fc0e4086f4fc8d7f4397ebe9076451c65bc4c549a61d WatchSource:0}: Error finding container 52380ff5c1ca82436c37fc0e4086f4fc8d7f4397ebe9076451c65bc4c549a61d: Status 404 returned error can't find the container with id 52380ff5c1ca82436c37fc0e4086f4fc8d7f4397ebe9076451c65bc4c549a61d Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.920014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jnsg2" Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.926726 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e49321b_6027_4071_8cae_7ab0f3aea156.slice/crio-96aaa7ec1552997051afa4323a53ecbc23d07f7843ed99e2daabbee4701a5669 WatchSource:0}: Error finding container 96aaa7ec1552997051afa4323a53ecbc23d07f7843ed99e2daabbee4701a5669: Status 404 returned error can't find the container with id 96aaa7ec1552997051afa4323a53ecbc23d07f7843ed99e2daabbee4701a5669 Jul 23 01:28:51.944751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.944477 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzcsh" Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.952771 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c6a7c0_ff87_4547_a7ff_e1cf2f4dedf6.slice/crio-92c42485ca492c0fdf59e2e595ce4d93d0e170a15b91c86a05ab6061e275038b WatchSource:0}: Error finding container 92c42485ca492c0fdf59e2e595ce4d93d0e170a15b91c86a05ab6061e275038b: Status 404 returned error can't find the container with id 92c42485ca492c0fdf59e2e595ce4d93d0e170a15b91c86a05ab6061e275038b Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.961604 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:51.976945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dh5gm" Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:51.984811 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88247bad_8b16_464f_8958_de0736e36ee2.slice/crio-186d9cd4ea83f1fb1a88c89948fe561ff3470692be1326f00b6036aca1bb7ac4 WatchSource:0}: Error finding container 186d9cd4ea83f1fb1a88c89948fe561ff3470692be1326f00b6036aca1bb7ac4: Status 404 returned error can't find the container with id 186d9cd4ea83f1fb1a88c89948fe561ff3470692be1326f00b6036aca1bb7ac4 Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.063842 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.071478 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:52.071797 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266c9325_3995_4e57_9222_e616ba4721ed.slice/crio-01dda8097df63ec2094310ed43298edd1b1286f48b3b9f0a0b1b6d5a01dbbda8 WatchSource:0}: Error finding container 01dda8097df63ec2094310ed43298edd1b1286f48b3b9f0a0b1b6d5a01dbbda8: Status 404 returned error can't find the container with id 01dda8097df63ec2094310ed43298edd1b1286f48b3b9f0a0b1b6d5a01dbbda8 Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:52.081316 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed4e708_1edc_4049_95b1_2ad8bbec3b1c.slice/crio-69fa189d714ac9c424f6254551d98b4e2e7dbb9e7e5eef16a32f8d92cc12ef5e WatchSource:0}: Error finding container 69fa189d714ac9c424f6254551d98b4e2e7dbb9e7e5eef16a32f8d92cc12ef5e: Status 404 returned error can't find the container with id 69fa189d714ac9c424f6254551d98b4e2e7dbb9e7e5eef16a32f8d92cc12ef5e Jul 23 01:28:52.126647 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:28:52.102239 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1694d2d6_8f82_4d67_85fb_8d40817f9992.slice/crio-047561df954699cae08ef2a2d1efaa16a0b7f9e7ba450ec855b0b23f2e92638c WatchSource:0}: Error finding container 047561df954699cae08ef2a2d1efaa16a0b7f9e7ba450ec855b0b23f2e92638c: Status 404 returned error can't find the container with id 047561df954699cae08ef2a2d1efaa16a0b7f9e7ba450ec855b0b23f2e92638c Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.192410 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.192571 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.192634 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:28:53.19261798 +0000 UTC m=+2.972141891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.293153 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.293235 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293378 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293442 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:28:53.293422943 +0000 UTC m=+3.072946874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293850 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293870 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293883 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:52.297812 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.293930 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:53.293915007 +0000 UTC m=+3.073438937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:52.554528 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.506026 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:52.554528 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.539260 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:52.633020 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.619582 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 01:23:51 +0000 UTC" deadline="2028-03-28 00:00:24.824766777 +0000 UTC" Jul 23 01:28:52.633020 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.619635 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14734h31m32.205135536s" Jul 23 01:28:52.766168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.765615 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:52.766168 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:52.765743 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:28:52.779648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.779510 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerStarted","Data":"1d4135345cb11a3ec3a47fd6581fdfd52dceafab3b8a6e70ddf88f4a15a2a6a2"} Jul 23 01:28:52.785278 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.785233 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-85j9x" event={"ID":"266c9325-3995-4e57-9222-e616ba4721ed","Type":"ContainerStarted","Data":"01dda8097df63ec2094310ed43298edd1b1286f48b3b9f0a0b1b6d5a01dbbda8"} Jul 23 01:28:52.808083 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.807981 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" event={"ID":"a8152f8ad5810793ce03a5b897a6076b","Type":"ContainerStarted","Data":"fa09986f6d4c8f247676f04fe99ccc8fafdcd88713c1fd983df734b01ae97070"} Jul 23 01:28:52.827302 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.827263 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"047561df954699cae08ef2a2d1efaa16a0b7f9e7ba450ec855b0b23f2e92638c"} Jul 23 01:28:52.847113 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.847076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" event={"ID":"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c","Type":"ContainerStarted","Data":"69fa189d714ac9c424f6254551d98b4e2e7dbb9e7e5eef16a32f8d92cc12ef5e"} Jul 23 01:28:52.850468 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.850382 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh5gm" event={"ID":"88247bad-8b16-464f-8958-de0736e36ee2","Type":"ContainerStarted","Data":"186d9cd4ea83f1fb1a88c89948fe561ff3470692be1326f00b6036aca1bb7ac4"} Jul 23 01:28:52.866654 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.866557 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" event={"ID":"f80e49a0-6d13-4628-930d-0b65a47c9033","Type":"ContainerStarted","Data":"7e7d1be363321388a8d8207449f91b8a811fccb73f5943b94980475310e1a310"} Jul 23 01:28:52.873593 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.873559 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzcsh" event={"ID":"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6","Type":"ContainerStarted","Data":"92c42485ca492c0fdf59e2e595ce4d93d0e170a15b91c86a05ab6061e275038b"} Jul 23 01:28:52.884095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.884026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jnsg2" event={"ID":"2e49321b-6027-4071-8cae-7ab0f3aea156","Type":"ContainerStarted","Data":"96aaa7ec1552997051afa4323a53ecbc23d07f7843ed99e2daabbee4701a5669"} Jul 23 01:28:52.895157 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.895076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2x8kp" event={"ID":"7032703b-c778-409b-a540-5d373902d38e","Type":"ContainerStarted","Data":"52380ff5c1ca82436c37fc0e4086f4fc8d7f4397ebe9076451c65bc4c549a61d"} Jul 23 01:28:52.913639 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:52.913597 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" event={"ID":"233b0e2a1e1f48ac6102330b8172b615","Type":"ContainerStarted","Data":"a5c694cbb936fedc7837a7f13cd1adb30fa348a2f3d1582e22538d591684098c"} Jul 23 01:28:53.200819 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.200374 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:53.200819 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.200552 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:53.200819 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.200616 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:28:55.200597407 +0000 UTC m=+4.980121320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.301491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.301557 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.301707 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.301773 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:28:55.301751293 +0000 UTC m=+5.081275203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.302193 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.302219 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.302232 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:53.302387 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.302283 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:55.30226804 +0000 UTC m=+5.081791959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:53.619999 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.619901 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 01:23:51 +0000 UTC" deadline="2028-02-06 11:47:38.736128568 +0000 UTC" Jul 23 01:28:53.619999 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.619953 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13522h18m45.116181392s" Jul 23 01:28:53.763745 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.763711 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:53.763928 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.763843 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:28:53.764000 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:53.763927 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:53.764049 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:53.764016 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:28:54.764360 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:54.764322 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:54.764800 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:54.764509 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:28:55.215960 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:55.215919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:55.216149 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.216094 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:55.216238 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.216160 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:28:59.216140374 +0000 UTC m=+8.995664284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:55.317841 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:55.317238 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:55.317841 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:55.317301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:55.317841 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.317478 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:55.317841 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.317540 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:28:59.317522597 +0000 UTC m=+9.097046507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:55.318167 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.317946 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:55.318167 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.317966 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:55.318167 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.317979 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:55.318167 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.318026 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:59.318010143 +0000 UTC m=+9.097534055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:55.763718 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:55.763143 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:55.763718 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:55.763215 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:55.763718 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.763317 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:28:55.763718 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:55.763502 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:28:56.763899 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:56.763395 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:56.763899 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:56.763533 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:28:57.763779 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:57.763746 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:57.763944 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:57.763872 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:28:57.764322 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:57.764281 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:57.764428 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:57.764389 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:28:58.763496 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:58.763466 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:58.763661 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:58.763606 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:28:59.253917 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:59.253878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:59.254418 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.254086 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:59.254418 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.254153 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:29:07.254133188 +0000 UTC m=+17.033657116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:59.355088 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:59.355021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:28:59.355298 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:59.355146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:59.355298 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355276 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:59.355298 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355291 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:59.355517 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355305 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:59.355517 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355317 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:59.355517 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355349 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:29:07.355329982 +0000 UTC m=+17.134853910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:59.355517 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.355397 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:07.35538547 +0000 UTC m=+17.134909381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:59.763676 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:59.763541 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:28:59.763855 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.763700 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:28:59.764195 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:28:59.764035 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:28:59.764195 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:28:59.764130 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:00.764709 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:00.764672 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:00.765120 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:00.764801 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:01.764049 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:01.763776 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:01.764228 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:01.763826 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:01.764228 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:01.764116 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:01.764348 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:01.764253 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:02.763551 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:02.763517 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:02.763995 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:02.763647 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:03.763729 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:03.763655 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:03.764141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:03.763664 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:03.764141 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:03.763770 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:03.764424 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:03.764349 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:04.763406 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:04.763371 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:04.763584 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:04.763511 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:05.763707 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:05.763670 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:05.763707 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:05.763691 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:05.764168 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:05.763811 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:05.764168 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:05.763915 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:06.766393 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:06.766369 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:06.766762 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:06.766460 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:07.317947 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:07.317909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:07.318113 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.318086 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:29:07.318204 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.318186 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:29:23.318154149 +0000 UTC m=+33.097678056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:29:07.418722 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:07.418628 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:07.418724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418823 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418874 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418894 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418911 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418897 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:29:23.41887543 +0000 UTC m=+33.198399338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:29:07.419024 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.418972 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:23.418956269 +0000 UTC m=+33.198480179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:29:07.763733 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:07.763651 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:07.763733 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:07.763672 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:07.763922 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.763783 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:07.763922 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:07.763902 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:08.763030 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:08.762993 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:08.763441 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:08.763137 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:09.763673 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:09.763633 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:09.764110 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:09.763633 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:09.764110 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:09.763796 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:09.764110 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:09.763921 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:10.766160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:10.766131 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:10.766574 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:10.766252 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:11.762953 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.762926 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:11.762953 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.762949 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:11.763071 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:11.763034 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:11.763199 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:11.763153 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:11.961646 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.961579 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"a4d4ee78d2e8092cb68a2726816efc5ede9516b2e0a6d36a545fd2309bce6dc3"} Jul 23 01:29:11.965997 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.965234 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" event={"ID":"7ed4e708-1edc-4049-95b1-2ad8bbec3b1c","Type":"ContainerStarted","Data":"62937be3b5c5cce07730d96208c8ebc97c0ab954966a77b3b7a55af6f2e3fa4d"} Jul 23 01:29:11.968454 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.967880 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jnsg2" event={"ID":"2e49321b-6027-4071-8cae-7ab0f3aea156","Type":"ContainerStarted","Data":"9fb89447d3b663a646ca22f258f8a64c470927e319e850473569622409b777b5"} Jul 23 01:29:11.978162 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.978131 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" event={"ID":"233b0e2a1e1f48ac6102330b8172b615","Type":"ContainerStarted","Data":"f369970967b50e24f721e9389bc317f3c813a0caa62cf289ce4a305abf0b842c"} Jul 23 01:29:11.982123 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.981614 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qtrvb" podStartSLOduration=1.557764878 podStartE2EDuration="20.981601377s" totalImagesPullingTime="19.423836499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:11.981038836 +0000 UTC m=+21.760562766" watchObservedRunningTime="2026-07-23 01:29:11.981601377 +0000 UTC m=+21.761125305" Jul 23 01:29:11.994580 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:11.994527 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-186.ec2.internal" podStartSLOduration=20.994508278 podStartE2EDuration="20.994508278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:11.993603219 +0000 UTC m=+21.773127148" watchObservedRunningTime="2026-07-23 01:29:11.994508278 +0000 UTC m=+21.774032207" Jul 23 01:29:12.010654 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.010612 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jnsg2" podStartSLOduration=2.3969292700000002 podStartE2EDuration="22.010596894s" totalImagesPullingTime="19.613667624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:12.010266675 +0000 UTC m=+21.789790604" watchObservedRunningTime="2026-07-23 01:29:12.010596894 +0000 UTC m=+21.790120823" Jul 23 01:29:12.763967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.763783 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:12.764137 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:12.764066 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:12.982398 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.982353 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dh5gm" event={"ID":"88247bad-8b16-464f-8958-de0736e36ee2","Type":"ContainerStarted","Data":"e309fed6fa6af64fa7e0bfaed98d68f8e92625cc9d72d3670ad78fc2e28cf624"} Jul 23 01:29:12.984226 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.984189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" event={"ID":"f80e49a0-6d13-4628-930d-0b65a47c9033","Type":"ContainerStarted","Data":"84676546d798a2b0a1b09805cc4f4063de7f4c8bb30318dc7a2aace90b9dc9f1"} Jul 23 01:29:12.986605 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.986577 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzcsh" event={"ID":"29c6a7c0-ff87-4547-a7ff-e1cf2f4dedf6","Type":"ContainerStarted","Data":"e20a33659a7478a3a2ea197d01d46425e16caa001640abfc200397fda4ffc454"} Jul 23 01:29:12.988053 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.988026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2x8kp" event={"ID":"7032703b-c778-409b-a540-5d373902d38e","Type":"ContainerStarted","Data":"8292c911b40ef645da12195a8df0e263ded4ebced737687753aa49ff3b5643d8"} Jul 23 01:29:12.989550 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.989523 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="7468f4db4d0045ea078115de4d3b153a77c5fc218380ddcfc578f4274be4ed59" exitCode=0 Jul 23 01:29:12.989677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.989601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"7468f4db4d0045ea078115de4d3b153a77c5fc218380ddcfc578f4274be4ed59"} Jul 23 01:29:12.990977 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.990959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-85j9x" event={"ID":"266c9325-3995-4e57-9222-e616ba4721ed","Type":"ContainerStarted","Data":"f0d910af0265c393ea033b968754073428bf0f002840ab54ce487a6a56d7c0c5"} Jul 23 01:29:12.992442 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.992419 2554 generic.go:356] "Generic (PLEG): container finished" podID="a8152f8ad5810793ce03a5b897a6076b" containerID="73409223ef71b5c2aece5467e295783e21f6dbbe154525f4d66e33e4c34ca248" exitCode=0 Jul 23 01:29:12.992534 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.992500 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" event={"ID":"a8152f8ad5810793ce03a5b897a6076b","Type":"ContainerDied","Data":"73409223ef71b5c2aece5467e295783e21f6dbbe154525f4d66e33e4c34ca248"} Jul 23 01:29:12.995223 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.995205 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"c4cbbb92753918327f91493b22051e0d66df442e00aad7382c56de5734219588"} Jul 23 01:29:12.995324 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.995271 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"22832493b30db1bddc04f0fac255bb47a9d8f030601643aebc5fa25d00511e6c"} Jul 23 01:29:12.995388 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.995344 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"a05369ef284b9ad5fd9138ddc27b0192aec53e4577f578a034eea551145448d7"} Jul 23 01:29:12.995444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.995395 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"526b40d1aa3966877f155346a9a381bc2909eb28e339a593d4c8bcf7b1e612b9"} Jul 23 01:29:12.995502 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.995452 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"8b1e31b9244353adac036a8df22f26ad88e1fbea43ceb87c4ff3221a4d676875"} Jul 23 01:29:12.996314 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:12.996277 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dh5gm" podStartSLOduration=3.475682201 podStartE2EDuration="22.996263347s" totalImagesPullingTime="19.520581146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:12.996051793 +0000 UTC m=+22.775575722" watchObservedRunningTime="2026-07-23 01:29:12.996263347 +0000 UTC m=+22.775787276" Jul 23 01:29:13.022508 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.022463 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vzcsh" podStartSLOduration=3.520243543 podStartE2EDuration="23.02244969s" totalImagesPullingTime="19.502206147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:13.022335564 +0000 UTC m=+22.801859492" watchObservedRunningTime="2026-07-23 01:29:13.02244969 +0000 UTC m=+22.801973619" Jul 23 01:29:13.035426 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.035381 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-85j9x" podStartSLOduration=2.6041431040000003 podStartE2EDuration="22.035368826s" totalImagesPullingTime="19.431225722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:13.035261881 +0000 UTC m=+22.814785809" watchObservedRunningTime="2026-07-23 01:29:13.035368826 +0000 UTC m=+22.814892755" Jul 23 01:29:13.047666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.047621 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2x8kp" podStartSLOduration=3.493931623 podStartE2EDuration="23.04760775s" totalImagesPullingTime="19.553676127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:13.04740509 +0000 UTC m=+22.826929019" watchObservedRunningTime="2026-07-23 01:29:13.04760775 +0000 UTC m=+22.827131678" Jul 23 01:29:13.457743 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.457719 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 01:29:13.676614 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.676465 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T01:29:13.457737741Z","UUID":"71b7c9a0-f799-434b-8ed1-69d6a24ebf68","Handler":null,"Name":"","Endpoint":""} Jul 23 01:29:13.678165 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.678145 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 01:29:13.678318 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.678205 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 01:29:13.763132 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.763095 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:13.763323 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:13.763243 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:13.763323 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.763298 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:13.763435 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:13.763413 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:13.909039 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.909002 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:29:13.909658 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.909635 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:29:13.999798 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:13.999705 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" event={"ID":"a8152f8ad5810793ce03a5b897a6076b","Type":"ContainerStarted","Data":"27a133808e5be7fa956e7e95af4f71c8b2c3bc65636a19a35013004b5d7f1ee0"} Jul 23 01:29:14.001773 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:14.001723 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" event={"ID":"f80e49a0-6d13-4628-930d-0b65a47c9033","Type":"ContainerStarted","Data":"b33963152393657d229dafcc5b26a45ddf75cc9e8e7e0f38221a21b013a1fc56"} Jul 23 01:29:14.002418 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:14.002393 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:29:14.002925 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:14.002906 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-85j9x" Jul 23 01:29:14.015784 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:14.015737 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-186.ec2.internal" podStartSLOduration=23.015723006 podStartE2EDuration="23.015723006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:14.015295297 +0000 UTC m=+23.794819230" watchObservedRunningTime="2026-07-23 01:29:14.015723006 +0000 UTC m=+23.795246935" Jul 23 01:29:14.763075 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:14.763042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:14.763279 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:14.763227 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:15.008473 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.008394 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"af4cacf115457189ec7f3d1727e4111124c062b5cfddf44772b8276d85cbb4cd"} Jul 23 01:29:15.010905 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.010868 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" event={"ID":"f80e49a0-6d13-4628-930d-0b65a47c9033","Type":"ContainerStarted","Data":"9a253ede27a6c4f2c7c7047fec36363e13cd84745f18172d47f4a85ad3e3fe60"} Jul 23 01:29:15.030043 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.029992 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nd9sb" podStartSLOduration=7.500691032 podStartE2EDuration="25.029971572s" totalImagesPullingTime="17.52928054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:15.029365593 +0000 UTC m=+24.808889524" watchObservedRunningTime="2026-07-23 01:29:15.029971572 +0000 UTC m=+24.809495499" Jul 23 01:29:15.763796 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.763761 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:15.763970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.763773 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:15.763970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:15.763873 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:15.763970 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:15.763878 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:15.763970 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:15.763935 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:15.764119 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:15.764013 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:17.763451 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:17.763379 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:17.763451 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:17.763421 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:17.764161 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:17.763385 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:17.764161 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:17.763489 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:17.764161 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:17.763545 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:17.764161 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:17.763625 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:18.020807 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.020724 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" event={"ID":"1694d2d6-8f82-4d67-85fb-8d40817f9992","Type":"ContainerStarted","Data":"1e91ca8e18266c5c9580e999b91751754d71f683c62f3bc3afbe30a76ae52560"} Jul 23 01:29:18.021070 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.021025 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:18.022613 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.022589 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="cf78b274075094ba40fb5db0e4fbdc8981eb0dd8d0748d814151089775d9b682" exitCode=0 Jul 23 01:29:18.022704 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.022622 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"cf78b274075094ba40fb5db0e4fbdc8981eb0dd8d0748d814151089775d9b682"} Jul 23 01:29:18.036245 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.036223 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:18.045148 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:18.045112 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" podStartSLOduration=8.328210723 podStartE2EDuration="28.045098973s" totalImagesPullingTime="19.71688825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:18.043801143 +0000 UTC m=+27.823325072" watchObservedRunningTime="2026-07-23 01:29:18.045098973 +0000 UTC m=+27.824622900" Jul 23 01:29:19.025009 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.024843 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:19.025431 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.025055 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:19.039949 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.039925 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:19.763831 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.763755 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:19.763996 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.763762 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:19.763996 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:19.763865 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:19.763996 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:19.763762 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:19.763996 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:19.763966 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:19.764160 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:19.764031 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:20.029188 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:20.029101 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="3e2c64f7ae6ca03f110fe8d97db39d4364dd8c45561fbef868891a5c5d364c7f" exitCode=0 Jul 23 01:29:20.029539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:20.029189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"3e2c64f7ae6ca03f110fe8d97db39d4364dd8c45561fbef868891a5c5d364c7f"} Jul 23 01:29:21.763513 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:21.763430 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:21.763901 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:21.763433 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:21.763901 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:21.763542 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:21.763901 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:21.763553 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:21.763901 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:21.763662 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:21.763901 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:21.763752 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:22.037077 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.037049 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="d517600fcb594246bb2ce45b136ae873d8f72387f29c1e630e3ab53be2484dad" exitCode=0 Jul 23 01:29:22.037189 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.037094 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"d517600fcb594246bb2ce45b136ae873d8f72387f29c1e630e3ab53be2484dad"} Jul 23 01:29:22.161951 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.161922 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdgzl"] Jul 23 01:29:22.162100 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.162013 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:22.162140 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:22.162092 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:22.164873 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.164846 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qtsd"] Jul 23 01:29:22.165099 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.165062 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:22.165208 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:22.165190 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:22.165657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.165635 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbvvn"] Jul 23 01:29:22.165738 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:22.165724 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:22.165847 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:22.165825 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:23.343295 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.343259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:23.343709 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.343517 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:29:23.343709 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.343593 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs podName:c44e022b-202a-48e4-8c9f-23ef0751decf nodeName:}" failed. No retries permitted until 2026-07-23 01:29:55.343573672 +0000 UTC m=+65.123097584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs") pod "network-metrics-daemon-rbvvn" (UID: "c44e022b-202a-48e4-8c9f-23ef0751decf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:29:23.444293 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.444245 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:23.444467 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.444320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:23.444467 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444380 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:29:23.444467 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444403 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:29:23.444467 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444416 2554 projected.go:196] Error preparing data for projected volume kube-api-access-bkqwz for pod openshift-network-diagnostics/network-check-target-kdgzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:29:23.444467 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444446 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:29:23.444729 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444472 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz podName:92e1ed67-e3ad-422a-bbd0-8aab66787164 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:55.444457176 +0000 UTC m=+65.223981082 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkqwz" (UniqueName: "kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz") pod "network-check-target-kdgzl" (UID: "92e1ed67-e3ad-422a-bbd0-8aab66787164") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:29:23.444729 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.444493 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret podName:3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e nodeName:}" failed. No retries permitted until 2026-07-23 01:29:55.444479566 +0000 UTC m=+65.224003473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret") pod "global-pull-secret-syncer-6qtsd" (UID: "3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:29:23.763875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.763801 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:23.764023 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.763801 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:23.764023 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.763922 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbvvn" podUID="c44e022b-202a-48e4-8c9f-23ef0751decf" Jul 23 01:29:23.764023 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:23.763801 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:23.764023 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.764008 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdgzl" podUID="92e1ed67-e3ad-422a-bbd0-8aab66787164" Jul 23 01:29:23.764199 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:23.764096 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qtsd" podUID="3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e" Jul 23 01:29:25.562825 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.562586 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-186.ec2.internal" event="NodeReady" Jul 23 01:29:25.563266 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.562897 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 01:29:25.614956 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.614927 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5tgvg"] Jul 23 01:29:25.617779 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.617748 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xj2xk"] Jul 23 01:29:25.617912 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.617885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.619624 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.619601 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.620562 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.620539 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjsgq\"" Jul 23 01:29:25.620678 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.620575 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 01:29:25.620678 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.620631 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 01:29:25.621943 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.621919 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 01:29:25.621943 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.621938 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 01:29:25.622111 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.621959 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl24z\"" Jul 23 01:29:25.622762 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.622746 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 01:29:25.627086 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.626503 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5tgvg"] Jul 23 01:29:25.629011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.628961 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xj2xk"] Jul 23 01:29:25.709983 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.709954 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhd9"] Jul 23 01:29:25.713521 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.713493 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.716761 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.716745 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 01:29:25.716882 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.716747 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 01:29:25.717057 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.717038 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hds2f\"" Jul 23 01:29:25.717538 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.717519 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 01:29:25.717631 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.717531 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 01:29:25.724276 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.724258 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhd9"] Jul 23 01:29:25.762215 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-metrics-tls\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.762345 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762241 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmnj\" (UniqueName: \"kubernetes.io/projected/9d61d55c-06e0-4453-b29d-c95619694ee9-kube-api-access-mwmnj\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.762345 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762273 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-config-volume\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.762345 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762307 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-tmp-dir\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.762503 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762422 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4vx\" (UniqueName: \"kubernetes.io/projected/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-kube-api-access-ng4vx\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.762503 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.762451 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d61d55c-06e0-4453-b29d-c95619694ee9-cert\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.763208 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.763163 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:25.763286 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.763270 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:25.763341 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.763293 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:25.766374 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766350 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 01:29:25.766476 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766397 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 01:29:25.766545 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766473 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lg5p2\"" Jul 23 01:29:25.766545 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766473 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 01:29:25.766645 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766475 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frx6l\"" Jul 23 01:29:25.766645 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.766547 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 01:29:25.863041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.862952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-metrics-tls\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.863041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1c65a8-d97e-4c96-8910-bf8bd928063e-data-volume\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.863285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863123 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64gs\" (UniqueName: \"kubernetes.io/projected/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-api-access-d64gs\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.863285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863200 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmnj\" (UniqueName: \"kubernetes.io/projected/9d61d55c-06e0-4453-b29d-c95619694ee9-kube-api-access-mwmnj\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.863285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-config-volume\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.863285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863247 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-tmp-dir\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.863466 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863410 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.863466 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863453 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca1c65a8-d97e-4c96-8910-bf8bd928063e-crio-socket\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.863568 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca1c65a8-d97e-4c96-8910-bf8bd928063e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.863568 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863523 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4vx\" (UniqueName: \"kubernetes.io/projected/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-kube-api-access-ng4vx\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.863668 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863576 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d61d55c-06e0-4453-b29d-c95619694ee9-cert\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.863871 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.863852 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-config-volume\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.864087 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.864060 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-tmp-dir\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.867475 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.867454 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-metrics-tls\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.867611 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.867596 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d61d55c-06e0-4453-b29d-c95619694ee9-cert\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.871378 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.871359 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4vx\" (UniqueName: \"kubernetes.io/projected/ae7f258d-cdf1-42a0-8783-3b9fc4c80f63-kube-api-access-ng4vx\") pod \"dns-default-5tgvg\" (UID: \"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63\") " pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.871475 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.871387 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmnj\" (UniqueName: \"kubernetes.io/projected/9d61d55c-06e0-4453-b29d-c95619694ee9-kube-api-access-mwmnj\") pod \"ingress-canary-xj2xk\" (UID: \"9d61d55c-06e0-4453-b29d-c95619694ee9\") " pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.931469 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.931427 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:25.937225 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.937202 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xj2xk" Jul 23 01:29:25.964687 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.964652 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64gs\" (UniqueName: \"kubernetes.io/projected/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-api-access-d64gs\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.964869 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.964842 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.964993 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.964878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca1c65a8-d97e-4c96-8910-bf8bd928063e-crio-socket\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.964993 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.964904 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca1c65a8-d97e-4c96-8910-bf8bd928063e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.964993 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.964957 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1c65a8-d97e-4c96-8910-bf8bd928063e-data-volume\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.965145 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.965120 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca1c65a8-d97e-4c96-8910-bf8bd928063e-crio-socket\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.965345 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.965326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1c65a8-d97e-4c96-8910-bf8bd928063e-data-volume\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.965519 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.965498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.967638 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.967614 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca1c65a8-d97e-4c96-8910-bf8bd928063e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:25.973413 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:25.973394 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64gs\" (UniqueName: \"kubernetes.io/projected/ca1c65a8-d97e-4c96-8910-bf8bd928063e-kube-api-access-d64gs\") pod \"insights-runtime-extractor-wkhd9\" (UID: \"ca1c65a8-d97e-4c96-8910-bf8bd928063e\") " pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:26.023663 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:26.023623 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhd9" Jul 23 01:29:26.359846 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:26.359568 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhd9"] Jul 23 01:29:26.360426 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:26.360268 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5tgvg"] Jul 23 01:29:26.363788 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:26.363761 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7f258d_cdf1_42a0_8783_3b9fc4c80f63.slice/crio-e9270b204c00bd7444a187cf42fa7234989ab5506b80d1b86fb9857747ff34ee WatchSource:0}: Error finding container e9270b204c00bd7444a187cf42fa7234989ab5506b80d1b86fb9857747ff34ee: Status 404 returned error can't find the container with id e9270b204c00bd7444a187cf42fa7234989ab5506b80d1b86fb9857747ff34ee Jul 23 01:29:26.364972 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:26.364946 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1c65a8_d97e_4c96_8910_bf8bd928063e.slice/crio-1e5f2a83adf39e71f65ab2f591cc485547f9090d47941f4e8eb1d83e8499a9f6 WatchSource:0}: Error finding container 1e5f2a83adf39e71f65ab2f591cc485547f9090d47941f4e8eb1d83e8499a9f6: Status 404 returned error can't find the container with id 1e5f2a83adf39e71f65ab2f591cc485547f9090d47941f4e8eb1d83e8499a9f6 Jul 23 01:29:26.372586 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:26.372558 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xj2xk"] Jul 23 01:29:26.375795 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:26.375771 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d61d55c_06e0_4453_b29d_c95619694ee9.slice/crio-af0b7c4b87f8a7851723a998da3f9893c2ea43ae8b9c44c64c7094341d0b6545 WatchSource:0}: Error finding container af0b7c4b87f8a7851723a998da3f9893c2ea43ae8b9c44c64c7094341d0b6545: Status 404 returned error can't find the container with id af0b7c4b87f8a7851723a998da3f9893c2ea43ae8b9c44c64c7094341d0b6545 Jul 23 01:29:27.055576 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:27.055534 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="d4e817b8c237334e875f64ef78455ed5d0e0af91bc7ffc834cae61f5ca54b7d7" exitCode=0 Jul 23 01:29:27.056380 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:27.055631 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"d4e817b8c237334e875f64ef78455ed5d0e0af91bc7ffc834cae61f5ca54b7d7"} Jul 23 01:29:27.058867 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:27.058783 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xj2xk" event={"ID":"9d61d55c-06e0-4453-b29d-c95619694ee9","Type":"ContainerStarted","Data":"af0b7c4b87f8a7851723a998da3f9893c2ea43ae8b9c44c64c7094341d0b6545"} Jul 23 01:29:27.062751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:27.062680 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tgvg" event={"ID":"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63","Type":"ContainerStarted","Data":"e9270b204c00bd7444a187cf42fa7234989ab5506b80d1b86fb9857747ff34ee"} Jul 23 01:29:27.067293 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:27.067232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhd9" event={"ID":"ca1c65a8-d97e-4c96-8910-bf8bd928063e","Type":"ContainerStarted","Data":"1e5f2a83adf39e71f65ab2f591cc485547f9090d47941f4e8eb1d83e8499a9f6"} Jul 23 01:29:28.074575 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:28.074537 2554 generic.go:356] "Generic (PLEG): container finished" podID="a0df5ea7-1285-44e5-894d-c4ecbdfac36b" containerID="024c4d48cf4f12f619a976b617f9efd2a7ae5fd888ba1c0d87002b5f4108fa76" exitCode=0 Jul 23 01:29:28.075011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:28.074594 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerDied","Data":"024c4d48cf4f12f619a976b617f9efd2a7ae5fd888ba1c0d87002b5f4108fa76"} Jul 23 01:29:29.081552 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:29.081515 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv78p" event={"ID":"a0df5ea7-1285-44e5-894d-c4ecbdfac36b","Type":"ContainerStarted","Data":"81f673ac07685fdaf714aebfaee59609bb742d85fa5e86068e4e72e829439804"} Jul 23 01:29:29.119617 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:29.119552 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tv78p" podStartSLOduration=9.010451147 podStartE2EDuration="39.119531123s" totalImagesPullingTime="30.109079976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:29.118395496 +0000 UTC m=+38.897919439" watchObservedRunningTime="2026-07-23 01:29:29.119531123 +0000 UTC m=+38.899055077" Jul 23 01:29:30.676885 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.676849 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:29:30.791608 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.791574 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.795848 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.795759 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:29:30.795994 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.795869 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z9qsp\"" Jul 23 01:29:30.795994 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.795877 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 01:29:30.795994 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.795884 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 01:29:30.795994 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.795909 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 01:29:30.796235 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.796141 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 01:29:30.796235 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.796231 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 01:29:30.796335 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.796238 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 01:29:30.796335 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.796247 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 01:29:30.910967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.910926 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.911141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.910977 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.911141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.911000 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.911141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.911020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.911141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.911102 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:30.911359 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:30.911163 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq762\" (UniqueName: \"kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012266 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012266 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012220 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq762\" (UniqueName: \"kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012381 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012401 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012422 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.012588 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.012445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.013202 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.013157 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.013202 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.013192 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.013379 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.013302 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.016517 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.016489 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.016517 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.016498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.020187 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.020156 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq762\" (UniqueName: \"kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762\") pod \"console-79f4b4cb66-wbjnn\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.102976 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.102942 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:31.535497 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.535462 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2"] Jul 23 01:29:31.637981 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.637952 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2"] Jul 23 01:29:31.638129 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.638056 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.640672 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.640651 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 23 01:29:31.640932 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.640907 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 01:29:31.640932 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.640912 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 23 01:29:31.641067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.640977 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 01:29:31.641933 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.641913 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7dqpr\"" Jul 23 01:29:31.642066 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.642031 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 01:29:31.818433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.818355 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3870bcb-4e65-44fd-bad0-dea55260755a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.818808 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.818464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.818808 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.818491 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwnc\" (UniqueName: \"kubernetes.io/projected/f3870bcb-4e65-44fd-bad0-dea55260755a-kube-api-access-wzwnc\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.818808 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.818527 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.919492 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.919463 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3870bcb-4e65-44fd-bad0-dea55260755a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.919666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.919645 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.919708 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.919676 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwnc\" (UniqueName: \"kubernetes.io/projected/f3870bcb-4e65-44fd-bad0-dea55260755a-kube-api-access-wzwnc\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.919708 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.919702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.919827 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:31.919813 2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 23 01:29:31.919887 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:31.919878 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls podName:f3870bcb-4e65-44fd-bad0-dea55260755a nodeName:}" failed. No retries permitted until 2026-07-23 01:29:32.419859121 +0000 UTC m=+42.199383044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-xb7m2" (UID: "f3870bcb-4e65-44fd-bad0-dea55260755a") : secret "prometheus-operator-tls" not found Jul 23 01:29:31.920165 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.920148 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3870bcb-4e65-44fd-bad0-dea55260755a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.921879 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.921861 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:31.927901 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:31.927882 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwnc\" (UniqueName: \"kubernetes.io/projected/f3870bcb-4e65-44fd-bad0-dea55260755a-kube-api-access-wzwnc\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:32.423598 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:32.423565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:32.425968 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:32.425945 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3870bcb-4e65-44fd-bad0-dea55260755a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xb7m2\" (UID: \"f3870bcb-4e65-44fd-bad0-dea55260755a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:32.546474 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:32.546443 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" Jul 23 01:29:32.925997 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:32.925971 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2"] Jul 23 01:29:32.934202 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:32.934158 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3870bcb_4e65_44fd_bad0_dea55260755a.slice/crio-358c5062e2676c119caba7c71d05788cf29c109000f3cd6c5d41d639a1d83aa2 WatchSource:0}: Error finding container 358c5062e2676c119caba7c71d05788cf29c109000f3cd6c5d41d639a1d83aa2: Status 404 returned error can't find the container with id 358c5062e2676c119caba7c71d05788cf29c109000f3cd6c5d41d639a1d83aa2 Jul 23 01:29:32.939498 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:32.939476 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:29:32.943463 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:32.943440 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849fc9bd_2902_4e99_b6e3_e1206bedff07.slice/crio-3e9a0e0aa53b6c3efbb032c8af54c03196e71549174043db0ed527eb0fd4cab0 WatchSource:0}: Error finding container 3e9a0e0aa53b6c3efbb032c8af54c03196e71549174043db0ed527eb0fd4cab0: Status 404 returned error can't find the container with id 3e9a0e0aa53b6c3efbb032c8af54c03196e71549174043db0ed527eb0fd4cab0 Jul 23 01:29:33.094322 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:33.094235 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xj2xk" event={"ID":"9d61d55c-06e0-4453-b29d-c95619694ee9","Type":"ContainerStarted","Data":"f85de0d38f21d1038022bf648de6c55f908048d0c5e78a6bfd55f5dea8b2bad2"} Jul 23 01:29:33.095583 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:33.095554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f4b4cb66-wbjnn" event={"ID":"849fc9bd-2902-4e99-b6e3-e1206bedff07","Type":"ContainerStarted","Data":"3e9a0e0aa53b6c3efbb032c8af54c03196e71549174043db0ed527eb0fd4cab0"} Jul 23 01:29:33.097055 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:33.097030 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tgvg" event={"ID":"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63","Type":"ContainerStarted","Data":"7f04c44f3223e6b20eae2c13864d8a32f69f874d560616fa3ba20eaab5252f64"} Jul 23 01:29:33.098137 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:33.098113 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" event={"ID":"f3870bcb-4e65-44fd-bad0-dea55260755a","Type":"ContainerStarted","Data":"358c5062e2676c119caba7c71d05788cf29c109000f3cd6c5d41d639a1d83aa2"} Jul 23 01:29:33.109416 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:33.109375 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xj2xk" podStartSLOduration=1.721823833 podStartE2EDuration="8.109361011s" totalImagesPullingTime="6.387537178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:33.108333701 +0000 UTC m=+42.887857630" watchObservedRunningTime="2026-07-23 01:29:33.109361011 +0000 UTC m=+42.888885007" Jul 23 01:29:34.103953 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:34.103911 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhd9" event={"ID":"ca1c65a8-d97e-4c96-8910-bf8bd928063e","Type":"ContainerStarted","Data":"54db4320faa9e61cb15ddb7960b4bf182b8ad3292bb4436c1e7b5caa96ed987f"} Jul 23 01:29:34.107818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:34.107787 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tgvg" event={"ID":"ae7f258d-cdf1-42a0-8783-3b9fc4c80f63","Type":"ContainerStarted","Data":"cf8c71963758f898a5cef5bcd4db19131b9c4a4ea42e4d081a33bd5e988c0124"} Jul 23 01:29:34.107966 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:34.107862 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:34.127063 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:34.127008 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5tgvg" podStartSLOduration=2.727081326 podStartE2EDuration="9.126986696s" totalImagesPullingTime="6.39990537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:34.125107505 +0000 UTC m=+43.904631434" watchObservedRunningTime="2026-07-23 01:29:34.126986696 +0000 UTC m=+43.906510606" Jul 23 01:29:35.119877 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:35.118456 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" event={"ID":"f3870bcb-4e65-44fd-bad0-dea55260755a","Type":"ContainerStarted","Data":"cabb7cb9db1770989c9f20c9c3598a92f93f5796d2680a2bb3e0d21323eba68c"} Jul 23 01:29:35.119877 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:35.118513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" event={"ID":"f3870bcb-4e65-44fd-bad0-dea55260755a","Type":"ContainerStarted","Data":"1b51ed88e5a8c109242bf33e51408d9496a9372399803b76af7b275ce523c722"} Jul 23 01:29:35.150615 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:35.150547 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xb7m2" podStartSLOduration=2.238703743 podStartE2EDuration="4.150525024s" totalImagesPullingTime="1.911821281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:35.14974533 +0000 UTC m=+44.929269262" watchObservedRunningTime="2026-07-23 01:29:35.150525024 +0000 UTC m=+44.930048953" Jul 23 01:29:36.902822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.902771 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn"] Jul 23 01:29:36.925836 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.925809 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn"] Jul 23 01:29:36.926107 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.925865 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg5h"] Jul 23 01:29:36.926107 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.926027 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:36.929093 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.929074 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 01:29:36.929247 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.929228 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9fmfl\"" Jul 23 01:29:36.929327 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.929316 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 01:29:36.935580 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.935559 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:36.935948 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.935924 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wrvrm"] Jul 23 01:29:36.938296 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.938278 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 01:29:36.939015 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.938900 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 01:29:36.939015 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.938963 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kd8n9\"" Jul 23 01:29:36.940335 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.940312 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 01:29:36.949522 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.949501 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg5h"] Jul 23 01:29:36.949612 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.949601 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:36.952268 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.952249 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 01:29:36.952698 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.952678 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 01:29:36.952800 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.952697 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nh4h7\"" Jul 23 01:29:36.952956 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:36.952940 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 01:29:37.063066 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063024 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-textfile\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063066 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063067 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063089 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtkl\" (UniqueName: \"kubernetes.io/projected/730ed4ae-093f-444e-92ee-1e8c65dfe798-kube-api-access-fmtkl\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-root\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063254 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063282 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063304 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.063333 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063328 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-sys\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063389 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063422 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/20020c4c-bfb4-404b-9efe-fbd15bed17e8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063456 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063511 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063542 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcn4w\" (UniqueName: \"kubernetes.io/projected/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-kube-api-access-hcn4w\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063580 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/730ed4ae-093f-444e-92ee-1e8c65dfe798-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.063643 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-wtmp\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.064167 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063639 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-metrics-client-ca\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.064167 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063680 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.064167 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.063738 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsgj\" (UniqueName: \"kubernetes.io/projected/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-api-access-czsgj\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.165204 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165114 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165204 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165159 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcn4w\" (UniqueName: \"kubernetes.io/projected/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-kube-api-access-hcn4w\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165438 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.165298 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 01:29:37.165438 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/730ed4ae-093f-444e-92ee-1e8c65dfe798-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.165438 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.165377 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls podName:da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:37.665353535 +0000 UTC m=+47.444877462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls") pod "node-exporter-wrvrm" (UID: "da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023") : secret "node-exporter-tls" not found Jul 23 01:29:37.165438 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165412 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-wtmp\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165438 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-metrics-client-ca\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165517 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsgj\" (UniqueName: \"kubernetes.io/projected/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-api-access-czsgj\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-textfile\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165570 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtkl\" (UniqueName: \"kubernetes.io/projected/730ed4ae-093f-444e-92ee-1e8c65dfe798-kube-api-access-fmtkl\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165630 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-root\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.165679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165678 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165703 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-wtmp\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165733 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165773 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-sys\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165841 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/20020c4c-bfb4-404b-9efe-fbd15bed17e8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.165924 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166031 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166012 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-metrics-client-ca\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166108 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-sys\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166198 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/730ed4ae-093f-444e-92ee-1e8c65dfe798-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.166304 2554 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.166354 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls podName:20020c4c-bfb4-404b-9efe-fbd15bed17e8 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:37.666337962 +0000 UTC m=+47.445861876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-vwg5h" (UID: "20020c4c-bfb4-404b-9efe-fbd15bed17e8") : secret "kube-state-metrics-tls" not found Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166408 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/20020c4c-bfb4-404b-9efe-fbd15bed17e8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166464 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166494 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166468 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-root\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.166873 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.166542 2554 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 23 01:29:37.166873 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:37.166589 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls podName:730ed4ae-093f-444e-92ee-1e8c65dfe798 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:37.666574378 +0000 UTC m=+47.446098301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-t65hn" (UID: "730ed4ae-093f-444e-92ee-1e8c65dfe798") : secret "openshift-state-metrics-tls" not found Jul 23 01:29:37.166873 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166626 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.167019 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-textfile\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.167019 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.166953 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20020c4c-bfb4-404b-9efe-fbd15bed17e8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.168468 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.168445 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.169098 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.169076 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.169304 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.169280 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.174156 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.174137 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcn4w\" (UniqueName: \"kubernetes.io/projected/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-kube-api-access-hcn4w\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.175343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.175300 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtkl\" (UniqueName: \"kubernetes.io/projected/730ed4ae-093f-444e-92ee-1e8c65dfe798-kube-api-access-fmtkl\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.175753 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.175731 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsgj\" (UniqueName: \"kubernetes.io/projected/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-api-access-czsgj\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.671579 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.671542 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.671769 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.671619 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.671769 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.671661 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.674454 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.674403 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/730ed4ae-093f-444e-92ee-1e8c65dfe798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-t65hn\" (UID: \"730ed4ae-093f-444e-92ee-1e8c65dfe798\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.674598 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.674451 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023-node-exporter-tls\") pod \"node-exporter-wrvrm\" (UID: \"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023\") " pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.674598 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.674515 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/20020c4c-bfb4-404b-9efe-fbd15bed17e8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg5h\" (UID: \"20020c4c-bfb4-404b-9efe-fbd15bed17e8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.836722 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.836692 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" Jul 23 01:29:37.846752 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.846729 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" Jul 23 01:29:37.858913 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.858889 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrvrm" Jul 23 01:29:37.980042 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:37.979950 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:29:38.001847 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.001797 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:29:38.002032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.002014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.005983 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.005942 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 01:29:38.006208 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.006189 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 01:29:38.006558 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.006331 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 01:29:38.006558 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.006447 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 01:29:38.006558 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.006547 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 01:29:38.007291 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.007270 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 01:29:38.007393 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.007314 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-shlfr\"" Jul 23 01:29:38.007393 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.007314 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 01:29:38.007571 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.007541 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 01:29:38.007671 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.007616 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 01:29:38.176879 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.176846 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.176909 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.176945 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.176983 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177054 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177093 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177157 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177229 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf476\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177280 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177344 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177755 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177368 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.177755 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.177414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278318 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278318 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278289 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278336 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278437 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278817 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278672 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.278817 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.278715 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.279863 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.279891 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.279950 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.280000 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.280032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf476\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280199 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.280057 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.280959 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.280575 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.281496 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.281446 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.281496 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.281475 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.281651 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.281557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.282934 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.282907 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.283299 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.283257 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.283886 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.283705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.283886 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.283845 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.284397 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.284357 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.284706 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.284683 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.290905 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.290886 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf476\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476\") pod \"alertmanager-main-0\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.316751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.316719 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:29:38.877482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.877447 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-665f54d6bd-vwc8s"] Jul 23 01:29:38.902575 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.902543 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665f54d6bd-vwc8s"] Jul 23 01:29:38.902725 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.902657 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.905285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905263 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 01:29:38.905285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905285 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 01:29:38.905482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905268 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 01:29:38.905482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905390 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 01:29:38.905482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905440 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 01:29:38.905666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905630 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b4s4d34rn97bi\"" Jul 23 01:29:38.905747 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.905727 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7wlds\"" Jul 23 01:29:38.965048 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:38.964260 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2a0ccb_f8b6_4f12_b7d9_3231e1c2b023.slice/crio-8a2e787109d6d062e0a32c413313cddc99a5ed4e212f027e4535363e9143f1ee WatchSource:0}: Error finding container 8a2e787109d6d062e0a32c413313cddc99a5ed4e212f027e4535363e9143f1ee: Status 404 returned error can't find the container with id 8a2e787109d6d062e0a32c413313cddc99a5ed4e212f027e4535363e9143f1ee Jul 23 01:29:38.985082 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.984961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985082 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.984998 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985082 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985026 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985149 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6037316-e72e-4649-8ca2-d156f1964464-metrics-client-ca\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985211 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-grpc-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985264 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985312 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/b6037316-e72e-4649-8ca2-d156f1964464-kube-api-access-9q2xj\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:38.985603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:38.985406 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.086396 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086296 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.086396 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086342 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.086396 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087478 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086773 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6037316-e72e-4649-8ca2-d156f1964464-metrics-client-ca\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087478 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-grpc-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087478 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087478 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086908 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/b6037316-e72e-4649-8ca2-d156f1964464-kube-api-access-9q2xj\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087478 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.086997 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.087797 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.087702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6037316-e72e-4649-8ca2-d156f1964464-metrics-client-ca\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.092315 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.092287 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-grpc-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.092597 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.092574 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.092784 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.092764 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-tls\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.093286 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.093265 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.093374 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.093265 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.095202 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.095056 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6037316-e72e-4649-8ca2-d156f1964464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.098229 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.098170 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/b6037316-e72e-4649-8ca2-d156f1964464-kube-api-access-9q2xj\") pod \"thanos-querier-665f54d6bd-vwc8s\" (UID: \"b6037316-e72e-4649-8ca2-d156f1964464\") " pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.105840 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.105270 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg5h"] Jul 23 01:29:39.108650 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:39.108617 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20020c4c_bfb4_404b_9efe_fbd15bed17e8.slice/crio-f94c4528ce12e55af3483ef09b9c8a8b5f4119cb4f3d57a1acc7a68f39df9e85 WatchSource:0}: Error finding container f94c4528ce12e55af3483ef09b9c8a8b5f4119cb4f3d57a1acc7a68f39df9e85: Status 404 returned error can't find the container with id f94c4528ce12e55af3483ef09b9c8a8b5f4119cb4f3d57a1acc7a68f39df9e85 Jul 23 01:29:39.128129 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.128058 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn"] Jul 23 01:29:39.131676 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.131553 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:29:39.136698 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:39.136623 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae276df8_8747_4a12_bf0d_b980fab6696d.slice/crio-5026676d23d4f07e5ad43ae226717e05074b81f6ab1d07981babba4468d9f9bf WatchSource:0}: Error finding container 5026676d23d4f07e5ad43ae226717e05074b81f6ab1d07981babba4468d9f9bf: Status 404 returned error can't find the container with id 5026676d23d4f07e5ad43ae226717e05074b81f6ab1d07981babba4468d9f9bf Jul 23 01:29:39.137911 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.137883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f4b4cb66-wbjnn" event={"ID":"849fc9bd-2902-4e99-b6e3-e1206bedff07","Type":"ContainerStarted","Data":"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d"} Jul 23 01:29:39.139069 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.139043 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" event={"ID":"20020c4c-bfb4-404b-9efe-fbd15bed17e8","Type":"ContainerStarted","Data":"f94c4528ce12e55af3483ef09b9c8a8b5f4119cb4f3d57a1acc7a68f39df9e85"} Jul 23 01:29:39.140288 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.140269 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrvrm" event={"ID":"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023","Type":"ContainerStarted","Data":"8a2e787109d6d062e0a32c413313cddc99a5ed4e212f027e4535363e9143f1ee"} Jul 23 01:29:39.181308 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.181256 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f4b4cb66-wbjnn" podStartSLOduration=3.166484946 podStartE2EDuration="9.18123838s" totalImagesPullingTime="6.014753434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:39.180576662 +0000 UTC m=+48.960100592" watchObservedRunningTime="2026-07-23 01:29:39.18123838 +0000 UTC m=+48.960762311" Jul 23 01:29:39.212374 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.212346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:39.441504 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.441402 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665f54d6bd-vwc8s"] Jul 23 01:29:39.452407 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:39.452092 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6037316_e72e_4649_8ca2_d156f1964464.slice/crio-f51b6765ea3c55d92af52231becfe320811269be7a83166c329a88ade0bbd404 WatchSource:0}: Error finding container f51b6765ea3c55d92af52231becfe320811269be7a83166c329a88ade0bbd404: Status 404 returned error can't find the container with id f51b6765ea3c55d92af52231becfe320811269be7a83166c329a88ade0bbd404 Jul 23 01:29:39.664057 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.663984 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:29:39.682831 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.682802 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:29:39.682944 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.682877 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.691352 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.691333 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 01:29:39.692878 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.692858 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.692932 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.692891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.692932 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.692919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.692991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.692950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.693020 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.692990 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6fx\" (UniqueName: \"kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.693052 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.693021 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.693113 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.693097 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.793759 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793720 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.793923 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793794 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.793923 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.793923 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.793923 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793908 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.794138 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6fx\" (UniqueName: \"kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.794138 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.793983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.795517 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.795484 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.795863 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.795841 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.797217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.796595 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.797217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.796700 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.797217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.796714 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.800554 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.800529 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.807816 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.807769 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6fx\" (UniqueName: \"kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx\") pod \"console-7656b87bdb-zjj9s\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:39.992473 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:39.992381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:40.146918 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.146870 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"5026676d23d4f07e5ad43ae226717e05074b81f6ab1d07981babba4468d9f9bf"} Jul 23 01:29:40.150364 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.150337 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" event={"ID":"730ed4ae-093f-444e-92ee-1e8c65dfe798","Type":"ContainerStarted","Data":"08b056df9c5db13b153d140631ef42d002983ebf453abe8b8db2786400fe7eaa"} Jul 23 01:29:40.150502 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.150393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" event={"ID":"730ed4ae-093f-444e-92ee-1e8c65dfe798","Type":"ContainerStarted","Data":"9745817b902303763b4fcef76679b58226ba5109c11beaeaf6fcdf41a09e34a9"} Jul 23 01:29:40.150502 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.150414 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" event={"ID":"730ed4ae-093f-444e-92ee-1e8c65dfe798","Type":"ContainerStarted","Data":"145a0fe961529ab32cb03a83c3cf53e979f25990f531a9a555860d013d63380b"} Jul 23 01:29:40.152772 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.152742 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhd9" event={"ID":"ca1c65a8-d97e-4c96-8910-bf8bd928063e","Type":"ContainerStarted","Data":"edb5bb59fa7c12e859c508a2b8859c273d03e3399697c7c3f6bd81fc98a94ded"} Jul 23 01:29:40.154588 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.154535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"f51b6765ea3c55d92af52231becfe320811269be7a83166c329a88ade0bbd404"} Jul 23 01:29:40.195801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:40.195657 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:29:40.555286 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:40.555246 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012c7c9a_5417_4044_a94a_03c39f70f056.slice/crio-b7b7aa180aeedfbfed87fc371d2d2f1d67ebafdfdfe6f8921a444081539a84a7 WatchSource:0}: Error finding container b7b7aa180aeedfbfed87fc371d2d2f1d67ebafdfdfe6f8921a444081539a84a7: Status 404 returned error can't find the container with id b7b7aa180aeedfbfed87fc371d2d2f1d67ebafdfdfe6f8921a444081539a84a7 Jul 23 01:29:41.104073 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.103978 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:41.104073 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.104069 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:41.110801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.110770 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:41.160293 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.160254 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7656b87bdb-zjj9s" event={"ID":"012c7c9a-5417-4044-a94a-03c39f70f056","Type":"ContainerStarted","Data":"b7b7aa180aeedfbfed87fc371d2d2f1d67ebafdfdfe6f8921a444081539a84a7"} Jul 23 01:29:41.166026 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.165998 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:29:41.213647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.213618 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c5db6549d-cgvbw"] Jul 23 01:29:41.227043 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.227018 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5db6549d-cgvbw"] Jul 23 01:29:41.227196 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.227125 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.229763 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.229736 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 01:29:41.229888 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.229859 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 01:29:41.230013 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.229994 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4ws8g\"" Jul 23 01:29:41.230252 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.230083 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 01:29:41.230535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.230514 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1tlbmrad64cd7\"" Jul 23 01:29:41.230646 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.230592 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 01:29:41.309621 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309585 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.309801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25f4676-44eb-492e-8fef-de08d5bac273-audit-log\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.309801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309678 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-client-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.309801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309735 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44dn\" (UniqueName: \"kubernetes.io/projected/c25f4676-44eb-492e-8fef-de08d5bac273-kube-api-access-c44dn\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.309970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309800 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-client-certs\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.309970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309926 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-metrics-server-audit-profiles\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.310067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.309980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-tls\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.410780 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.410704 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-client-certs\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.410946 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.410877 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-metrics-server-audit-profiles\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.410946 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.410929 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-tls\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.410968 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.411009 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25f4676-44eb-492e-8fef-de08d5bac273-audit-log\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.411032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-client-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411253 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.411072 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44dn\" (UniqueName: \"kubernetes.io/projected/c25f4676-44eb-492e-8fef-de08d5bac273-kube-api-access-c44dn\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411858 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.411787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25f4676-44eb-492e-8fef-de08d5bac273-audit-log\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.411965 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.411913 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.412445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.412396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25f4676-44eb-492e-8fef-de08d5bac273-metrics-server-audit-profiles\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.415223 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.414439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-client-ca-bundle\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.415223 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.414474 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-tls\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.415223 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.415119 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25f4676-44eb-492e-8fef-de08d5bac273-secret-metrics-server-client-certs\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.420463 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.420418 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44dn\" (UniqueName: \"kubernetes.io/projected/c25f4676-44eb-492e-8fef-de08d5bac273-kube-api-access-c44dn\") pod \"metrics-server-6c5db6549d-cgvbw\" (UID: \"c25f4676-44eb-492e-8fef-de08d5bac273\") " pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.540508 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.540470 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:29:41.680472 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.680398 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh"] Jul 23 01:29:41.688815 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.688776 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:41.693741 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.690755 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh"] Jul 23 01:29:41.694856 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.694011 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 23 01:29:41.694856 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.694287 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fczt7\"" Jul 23 01:29:41.713773 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.713586 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:41.713773 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.713705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qnk\" (UniqueName: \"kubernetes.io/projected/91cfa4ae-e721-43d8-974a-c645889858a0-kube-api-access-v9qnk\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:41.815095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.815061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:41.815322 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.815167 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qnk\" (UniqueName: \"kubernetes.io/projected/91cfa4ae-e721-43d8-974a-c645889858a0-kube-api-access-v9qnk\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:41.815824 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:41.815524 2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 23 01:29:41.815824 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:29:41.815605 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert podName:91cfa4ae-e721-43d8-974a-c645889858a0 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:42.315586167 +0000 UTC m=+52.095110075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-rpbdh" (UID: "91cfa4ae-e721-43d8-974a-c645889858a0") : secret "monitoring-plugin-cert" not found Jul 23 01:29:41.825117 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:41.825084 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qnk\" (UniqueName: \"kubernetes.io/projected/91cfa4ae-e721-43d8-974a-c645889858a0-kube-api-access-v9qnk\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:42.125288 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:42.125259 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5tgvg" Jul 23 01:29:42.320527 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:42.320495 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:42.322832 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:42.322786 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91cfa4ae-e721-43d8-974a-c645889858a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rpbdh\" (UID: \"91cfa4ae-e721-43d8-974a-c645889858a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:42.608002 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:42.607968 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:43.140671 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.140631 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:29:43.222960 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.222931 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:29:43.223149 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.223045 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.226041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.226015 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 01:29:43.226206 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.226083 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9d6lc\"" Jul 23 01:29:43.226352 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.226335 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 01:29:43.227116 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227087 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 01:29:43.227223 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227207 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 01:29:43.227426 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227339 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 01:29:43.227426 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227379 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 01:29:43.227630 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227615 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 01:29:43.227630 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227621 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 01:29:43.227768 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227646 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 01:29:43.227768 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227617 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 01:29:43.227884 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227791 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 01:29:43.227938 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.227901 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 01:29:43.228021 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.228000 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1jj53rfahqf96\"" Jul 23 01:29:43.231105 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.231083 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 01:29:43.235221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.235204 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 01:29:43.329595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329551 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.329789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329653 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.329789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.329789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.329789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329747 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.329789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329773 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtvd\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329869 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329911 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329949 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.329987 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330021 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330076 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330115 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330146 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330184 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330208 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.330485 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.330268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432086 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432163 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432328 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432376 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtvd\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432486 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432507 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432547 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432571 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.432992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.432958 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433015 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433076 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433113 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433168 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433209 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433241 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433266 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433281 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433299 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.433599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.434060 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.433921 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.434875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.434479 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.435811 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.435462 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.437347 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.437324 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.437713 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.437684 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.437992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.437965 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.438289 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.438264 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.438767 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.438742 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.438767 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.438748 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.439791 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.439728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.440522 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.440500 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.440600 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.440562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.440600 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.440590 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.441267 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.441242 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.441832 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.441808 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.453718 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.453694 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtvd\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd\") pod \"prometheus-k8s-0\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:43.537446 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:43.537410 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:44.890364 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:44.887879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:29:45.109838 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.109767 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh"] Jul 23 01:29:45.113037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.113008 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5db6549d-cgvbw"] Jul 23 01:29:45.136306 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:45.136273 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25f4676_44eb_492e_8fef_de08d5bac273.slice/crio-2d1bda5159eb60a23f0cc2bc63cda887c22c9224fac78ee5b84c4c52d979b915 WatchSource:0}: Error finding container 2d1bda5159eb60a23f0cc2bc63cda887c22c9224fac78ee5b84c4c52d979b915: Status 404 returned error can't find the container with id 2d1bda5159eb60a23f0cc2bc63cda887c22c9224fac78ee5b84c4c52d979b915 Jul 23 01:29:45.136600 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:45.136568 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cfa4ae_e721_43d8_974a_c645889858a0.slice/crio-65c393980ba19f78159c7a12b4128c2ea4c9e626a13626651710f4d0a97683ab WatchSource:0}: Error finding container 65c393980ba19f78159c7a12b4128c2ea4c9e626a13626651710f4d0a97683ab: Status 404 returned error can't find the container with id 65c393980ba19f78159c7a12b4128c2ea4c9e626a13626651710f4d0a97683ab Jul 23 01:29:45.179929 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.179897 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" event={"ID":"20020c4c-bfb4-404b-9efe-fbd15bed17e8","Type":"ContainerStarted","Data":"86bca01c6a927226cfa0f4cba511642728c852993f5d3b740a5cd53c7350ccb2"} Jul 23 01:29:45.180074 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.179969 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" event={"ID":"20020c4c-bfb4-404b-9efe-fbd15bed17e8","Type":"ContainerStarted","Data":"0d368d553da4e9b24da8d7d970e6ff7f72f9615c83438abd7eddcab5e76dac98"} Jul 23 01:29:45.180074 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.180000 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" event={"ID":"20020c4c-bfb4-404b-9efe-fbd15bed17e8","Type":"ContainerStarted","Data":"8025f5217ac2e71079585939455b1b9a437dd691170b93dd75225fdd5587c42c"} Jul 23 01:29:45.181992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.181967 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142" exitCode=0 Jul 23 01:29:45.182119 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.182048 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142"} Jul 23 01:29:45.183776 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.183680 2554 generic.go:356] "Generic (PLEG): container finished" podID="da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023" containerID="344613d0ebcc886f93e4537d0dcf7ea63a691ec3b25c0a582cbc5722ca628405" exitCode=0 Jul 23 01:29:45.183776 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.183706 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrvrm" event={"ID":"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023","Type":"ContainerDied","Data":"344613d0ebcc886f93e4537d0dcf7ea63a691ec3b25c0a582cbc5722ca628405"} Jul 23 01:29:45.186136 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.186114 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" event={"ID":"730ed4ae-093f-444e-92ee-1e8c65dfe798","Type":"ContainerStarted","Data":"95cfc62fb02d0c06ccc8c1101c087b95bb29f202f06ef59a90485dcb4f35f243"} Jul 23 01:29:45.188442 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.188301 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7656b87bdb-zjj9s" event={"ID":"012c7c9a-5417-4044-a94a-03c39f70f056","Type":"ContainerStarted","Data":"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497"} Jul 23 01:29:45.189991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.189969 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" exitCode=0 Jul 23 01:29:45.190088 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.190005 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} Jul 23 01:29:45.190088 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.190072 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"a8f02c1208ac9ffe7d9b146025859313de842f3bbe846f416dfbed3ccd668d5d"} Jul 23 01:29:45.191495 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.191442 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" event={"ID":"91cfa4ae-e721-43d8-974a-c645889858a0","Type":"ContainerStarted","Data":"65c393980ba19f78159c7a12b4128c2ea4c9e626a13626651710f4d0a97683ab"} Jul 23 01:29:45.199857 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.199809 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg5h" podStartSLOduration=3.667396955 podStartE2EDuration="9.199792361s" totalImagesPullingTime="5.532395406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:45.199365143 +0000 UTC m=+54.978889073" watchObservedRunningTime="2026-07-23 01:29:45.199792361 +0000 UTC m=+54.979316291" Jul 23 01:29:45.200145 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.200119 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhd9" event={"ID":"ca1c65a8-d97e-4c96-8910-bf8bd928063e","Type":"ContainerStarted","Data":"47ecd3af643889612c9f0f01a1e16f6e3e02888ccd07975ebeda475a7043127c"} Jul 23 01:29:45.202652 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.202629 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"73f46a1706db8a51644b189f79e2bf17c5730fcba2ba9827f50af4bd6354cf93"} Jul 23 01:29:45.202751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.202696 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"05d7c67812195b9b6d3d22b0ecf7e9e082940bbaa33c742a31faca1b4d5be908"} Jul 23 01:29:45.202751 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.202739 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"11300de1e178982742a9b70965262e33d76535bbff6175c53e2a0d71182c7cce"} Jul 23 01:29:45.208220 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.206989 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" event={"ID":"c25f4676-44eb-492e-8fef-de08d5bac273","Type":"ContainerStarted","Data":"2d1bda5159eb60a23f0cc2bc63cda887c22c9224fac78ee5b84c4c52d979b915"} Jul 23 01:29:45.265651 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.265613 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7656b87bdb-zjj9s" podStartSLOduration=6.265599313 podStartE2EDuration="6.265599313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:45.265272115 +0000 UTC m=+55.044796043" watchObservedRunningTime="2026-07-23 01:29:45.265599313 +0000 UTC m=+55.045123241" Jul 23 01:29:45.304035 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.303976 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-t65hn" podStartSLOduration=4.175010093 podStartE2EDuration="9.303952689s" totalImagesPullingTime="5.128942596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:45.302363368 +0000 UTC m=+55.081887298" watchObservedRunningTime="2026-07-23 01:29:45.303952689 +0000 UTC m=+55.083476621" Jul 23 01:29:45.321040 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:45.320996 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkhd9" podStartSLOduration=2.305917979 podStartE2EDuration="20.320980685s" totalImagesPullingTime="18.015062706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:29:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:45.320429546 +0000 UTC m=+55.099953475" watchObservedRunningTime="2026-07-23 01:29:45.320980685 +0000 UTC m=+55.100504630" Jul 23 01:29:46.220723 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:46.220684 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrvrm" event={"ID":"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023","Type":"ContainerStarted","Data":"acc3a5bf8c6a3f3d3d7c6f880d21efa40941af02dda65e4893f56016ae1f14c9"} Jul 23 01:29:46.221280 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:46.220746 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrvrm" event={"ID":"da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023","Type":"ContainerStarted","Data":"ae037a02b3bd5ced84a29c3b603ce0f629323479f7eb360be769b36bf03e5ba2"} Jul 23 01:29:46.244076 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:46.244018 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wrvrm" podStartSLOduration=4.387082045 podStartE2EDuration="10.243993984s" totalImagesPullingTime="5.67687239s" totalInitContainerRuntime="180.039549ms" isStatefulPod=true podCreationTimestamp="2026-07-23 01:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:46.242800751 +0000 UTC m=+56.022324703" watchObservedRunningTime="2026-07-23 01:29:46.243993984 +0000 UTC m=+56.023517915" Jul 23 01:29:46.672491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:46.672452 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:29:49.235208 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.235102 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f"} Jul 23 01:29:49.236616 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.236588 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" event={"ID":"91cfa4ae-e721-43d8-974a-c645889858a0","Type":"ContainerStarted","Data":"236a5d08b11f39c45521927ef846b7a1dc7a223c2299062b87e4d51dfa41ea1b"} Jul 23 01:29:49.236820 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.236800 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:49.239266 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.239242 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"085f8526ff869aec47e1060d2e56b5a4afcc20813cfb6d179952270fb2bab2ff"} Jul 23 01:29:49.240910 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.240883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" event={"ID":"c25f4676-44eb-492e-8fef-de08d5bac273","Type":"ContainerStarted","Data":"109d600feea27f0ee8c097230eb358cb656343cd42d2f5215e3958f9d1cbceff"} Jul 23 01:29:49.242294 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.242275 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" Jul 23 01:29:49.252439 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.252403 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rpbdh" podStartSLOduration=4.457690489 podStartE2EDuration="8.252390094s" totalImagesPullingTime="3.794699605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:49.251536831 +0000 UTC m=+59.031060761" watchObservedRunningTime="2026-07-23 01:29:49.252390094 +0000 UTC m=+59.031914022" Jul 23 01:29:49.269580 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.269539 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" podStartSLOduration=4.473231537 podStartE2EDuration="8.269524496s" totalImagesPullingTime="3.796292959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:49.267543502 +0000 UTC m=+59.047067433" watchObservedRunningTime="2026-07-23 01:29:49.269524496 +0000 UTC m=+59.049048425" Jul 23 01:29:49.992471 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:49.992445 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:29:50.248885 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.248788 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e"} Jul 23 01:29:50.248885 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.248862 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd"} Jul 23 01:29:50.249337 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.248901 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11"} Jul 23 01:29:50.249337 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.248941 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0"} Jul 23 01:29:50.249337 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.248970 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerStarted","Data":"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29"} Jul 23 01:29:50.251776 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.251747 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} Jul 23 01:29:50.251906 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.251865 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} Jul 23 01:29:50.251906 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.251900 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} Jul 23 01:29:50.251985 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.251930 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} Jul 23 01:29:50.251985 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.251967 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} Jul 23 01:29:50.252056 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.252001 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerStarted","Data":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} Jul 23 01:29:50.254862 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.254823 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"05022c610a1840e60fd956e7cb6a5306530dab50327f09161b025c536f40a8d8"} Jul 23 01:29:50.254967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.254892 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" event={"ID":"b6037316-e72e-4649-8ca2-d156f1964464","Type":"ContainerStarted","Data":"268a01e5c6a0052b8f0a1909ae43c5636b0df4d2f5f98c318c51fafc65b8d9d2"} Jul 23 01:29:50.274817 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.274762 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.000255519 podStartE2EDuration="13.274749538s" totalImagesPullingTime="9.261634856s" totalInitContainerRuntime="12.859163ms" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:50.273204828 +0000 UTC m=+60.052728757" watchObservedRunningTime="2026-07-23 01:29:50.274749538 +0000 UTC m=+60.054273467" Jul 23 01:29:50.295492 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.295431 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" podStartSLOduration=5.777311838 podStartE2EDuration="12.295415433s" totalImagesPullingTime="6.518103595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:50.295057597 +0000 UTC m=+60.074581541" watchObservedRunningTime="2026-07-23 01:29:50.295415433 +0000 UTC m=+60.074939361" Jul 23 01:29:50.325091 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:50.325031 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.167096064 podStartE2EDuration="7.324966153s" totalImagesPullingTime="4.131882237s" totalInitContainerRuntime="25.987852ms" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:50.323241096 +0000 UTC m=+60.102765026" watchObservedRunningTime="2026-07-23 01:29:50.324966153 +0000 UTC m=+60.104490084" Jul 23 01:29:51.042496 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:51.042459 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wsn" Jul 23 01:29:51.259723 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:51.259691 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:51.264978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:51.264958 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-665f54d6bd-vwc8s" Jul 23 01:29:53.538113 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:53.538080 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:29:55.357651 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.357617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:55.360089 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.360068 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 01:29:55.370296 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.370277 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44e022b-202a-48e4-8c9f-23ef0751decf-metrics-certs\") pod \"network-metrics-daemon-rbvvn\" (UID: \"c44e022b-202a-48e4-8c9f-23ef0751decf\") " pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:55.458927 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.458895 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:55.459082 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.458940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:55.462045 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.462027 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 01:29:55.462089 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.462043 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 01:29:55.471445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.471411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e-original-pull-secret\") pod \"global-pull-secret-syncer-6qtsd\" (UID: \"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e\") " pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:55.471898 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.471883 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 01:29:55.473912 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.473889 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qtsd" Jul 23 01:29:55.482384 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.482366 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqwz\" (UniqueName: \"kubernetes.io/projected/92e1ed67-e3ad-422a-bbd0-8aab66787164-kube-api-access-bkqwz\") pod \"network-check-target-kdgzl\" (UID: \"92e1ed67-e3ad-422a-bbd0-8aab66787164\") " pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:55.483287 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.483272 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frx6l\"" Jul 23 01:29:55.489586 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.489559 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lg5p2\"" Jul 23 01:29:55.491619 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.491599 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbvvn" Jul 23 01:29:55.498101 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.498078 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:29:55.621579 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.621544 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qtsd"] Jul 23 01:29:55.625452 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:55.625410 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe3efc5_d6e1_44e4_9f63_ff1cff1e4e4e.slice/crio-77d13ba70d6cf236ad347c2a3d962ab2ff7bf144e262caa62e2ec67a0509841c WatchSource:0}: Error finding container 77d13ba70d6cf236ad347c2a3d962ab2ff7bf144e262caa62e2ec67a0509841c: Status 404 returned error can't find the container with id 77d13ba70d6cf236ad347c2a3d962ab2ff7bf144e262caa62e2ec67a0509841c Jul 23 01:29:55.651049 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.647694 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbvvn"] Jul 23 01:29:55.664731 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:55.664705 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdgzl"] Jul 23 01:29:55.667364 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:29:55.667343 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e1ed67_e3ad_422a_bbd0_8aab66787164.slice/crio-98df84cd009e228c832901af9406eb3644edeba7da43cb1b2ff8095e79d50140 WatchSource:0}: Error finding container 98df84cd009e228c832901af9406eb3644edeba7da43cb1b2ff8095e79d50140: Status 404 returned error can't find the container with id 98df84cd009e228c832901af9406eb3644edeba7da43cb1b2ff8095e79d50140 Jul 23 01:29:56.282295 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:56.282254 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdgzl" event={"ID":"92e1ed67-e3ad-422a-bbd0-8aab66787164","Type":"ContainerStarted","Data":"98df84cd009e228c832901af9406eb3644edeba7da43cb1b2ff8095e79d50140"} Jul 23 01:29:56.284509 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:56.284472 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbvvn" event={"ID":"c44e022b-202a-48e4-8c9f-23ef0751decf","Type":"ContainerStarted","Data":"d47a5901641bc5791a6e3184ce13ebf7b2c64e665f1e837e989fbb2accec9d2f"} Jul 23 01:29:56.286768 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:56.286715 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qtsd" event={"ID":"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e","Type":"ContainerStarted","Data":"77d13ba70d6cf236ad347c2a3d962ab2ff7bf144e262caa62e2ec67a0509841c"} Jul 23 01:29:57.294882 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:57.294825 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbvvn" event={"ID":"c44e022b-202a-48e4-8c9f-23ef0751decf","Type":"ContainerStarted","Data":"5306a9bce94cfb9bc7388edb6a1161488f269d1617637ca26aaa62401f84baff"} Jul 23 01:29:57.295366 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:57.294902 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbvvn" event={"ID":"c44e022b-202a-48e4-8c9f-23ef0751decf","Type":"ContainerStarted","Data":"f1b2cc90a3e5500676d72d7fc7e5808a1b3053bb749e21c6c421f3f08e6e2811"} Jul 23 01:29:57.311888 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:57.311829 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rbvvn" podStartSLOduration=65.205957013 podStartE2EDuration="1m6.311807789s" totalImagesPullingTime="1.105850776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:57.310155491 +0000 UTC m=+67.089679420" watchObservedRunningTime="2026-07-23 01:29:57.311807789 +0000 UTC m=+67.091331717" Jul 23 01:29:57.570604 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:29:57.570569 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:30:01.315937 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.315894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qtsd" event={"ID":"3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e","Type":"ContainerStarted","Data":"e9912ed1950e1dcf4e9fff95bf67ac60e7b8cc4a0ceeae5cfa9cf80ddc420f7d"} Jul 23 01:30:01.317498 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.317467 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdgzl" event={"ID":"92e1ed67-e3ad-422a-bbd0-8aab66787164","Type":"ContainerStarted","Data":"59c048f7eb90f4563fce53a678e6e1268adc72ed302b7e56511922633f55c623"} Jul 23 01:30:01.317596 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.317568 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:30:01.331267 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.331226 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6qtsd" podStartSLOduration=65.236060165 podStartE2EDuration="1m10.331213947s" totalImagesPullingTime="5.095153782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:01.329832459 +0000 UTC m=+71.109356385" watchObservedRunningTime="2026-07-23 01:30:01.331213947 +0000 UTC m=+71.110737934" Jul 23 01:30:01.344810 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.344768 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kdgzl" podStartSLOduration=66.302600428 podStartE2EDuration="1m11.344753698s" totalImagesPullingTime="5.04215327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:01.344208443 +0000 UTC m=+71.123732373" watchObservedRunningTime="2026-07-23 01:30:01.344753698 +0000 UTC m=+71.124277627" Jul 23 01:30:01.541090 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.541059 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:30:01.541284 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:01.541140 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:30:12.253669 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.253604 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7656b87bdb-zjj9s" podUID="012c7c9a-5417-4044-a94a-03c39f70f056" containerName="console" containerID="cri-o://5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497" gracePeriod=15 Jul 23 01:30:12.530516 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.530449 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:30:12.613866 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.613834 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.613874 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr6fx\" (UniqueName: \"kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.613925 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.613941 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.613967 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614016 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614399 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614099 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca\") pod \"012c7c9a-5417-4044-a94a-03c39f70f056\" (UID: \"012c7c9a-5417-4044-a94a-03c39f70f056\") " Jul 23 01:30:12.614460 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614420 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config" (OuterVolumeSpecName: "console-config") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:12.614460 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614429 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:12.614636 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614595 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:12.614714 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.614696 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca" (OuterVolumeSpecName: "service-ca") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:12.616266 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.616233 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx" (OuterVolumeSpecName: "kube-api-access-mr6fx") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "kube-api-access-mr6fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:12.616385 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.616289 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:12.616385 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.616351 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "012c7c9a-5417-4044-a94a-03c39f70f056" (UID: "012c7c9a-5417-4044-a94a-03c39f70f056"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:12.715610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715575 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-oauth-serving-cert\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715603 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-trusted-ca-bundle\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715614 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-service-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715610 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715623 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-oauth-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715851 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715631 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6fx\" (UniqueName: \"kubernetes.io/projected/012c7c9a-5417-4044-a94a-03c39f70f056-kube-api-access-mr6fx\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715851 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715640 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/012c7c9a-5417-4044-a94a-03c39f70f056-console-serving-cert\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:12.715851 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:12.715649 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/012c7c9a-5417-4044-a94a-03c39f70f056-console-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:13.366515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.366479 2554 generic.go:356] "Generic (PLEG): container finished" podID="012c7c9a-5417-4044-a94a-03c39f70f056" containerID="5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497" exitCode=0 Jul 23 01:30:13.367018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.366530 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7656b87bdb-zjj9s" Jul 23 01:30:13.367018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.366567 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7656b87bdb-zjj9s" event={"ID":"012c7c9a-5417-4044-a94a-03c39f70f056","Type":"ContainerDied","Data":"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497"} Jul 23 01:30:13.367018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.366635 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7656b87bdb-zjj9s" event={"ID":"012c7c9a-5417-4044-a94a-03c39f70f056","Type":"ContainerDied","Data":"b7b7aa180aeedfbfed87fc371d2d2f1d67ebafdfdfe6f8921a444081539a84a7"} Jul 23 01:30:13.367018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.366658 2554 scope.go:117] "RemoveContainer" containerID="5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497" Jul 23 01:30:13.375698 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.375682 2554 scope.go:117] "RemoveContainer" containerID="5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497" Jul 23 01:30:13.375952 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:13.375934 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497\": container with ID starting with 5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497 not found: ID does not exist" containerID="5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497" Jul 23 01:30:13.376018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.375966 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497"} err="failed to get container status \"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497\": rpc error: code = NotFound desc = could not find container \"5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497\": container with ID starting with 5d9e7400da2cc6c2b2c78bd83603bfc222f2c3783063b4d96c2d398dd0234497 not found: ID does not exist" Jul 23 01:30:13.381518 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.381487 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:30:13.387940 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:13.387920 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7656b87bdb-zjj9s"] Jul 23 01:30:14.768793 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:14.768754 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012c7c9a-5417-4044-a94a-03c39f70f056" path="/var/lib/kubelet/pods/012c7c9a-5417-4044-a94a-03c39f70f056/volumes" Jul 23 01:30:21.546219 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:21.546168 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:30:21.549967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:21.549947 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c5db6549d-cgvbw" Jul 23 01:30:22.600219 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:22.600144 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f4b4cb66-wbjnn" podUID="849fc9bd-2902-4e99-b6e3-e1206bedff07" containerName="console" containerID="cri-o://ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d" gracePeriod=15 Jul 23 01:30:23.375451 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.375417 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:30:23.405703 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405679 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.405856 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405759 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq762\" (UniqueName: \"kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.405856 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405795 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.405970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405878 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.405970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405908 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.405970 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.405934 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config\") pod \"849fc9bd-2902-4e99-b6e3-e1206bedff07\" (UID: \"849fc9bd-2902-4e99-b6e3-e1206bedff07\") " Jul 23 01:30:23.406547 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.406426 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:23.406547 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.406447 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config" (OuterVolumeSpecName: "console-config") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:23.406733 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.406600 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca" (OuterVolumeSpecName: "service-ca") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:23.408117 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.408089 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:23.408482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.408456 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:23.408482 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.408467 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762" (OuterVolumeSpecName: "kube-api-access-sq762") pod "849fc9bd-2902-4e99-b6e3-e1206bedff07" (UID: "849fc9bd-2902-4e99-b6e3-e1206bedff07"). InnerVolumeSpecName "kube-api-access-sq762". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:23.410661 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.410633 2554 generic.go:356] "Generic (PLEG): container finished" podID="849fc9bd-2902-4e99-b6e3-e1206bedff07" containerID="ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d" exitCode=0 Jul 23 01:30:23.410771 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.410674 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f4b4cb66-wbjnn" Jul 23 01:30:23.410771 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.410717 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f4b4cb66-wbjnn" event={"ID":"849fc9bd-2902-4e99-b6e3-e1206bedff07","Type":"ContainerDied","Data":"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d"} Jul 23 01:30:23.410869 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.410775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f4b4cb66-wbjnn" event={"ID":"849fc9bd-2902-4e99-b6e3-e1206bedff07","Type":"ContainerDied","Data":"3e9a0e0aa53b6c3efbb032c8af54c03196e71549174043db0ed527eb0fd4cab0"} Jul 23 01:30:23.410869 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.410804 2554 scope.go:117] "RemoveContainer" containerID="ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d" Jul 23 01:30:23.423804 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.423787 2554 scope.go:117] "RemoveContainer" containerID="ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d" Jul 23 01:30:23.424056 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:23.424036 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d\": container with ID starting with ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d not found: ID does not exist" containerID="ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d" Jul 23 01:30:23.424124 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.424064 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d"} err="failed to get container status \"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d\": rpc error: code = NotFound desc = could not find container \"ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d\": container with ID starting with ac84863149295fa7d847c7b2e72df72768b7935c431dfdbdf7bab2a680984c0d not found: ID does not exist" Jul 23 01:30:23.433772 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.433752 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:30:23.435839 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.435820 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f4b4cb66-wbjnn"] Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507347 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-oauth-serving-cert\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507377 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-service-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507389 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507397 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-serving-cert\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507405 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq762\" (UniqueName: \"kubernetes.io/projected/849fc9bd-2902-4e99-b6e3-e1206bedff07-kube-api-access-sq762\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:23.507421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:23.507413 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/849fc9bd-2902-4e99-b6e3-e1206bedff07-console-oauth-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:24.768770 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:24.768733 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849fc9bd-2902-4e99-b6e3-e1206bedff07" path="/var/lib/kubelet/pods/849fc9bd-2902-4e99-b6e3-e1206bedff07/volumes" Jul 23 01:30:32.324795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:32.324755 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kdgzl" Jul 23 01:30:34.097078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:34.097022 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:34.116503 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:34.116472 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:34.480249 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:34.480161 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:47.227572 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.227536 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:47.228078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.227846 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="alertmanager" containerID="cri-o://8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f" gracePeriod=120 Jul 23 01:30:47.228078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.227920 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-metric" containerID="cri-o://a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd" gracePeriod=120 Jul 23 01:30:47.228078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.228000 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy" containerID="cri-o://260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11" gracePeriod=120 Jul 23 01:30:47.228078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.227995 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="config-reloader" containerID="cri-o://7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29" gracePeriod=120 Jul 23 01:30:47.228078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.227944 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="prom-label-proxy" containerID="cri-o://a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e" gracePeriod=120 Jul 23 01:30:47.228369 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.228000 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-web" containerID="cri-o://f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0" gracePeriod=120 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520596 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e" exitCode=0 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520623 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd" exitCode=0 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520631 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11" exitCode=0 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520637 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29" exitCode=0 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520642 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f" exitCode=0 Jul 23 01:30:47.520682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520667 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e"} Jul 23 01:30:47.520978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520731 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd"} Jul 23 01:30:47.520978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520759 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11"} Jul 23 01:30:47.520978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520783 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29"} Jul 23 01:30:47.520978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:47.520806 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f"} Jul 23 01:30:48.475148 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.475121 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:48.528811 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.528777 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerID="f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0" exitCode=0 Jul 23 01:30:48.528978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.528851 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0"} Jul 23 01:30:48.528978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.528916 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae276df8-8747-4a12-bf0d-b980fab6696d","Type":"ContainerDied","Data":"5026676d23d4f07e5ad43ae226717e05074b81f6ab1d07981babba4468d9f9bf"} Jul 23 01:30:48.528978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.528861 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:48.528978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.528952 2554 scope.go:117] "RemoveContainer" containerID="a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e" Jul 23 01:30:48.541298 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.541272 2554 scope.go:117] "RemoveContainer" containerID="a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd" Jul 23 01:30:48.548966 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.548944 2554 scope.go:117] "RemoveContainer" containerID="260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11" Jul 23 01:30:48.556565 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.556541 2554 scope.go:117] "RemoveContainer" containerID="f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0" Jul 23 01:30:48.563455 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.563432 2554 scope.go:117] "RemoveContainer" containerID="7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29" Jul 23 01:30:48.570436 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.570418 2554 scope.go:117] "RemoveContainer" containerID="8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f" Jul 23 01:30:48.577562 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.577543 2554 scope.go:117] "RemoveContainer" containerID="3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142" Jul 23 01:30:48.584476 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.584459 2554 scope.go:117] "RemoveContainer" containerID="a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e" Jul 23 01:30:48.584744 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.584726 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e\": container with ID starting with a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e not found: ID does not exist" containerID="a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e" Jul 23 01:30:48.584811 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.584755 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e"} err="failed to get container status \"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e\": rpc error: code = NotFound desc = could not find container \"a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e\": container with ID starting with a6e2271c27352b84b1c0595becced1cb5d38c0fdb3f741f4eb02a591b2c98d1e not found: ID does not exist" Jul 23 01:30:48.584811 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.584781 2554 scope.go:117] "RemoveContainer" containerID="a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd" Jul 23 01:30:48.585015 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.584993 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd\": container with ID starting with a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd not found: ID does not exist" containerID="a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd" Jul 23 01:30:48.585099 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585020 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd"} err="failed to get container status \"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd\": rpc error: code = NotFound desc = could not find container \"a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd\": container with ID starting with a17728c7f082e9fcc4c051cd8fd29a01602388c69c21e9d0c74ebbb70d5249cd not found: ID does not exist" Jul 23 01:30:48.585099 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585037 2554 scope.go:117] "RemoveContainer" containerID="260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11" Jul 23 01:30:48.585283 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.585269 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11\": container with ID starting with 260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11 not found: ID does not exist" containerID="260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11" Jul 23 01:30:48.585338 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585286 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11"} err="failed to get container status \"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11\": rpc error: code = NotFound desc = could not find container \"260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11\": container with ID starting with 260415129a93947258a2922b464bb6e663044683862e47040acb0a49692edb11 not found: ID does not exist" Jul 23 01:30:48.585338 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585299 2554 scope.go:117] "RemoveContainer" containerID="f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0" Jul 23 01:30:48.585524 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.585506 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0\": container with ID starting with f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0 not found: ID does not exist" containerID="f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0" Jul 23 01:30:48.585584 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585531 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0"} err="failed to get container status \"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0\": rpc error: code = NotFound desc = could not find container \"f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0\": container with ID starting with f4bf34f5aae2cad6dbd061a4e06139708eb3d3e0e5a0d424d7aa0da273555ff0 not found: ID does not exist" Jul 23 01:30:48.585584 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585552 2554 scope.go:117] "RemoveContainer" containerID="7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29" Jul 23 01:30:48.585793 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.585777 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29\": container with ID starting with 7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29 not found: ID does not exist" containerID="7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29" Jul 23 01:30:48.585843 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585798 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29"} err="failed to get container status \"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29\": rpc error: code = NotFound desc = could not find container \"7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29\": container with ID starting with 7446ff3b0f256e007e603fb8e0f3b704e55b4024a2580d456d10704503ce3a29 not found: ID does not exist" Jul 23 01:30:48.585843 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.585821 2554 scope.go:117] "RemoveContainer" containerID="8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f" Jul 23 01:30:48.586010 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.585994 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f\": container with ID starting with 8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f not found: ID does not exist" containerID="8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f" Jul 23 01:30:48.586072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.586016 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f"} err="failed to get container status \"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f\": rpc error: code = NotFound desc = could not find container \"8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f\": container with ID starting with 8ad7c3cf8e2947f3f58c1616d41f23e74ee0e9747dc09b92720384a8c49cab7f not found: ID does not exist" Jul 23 01:30:48.586072 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.586038 2554 scope.go:117] "RemoveContainer" containerID="3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142" Jul 23 01:30:48.586292 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:48.586275 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142\": container with ID starting with 3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142 not found: ID does not exist" containerID="3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142" Jul 23 01:30:48.586342 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.586298 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142"} err="failed to get container status \"3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142\": rpc error: code = NotFound desc = could not find container \"3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142\": container with ID starting with 3c2c43eb37339df9141fbf65e2ed624586cb2bc502030a2a8f588d0f22a33142 not found: ID does not exist" Jul 23 01:30:48.633299 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633266 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633321 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633347 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633381 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf476\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633401 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633417 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633465 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633502 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633536 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633563 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633616 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633645 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633675 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle\") pod \"ae276df8-8747-4a12-bf0d-b980fab6696d\" (UID: \"ae276df8-8747-4a12-bf0d-b980fab6696d\") " Jul 23 01:30:48.633777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.633740 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:48.634433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.634096 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-metrics-client-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.634433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.634162 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:48.634433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.634381 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:30:48.636583 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.636539 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.636583 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.636571 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476" (OuterVolumeSpecName: "kube-api-access-cf476") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "kube-api-access-cf476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:48.636583 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.636655 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume" (OuterVolumeSpecName: "config-volume") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.637085 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.637060 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:48.637648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.637612 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.637839 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.637810 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out" (OuterVolumeSpecName: "config-out") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:30:48.637992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.637953 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.638422 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.638396 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.641564 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.641429 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.648669 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.648635 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config" (OuterVolumeSpecName: "web-config") pod "ae276df8-8747-4a12-bf0d-b980fab6696d" (UID: "ae276df8-8747-4a12-bf0d-b980fab6696d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:48.735170 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735132 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735170 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735165 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-config-out\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735170 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735194 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-tls-assets\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735204 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-main-tls\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735213 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735222 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735231 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735240 2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-config-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735249 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf476\" (UniqueName: \"kubernetes.io/projected/ae276df8-8747-4a12-bf0d-b980fab6696d-kube-api-access-cf476\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735256 2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-cluster-tls-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735265 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae276df8-8747-4a12-bf0d-b980fab6696d-web-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.735414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.735272 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae276df8-8747-4a12-bf0d-b980fab6696d-alertmanager-main-db\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:48.849103 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.849056 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:48.854368 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.854321 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:48.881493 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881449 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:48.881771 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881759 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="alertmanager" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881774 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="alertmanager" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881782 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012c7c9a-5417-4044-a94a-03c39f70f056" containerName="console" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881788 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="012c7c9a-5417-4044-a94a-03c39f70f056" containerName="console" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881797 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-web" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881803 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-web" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881810 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-metric" Jul 23 01:30:48.881818 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881816 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-metric" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881825 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881830 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881843 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="config-reloader" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881848 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="config-reloader" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881856 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849fc9bd-2902-4e99-b6e3-e1206bedff07" containerName="console" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881861 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="849fc9bd-2902-4e99-b6e3-e1206bedff07" containerName="console" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881867 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="init-config-reloader" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881872 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="init-config-reloader" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881877 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="prom-label-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881882 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="prom-label-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881921 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="012c7c9a-5417-4044-a94a-03c39f70f056" containerName="console" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881929 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="config-reloader" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881936 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-web" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881942 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="prom-label-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881948 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="alertmanager" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881952 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881957 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="849fc9bd-2902-4e99-b6e3-e1206bedff07" containerName="console" Jul 23 01:30:48.882011 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.881963 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" containerName="kube-rbac-proxy-metric" Jul 23 01:30:48.886780 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.885873 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:48.888689 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888651 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 01:30:48.888822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888684 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 01:30:48.888822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888717 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 01:30:48.888822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888699 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 01:30:48.888822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888781 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 01:30:48.888822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888819 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 01:30:48.889061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.888893 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 01:30:48.889222 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.889204 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-shlfr\"" Jul 23 01:30:48.889540 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.889523 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 01:30:48.893677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.893654 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 01:30:48.898560 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:48.898540 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:49.038417 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038378 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038587 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038430 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-web-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038587 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038459 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038587 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038587 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038550 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-config-out\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vh2\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-kube-api-access-65vh2\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038707 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038722 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.038795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038754 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.039027 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.038815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-config-volume\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139546 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-config-volume\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139546 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139503 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139546 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139531 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-web-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139546 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139550 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139589 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139614 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139638 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-config-out\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vh2\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-kube-api-access-65vh2\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139760 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139786 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.139875 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.139826 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.140644 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.140611 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.140795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.140771 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/339b2ae1-782e-408e-8957-6eb5b8116b61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.141065 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.141043 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.142628 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.142595 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-web-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.142734 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.142630 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-config-volume\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.142843 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.142821 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.142843 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.142836 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.142979 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.142835 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.143230 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.143211 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.143307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.143235 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.143435 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.143417 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/339b2ae1-782e-408e-8957-6eb5b8116b61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.144381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.144363 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/339b2ae1-782e-408e-8957-6eb5b8116b61-config-out\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.149942 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.149923 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vh2\" (UniqueName: \"kubernetes.io/projected/339b2ae1-782e-408e-8957-6eb5b8116b61-kube-api-access-65vh2\") pod \"alertmanager-main-0\" (UID: \"339b2ae1-782e-408e-8957-6eb5b8116b61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.196027 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.195979 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 01:30:49.335617 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.335587 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 01:30:49.338528 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:30:49.338499 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339b2ae1_782e_408e_8957_6eb5b8116b61.slice/crio-bd65415955c4dcaa889a6bb5531571fc2445a94b6004e69d54ccbde7fec9941d WatchSource:0}: Error finding container bd65415955c4dcaa889a6bb5531571fc2445a94b6004e69d54ccbde7fec9941d: Status 404 returned error can't find the container with id bd65415955c4dcaa889a6bb5531571fc2445a94b6004e69d54ccbde7fec9941d Jul 23 01:30:49.536537 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.536502 2554 generic.go:356] "Generic (PLEG): container finished" podID="339b2ae1-782e-408e-8957-6eb5b8116b61" containerID="3e573773e85e855d09a93754393e15c04ee1dc6045fc8d35a64cf4a1872c442e" exitCode=0 Jul 23 01:30:49.536910 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.536571 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerDied","Data":"3e573773e85e855d09a93754393e15c04ee1dc6045fc8d35a64cf4a1872c442e"} Jul 23 01:30:49.536910 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:49.536625 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"bd65415955c4dcaa889a6bb5531571fc2445a94b6004e69d54ccbde7fec9941d"} Jul 23 01:30:50.548301 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"104afb08991f0510a676b16dde798d3117dd8b050a14c3214f5b54fe3c4682ae"} Jul 23 01:30:50.548648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548307 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"8d919ce64f0da205907fd770c62575656dbd3a2fe9494b94589cf07cae6222dc"} Jul 23 01:30:50.548648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548334 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"de367eda4e4b0d56c39e03b0832a8bb9c1347eafbe8de48aa79ebcc5575f2756"} Jul 23 01:30:50.548648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548359 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"d29b6865c120f12ec67861c325093e8bc8aa006e14eb888e6e6869077e7b9b17"} Jul 23 01:30:50.548648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548380 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"fa030c5e2656fc277d1d857abb9406369609f53a34955ee2b30c2c7f080f1028"} Jul 23 01:30:50.548648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.548403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"339b2ae1-782e-408e-8957-6eb5b8116b61","Type":"ContainerStarted","Data":"e9bbdcee199798782d40582bfda43d5fc02f2656f3b06c1df50828bd572707ad"} Jul 23 01:30:50.576838 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.576788 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.560778423 podStartE2EDuration="2.57677305s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.994627ms" isStatefulPod=false podCreationTimestamp="2026-07-23 01:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:50.575337872 +0000 UTC m=+120.354861819" watchObservedRunningTime="2026-07-23 01:30:50.57677305 +0000 UTC m=+120.356296977" Jul 23 01:30:50.771057 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:50.771021 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae276df8-8747-4a12-bf0d-b980fab6696d" path="/var/lib/kubelet/pods/ae276df8-8747-4a12-bf0d-b980fab6696d/volumes" Jul 23 01:30:51.220641 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.220606 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-765848f968-j4mrz"] Jul 23 01:30:51.225023 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.224996 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.227408 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.227381 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 01:30:51.227655 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.227636 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 01:30:51.227744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.227720 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 01:30:51.227911 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.227755 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 01:30:51.227911 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.227787 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s76nk\"" Jul 23 01:30:51.228655 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.228640 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 01:30:51.234593 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.234568 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 01:30:51.235353 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.235333 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-765848f968-j4mrz"] Jul 23 01:30:51.356259 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356218 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356270 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-metrics-client-ca\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356369 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-federate-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356393 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356668 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356448 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv9w\" (UniqueName: \"kubernetes.io/projected/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-kube-api-access-jzv9w\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356668 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.356668 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.356536 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-serving-certs-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457432 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457390 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457432 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-serving-certs-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457486 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-metrics-client-ca\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-federate-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457605 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.457699 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.457666 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv9w\" (UniqueName: \"kubernetes.io/projected/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-kube-api-access-jzv9w\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.458372 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.458340 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-metrics-client-ca\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.458846 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.458823 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-serving-certs-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.458947 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.458889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.460769 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.460718 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.460857 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.460785 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-federate-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.461458 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.461415 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-telemeter-client-tls\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.461930 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.461833 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.465428 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.465401 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv9w\" (UniqueName: \"kubernetes.io/projected/c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab-kube-api-access-jzv9w\") pod \"telemeter-client-765848f968-j4mrz\" (UID: \"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab\") " pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.537285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.537161 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" Jul 23 01:30:51.572136 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572083 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:51.572595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572473 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy" containerID="cri-o://c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" gracePeriod=600 Jul 23 01:30:51.572595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572524 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="config-reloader" containerID="cri-o://6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" gracePeriod=600 Jul 23 01:30:51.572595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572501 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="thanos-sidecar" containerID="cri-o://d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" gracePeriod=600 Jul 23 01:30:51.572759 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572528 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-web" containerID="cri-o://58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" gracePeriod=600 Jul 23 01:30:51.572759 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572570 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" gracePeriod=600 Jul 23 01:30:51.572759 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.572479 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="prometheus" containerID="cri-o://b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" gracePeriod=600 Jul 23 01:30:51.710747 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.710717 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-765848f968-j4mrz"] Jul 23 01:30:51.713974 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:30:51.713943 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f027f_c2bd_4c27_a7a0_2b6e6c80d3ab.slice/crio-07f594239114c0adaede763e2279d19b7ace14b33dad3c1b442f71f784b87477 WatchSource:0}: Error finding container 07f594239114c0adaede763e2279d19b7ace14b33dad3c1b442f71f784b87477: Status 404 returned error can't find the container with id 07f594239114c0adaede763e2279d19b7ace14b33dad3c1b442f71f784b87477 Jul 23 01:30:51.823973 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.823945 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:51.965021 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.964982 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965046 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965068 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965091 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965114 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965269 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965326 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965368 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965401 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965440 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965445 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965469 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965511 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965539 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965563 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965607 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965637 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965653 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965667 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965715 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.965742 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.965746 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qtvd\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd\") pod \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\" (UID: \"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40\") " Jul 23 01:30:51.966285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.966061 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:51.966285 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.966079 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:51.966878 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.966843 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.967203 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.967144 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.967203 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.967149 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.968434 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.968383 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.968517 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.968459 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.968630 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:30:51.969041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.968792 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969007 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:30:51.969248 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969091 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config" (OuterVolumeSpecName: "config") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969248 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969215 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969375 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969354 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969549 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969522 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.969549 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969553 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:51.970017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.969992 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.970091 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.970071 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:51.970273 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.970257 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd" (OuterVolumeSpecName: "kube-api-access-9qtvd") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "kube-api-access-9qtvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:30:51.970868 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.970853 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out" (OuterVolumeSpecName: "config-out") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:30:51.981076 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:51.981046 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config" (OuterVolumeSpecName: "web-config") pod "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" (UID: "8e1da72b-1f99-4aa3-bb9c-1561c2d38b40"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066825 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config-out\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066859 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066872 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066882 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-db\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066891 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-metrics-client-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066901 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtvd\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-kube-api-access-9qtvd\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066911 2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-metrics-client-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.066922 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066925 2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-grpc-tls\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066936 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066945 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-tls-assets\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066953 2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066961 2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-kube-rbac-proxy\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066969 2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066977 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-web-config\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066985 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.066994 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.067002 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.067307 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.067012 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:30:52.568749 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.568709 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" event={"ID":"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab","Type":"ContainerStarted","Data":"07f594239114c0adaede763e2279d19b7ace14b33dad3c1b442f71f784b87477"} Jul 23 01:30:52.572624 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572594 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" exitCode=0 Jul 23 01:30:52.572624 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572620 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" exitCode=0 Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572630 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" exitCode=0 Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572638 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" exitCode=0 Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572646 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" exitCode=0 Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572654 2554 generic.go:356] "Generic (PLEG): container finished" podID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" exitCode=0 Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572677 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572703 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572742 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572813 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572853 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} Jul 23 01:30:52.573014 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.572876 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e1da72b-1f99-4aa3-bb9c-1561c2d38b40","Type":"ContainerDied","Data":"a8f02c1208ac9ffe7d9b146025859313de842f3bbe846f416dfbed3ccd668d5d"} Jul 23 01:30:52.583061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.583041 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.591138 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.591097 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.599306 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.599285 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:52.599719 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.599703 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.604115 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.604086 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:52.608540 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.608516 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.616945 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.616904 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.625998 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.625978 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:52.626270 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626243 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.626439 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626423 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626442 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626457 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="config-reloader" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626466 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="config-reloader" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626481 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-thanos" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626491 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-thanos" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626506 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-web" Jul 23 01:30:52.626515 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626515 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-web" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626541 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="init-config-reloader" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626550 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="init-config-reloader" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626560 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="prometheus" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626569 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="prometheus" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626578 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="thanos-sidecar" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626586 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="thanos-sidecar" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626682 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="config-reloader" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626695 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-web" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626707 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626717 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="kube-rbac-proxy-thanos" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626728 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="thanos-sidecar" Jul 23 01:30:52.626834 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.626737 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" containerName="prometheus" Jul 23 01:30:52.630931 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.630901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.633433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.633411 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9d6lc\"" Jul 23 01:30:52.633571 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.633475 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 01:30:52.633571 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.633479 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 01:30:52.634096 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634019 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 01:30:52.634096 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634041 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 01:30:52.634096 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634072 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 01:30:52.634096 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634081 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 01:30:52.634581 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634021 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 01:30:52.634581 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634283 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 01:30:52.634581 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634296 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 01:30:52.634581 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634367 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1jj53rfahqf96\"" Jul 23 01:30:52.634581 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634554 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 01:30:52.635055 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634895 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 01:30:52.635055 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.634958 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 01:30:52.635441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.635416 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.635805 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.635840 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.635867 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.636130 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.636158 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.636253 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.636202 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.636675 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.636649 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.636745 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.636683 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.636745 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.636708 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.636994 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.636976 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.637087 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.637001 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.637087 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.637021 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.637342 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.637324 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 01:30:52.637680 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.637636 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.637763 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.637672 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.637763 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.637694 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.638235 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.637977 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.638235 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.638004 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.638235 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.638022 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.638458 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:30:52.638431 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.638537 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.638465 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.638537 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.638484 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.639078 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639052 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.639204 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639079 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.639477 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639444 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.639477 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639477 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.639880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639775 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.639880 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.639801 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.640127 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640097 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.640127 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640123 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.640445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640407 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.640445 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640429 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.640725 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640695 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.640817 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640726 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.640817 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.640783 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 01:30:52.641286 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.641260 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.641392 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.641289 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.642424 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642400 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.642521 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642426 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.642682 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642656 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.642770 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642683 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.642964 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642941 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.643084 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.642965 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.643362 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643337 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.643455 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643363 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.643654 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643630 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.643730 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643655 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.643914 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643893 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.644034 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.643916 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.644254 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.644211 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.644254 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.644240 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.644509 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.644486 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.644571 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.644513 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.644571 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.644535 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:52.645129 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.645031 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.645129 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.645056 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.647786 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.647321 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.647786 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.647345 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.648055 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648016 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.648055 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648040 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.648689 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648657 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.648689 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648683 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.648962 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648934 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.649069 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.648963 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.649260 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649236 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.649322 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649263 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.649499 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649473 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.649572 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649500 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.649746 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649721 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.649822 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649746 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.650021 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.649999 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.650087 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650023 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.650343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650318 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.650343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650341 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.650595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650565 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.650595 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650591 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.650853 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650833 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.650940 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.650854 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.651288 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651155 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.651288 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651210 2554 scope.go:117] "RemoveContainer" containerID="2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e" Jul 23 01:30:52.651449 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651427 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e"} err="failed to get container status \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": rpc error: code = NotFound desc = could not find container \"2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e\": container with ID starting with 2e0c1c0b80c68e0a9a3979a3e7648152cecee681625332f155dc614bf3d5d28e not found: ID does not exist" Jul 23 01:30:52.651505 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651452 2554 scope.go:117] "RemoveContainer" containerID="c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231" Jul 23 01:30:52.651685 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651662 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231"} err="failed to get container status \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": rpc error: code = NotFound desc = could not find container \"c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231\": container with ID starting with c415fbc3fbae316dc67deb52bf148d9013170f6f8bcbb7df01e1ecbfc736b231 not found: ID does not exist" Jul 23 01:30:52.651754 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.651687 2554 scope.go:117] "RemoveContainer" containerID="58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91" Jul 23 01:30:52.652046 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652022 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91"} err="failed to get container status \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": rpc error: code = NotFound desc = could not find container \"58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91\": container with ID starting with 58ab7742030be477525bce0aad8bc921f952f305769219324395acdd4426ea91 not found: ID does not exist" Jul 23 01:30:52.652133 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652048 2554 scope.go:117] "RemoveContainer" containerID="d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e" Jul 23 01:30:52.652308 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652287 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e"} err="failed to get container status \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": rpc error: code = NotFound desc = could not find container \"d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e\": container with ID starting with d5147004542da7ad37d891374735cd0f12f7d92c10f88110453e556605543d8e not found: ID does not exist" Jul 23 01:30:52.652308 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652310 2554 scope.go:117] "RemoveContainer" containerID="6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f" Jul 23 01:30:52.652528 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652509 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f"} err="failed to get container status \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": rpc error: code = NotFound desc = could not find container \"6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f\": container with ID starting with 6cb47035c6df54ded1186f67833ee33820facd8b9e5fafda42536401f45ded8f not found: ID does not exist" Jul 23 01:30:52.652528 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652528 2554 scope.go:117] "RemoveContainer" containerID="b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef" Jul 23 01:30:52.652769 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652752 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef"} err="failed to get container status \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": rpc error: code = NotFound desc = could not find container \"b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef\": container with ID starting with b84bfb21cc0f152804d81cbbacd9210203108e70fecfe3118a136dc60bb2d6ef not found: ID does not exist" Jul 23 01:30:52.652838 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652769 2554 scope.go:117] "RemoveContainer" containerID="ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e" Jul 23 01:30:52.653016 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.652989 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e"} err="failed to get container status \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": rpc error: code = NotFound desc = could not find container \"ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e\": container with ID starting with ca64fed709f513abcda1e82912c9ee100d2409ca57d5799833f1ce156a08fb4e not found: ID does not exist" Jul 23 01:30:52.770462 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.770378 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1da72b-1f99-4aa3-bb9c-1561c2d38b40" path="/var/lib/kubelet/pods/8e1da72b-1f99-4aa3-bb9c-1561c2d38b40/volumes" Jul 23 01:30:52.773044 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773056 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773077 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773103 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773128 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773194 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773205 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773251 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773301 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773357 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773386 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmw8\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-kube-api-access-stmw8\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773479 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773438 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773620 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773664 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773693 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.773795 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.773749 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874306 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874306 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874308 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874399 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874426 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874491 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874495 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmw8\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-kube-api-access-stmw8\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874526 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874549 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874577 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874601 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874630 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874656 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874693 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.874744 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874733 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874758 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874792 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874820 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.874871 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.875221 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.875144 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.878141 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.877988 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.878615 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.878411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.879008 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.878716 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.879096 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.879284 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879216 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879423 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879807 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879825 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.879995 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880160 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.880095 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.880470 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.880440 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.882195 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.880961 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.883268 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.882422 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-config\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.883268 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.882908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.883268 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.883064 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.884556 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.884529 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.886775 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.886717 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.888118 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.888098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmw8\" (UniqueName: \"kubernetes.io/projected/78f591f9-f4b4-420c-bf53-1c2ff18d21a8-kube-api-access-stmw8\") pod \"prometheus-k8s-0\" (UID: \"78f591f9-f4b4-420c-bf53-1c2ff18d21a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:52.945110 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:52.945073 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:30:53.095093 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:53.095058 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 01:30:53.098241 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:30:53.098207 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f591f9_f4b4_420c_bf53_1c2ff18d21a8.slice/crio-7040d3cfcddb1bfd3c72b42fa56674996470adc1a82d43d43874b025c4701966 WatchSource:0}: Error finding container 7040d3cfcddb1bfd3c72b42fa56674996470adc1a82d43d43874b025c4701966: Status 404 returned error can't find the container with id 7040d3cfcddb1bfd3c72b42fa56674996470adc1a82d43d43874b025c4701966 Jul 23 01:30:53.578965 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:53.578923 2554 generic.go:356] "Generic (PLEG): container finished" podID="78f591f9-f4b4-420c-bf53-1c2ff18d21a8" containerID="cf875893fedd0cfbe7f68de9ece03e0b9040584831ac5fbb2c98a15f6b7f5f2e" exitCode=0 Jul 23 01:30:53.579429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:53.579028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerDied","Data":"cf875893fedd0cfbe7f68de9ece03e0b9040584831ac5fbb2c98a15f6b7f5f2e"} Jul 23 01:30:53.579429 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:53.579111 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"7040d3cfcddb1bfd3c72b42fa56674996470adc1a82d43d43874b025c4701966"} Jul 23 01:30:54.586382 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.586344 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" event={"ID":"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab","Type":"ContainerStarted","Data":"4b835f99babf666922e8160f350554757452349ccef704f5492b3d88a173f384"} Jul 23 01:30:54.586827 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.586402 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" event={"ID":"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab","Type":"ContainerStarted","Data":"b58b98bd47c608cb208b41b77b29ce528ec821d59248c3247b5f4760c6534b15"} Jul 23 01:30:54.586827 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.586427 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" event={"ID":"c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab","Type":"ContainerStarted","Data":"bbdb9e85c96bac98d7a8032e1df571fb7f3c561b79bddaab7bb81f68243fbcb4"} Jul 23 01:30:54.589659 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589635 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"2d1e1bef15b18d73214143f30bb5df358702ca9142e08cf727ade550d00e44de"} Jul 23 01:30:54.589796 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589702 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"abbb14d99eb63b8923df121a925fda5e93ba25aad07970cfd8277bb426658bbf"} Jul 23 01:30:54.589796 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589748 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"01b267ce75edc4bcce488a5777853afaa26c225dca8bdec73c432fef2ccfaed0"} Jul 23 01:30:54.589796 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589786 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"e06e05d668c7a49188110bb9eec86b52ac6382000e26c1fae97402b0d93c09ae"} Jul 23 01:30:54.589955 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589829 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"a4232f584bbcc6a3d4378fbc3497b79fdd1f19fce04fbe01afcdde93a1bed831"} Jul 23 01:30:54.589955 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.589870 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78f591f9-f4b4-420c-bf53-1c2ff18d21a8","Type":"ContainerStarted","Data":"54bfa61cd53bc953d98aeaef914fc3857c20653cca0ca0706b19e1bb42bf2232"} Jul 23 01:30:54.606984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.606936 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-765848f968-j4mrz" podStartSLOduration=1.6009072359999998 podStartE2EDuration="3.606909884s" totalImagesPullingTime="2.006002648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:54.605580644 +0000 UTC m=+124.385104573" watchObservedRunningTime="2026-07-23 01:30:54.606909884 +0000 UTC m=+124.386433812" Jul 23 01:30:54.633086 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:54.633023 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.612665031 podStartE2EDuration="2.633003675s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.338644ms" isStatefulPod=false podCreationTimestamp="2026-07-23 01:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:54.631660785 +0000 UTC m=+124.411184745" watchObservedRunningTime="2026-07-23 01:30:54.633003675 +0000 UTC m=+124.412527604" Jul 23 01:30:57.945789 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:30:57.945745 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:31:52.945497 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:31:52.945449 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:31:52.960872 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:31:52.960845 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:31:53.834499 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:31:53.834473 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 01:33:40.988444 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.988410 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m9j8j"] Jul 23 01:33:40.991565 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.991547 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:40.993971 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.993946 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 01:33:40.994082 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.993995 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 01:33:40.995104 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.995086 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2m5c5\"" Jul 23 01:33:40.998584 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:40.998564 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m9j8j"] Jul 23 01:33:41.100465 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.100434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcs6p\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-kube-api-access-bcs6p\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.100626 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.100485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.201152 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.201116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.201329 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.201227 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcs6p\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-kube-api-access-bcs6p\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.209433 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.209402 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.209597 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.209577 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcs6p\" (UniqueName: \"kubernetes.io/projected/bd6eeb35-dd91-4864-a9a7-1d4f379a7845-kube-api-access-bcs6p\") pod \"cert-manager-webhook-6486ffd997-m9j8j\" (UID: \"bd6eeb35-dd91-4864-a9a7-1d4f379a7845\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.310362 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.310288 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:41.434102 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:41.434071 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m9j8j"] Jul 23 01:33:41.439887 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:33:41.439862 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6eeb35_dd91_4864_a9a7_1d4f379a7845.slice/crio-e3e35b0832b41e121a5a42bda188887e6f303eaf35fe54364bb3b9f0b9f5a3ba WatchSource:0}: Error finding container e3e35b0832b41e121a5a42bda188887e6f303eaf35fe54364bb3b9f0b9f5a3ba: Status 404 returned error can't find the container with id e3e35b0832b41e121a5a42bda188887e6f303eaf35fe54364bb3b9f0b9f5a3ba Jul 23 01:33:42.236966 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:42.236924 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" event={"ID":"bd6eeb35-dd91-4864-a9a7-1d4f379a7845","Type":"ContainerStarted","Data":"e3e35b0832b41e121a5a42bda188887e6f303eaf35fe54364bb3b9f0b9f5a3ba"} Jul 23 01:33:45.251618 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:45.251577 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" event={"ID":"bd6eeb35-dd91-4864-a9a7-1d4f379a7845","Type":"ContainerStarted","Data":"7963c1b40d47e5fb8cd0813cfaa89ea824d7f5fb1efee0d62781bfd4ad50685b"} Jul 23 01:33:45.251976 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:45.251716 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:45.266678 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:45.266635 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" podStartSLOduration=2.258048838 podStartE2EDuration="5.266622136s" totalImagesPullingTime="3.008573298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:33:45.265783306 +0000 UTC m=+295.045307237" watchObservedRunningTime="2026-07-23 01:33:45.266622136 +0000 UTC m=+295.046146065" Jul 23 01:33:50.600126 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.600097 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 01:33:50.968973 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.968896 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj"] Jul 23 01:33:50.971811 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.971794 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:50.974217 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.974192 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 01:33:50.974339 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.974193 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:33:50.975258 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.975243 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-59f8b\"" Jul 23 01:33:50.980430 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:50.980408 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj"] Jul 23 01:33:51.092306 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.092271 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvjl\" (UniqueName: \"kubernetes.io/projected/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-kube-api-access-nhvjl\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.092483 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.092328 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.193604 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.193556 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.193802 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.193656 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvjl\" (UniqueName: \"kubernetes.io/projected/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-kube-api-access-nhvjl\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.194002 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.193978 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.202645 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.202625 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvjl\" (UniqueName: \"kubernetes.io/projected/5f25e40d-4fb3-4ca6-b5cd-063f236430f0-kube-api-access-nhvjl\") pod \"openshift-lws-operator-fd8ccff4c-ln5bj\" (UID: \"5f25e40d-4fb3-4ca6-b5cd-063f236430f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.257839 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.257768 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m9j8j" Jul 23 01:33:51.281522 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.281493 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" Jul 23 01:33:51.410018 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:51.409991 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj"] Jul 23 01:33:51.412271 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:33:51.412242 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f25e40d_4fb3_4ca6_b5cd_063f236430f0.slice/crio-cccefb591f9b428aec2a70230e30ea7067990018c76ba48903a983457ef50fe6 WatchSource:0}: Error finding container cccefb591f9b428aec2a70230e30ea7067990018c76ba48903a983457ef50fe6: Status 404 returned error can't find the container with id cccefb591f9b428aec2a70230e30ea7067990018c76ba48903a983457ef50fe6 Jul 23 01:33:52.280871 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:52.280835 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" event={"ID":"5f25e40d-4fb3-4ca6-b5cd-063f236430f0","Type":"ContainerStarted","Data":"cccefb591f9b428aec2a70230e30ea7067990018c76ba48903a983457ef50fe6"} Jul 23 01:33:54.291089 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:54.291057 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" event={"ID":"5f25e40d-4fb3-4ca6-b5cd-063f236430f0","Type":"ContainerStarted","Data":"e7c23c2b237b01f087e79e43a351d09b6822c0b30737a5c994145cfcb5619884"} Jul 23 01:33:54.308127 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:33:54.308077 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln5bj" podStartSLOduration=1.62383384 podStartE2EDuration="4.308064303s" totalImagesPullingTime="2.684230463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:33:54.306820116 +0000 UTC m=+304.086344046" watchObservedRunningTime="2026-07-23 01:33:54.308064303 +0000 UTC m=+304.087588236" Jul 23 01:34:10.731539 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.731502 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq"] Jul 23 01:34:10.737775 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.737754 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.740282 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.740258 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 01:34:10.740412 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.740307 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zdsb9\"" Jul 23 01:34:10.740412 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.740328 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 01:34:10.740592 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.740483 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 01:34:10.740592 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.740497 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 01:34:10.748646 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.748621 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq"] Jul 23 01:34:10.759996 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.759970 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.760107 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.760070 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.760204 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.760119 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.760204 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.760167 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.760317 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.760265 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/f36abc8b-5a11-448d-88ee-3d70a9db2c22-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.860855 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.860822 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.860855 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.860862 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.861067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.860889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.861067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.860925 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/f36abc8b-5a11-448d-88ee-3d70a9db2c22-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.861067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.860954 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.861329 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.861298 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.863358 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.863335 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f36abc8b-5a11-448d-88ee-3d70a9db2c22-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.863452 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.863369 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.863519 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.863505 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f36abc8b-5a11-448d-88ee-3d70a9db2c22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:10.869208 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:10.869189 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/f36abc8b-5a11-448d-88ee-3d70a9db2c22-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq\" (UID: \"f36abc8b-5a11-448d-88ee-3d70a9db2c22\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:11.050587 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:11.050502 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:11.202263 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:11.202235 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq"] Jul 23 01:34:11.204930 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:34:11.204902 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36abc8b_5a11_448d_88ee_3d70a9db2c22.slice/crio-63a29afa7846bda006748f1b1c51b92f50acaba63490fe153ddf7ffab6d751a9 WatchSource:0}: Error finding container 63a29afa7846bda006748f1b1c51b92f50acaba63490fe153ddf7ffab6d751a9: Status 404 returned error can't find the container with id 63a29afa7846bda006748f1b1c51b92f50acaba63490fe153ddf7ffab6d751a9 Jul 23 01:34:11.206689 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:11.206672 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:34:11.362441 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:11.362402 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" event={"ID":"f36abc8b-5a11-448d-88ee-3d70a9db2c22","Type":"ContainerStarted","Data":"63a29afa7846bda006748f1b1c51b92f50acaba63490fe153ddf7ffab6d751a9"} Jul 23 01:34:15.382312 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:15.382272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" event={"ID":"f36abc8b-5a11-448d-88ee-3d70a9db2c22","Type":"ContainerStarted","Data":"35873f9bd2674b30665c3807ec5e4b92be81b46aa9b1796d8989c98a403aa833"} Jul 23 01:34:16.387911 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:16.387871 2554 generic.go:356] "Generic (PLEG): container finished" podID="f36abc8b-5a11-448d-88ee-3d70a9db2c22" containerID="35873f9bd2674b30665c3807ec5e4b92be81b46aa9b1796d8989c98a403aa833" exitCode=0 Jul 23 01:34:16.388362 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:16.387952 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" event={"ID":"f36abc8b-5a11-448d-88ee-3d70a9db2c22","Type":"ContainerDied","Data":"35873f9bd2674b30665c3807ec5e4b92be81b46aa9b1796d8989c98a403aa833"} Jul 23 01:34:17.395412 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:17.395373 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" event={"ID":"f36abc8b-5a11-448d-88ee-3d70a9db2c22","Type":"ContainerStarted","Data":"cc1669cccfc832723999280273fcba665d6aed3de2f655ff83e7b97a6a50b213"} Jul 23 01:34:17.395777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:17.395454 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:17.417562 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:17.417514 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" podStartSLOduration=2.370946952 podStartE2EDuration="7.417498638s" totalImagesPullingTime="3.925941493s" totalInitContainerRuntime="1.120610193s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:17.414545843 +0000 UTC m=+327.194069773" watchObservedRunningTime="2026-07-23 01:34:17.417498638 +0000 UTC m=+327.197022618" Jul 23 01:34:28.401750 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:28.401720 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq" Jul 23 01:34:29.970521 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.970486 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt"] Jul 23 01:34:29.975421 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.975398 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:29.979849 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.979808 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 01:34:29.979849 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.979825 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 01:34:29.980054 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.979864 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 01:34:29.980054 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.979808 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pd6hf\"" Jul 23 01:34:29.985923 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:29.985903 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt"] Jul 23 01:34:30.020808 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.020774 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpk9\" (UniqueName: \"kubernetes.io/projected/42972945-0cf2-4f9f-bd40-1c9db6018e89-kube-api-access-mkpk9\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.020991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.020845 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42972945-0cf2-4f9f-bd40-1c9db6018e89-manager-config\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.020991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.020864 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-metrics-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.020991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.020963 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.121927 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.121893 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.122102 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.121937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpk9\" (UniqueName: \"kubernetes.io/projected/42972945-0cf2-4f9f-bd40-1c9db6018e89-kube-api-access-mkpk9\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.122102 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.121986 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42972945-0cf2-4f9f-bd40-1c9db6018e89-manager-config\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.122102 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.122013 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-metrics-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.122659 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.122636 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42972945-0cf2-4f9f-bd40-1c9db6018e89-manager-config\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.124389 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.124359 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.124484 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.124390 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42972945-0cf2-4f9f-bd40-1c9db6018e89-metrics-cert\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.147229 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.147200 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpk9\" (UniqueName: \"kubernetes.io/projected/42972945-0cf2-4f9f-bd40-1c9db6018e89-kube-api-access-mkpk9\") pod \"lws-controller-manager-7564d6dfb8-cv8pt\" (UID: \"42972945-0cf2-4f9f-bd40-1c9db6018e89\") " pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.285497 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.285413 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:30.414196 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.414154 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt"] Jul 23 01:34:30.450887 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.450851 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" event={"ID":"42972945-0cf2-4f9f-bd40-1c9db6018e89","Type":"ContainerStarted","Data":"c27e3d07b9bd6c9b969d8bf54f03821006dfb40cb0a5e31eb8fcc045670c7bb9"} Jul 23 01:34:30.526334 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.526308 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt"] Jul 23 01:34:30.529240 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.529222 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.531907 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.531708 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 01:34:30.532161 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.532139 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cjnxx\"" Jul 23 01:34:30.532283 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.532195 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 01:34:30.532283 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.532204 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 01:34:30.532951 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.532936 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 01:34:30.540402 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.540382 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt"] Jul 23 01:34:30.626508 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.626480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.626508 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.626509 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkfd\" (UniqueName: \"kubernetes.io/projected/b5d66029-e76e-4d2f-9c5a-80d51243a316-kube-api-access-rkkfd\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.626678 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.626639 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d66029-e76e-4d2f-9c5a-80d51243a316-tmp\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.727887 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.727851 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.727887 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.727888 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkfd\" (UniqueName: \"kubernetes.io/projected/b5d66029-e76e-4d2f-9c5a-80d51243a316-kube-api-access-rkkfd\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.728128 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.727975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d66029-e76e-4d2f-9c5a-80d51243a316-tmp\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.728128 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:34:30.728015 2554 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 23 01:34:30.728128 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:34:30.728122 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs podName:b5d66029-e76e-4d2f-9c5a-80d51243a316 nodeName:}" failed. No retries permitted until 2026-07-23 01:34:31.228101579 +0000 UTC m=+341.007625500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs") pod "kube-auth-proxy-85497ff65b-6wgbt" (UID: "b5d66029-e76e-4d2f-9c5a-80d51243a316") : secret "kube-auth-proxy-tls" not found Jul 23 01:34:30.730101 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.730083 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d66029-e76e-4d2f-9c5a-80d51243a316-tmp\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:30.743025 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:30.743006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkfd\" (UniqueName: \"kubernetes.io/projected/b5d66029-e76e-4d2f-9c5a-80d51243a316-kube-api-access-rkkfd\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:31.233325 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:31.233287 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:31.235698 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:31.235675 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66029-e76e-4d2f-9c5a-80d51243a316-tls-certs\") pod \"kube-auth-proxy-85497ff65b-6wgbt\" (UID: \"b5d66029-e76e-4d2f-9c5a-80d51243a316\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:31.440317 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:31.440281 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" Jul 23 01:34:31.594741 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:31.594710 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt"] Jul 23 01:34:31.596869 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:34:31.596832 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d66029_e76e_4d2f_9c5a_80d51243a316.slice/crio-fe988f0cd4f0f83774382b105a3f2333d61d4143354778c73f68a37da219dcff WatchSource:0}: Error finding container fe988f0cd4f0f83774382b105a3f2333d61d4143354778c73f68a37da219dcff: Status 404 returned error can't find the container with id fe988f0cd4f0f83774382b105a3f2333d61d4143354778c73f68a37da219dcff Jul 23 01:34:32.464950 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:32.464857 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" event={"ID":"42972945-0cf2-4f9f-bd40-1c9db6018e89","Type":"ContainerStarted","Data":"b7c50c18d0b79391ba7cb4f0e4d0ff44ce49ef88d2263addbb51a74ff6183ae4"} Jul 23 01:34:32.464950 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:32.464925 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:34:32.466252 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:32.466225 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" event={"ID":"b5d66029-e76e-4d2f-9c5a-80d51243a316","Type":"ContainerStarted","Data":"fe988f0cd4f0f83774382b105a3f2333d61d4143354778c73f68a37da219dcff"} Jul 23 01:34:32.482228 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:32.482152 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" podStartSLOduration=1.7817529429999999 podStartE2EDuration="3.482138532s" totalImagesPullingTime="1.700385589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:32.481455274 +0000 UTC m=+342.260979204" watchObservedRunningTime="2026-07-23 01:34:32.482138532 +0000 UTC m=+342.261662460" Jul 23 01:34:34.477245 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:34.477210 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" event={"ID":"b5d66029-e76e-4d2f-9c5a-80d51243a316","Type":"ContainerStarted","Data":"e7becd333a61a866186a2d8d81f07e12a4440ec0c5eb0fcb049c7535fe3d739e"} Jul 23 01:34:34.493261 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:34.493211 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85497ff65b-6wgbt" podStartSLOduration=2.208046315 podStartE2EDuration="4.493196206s" totalImagesPullingTime="2.285149891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:34.491812415 +0000 UTC m=+344.271336345" watchObservedRunningTime="2026-07-23 01:34:34.493196206 +0000 UTC m=+344.272720126" Jul 23 01:34:43.473559 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:34:43.473483 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7564d6dfb8-cv8pt" Jul 23 01:36:12.113687 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.113652 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff"] Jul 23 01:36:12.115963 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.115944 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.119804 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.119785 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 01:36:12.120059 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.120040 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ggqtr\"" Jul 23 01:36:12.120142 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.120091 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 01:36:12.139462 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.139433 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff"] Jul 23 01:36:12.217331 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.217279 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjw8f\" (UniqueName: \"kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.217496 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.217393 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.318683 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.318641 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.318835 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.318738 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjw8f\" (UniqueName: \"kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.319008 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.318971 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.329559 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.329535 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjw8f\" (UniqueName: \"kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.425841 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.425750 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:12.553815 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.553778 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff"] Jul 23 01:36:12.556546 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:36:12.556519 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257f0094_d9b5_485a_a479_81e6bacd0759.slice/crio-ad1a4de4b6e47d04a23c353d155b0703b4a1b4b103eb434ed94ee42eabf3f3c0 WatchSource:0}: Error finding container ad1a4de4b6e47d04a23c353d155b0703b4a1b4b103eb434ed94ee42eabf3f3c0: Status 404 returned error can't find the container with id ad1a4de4b6e47d04a23c353d155b0703b4a1b4b103eb434ed94ee42eabf3f3c0 Jul 23 01:36:12.907267 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:12.907232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" event={"ID":"257f0094-d9b5-485a-a479-81e6bacd0759","Type":"ContainerStarted","Data":"ad1a4de4b6e47d04a23c353d155b0703b4a1b4b103eb434ed94ee42eabf3f3c0"} Jul 23 01:36:18.944109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:18.944076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" event={"ID":"257f0094-d9b5-485a-a479-81e6bacd0759","Type":"ContainerStarted","Data":"02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9"} Jul 23 01:36:18.944521 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:18.944167 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:18.962964 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:18.962903 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" podStartSLOduration=1.244988585 podStartE2EDuration="6.96288799s" totalImagesPullingTime="5.717899405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:18.961413792 +0000 UTC m=+448.740937749" watchObservedRunningTime="2026-07-23 01:36:18.96288799 +0000 UTC m=+448.742411919" Jul 23 01:36:29.951836 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:29.951801 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:30.912338 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:30.912305 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr"] Jul 23 01:36:30.920216 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:30.920194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:30.925681 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:30.925657 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr"] Jul 23 01:36:31.081159 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.081127 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.081526 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.081206 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jptkz\" (UniqueName: \"kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.182467 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.182374 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.182467 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.182434 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jptkz\" (UniqueName: \"kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.182790 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.182768 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.191473 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.191445 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jptkz\" (UniqueName: \"kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.231463 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.231432 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:31.368760 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.368733 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr"] Jul 23 01:36:31.371436 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:36:31.371407 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59422a55_e482_452a_a4a2_d01a847156f9.slice/crio-1305d601d4928891c2804f5bc412455cf6b414f4a84a0f45e4c84bfda8b705be WatchSource:0}: Error finding container 1305d601d4928891c2804f5bc412455cf6b414f4a84a0f45e4c84bfda8b705be: Status 404 returned error can't find the container with id 1305d601d4928891c2804f5bc412455cf6b414f4a84a0f45e4c84bfda8b705be Jul 23 01:36:31.561448 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.561402 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr"] Jul 23 01:36:31.567885 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.567852 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr"] Jul 23 01:36:31.577979 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.577951 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff"] Jul 23 01:36:31.578155 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.578134 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" containerName="manager" containerID="cri-o://02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9" gracePeriod=2 Jul 23 01:36:31.587541 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.587513 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:31.588095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.588069 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59422a55-e482-452a-a4a2-d01a847156f9" containerName="manager" Jul 23 01:36:31.588095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.588094 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="59422a55-e482-452a-a4a2-d01a847156f9" containerName="manager" Jul 23 01:36:31.588289 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.588200 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="59422a55-e482-452a-a4a2-d01a847156f9" containerName="manager" Jul 23 01:36:31.592017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.591994 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.596992 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:31.596958 2554 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"257f0094-d9b5-485a-a479-81e6bacd0759\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-752ff\": pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:31.597382 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.597357 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff"] Jul 23 01:36:31.599093 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:31.599063 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:31.605582 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.605553 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:31.606142 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.606125 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" containerName="manager" Jul 23 01:36:31.606220 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.606145 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" containerName="manager" Jul 23 01:36:31.606294 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.606283 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" containerName="manager" Jul 23 01:36:31.610164 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.610145 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.611658 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.611636 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:31.625749 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:31.625715 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:31.627132 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.627017 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:31.687801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.687720 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9q7\" (UniqueName: \"kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.687801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.687761 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.687801 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.687787 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9vr\" (UniqueName: \"kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.688021 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.687874 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.788597 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.788568 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9q7\" (UniqueName: \"kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.788766 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.788605 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.788766 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.788632 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9vr\" (UniqueName: \"kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.788766 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.788658 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.789042 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.789022 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.789102 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.789021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.797061 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.797035 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9q7\" (UniqueName: \"kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pwch5\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.797283 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.797260 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9vr\" (UniqueName: \"kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rcm9s\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.806992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.806972 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:31.809129 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:31.809107 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:31.889647 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.889617 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume\") pod \"257f0094-d9b5-485a-a479-81e6bacd0759\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " Jul 23 01:36:31.889816 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.889669 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjw8f\" (UniqueName: \"kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f\") pod \"257f0094-d9b5-485a-a479-81e6bacd0759\" (UID: \"257f0094-d9b5-485a-a479-81e6bacd0759\") " Jul 23 01:36:31.890095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.890064 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "257f0094-d9b5-485a-a479-81e6bacd0759" (UID: "257f0094-d9b5-485a-a479-81e6bacd0759"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:36:31.891766 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.891747 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f" (OuterVolumeSpecName: "kube-api-access-mjw8f") pod "257f0094-d9b5-485a-a479-81e6bacd0759" (UID: "257f0094-d9b5-485a-a479-81e6bacd0759"). InnerVolumeSpecName "kube-api-access-mjw8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:31.948328 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.948247 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:31.956327 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.956300 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:31.990815 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.990779 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f0094-d9b5-485a-a479-81e6bacd0759-extensions-socket-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:31.990815 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:31.990816 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjw8f\" (UniqueName: \"kubernetes.io/projected/257f0094-d9b5-485a-a479-81e6bacd0759-kube-api-access-mjw8f\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:32.011589 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.011521 2554 generic.go:356] "Generic (PLEG): container finished" podID="257f0094-d9b5-485a-a479-81e6bacd0759" containerID="02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9" exitCode=0 Jul 23 01:36:32.011778 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.011596 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:32.011778 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.011632 2554 scope.go:117] "RemoveContainer" containerID="02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9" Jul 23 01:36:32.014221 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.014138 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:32.015458 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.015148 2554 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" container="manager" Jul 23 01:36:32.018651 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.018494 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:32.020424 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.020391 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="59422a55-e482-452a-a4a2-d01a847156f9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:32.026967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.026888 2554 scope.go:117] "RemoveContainer" containerID="02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9" Jul 23 01:36:32.027582 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.027392 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9\": container with ID starting with 02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9 not found: ID does not exist" containerID="02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9" Jul 23 01:36:32.027582 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.027439 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9"} err="failed to get container status \"02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9\": rpc error: code = NotFound desc = could not find container \"02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9\": container with ID starting with 02e1352a9d5b75acc039c02acfd4ceaee239f851ad9311013d98d6f95fe339b9 not found: ID does not exist" Jul 23 01:36:32.032022 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.031839 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="59422a55-e482-452a-a4a2-d01a847156f9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:32.033561 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:32.033535 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-752ff\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-752ff" Jul 23 01:36:32.104966 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.104925 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:32.108571 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:36:32.108547 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947ed7b8_4b33_477d_a044_de1aa33a7d9f.slice/crio-493e4dcde9c1e57134ac86ffd6b8adbc962d7035bfce980d7996d3cd99fc9dbe WatchSource:0}: Error finding container 493e4dcde9c1e57134ac86ffd6b8adbc962d7035bfce980d7996d3cd99fc9dbe: Status 404 returned error can't find the container with id 493e4dcde9c1e57134ac86ffd6b8adbc962d7035bfce980d7996d3cd99fc9dbe Jul 23 01:36:32.125136 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.125114 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:32.130151 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:36:32.130113 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8310df4_0f11_419c_aae6_9cc730b294a7.slice/crio-7b9a564a7fb6c63ac241e9c1cf11697984e6e2ba6003f87a9839f05fa6e0d966 WatchSource:0}: Error finding container 7b9a564a7fb6c63ac241e9c1cf11697984e6e2ba6003f87a9839f05fa6e0d966: Status 404 returned error can't find the container with id 7b9a564a7fb6c63ac241e9c1cf11697984e6e2ba6003f87a9839f05fa6e0d966 Jul 23 01:36:32.768758 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:32.768726 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257f0094-d9b5-485a-a479-81e6bacd0759" path="/var/lib/kubelet/pods/257f0094-d9b5-485a-a479-81e6bacd0759/volumes" Jul 23 01:36:33.022051 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.021954 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" event={"ID":"947ed7b8-4b33-477d-a044-de1aa33a7d9f","Type":"ContainerStarted","Data":"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a"} Jul 23 01:36:33.022245 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.022130 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:33.022331 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.022238 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" event={"ID":"947ed7b8-4b33-477d-a044-de1aa33a7d9f","Type":"ContainerStarted","Data":"493e4dcde9c1e57134ac86ffd6b8adbc962d7035bfce980d7996d3cd99fc9dbe"} Jul 23 01:36:33.023812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.023784 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" podUID="59422a55-e482-452a-a4a2-d01a847156f9" containerName="manager" containerID="cri-o://ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136" gracePeriod=2 Jul 23 01:36:33.026499 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.026473 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" event={"ID":"f8310df4-0f11-419c-aae6-9cc730b294a7","Type":"ContainerStarted","Data":"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400"} Jul 23 01:36:33.026586 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.026523 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" event={"ID":"f8310df4-0f11-419c-aae6-9cc730b294a7","Type":"ContainerStarted","Data":"7b9a564a7fb6c63ac241e9c1cf11697984e6e2ba6003f87a9839f05fa6e0d966"} Jul 23 01:36:33.026586 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.026546 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:33.044477 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.044432 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" podStartSLOduration=2.044413428 podStartE2EDuration="2.044413428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:33.042712111 +0000 UTC m=+462.822236041" watchObservedRunningTime="2026-07-23 01:36:33.044413428 +0000 UTC m=+462.823937357" Jul 23 01:36:33.065286 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.065241 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" podStartSLOduration=2.065210493 podStartE2EDuration="2.065210493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:33.063631735 +0000 UTC m=+462.843155697" watchObservedRunningTime="2026-07-23 01:36:33.065210493 +0000 UTC m=+462.844734421" Jul 23 01:36:33.266768 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.266737 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:33.268965 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:33.268933 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="59422a55-e482-452a-a4a2-d01a847156f9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:33.403458 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.403426 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jptkz\" (UniqueName: \"kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz\") pod \"59422a55-e482-452a-a4a2-d01a847156f9\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " Jul 23 01:36:33.403625 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.403470 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume\") pod \"59422a55-e482-452a-a4a2-d01a847156f9\" (UID: \"59422a55-e482-452a-a4a2-d01a847156f9\") " Jul 23 01:36:33.403770 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.403747 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "59422a55-e482-452a-a4a2-d01a847156f9" (UID: "59422a55-e482-452a-a4a2-d01a847156f9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:36:33.405508 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.405483 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz" (OuterVolumeSpecName: "kube-api-access-jptkz") pod "59422a55-e482-452a-a4a2-d01a847156f9" (UID: "59422a55-e482-452a-a4a2-d01a847156f9"). InnerVolumeSpecName "kube-api-access-jptkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:33.505085 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.505051 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jptkz\" (UniqueName: \"kubernetes.io/projected/59422a55-e482-452a-a4a2-d01a847156f9-kube-api-access-jptkz\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:33.505085 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:33.505080 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59422a55-e482-452a-a4a2-d01a847156f9-extensions-socket-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.032864 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.032831 2554 generic.go:356] "Generic (PLEG): container finished" podID="59422a55-e482-452a-a4a2-d01a847156f9" containerID="ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136" exitCode=2 Jul 23 01:36:34.033037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.032881 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:34.033037 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.032939 2554 scope.go:117] "RemoveContainer" containerID="ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136" Jul 23 01:36:34.035437 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:34.035407 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="59422a55-e482-452a-a4a2-d01a847156f9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:34.042085 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.042070 2554 scope.go:117] "RemoveContainer" containerID="ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136" Jul 23 01:36:34.042371 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:34.042353 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136\": container with ID starting with ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136 not found: ID does not exist" containerID="ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136" Jul 23 01:36:34.042423 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.042380 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136"} err="failed to get container status \"ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136\": rpc error: code = NotFound desc = could not find container \"ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136\": container with ID starting with ca59ff4bb4b9464835edf9946ce1285c6b9c5aaeae1072a0d8d7139d74ff4136 not found: ID does not exist" Jul 23 01:36:34.043512 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:34.043489 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-wlfmr\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="59422a55-e482-452a-a4a2-d01a847156f9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wlfmr" Jul 23 01:36:34.768968 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:34.768935 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59422a55-e482-452a-a4a2-d01a847156f9" path="/var/lib/kubelet/pods/59422a55-e482-452a-a4a2-d01a847156f9/volumes" Jul 23 01:36:44.036039 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.036011 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:44.036551 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.036079 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:44.088313 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.088284 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:44.088492 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.088469 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" podUID="f8310df4-0f11-419c-aae6-9cc730b294a7" containerName="manager" containerID="cri-o://a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400" gracePeriod=10 Jul 23 01:36:44.338553 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.338531 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:44.435239 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.435208 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:36:44.435599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.435586 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8310df4-0f11-419c-aae6-9cc730b294a7" containerName="manager" Jul 23 01:36:44.435683 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.435601 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8310df4-0f11-419c-aae6-9cc730b294a7" containerName="manager" Jul 23 01:36:44.435683 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.435663 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8310df4-0f11-419c-aae6-9cc730b294a7" containerName="manager" Jul 23 01:36:44.438570 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.438552 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.449703 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.449675 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:36:44.501114 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.501086 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume\") pod \"f8310df4-0f11-419c-aae6-9cc730b294a7\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " Jul 23 01:36:44.501293 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.501132 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9vr\" (UniqueName: \"kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr\") pod \"f8310df4-0f11-419c-aae6-9cc730b294a7\" (UID: \"f8310df4-0f11-419c-aae6-9cc730b294a7\") " Jul 23 01:36:44.501504 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.501479 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f8310df4-0f11-419c-aae6-9cc730b294a7" (UID: "f8310df4-0f11-419c-aae6-9cc730b294a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:36:44.503202 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.503162 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr" (OuterVolumeSpecName: "kube-api-access-rm9vr") pod "f8310df4-0f11-419c-aae6-9cc730b294a7" (UID: "f8310df4-0f11-419c-aae6-9cc730b294a7"). InnerVolumeSpecName "kube-api-access-rm9vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:44.602287 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.602255 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.602443 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.602314 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2jb\" (UniqueName: \"kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.602443 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.602428 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8310df4-0f11-419c-aae6-9cc730b294a7-extensions-socket-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:44.602533 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.602455 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9vr\" (UniqueName: \"kubernetes.io/projected/f8310df4-0f11-419c-aae6-9cc730b294a7-kube-api-access-rm9vr\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:44.703083 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.703046 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2jb\" (UniqueName: \"kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.703269 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.703121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.703456 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.703440 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.712107 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.712082 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2jb\" (UniqueName: \"kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb\") pod \"kuadrant-operator-controller-manager-86878c769f-hhnqx\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.748667 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.748638 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:44.876697 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:44.876667 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:36:44.879262 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:36:44.879234 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03efa629_27fc_4fc4_ab12_d8fff583d95e.slice/crio-ff96b5b8d86059bb388ed94228491c1572bd9ce7fbd73cb8fc080de0664c72c2 WatchSource:0}: Error finding container ff96b5b8d86059bb388ed94228491c1572bd9ce7fbd73cb8fc080de0664c72c2: Status 404 returned error can't find the container with id ff96b5b8d86059bb388ed94228491c1572bd9ce7fbd73cb8fc080de0664c72c2 Jul 23 01:36:45.086467 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.086427 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" event={"ID":"03efa629-27fc-4fc4-ab12-d8fff583d95e","Type":"ContainerStarted","Data":"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21"} Jul 23 01:36:45.086918 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.086487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" event={"ID":"03efa629-27fc-4fc4-ab12-d8fff583d95e","Type":"ContainerStarted","Data":"ff96b5b8d86059bb388ed94228491c1572bd9ce7fbd73cb8fc080de0664c72c2"} Jul 23 01:36:45.086918 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.086605 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:45.087709 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.087684 2554 generic.go:356] "Generic (PLEG): container finished" podID="f8310df4-0f11-419c-aae6-9cc730b294a7" containerID="a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400" exitCode=0 Jul 23 01:36:45.087793 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.087713 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" Jul 23 01:36:45.087830 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.087716 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" event={"ID":"f8310df4-0f11-419c-aae6-9cc730b294a7","Type":"ContainerDied","Data":"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400"} Jul 23 01:36:45.087863 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.087850 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s" event={"ID":"f8310df4-0f11-419c-aae6-9cc730b294a7","Type":"ContainerDied","Data":"7b9a564a7fb6c63ac241e9c1cf11697984e6e2ba6003f87a9839f05fa6e0d966"} Jul 23 01:36:45.087901 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.087874 2554 scope.go:117] "RemoveContainer" containerID="a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400" Jul 23 01:36:45.096228 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.096069 2554 scope.go:117] "RemoveContainer" containerID="a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400" Jul 23 01:36:45.096346 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:45.096325 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400\": container with ID starting with a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400 not found: ID does not exist" containerID="a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400" Jul 23 01:36:45.096384 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.096353 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400"} err="failed to get container status \"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400\": rpc error: code = NotFound desc = could not find container \"a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400\": container with ID starting with a9c0cbc4656dac72af89df97b33527ffe635348b80cb1ba7f840663d291e4400 not found: ID does not exist" Jul 23 01:36:45.106677 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.106628 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" podStartSLOduration=1.10661527 podStartE2EDuration="1.10661527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:45.10513441 +0000 UTC m=+474.884658341" watchObservedRunningTime="2026-07-23 01:36:45.10661527 +0000 UTC m=+474.886139198" Jul 23 01:36:45.120930 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.120907 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:45.125187 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:45.125145 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rcm9s"] Jul 23 01:36:46.769032 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:46.768999 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8310df4-0f11-419c-aae6-9cc730b294a7" path="/var/lib/kubelet/pods/f8310df4-0f11-419c-aae6-9cc730b294a7/volumes" Jul 23 01:36:56.095796 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.095762 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:36:56.160095 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.160055 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:56.160381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.160322 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" podUID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" containerName="manager" containerID="cri-o://7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a" gracePeriod=10 Jul 23 01:36:56.413659 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.413637 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:56.499039 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.499004 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9q7\" (UniqueName: \"kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7\") pod \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " Jul 23 01:36:56.499237 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.499054 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume\") pod \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\" (UID: \"947ed7b8-4b33-477d-a044-de1aa33a7d9f\") " Jul 23 01:36:56.499489 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.499463 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "947ed7b8-4b33-477d-a044-de1aa33a7d9f" (UID: "947ed7b8-4b33-477d-a044-de1aa33a7d9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:36:56.501066 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.501040 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7" (OuterVolumeSpecName: "kube-api-access-dm9q7") pod "947ed7b8-4b33-477d-a044-de1aa33a7d9f" (UID: "947ed7b8-4b33-477d-a044-de1aa33a7d9f"). InnerVolumeSpecName "kube-api-access-dm9q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:56.599827 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.599793 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9q7\" (UniqueName: \"kubernetes.io/projected/947ed7b8-4b33-477d-a044-de1aa33a7d9f-kube-api-access-dm9q7\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:56.599827 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:56.599820 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/947ed7b8-4b33-477d-a044-de1aa33a7d9f-extensions-socket-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:36:57.144681 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.144644 2554 generic.go:356] "Generic (PLEG): container finished" podID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" containerID="7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a" exitCode=0 Jul 23 01:36:57.145125 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.144689 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" Jul 23 01:36:57.145125 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.144718 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" event={"ID":"947ed7b8-4b33-477d-a044-de1aa33a7d9f","Type":"ContainerDied","Data":"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a"} Jul 23 01:36:57.145125 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.144762 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5" event={"ID":"947ed7b8-4b33-477d-a044-de1aa33a7d9f","Type":"ContainerDied","Data":"493e4dcde9c1e57134ac86ffd6b8adbc962d7035bfce980d7996d3cd99fc9dbe"} Jul 23 01:36:57.145125 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.144784 2554 scope.go:117] "RemoveContainer" containerID="7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a" Jul 23 01:36:57.153009 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.152989 2554 scope.go:117] "RemoveContainer" containerID="7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a" Jul 23 01:36:57.153281 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:36:57.153261 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a\": container with ID starting with 7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a not found: ID does not exist" containerID="7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a" Jul 23 01:36:57.153324 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.153289 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a"} err="failed to get container status \"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a\": rpc error: code = NotFound desc = could not find container \"7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a\": container with ID starting with 7f34d0cbc094bcb642bf7986da248e743f7085a811b5d7c47e25ce188c3e376a not found: ID does not exist" Jul 23 01:36:57.187800 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.187767 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:57.192281 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:57.192255 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pwch5"] Jul 23 01:36:58.768992 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:36:58.768947 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" path="/var/lib/kubelet/pods/947ed7b8-4b33-477d-a044-de1aa33a7d9f/volumes" Jul 23 01:37:16.834825 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.834790 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:37:16.835381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.835164 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" containerName="manager" Jul 23 01:37:16.835381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.835201 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" containerName="manager" Jul 23 01:37:16.835381 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.835271 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="947ed7b8-4b33-477d-a044-de1aa33a7d9f" containerName="manager" Jul 23 01:37:16.838414 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.838395 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:16.840737 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.840719 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 01:37:16.840839 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.840744 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x2gdz\"" Jul 23 01:37:16.847145 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.847124 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:37:16.872067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.872041 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:37:16.980077 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.980044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zzz\" (UniqueName: \"kubernetes.io/projected/7f9bc72a-4e2c-481b-96a6-d494f0796538-kube-api-access-69zzz\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:16.980278 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:16.980132 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f9bc72a-4e2c-481b-96a6-d494f0796538-config-file\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.081603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.081572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zzz\" (UniqueName: \"kubernetes.io/projected/7f9bc72a-4e2c-481b-96a6-d494f0796538-kube-api-access-69zzz\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.081765 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.081643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f9bc72a-4e2c-481b-96a6-d494f0796538-config-file\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.082461 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.082439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f9bc72a-4e2c-481b-96a6-d494f0796538-config-file\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.089371 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.089315 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zzz\" (UniqueName: \"kubernetes.io/projected/7f9bc72a-4e2c-481b-96a6-d494f0796538-kube-api-access-69zzz\") pod \"limitador-limitador-54d6d988d5-l8zbz\" (UID: \"7f9bc72a-4e2c-481b-96a6-d494f0796538\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.149336 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.149300 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:17.285068 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:17.285047 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:37:17.287430 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:37:17.287402 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9bc72a_4e2c_481b_96a6_d494f0796538.slice/crio-9bf70947fc7c028d4233a1e45012b04e46980f521386d9796b62e2fb17317fa7 WatchSource:0}: Error finding container 9bf70947fc7c028d4233a1e45012b04e46980f521386d9796b62e2fb17317fa7: Status 404 returned error can't find the container with id 9bf70947fc7c028d4233a1e45012b04e46980f521386d9796b62e2fb17317fa7 Jul 23 01:37:18.247615 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:18.247576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" event={"ID":"7f9bc72a-4e2c-481b-96a6-d494f0796538","Type":"ContainerStarted","Data":"9bf70947fc7c028d4233a1e45012b04e46980f521386d9796b62e2fb17317fa7"} Jul 23 01:37:20.259154 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:20.259116 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" event={"ID":"7f9bc72a-4e2c-481b-96a6-d494f0796538","Type":"ContainerStarted","Data":"25b5de2bd5f8cf7905249c242fde45c1a53f9fef63f3650adeab083795caebd9"} Jul 23 01:37:20.259554 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:20.259278 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:37:20.275017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:20.274964 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" podStartSLOduration=1.807235513 podStartE2EDuration="4.274947956s" totalImagesPullingTime="2.467712443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:37:20.274793912 +0000 UTC m=+510.054317844" watchObservedRunningTime="2026-07-23 01:37:20.274947956 +0000 UTC m=+510.054471886" Jul 23 01:37:31.264300 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:37:31.264272 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l8zbz" Jul 23 01:38:30.265787 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:38:30.265755 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:38:49.399231 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:38:49.399199 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:01.364448 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:01.364414 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:21.970002 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:21.969922 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:36.070212 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:36.070163 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:44.987529 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.987498 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr"] Jul 23 01:39:44.990841 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.990824 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:44.993109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.993089 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ktgld\"" Jul 23 01:39:44.993250 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.993090 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 01:39:44.994038 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.994018 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 01:39:44.994121 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:44.994037 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 01:39:45.000902 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.000879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr"] Jul 23 01:39:45.011514 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011482 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.011666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011535 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.011666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011567 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7affb312-1890-469c-b25d-acf9dc49bc25-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.011666 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.011812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.011812 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.011702 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4gx\" (UniqueName: \"kubernetes.io/projected/7affb312-1890-469c-b25d-acf9dc49bc25-kube-api-access-mz4gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112439 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112404 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112439 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112442 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112462 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4gx\" (UniqueName: \"kubernetes.io/projected/7affb312-1890-469c-b25d-acf9dc49bc25-kube-api-access-mz4gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112511 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112531 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112670 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112555 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7affb312-1890-469c-b25d-acf9dc49bc25-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112899 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112872 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112947 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112878 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.112947 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.112934 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.115084 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.115061 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7affb312-1890-469c-b25d-acf9dc49bc25-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.115206 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.115092 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7affb312-1890-469c-b25d-acf9dc49bc25-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.120908 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.120886 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4gx\" (UniqueName: \"kubernetes.io/projected/7affb312-1890-469c-b25d-acf9dc49bc25-kube-api-access-mz4gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xchvr\" (UID: \"7affb312-1890-469c-b25d-acf9dc49bc25\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.302990 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.302886 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:45.432985 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.432965 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr"] Jul 23 01:39:45.435266 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:39:45.435232 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7affb312_1890_469c_b25d_acf9dc49bc25.slice/crio-2ea784e1c778417ff844c5763b055d86b7e9404ae3fb075437caed5f3dfbbe9e WatchSource:0}: Error finding container 2ea784e1c778417ff844c5763b055d86b7e9404ae3fb075437caed5f3dfbbe9e: Status 404 returned error can't find the container with id 2ea784e1c778417ff844c5763b055d86b7e9404ae3fb075437caed5f3dfbbe9e Jul 23 01:39:45.436564 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.436548 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:39:45.930604 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:45.930562 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" event={"ID":"7affb312-1890-469c-b25d-acf9dc49bc25","Type":"ContainerStarted","Data":"2ea784e1c778417ff844c5763b055d86b7e9404ae3fb075437caed5f3dfbbe9e"} Jul 23 01:39:46.936648 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:46.936607 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" event={"ID":"7affb312-1890-469c-b25d-acf9dc49bc25","Type":"ContainerStarted","Data":"ddfa46f22f5cb5a7ea62322237c69c3d7807e3032e1e51881612538fe0e2bad0"} Jul 23 01:39:46.954613 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:46.954558 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" podStartSLOduration=1.922042511 podStartE2EDuration="2.954542134s" totalImagesPullingTime="1.032499623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:39:46.952670149 +0000 UTC m=+656.732194077" watchObservedRunningTime="2026-07-23 01:39:46.954542134 +0000 UTC m=+656.734066062" Jul 23 01:39:47.067051 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:47.067019 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:52.567708 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.567672 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm"] Jul 23 01:39:52.571406 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.571383 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.573827 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.573806 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 23 01:39:52.581657 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.581635 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm"] Jul 23 01:39:52.675840 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.675807 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dlpx\" (UniqueName: \"kubernetes.io/projected/06f82281-18c2-4760-bb30-edfce31c015e-kube-api-access-4dlpx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.676017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.675862 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f82281-18c2-4760-bb30-edfce31c015e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.676017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.675962 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.676017 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.676008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.676170 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.676072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.676170 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.676122 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776759 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776730 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f82281-18c2-4760-bb30-edfce31c015e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776920 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776776 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776920 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776798 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776920 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776823 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776920 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776841 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.776920 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.776865 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlpx\" (UniqueName: \"kubernetes.io/projected/06f82281-18c2-4760-bb30-edfce31c015e-kube-api-access-4dlpx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.777168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.777150 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.777272 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.777221 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.777312 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.777291 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.779020 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.778995 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f82281-18c2-4760-bb30-edfce31c015e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.779428 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.779407 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f82281-18c2-4760-bb30-edfce31c015e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.784434 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.784417 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dlpx\" (UniqueName: \"kubernetes.io/projected/06f82281-18c2-4760-bb30-edfce31c015e-kube-api-access-4dlpx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm\" (UID: \"06f82281-18c2-4760-bb30-edfce31c015e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:52.881599 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:52.881525 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:39:53.012502 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:53.012468 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm"] Jul 23 01:39:53.013210 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:39:53.013146 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f82281_18c2_4760_bb30_edfce31c015e.slice/crio-ff55422e36476cdbb4ce52c54d510eb3b5002580419763a42cb1518161e78948 WatchSource:0}: Error finding container ff55422e36476cdbb4ce52c54d510eb3b5002580419763a42cb1518161e78948: Status 404 returned error can't find the container with id ff55422e36476cdbb4ce52c54d510eb3b5002580419763a42cb1518161e78948 Jul 23 01:39:53.970704 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:53.970668 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" event={"ID":"06f82281-18c2-4760-bb30-edfce31c015e","Type":"ContainerStarted","Data":"de001e17ff4a267050b75ac9eb08ef1fd7da0c3d73bd8736f5d44cfa6a2899fe"} Jul 23 01:39:53.970704 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:53.970710 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" event={"ID":"06f82281-18c2-4760-bb30-edfce31c015e","Type":"ContainerStarted","Data":"ff55422e36476cdbb4ce52c54d510eb3b5002580419763a42cb1518161e78948"} Jul 23 01:39:53.987905 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:53.987856 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" podStartSLOduration=1.8156245709999999 podStartE2EDuration="1.987842401s" totalImagesPullingTime="172.21783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:39:53.986478197 +0000 UTC m=+663.766002117" watchObservedRunningTime="2026-07-23 01:39:53.987842401 +0000 UTC m=+663.767366307" Jul 23 01:39:55.303855 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:55.303825 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:55.304260 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:55.303903 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:55.320408 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:55.320384 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:39:55.862315 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:55.862280 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:39:55.993192 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:39:55.993151 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xchvr" Jul 23 01:40:02.882596 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:40:02.882555 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:40:02.882596 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:40:02.882606 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:40:02.895358 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:40:02.895331 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:40:03.031842 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:40:03.031809 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm" Jul 23 01:41:33.071134 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:41:33.071094 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:41:42.556603 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:41:42.556565 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:41:52.463512 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:41:52.463478 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:43:35.773484 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:43:35.773449 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:43:43.263118 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:43:43.263038 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:44:14.458378 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:44:14.458326 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:44:24.363708 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:44:24.363676 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:44:32.666916 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:44:32.666881 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:44:41.564067 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:44:41.564030 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:44:53.071447 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:44:53.071412 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:45:02.174729 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:45:02.174681 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:45:13.264918 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:45:13.264840 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:45:23.064191 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:45:23.064151 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:45:45.772950 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:45:45.772917 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:45:57.565065 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:45:57.565029 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:46:20.071621 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:46:20.071588 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:46:30.262802 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:46:30.262770 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:46:45.573559 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:46:45.573524 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:46:55.369947 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:46:55.369897 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:47:06.369012 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:06.368974 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:47:15.968488 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:15.968456 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:47:25.183556 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:25.183518 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:47:50.262757 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:50.262716 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:47:50.263344 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:50.262968 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" podUID="03efa629-27fc-4fc4-ab12-d8fff583d95e" containerName="manager" containerID="cri-o://9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21" gracePeriod=10 Jul 23 01:47:53.617002 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.616975 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:47:53.688844 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.688764 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume\") pod \"03efa629-27fc-4fc4-ab12-d8fff583d95e\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " Jul 23 01:47:53.688844 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.688818 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv2jb\" (UniqueName: \"kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb\") pod \"03efa629-27fc-4fc4-ab12-d8fff583d95e\" (UID: \"03efa629-27fc-4fc4-ab12-d8fff583d95e\") " Jul 23 01:47:53.689191 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.689150 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "03efa629-27fc-4fc4-ab12-d8fff583d95e" (UID: "03efa629-27fc-4fc4-ab12-d8fff583d95e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:47:53.690756 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.690733 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb" (OuterVolumeSpecName: "kube-api-access-mv2jb") pod "03efa629-27fc-4fc4-ab12-d8fff583d95e" (UID: "03efa629-27fc-4fc4-ab12-d8fff583d95e"). InnerVolumeSpecName "kube-api-access-mv2jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:47:53.790167 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.790132 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03efa629-27fc-4fc4-ab12-d8fff583d95e-extensions-socket-volume\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:47:53.790167 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:53.790158 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2jb\" (UniqueName: \"kubernetes.io/projected/03efa629-27fc-4fc4-ab12-d8fff583d95e-kube-api-access-mv2jb\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:47:54.211942 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.211905 2554 generic.go:356] "Generic (PLEG): container finished" podID="03efa629-27fc-4fc4-ab12-d8fff583d95e" containerID="9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21" exitCode=0 Jul 23 01:47:54.212111 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.211953 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" Jul 23 01:47:54.212111 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.211993 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" event={"ID":"03efa629-27fc-4fc4-ab12-d8fff583d95e","Type":"ContainerDied","Data":"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21"} Jul 23 01:47:54.212111 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.212045 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx" event={"ID":"03efa629-27fc-4fc4-ab12-d8fff583d95e","Type":"ContainerDied","Data":"ff96b5b8d86059bb388ed94228491c1572bd9ce7fbd73cb8fc080de0664c72c2"} Jul 23 01:47:54.212111 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.212088 2554 scope.go:117] "RemoveContainer" containerID="9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21" Jul 23 01:47:54.221512 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.221495 2554 scope.go:117] "RemoveContainer" containerID="9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21" Jul 23 01:47:54.221755 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:47:54.221736 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21\": container with ID starting with 9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21 not found: ID does not exist" containerID="9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21" Jul 23 01:47:54.221821 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.221763 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21"} err="failed to get container status \"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21\": rpc error: code = NotFound desc = could not find container \"9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21\": container with ID starting with 9004558b8a81df23702e854ae68875c4f854d5cb906ad9df92a32c2b21d07f21 not found: ID does not exist" Jul 23 01:47:54.233684 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.233663 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:47:54.239197 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.239163 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hhnqx"] Jul 23 01:47:54.769395 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:47:54.769362 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03efa629-27fc-4fc4-ab12-d8fff583d95e" path="/var/lib/kubelet/pods/03efa629-27fc-4fc4-ab12-d8fff583d95e/volumes" Jul 23 01:48:51.378981 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.378945 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l"] Jul 23 01:48:51.379396 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.379381 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03efa629-27fc-4fc4-ab12-d8fff583d95e" containerName="manager" Jul 23 01:48:51.379440 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.379400 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="03efa629-27fc-4fc4-ab12-d8fff583d95e" containerName="manager" Jul 23 01:48:51.379477 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.379470 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="03efa629-27fc-4fc4-ab12-d8fff583d95e" containerName="manager" Jul 23 01:48:51.382660 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.382639 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.385088 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.385071 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ggqtr\"" Jul 23 01:48:51.400301 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.400279 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l"] Jul 23 01:48:51.464944 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.464919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbb26eba-8089-4067-b0b2-082f6e5e06b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.465109 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.464964 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsnr\" (UniqueName: \"kubernetes.io/projected/dbb26eba-8089-4067-b0b2-082f6e5e06b9-kube-api-access-pqsnr\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.565790 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.565756 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbb26eba-8089-4067-b0b2-082f6e5e06b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.565945 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.565813 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsnr\" (UniqueName: \"kubernetes.io/projected/dbb26eba-8089-4067-b0b2-082f6e5e06b9-kube-api-access-pqsnr\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.566203 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.566160 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbb26eba-8089-4067-b0b2-082f6e5e06b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.574231 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.574207 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsnr\" (UniqueName: \"kubernetes.io/projected/dbb26eba-8089-4067-b0b2-082f6e5e06b9-kube-api-access-pqsnr\") pod \"kuadrant-operator-controller-manager-86878c769f-zh57l\" (UID: \"dbb26eba-8089-4067-b0b2-082f6e5e06b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.693317 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.693244 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:51.821379 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:51.821355 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l"] Jul 23 01:48:51.824125 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:48:51.824100 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb26eba_8089_4067_b0b2_082f6e5e06b9.slice/crio-43d84dbc09ca45868fb6d63b803205807425134c06b05fb5d68b62eaa52e4b63 WatchSource:0}: Error finding container 43d84dbc09ca45868fb6d63b803205807425134c06b05fb5d68b62eaa52e4b63: Status 404 returned error can't find the container with id 43d84dbc09ca45868fb6d63b803205807425134c06b05fb5d68b62eaa52e4b63 Jul 23 01:48:52.492750 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:52.492712 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" event={"ID":"dbb26eba-8089-4067-b0b2-082f6e5e06b9","Type":"ContainerStarted","Data":"1107921bc51267e2bd1b0a5e392dbdbb70833a3bda33a4df5ad2b5926b0700aa"} Jul 23 01:48:52.493134 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:52.492777 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:48:52.493134 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:52.492797 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" event={"ID":"dbb26eba-8089-4067-b0b2-082f6e5e06b9","Type":"ContainerStarted","Data":"43d84dbc09ca45868fb6d63b803205807425134c06b05fb5d68b62eaa52e4b63"} Jul 23 01:48:52.513904 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:48:52.513851 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" podStartSLOduration=1.513837122 podStartE2EDuration="1.513837122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:48:52.51220671 +0000 UTC m=+1202.291730641" watchObservedRunningTime="2026-07-23 01:48:52.513837122 +0000 UTC m=+1202.293361126" Jul 23 01:49:03.500122 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:03.500091 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zh57l" Jul 23 01:49:13.171774 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:13.171740 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:49:25.065715 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:25.065672 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:49:32.464957 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:32.464920 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:49:45.170099 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:45.170026 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:49:54.566474 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:49:54.566437 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:05.773679 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:05.773642 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:14.468930 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:14.468896 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:25.063071 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:25.063037 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:36.268724 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:36.268693 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:44.969725 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:44.969690 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:50:55.569860 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:50:55.569829 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:51:17.074470 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:17.074391 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:51:27.170091 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:27.170058 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:51:31.886457 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:31.886411 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:51:31.889964 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:31.889940 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:31.892319 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:31.892301 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-98bdf318-kserve-self-signed-certs\"" Jul 23 01:51:31.899402 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:31.899383 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:51:32.000091 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000055 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.000091 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000096 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.000343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000114 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9hc\" (UniqueName: \"kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.000343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000224 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.000343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000279 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.000343 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.000296 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100802 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100768 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100814 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9hc\" (UniqueName: \"kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100869 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.100984 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.100917 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.101281 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.101256 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.101363 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.101308 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.101423 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.101401 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.103186 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.103151 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.103339 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.103321 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.108337 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.108313 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9hc\" (UniqueName: \"kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc\") pod \"e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.200016 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.199939 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:32.334360 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:32.334334 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:51:32.336809 ip-10-0-136-186 kubenswrapper[2554]: W0723 01:51:32.336781 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod907fae9f_7e2f_41dd_9a07_30310b4c87b7.slice/crio-bab40d5fa4eb6d804da93992b70f3eeb0adb40b409bd432611b8373d9587ae51 WatchSource:0}: Error finding container bab40d5fa4eb6d804da93992b70f3eeb0adb40b409bd432611b8373d9587ae51: Status 404 returned error can't find the container with id bab40d5fa4eb6d804da93992b70f3eeb0adb40b409bd432611b8373d9587ae51 Jul 23 01:51:33.254535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:33.254502 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" event={"ID":"907fae9f-7e2f-41dd-9a07-30310b4c87b7","Type":"ContainerStarted","Data":"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049"} Jul 23 01:51:33.254535 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:33.254541 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" event={"ID":"907fae9f-7e2f-41dd-9a07-30310b4c87b7","Type":"ContainerStarted","Data":"bab40d5fa4eb6d804da93992b70f3eeb0adb40b409bd432611b8373d9587ae51"} Jul 23 01:51:33.271335 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:33.271276 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" podStartSLOduration=2.271259932 podStartE2EDuration="2.271259932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:51:33.269628534 +0000 UTC m=+1363.049152465" watchObservedRunningTime="2026-07-23 01:51:33.271259932 +0000 UTC m=+1363.050783860" Jul 23 01:51:42.200589 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:42.200549 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:42.201041 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:42.200604 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:42.213224 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:42.213196 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:42.309234 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:42.309203 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:51:45.264807 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:45.264770 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:51:58.591142 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:51:58.591108 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:52:00.965022 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:00.964986 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:52:13.613548 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.613506 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" podUID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" containerName="main" containerID="cri-o://0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049" gracePeriod=45 Jul 23 01:52:13.858718 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.858690 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:52:13.971340 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.971251 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:52:13.984967 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.984931 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr9hc\" (UniqueName: \"kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985093 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.984982 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985093 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985021 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985093 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985039 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985243 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985094 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985243 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985155 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs\") pod \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\" (UID: \"907fae9f-7e2f-41dd-9a07-30310b4c87b7\") " Jul 23 01:52:13.985359 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985321 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:52:13.985359 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985332 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:52:13.985359 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985343 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home" (OuterVolumeSpecName: "home") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:52:13.985519 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985498 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:13.985519 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985512 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:13.985607 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.985550 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:13.987346 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.987315 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:52:13.987346 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.987331 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc" (OuterVolumeSpecName: "kube-api-access-rr9hc") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "kube-api-access-rr9hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:52:13.987490 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:13.987323 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm" (OuterVolumeSpecName: "dshm") pod "907fae9f-7e2f-41dd-9a07-30310b4c87b7" (UID: "907fae9f-7e2f-41dd-9a07-30310b4c87b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:52:14.086732 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.086690 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907fae9f-7e2f-41dd-9a07-30310b4c87b7-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:14.086732 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.086723 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9hc\" (UniqueName: \"kubernetes.io/projected/907fae9f-7e2f-41dd-9a07-30310b4c87b7-kube-api-access-rr9hc\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:14.086732 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.086733 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907fae9f-7e2f-41dd-9a07-30310b4c87b7-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 01:52:14.456968 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.456932 2554 generic.go:356] "Generic (PLEG): container finished" podID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" containerID="0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049" exitCode=0 Jul 23 01:52:14.457168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.456980 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" Jul 23 01:52:14.457168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.457014 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" event={"ID":"907fae9f-7e2f-41dd-9a07-30310b4c87b7","Type":"ContainerDied","Data":"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049"} Jul 23 01:52:14.457168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.457060 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm" event={"ID":"907fae9f-7e2f-41dd-9a07-30310b4c87b7","Type":"ContainerDied","Data":"bab40d5fa4eb6d804da93992b70f3eeb0adb40b409bd432611b8373d9587ae51"} Jul 23 01:52:14.457168 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.457082 2554 scope.go:117] "RemoveContainer" containerID="0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049" Jul 23 01:52:14.466628 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.466612 2554 scope.go:117] "RemoveContainer" containerID="0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049" Jul 23 01:52:14.466934 ip-10-0-136-186 kubenswrapper[2554]: E0723 01:52:14.466917 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049\": container with ID starting with 0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049 not found: ID does not exist" containerID="0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049" Jul 23 01:52:14.466997 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.466941 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049"} err="failed to get container status \"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049\": rpc error: code = NotFound desc = could not find container \"0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049\": container with ID starting with 0f4e0dff42518aa16f854561aa958a598607df0f59a599ae81a97da0fd210049 not found: ID does not exist" Jul 23 01:52:14.479305 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.479278 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:52:14.482978 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.482955 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-98bdf318-kserve-9cf5c559-b6qxm"] Jul 23 01:52:14.769777 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:14.769682 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" path="/var/lib/kubelet/pods/907fae9f-7e2f-41dd-9a07-30310b4c87b7/volumes" Jul 23 01:52:20.663968 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:20.663932 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:52:36.165374 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:36.165343 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:52:44.972925 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:44.972846 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:52:55.070025 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:52:55.069990 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:02.665867 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:02.665830 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:13.171269 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:13.171240 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:22.264406 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:22.264364 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:35.476881 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:35.476844 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:44.569767 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:44.569733 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:53:54.663991 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:53:54.663955 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:54:06.768974 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:54:06.768932 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:54:15.878512 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:54:15.878441 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:54:22.066687 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:54:22.066652 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:54:38.786262 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:54:38.786234 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:54:50.480225 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:54:50.480186 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:55:01.287736 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:55:01.287699 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:55:09.389430 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:55:09.389393 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:55:23.286341 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:55:23.286304 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:55:32.190731 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:55:32.190695 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:55:46.971415 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:55:46.971330 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:56:20.471916 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:56:20.471881 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:56:31.063089 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:56:31.063057 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:57:04.360767 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:57:04.360732 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:57:14.966006 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:57:14.965920 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:57:48.167793 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:57:48.167763 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:58:00.475803 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:58:00.475762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:58:32.365775 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:58:32.365738 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 01:59:58.380027 ip-10-0-136-186 kubenswrapper[2554]: I0723 01:59:58.379994 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:00:10.070091 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:00:10.070061 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:00:21.966288 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:00:21.966211 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:00:38.862665 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:00:38.862632 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:01:13.584596 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:13.584557 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:01:21.187684 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.187644 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:01:21.188134 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.188059 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" containerName="main" Jul 23 02:01:21.188134 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.188076 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" containerName="main" Jul 23 02:01:21.188259 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.188212 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="907fae9f-7e2f-41dd-9a07-30310b4c87b7" containerName="main" Jul 23 02:01:21.191504 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.191482 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.194256 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.194233 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d207b89\"/\"kube-root-ca.crt\"" Jul 23 02:01:21.195414 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.195394 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d207b89\"/\"e2e-lifecycle-model-6d207b89-kserve-self-signed-certs\"" Jul 23 02:01:21.195533 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.195442 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d207b89\"/\"openshift-service-ca.crt\"" Jul 23 02:01:21.195533 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.195468 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d207b89\"/\"default-dockercfg-qpqf9\"" Jul 23 02:01:21.202339 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.202318 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:01:21.345882 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.345841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.346056 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.345891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27mm\" (UniqueName: \"kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.346056 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.345948 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.346056 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.346024 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.346056 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.346040 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.346232 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.346079 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.446734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446638 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27mm\" (UniqueName: \"kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.446734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.446734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447037 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446748 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447037 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446785 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447037 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.446922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447250 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.447124 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447314 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.447243 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.447314 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.447255 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.449073 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.449052 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.449269 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.449253 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.454692 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.454675 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27mm\" (UniqueName: \"kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm\") pod \"e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.502690 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.502653 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:21.635016 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:21.634991 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:01:21.637100 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:01:21.637072 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00394a3e_a699_48e2_991b_21736f5d15f4.slice/crio-92a3ee5cfc946bceb78da5973d324dedd4c67d92b9b1daa8b0cd5797e903c7fe WatchSource:0}: Error finding container 92a3ee5cfc946bceb78da5973d324dedd4c67d92b9b1daa8b0cd5797e903c7fe: Status 404 returned error can't find the container with id 92a3ee5cfc946bceb78da5973d324dedd4c67d92b9b1daa8b0cd5797e903c7fe Jul 23 02:01:22.052834 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:22.052802 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" event={"ID":"00394a3e-a699-48e2-991b-21736f5d15f4","Type":"ContainerStarted","Data":"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f"} Jul 23 02:01:22.053025 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:22.052843 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" event={"ID":"00394a3e-a699-48e2-991b-21736f5d15f4","Type":"ContainerStarted","Data":"92a3ee5cfc946bceb78da5973d324dedd4c67d92b9b1daa8b0cd5797e903c7fe"} Jul 23 02:01:22.070565 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:22.070510 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" podStartSLOduration=1.070491697 podStartE2EDuration="1.070491697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:01:22.070304557 +0000 UTC m=+1951.849828487" watchObservedRunningTime="2026-07-23 02:01:22.070491697 +0000 UTC m=+1951.850015607" Jul 23 02:01:25.477923 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:25.477882 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:01:31.503410 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:31.503367 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:31.503864 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:31.503588 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:31.516247 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:31.516222 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:32.111048 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:32.111019 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:01:37.776777 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:37.776743 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:01:55.586004 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:01:55.585909 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:02:06.174886 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:06.174839 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:02:08.524327 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:08.524290 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:02:23.545698 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.545643 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" podUID="00394a3e-a699-48e2-991b-21736f5d15f4" containerName="main" containerID="cri-o://cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f" gracePeriod=45 Jul 23 02:02:23.788340 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.788315 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:02:23.907251 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907216 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907422 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907264 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x27mm\" (UniqueName: \"kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907422 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907292 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907422 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907317 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907555 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907522 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907641 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907529 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:02:23.907703 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907651 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home\") pod \"00394a3e-a699-48e2-991b-21736f5d15f4\" (UID: \"00394a3e-a699-48e2-991b-21736f5d15f4\") " Jul 23 02:02:23.907762 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907548 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:02:23.907859 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907836 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home" (OuterVolumeSpecName: "home") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:02:23.908000 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.907979 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:23.908000 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.908003 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:23.908137 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.908016 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:23.909536 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.909512 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm" (OuterVolumeSpecName: "dshm") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:02:23.909629 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.909555 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:02:23.909629 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:23.909568 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm" (OuterVolumeSpecName: "kube-api-access-x27mm") pod "00394a3e-a699-48e2-991b-21736f5d15f4" (UID: "00394a3e-a699-48e2-991b-21736f5d15f4"). InnerVolumeSpecName "kube-api-access-x27mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:02:24.008721 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.008688 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27mm\" (UniqueName: \"kubernetes.io/projected/00394a3e-a699-48e2-991b-21736f5d15f4-kube-api-access-x27mm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:24.008721 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.008717 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00394a3e-a699-48e2-991b-21736f5d15f4-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:24.008721 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.008726 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00394a3e-a699-48e2-991b-21736f5d15f4-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:02:24.353609 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.353574 2554 generic.go:356] "Generic (PLEG): container finished" podID="00394a3e-a699-48e2-991b-21736f5d15f4" containerID="cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f" exitCode=0 Jul 23 02:02:24.353786 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.353616 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" Jul 23 02:02:24.353786 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.353653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" event={"ID":"00394a3e-a699-48e2-991b-21736f5d15f4","Type":"ContainerDied","Data":"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f"} Jul 23 02:02:24.353786 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.353693 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7" event={"ID":"00394a3e-a699-48e2-991b-21736f5d15f4","Type":"ContainerDied","Data":"92a3ee5cfc946bceb78da5973d324dedd4c67d92b9b1daa8b0cd5797e903c7fe"} Jul 23 02:02:24.353786 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.353710 2554 scope.go:117] "RemoveContainer" containerID="cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f" Jul 23 02:02:24.362593 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.362578 2554 scope.go:117] "RemoveContainer" containerID="cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f" Jul 23 02:02:24.362865 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:02:24.362849 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f\": container with ID starting with cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f not found: ID does not exist" containerID="cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f" Jul 23 02:02:24.362923 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.362872 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f"} err="failed to get container status \"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f\": rpc error: code = NotFound desc = could not find container \"cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f\": container with ID starting with cda778e3561036effb0c03c2f892118f155ded6a0053452b976fbbf307f0d07f not found: ID does not exist" Jul 23 02:02:24.374528 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.374507 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:02:24.378019 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.377998 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6d207b89/e2e-lifecycle-model-6d207b89-kserve-6bd757d5c5-lsqf7"] Jul 23 02:02:24.769162 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:24.769084 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00394a3e-a699-48e2-991b-21736f5d15f4" path="/var/lib/kubelet/pods/00394a3e-a699-48e2-991b-21736f5d15f4/volumes" Jul 23 02:02:54.977714 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:02:54.977670 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:03:06.976593 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:03:06.976560 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:03:54.868806 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:03:54.868772 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:04:03.872660 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:03.872624 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:04:09.967818 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.967784 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:04:09.968226 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.968209 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00394a3e-a699-48e2-991b-21736f5d15f4" containerName="main" Jul 23 02:04:09.968226 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.968221 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="00394a3e-a699-48e2-991b-21736f5d15f4" containerName="main" Jul 23 02:04:09.968321 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.968286 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="00394a3e-a699-48e2-991b-21736f5d15f4" containerName="main" Jul 23 02:04:09.970808 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.970788 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:09.973614 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.973592 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99eab2\"/\"openshift-service-ca.crt\"" Jul 23 02:04:09.973704 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.973634 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99eab2\"/\"kube-root-ca.crt\"" Jul 23 02:04:09.973704 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.973634 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99eab2\"/\"default-dockercfg-kk85q\"" Jul 23 02:04:09.974709 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.974691 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99eab2\"/\"test-model-99eab2-kserve-self-signed-certs\"" Jul 23 02:04:09.982212 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:09.981850 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:04:10.077897 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.077865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.078061 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.077912 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.078061 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.077930 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.078061 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.078018 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.078168 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.078061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.078168 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.078106 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4v4d\" (UniqueName: \"kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179444 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179568 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179459 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179568 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179640 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179591 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179680 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179743 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4v4d\" (UniqueName: \"kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179801 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179781 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179953 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179933 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.179995 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.179959 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.181677 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.181654 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.182050 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.182030 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.191306 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.191284 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4v4d\" (UniqueName: \"kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d\") pod \"test-model-99eab2-kserve-5974cfbc45-trj9b\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.281331 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.281266 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:10.410516 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.410490 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:04:10.413701 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:04:10.413673 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2032e99_8218_4760_b53d_a4526377e6d0.slice/crio-e03f34e764bc208dd9c813569b90ec1a7a8bf76ef3acd170bae2c1764f16d1b8 WatchSource:0}: Error finding container e03f34e764bc208dd9c813569b90ec1a7a8bf76ef3acd170bae2c1764f16d1b8: Status 404 returned error can't find the container with id e03f34e764bc208dd9c813569b90ec1a7a8bf76ef3acd170bae2c1764f16d1b8 Jul 23 02:04:10.849916 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.849883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" event={"ID":"b2032e99-8218-4760-b53d-a4526377e6d0","Type":"ContainerStarted","Data":"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7"} Jul 23 02:04:10.849916 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.849919 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" event={"ID":"b2032e99-8218-4760-b53d-a4526377e6d0","Type":"ContainerStarted","Data":"e03f34e764bc208dd9c813569b90ec1a7a8bf76ef3acd170bae2c1764f16d1b8"} Jul 23 02:04:10.869268 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:10.869219 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" podStartSLOduration=1.869205858 podStartE2EDuration="1.869205858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:04:10.867778988 +0000 UTC m=+2120.647302943" watchObservedRunningTime="2026-07-23 02:04:10.869205858 +0000 UTC m=+2120.648729787" Jul 23 02:04:12.666363 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:12.666322 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:04:20.281904 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:20.281855 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:20.282322 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:20.281927 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:20.294264 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:20.294241 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:20.908251 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:20.908222 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:04:26.364221 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:26.364187 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:04:40.478560 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:04:40.478524 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:05:16.370577 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:16.370539 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:05:20.969927 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:20.969893 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:05:31.390923 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.390849 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" podUID="b2032e99-8218-4760-b53d-a4526377e6d0" containerName="main" containerID="cri-o://077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7" gracePeriod=45 Jul 23 02:05:31.645834 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.645770 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:05:31.819028 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.818992 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819028 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819034 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819284 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819080 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819284 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819136 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4v4d\" (UniqueName: \"kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819284 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819168 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819284 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819230 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache\") pod \"b2032e99-8218-4760-b53d-a4526377e6d0\" (UID: \"b2032e99-8218-4760-b53d-a4526377e6d0\") " Jul 23 02:05:31.819508 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819426 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home" (OuterVolumeSpecName: "home") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:05:31.819603 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819583 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:31.819658 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819595 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:05:31.819723 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.819657 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:05:31.821410 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.821371 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:05:31.821516 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.821457 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm" (OuterVolumeSpecName: "dshm") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:05:31.821516 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.821456 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d" (OuterVolumeSpecName: "kube-api-access-w4v4d") pod "b2032e99-8218-4760-b53d-a4526377e6d0" (UID: "b2032e99-8218-4760-b53d-a4526377e6d0"). InnerVolumeSpecName "kube-api-access-w4v4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:05:31.920545 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.920463 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4v4d\" (UniqueName: \"kubernetes.io/projected/b2032e99-8218-4760-b53d-a4526377e6d0-kube-api-access-w4v4d\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:31.920545 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.920491 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:31.920545 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.920503 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:31.920545 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.920511 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2032e99-8218-4760-b53d-a4526377e6d0-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:31.920545 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:31.920520 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2032e99-8218-4760-b53d-a4526377e6d0-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:05:32.248375 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.248283 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2032e99-8218-4760-b53d-a4526377e6d0" containerID="077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7" exitCode=0 Jul 23 02:05:32.248375 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.248353 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" Jul 23 02:05:32.248637 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.248378 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" event={"ID":"b2032e99-8218-4760-b53d-a4526377e6d0","Type":"ContainerDied","Data":"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7"} Jul 23 02:05:32.248637 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.248427 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b" event={"ID":"b2032e99-8218-4760-b53d-a4526377e6d0","Type":"ContainerDied","Data":"e03f34e764bc208dd9c813569b90ec1a7a8bf76ef3acd170bae2c1764f16d1b8"} Jul 23 02:05:32.248637 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.248449 2554 scope.go:117] "RemoveContainer" containerID="077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7" Jul 23 02:05:32.258642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.258625 2554 scope.go:117] "RemoveContainer" containerID="077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7" Jul 23 02:05:32.258925 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:05:32.258907 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7\": container with ID starting with 077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7 not found: ID does not exist" containerID="077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7" Jul 23 02:05:32.258997 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.258931 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7"} err="failed to get container status \"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7\": rpc error: code = NotFound desc = could not find container \"077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7\": container with ID starting with 077bfdc4c7cac234691ef98964e2acb0a8abc4e56a5f6013ffeeab30d9cf6cc7 not found: ID does not exist" Jul 23 02:05:32.271246 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.271222 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:05:32.273995 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.273974 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-99eab2/test-model-99eab2-kserve-5974cfbc45-trj9b"] Jul 23 02:05:32.768862 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:32.768824 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2032e99-8218-4760-b53d-a4526377e6d0" path="/var/lib/kubelet/pods/b2032e99-8218-4760-b53d-a4526377e6d0/volumes" Jul 23 02:05:36.475044 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:36.475005 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:05:52.861417 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:05:52.861379 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:06:11.966253 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:11.966212 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:06:27.469815 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:27.469739 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:06:51.673454 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:51.673420 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:06:56.713148 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.713118 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b"] Jul 23 02:06:56.713568 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.713553 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2032e99-8218-4760-b53d-a4526377e6d0" containerName="main" Jul 23 02:06:56.713624 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.713570 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2032e99-8218-4760-b53d-a4526377e6d0" containerName="main" Jul 23 02:06:56.713679 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.713663 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2032e99-8218-4760-b53d-a4526377e6d0" containerName="main" Jul 23 02:06:56.716692 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.716674 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:06:56.719113 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.719088 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:06:56.719211 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.719112 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 02:06:56.723387 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.723366 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b"] Jul 23 02:06:56.799010 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.798973 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b\" (UID: \"4617502b-c7f8-4f62-ae03-b4cfa919989e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:06:56.900452 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.900415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b\" (UID: \"4617502b-c7f8-4f62-ae03-b4cfa919989e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:06:56.901054 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:56.901032 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b\" (UID: \"4617502b-c7f8-4f62-ae03-b4cfa919989e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:06:57.028537 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:57.028436 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:06:57.165691 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:57.165656 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b"] Jul 23 02:06:57.169335 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:06:57.169300 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4617502b_c7f8_4f62_ae03_b4cfa919989e.slice/crio-275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df WatchSource:0}: Error finding container 275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df: Status 404 returned error can't find the container with id 275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df Jul 23 02:06:57.171363 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:57.171342 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 02:06:57.668398 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:06:57.668362 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" event={"ID":"4617502b-c7f8-4f62-ae03-b4cfa919989e","Type":"ContainerStarted","Data":"275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df"} Jul 23 02:07:01.691683 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:01.691647 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" event={"ID":"4617502b-c7f8-4f62-ae03-b4cfa919989e","Type":"ContainerStarted","Data":"df6704fc499ecdd44781b374ab011fac8bd64e536559ca979054b4212551a677"} Jul 23 02:07:01.707483 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:01.707430 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" podStartSLOduration=1.684257404 podStartE2EDuration="5.707413913s" totalImagesPullingTime="4.023156509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:07:01.705421096 +0000 UTC m=+2291.484945025" watchObservedRunningTime="2026-07-23 02:07:01.707413913 +0000 UTC m=+2291.486937843" Jul 23 02:07:02.697701 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:02.697666 2554 generic.go:356] "Generic (PLEG): container finished" podID="4617502b-c7f8-4f62-ae03-b4cfa919989e" containerID="df6704fc499ecdd44781b374ab011fac8bd64e536559ca979054b4212551a677" exitCode=0 Jul 23 02:07:02.698157 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:02.697748 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" event={"ID":"4617502b-c7f8-4f62-ae03-b4cfa919989e","Type":"ContainerDied","Data":"df6704fc499ecdd44781b374ab011fac8bd64e536559ca979054b4212551a677"} Jul 23 02:07:03.831409 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:03.831383 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:07:03.864933 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:03.864891 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca\") pod \"4617502b-c7f8-4f62-ae03-b4cfa919989e\" (UID: \"4617502b-c7f8-4f62-ae03-b4cfa919989e\") " Jul 23 02:07:03.865297 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:03.865270 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4617502b-c7f8-4f62-ae03-b4cfa919989e" (UID: "4617502b-c7f8-4f62-ae03-b4cfa919989e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:07:03.966330 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:03.966243 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4617502b-c7f8-4f62-ae03-b4cfa919989e-maas-api-service-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:07:04.707511 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:04.707482 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" Jul 23 02:07:04.707693 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:04.707477 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b" event={"ID":"4617502b-c7f8-4f62-ae03-b4cfa919989e","Type":"ContainerDied","Data":"275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df"} Jul 23 02:07:04.707693 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:04.707606 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275e0258d7986cfe415d66fab385c982503b13df83248eaf4badc599b012e3df" Jul 23 02:07:16.790529 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:16.790493 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b"] Jul 23 02:07:16.794502 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:16.794479 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf5ce0-txm2b"] Jul 23 02:07:17.166108 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:17.166070 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:07:18.768742 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:18.768711 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4617502b-c7f8-4f62-ae03-b4cfa919989e" path="/var/lib/kubelet/pods/4617502b-c7f8-4f62-ae03-b4cfa919989e/volumes" Jul 23 02:07:31.782859 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:31.782828 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:07:40.270441 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:40.270407 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:07:50.583226 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.583134 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:07:50.583602 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.583569 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4617502b-c7f8-4f62-ae03-b4cfa919989e" containerName="revoke-keys" Jul 23 02:07:50.583602 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.583584 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4617502b-c7f8-4f62-ae03-b4cfa919989e" containerName="revoke-keys" Jul 23 02:07:50.583675 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.583661 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4617502b-c7f8-4f62-ae03-b4cfa919989e" containerName="revoke-keys" Jul 23 02:07:50.591858 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.591834 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.595719 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.595693 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"auth-test-model-1dd296-kserve-self-signed-certs\"" Jul 23 02:07:50.595858 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.595693 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"default-dockercfg-jx4h2\"" Jul 23 02:07:50.595858 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.595696 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"kube-root-ca.crt\"" Jul 23 02:07:50.595858 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.595699 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"openshift-service-ca.crt\"" Jul 23 02:07:50.599768 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.599735 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:07:50.694353 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694320 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.694518 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694371 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.694518 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694388 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.694518 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694408 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.694518 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694454 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.694518 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.694499 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf78v\" (UniqueName: \"kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795127 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795099 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795127 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795129 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795153 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795249 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf78v\" (UniqueName: \"kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795332 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795592 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795553 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795685 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795596 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.795728 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.795683 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.797391 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.797370 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.797569 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.797543 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.804947 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.804921 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf78v\" (UniqueName: \"kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v\") pod \"auth-test-model-1dd296-kserve-59bf6c4667-4lsh9\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:50.905349 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:50.905274 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:07:51.039367 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:51.039294 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:07:51.041749 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:07:51.041720 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f1d33f0_e625_417f_94ea_a3130ee44803.slice/crio-c14004583612455872b5ae37b67f785f976f222a21d43fb5738ab8010ed2f2bb WatchSource:0}: Error finding container c14004583612455872b5ae37b67f785f976f222a21d43fb5738ab8010ed2f2bb: Status 404 returned error can't find the container with id c14004583612455872b5ae37b67f785f976f222a21d43fb5738ab8010ed2f2bb Jul 23 02:07:51.935656 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:51.935620 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" event={"ID":"2f1d33f0-e625-417f-94ea-a3130ee44803","Type":"ContainerStarted","Data":"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636"} Jul 23 02:07:51.935656 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:51.935661 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" event={"ID":"2f1d33f0-e625-417f-94ea-a3130ee44803","Type":"ContainerStarted","Data":"c14004583612455872b5ae37b67f785f976f222a21d43fb5738ab8010ed2f2bb"} Jul 23 02:07:51.955397 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:51.955349 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" podStartSLOduration=1.955332795 podStartE2EDuration="1.955332795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:07:51.955202923 +0000 UTC m=+2341.734726853" watchObservedRunningTime="2026-07-23 02:07:51.955332795 +0000 UTC m=+2341.734856726" Jul 23 02:07:53.876418 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:07:53.876385 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:08:00.906471 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:00.906437 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:08:00.906851 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:00.906484 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:08:00.921291 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:00.921262 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:08:00.990693 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:00.990660 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:08:13.765917 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:13.765882 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:08:30.668010 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:30.667970 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:08:49.580659 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:08:49.580619 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:09:05.372275 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:09:05.372241 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:09:24.067532 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:09:24.067446 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:09:44.265580 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:09:44.265548 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:10:02.968121 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:02.968088 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:10:19.192370 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:19.192334 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:10:21.476819 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:21.476780 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:10:22.689732 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.689700 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:10:22.693277 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.693255 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.695604 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.695582 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"sub-test-model-1dd296-kserve-self-signed-certs\"" Jul 23 02:10:22.702277 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.702255 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:10:22.801814 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801782 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.801957 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801825 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.801957 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801849 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8bb\" (UniqueName: \"kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.801957 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801907 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.801957 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.802091 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.801977 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902548 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902520 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8bb\" (UniqueName: \"kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902703 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902577 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902703 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902638 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902703 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902676 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902856 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902713 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.902906 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.902856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.903066 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.903043 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.903133 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.903062 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.903209 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.903194 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.904974 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.904941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.905249 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.905233 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:22.910714 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:22.910691 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8bb\" (UniqueName: \"kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb\") pod \"sub-test-model-1dd296-kserve-794c4d86f5-hnvdx\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:23.003084 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:23.002996 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:23.135853 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:23.135824 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:10:23.137541 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:10:23.137512 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52374aaf_8b0e_43ed_b0ca_de63812458f7.slice/crio-c76cdf9a0193e42fe262adb3b30667a0eb7172a86124697b0b249f6c902ad1d2 WatchSource:0}: Error finding container c76cdf9a0193e42fe262adb3b30667a0eb7172a86124697b0b249f6c902ad1d2: Status 404 returned error can't find the container with id c76cdf9a0193e42fe262adb3b30667a0eb7172a86124697b0b249f6c902ad1d2 Jul 23 02:10:23.659352 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:23.659311 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" event={"ID":"52374aaf-8b0e-43ed-b0ca-de63812458f7","Type":"ContainerStarted","Data":"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378"} Jul 23 02:10:23.659523 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:23.659360 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" event={"ID":"52374aaf-8b0e-43ed-b0ca-de63812458f7","Type":"ContainerStarted","Data":"c76cdf9a0193e42fe262adb3b30667a0eb7172a86124697b0b249f6c902ad1d2"} Jul 23 02:10:23.677360 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:23.677312 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" podStartSLOduration=1.677297146 podStartE2EDuration="1.677297146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:10:23.676912619 +0000 UTC m=+2493.456436548" watchObservedRunningTime="2026-07-23 02:10:23.677297146 +0000 UTC m=+2493.456821075" Jul 23 02:10:33.003745 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:33.003709 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:33.004140 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:33.003759 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:33.015961 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:33.015934 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:33.716940 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:33.716902 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:10:34.212528 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.212486 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" podUID="2f1d33f0-e625-417f-94ea-a3130ee44803" containerName="main" containerID="cri-o://bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636" gracePeriod=45 Jul 23 02:10:34.462636 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.462577 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:10:34.510269 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510234 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510433 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510311 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510433 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510335 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510433 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510350 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510433 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510385 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf78v\" (UniqueName: \"kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510433 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510413 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache\") pod \"2f1d33f0-e625-417f-94ea-a3130ee44803\" (UID: \"2f1d33f0-e625-417f-94ea-a3130ee44803\") " Jul 23 02:10:34.510706 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510481 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home" (OuterVolumeSpecName: "home") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:10:34.510706 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510609 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:10:34.510706 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510666 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:10:34.510873 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510741 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.510873 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510760 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.510873 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.510775 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.512771 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.512736 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v" (OuterVolumeSpecName: "kube-api-access-jf78v") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "kube-api-access-jf78v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:10:34.513024 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.512997 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:10:34.513105 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.513000 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm" (OuterVolumeSpecName: "dshm") pod "2f1d33f0-e625-417f-94ea-a3130ee44803" (UID: "2f1d33f0-e625-417f-94ea-a3130ee44803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:10:34.611859 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.611827 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1d33f0-e625-417f-94ea-a3130ee44803-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.611859 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.611855 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1d33f0-e625-417f-94ea-a3130ee44803-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.612040 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.611870 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf78v\" (UniqueName: \"kubernetes.io/projected/2f1d33f0-e625-417f-94ea-a3130ee44803-kube-api-access-jf78v\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:10:34.713510 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.713417 2554 generic.go:356] "Generic (PLEG): container finished" podID="2f1d33f0-e625-417f-94ea-a3130ee44803" containerID="bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636" exitCode=0 Jul 23 02:10:34.713510 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.713463 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" Jul 23 02:10:34.713734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.713498 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" event={"ID":"2f1d33f0-e625-417f-94ea-a3130ee44803","Type":"ContainerDied","Data":"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636"} Jul 23 02:10:34.713734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.713550 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9" event={"ID":"2f1d33f0-e625-417f-94ea-a3130ee44803","Type":"ContainerDied","Data":"c14004583612455872b5ae37b67f785f976f222a21d43fb5738ab8010ed2f2bb"} Jul 23 02:10:34.713734 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.713569 2554 scope.go:117] "RemoveContainer" containerID="bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636" Jul 23 02:10:34.724646 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.724628 2554 scope.go:117] "RemoveContainer" containerID="bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636" Jul 23 02:10:34.724911 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:10:34.724892 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636\": container with ID starting with bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636 not found: ID does not exist" containerID="bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636" Jul 23 02:10:34.724963 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.724919 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636"} err="failed to get container status \"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636\": rpc error: code = NotFound desc = could not find container \"bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636\": container with ID starting with bec5a6457da17357ef48171c959fe2262e26ec0c114e6fe064028f5a0f360636 not found: ID does not exist" Jul 23 02:10:34.735591 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.735567 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:10:34.739869 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.739841 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/auth-test-model-1dd296-kserve-59bf6c4667-4lsh9"] Jul 23 02:10:34.768789 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:34.768758 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1d33f0-e625-417f-94ea-a3130ee44803" path="/var/lib/kubelet/pods/2f1d33f0-e625-417f-94ea-a3130ee44803/volumes" Jul 23 02:10:35.265219 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:35.265186 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:10:50.692526 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.692446 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:10:50.692969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.692860 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1d33f0-e625-417f-94ea-a3130ee44803" containerName="main" Jul 23 02:10:50.692969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.692873 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1d33f0-e625-417f-94ea-a3130ee44803" containerName="main" Jul 23 02:10:50.692969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.692937 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1d33f0-e625-417f-94ea-a3130ee44803" containerName="main" Jul 23 02:10:50.697145 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.697125 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.702868 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.702840 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5fbcb9\"/\"kube-root-ca.crt\"" Jul 23 02:10:50.703124 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.703108 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5fbcb9\"/\"default-dockercfg-x66bf\"" Jul 23 02:10:50.704070 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.704051 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5fbcb9\"/\"openshift-service-ca.crt\"" Jul 23 02:10:50.704271 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.704161 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5fbcb9\"/\"sub-test-model-5fbcb9-kserve-self-signed-certs\"" Jul 23 02:10:50.714774 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.714727 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:10:50.756859 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.756832 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.756969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.756885 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzkt\" (UniqueName: \"kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.756969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.756944 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.757051 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.756973 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.757051 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.756998 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.757115 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.757071 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.857630 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.857599 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.857780 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.857665 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzkt\" (UniqueName: \"kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.857857 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.857831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.857917 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.857887 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.857973 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.857940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.858106 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.858027 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.858276 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.858230 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.858276 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.858265 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.858442 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.858338 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.859886 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.859864 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.860207 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.860190 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:50.876068 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:50.876042 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzkt\" (UniqueName: \"kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt\") pod \"sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:51.007111 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:51.007042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:10:51.143330 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:51.142867 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:10:51.801413 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:51.801374 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" event={"ID":"2d45a535-e281-4f50-94cc-0d5ec8098fcb","Type":"ContainerStarted","Data":"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859"} Jul 23 02:10:51.801789 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:51.801422 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" event={"ID":"2d45a535-e281-4f50-94cc-0d5ec8098fcb","Type":"ContainerStarted","Data":"22b39d453baf33284398fa94e743abb6d5281692ca0453e15cefcfe7983ee7ed"} Jul 23 02:10:51.819383 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:51.819341 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" podStartSLOduration=1.819325759 podStartE2EDuration="1.819325759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:10:51.818293131 +0000 UTC m=+2521.597817072" watchObservedRunningTime="2026-07-23 02:10:51.819325759 +0000 UTC m=+2521.598849688" Jul 23 02:10:54.772283 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:10:54.772251 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:11:01.007575 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:01.007541 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:01.008034 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:01.007616 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:01.020691 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:01.020665 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:01.861416 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:01.861384 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:08.074090 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:08.074056 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:11:23.606230 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:23.606187 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:11:24.688981 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:24.688945 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:11:28.173446 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:28.173410 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:11:38.626445 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:38.626382 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" podUID="52374aaf-8b0e-43ed-b0ca-de63812458f7" containerName="main" containerID="cri-o://1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378" gracePeriod=45 Jul 23 02:11:38.884865 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:38.884806 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:11:39.002447 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002416 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002493 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002541 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002564 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002613 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k8bb\" (UniqueName: \"kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002642 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002641 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm\") pod \"52374aaf-8b0e-43ed-b0ca-de63812458f7\" (UID: \"52374aaf-8b0e-43ed-b0ca-de63812458f7\") " Jul 23 02:11:39.002928 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002668 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:39.002928 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002811 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:39.002928 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002855 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home" (OuterVolumeSpecName: "home") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:39.002928 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002919 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.003072 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.002940 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.004666 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.004645 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm" (OuterVolumeSpecName: "dshm") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:39.004666 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.004656 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:11:39.004832 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.004703 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb" (OuterVolumeSpecName: "kube-api-access-5k8bb") pod "52374aaf-8b0e-43ed-b0ca-de63812458f7" (UID: "52374aaf-8b0e-43ed-b0ca-de63812458f7"). InnerVolumeSpecName "kube-api-access-5k8bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:11:39.026363 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.026335 2554 generic.go:356] "Generic (PLEG): container finished" podID="52374aaf-8b0e-43ed-b0ca-de63812458f7" containerID="1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378" exitCode=0 Jul 23 02:11:39.026480 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.026392 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" Jul 23 02:11:39.026480 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.026393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" event={"ID":"52374aaf-8b0e-43ed-b0ca-de63812458f7","Type":"ContainerDied","Data":"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378"} Jul 23 02:11:39.026562 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.026503 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx" event={"ID":"52374aaf-8b0e-43ed-b0ca-de63812458f7","Type":"ContainerDied","Data":"c76cdf9a0193e42fe262adb3b30667a0eb7172a86124697b0b249f6c902ad1d2"} Jul 23 02:11:39.026562 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.026524 2554 scope.go:117] "RemoveContainer" containerID="1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378" Jul 23 02:11:39.036116 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.036098 2554 scope.go:117] "RemoveContainer" containerID="1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378" Jul 23 02:11:39.036436 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:11:39.036411 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378\": container with ID starting with 1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378 not found: ID does not exist" containerID="1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378" Jul 23 02:11:39.036540 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.036446 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378"} err="failed to get container status \"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378\": rpc error: code = NotFound desc = could not find container \"1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378\": container with ID starting with 1930449ec0a5ab9e4da59ae5dd341d4445d434fa76f75474345c1f079e941378 not found: ID does not exist" Jul 23 02:11:39.048280 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.048256 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:11:39.052834 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.052811 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/sub-test-model-1dd296-kserve-794c4d86f5-hnvdx"] Jul 23 02:11:39.103907 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.103871 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.103907 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.103903 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8bb\" (UniqueName: \"kubernetes.io/projected/52374aaf-8b0e-43ed-b0ca-de63812458f7-kube-api-access-5k8bb\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.103907 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.103915 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52374aaf-8b0e-43ed-b0ca-de63812458f7-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.104121 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.103926 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52374aaf-8b0e-43ed-b0ca-de63812458f7-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:39.710477 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.710434 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" podUID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" containerName="main" containerID="cri-o://90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859" gracePeriod=45 Jul 23 02:11:39.953744 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:39.953719 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:40.033742 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.033642 2554 generic.go:356] "Generic (PLEG): container finished" podID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" containerID="90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859" exitCode=0 Jul 23 02:11:40.033742 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.033691 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" Jul 23 02:11:40.033973 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.033733 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" event={"ID":"2d45a535-e281-4f50-94cc-0d5ec8098fcb","Type":"ContainerDied","Data":"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859"} Jul 23 02:11:40.033973 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.033776 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h" event={"ID":"2d45a535-e281-4f50-94cc-0d5ec8098fcb","Type":"ContainerDied","Data":"22b39d453baf33284398fa94e743abb6d5281692ca0453e15cefcfe7983ee7ed"} Jul 23 02:11:40.033973 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.033793 2554 scope.go:117] "RemoveContainer" containerID="90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859" Jul 23 02:11:40.042224 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.042206 2554 scope.go:117] "RemoveContainer" containerID="90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859" Jul 23 02:11:40.042447 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:11:40.042432 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859\": container with ID starting with 90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859 not found: ID does not exist" containerID="90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859" Jul 23 02:11:40.042498 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.042455 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859"} err="failed to get container status \"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859\": rpc error: code = NotFound desc = could not find container \"90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859\": container with ID starting with 90e98144d9d83c0b530f439a290efea0faf4f3e3a199321c0affc90884c08859 not found: ID does not exist" Jul 23 02:11:40.114248 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114216 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114407 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114261 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114407 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114292 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114407 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114352 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114407 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114403 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114623 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114436 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gzkt\" (UniqueName: \"kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt\") pod \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\" (UID: \"2d45a535-e281-4f50-94cc-0d5ec8098fcb\") " Jul 23 02:11:40.114623 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114549 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home" (OuterVolumeSpecName: "home") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:40.114623 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114560 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:40.114774 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114694 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:40.114868 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114850 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.114936 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114873 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.114936 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.114887 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.116677 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.116651 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm" (OuterVolumeSpecName: "dshm") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:11:40.116677 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.116667 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt" (OuterVolumeSpecName: "kube-api-access-7gzkt") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "kube-api-access-7gzkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:11:40.116817 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.116706 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d45a535-e281-4f50-94cc-0d5ec8098fcb" (UID: "2d45a535-e281-4f50-94cc-0d5ec8098fcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:11:40.216427 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.216397 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d45a535-e281-4f50-94cc-0d5ec8098fcb-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.216427 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.216424 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzkt\" (UniqueName: \"kubernetes.io/projected/2d45a535-e281-4f50-94cc-0d5ec8098fcb-kube-api-access-7gzkt\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.216427 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.216434 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d45a535-e281-4f50-94cc-0d5ec8098fcb-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:11:40.354885 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.354855 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:11:40.357947 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.357925 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5fbcb9/sub-test-model-5fbcb9-kserve-9c9b97fb5-pkm2h"] Jul 23 02:11:40.769415 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.769337 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" path="/var/lib/kubelet/pods/2d45a535-e281-4f50-94cc-0d5ec8098fcb/volumes" Jul 23 02:11:40.769756 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:40.769722 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52374aaf-8b0e-43ed-b0ca-de63812458f7" path="/var/lib/kubelet/pods/52374aaf-8b0e-43ed-b0ca-de63812458f7/volumes" Jul 23 02:11:59.189728 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.189695 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190057 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52374aaf-8b0e-43ed-b0ca-de63812458f7" containerName="main" Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190069 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52374aaf-8b0e-43ed-b0ca-de63812458f7" containerName="main" Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190086 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" containerName="main" Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190091 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" containerName="main" Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190147 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52374aaf-8b0e-43ed-b0ca-de63812458f7" containerName="main" Jul 23 02:11:59.190217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.190156 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d45a535-e281-4f50-94cc-0d5ec8098fcb" containerName="main" Jul 23 02:11:59.193257 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.193237 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.195742 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.195721 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"openshift-service-ca.crt\"" Jul 23 02:11:59.196738 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.196685 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"kube-root-ca.crt\"" Jul 23 02:11:59.196858 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.196804 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"default-dockercfg-jx4h2\"" Jul 23 02:11:59.196930 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.196852 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1dd296\"/\"rate-test-model-1dd296-kserve-self-signed-certs\"" Jul 23 02:11:59.203169 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.203146 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:11:59.272721 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.272894 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272732 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.272894 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272752 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.272894 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.272894 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272842 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlpl\" (UniqueName: \"kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.272894 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.272868 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373506 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373477 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373634 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373514 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlpl\" (UniqueName: \"kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373634 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373533 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373634 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373592 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373634 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373629 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373852 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373655 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373852 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.373975 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373960 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.374024 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.373996 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.375814 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.375788 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.376080 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.376061 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.382374 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.382357 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlpl\" (UniqueName: \"kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl\") pod \"rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.503105 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.503030 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:11:59.635235 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:11:59.635210 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:11:59.637560 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:11:59.637531 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1466e80_5c8a_49aa_bb33_aa7d4f8f9f9a.slice/crio-643ceab79c1eef821ac3ad3fb5a9c4dfe581e3fa98623a6b6e7a8daa9898ad7c WatchSource:0}: Error finding container 643ceab79c1eef821ac3ad3fb5a9c4dfe581e3fa98623a6b6e7a8daa9898ad7c: Status 404 returned error can't find the container with id 643ceab79c1eef821ac3ad3fb5a9c4dfe581e3fa98623a6b6e7a8daa9898ad7c Jul 23 02:12:00.131237 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:00.131201 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" event={"ID":"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a","Type":"ContainerStarted","Data":"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d"} Jul 23 02:12:00.131237 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:00.131242 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" event={"ID":"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a","Type":"ContainerStarted","Data":"643ceab79c1eef821ac3ad3fb5a9c4dfe581e3fa98623a6b6e7a8daa9898ad7c"} Jul 23 02:12:00.148388 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:00.148340 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" podStartSLOduration=1.148323976 podStartE2EDuration="1.148323976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:12:00.146945543 +0000 UTC m=+2589.926469472" watchObservedRunningTime="2026-07-23 02:12:00.148323976 +0000 UTC m=+2589.927847904" Jul 23 02:12:02.866560 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:02.866524 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:12:09.503763 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:09.503723 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:12:09.504168 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:09.503791 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:12:09.516561 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:09.516533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:12:10.191251 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:10.191226 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:12:26.362275 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:26.362198 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:12:42.473868 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:42.473834 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:12:57.172266 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:12:57.172232 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:13:04.735507 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:04.735473 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:13:06.623314 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.623282 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7724-140487957150512"] Jul 23 02:13:06.626415 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.626398 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:06.628867 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.628846 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqlmt\"" Jul 23 02:13:06.633821 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.633800 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957150512"] Jul 23 02:13:06.706427 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.706386 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4m5j\" (UniqueName: \"kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j\") pod \"test-curl-7724-140487957150512\" (UID: \"5a27a450-ccaa-46d4-89fa-6264ff06ef5d\") " pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:06.807376 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.807346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4m5j\" (UniqueName: \"kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j\") pod \"test-curl-7724-140487957150512\" (UID: \"5a27a450-ccaa-46d4-89fa-6264ff06ef5d\") " pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:06.821777 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.821751 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4m5j\" (UniqueName: \"kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j\") pod \"test-curl-7724-140487957150512\" (UID: \"5a27a450-ccaa-46d4-89fa-6264ff06ef5d\") " pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:06.936237 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:06.936127 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:07.082738 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:07.082711 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957150512"] Jul 23 02:13:07.084567 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:13:07.084537 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a27a450_ccaa_46d4_89fa_6264ff06ef5d.slice/crio-9f92ba5772dfa94d87bda2328a6819b5c838e842f806d674649d394768255c2c WatchSource:0}: Error finding container 9f92ba5772dfa94d87bda2328a6819b5c838e842f806d674649d394768255c2c: Status 404 returned error can't find the container with id 9f92ba5772dfa94d87bda2328a6819b5c838e842f806d674649d394768255c2c Jul 23 02:13:07.086187 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:07.086158 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 02:13:07.457439 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:07.457390 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957150512" event={"ID":"5a27a450-ccaa-46d4-89fa-6264ff06ef5d","Type":"ContainerStarted","Data":"9f92ba5772dfa94d87bda2328a6819b5c838e842f806d674649d394768255c2c"} Jul 23 02:13:08.463712 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:08.463676 2554 generic.go:356] "Generic (PLEG): container finished" podID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" containerID="08bbe6c679c835dc1028a700ba085c8d3fe636c2727c754684bd2caedfe65d7e" exitCode=0 Jul 23 02:13:08.463712 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:08.463717 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957150512" event={"ID":"5a27a450-ccaa-46d4-89fa-6264ff06ef5d","Type":"ContainerDied","Data":"08bbe6c679c835dc1028a700ba085c8d3fe636c2727c754684bd2caedfe65d7e"} Jul 23 02:13:09.601304 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.601277 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:09.633971 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.633943 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7724-140487957150512_5a27a450-ccaa-46d4-89fa-6264ff06ef5d/test-curl-7724-140487957150512/0.log" Jul 23 02:13:09.654773 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.654747 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7724-140487957150512"] Jul 23 02:13:09.658777 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.658755 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7724-140487957150512"] Jul 23 02:13:09.737219 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.737127 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4m5j\" (UniqueName: \"kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j\") pod \"5a27a450-ccaa-46d4-89fa-6264ff06ef5d\" (UID: \"5a27a450-ccaa-46d4-89fa-6264ff06ef5d\") " Jul 23 02:13:09.739244 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.739222 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j" (OuterVolumeSpecName: "kube-api-access-c4m5j") pod "5a27a450-ccaa-46d4-89fa-6264ff06ef5d" (UID: "5a27a450-ccaa-46d4-89fa-6264ff06ef5d"). InnerVolumeSpecName "kube-api-access-c4m5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:09.838035 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:09.838001 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4m5j\" (UniqueName: \"kubernetes.io/projected/5a27a450-ccaa-46d4-89fa-6264ff06ef5d-kube-api-access-c4m5j\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:10.474294 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:10.474265 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:10.474294 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:10.474291 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f92ba5772dfa94d87bda2328a6819b5c838e842f806d674649d394768255c2c" Jul 23 02:13:10.484048 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:13:10.484019 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7724-140487957150512\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:10.767592 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:13:10.767497 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7724-140487957150512\" is forbidden: User \"system:node:ip-10-0-136-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-186.ec2.internal' and this object" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" pod="opendatahub/test-curl-7724-140487957150512" Jul 23 02:13:10.768825 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:10.768798 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" path="/var/lib/kubelet/pods/5a27a450-ccaa-46d4-89fa-6264ff06ef5d/volumes" Jul 23 02:13:12.665519 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:12.665480 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:13:18.778476 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.778449 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:18.778950 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.778801 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" containerName="test-curl-7724-140487957150512" Jul 23 02:13:18.778950 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.778813 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" containerName="test-curl-7724-140487957150512" Jul 23 02:13:18.778950 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.778890 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a27a450-ccaa-46d4-89fa-6264ff06ef5d" containerName="test-curl-7724-140487957150512" Jul 23 02:13:18.782129 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.782114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:18.784551 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.784529 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqlmt\"" Jul 23 02:13:18.787563 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.787544 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:18.808847 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.808827 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvp8\" (UniqueName: \"kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8\") pod \"test-curl-7724-140487957268784\" (UID: \"101d667e-1898-4963-a654-1120f841274b\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:18.909371 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.909343 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvp8\" (UniqueName: \"kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8\") pod \"test-curl-7724-140487957268784\" (UID: \"101d667e-1898-4963-a654-1120f841274b\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:18.918029 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:18.918008 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvp8\" (UniqueName: \"kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8\") pod \"test-curl-7724-140487957268784\" (UID: \"101d667e-1898-4963-a654-1120f841274b\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:19.091973 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:19.091948 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:19.426965 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:19.426936 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:19.429323 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:13:19.429295 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod101d667e_1898_4963_a654_1120f841274b.slice/crio-ae044ea93cc76b2179f8898637fb294bda3667c6e9d0408c054c3b7b62e0a619 WatchSource:0}: Error finding container ae044ea93cc76b2179f8898637fb294bda3667c6e9d0408c054c3b7b62e0a619: Status 404 returned error can't find the container with id ae044ea93cc76b2179f8898637fb294bda3667c6e9d0408c054c3b7b62e0a619 Jul 23 02:13:19.523354 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:19.523326 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957268784" event={"ID":"101d667e-1898-4963-a654-1120f841274b","Type":"ContainerStarted","Data":"ae044ea93cc76b2179f8898637fb294bda3667c6e9d0408c054c3b7b62e0a619"} Jul 23 02:13:19.755373 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:19.755336 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" podUID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" containerName="main" containerID="cri-o://720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d" gracePeriod=45 Jul 23 02:13:19.997839 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:19.997818 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:13:20.019967 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.019943 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnlpl\" (UniqueName: \"kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020109 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.019991 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020109 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020020 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020109 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020072 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020272 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020140 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020272 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020186 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir\") pod \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\" (UID: \"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a\") " Jul 23 02:13:20.020716 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020337 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:13:20.020716 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020408 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home" (OuterVolumeSpecName: "home") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:13:20.020716 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020528 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:13:20.020716 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020568 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-model-cache\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.020716 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.020587 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-home\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.022119 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.022092 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl" (OuterVolumeSpecName: "kube-api-access-wnlpl") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "kube-api-access-wnlpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:20.022264 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.022140 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:13:20.022443 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.022414 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm" (OuterVolumeSpecName: "dshm") pod "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" (UID: "b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 02:13:20.121645 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.121603 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tls-certs\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.121781 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.121660 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-tmp-dir\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.121781 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.121671 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnlpl\" (UniqueName: \"kubernetes.io/projected/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-kube-api-access-wnlpl\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.121781 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.121682 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a-dshm\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:20.529386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.529350 2554 generic.go:356] "Generic (PLEG): container finished" podID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" containerID="720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d" exitCode=0 Jul 23 02:13:20.529552 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.529395 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" Jul 23 02:13:20.529552 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.529498 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" event={"ID":"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a","Type":"ContainerDied","Data":"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d"} Jul 23 02:13:20.529690 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.529555 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6" event={"ID":"b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a","Type":"ContainerDied","Data":"643ceab79c1eef821ac3ad3fb5a9c4dfe581e3fa98623a6b6e7a8daa9898ad7c"} Jul 23 02:13:20.529690 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.529610 2554 scope.go:117] "RemoveContainer" containerID="720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d" Jul 23 02:13:20.531377 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.531349 2554 generic.go:356] "Generic (PLEG): container finished" podID="101d667e-1898-4963-a654-1120f841274b" containerID="606db911498859045118a86a027829b52de1a7af8af42b359add27898f8d7edb" exitCode=0 Jul 23 02:13:20.531494 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.531419 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957268784" event={"ID":"101d667e-1898-4963-a654-1120f841274b","Type":"ContainerDied","Data":"606db911498859045118a86a027829b52de1a7af8af42b359add27898f8d7edb"} Jul 23 02:13:20.540602 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.540585 2554 scope.go:117] "RemoveContainer" containerID="720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d" Jul 23 02:13:20.540840 ip-10-0-136-186 kubenswrapper[2554]: E0723 02:13:20.540820 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d\": container with ID starting with 720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d not found: ID does not exist" containerID="720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d" Jul 23 02:13:20.540897 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.540847 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d"} err="failed to get container status \"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d\": rpc error: code = NotFound desc = could not find container \"720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d\": container with ID starting with 720db730b1deb4e3bf892b64cbbc9aac20edf5a4131f10c37dbe38a1d797bd9d not found: ID does not exist" Jul 23 02:13:20.558281 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.558261 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:13:20.564023 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.564004 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1dd296/rate-test-model-1dd296-kserve-668f6c6fcf-xq2q6"] Jul 23 02:13:20.769745 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:20.769712 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" path="/var/lib/kubelet/pods/b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a/volumes" Jul 23 02:13:21.670503 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.670480 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:21.689909 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.689885 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7724-140487957268784_101d667e-1898-4963-a654-1120f841274b/test-curl-7724-140487957268784/0.log" Jul 23 02:13:21.712383 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.712358 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:21.714039 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.714022 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:21.736863 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.736837 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kvp8\" (UniqueName: \"kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8\") pod \"101d667e-1898-4963-a654-1120f841274b\" (UID: \"101d667e-1898-4963-a654-1120f841274b\") " Jul 23 02:13:21.738718 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.738697 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8" (OuterVolumeSpecName: "kube-api-access-9kvp8") pod "101d667e-1898-4963-a654-1120f841274b" (UID: "101d667e-1898-4963-a654-1120f841274b"). InnerVolumeSpecName "kube-api-access-9kvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:21.837436 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.837407 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvp8\" (UniqueName: \"kubernetes.io/projected/101d667e-1898-4963-a654-1120f841274b-kube-api-access-9kvp8\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:21.837970 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.837948 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:21.838359 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838346 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101d667e-1898-4963-a654-1120f841274b" containerName="test-curl-7724-140487957268784" Jul 23 02:13:21.838394 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838362 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="101d667e-1898-4963-a654-1120f841274b" containerName="test-curl-7724-140487957268784" Jul 23 02:13:21.838394 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838372 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" containerName="main" Jul 23 02:13:21.838394 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838378 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" containerName="main" Jul 23 02:13:21.838487 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838444 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1466e80-5c8a-49aa-bb33-aa7d4f8f9f9a" containerName="main" Jul 23 02:13:21.838487 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.838453 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="101d667e-1898-4963-a654-1120f841274b" containerName="test-curl-7724-140487957268784" Jul 23 02:13:21.841350 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.841336 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:21.845357 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.845332 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:21.938507 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:21.938475 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhnd\" (UniqueName: \"kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd\") pod \"test-curl-7724-140487957268784\" (UID: \"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:22.039037 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.039002 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhnd\" (UniqueName: \"kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd\") pod \"test-curl-7724-140487957268784\" (UID: \"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:22.047589 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.047567 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhnd\" (UniqueName: \"kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd\") pod \"test-curl-7724-140487957268784\" (UID: \"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f\") " pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:22.151384 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.151312 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:22.275798 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.275758 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:22.278669 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:13:22.278641 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68fe641e_918c_4e3c_9efa_4ab6cbef3a0f.slice/crio-c11fb331e21b32d11d8e62f90b4f9757b79250e7389085e9075404a367f8c943 WatchSource:0}: Error finding container c11fb331e21b32d11d8e62f90b4f9757b79250e7389085e9075404a367f8c943: Status 404 returned error can't find the container with id c11fb331e21b32d11d8e62f90b4f9757b79250e7389085e9075404a367f8c943 Jul 23 02:13:22.543979 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.543950 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:22.543979 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.543966 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae044ea93cc76b2179f8898637fb294bda3667c6e9d0408c054c3b7b62e0a619" Jul 23 02:13:22.545152 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.545132 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957268784" event={"ID":"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f","Type":"ContainerStarted","Data":"c11fb331e21b32d11d8e62f90b4f9757b79250e7389085e9075404a367f8c943"} Jul 23 02:13:22.559836 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.559799 2554 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7724-140487957268784" oldPodUID="101d667e-1898-4963-a654-1120f841274b" podUID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" Jul 23 02:13:22.774351 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:22.770813 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101d667e-1898-4963-a654-1120f841274b" path="/var/lib/kubelet/pods/101d667e-1898-4963-a654-1120f841274b/volumes" Jul 23 02:13:23.552218 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:23.552164 2554 generic.go:356] "Generic (PLEG): container finished" podID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" containerID="70f95543019a1a1fdac9e9b7edff39fa36067e5b09c3d39f5e5a1a443abba88b" exitCode=0 Jul 23 02:13:23.552391 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:23.552246 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7724-140487957268784" event={"ID":"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f","Type":"ContainerDied","Data":"70f95543019a1a1fdac9e9b7edff39fa36067e5b09c3d39f5e5a1a443abba88b"} Jul 23 02:13:24.687611 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.687590 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:24.708969 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.708941 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7724-140487957268784_68fe641e-918c-4e3c-9efa-4ab6cbef3a0f/test-curl-7724-140487957268784/0.log" Jul 23 02:13:24.731364 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.731339 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:24.736573 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.736553 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7724-140487957268784"] Jul 23 02:13:24.763323 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.763295 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dhnd\" (UniqueName: \"kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd\") pod \"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f\" (UID: \"68fe641e-918c-4e3c-9efa-4ab6cbef3a0f\") " Jul 23 02:13:24.765361 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.765334 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd" (OuterVolumeSpecName: "kube-api-access-9dhnd") pod "68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" (UID: "68fe641e-918c-4e3c-9efa-4ab6cbef3a0f"). InnerVolumeSpecName "kube-api-access-9dhnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:24.769521 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.769500 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" path="/var/lib/kubelet/pods/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f/volumes" Jul 23 02:13:24.864370 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.864346 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhnd\" (UniqueName: \"kubernetes.io/projected/68fe641e-918c-4e3c-9efa-4ab6cbef3a0f-kube-api-access-9dhnd\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:24.928414 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.928389 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5"] Jul 23 02:13:24.928876 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.928855 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" containerName="test-curl-7724-140487957268784" Jul 23 02:13:24.928922 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.928881 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" containerName="test-curl-7724-140487957268784" Jul 23 02:13:24.928984 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.928970 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fe641e-918c-4e3c-9efa-4ab6cbef3a0f" containerName="test-curl-7724-140487957268784" Jul 23 02:13:24.932054 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.932038 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:24.934321 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.934302 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:13:24.934424 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.934391 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 02:13:24.938238 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.938214 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5"] Jul 23 02:13:24.965744 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:24.965719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5\" (UID: \"e180f068-b4f2-41c8-9a51-9c3a114b94f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:25.067261 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.067228 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5\" (UID: \"e180f068-b4f2-41c8-9a51-9c3a114b94f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:25.067784 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.067765 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5\" (UID: \"e180f068-b4f2-41c8-9a51-9c3a114b94f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:25.244241 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.244153 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:25.564267 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.564244 2554 scope.go:117] "RemoveContainer" containerID="70f95543019a1a1fdac9e9b7edff39fa36067e5b09c3d39f5e5a1a443abba88b" Jul 23 02:13:25.564267 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.564261 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7724-140487957268784" Jul 23 02:13:25.578009 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:25.577990 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5"] Jul 23 02:13:25.580116 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:13:25.580079 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode180f068_b4f2_41c8_9a51_9c3a114b94f9.slice/crio-ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0 WatchSource:0}: Error finding container ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0: Status 404 returned error can't find the container with id ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0 Jul 23 02:13:26.571456 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:26.571364 2554 generic.go:356] "Generic (PLEG): container finished" podID="e180f068-b4f2-41c8-9a51-9c3a114b94f9" containerID="e48dccb7fee077d04a0576b65d69d6b6300aeb2ef9a3c10e545cbe74aef22335" exitCode=0 Jul 23 02:13:26.571456 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:26.571427 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" event={"ID":"e180f068-b4f2-41c8-9a51-9c3a114b94f9","Type":"ContainerDied","Data":"e48dccb7fee077d04a0576b65d69d6b6300aeb2ef9a3c10e545cbe74aef22335"} Jul 23 02:13:26.571878 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:26.571466 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" event={"ID":"e180f068-b4f2-41c8-9a51-9c3a114b94f9","Type":"ContainerStarted","Data":"ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0"} Jul 23 02:13:27.706090 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:27.706064 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:27.792120 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:27.792082 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca\") pod \"e180f068-b4f2-41c8-9a51-9c3a114b94f9\" (UID: \"e180f068-b4f2-41c8-9a51-9c3a114b94f9\") " Jul 23 02:13:27.792436 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:27.792415 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e180f068-b4f2-41c8-9a51-9c3a114b94f9" (UID: "e180f068-b4f2-41c8-9a51-9c3a114b94f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:13:27.893629 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:27.893563 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e180f068-b4f2-41c8-9a51-9c3a114b94f9-maas-api-service-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:28.582277 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:28.582244 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" event={"ID":"e180f068-b4f2-41c8-9a51-9c3a114b94f9","Type":"ContainerDied","Data":"ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0"} Jul 23 02:13:28.582277 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:28.582268 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5" Jul 23 02:13:28.582502 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:28.582290 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef82e5ec3478e9b32027ab6547c791630c630c76a9db2d43ccf1922c8f6d27b0" Jul 23 02:13:39.374259 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:39.374222 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:13:40.040267 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:40.040215 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5"] Jul 23 02:13:40.042056 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:40.042027 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1dd296-7pxb5"] Jul 23 02:13:40.768781 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:40.768747 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e180f068-b4f2-41c8-9a51-9c3a114b94f9" path="/var/lib/kubelet/pods/e180f068-b4f2-41c8-9a51-9c3a114b94f9/volumes" Jul 23 02:13:47.143295 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.143214 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm"] Jul 23 02:13:47.143801 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.143786 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e180f068-b4f2-41c8-9a51-9c3a114b94f9" containerName="revoke-keys" Jul 23 02:13:47.143862 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.143805 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e180f068-b4f2-41c8-9a51-9c3a114b94f9" containerName="revoke-keys" Jul 23 02:13:47.143931 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.143920 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e180f068-b4f2-41c8-9a51-9c3a114b94f9" containerName="revoke-keys" Jul 23 02:13:47.149540 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.149516 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:47.152162 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.152137 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 02:13:47.152288 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.152193 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:13:47.153880 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.153857 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm"] Jul 23 02:13:47.268011 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.267976 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm\" (UID: \"195a4e87-0fc7-4a88-b3c0-574082a23c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:47.368675 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.368636 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm\" (UID: \"195a4e87-0fc7-4a88-b3c0-574082a23c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:47.369285 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.369265 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm\" (UID: \"195a4e87-0fc7-4a88-b3c0-574082a23c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:47.463120 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.463029 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:47.601514 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.601493 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm"] Jul 23 02:13:47.604624 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:13:47.604594 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195a4e87_0fc7_4a88_b3c0_574082a23c31.slice/crio-a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb WatchSource:0}: Error finding container a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb: Status 404 returned error can't find the container with id a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb Jul 23 02:13:47.681130 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:47.681099 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" event={"ID":"195a4e87-0fc7-4a88-b3c0-574082a23c31","Type":"ContainerStarted","Data":"a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb"} Jul 23 02:13:48.687454 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:48.687419 2554 generic.go:356] "Generic (PLEG): container finished" podID="195a4e87-0fc7-4a88-b3c0-574082a23c31" containerID="da7e36091d89984b9383e0e289698c0fe2fa06286b3275a4e0f9908f8172db73" exitCode=0 Jul 23 02:13:48.688004 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:48.687478 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" event={"ID":"195a4e87-0fc7-4a88-b3c0-574082a23c31","Type":"ContainerDied","Data":"da7e36091d89984b9383e0e289698c0fe2fa06286b3275a4e0f9908f8172db73"} Jul 23 02:13:49.824572 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:49.824546 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:49.993846 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:49.993757 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca\") pod \"195a4e87-0fc7-4a88-b3c0-574082a23c31\" (UID: \"195a4e87-0fc7-4a88-b3c0-574082a23c31\") " Jul 23 02:13:49.994098 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:49.994075 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "195a4e87-0fc7-4a88-b3c0-574082a23c31" (UID: "195a4e87-0fc7-4a88-b3c0-574082a23c31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:13:50.094546 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:50.094510 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a4e87-0fc7-4a88-b3c0-574082a23c31-maas-api-service-ca\") on node \"ip-10-0-136-186.ec2.internal\" DevicePath \"\"" Jul 23 02:13:50.697799 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:50.697769 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" Jul 23 02:13:50.697976 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:50.697805 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm" event={"ID":"195a4e87-0fc7-4a88-b3c0-574082a23c31","Type":"ContainerDied","Data":"a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb"} Jul 23 02:13:50.697976 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:50.697859 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71efcb8deea6d5adc3cbcfbe34ea21f3d55597ef4780eec663cea822913e3eb" Jul 23 02:13:50.942830 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:13:50.942799 2554 scope.go:117] "RemoveContainer" containerID="df6704fc499ecdd44781b374ab011fac8bd64e536559ca979054b4212551a677" Jul 23 02:14:00.468692 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:00.468661 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l8zbz"] Jul 23 02:14:02.237977 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:02.237940 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm"] Jul 23 02:14:02.240449 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:02.240423 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5fbcb9-pkgjm"] Jul 23 02:14:02.769437 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:02.769397 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195a4e87-0fc7-4a88-b3c0-574082a23c31" path="/var/lib/kubelet/pods/195a4e87-0fc7-4a88-b3c0-574082a23c31/volumes" Jul 23 02:14:16.201038 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:16.201006 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq_f36abc8b-5a11-448d-88ee-3d70a9db2c22/copy-manifests/0.log" Jul 23 02:14:16.207655 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:16.207628 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq_f36abc8b-5a11-448d-88ee-3d70a9db2c22/manager/0.log" Jul 23 02:14:19.103361 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:19.103333 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zh57l_dbb26eba-8089-4067-b0b2-082f6e5e06b9/manager/0.log" Jul 23 02:14:19.218217 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:19.218190 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l8zbz_7f9bc72a-4e2c-481b-96a6-d494f0796538/limitador/0.log" Jul 23 02:14:19.963279 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:19.963247 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-6wgbt_b5d66029-e76e-4d2f-9c5a-80d51243a316/kube-auth-proxy/0.log" Jul 23 02:14:21.418737 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:21.418707 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xchvr_7affb312-1890-469c-b25d-acf9dc49bc25/main/0.log" Jul 23 02:14:21.531347 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:21.531316 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xjvgm_06f82281-18c2-4760-bb30-edfce31c015e/main/0.log" Jul 23 02:14:30.857779 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:30.857745 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6qtsd_3fe3efc5-d6e1-44e4-9f63-ff1cff1e4e4e/global-pull-secret-syncer/0.log" Jul 23 02:14:31.039124 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:31.039094 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-85j9x_266c9325-3995-4e57-9222-e616ba4721ed/konnectivity-agent/0.log" Jul 23 02:14:31.138622 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:31.138529 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-186.ec2.internal_233b0e2a1e1f48ac6102330b8172b615/haproxy/0.log" Jul 23 02:14:36.713028 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:36.712993 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zh57l_dbb26eba-8089-4067-b0b2-082f6e5e06b9/manager/0.log" Jul 23 02:14:36.732700 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:36.732671 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l8zbz_7f9bc72a-4e2c-481b-96a6-d494f0796538/limitador/0.log" Jul 23 02:14:38.342742 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.342712 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/alertmanager/0.log" Jul 23 02:14:38.378633 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.378609 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/config-reloader/0.log" Jul 23 02:14:38.398036 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.398014 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/kube-rbac-proxy-web/0.log" Jul 23 02:14:38.418108 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.418091 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/kube-rbac-proxy/0.log" Jul 23 02:14:38.437428 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.437408 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/kube-rbac-proxy-metric/0.log" Jul 23 02:14:38.455977 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.455959 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/prom-label-proxy/0.log" Jul 23 02:14:38.476315 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.476297 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_339b2ae1-782e-408e-8957-6eb5b8116b61/init-config-reloader/0.log" Jul 23 02:14:38.590731 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.590701 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg5h_20020c4c-bfb4-404b-9efe-fbd15bed17e8/kube-state-metrics/0.log" Jul 23 02:14:38.639614 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.639539 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg5h_20020c4c-bfb4-404b-9efe-fbd15bed17e8/kube-rbac-proxy-main/0.log" Jul 23 02:14:38.676695 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.676667 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg5h_20020c4c-bfb4-404b-9efe-fbd15bed17e8/kube-rbac-proxy-self/0.log" Jul 23 02:14:38.717761 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.717739 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c5db6549d-cgvbw_c25f4676-44eb-492e-8fef-de08d5bac273/metrics-server/0.log" Jul 23 02:14:38.745337 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.745311 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-rpbdh_91cfa4ae-e721-43d8-974a-c645889858a0/monitoring-plugin/0.log" Jul 23 02:14:38.929522 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.929439 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrvrm_da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023/node-exporter/0.log" Jul 23 02:14:38.948231 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.948211 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrvrm_da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023/kube-rbac-proxy/0.log" Jul 23 02:14:38.970447 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.970426 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrvrm_da2a0ccb-f8b6-4f12-b7d9-3231e1c2b023/init-textfile/0.log" Jul 23 02:14:38.993509 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:38.993486 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-t65hn_730ed4ae-093f-444e-92ee-1e8c65dfe798/kube-rbac-proxy-main/0.log" Jul 23 02:14:39.015739 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.015715 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-t65hn_730ed4ae-093f-444e-92ee-1e8c65dfe798/kube-rbac-proxy-self/0.log" Jul 23 02:14:39.038212 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.038162 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-t65hn_730ed4ae-093f-444e-92ee-1e8c65dfe798/openshift-state-metrics/0.log" Jul 23 02:14:39.079319 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.079291 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/prometheus/0.log" Jul 23 02:14:39.100166 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.100128 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/config-reloader/0.log" Jul 23 02:14:39.119779 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.119756 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/thanos-sidecar/0.log" Jul 23 02:14:39.138459 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.138436 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/kube-rbac-proxy-web/0.log" Jul 23 02:14:39.159423 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.159404 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/kube-rbac-proxy/0.log" Jul 23 02:14:39.180509 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.180443 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/kube-rbac-proxy-thanos/0.log" Jul 23 02:14:39.200252 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.200228 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78f591f9-f4b4-420c-bf53-1c2ff18d21a8/init-config-reloader/0.log" Jul 23 02:14:39.233825 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.233799 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-xb7m2_f3870bcb-4e65-44fd-bad0-dea55260755a/prometheus-operator/0.log" Jul 23 02:14:39.248701 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.248682 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-xb7m2_f3870bcb-4e65-44fd-bad0-dea55260755a/kube-rbac-proxy/0.log" Jul 23 02:14:39.301158 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.301141 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765848f968-j4mrz_c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab/telemeter-client/0.log" Jul 23 02:14:39.324875 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.324853 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765848f968-j4mrz_c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab/reload/0.log" Jul 23 02:14:39.344157 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.344139 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765848f968-j4mrz_c01f027f-c2bd-4c27-a7a0-2b6e6c80d3ab/kube-rbac-proxy/0.log" Jul 23 02:14:39.386101 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.386081 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/thanos-query/0.log" Jul 23 02:14:39.406946 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.406928 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/kube-rbac-proxy-web/0.log" Jul 23 02:14:39.426336 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.426319 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/kube-rbac-proxy/0.log" Jul 23 02:14:39.452515 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.452460 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/prom-label-proxy/0.log" Jul 23 02:14:39.473766 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.473746 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/kube-rbac-proxy-rules/0.log" Jul 23 02:14:39.494216 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:39.494200 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665f54d6bd-vwc8s_b6037316-e72e-4649-8ca2-d156f1964464/kube-rbac-proxy-metrics/0.log" Jul 23 02:14:40.656818 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.656787 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86"] Jul 23 02:14:40.657235 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.657169 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195a4e87-0fc7-4a88-b3c0-574082a23c31" containerName="revoke-keys" Jul 23 02:14:40.657235 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.657193 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="195a4e87-0fc7-4a88-b3c0-574082a23c31" containerName="revoke-keys" Jul 23 02:14:40.657327 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.657258 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="195a4e87-0fc7-4a88-b3c0-574082a23c31" containerName="revoke-keys" Jul 23 02:14:40.660595 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.660572 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.663126 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.663108 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54hzb\"/\"openshift-service-ca.crt\"" Jul 23 02:14:40.664096 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.664079 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54hzb\"/\"kube-root-ca.crt\"" Jul 23 02:14:40.664187 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.664080 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-54hzb\"/\"default-dockercfg-nz4lg\"" Jul 23 02:14:40.669120 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.669101 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86"] Jul 23 02:14:40.845077 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.845043 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-podres\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.845077 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.845091 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-proc\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.845326 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.845136 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-sys\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.845326 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.845210 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-lib-modules\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.845326 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.845282 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbz6\" (UniqueName: \"kubernetes.io/projected/6595412e-2342-4b80-8577-e384beaa8875-kube-api-access-ltbz6\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946152 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-sys\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946152 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946109 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-lib-modules\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbz6\" (UniqueName: \"kubernetes.io/projected/6595412e-2342-4b80-8577-e384beaa8875-kube-api-access-ltbz6\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946211 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-sys\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-podres\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946278 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-proc\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946319 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-lib-modules\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946386 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946367 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-proc\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.946582 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.946392 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6595412e-2342-4b80-8577-e384beaa8875-podres\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.954487 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.954457 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbz6\" (UniqueName: \"kubernetes.io/projected/6595412e-2342-4b80-8577-e384beaa8875-kube-api-access-ltbz6\") pod \"perf-node-gather-daemonset-jnw86\" (UID: \"6595412e-2342-4b80-8577-e384beaa8875\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:40.971513 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:40.971488 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:41.099574 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:41.099516 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86"] Jul 23 02:14:41.102119 ip-10-0-136-186 kubenswrapper[2554]: W0723 02:14:41.102096 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6595412e_2342_4b80_8577_e384beaa8875.slice/crio-e9c37b4641badd85d279766ed075b4d7de66c3f9124cdecb2c94f5aaa938d162 WatchSource:0}: Error finding container e9c37b4641badd85d279766ed075b4d7de66c3f9124cdecb2c94f5aaa938d162: Status 404 returned error can't find the container with id e9c37b4641badd85d279766ed075b4d7de66c3f9124cdecb2c94f5aaa938d162 Jul 23 02:14:41.936999 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:41.936965 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" event={"ID":"6595412e-2342-4b80-8577-e384beaa8875","Type":"ContainerStarted","Data":"053fe3618f61c03b88bf35823f1a149f6477437cfed3a8a71871444f1a3254da"} Jul 23 02:14:41.937432 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:41.937018 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" event={"ID":"6595412e-2342-4b80-8577-e384beaa8875","Type":"ContainerStarted","Data":"e9c37b4641badd85d279766ed075b4d7de66c3f9124cdecb2c94f5aaa938d162"} Jul 23 02:14:41.937432 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:41.937061 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:41.953928 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:41.953880 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" podStartSLOduration=1.953864279 podStartE2EDuration="1.953864279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 02:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:14:41.95177152 +0000 UTC m=+2751.731295451" watchObservedRunningTime="2026-07-23 02:14:41.953864279 +0000 UTC m=+2751.733388207" Jul 23 02:14:43.536470 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:43.536443 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5tgvg_ae7f258d-cdf1-42a0-8783-3b9fc4c80f63/dns/0.log" Jul 23 02:14:43.555658 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:43.555635 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5tgvg_ae7f258d-cdf1-42a0-8783-3b9fc4c80f63/kube-rbac-proxy/0.log" Jul 23 02:14:43.627916 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:43.627890 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dh5gm_88247bad-8b16-464f-8958-de0736e36ee2/dns-node-resolver/0.log" Jul 23 02:14:44.242335 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:44.242307 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2x8kp_7032703b-c778-409b-a540-5d373902d38e/node-ca/0.log" Jul 23 02:14:45.332200 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:45.332152 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-6wgbt_b5d66029-e76e-4d2f-9c5a-80d51243a316/kube-auth-proxy/0.log" Jul 23 02:14:46.215260 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:46.215230 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xj2xk_9d61d55c-06e0-4453-b29d-c95619694ee9/serve-healthcheck-canary/0.log" Jul 23 02:14:46.775285 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:46.775255 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhd9_ca1c65a8-d97e-4c96-8910-bf8bd928063e/kube-rbac-proxy/0.log" Jul 23 02:14:46.794812 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:46.794784 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhd9_ca1c65a8-d97e-4c96-8910-bf8bd928063e/exporter/0.log" Jul 23 02:14:46.815373 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:46.815338 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhd9_ca1c65a8-d97e-4c96-8910-bf8bd928063e/extractor/0.log" Jul 23 02:14:47.951233 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:47.951207 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-jnw86" Jul 23 02:14:49.516372 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:49.516339 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq_f36abc8b-5a11-448d-88ee-3d70a9db2c22/manager/0.log" Jul 23 02:14:49.542709 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:49.542684 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-ht5lq_f36abc8b-5a11-448d-88ee-3d70a9db2c22/copy-manifests/0.log" Jul 23 02:14:51.000429 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:51.000344 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7564d6dfb8-cv8pt_42972945-0cf2-4f9f-bd40-1c9db6018e89/manager/0.log" Jul 23 02:14:51.053450 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:51.053423 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ln5bj_5f25e40d-4fb3-4ca6-b5cd-063f236430f0/openshift-lws-operator/0.log" Jul 23 02:14:51.668798 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:51.668770 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-6wgbt_b5d66029-e76e-4d2f-9c5a-80d51243a316/kube-auth-proxy/0.log" Jul 23 02:14:53.074661 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:53.074629 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-6wgbt_b5d66029-e76e-4d2f-9c5a-80d51243a316/kube-auth-proxy/0.log" Jul 23 02:14:57.803842 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.803816 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/kube-multus-additional-cni-plugins/0.log" Jul 23 02:14:57.823234 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.823213 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/egress-router-binary-copy/0.log" Jul 23 02:14:57.843019 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.842954 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/cni-plugins/0.log" Jul 23 02:14:57.861041 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.861017 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/bond-cni-plugin/0.log" Jul 23 02:14:57.881621 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.881601 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/routeoverride-cni/0.log" Jul 23 02:14:57.899755 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.899736 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/whereabouts-cni-bincopy/0.log" Jul 23 02:14:57.923268 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:57.923251 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv78p_a0df5ea7-1285-44e5-894d-c4ecbdfac36b/whereabouts-cni/0.log" Jul 23 02:14:58.156755 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:58.156676 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jnsg2_2e49321b-6027-4071-8cae-7ab0f3aea156/kube-multus/0.log" Jul 23 02:14:58.256588 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:58.256555 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rbvvn_c44e022b-202a-48e4-8c9f-23ef0751decf/network-metrics-daemon/0.log" Jul 23 02:14:58.272102 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:58.272069 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rbvvn_c44e022b-202a-48e4-8c9f-23ef0751decf/kube-rbac-proxy/0.log" Jul 23 02:14:59.347400 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:59.347360 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wsn_1694d2d6-8f82-4d67-85fb-8d40817f9992/ovn-controller/0.log" Jul 23 02:14:59.363046 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:59.363021 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wsn_1694d2d6-8f82-4d67-85fb-8d40817f9992/ovn-acl-logging/0.log" Jul 23 02:14:59.384772 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:59.384722 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wsn_1694d2d6-8f82-4d67-85fb-8d40817f9992/kube-rbac-proxy-node/0.log" Jul 23 02:14:59.406849 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:59.406829 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wsn_1694d2d6-8f82-4d67-85fb-8d40817f9992/kube-rbac-proxy-ovn-metrics/0.log" Jul 23 02:14:59.428438 ip-10-0-136-186 kubenswrapper[2554]: I0723 02:14:59.428410 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wsn_1694d2d6-8f82-4d67-85fb-8d40817f9992/northd/0.log"